## Why
Compaction analytics adds retained image count and compaction summary
output tokens for v1.5 specifically.
## What changed
- Add nullable `retained_image_count` and `compaction_summary_tokens`
fields to `codex_compaction_event`.
- Populate them only for `responses_compaction_v2`: retained images come
from the retained v2 compacted history, and summary tokens come from
`response.completed.token_usage.output_tokens`.
- Leave local and legacy remote compaction events as `null` for these
detail fields.
## Verification
- `just fmt`
- `just fix -p codex-core`
- `just test -p codex-core
build_v2_compacted_history_counts_retained_input_images`
- `git diff --check`
## Summary
- accept non-empty model-defined reasoning effort values while
preserving built-in effort behavior
- propagate the non-Copy effort type through core, app-server, TUI,
telemetry, and persistence call sites
- preserve string wire encoding and expose an open-string schema for
clients
- update model selection and shortcut behavior for model-advertised
effort values
## Root cause
`ReasoningEffort` gained a string-backed custom variant, so it could no
longer implement `Copy` or rely on derived closed-enum serialization.
Existing consumers still moved effort values from shared references and
assumed a fixed built-in value set.
## Validation
- `just fmt`
- Local tests and compilation were not run per request; relying on CI.
## Why
The Responses websocket client no longer needs to send a follow-up
`response.processed` request after a turn response has already been
recorded. Keeping that extra acknowledgement path adds feature-gated
control flow and a second websocket request shape that no longer carries
useful behavior.
## What Changed
- Removed the `response.processed` websocket request type and sender.
- Removed the `responses_websocket_response_processed` feature flag and
schema entry.
- Removed turn and remote-compaction plumbing that only tracked response
IDs to send the acknowledgement.
- Removed tests that existed solely to cover the deleted feature path.
## Validation
- `just fix -p codex-core -p codex-api -p codex-features`
## Why
Compaction analytics need token counts that better represent the request
being compacted. The existing session snapshot can diverge from the
actual remote compaction request after output rewriting, and remote v2
can use server-side Responses usage when available.
## What changed
- Add an optional `active_context_tokens_before` override to
`CompactionAnalyticsAttempt::track(...)` for remote compaction when it
has a better before-token value than the begin-time session snapshot.
The local `/compact` path passes no override.
- For remote v1 `responses_compact`, subtract the estimated token delta
from pre-compaction output rewriting from the session snapshot, capped
by locally-added tokens since the last successful API response.
- For remote v2 `responses_compaction_v2`, use the same bounded
output-rewrite fallback as remote v1, then overwrite
`active_context_tokens_before` with server `token_usage.input_tokens`
from the `response.completed` event when present.
- Keep the existing v2 compaction-output validation while carrying the
completed response token usage through `collect_compaction_output`.
## Verification
- `just fmt`
- `just test -p codex-core
collect_compaction_output_accepts_additional_output_items`
- `git diff --check`
## Why
When trying to fit history under compaction limit rewrite output items
instead of removing them entirely. Otherwise we're breaking
incrementality in relation to the previous response.
## Summary
- add analytics-only `CodexErr` telemetry to `codex_turn_event` while
leaving existing `turn_error` unchanged
- record terminal `CodexErr` facts from core immediately before the
existing turn error event is sent
- emit source-truth `codex_error_*` fields for downstream analytics,
including the raw `CodexErr::InvalidRequest(String)` message as
`codex_error_subreason`
## Validation
- `just test -p codex-analytics`
- attempted `just test -p codex-core`, but the local run timed out
across unrelated integration suites in this environment and is not being
used as validation
## Summary
- Add `request_kind` values for foreground turn, startup prewarm,
compaction, and detached memory model requests.
- Attach compaction dispatch metadata to local Responses, legacy
`/v1/responses/compact`, and remote v2 compact requests.
- Add the existing logical context-window identifier as `window_id` on
turn-owned model request metadata.
- Keep identity fields optional for detached memory requests, while
still emitting `request_kind="memory"` in non-git/no-sandbox workspaces.
## Root Cause
`x-codex-turn-metadata` has more than one producer. Foreground turns and
compaction requests own a real turn and should carry that turn identity.
Detached memory stage-one requests do not own a foreground turn, so
absent identity fields are valid rather than missing data. Startup
websocket prewarm is also a model request, but it has `generate=false`
and must not be counted as a foreground turn.
`thread_source` or session source identifies where a thread came from
(for example review, guardian, or another subagent). `request_kind`
identifies what the current outbound model request is doing (`turn`,
`prewarm`, `compaction`, or `memory`). A review or guardian thread can
issue either a normal turn request or a compaction request, so source
cannot replace request kind.
## Behavior / Impact
- Ordinary foreground requests send `request_kind="turn"`, their real
identity fields, and `window_id="<thread_id>:<window_generation>"`.
- Startup websocket warmup requests send `request_kind="prewarm"` so
they are not counted as foreground turns.
- Compaction requests send `request_kind="compaction"`, their real
owning turn identity, the existing `window_id`, and
`compaction.{trigger,reason,implementation,phase,strategy}`.
- Detached memory stage-one requests send `request_kind="memory"`
without `session_id`, `thread_id`, `turn_id`, or `window_id`; when no
workspace metadata exists, the kind-only header is still emitted.
- `session_id`, `thread_id`, `turn_id`, and `window_id` remain optional
in the header schema because detached memory requests do not own a
foreground turn or context window.
- `window_id` is not a new ID system: it is copied from the already-sent
`x-codex-window-id` / WS client metadata value at model-request dispatch
time.
- Existing `x-codex-window-id` HTTP/WS emission, value format,
generation advancement, resume behavior, and fork reset behavior are
unchanged.
- `request_kind`, `window_id`, and upstream turn-owned identity fields
remain schema-owned; input `responsesapi_client_metadata` cannot replace
their canonical values.
- No table, DAG, export, app-server API, or MCP `_meta` schema changes
are included.
A compaction attempt stopped by a pre-compact hook issues no model
request and therefore has no request header; its outcome remains in
analytics events. Status, error, duration, and token deltas also remain
analytics fields rather than request-header fields.
Future detached-memory attribution using a real initiating turn ID as
`trigger_turn_id` is intentionally not part of this PR.
## Sync With Main
- Final pushed head `716342e79` is rebased onto `origin/main@0d37db4b2`.
- The metadata conflict came from upstream `#24160`, which added
`forked_from_thread_id` on the same `turn_metadata` surface. Resolution
preserves that field and its protection from client metadata override
alongside this PR's request-kind, compaction, and window-id fields.
- While resolving the overlapping commits, I removed an accidental
recursive model-request overlay and a duplicate detached-memory header
builder before completing the rebase.
## Latency / User Experience Boundary
- Foreground turns perform no new filesystem, git, or network work. New
fields are inserted into metadata already serialized for outgoing
requests.
- Compaction issues the same model/HTTP requests with the same prompt,
model, service tier, and sampling settings; only metadata bytes change.
- Startup prewarm already sent metadata; it is now correctly classified
as `prewarm`.
- Non-git detached memory now sends a small kind-only metadata header
rather than no header.
- This client diff adds no user-visible latency mechanism beyond
negligible serialization and header bytes on already-existing requests.
## Validation
On conflict-resolved head `1d35c2cfb` based on `origin/main@487521733`:
- `just fmt` (passed)
- `just fix -p codex-core` (passed)
- `git diff --check origin/main...HEAD` (passed)
- `just test -p codex-core -E 'test(turn_metadata) |
test(websocket_first_turn_uses_startup_prewarm_and_create) |
test(responses_stream_includes_turn_metadata_header_for_git_workspace_e2e)
|
test(responses_websocket_forwards_turn_metadata_on_initial_and_incremental_create)
| test(remote_compact_v2_retries_failures_with_stream_retry_budget) |
test(window_id_advances_after_compact_persists_on_resume_and_resets_on_fork)'`
(`23 passed`; `bench-smoke` passed)
- `just test -p codex-app-server -E
'test(turn_start_forwards_client_metadata_to_responses_request_v2) |
test(turn_start_forwards_client_metadata_to_responses_websocket_request_body_v2)
| test(auto_compaction_remote_emits_started_and_completed_items)'` (`3
passed`; `bench-smoke` passed)
- `just test -p codex-memories-write` (`29 passed`; `bench-smoke`
passed)
## Why
#23951 added remote compaction v2 retries, but it left the retry and WS
-> HTTPS fallback behavior duplicated between normal Responses turns and
compaction. This follow-up centralizes the common retry handling so
future changes to fallback, retry delay, retry notifications, and retry
sleep do not have to be kept in sync across both callsites.
## What changed
- Added `core/src/responses_retry.rs` with a shared handler for
retryable Responses stream errors.
- Reused that handler from normal turn sampling and remote compaction
v2.
- Kept each callsite responsible for its retry budget: normal turns
still use `stream_max_retries`, while compaction v2 still uses
`min(stream_max_retries, 2)`.
- Preserved caller-specific behavior around non-retryable errors,
context-window errors, usage-limit errors, and compact-specific final
failure logging.
The shared handler now owns:
- WS -> HTTPS fallback warning emission
- retry delay selection, including server-requested stream retry delay
- retry logging
- first-WebSocket-retry notification suppression
- `Reconnecting... n/max` stream-error notification
- sleeping before the next retry attempt
## Verification
- `cargo test -p codex-core remote_compact_v2`
- `cargo test -p codex-core websocket_fallback`
- `just fix -p codex-core`
Did not run the full workspace test suite.
---------
Co-authored-by: jif-oai <jif@openai.com>
## What changed
- Add a distinct `responses_compaction_v2` value for
`CodexCompactionEvent.implementation`.
- Emit that value from the remote compaction v2 path.
- Keep local compaction as `responses` and legacy `/responses/compact`
as `responses_compact`.
## Why
Remote compaction v2 and local prompt-based compaction were both
reported as `responses`, which made the analytics table collapse two
different compaction mechanisms into one implementation bucket.
## Validation
- `just fmt`
- `just test -p codex-analytics`
`just test -p codex-core` was started locally, but this PR is
intentionally being pushed for CI to finish the remaining validation.
## Why
[Recent PR](https://github.com/openai/codex/pull/22709) removed
`trace_id` from `TurnContextItem`.
## What changed
- Add to `TurnStartedEvent` so rollout consumers can correlate turns
with telemetry traces.
- Note that the branch name is out of date because I originally re-added
to `TurnContextItem`, but we decided to move it to `TurnStartedEvent`.
## Verification
- `cargo test -p codex-protocol`
- `cargo test -p codex-core --lib
regular_turn_emits_turn_started_without_waiting_for_startup_prewarm`
- `cargo test -p codex-core --test all
emits_warning_when_resumed_model_differs`
- `cargo test -p codex-rollout`
- `cargo test -p codex-state`
## Why
Remote compaction v2 sends a normal `/responses` request with a
compaction trigger. It should follow the retry semantics used by normal
Responses streaming calls for transient stream/request failures, while
keeping a smaller per-transport retry budget because compact attempts
can run much longer than normal turns.
## What changed
- Add a v2 compaction retry loop that uses `stream_max_retries`,
matching normal Responses turn retry mechanics.
- Cap the compact v2 retry budget at 2 retries per transport with
`min(stream_max_retries, 2)`.
- Retry retryable request-open and post-open stream collection failures
through the same loop.
- Use the existing 200ms exponential backoff and requested retry delay
handling used by normal turn retries.
- Emit the same `Reconnecting... n/max` stream-error notification
pattern.
- Fall back from WebSockets to HTTPS after the compact v2 stream retry
budget is exhausted, then reset the retry counter for HTTPS.
- Keep final remote-compaction failure logging after retries/fallback
are exhausted.
- Treat compact stream EOF before `response.completed` as a retryable
stream failure.
- Add compact v2 regression coverage with `request_max_retries = 0` and
`stream_max_retries = 2`, covering both request-open failure and
opened-stream EOF in one end-to-end test.
## Tests
- `just fmt`
- `cargo test -p codex-core remote_compact_v2`
- `just fix -p codex-core`
## Why
Remote compaction now has two implementations: the existing
server-rebuilt v1 path and the newer client-rebuilt v2 path behind
`remote_compaction_v2`. The v1 path bounds retained
user/developer/system history before installing the compaction item,
while v2 was previously carrying the full retained history forward. That
made the two paths diverge for large pre-compaction transcripts even
though they are meant to preserve the same compaction contract.
This aligns v2 with the retained-history budget expected from v1 so
switching the feature flag does not materially change which
pre-compaction messages survive into the rebuilt history.
## What changed
- Apply a retained-message character budget while rebuilding v2
compacted history in `core/src/compact_remote_v2.rs`.
- Keep newest retained messages first, truncate the boundary message
with the shared `truncate_text(...)` helper, and drop older retained
messages once the budget is exhausted.
- Preserve non-text retained message content such as images while
truncating text content.
- Use the current `64_000` token retained-message default translated to
the existing `4x` character budget.
## Testing
- `cargo test -p codex-core compact_remote_v2::tests::`
- Added focused coverage for newest-first retention and truncating
multipart retained messages without dropping images.
## Why
Remote compaction v2 is the `/responses` implementation of
session-history compaction, but it still needs to preserve the
observable contract of the legacy `/responses/compact` path. In
particular, users and integrations that rely on `PreCompact` and
`PostCompact` hooks should not see different behavior when
`remote_compaction_v2` is enabled.
## What Changed
- Runs `PreCompact` before issuing the remote compaction v2 request,
including `Interrupted` analytics when a pre-hook stops execution.
- Runs `PostCompact` after a successful v2 compaction and aborts the
turn if the post-hook stops execution.
- Adds `compact_remote_parity` coverage that compares legacy and v2
compaction across manual transcript shapes, automatic pre-turn
compaction, automatic mid-turn compaction, hook payloads, replacement
history, follow-up request payloads, and API-key `service_tier=fast`
behavior.
- Registers the new parity suite under `core/tests/suite`.
Relevant code:
-
[`compact_remote_v2.rs`](https://github.com/openai/codex/blob/af63745cb502183a6fc447d0240f8150934d70b7/codex-rs/core/src/compact_remote_v2.rs)
-
[`compact_remote_parity.rs`](https://github.com/openai/codex/blob/af63745cb502183a6fc447d0240f8150934d70b7/codex-rs/core/tests/suite/compact_remote_parity.rs)
## Verification
- Added `core/tests/suite/compact_remote_parity.rs` to assert parity
between legacy remote compaction and remote compaction v2 for the
affected request, hook, rollout-history, and follow-up paths.
- Existing `compact_remote_v2` unit coverage still exercises v2
replacement-history retention and compaction-output collection.
## Why
Remote compaction v2 was still using `context_compaction` as both the
request trigger and the compacted output shape. The Responses API now
has the landed contract for this flow: Codex sends a dedicated `{
"type": "compaction_trigger" }` input item, and the backend returns the
standard `compaction` output item with encrypted content.
This aligns the v2 path with that wire contract while preserving the
existing local compacted-history post-processing behavior.
## What changed
- Add `ResponseItem::CompactionTrigger` and regenerate the app-server
protocol schema fixtures.
- Send `compaction_trigger` from `remote_compaction_v2` instead of a
payload-less `context_compaction`.
- Collect exactly one backend `compaction` output item, then reuse the
existing compacted-history rebuilding path.
- Treat the trigger item as a transient request marker rather than model
output or persisted rollout/memory content.
## Verification
- `cargo test -p codex-protocol compaction_trigger`
- `cargo test -p codex-core remote_compact_v2`
- `cargo test -p codex-core compact_remote_v2`
- `cargo test -p codex-core
responses_websocket_sends_response_processed_after_remote_compaction_v2`
- `just write-app-server-schema`
- `cargo test -p codex-app-server-protocol schema_fixtures`
## Why
Recent session history showed no active use of the raw `shell`,
`local_shell`, or `container.exec` execution surfaces. Keeping those
handlers/specs wired into core leaves duplicate shell execution paths
alongside the supported `shell_command` and unified exec tools.
## What changed
- Removed the raw `shell` handler/spec and its `ShellToolCallParams`
protocol helper.
- Removed the legacy `local_shell` and `container.exec` handler/spec
plumbing while preserving persisted-history compatibility for old
response items.
- Normalized model/config `default` and `local` shell selections to
`shell_command`.
- Pruned tests that exercised removed raw-shell/local-shell/apply-patch
variants and kept coverage on `shell_command`, unified exec, and
freeform `apply_patch`.
## Verification
- `git diff --check`
- `cargo test -p codex-protocol`
- `cargo test -p codex-tools`
- `cargo test -p codex-core tools::handlers::shell`
- `cargo test -p codex-core tools::spec`
- `cargo test -p codex-core tools::router`
- `cargo test -p codex-core
active_call_preserves_triggering_command_context`
- `cargo test -p codex-core guardian_tests`
- `cargo test -p codex-core --test all shell_serialization`
- `cargo test -p codex-core --test all apply_patch_cli`
- `cargo test -p codex-core --test all shell_command_`
- `cargo test -p codex-core --test all local_shell`
- `cargo test -p codex-core --test all otel::`
- `cargo test -p codex-core --test all hooks::`
- `just fix -p codex-core`
- `just fix -p codex-tools`
## Why
Remote compaction v2 consumes a normal Responses stream, but that
compaction-specific stream consumer dropped the `response.completed` id.
As a result, the `responses_websocket_response_processed` lifecycle
notification was emitted for normal turn sampling but not after a v2
remote compaction response was fully processed.
## What changed
- Return the completed response id alongside the v2 `context_compaction`
output item.
- After v2 compacted history is installed, send `response.processed`
through the same websocket session when the feature is enabled.
- Add websocket regression coverage for a remote compaction v2 request
followed by `response.processed`.
## Verification
- `cargo test -p codex-core --test all
responses_websocket_sends_response_processed_after_remote_compaction_v2
-- --nocapture`
- `cargo test -p codex-core
collect_context_compaction_output_accepts_additional_output_items --
--nocapture`
## Why
This adds the `remote_compaction_v2` client path so remote compaction
can run through the normal Responses stream and install a
`context_compaction` item that trigger a compaction.
The goal is to migrate some of the compaction logic on the client side
We keeps the v2 transport behind a feature flag while letting follow-up
requests reuse the compacted context instead of falling back to the
legacy compaction item shape.
## What changed
- add `ResponseItem::ContextCompaction` and refresh the generated
app-server / schema / TypeScript fixtures that expose response items on
the wire
- add `core/src/compact_remote_v2.rs` to send compaction through the
standard streamed Responses client, require exactly one
`context_compaction` output item, and install that item into compacted
history
- route manual compact and auto-compaction through the v2 path when
`remote_compaction_v2` is enabled, while keeping the existing remote
compaction path as the fallback
- preserve the new item type across history retention, follow-up request
construction, telemetry, rollout persistence, and rollout-trace
normalization
- add targeted coverage for the feature flag, `context_compaction`
serialization, rollout-trace normalization, and remote-compaction
follow-up behavior
## Verification
- added protocol tests for `context_compaction`
serialization/deserialization in `protocol/src/models.rs`
- added rollout-trace coverage for `context_compaction` normalization in
`rollout-trace/src/reducer/conversation_tests.rs`
- added remote compaction integration coverage for v2 follow-up reuse
and mixed compaction output streams in
`core/tests/suite/compact_remote.rs`
---------
Co-authored-by: Codex <noreply@openai.com>
rhan-oai ·
Ahmed Ibrahim ·
pakrym-oai ·
ningyi-oai ·
mchen-oai ·
jif-oai ·
sayan-oai ·