When using Responses Lite, we should all use `additional_tools` and a
developer item instead of the top level tools array & instructions
field. This keeps things 1-to-1.
Forced namespacing for _all_ tools will land in a following PR after
some coordination & fixes in Responses API (around collisions & return
items).
The goal is to eventually expand the scope of this to _all_ requests
from codex, but that will require larger coordination across providers &
slower rollout.
## Summary
- read the request-scoped safety-buffering treatment from HTTP response
headers and per-turn WebSocket metadata through one shared header parser
- combine that treatment with Responses API safety-buffering signals
- propagate `showBufferingUi` and nullable `fasterModel` through the
existing `model/safetyBuffering/updated` app-server notification
- update the app-server documentation and generated JSON and TypeScript
schemas
The public implementation contains no model mapping or real model
identifier. Tests and protocol examples use generic `current-model` and
`faster-model` placeholders only.
## Dependencies
- server-side treatment evaluation:
https://github.com/openai/openai/pull/1060247
- initial Responses API safety-buffering propagation:
https://github.com/openai/codex/pull/29371
- Codex App UI: https://github.com/openai/openai/pull/1057789
## Validation
- Codex API tests: 129 passed
- focused Codex core safety-buffering integration test passed
- app-server protocol tests passed after regenerating schema fixtures
- Clippy fix and repository formatting completed successfully
The broader app-server run compiled all changed crates and completed
with 1,269 passing tests. Its remaining failures were unrelated
environment limitations: macOS sandbox application was denied, one
expected test binary was unavailable, and several existing subprocess
tests timed out as a result.
## Description
This PR cuts Codex over from generic `ResponseItem.metadata` (introduced
here: https://github.com/openai/codex/pull/28355) to
`ResponseItem.internal_chat_message_metadata_passthrough`, which is the
blessed path and has strongly-typed keys.
For now we have to drop this MAv2 usage of `metadata`:
https://github.com/openai/codex/pull/28561 until we figure out where
that should live.
## Why
Every successful Responses WebSocket event currently produces three
local log records: the full payload at TRACE, an OpenTelemetry log
event, and an OpenTelemetry trace event.
On busy threads these records fill the 1,000-row log partition in
seconds and cause continuous SQLite insert-and-prune churn.
Related to
https://openai.slack.com/archives/C095U48JNL9/p1782128972644209
## What changed
- Stop logging each successful Responses WebSocket payload at TRACE.
- Stop emitting `codex.websocket_event` as OpenTelemetry log and trace
events.
- Keep WebSocket event counters, duration metrics, response timing
metrics, parsing, and error handling.
Responses API safety buffering metadata currently stops at the transport
boundary, so app-server clients cannot render the in-progress safety
review state.
This change:
- decodes and deduplicates `safety_buffering` metadata from Responses
API SSE and WebSocket events without suppressing the original response
event
- emits a typed core event containing the requested model plus backend
use cases and reasons
- forwards that event as `turn/safetyBuffering/updated` through
app-server v2 and updates generated protocol schemas
- keeps the side-channel event out of persisted rollouts and turn timing
This supports the Codex Apps buffering UX and depends on the Responses
API backend work in https://github.com/openai/openai/pull/1044569 and
https://github.com/openai/openai/pull/1044571.
Validation:
- focused `codex-core` safety-buffering integration test passes
- `cargo check -p codex-core -p codex-app-server -p
codex-app-server-protocol`
- `just fix -p codex-api -p codex-protocol -p codex-core -p
codex-app-server-protocol -p codex-app-server -p codex-rollout -p
codex-rollout-trace -p codex-otel`
- `just fmt`
- broad package test run: 4,430/4,492 passed; 62 unrelated
local-environment/concurrency failures involved unavailable test
binaries, MCP subprocess setup, and app-server timeouts
## Description
This PR adds an optional `metadata` field to `ResponseItem` for
Responses API calls. Only mechanical plumbing, no actual values
populated and sent yet. Turns out just adding a new field to
`ResponseItem` has quite a large blast radius already.
This change is backwards compatible because `metadata` is optional and
omitted when absent, so existing response items and rollout history
without it still deserialize and requests that do not set it keep the
same wire shape. For provider compatibility, we strip out `metadata`
before non-OpenAI Responses requests so Azure and AWS Bedrock never see
this field.
My followup PR here will actually make use of it to start storing and
passing along `turn_id`: https://github.com/openai/codex/pull/28360
## What changed
- Added `ResponseItemMetadata` with optional `turn_id`, plus optional
`metadata` on Responses API item variants and inter-agent communication.
- Preserved item metadata through response-item rewrites such as
truncation, missing tool-output synthesis, compaction history
rebuilding, visible-history conversion, rollout/resume, and generated
app-server schemas/types.
- Strip item metadata from non-OpenAI Responses requests while
preserving it for OpenAI-shaped requests.
- Updated the mechanical fixture/test construction churn required by the
new optional field.
## Why
Responses WebSocket requests were encoded in two steps: first into a
full `serde_json::Value`, then again into the JSON string sent over the
socket.
That walks the full request twice and keeps an extra JSON tree alive.
These requests can contain the complete conversation history and tool
schemas, so the extra work grows with the request size.
## What changed
- serialize `ResponsesWsRequest` directly to the wire string
- pass that string through the existing WebSocket stream and send path
- keep the existing error mapping, tracing, send timeout, and telemetry
behavior
- compare the new wire JSON with the previous `to_value` payload in a
focused test
## Performance
I measured both paths in an optimized temporary test using a
6,324,180-byte request: 4 MiB of history plus 256 tools with 8 KiB
descriptions. Each path ran 100 times.
- previous `to_value` + `to_string`: 209 ms total, 2.09 ms per request
- direct `to_string`: 174 ms total, 1.74 ms per request
- difference: about 17% faster, or 0.35 ms per request
The direct path also removes one full temporary `serde_json::Value`
tree. For this mostly string-backed payload, that avoids roughly one
payload-sized copy plus the JSON node overhead. The exact memory saving
depends on the request shape.
The temporary benchmark was removed before committing.
## Validation
- `just test -p codex-api` — 125 passed
- `just fix -p codex-api`
## Context
Turn state is scoped to one logical turn, but the WebSocket path
currently exchanges it through upgrade headers, which are scoped to the
physical connection. A connection may be reused across turns, so its
handshake cannot represent the turn lifecycle reliably.
## Change
Exchange turn state on each WebSocket response request instead:
- send an established value in `response.create.client_metadata`
- read the returned value from the existing `response.metadata` event
- retain the first value in the turn-scoped `ModelClientSession`
`OnceLock`
- start the next logical turn without state, even when it reuses the
same WebSocket connection
This gives WebSocket requests the same first-value-wins contract as the
existing HTTP path.
## Test plan
Integration coverage verifies that:
- WebSocket replays returned state on same-turn follow-ups
- later response metadata does not replace the first value
- state resets at the logical turn boundary without requiring a
reconnect
CI validates the full change.
## Stack
This is 1/2. #28002 builds on this request-scoped transport to carry
established state through compact requests.
## Why
First-party backends can supply turn-scoped moderation metadata that
app-server clients need for client-side presentation. Exposing this as
an experimental typed notification lets opted-in clients consume it
without interpreting raw Responses API events.
## What changed
- forward `response.metadata.openai_chatgpt_moderation_metadata` from
Responses API SSE and WebSocket streams as turn-scoped moderation
metadata
- emit the experimental app-server v2 `turn/moderationMetadata`
notification with `{ threadId, turnId, metadata }`
- add app-server integration coverage for the typed moderation metadata
notification
## Testing
- `just test -p codex-core
build_ws_client_metadata_includes_window_lineage_and_turn_metadata`
- `just test -p codex-core` (fails locally: 46 failures and 1 timeout,
primarily missing `test_stdio_server` and shell snapshot timeouts)
- `just test -p codex-app-server-protocol`
- `just test -p codex-app-server
turn_moderation_metadata_emits_typed_notification_v2`
- `just test -p codex-app-server` (fails locally: 792 passed, 10 failed,
and 5 timed out; failures are in existing environment-sensitive tests,
primarily because nested macOS `sandbox-exec` is not permitted)
- `just write-app-server-schema --experimental --schema-root
/tmp/codex-app-server-schema-experimental`
## Why
The Responses websocket client no longer needs to send a follow-up
`response.processed` request after a turn response has already been
recorded. Keeping that extra acknowledgement path adds feature-gated
control flow and a second websocket request shape that no longer carries
useful behavior.
## What Changed
- Removed the `response.processed` websocket request type and sender.
- Removed the `responses_websocket_response_processed` feature flag and
schema entry.
- Removed turn and remote-compaction plumbing that only tracked response
IDs to send the acknowledgement.
- Removed tests that existed solely to cover the deleted feature path.
## Validation
- `just fix -p codex-core -p codex-api -p codex-features`
## Why
Users and support need a single command that captures the local Codex
runtime, configuration, auth, terminal, network, and state shape without
asking the user to know which diagnostic depth to choose first. `codex
doctor` now runs the useful checks by default and makes the detailed
human output the default because the command is usually run when someone
already needs context.
The command also targets concrete support failure modes we have seen
while iterating on the design:
- update-target mismatches like #21956, where the installed package
manager target can differ from the running executable
- terminal and multiplexer issues that depend on `TERM`, tmux/zellij
state, color handling, and TTY metadata
- provider-specific HTTP/WebSocket connectivity, including ChatGPT
WebSocket handshakes and API-key/provider endpoint reachability
- local state/log SQLite integrity problems and large rollout
directories
- feedback reports that need an attached, redacted diagnostic snapshot
without asking the user to run a second command
## What Changed
- Adds `codex doctor` as a grouped CLI diagnostic report with default
detailed output and `--summary` for the compact view.
- Adds stable report sections for Environment, Configuration, Updates,
Connectivity, and Background Server, plus a top Notes block that
promotes anomalies such as available updates, large rollout directories,
optional MCP issues, and mixed auth signals.
- Adds runtime provenance, install consistency, bundled/system search
readiness, terminal/multiplexer metadata, `config.toml` parse status,
auth mode details, sandbox details, feature flag summaries, update
cache/latest-version state, app-server daemon state, SQLite integrity
checks, rollout statistics, and provider-aware network diagnostics.
- Adds ChatGPT WebSocket diagnostics that report the negotiated HTTP
upgrade as `HTTP 101 Switching Protocols` and include timeout, DNS,
auth, and provider context in detailed output.
- Makes reachability provider-aware: API-key OpenAI setups check the API
endpoint, ChatGPT auth checks the ChatGPT path, and custom/AWS/local
providers check configured HTTP endpoints when available.
- Adds structured, redacted JSON output where `checks` is keyed by check
id and `details` is a key/value object for support tooling.
- Integrates doctor with feedback uploads by attaching a best-effort
`codex-doctor-report.json` report and adding derived Sentry tags for
overall status and failing/warning checks.
- Updates the TUI feedback consent copy so users can see that the doctor
report is included when logs/diagnostics are uploaded.
- Updates the CLI bug issue template to ask reporters for `codex doctor
--json` and render pasted reports as JSON.
## Example Output
The examples below are sanitized from local smoke runs with `--no-color`
so the structure is reviewable in plain text.
### `codex doctor`
```text
Codex Doctor v0.0.0 · macos-aarch64
Notes
↑ updates 0.130.0 available (current 0.0.0, dismissed 0.128.0)
⚠ rollouts 1,526 active files · 2.53 GB on disk
⚠ mcp MCP configuration has optional issues
⚠ auth mixed auth signals: ChatGPT login plus API key env var; HTTP reachability uses API-key mode
─────────────────────────────────────────────────────────────
Environment
✓ runtime local debug build
version 0.0.0
install method other
commit unknown
executable ~/code/codex.fcoury-doct…x-rs/target/debug/codex
✓ install consistent
context other
managed by npm: no · bun: no · package root —
PATH entries (2) ~/.local/share/mise/installs/node/24/bin/codex
~/.local/share/mise/shims/codex
✓ search ripgrep 15.1.0 (system, `rg`)
✓ terminal Ghostty 1.3.2-main-+b0f827665 · tmux 3.6a · TERM=xterm-256color
terminal Ghostty
TERM_PROGRAM ghostty
terminal version 1.3.2-main-+b0f827665
TERM xterm-256color
multiplexer tmux 3.6a
tmux extended-keys on
tmux allow-passthrough on
tmux set-clipboard on
✓ state databases healthy
CODEX_HOME ~/.codex (dir)
state DB ~/.codex/state_5.sqlite (file) · integrity ok
log DB ~/.codex/logs_2.sqlite (file) · integrity ok
active rollouts 1,526 files · 2.53 GB (avg 1.70 MB)
archived rollouts 8 files · 3.84 MB (avg 491.11 KB)
Configuration
✓ config loaded
model gpt-5.5 · openai
cwd ~/code/codex.fcoury-doctor/codex-rs
config.toml ~/.codex/config.toml
config.toml parse ok
MCP servers 1
feature flags 36 enabled · 7 overridden (full list with --all)
overrides code_mode, code_mode_only, memories, chronicle, goals, remote_control, prevent_idle_sleep
✓ auth auth is configured
auth storage mode File
auth file ~/.codex/auth.json
auth env vars present OPENAI_API_KEY
stored auth mode chatgpt
stored API key false
stored ChatGPT tokens true
stored agent identity false
⚠ mcp MCP configuration has optional issues — Set the missing MCP env vars or disable the affected server.
configured servers 1
disabled servers 0
streamable_http servers 1
optional reachability openaiDeveloperDocs: https://developers.openai.com/mcp (HEAD connect failed; GET connect failed)
✓ sandbox restricted fs + restricted network · approval OnRequest
approval policy OnRequest
filesystem sandbox restricted
network sandbox restricted
Connectivity
✓ network network-related environment looks readable
✓ websocket connected (HTTP 101 Switching Protocols) · 15s timeout
model provider openai
provider name OpenAI
wire API responses
supports websockets true
connect timeout 15000 ms
auth mode chatgpt
endpoint wss://chatgpt.com/backend-api/<redacted>
DNS 2 IPv4, 2 IPv6, first IPv6
handshake result HTTP 101 Switching Protocols
✗ reachability one or more required provider endpoints are unreachable over HTTP — Check proxy, VPN, firewall, DNS, and custom CA configuration.
reachability mode API key auth
openai API https://api.openai.com/v1 connect failed (required)
Background Server
○ app-server not running (ephemeral mode)
─────────────────────────────────────────────────────────────
11 ok · 1 idle · 4 notes · 1 warn · 1 fail failed
--summary compact output --all expand truncated lists
--json redacted report
```
### `codex doctor --summary`
```text
Codex Doctor v0.0.0 · macos-aarch64
Notes
↑ updates 0.130.0 available (current 0.0.0, dismissed 0.128.0)
⚠ rollouts 1,526 active files · 2.53 GB on disk
⚠ mcp MCP configuration has optional issues
⚠ auth mixed auth signals: ChatGPT login plus API key env var; HTTP reachability uses API-key mode
─────────────────────────────────────────────────────────────
Environment
✓ runtime local debug build
✓ install consistent
✓ search ripgrep 15.1.0 (system, `rg`)
✓ terminal Ghostty 1.3.2-main-+b0f827665 · tmux 3.6a · TERM=xterm-256color
✓ state databases healthy
Configuration
✓ config loaded
✓ auth auth is configured
⚠ mcp MCP configuration has optional issues — Set the missing MCP env vars or disable the affected server.
✓ sandbox restricted fs + restricted network · approval OnRequest
Updates
✓ updates update configuration is locally consistent
Connectivity
✓ network network-related environment looks readable
✓ websocket connected (HTTP 101 Switching Protocols) · 15s timeout
✗ reachability one or more required provider endpoints are unreachable over HTTP — Check proxy, VPN, firewall, DNS, and custom CA configuration.
Background Server
○ app-server not running (ephemeral mode)
─────────────────────────────────────────────────────────────
11 ok · 1 idle · 4 notes · 1 warn · 1 fail failed
Run codex doctor without --summary for detailed diagnostics.
--all expand truncated lists --json redacted report
```
### `codex doctor --json` shape
```json
{
"schema_version": 1,
"overall_status": "fail",
"checks": {
"runtime.provenance": {
"id": "runtime.provenance",
"category": "Environment",
"status": "ok",
"summary": "local debug build",
"details": {
"version": "0.0.0",
"install method": "other",
"commit": "unknown"
}
},
"sandbox.helpers": {
"id": "sandbox.helpers",
"category": "Configuration",
"status": "ok",
"summary": "restricted fs + restricted network · approval OnRequest",
"details": {
"approval policy": "OnRequest",
"filesystem sandbox": "restricted",
"network sandbox": "restricted"
}
}
}
}
```
### `/feedback` new sentry attachment
<img width="938" height="798" alt="CleanShot 2026-05-13 at 15 36 14"
src="https://github.com/user-attachments/assets/715e62e0-d7b4-4fea-a35a-fd5d5d33c4c0"
/>
### New section in CLI issue template
<img width="1164" height="435" alt="CleanShot 2026-05-13 at 15 47 24"
src="https://github.com/user-attachments/assets/9081dc25-a28c-4afa-8ba1-e299c2b4031d"
/>
## How to Test
1. Run `cargo run --bin codex -- doctor --no-color`.
2. Confirm the detailed report is the default and includes promoted
Notes, grouped sections, terminal details, state DB integrity, rollout
stats, provider reachability, WebSocket diagnostics, and app-server
status.
3. Run `cargo run --bin codex -- doctor --summary --no-color`.
4. Confirm the compact view keeps the same sections and summary counts
but omits detailed key/value rows.
5. Run `cargo run --bin codex -- doctor --json`.
6. Confirm the output is redacted JSON, `checks` is an object keyed by
check id, and each check's `details` is a key/value object.
7. Preview the CLI bug issue template and confirm the `Codex doctor
report` field appears after the terminal field, asks for `codex doctor
--json`, and renders pasted output as JSON.
8. Start a feedback flow that includes logs.
9. Confirm the upload consent copy lists `codex-doctor-report.json`
alongside the log attachments.
Targeted tests:
- `cargo test -p codex-cli doctor`
- `cargo test -p codex-app-server
doctor_report_tags_summarize_status_counts`
- `cargo test -p codex-feedback`
- `cargo test -p codex-tui feedback_view`
- `just argument-comment-lint`
- `git diff --check`
## Summary
- Add a `response.processed` websocket request payload and sender for
Responses API websockets.
- Send `response.processed` from `try_run_sampling_request` after a
response completes, local turn processing succeeds, and the
session-owned feature flag is enabled.
- Add websocket coverage for both enabled and disabled feature-flag
behavior.
## Validation
- `just fmt`
- `cargo test -p codex-core response_processed`
- `cargo test -p codex-api responses_websocket`
- `cargo test -p codex-features
responses_websocket_response_processed_is_under_development`
- `git diff --check`
- `just fix -p codex-api -p codex-core -p codex-features`
- `git diff --check origin/main...HEAD`
## Why
We saw Responses websocket sessions recover only after a long quiet
period when the server had already logged the websocket as disconnected.
The normal connect path is already bounded by
`websocket_connect_timeout_ms`, but the first request send on an
established websocket reused only the receive-side idle timeout after
the write completed. If the socket write/pump stalls, the client can sit
in `ws_stream.send(...)` without reaching the existing receive timeout.
## Why
Rollout traces need an identifier that can be used to correlate a Codex
inference with upstream Responses API, proxy, and engine logs. The
reduced trace model already exposed `upstream_request_id`, but it was
being populated from the Responses API `response.id`. That value is
useful for `previous_response_id` chaining, but it is not the transport
request id that upstream systems key on.
This PR separates those concepts so trace consumers can reliably answer
both questions:
- which Responses API response did this inference produce?
- which upstream request handled it?
## Structure
The change keeps the upstream request id at the same lifecycle level as
the provider stream:
- `codex-api` captures the `x-request-id` HTTP response header when the
SSE stream is created and exposes it on `ResponseStream`. Fixture and
websocket streams set the field to `None` because they do not have that
HTTP response header.
- `codex-core` carries that stream-level id into `InferenceTraceAttempt`
when recording terminal stream outcomes. Completed, failed, cancelled,
dropped-stream, and pre-response error paths all record the id when it
is available.
- `rollout-trace` now records both identifiers in raw terminal inference
events and response payloads: `response_id` for the Responses API
`response.id`, and `upstream_request_id` for `x-request-id`.
- The reducer stores both fields on `InferenceCall`. It also uses
`response_id` for `previous_response_id` conversation linking, which
removes the old accidental dependency on the misnamed
`upstream_request_id` field.
- Terminal inference reduction now consumes the full terminal payload
(`InferenceCompleted`, `InferenceFailed`, or `InferenceCancelled`) in
one place. That keeps status, partial payloads, response ids, and
upstream request ids consistent across success, failure, cancellation,
and late stream-mapper events.
## Why This Shape
`x-request-id` is a property of the HTTP/provider response envelope, not
an SSE event. Capturing it once in `codex-api` and plumbing it through
terminal trace recording avoids trying to infer the value from stream
contents, and it preserves the id even when the stream fails or is
cancelled after only partial output.
Keeping `response_id` separate from `upstream_request_id` also makes the
reduced trace model less surprising: `response_id` remains the
conversation-continuation id, while `upstream_request_id` is the
operational correlation id for upstream debugging.
## Validation
The PR updates trace and reducer coverage for:
- reading `x-request-id` from SSE response headers;
- storing the true upstream request id on completed inference calls;
- preserving upstream request ids for cancelled and late-cancelled
inference streams;
- keeping `previous_response_id` reconstruction tied to `response_id`
rather than transport request ids.
## Why
This PR prepares the stack to enable Clippy await-holding lints that
were left disabled in #18178. The mechanical lock-scope cleanup is
handled separately; this PR is the documentation/configuration layer for
the remaining await-across-guard sites.
Without explicit annotations, reviewers and future maintainers cannot
tell whether an await-holding warning is a real concurrency smell or an
intentional serialization boundary.
## What changed
- Configures `clippy.toml` so `await_holding_invalid_type` also covers
`tokio::sync::{MutexGuard,RwLockReadGuard,RwLockWriteGuard}`.
- Adds targeted `#[expect(clippy::await_holding_invalid_type, reason =
...)]` annotations for intentional async guard lifetimes.
- Documents the main categories of intentional cases: active-turn state
transitions that must remain atomic, session-owned MCP manager accesses,
remote-control websocket serialization, JS REPL kernel/process
serialization, OAuth persistence, external bearer token refresh
serialization, and tests that intentionally serialize shared global or
session-owned state.
- For external bearer token refresh, documents the existing
serialization boundary: holding `cached_token` across the provider
command prevents concurrent cache misses from starting duplicate refresh
commands, and the current behavior is small enough that an explicit
expectation is easier to maintain than adding another synchronization
primitive.
## Verification
- `cargo clippy -p codex-login --all-targets`
- `cargo clippy -p codex-connectors --all-targets`
- `cargo clippy -p codex-core --all-targets`
- The follow-up PR #18698 enables `await_holding_invalid_type` and
`await_holding_lock` as workspace `deny` lints, so any undocumented
remaining offender will fail Clippy.
---
[//]: # (BEGIN SAPLING FOOTER)
Stack created with [Sapling](https://sapling-scm.com). Best reviewed
with [ReviewStack](https://reviewstack.dev/openai/codex/pull/18423).
* #18698
* __->__ #18423
## Summary
- Add `codex-model-provider` as the runtime home for model-provider
behavior that does not belong in `codex-core`, `codex-login`, or
`codex-api`.
- The new crate wraps configured `ModelProviderInfo` in a
`ModelProvider` trait object that can resolve the API provider config,
provider-scoped auth manager, and request auth provider for each call.
- This centralizes provider auth behavior in one place today, and gives
us an extension point for future provider-specific auth, model listing,
request setup, and related runtime behavior.
## Tests
Ran tests manually to make sure that provider auth under different
configs still work as expected.
---------
Co-authored-by: pakrym-oai <pakrym@openai.com>
## Summary
- Move auth header construction into the
`AuthProvider::add_auth_headers` contract.
- Inline `CoreAuthProvider` header mutation in its provider impl and
remove the shared header-map helper.
- Update HTTP, websocket, file upload, sideband websocket, and test auth
callsites to use the provider method.
- Add direct coverage for `CoreAuthProvider` auth header mutation.
## Testing
- `just fmt`
- `cargo test -p codex-api`
- `cargo test -p codex-core
client::tests::auth_request_telemetry_context_tracks_attached_auth_and_retry_phase`
- `cargo test -p codex-core` failed on unrelated/reproducible
`tools::handlers::multi_agents::tests::multi_agent_v2_followup_task_interrupts_busy_child_without_losing_message`
---------
Co-authored-by: Celia Chen <celia@openai.com>
## Summary
- reduce public module visibility across Rust crates, preferring private
or crate-private modules with explicit crate-root public exports
- update external call sites and tests to use the intended public crate
APIs instead of reaching through module trees
- add the module visibility guideline to AGENTS.md
## Validation
- `cargo check --workspace --all-targets --message-format=short` passed
before the final fix/format pass
- `just fix` completed successfully
- `just fmt` completed successfully
- `git diff --check` passed
CXC-392
[With
401](https://openai.sentry.io/issues/7333870443/?project=4510195390611458&query=019ce8f8-560c-7f10-a00a-c59553740674&referrer=issue-stream)
<img width="1909" height="555" alt="401 auth tags in Sentry"
src="https://github.com/user-attachments/assets/412ea950-61c4-4780-9697-15c270971ee3"
/>
- auth_401_*: preserved facts from the latest unauthorized response snapshot
- auth_*: latest auth-related facts from the latest request attempt
- auth_recovery_*: unauthorized recovery state and follow-up result
Without 401
<img width="1917" height="522" alt="happy-path auth tags in Sentry"
src="https://github.com/user-attachments/assets/3381ed28-8022-43b0-b6c0-623a630e679f"
/>
###### Summary
- Add client-visible 401 diagnostics for auth attachment, upstream auth classification, and 401 request id / cf-ray correlation.
- Record unauthorized recovery mode, phase, outcome, and retry/follow-up status without changing auth behavior.
- Surface the highest-signal auth and recovery fields on uploaded client bug reports so they are usable in Sentry.
- Preserve original unauthorized evidence under `auth_401_*` while keeping follow-up result tags separate.
###### Rationale (from spec findings)
- The dominant bucket needed proof of whether the client attached auth before send or upstream still classified the request as missing auth.
- Client uploads needed to show whether unauthorized recovery ran and what the client tried next.
- Request id and cf-ray needed to be preserved on the unauthorized response so server-side correlation is immediate.
- The bug-report path needed the same auth evidence as the request telemetry path, otherwise the observability would not be operationally useful.
###### Scope
- Add auth 401 and unauthorized-recovery observability in `codex-rs/core`, `codex-rs/codex-api`, and `codex-rs/otel`, including feedback-tag surfacing.
- Keep auth semantics, refresh behavior, retry behavior, endpoint classification, and geo-denial follow-up work out of this PR.
###### Trade-offs
- This exports only safe auth evidence: header presence/name, upstream auth classification, request ids, and recovery state. It does not export token values or raw upstream bodies.
- This keeps websocket connection reuse as a transport clue because it can help distinguish stale reused sessions from fresh reconnects.
- Misroute/base-url classification and geo-denial are intentionally deferred to a separate follow-up PR so this review stays focused on the dominant auth 401 bucket.
###### Client follow-up
- PR 2 will add misroute/provider and geo-denial observability plus the matching feedback-tag surfacing.
- A separate host/app-server PR should log auth-decision inputs so pre-send host auth state can be correlated with client request evidence.
- `device_id` remains intentionally separate until there is a safe existing source on the feedback upload path.
###### Testing
- `cargo test -p codex-core refresh_available_models_sorts_by_priority`
- `cargo test -p codex-core emit_feedback_request_tags_`
- `cargo test -p codex-core emit_feedback_auth_recovery_tags_`
- `cargo test -p codex-core auth_request_telemetry_context_tracks_attached_auth_and_retry_phase`
- `cargo test -p codex-core extract_response_debug_context_decodes_identity_headers`
- `cargo test -p codex-core identity_auth_details`
- `cargo test -p codex-core telemetry_error_messages_preserve_non_http_details`
- `cargo test -p codex-core --all-features --no-run`
- `cargo test -p codex-otel otel_export_routing_policy_routes_api_request_auth_observability`
- `cargo test -p codex-otel otel_export_routing_policy_routes_websocket_connect_auth_observability`
- `cargo test -p codex-otel otel_export_routing_policy_routes_websocket_request_transport_observability`
## Description
This PR expands tracing coverage across app-server thread startup, core
session initialization, and the Responses transport layer. It also gives
core dispatch spans stable operation-specific names so traces are easier
to follow than the old generic `submission_dispatch` spans.
Also use `fmt::Display` for types that we serialize in traces so we send
strings instead of rust types
## Stacked PRs
This work is now effectively split across two steps:
- #14178: add custom CA support for browser and device-code login flows,
docs, and hermetic subprocess tests
- #14239: extend that shared custom CA handling across Codex HTTPS
clients and secure websocket TLS
Note: #14240 was merged into this branch while it was stacked on top of
this PR. This PR now subsumes that websocket follow-up and should be
treated as the combined change.
Builds on top of #14178.
## Problem
Custom CA support landed first in the login path, but the real
requirement is broader. Codex constructs outbound TLS clients in
multiple places, and both HTTPS and secure websocket paths can fail
behind enterprise TLS interception if they do not honor
`CODEX_CA_CERTIFICATE` or `SSL_CERT_FILE` consistently.
This PR broadens the shared custom-CA logic beyond login and applies the
same policy to websocket TLS, so the enterprise-proxy story is no longer
split between “HTTPS works” and “websockets still fail”.
## What This Delivers
Custom CA support is no longer limited to login. Codex outbound HTTPS
clients and secure websocket connections can now honor the same
`CODEX_CA_CERTIFICATE` / `SSL_CERT_FILE` configuration, so enterprise
proxy/intercept setups work more consistently end-to-end.
For users and operators, nothing new needs to be configured beyond the
same CA env vars introduced in #14178. The change is that more of Codex
now respects them, including websocket-backed flows that were previously
still using default trust roots.
I also manually validated the proxy path locally with mitmproxy using:
`CODEX_CA_CERTIFICATE=~/.mitmproxy/mitmproxy-ca-cert.pem
HTTPS_PROXY=http://127.0.0.1:8080 just codex`
with mitmproxy installed via `brew install mitmproxy` and configured as
the macOS system proxy.
## Mental model
`codex-client` is now the owner of shared custom-CA policy for outbound
TLS client construction. Reqwest callers start from the builder
configuration they already need, then pass that builder through
`build_reqwest_client_with_custom_ca(...)`. Websocket callers ask the
same module for a rustls client config when a custom CA bundle is
configured.
The env precedence is the same everywhere:
- `CODEX_CA_CERTIFICATE` wins
- otherwise fall back to `SSL_CERT_FILE`
- otherwise use system roots
The helper is intentionally narrow. It loads every usable certificate
from the configured PEM bundle into the appropriate root store and
returns either a configured transport or a typed error that explains
what went wrong.
## Non-goals
This does not add handshake-level integration tests against a live TLS
endpoint. It does not validate that the configured bundle forms a
meaningful certificate chain. It also does not try to force every
transport in the repo through one abstraction; it extends the shared CA
policy across the reqwest and websocket paths that actually needed it.
## Tradeoffs
The main tradeoff is centralizing CA behavior in `codex-client` while
still leaving adoption up to call sites. That keeps the implementation
additive and reviewable, but it means the rule "outbound Codex TLS that
should honor enterprise roots must use the shared helper" is still
partly enforced socially rather than by types.
For websockets, the shared helper only builds an explicit rustls config
when a custom CA bundle is configured. When no override env var is set,
websocket callers still use their ordinary default connector path.
## Architecture
`codex-client::custom_ca` now owns CA bundle selection, PEM
normalization, mixed-section parsing, certificate extraction, typed
CA-loading errors, and optional rustls client-config construction for
websocket TLS.
The affected consumers now call into that shared helper directly rather
than carrying login-local CA behavior:
- backend-client
- cloud-tasks
- RMCP client paths that use `reqwest`
- TUI voice HTTP paths
- `codex-core` default reqwest client construction
- `codex-api` websocket clients for both responses and realtime
websocket connections
The subprocess CA probe, env-sensitive integration tests, and shared PEM
fixtures also live in `codex-client`, which is now the actual owner of
the behavior they exercise.
## Observability
The shared CA path logs:
- which environment variable selected the bundle
- which path was loaded
- how many certificates were accepted
- when `TRUSTED CERTIFICATE` labels were normalized
- when CRLs were ignored
- where client construction failed
Returned errors remain user-facing and include the relevant env var,
path, and remediation hint. That same error model now applies whether
the failure surfaced while building a reqwest client or websocket TLS
configuration.
## Tests
Pure unit tests in `codex-client` cover env precedence and PEM
normalization behavior. Real client construction remains in subprocess
tests so the suite can control process env and avoid the macOS seatbelt
panic path that motivated the hermetic test split.
The subprocess coverage verifies:
- `CODEX_CA_CERTIFICATE` precedence over `SSL_CERT_FILE`
- fallback to `SSL_CERT_FILE`
- single-cert and multi-cert bundles
- malformed and empty-file errors
- OpenSSL `TRUSTED CERTIFICATE` handling
- CRL tolerance for well-formed CRL sections
The websocket side is covered by the existing `codex-api` / `codex-core`
websocket test suites plus the manual mitmproxy validation above.
---------
Co-authored-by: Ivan Zakharchanka <3axap4eHko@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
## What changed
- Drop failed websocket connections immediately after a terminal stream
error instead of awaiting a graceful close handshake before forwarding
the error to the caller.
- Keep the success path and the closed-connection guard behavior
unchanged.
## Why this fixes the flake
- The failing integration test waits for the second websocket stream to
surface the model error before issuing a follow-up request.
- On slower runners, the old error path awaited
`ws_stream.close().await` before sending the error downstream. If that
close handshake stalled, the test kept waiting for an error that had
already happened server-side and nextest timed it out.
- Dropping the failed websocket immediately makes the terminal error
observable right away and marks the session closed so the next request
reconnects cleanly instead of depending on a best-effort close
handshake.
## Code or test?
- This is a production logic fix in `codex-api`. The existing websocket
integration test already exercises the regression path.
Add per-turn notice when a request is downgraded to a fallback model due
to cyber safety checks.
**Changes**
- codex-api: Emit a ServerModel event based on the openai-model response
header and/or response payload (SSE + WebSocket), including when the
model changes mid-stream.
- core: When the server-reported model differs from the requested model,
emit a single per-turn warning explaining the reroute to gpt-5.2 and
directing users to Trusted
Access verification and the cyber safety explainer.
- app-server (v2): Surface these cyber model-routing warnings as
synthetic userMessage items with text prefixed by Warning: (and document
this behavior).
When communicating over websockets, we can't rely on headers to deliver
rate limit information. This PR adds a `codex.rate_limits` event that
the server can pass to the client to inform them about rate limit usage.
The client parses this data the same way we parse rate limit headers in
HTTP mode.
This PR also wires up the etag and reasoning headers for websockets
Summary
- expose websocket telemetry hooks through the responses client so
request durations and event processing can be reported
- record websocket request/event metrics and emit runtime telemetry
events that the history UI now surfaces
- improve tests to cover websocket telemetry reporting and guard runtime
summary updates
<img width="824" height="79" alt="Screenshot 2026-01-31 at 5 28 12 PM"
src="https://github.com/user-attachments/assets/ea9a7965-d8b4-4e3c-a984-ef4fdc44c81d"
/>
- capture the header from SSE/WS handshakes, store it per
ModelClientSession using `Oncelock`, echo it on turn-scoped requests,
and add SSE+WS integration tests for within-turn persistence +
cross-turn reset.
- keep `x-codex-turn-state` sticky within a user turn to maintain
routing continuity for retries/tool follow-ups.
rka-oai ·
Francis Chalissery ·
Owen Lin ·
jif ·
Ahmed Ibrahim ·
carlc-oai ·
pakrym-oai ·
Felipe Coury ·
cassirer-openai ·
Eric Traut ·
Michael Bolin ·
Celia Chen ·
Colin Young ·
Josh McKinney ·
Fouad Matin ·
Anton Panasenko ·
Rasmus Rygaard ·