17 Commits

• Represent MCP authentication with an enum (#29924)

  ## Why
  
  MCP authentication has distinct OAuth and ChatGPT-session flows.
  Representing that choice as `use_chatgpt_auth` makes one flow implicit
  and allows the configuration model to express the distinction only
  through a boolean.
  
  ChatGPT credential forwarding also needs a first-party trust boundary. A
  configurable `chatgpt_base_url` controls routing, but must not grant an
  MCP server permission to receive session credentials.
  
  This change builds on #29733, where the boolean was introduced.
  
  ## What changed
  
  - Replace `use_chatgpt_auth` with an `auth` field backed by the
  exhaustive `McpServerAuth` enum.
  - Support `auth = "oauth"` and `auth = "chatgpt"`, with OAuth remaining
  the default.
  - Trust only the origin derived from the existing hardcoded
  `CHATGPT_CODEX_BASE_URL` when granting ChatGPT auth to an MCP server.
  - Keep configured bearer tokens and authorization headers ahead of the
  selected authentication flow.
  - Update config writers, schema output, fixtures, and integration-test
  setup to use the enum.
  
  ## Verification
  
  Integration coverage exercises the complete streamable HTTP startup path
  in two independent configurations:
  
  - A directly constructed MCP configuration verifies that matching an
  overridden `chatgpt_base_url` does not grant ChatGPT auth.
  - A persisted `config.toml` containing an attacker-controlled
  `chatgpt_base_url` and `auth = "chatgpt"` verifies the same boundary
  through normal config parsing.
  
  Both tests complete MCP initialization and tool listing and assert that
  the full captured request sequence contains no authorization headers.
  Separate integration coverage verifies that configured authorization
  takes precedence over ChatGPT auth.

  Ahmed Ibrahim · 2026-06-24 19:51:51 -07:00
  f8937b7d86
• Allow ChatGPT-hosted MCP servers to use session auth (#29733)

  ## Why
  
  ChatGPT session authentication was inferred from the reserved Codex Apps
  server name. That couples credential routing to Codex Apps-specific
  behavior and prevents other MCP endpoints hosted by ChatGPT from
  explicitly using the current session.
  
  The opt-in also needs a clear security boundary: an arbitrary MCP
  configuration must not be able to redirect ChatGPT credentials to
  another origin.
  
  ## What changed
  
  - Add `use_chatgpt_auth` to HTTP MCP server configuration, defaulting to
  `false`.
  - Honor the setting only when the parsed server URL has the same HTTP(S)
  origin as the configured `chatgpt_base_url`; otherwise remove the
  capability before startup.
  - Resolve bearer tokens and static or environment-backed authorization
  headers before selecting authentication, with configured authorization
  taking precedence over ChatGPT session auth.
  - Enable the setting for the built-in Codex Apps and hosted plugin
  runtime endpoints while keeping Codex Apps caching and tool
  normalization scoped to the reserved server.
  - Persist the setting through MCP config rewrite paths and expose it in
  the generated config schema.
  - Load the current login state for `codex mcp list` so reported auth
  status matches runtime behavior.
  
  ## Verification
  
  Core integration coverage exercises the complete streamable HTTP MCP
  startup path and verifies that:
  
  - a same-origin opted-in server receives the current ChatGPT access
  token;
  - an explicitly configured authorization header takes precedence;
  - a different-origin server completes MCP initialization and tool
  listing without receiving any ChatGPT authorization header.

  Ahmed Ibrahim · 2026-06-24 19:21:28 -07:00
  4c0706e24a
• core: add configurable <context_window_guidance> message (#29936)

  ## Why
  
  This PR adds a configurable `<context_window_guidance>` developer
  section immediately after `<context_window>`. Harness integrations need
  this section to give the model deployment-specific instructions for
  preparing for context-window transitions.
  
  ## What changed
  
  - Add an optional `features.token_budget.guidance_message` config with a
  1,000-byte runtime cap and generated schema support.
  - Render configured guidance as a developer `ContextualUserFragment`
  wrapped in `<context_window_guidance>` immediately after
  `<context_window>`.
  - Omit the section when guidance is unset, empty, or whitespace-only.
  - Preserve the resolved value in config locks and classify persisted
  guidance as contextual developer content.
  - Add integration coverage for rendered content and ordering.

  Michael Bolin · 2026-06-24 18:03:44 -07:00
  f15df624a6
• [codex] Remove auto-compaction opt-out (#29815)

  ## Summary
  
  - remove the default-on `auto_compaction` feature flag and generated
  config schema entries
  - restore unconditional pre-turn, model-switch/hash, and mid-turn
  automatic compaction
  - expose `new_context` whenever token-budget tooling is enabled
  - remove the disabled-auto-compaction integration coverage introduced by
  #28260
  
  ## Motivation
  
  Roll back the internal auto-compaction escape hatch added in #28260.
  Automatic compaction should no longer be suppressible with `--disable
  auto_compaction`; existing manual `/compact` behavior remains unchanged.
  
  ## Testing
  
  - `just write-config-schema`
  - `just test -p codex-features` — 53 passed
  - `just test -p codex-core 'suite::compact::'` — 36 passed
  - `just test -p codex-core
  suite::token_budget::new_context_tool_starts_new_window_before_follow_up`
  — 1 passed
  - `just fix -p codex-core -p codex-features`
  - `just fmt`
  - `just test -p codex-core` — 2,778 passed, 59 failed, 16 skipped;
  failures were outside the changed compaction paths and were dominated by
  missing first-party test binaries and shell-snapshot timeouts

  rhan-oai · 2026-06-24 00:15:04 -07:00
  2a320fedb5
• core: reset context for token budget compaction (#29743)

  ## Why
  
  When `Feature::TokenBudget` is enabled, compaction should behave like
  `new_context`: start a fresh context window with the standard injected
  context, without asking the server to summarize old history and without
  carrying prior user or assistant messages into the next model request.
  
  This is still a compaction operation from the client lifecycle
  perspective. Manual `/compact` and auto-compaction should keep the same
  observable side effects that clients and hooks expect, including compact
  hooks and `TurnItem::ContextCompaction`.
  
  ## What changed
  
  - Added `compact_token_budget` to run token-budget manual and inline
  auto-compaction through a shared compaction lifecycle.
  - Split pending `new_context` requests from forced context-window
  startup: `take_new_context_window_request()` consumes pending requests,
  and `start_new_context_window()` installs a fresh context window.
  - Routed token-budget manual `/compact` and inline auto-compaction to
  install a fresh context window locally instead of calling server/local
  summarization.
  - Preserved compact lifecycle side effects for token-budget compaction
  by running pre/post compact hooks and emitting `ContextCompaction` item
  start/completion events.
  - Updated token-budget tests to assert fresh window IDs, absence of
  server-side compaction calls, dropped prior transcript messages/tool
  output after reset, and compact hook/item lifecycle behavior.
  
  ## Testing
  
  - `just test -p codex-core
  token_budget_context_uses_new_window_after_compaction`
  - `just test -p codex-core token_budget_compaction_runs_compact_hooks`
  - `just test -p codex-core
  token_budget_mid_turn_auto_compaction_resets_before_active_follow_up`
  
  ---------
  
  Co-authored-by: pakrym-oai <pakrym@openai.com>

  Michael Bolin · 2026-06-23 16:59:04 -07:00
  32b65bbf7a
• fix: scope context remaining to body window (#29665)

  ## Why
  
  With `model_auto_compact_token_limit_scope = "body_after_prefix"`, the
  persistent prefix should not count against the active body window.
  `get_context_remaining` and the token-budget reminder should report the
  same usable body-after-prefix window that auto-compaction uses, rather
  than the total token count since the session began.
  
  This is stacked on #29664 so the mechanical move from `turn.rs` is
  isolated from the behavior fix.
  
  ## What
  
  - Extends `ContextWindowTokenStatus` with `context_remaining_tokens`.
  - Updates `get_context_remaining` to use the shared context-window
  accounting.
  - Adds integration coverage for body-after-prefix reminder timing and
  `get_context_remaining` output.
  
  ## Testing
  
  - `just test -p codex-core body_after_prefix_window`
  - `just test -p codex-core auto_compact_body_after_prefix`
  - `just fix -p codex-core`

  Michael Bolin · 2026-06-23 23:08:54 +00:00
  77e7ce1374
• core: wrap token budget window context (#29494)

  Token-budget initial context carries thread and context-window lineage
  that the model should treat as one structured context-window block.
  Wrapping it in `<context_window>` makes that boundary explicit while
  preserving the existing window id content.
  
  Before this change, the window identifiers were injected as an untagged
  developer text fragment:
  
  ```text
  Thread id <THREAD_ID>.
  First context window id: <FIRST_WINDOW_ID>
  Current context window id: <WINDOW_ID>
  Previous context window id: <PREVIOUS_WINDOW_ID>
  ```
  
  After this change, the same payload is wrapped as a context-window
  block:
  
  ```text
  <context_window>
  Thread id: <THREAD_ID>
  First context window id: <FIRST_WINDOW_ID>
  Current context window id: <WINDOW_ID>
  Previous context window id: <PREVIOUS_WINDOW_ID>
  </context_window>
  ```
  
  This adds shared `CONTEXT_WINDOW_*_TAG` protocol constants, updates
  `TokenBudgetContext` to render with those markers, treats the new
  wrapper as contextual developer content when mapping history, and
  refreshes the token-budget request-shape assertions and snapshot.
  
  Verification:
  - `just test -p codex-core token_budget`
  - `just test -p codex-core
  recognizes_context_window_as_contextual_developer_content`

  Michael Bolin · 2026-06-22 23:37:49 +00:00
  f1945de3b7
• [codex] Add internal auto-compaction opt-out (#28260)

  ## Summary
  
  - add a default-on `auto_compaction` feature flag as an internal escape
  hatch
  - skip pre-turn, model-switch/hash, and mid-turn automatic compaction
  when the flag is disabled
  - preserve manual `/compact` behavior and surface the existing
  context-window error when the provider runs out of room
  - add integration coverage for disabled pre-turn and mid-turn compaction
  
  ## Motivation
  
  Long-running SPO optimization rollouts need the option to preserve their
  full context and fail on context exhaustion instead of entering another
  compaction window. This deliberately uses the existing feature-flag
  mechanism rather than adding a dedicated public config or app-server
  API.
  
  Disable it with:
  
  ```sh
  codex --disable auto_compaction
  ```
  
  ## Testing
  
  - `just test -p codex-features` — 51 passed
  - `just test -p codex-core auto_compaction_feature_disabled` — 2 passed
  - `just fix -p codex-core -p codex-features`
  - `just write-config-schema`
  - `just test -p codex-core` — the new compaction tests passed; the
  overall local run had 54 unrelated environment failures, primarily
  missing first-party test binaries and shell-snapshot timeouts

  rhan-oai · 2026-06-21 20:11:50 -07:00
  b21f0e7a98
• [codex] simplify token budget context (#29295)

  ## Why
  
  The token-budget feature currently adds remaining-token messages
  whenever usage crosses the 25%, 50%, and 75% thresholds. Those periodic
  inserts create prompt churn without requiring action, while the
  near-compaction reminder and explicit `get_context_remaining` tool
  already cover actionable and on-demand budget information.
  
  The context-window lineage block is also easier to scan as plain labeled
  text than as a `<token_budget>`-wrapped fragment.
  
  ## What changed
  
  - Stop recording automatic remaining-token messages at percentage
  thresholds.
  - Render context-window lineage in `First`, `Current`, `Previous` order
  with colon-separated labels.
  - Omit the `Previous` line for the first context window.
  - Remove `<token_budget>` wrappers from newly rendered lineage,
  near-compaction reminders, and `get_context_remaining` output.
  - Keep recognizing legacy wrapped fragments so existing rollouts remain
  compatible.
  - Remove the post-sampling token snapshot that was only needed by the
  periodic threshold path.
  
  ## Testing
  
  - `just test -p codex-core token_budget` (11 tests passed)

  pakrym-oai · 2026-06-20 21:50:09 -07:00
  aaf737fa59
• [codex] add configurable token budget compaction reminder (#29255)

  ## Why
  
  The token-budget feature reports coarse remaining-context milestones,
  but it does not give the model a configurable wrap-up prompt before
  automatic compaction. A strict threshold-crossing check can also miss
  resumed or reconfigured windows that are already inside the threshold.
  
  ## What changed
  
  - Add structured `[features.token_budget]` configuration for an absolute
  `reminder_threshold_tokens` and bounded `reminder_message_template`;
  `{n_remaining}` is expanded when the reminder is delivered.
  - Compute remaining tokens against the next effective auto-compaction
  boundary, including scoped `body_after_prefix` accounting and the full
  context-window limit.
  - Make reminder delivery level-triggered before and after sampling, with
  one-shot state owned by `AutoCompactWindow` and re-armed on compaction,
  `new_context`, restore, or history replacement.
  - Leave the existing initial full-window token-budget context, 25/50/75%
  notices, and token-budget tools unchanged.
  - Persist the resolved feature configuration in the session config lock
  and regenerate the config schema.
  
  ## Validation
  
  - `just test -p codex-core token_budget`
  - `just test -p codex-core
  token_budget_reminder_emits_after_crossing_compaction_threshold`
  - `just test -p codex-core auto_compact_window`
  - `just test -p codex-core
  lock_contains_prompts_and_materializes_features`
  - `just test -p codex-features`
  - `just test -p codex-config`

  pakrym-oai · 2026-06-20 19:13:42 -07:00
  6df037d47f
• [codex] prototype mcp_history thread hint injection (#29259)

  ## Why
  
  Prototype whether the harness can invoke the `mcp_history` MCP while
  constructing full initial context and expose its thread hint to the
  model without requiring a model-issued tool call.
  
  The prototype builds on the context-window lineage added by #29256 and
  is now based directly on `main`.
  
  ## What changed
  
  - Call `mcp_history/thread_hint` with no arguments while building the
  full `<token_budget>` context.
  - Pass the current `threadId` through MCP request metadata, matching the
  normal MCP tool-call path.
  - Serialize only the unstructured `content` result and append it inside
  `<token_budget>` when the call succeeds.
  - Omit the additional context when the MCP call or content serialization
  fails.
  
  ## Prototype limitations
  
  - The direct call bypasses the normal model-initiated MCP approval,
  lifecycle-event, telemetry, and result-sanitization path.
  - The call has no prototype-specific timeout, result-size cap, or
  per-window cache.
  - MCP latency is added to full-context construction, including
  applicable compaction paths.
  
  ## Validation
  
  - `just test -p codex-core token_budget`

  pakrym-oai · 2026-06-20 18:02:02 -07:00
  b6d6be2a84
• core: add context window lineage IDs (#29256)

  ## Why
  
  The rendered `<token_budget>` fragment identifies the thread and current
  context window, but it does not expose enough lineage to identify the
  first window in the thread or the immediately preceding window. Those
  IDs also need to remain stable across compaction, resume, and rollback.
  
  ## What changed
  
  - Track first, previous, and current UUIDv7 context-window IDs in
  auto-compaction state.
  - Render `thread_id`, `first_window_id`, `previous_window_id`, and the
  current window ID in the full `<token_budget>` fragment.
  - Persist the first and previous window IDs in compacted rollout
  checkpoints and restore them during rollout reconstruction.
  - Preserve compatibility with older compacted records that do not
  contain the new optional fields.
  - Update focused state, rendering, reconstruction, rollback, and
  serialization coverage.
  
  ## Validation
  
  - `just test -p codex-core token_budget`
  - `just test -p codex-protocol compacted_item::tests`
  - `just test -p codex-core tracks_prefill_and_window_boundaries`
  - `just test -p codex-core
  reconstruct_history_uses_replacement_history_verbatim`
  - `just test -p codex-core
  thread_rollback_restores_cleared_reference_context_item_after_compaction`

  pakrym-oai · 2026-06-20 13:15:49 -07:00
  d1209bddfc
• core: add UUIDv7 context window IDs (#28953)

  ## Why
  
  The token-budget context currently identifies a context window by its
  thread-local sequence number. A UUIDv7 gives the model a stable opaque
  identity that remains fixed for a window and rotates when compaction or
  `new_context` starts the next one.
  
  ## What changed
  
  - Preserve the existing monotonic value as `window_number` and add a
  UUIDv7 `window_id` to `CompactedItem`.
  - Generate and rotate the UUID with auto-compaction window state,
  persist it alongside the number, and reconstruct it on resume and
  rollback.
  - Accept legacy compacted rollout records where the numeric `window_id`
  represented the window number.
  - Use the UUID only in token-budget context; existing request headers
  and metadata continue using `thread_id:window_number`.
  
  ## Testing
  
  - `just test -p codex-protocol compacted_item::tests`
  - `just test -p codex-core token_budget`

  pakrym-oai · 2026-06-18 17:00:49 -07:00
  5c12034e42
• Include thread id in token budget context (#27663)

  ## Why
  
  The token budget full-context fragment identifies the current context
  window, but not the thread that owns that window. Including the thread
  id makes the initial context-window metadata self-contained, and
  `get_context_remaining` also needs to be usable from Code Mode without
  forcing callers to parse the model-facing fragment string.
  
  ## What changed
  
  - Include the session thread id in the initial `<token_budget>` context
  fragment.
  - Expose `get_context_remaining` as a Code Mode nested tool while
  keeping `new_context` direct-model-only.
  - Keep direct model-facing `get_context_remaining` output as the
  existing `<token_budget>` text fragment.
  - Return only `tokens_left` from the Code Mode structured result for
  `get_context_remaining`.
  - Update token-budget integration tests and add Code Mode coverage for
  the structured result.
  
  ## Verification
  
  - `just test -p codex-core token_budget`
  - `just test -p codex-core
  code_mode_get_context_remaining_returns_structured_result`
  - `just test -p core_test_support redacted_text_mode_normalizes_uuids`

  pakrym-oai · 2026-06-11 15:10:29 -07:00
  7516eb5c70
• [codex] Add context remaining tool (#27518)

  ## Why
  
  The token budget feature can inject remaining-context notices into
  model-visible context, but the model does not have a direct way to ask
  for that same remaining-token fragment on demand.
  
  This PR adds a small model tool for the token budget feature so the
  model can request the current remaining context window message without
  duplicating the fragment format.
  
  ## What changed
  
  - Adds a `get_context_remaining` direct-model tool behind
  `Feature::TokenBudget`.
  - Renders the tool output through `TokenBudgetRemainingContext`,
  matching the existing budget message shape.
  - Registers the tool alongside `new_context` in the token budget tool
  set.
  - Adds integration coverage that verifies the tool is exposed and
  returns the same `<token_budget>` remaining fragment already present in
  context.
  
  ## Validation
  
  - `just test -p codex-core token_budget`

  pakrym-oai · 2026-06-11 04:17:10 +00:00
  dac5f07403
• [codex] Add new context window tool (#27488)

  ## Why
  
  The token budget feature tells the model how much room remains in the
  current context window. When the model decides the current window is no
  longer useful, it needs a way to ask Codex to start over with a fresh
  context window without spending tokens on a compaction summary.
  
  This PR adds that model-requestable escape hatch on top of #27438.
  
  ## What changed
  
  - Added a direct-model-only `new_context` tool behind
  `Feature::TokenBudget`.
  - Stores the tool request on `AutoCompactWindow` and consumes it after
  sampling so the next follow-up request in the same turn starts in the
  new window.
  - Starts the new window as a no-summary compaction checkpoint that
  contains only fresh initial context, not preserved conversation history.
  - Keeps the new window aligned with token-budget startup context,
  including the `Current context window Z` message.
  - Added integration coverage and a snapshot showing the same-turn
  `new_context` flow into a fresh full-context follow-up request.
  
  ## Validation
  
  - `just test -p codex-core token_budget`

  pakrym-oai · 2026-06-11 03:39:07 +00:00
  87ab01834a
• [codex] Add token budget context feature (#27438)

  ## Why
  
  The model should be able to see bounded context-window budget metadata
  when the `token_budget` feature is enabled. The full-window message is
  only injected with full context, while normal turns get a smaller
  follow-up only when reported usage first crosses a budget threshold.
  
  ## What changed
  
  - Added the `TokenBudget` feature flag.
  - Added `<token_budget>` developer fragments for full context-window
  metadata and current-window remaining tokens.
  - Inserted the threshold message during normal turn handling by
  comparing token usage before and after sampling, avoiding persistent
  threshold bookkeeping.
  - Added core integration coverage for full-context-only metadata and
  25/50/75 percent threshold messages.
  
  ## Verification
  
  - `just test -p codex-core token_budget`
  - `git diff --check`

  pakrym-oai · 2026-06-10 20:07:06 -07:00
  658af936fd