27 Commits

• Handle model-switch base instructions after compaction (#11659)

  Strip trailing <model_switch> during model-switch compaction request,
  and append <model_switch> after model switch compaction

  Charley Cunningham · 2026-02-13 19:02:53 -08:00
  67e577da53
• chore: persist turn_id in rollout session and make turn_id uuid based (#11246)

  Problem:
  1. turn id is constructed in-memory;
  2. on resuming threads, turn_id might not be unique;
  3. client cannot no the boundary of a turn from rollout files easily.
  
  This PR does three things:
  1. persist `task_started` and `task_complete` events;
  1. persist `turn_id` in rollout turn events;
  5. generate turn_id as unique uuids instead of incrementing it in
  memory.
  
  This helps us resolve the issue of clients wanting to have unique turn
  ids for resuming a thread, and knowing the boundry of each turn in
  rollout files.
  
  example debug logs
  ```
  2026-02-11T00:32:10.746876Z DEBUG codex_app_server_protocol::protocol::thread_history: built turn from rollout items turn_index=8 turn=Turn { id: "019c4a07-d809-74c3-bc4b-fd9618487b4b", items: [UserMessage { id: "item-24", content: [Text { text: "hi", text_elements: [] }] }, AgentMessage { id: "item-25", text: "Hi. I’m in the workspace with your current changes loaded and ready. Send the next task and I’ll execute it end-to-end." }], status: Completed, error: None }
  2026-02-11T00:32:10.746888Z DEBUG codex_app_server_protocol::protocol::thread_history: built turn from rollout items turn_index=9 turn=Turn { id: "019c4a18-1004-76c0-a0fb-a77610f6a9b8", items: [UserMessage { id: "item-26", content: [Text { text: "hello", text_elements: [] }] }, AgentMessage { id: "item-27", text: "Hello. Ready for the next change in `codex-rs`; I can continue from the current in-progress diff or start a new task." }], status: Completed, error: None }
  2026-02-11T00:32:10.746899Z DEBUG codex_app_server_protocol::protocol::thread_history: built turn from rollout items turn_index=10 turn=Turn { id: "019c4a19-41f0-7db0-ad78-74f1503baeb8", items: [UserMessage { id: "item-28", content: [Text { text: "hello", text_elements: [] }] }, AgentMessage { id: "item-29", text: "Hello. Send the specific change you want in `codex-rs`, and I’ll implement it and run the required checks." }], status: Completed, error: None }
  ```
  
  backward compatibility:
  if you try to resume an old session without task_started and
  task_complete event populated, the following happens:
  - If you resume and do nothing: those reconstructed historical IDs can
  differ next time you resume.
  - If you resume and send a new turn: the new turn gets a fresh UUID from
  live submission flow and is persisted, so that new turn’s ID is stable
  on later resumes.
  I think this behavior is fine, because we only care about deterministic
  turn id once a turn is triggered.

  Celia Chen · 2026-02-11 03:56:01 +00:00
  641d5268fa
• Strip unsupported images from prompt history to guard against model switch (#11349)

  - Make `ContextManager::for_prompt` modality-aware and strip input_image
  content when the active model is text-only.
  - Added a test for multi-model -> text-only model switch

  Ahmed Ibrahim · 2026-02-10 11:58:00 -08:00
  5e01450963
• core: add focused diagnostics for remote compaction overflows (#11133)

  ## Summary
  - add targeted remote-compaction failure diagnostics in compact_remote
  logging
  - log the specific values needed to explain overflow timing:
    - last_api_response_total_tokens
    - estimated_tokens_of_items_added_since_last_successful_api_response
    - estimated_bytes_of_items_added_since_last_successful_api_response
    - failing_compaction_request_body_bytes
  - simplify breakdown naming and remove
  last_api_response_total_bytes_estimate (it was an approximation and not
  useful for debugging)
  
  ## Why
  When compaction fails with context_length_exceeded, we need concrete,
  low-ambiguity numbers that map directly to:
  1) what the API most recently reported, and
  2) what local history added since then.
  
  This keeps the failure logs actionable without adding broad, noisy
  metrics.
  
  ## Testing
  - just fmt
  - cargo test -p codex-core

  Charley Cunningham · 2026-02-09 12:42:20 -08:00
  9450cd9ce5
• Treat compaction failure as failure state (#10927)

  - Return compaction errors from local and remote compaction flows.\n-
  Stop turns/tasks when auto-compaction fails instead of continuing
  execution.

  Ahmed Ibrahim · 2026-02-06 13:51:46 -08:00
  ba8b5d9018
• core: refresh developer instructions after compaction replacement history (#10574)

  ## Summary
  
  When replaying compacted history (especially `replacement_history` from
  remote compaction), we should not keep stale developer messages from
  older session state. This PR trims developer-
  role messages from compacted replacement history and reinjects fresh
  developer instructions derived from current turn/session state.
  
  This aligns compaction replay behavior with the intended "fresh
  instructions after summary" model.
  
  ## Problem
  
  Compaction replay had two paths:
  
  - `Compacted { replacement_history: None }`: rebuilt with fresh initial
  context
  - `Compacted { replacement_history: Some(...) }`: previously used raw
  replacement history as-is
  
  The second path could carry stale developer instructions
  (permissions/personality/collab-mode guidance) across session changes.
  
  ## What Changed
  
  ### 1) Added helper to refresh compacted developer instructions
  
  - **File:** `codex-rs/core/src/compact.rs`
  - **Function:** `refresh_compacted_developer_instructions(...)`
  
  Behavior:
  - remove all `ResponseItem::Message { role: "developer", .. }` from
  compacted history
  - append fresh developer messages from current
  `build_initial_context(...)`
  
  ### 2) Applied helper in remote compaction flow
  
  - **File:** `codex-rs/core/src/compact_remote.rs`
  - After receiving compact endpoint output, refresh developer
  instructions before replacing history and persisting
  `replacement_history`.
  
  ### 3) Applied helper while reconstructing history from rollout
  
  - **File:** `codex-rs/core/src/codex.rs`
  - In `reconstruct_history_from_rollout(...)`, when processing
  `Compacted` entries with `replacement_history`, refresh developer
  instructions instead of directly replacing with raw history.
  
  ## Non-Goals / Follow-up
  
  This PR does **not** address the existing first-turn-after-resume
  double-injection behavior.
  A follow-up PR will handle resume-time dedup/idempotence separately.
  
  If you want, I can also give you a shorter “squash-merge friendly”
  version of the description.
  
  ## Codex author
  `codex fork 019c25e6-706e-75d1-9198-688ec00a8256`

  Charley Cunningham · 2026-02-06 12:25:08 -08:00
  143daadb31
• Fix remote compaction estimator/payload instruction small mismatch (#10692)

  ## Summary
  This PR fixes a deterministic mismatch in remote compaction where
  pre-trim estimation and the `/v1/responses/compact` payload could use
  different base instructions.
  
  Before this change:
  - pre-trim estimation used model-derived instructions
  (`model_info.get_model_instructions(...)`)
  - compact payload used session base instructions
  (`sess.get_base_instructions()`)
  
  After this change:
  - remote pre-trim estimation and compact payload both use the same
  `BaseInstructions` instance from session state.
  
  ## Changes
  - Added a shared estimator entry point in `ContextManager`:
  - `estimate_token_count_with_base_instructions(&self, base_instructions:
  &BaseInstructions) -> Option<i64>`
  - Kept `estimate_token_count(&TurnContext)` as a thin wrapper that
  resolves model/personality instructions and delegates to the new helper.
  - Updated remote compaction flow to fetch base instructions once and
  reuse it for both:
    - trim preflight estimation
    - compact request payload construction
  - Added regression coverage for parity and behavior:
    - unit test verifying explicit-base estimator behavior
  - integration test proving remote compaction uses session override
  instructions and trims accordingly
  
  ## Why this matters
  This removes a deterministic divergence source where pre-trim could
  think the request fits while the actual compact request exceeded context
  because its instructions were longer/different.
  
  ## Scope
  In scope:
  - estimator/payload base-instructions parity in remote compaction
  
  Out of scope:
  - retry-on-`context_length_exceeded`
  - compaction threshold/headroom policy changes
  - broader trimming policy changes
  
  ## Codex author:
  `codex fork 019c2b24-c2df-7b31-a482-fb8cf7a28559`

  Charley Cunningham · 2026-02-04 23:24:06 -08:00
  dc7007beaa
• Session-level model client (#10664)

  Make ModelClient a session-scoped object.
  Move state that is session level onto the client, and make state that is
  per-turn explicit on corresponding methods.
  Stop taking a huge Config object, instead only pass in values that are
  actually needed.
  
  ---------
  
  Co-authored-by: Josh McKinney <joshka@openai.com>

  pakrym-oai · 2026-02-04 16:58:48 -08:00
  0e8d359da9
• Stop client from being state carrier (#10595)

  I'd like to make client session wide. This requires shedding all random
  state it has to carry.

  pakrym-oai · 2026-02-04 09:05:37 -08:00
  7f20357611
• Trim compaction input (#10374)

  Two fixes:
  
  1. Include trailing tool output in the total context size calculation.
  Otherwise when checking whether compaction should run we ignore newly
  added outputs.
  2. Trim trailing tool output/tool calls until we can fit the request
  into the model context size. Otherwise the compaction endpoint will fail
  to compact. We only trim items that can be reproduced again by the model
  (tool calls, tool call outputs).

  pakrym-oai · 2026-02-02 19:03:11 -08:00
  cbfd2a37cc
• Do not append items on override turn context (#10354)
  pakrym-oai · 2026-02-01 18:51:26 -08:00
  03fcd12e77
• Plan mode: stream proposed plans, emit plan items, and render in TUI (#9786)

  ## Summary
  - Stream proposed plans in Plan Mode using `<proposed_plan>` tags parsed
  in core, emitting plan deltas plus a plan `ThreadItem`, while stripping
  tags from normal assistant output.
  - Persist plan items and rebuild them on resume so proposed plans show
  in thread history.
  - Wire plan items/deltas through app-server protocol v2 and render a
  dedicated proposed-plan view in the TUI, including the “Implement this
  plan?” prompt only when a plan item is present.
  
  ## Changes
  
  ### Core (`codex-rs/core`)
  - Added a generic, line-based tag parser that buffers each line until it
  can disprove a tag prefix; implements auto-close on `finish()` for
  unterminated tags. `codex-rs/core/src/tagged_block_parser.rs`
  - Refactored proposed plan parsing to wrap the generic parser.
  `codex-rs/core/src/proposed_plan_parser.rs`
  - In plan mode, stream assistant deltas as:
    - **Normal text** → `AgentMessageContentDelta`
    - **Plan text** → `PlanDelta` + `TurnItem::Plan` start/completion  
    (`codex-rs/core/src/codex.rs`)
  - Final plan item content is derived from the completed assistant
  message (authoritative), not necessarily the concatenated deltas.
  - Strips `<proposed_plan>` blocks from assistant text in plan mode so
  tags don’t appear in normal messages.
  (`codex-rs/core/src/stream_events_utils.rs`)
  - Persist `ItemCompleted` events only for plan items for rollout replay.
  (`codex-rs/core/src/rollout/policy.rs`)
  - Guard `update_plan` tool in Plan Mode with a clear error message.
  (`codex-rs/core/src/tools/handlers/plan.rs`)
  - Updated Plan Mode prompt to:  
    - keep `<proposed_plan>` out of non-final reasoning/preambles  
    - require exact tag formatting  
    - allow only one `<proposed_plan>` block per turn  
    (`codex-rs/core/templates/collaboration_mode/plan.md`)
  
  ### Protocol / App-server protocol
  - Added `TurnItem::Plan` and `PlanDeltaEvent` to core protocol items.
  (`codex-rs/protocol/src/items.rs`, `codex-rs/protocol/src/protocol.rs`)
  - Added v2 `ThreadItem::Plan` and `PlanDeltaNotification` with
  EXPERIMENTAL markers and note that deltas may not match the final plan
  item. (`codex-rs/app-server-protocol/src/protocol/v2.rs`)
  - Added plan delta route in app-server protocol common mapping.
  (`codex-rs/app-server-protocol/src/protocol/common.rs`)
  - Rebuild plan items from persisted `ItemCompleted` events on resume.
  (`codex-rs/app-server-protocol/src/protocol/thread_history.rs`)
  
  ### App-server
  - Forward plan deltas to v2 clients and map core plan items to v2 plan
  items. (`codex-rs/app-server/src/bespoke_event_handling.rs`,
  `codex-rs/app-server/src/codex_message_processor.rs`)
  - Added v2 plan item tests.
  (`codex-rs/app-server/tests/suite/v2/plan_item.rs`)
  
  ### TUI
  - Added a dedicated proposed plan history cell with special background
  and padding, and moved “• Proposed Plan” outside the highlighted block.
  (`codex-rs/tui/src/history_cell.rs`, `codex-rs/tui/src/style.rs`)
  - Only show “Implement this plan?” when a plan item exists.
  (`codex-rs/tui/src/chatwidget.rs`,
  `codex-rs/tui/src/chatwidget/tests.rs`)
  
  <img width="831" height="847" alt="Screenshot 2026-01-29 at 7 06 24 PM"
  src="https://github.com/user-attachments/assets/69794c8c-f96b-4d36-92ef-c1f5c3a8f286"
  />
  
  ### Docs / Misc
  - Updated protocol docs to mention plan deltas.
  (`codex-rs/docs/protocol_v1.md`)
  - Minor plumbing updates in exec/debug clients to tolerate plan deltas.
  (`codex-rs/debug-client/src/reader.rs`, `codex-rs/exec/...`)
  
  ## Tests
  - Added core integration tests:
    - Plan mode strips plan from agent messages.
    - Missing `</proposed_plan>` closes at end-of-message.  
    (`codex-rs/core/tests/suite/items.rs`)
  - Added unit tests for generic tag parser (prefix buffering, non-tag
  lines, auto-close). (`codex-rs/core/src/tagged_block_parser.rs`)
  - Existing app-server plan item tests in v2.
  (`codex-rs/app-server/tests/suite/v2/plan_item.rs`)
  
  ## Notes / Behavior
  - Plan output no longer appears in standard assistant text in Plan Mode;
  it streams via `PlanDelta` and completes as a `TurnItem::Plan`.
  - The final plan item content is authoritative and may diverge from
  streamed deltas (documented as experimental).
  - Reasoning summaries are not filtered; prompt instructs the model not
  to include `<proposed_plan>` outside the final plan message.
  
  ## Codex Author
  `codex fork 019bec2d-b09d-7450-b292-d7bcdddcdbfb`

  Charley Cunningham · 2026-01-30 18:59:30 +00:00
  ec4a2d07e4
• compaction (#10034)

  # External (non-OpenAI) Pull Request Requirements
  
  Before opening this Pull Request, please read the dedicated
  "Contributing" markdown file or your PR may be closed:
  https://github.com/openai/codex/blob/main/docs/contributing.md
  
  If your PR conforms to our contribution guidelines, replace this text
  with a detailed and high quality description of your changes.
  
  Include a link to a bug report or enhancement request.

  Ahmed Ibrahim · 2026-01-28 11:36:11 -08:00
  b7edeee8ca
• feat(core) update Personality on turn (#9644)

  ## Summary
  Support updating Personality mid-Thread via UserTurn/OverwriteTurn. This
  is explicitly unused by the clients so far, to simplify PRs - app-server
  and tui implementations will be follow-ups.
  
  ## Testing
  - [x] added integration tests

  Dylan Hurd · 2026-01-22 12:04:23 -08:00
  8b3521ee77
• feat(personality) introduce model_personality config (#9459)

  ## Summary
  Introduces the concept of a config model_personality. I would consider
  this an MVP for testing out the feature. There are a number of
  follow-ups to this PR:
  
  - More sophisticated templating with validation
  - In-product experience to manage this
  
  ## Testing
  - [x] Testing locally

  Dylan Hurd · 2026-01-20 11:06:14 -08:00
  714151eb4e
• fix(core) Preserve base_instructions in SessionMeta (#9427)

  ## Summary
  This PR consolidates base_instructions onto SessionMeta /
  SessionConfiguration, so we ensure `base_instructions` is set once per
  session and should be (mostly) immutable, unless:
  - overridden by config on resume / fork
  - sub-agent tasks, like review or collab
  
  
  In a future PR, we should convert all references to `base_instructions`
  to consistently used the typed struct, so it's less likely that we put
  other strings there. See #9423. However, this PR is already quite
  complex, so I'm deferring that to a follow-up.
  
  ## Testing
  - [x] Added a resume test to assert that instructions are preserved. In
  particular, `resume_switches_models_preserves_base_instructions` fails
  against main.
  
  Existing test coverage thats assert base instructions are preserved
  across multiple requests in a session:
  - Manual compact keeps baseline instructions:
  core/tests/suite/compact.rs:199
  - Auto-compact keeps baseline instructions:
  core/tests/suite/compact.rs:1142
  - Prompt caching reuses the same instructions across two requests:
  core/tests/suite/prompt_caching.rs:150 and
  core/tests/suite/prompt_caching.rs:157
  - Prompt caching with explicit expected string across two requests:
  core/tests/suite/prompt_caching.rs:213 and
  core/tests/suite/prompt_caching.rs:222
  - Resume with model switch keeps original instructions:
  core/tests/suite/resume.rs:136
  - Compact/resume/fork uses request 0 instructions for later expected
  payloads: core/tests/suite/compact_resume_fork.rs:215

  Dylan Hurd · 2026-01-19 21:59:36 -08:00
  675f165c56
• renaming: task to turn (#8963)
  jif-oai · 2026-01-09 17:31:17 +00:00
  1aed01e99f
• clean: all history cloning (#8916)
  jif-oai · 2026-01-08 18:17:18 +00:00
  69898e3dba
• feat: add compaction event (#7289)
  jif-oai · 2025-11-25 16:12:14 +00:00
  9ba27cfa0a
• [app-server & core] introduce new codex error code and v2 app-server error events (#6938)

  This PR does two things:
  1. populate a new `codex_error_code` protocol in error events sent from
  core to client;
  2. old v1 core events `codex/event/stream_error` and `codex/event/error`
  will now both become `error`. We also show codex error code for
  turncompleted -> error status.
  
  new events in app server test:
  ```
  < {
  <   "method": "codex/event/stream_error",
  <   "params": {
  <     "conversationId": "019aa34c-0c14-70e0-9706-98520a760d67",
  <     "id": "0",
  <     "msg": {
  <       "codex_error_code": {
  <         "response_stream_disconnected": {
  <           "http_status_code": 401
  <         }
  <       },
  <       "message": "Reconnecting... 2/5",
  <       "type": "stream_error"
  <     }
  <   }
  < }
  
   {
  <   "method": "error",
  <   "params": {
  <     "error": {
  <       "codexErrorCode": {
  <         "responseStreamDisconnected": {
  <           "httpStatusCode": 401
  <         }
  <       },
  <       "message": "Reconnecting... 2/5"
  <     }
  <   }
  < }
  
  < {
  <   "method": "turn/completed",
  <   "params": {
  <     "turn": {
  <       "error": {
  <         "codexErrorCode": {
  <           "responseTooManyFailedAttempts": {
  <             "httpStatusCode": 401
  <           }
  <         },
  <         "message": "exceeded retry limit, last status: 401 Unauthorized, request id: 9a1b495a1a97ed3e-SJC"
  <       },
  <       "id": "0",
  <       "items": [],
  <       "status": "failed"
  <     }
  <   }
  < }
  ```

  Celia Chen · 2025-11-20 23:06:55 +00:00
  9bce050385
• Revert "[core] add optional status_code to error events (#6865)" (#6955)

  This reverts commit c2ec477d93.
  
  # External (non-OpenAI) Pull Request Requirements
  
  Before opening this Pull Request, please read the dedicated
  "Contributing" markdown file or your PR may be closed:
  https://github.com/openai/codex/blob/main/docs/contributing.md
  
  If your PR conforms to our contribution guidelines, replace this text
  with a detailed and high quality description of your changes.
  
  Include a link to a bug report or enhancement request.

  Celia Chen · 2025-11-20 01:26:14 +00:00
  72a1453ac5
• [core] add optional status_code to error events (#6865)

  We want to better uncover error status code for clients. Add an optional
  status_code to error events (thread error, error, stream error) so app
  server could uncover the status code from the client side later.
  
  in event log:
  ```
  < {
  <   "method": "codex/event/stream_error",
  <   "params": {
  <     "conversationId": "019a9a32-f576-7292-9711-8e57e8063536",
  <     "id": "0",
  <     "msg": {
  <       "message": "Reconnecting... 5/5",
  <       "status_code": 401,
  <       "type": "stream_error"
  <     }
  <   }
  < }
  < {
  <   "method": "codex/event/error",
  <   "params": {
  <     "conversationId": "019a9a32-f576-7292-9711-8e57e8063536",
  <     "id": "0",
  <     "msg": {
  <       "message": "exceeded retry limit, last status: 401 Unauthorized, request id: 9a0cb03a485067f7-SJC",
  <       "status_code": 401,
  <       "type": "error"
  <     }
  <   }
  < }
  ```

  Celia Chen · 2025-11-19 19:51:21 +00:00
  c2ec477d93
• chore: consolidate compaction token usage (#6894)
  jif-oai · 2025-11-19 11:26:01 +00:00
  3e9e1d993d
• Run remote auto compaction (#6879)
  pakrym-oai · 2025-11-19 00:43:58 -08:00
  75f38f16dd
• fix: local compaction (#6844)
  jif-oai · 2025-11-18 22:18:10 +00:00
  c56d0c159b
• Add the utility to truncate by tokens (#6746)

  - This PR is to make it on path for truncating by tokens. This path will
  be initially used by unified exec and context manager (responsible for
  MCP calls mainly).
  - We are exposing new config `calls_output_max_tokens`
  - Use `tokens` as the main budget unit but truncate based on the model
  family by Introducing `TruncationPolicy`.
  - Introduce `truncate_text` as a router for truncation based on the
  mode.
  
  In next PRs:
  - remove truncate_with_line_bytes_budget
  - Add the ability to the model to override the token budget.

  Ahmed Ibrahim · 2025-11-18 11:36:23 -08:00
  3de8790714
• feat: remote compaction (#6795)

  Co-authored-by: pakrym-oai <pakrym@openai.com>

  jif-oai · 2025-11-18 16:51:16 +00:00
  838531d3e4