Commit Graph

180 Commits

  • [MCP] Introduce an experimental official rust sdk based mcp client (#4252)
    The [official Rust
    SDK](https://github.com/modelcontextprotocol/rust-sdk/tree/57fc428c578a1a3fe851ee0838bf068bda120eb3)
    has come a long way since we first started our mcp client implementation
    5 months ago and, today, it is much more complete than our own
    stdio-only implementation.
    
    This PR introduces a new config flag `experimental_use_rmcp_client`
    which will use a new mcp client powered by the sdk instead of our own.
    
    To keep this PR simple, I've only implemented the same stdio MCP
    functionality that we had but will expand on it with future PRs.
    
    ---------
    
    Co-authored-by: pakrym-oai <pakrym@openai.com>
  • ref: state - 2 (#4229)
    Extracting tasks in a module and start abstraction behind a Trait (more
    to come on this but each task will be tackled in a dedicated PR)
    The goal was to drop the ActiveTask and to have a (potentially) set of
    tasks during each turn
  • ref: full state refactor (#4174)
    ## Current State Observations
    - `Session` currently holds many unrelated responsibilities (history,
    approval queues, task handles, rollout recorder, shell discovery, token
    tracking, etc.), making it hard to reason about ownership and lifetimes.
    - The anonymous `State` struct inside `codex.rs` mixes session-long data
    with turn-scoped queues and approval bookkeeping.
    - Turn execution (`run_task`) relies on ad-hoc local variables that
    should conceptually belong to a per-turn state object.
    - External modules (`codex::compact`, tests) frequently poke the raw
    `Session.state` mutex, which couples them to implementation details.
    - Interrupts, approvals, and rollout persistence all have bespoke
    cleanup paths, contributing to subtle bugs when a turn is aborted
    mid-flight.
    
    ## Desired End State
    - Keep a slim `Session` object that acts as the orchestrator and façade.
    It should expose a focused API (submit, approvals, interrupts, event
    emission) without storing unrelated fields directly.
    - Introduce a `state` module that encapsulates all mutable data
    structures:
    - `SessionState`: session-persistent data (history, approved commands,
    token/rate-limit info, maybe user preferences).
    - `ActiveTurn`: metadata for the currently running turn (sub-id, task
    kind, abort handle) and an `Arc<TurnState>`.
    - `TurnState`: all turn-scoped pieces (pending inputs, approval waiters,
    diff tracker, review history, auto-compact flags, last agent message,
    outstanding tool call bookkeeping).
    - Group long-lived helpers/managers into a dedicated `SessionServices`
    struct so `Session` does not accumulate "random" fields.
    - Provide clear, lock-safe APIs so other modules never touch raw
    mutexes.
    - Ensure every turn creates/drops a `TurnState` and that
    interrupts/finishes delegate cleanup to it.
  • Simplify tool implemetations (#4160)
    Use Result<String, FunctionCallError> for all tool handling code and
    rely on error propagation instead of creating failed items everywhere.
  • Send limits when getting rate limited (#4102)
    Users need visibility on rate limits when they are rate limited.
  • Add exec output-schema parameter (#4079)
    Adds structured output to `exec` via the `--structured-output`
    parameter.
  • chore: compact do not modify instructions (#4088)
    Keep the developer instruction and insert the summarisation message as a
    user message instead
  • Add notifier tests (#4064)
    Proposal:
    1. Use anyhow for tests and avoid unwrap
    2. Extract a helper for starting a test instance of codex
  • chore: clippy on redundant closure (#4058)
    Add redundant closure clippy rules and let Codex fix it by minimising
    FQP
  • timeouts for mcp tool calls (#3959)
    defaults to 60sec, overridable with MCP_TOOL_TIMEOUT or on a per-server
    basis in the config.
  • Forward Rate limits to the UI (#3965)
    We currently get information about rate limits in the response headers.
    We want to forward them to the clients to have better transparency.
    UI/UX plans have been discussed and this information is needed.
  • Use a unified shell tell to not break cache (#3814)
    Currently, we change the tool description according to the sandbox
    policy and approval policy. This breaks the cache when the user hits
    `/approvals`. This PR does the following:
    - Always use the shell with escalation parameter:
    - removes `create_shell_tool_for_sandbox` and always uses unified tool
    via `create_shell_tool`
    - Reject the func call when the model uses escalation parameter when it
    cannot.
  • fix: ensure cwd for conversation and sandbox are separate concerns (#3874)
    Previous to this PR, both of these functions take a single `cwd`:
    
    
    https://github.com/openai/codex/blob/71038381aa0f51aa62e1a2bcc7cbf26a05b141f3/codex-rs/core/src/seatbelt.rs#L19-L25
    
    
    https://github.com/openai/codex/blob/71038381aa0f51aa62e1a2bcc7cbf26a05b141f3/codex-rs/core/src/landlock.rs#L16-L23
    
    whereas `cwd` and `sandbox_cwd` should be set independently (fixed in
    this PR).
    
    Added `sandbox_distinguishes_command_and_policy_cwds()` to
    `codex-rs/exec/tests/suite/sandbox.rs` to verify this.
  • feat: /review (#3774)
    Adds `/review` action in TUI
    
    <img width="637" height="370" alt="Screenshot 2025-09-17 at 12 41 19 AM"
    src="https://github.com/user-attachments/assets/b1979a6e-844a-4b97-ab20-107c185aec1d"
    />
  • chore: use tokio mutex and async function to prevent blocking a worker (#3850)
    ### Why Use `tokio::sync::Mutex`
    
    `std::sync::Mutex` are not _async-aware_. As a result, they will block
    the entire thread instead of just yielding the task. Furthermore they
    can be poisoned which is not the case of `tokio` Mutex.
    This allows the Tokio runtime to continue running other tasks while
    waiting for the lock, preventing deadlocks and performance bottlenecks.
    
    In general, this is preferred in async environment
  • Add dev message upon review out (#3758)
    Proposal: We want to record a dev message like so:
    
    ```
    {
          "type": "message",
          "role": "user",
          "content": [
            {
              "type": "input_text",
              "text": "<user_action>
      <context>User initiated a review task. Here's the full review output from reviewer model. User may select one or more comments to resolve.</context>
      <action>review</action>
      <results>
      {findings_str}
      </results>
    </user_action>"
            }
          ]
        },
    ```
    
    Without showing in the chat transcript.
    
    Rough idea, but it fixes issue where the user finishes a review thread,
    and asks the parent "fix the rest of the review issues" thinking that
    the parent knows about it.
    
    ### Question: Why not a tool call?
    
    Because the agent didn't make the call, it was a human. + we haven't
    implemented sub-agents yet, and we'll need to think about the way we
    represent these human-led tool calls for the agent.
  • Review mode core updates (#3701)
    1. Adds the environment prompt (including cwd) to review thread
    2. Prepends the review prompt as a user message (temporary fix so the
    instructions are not replaced on backend)
    3. Sets reasoning to low
    4. Sets default review model to `gpt-5-codex`
  • fix: Record EnvironmentContext in SendUserTurn (#3678)
    ## Summary
    SendUserTurn has not been correctly handling updates to policies. While
    the tui protocol handles this in `Op::OverrideTurnContext`, the
    SendUserTurn should be appending `EnvironmentContext` messages when the
    sandbox settings change. MCP client behavior should match the cli
    behavior, so we update `SendUserTurn` message to match.
    
    ## Testing
    - [x] Added prompt caching tests
  • Include command output when sending timeout to model (#3576)
    Being able to see the output helps the model decide how to handle the
    timeout.
  • Handle resuming/forking after compact (#3533)
    We need to construct the history different when compact happens. For
    this, we need to just consider the history after compact and convert
    compact to a response item.
    
    This needs to change and use `build_compact_history` when this #3446 is
    merged.
  • Review Mode (Core) (#3401)
    ## 📝 Review Mode -- Core
    
    This PR introduces the Core implementation for Review mode:
    
    - New op `Op::Review { prompt: String }:` spawns a child review task
    with isolated context, a review‑specific system prompt, and a
    `Config.review_model`.
    - `EnteredReviewMode`: emitted when the child review session starts.
    Every event from this point onwards reflects the review session.
    - `ExitedReviewMode(Option<ReviewOutputEvent>)`: emitted when the review
    finishes or is interrupted, with optional structured findings:
    
    ```json
    {
      "findings": [
        {
          "title": "<≤ 80 chars, imperative>",
          "body": "<valid Markdown explaining *why* this is a problem; cite files/lines/functions>",
          "confidence_score": <float 0.0-1.0>,
          "priority": <int 0-3>,
          "code_location": {
            "absolute_file_path": "<file path>",
            "line_range": {"start": <int>, "end": <int>}
          }
        }
      ],
      "overall_correctness": "patch is correct" | "patch is incorrect",
      "overall_explanation": "<1-3 sentence explanation justifying the overall_correctness verdict>",
      "overall_confidence_score": <float 0.0-1.0>
    }
    ```
    
    ## Questions
    
    ### Why separate out its own message history?
    
    We want the review thread to match the training of our review models as
    much as possible -- that means using a custom prompt, removing user
    instructions, and starting a clean chat history.
    
    We also want to make sure the review thread doesn't leak into the parent
    thread.
    
    ### Why do this as a mode, vs. sub-agents?
    
    1. We want review to be a synchronous task, so it's fine for now to do a
    bespoke implementation.
    2. We're still unclear about the final structure for sub-agents. We'd
    prefer to land this quickly and then refactor into sub-agents without
    rushing that implementation.
  • feat: context compaction (#3446)
    ## Compact feature:
    1. Stops the model when the context window become too large
    2. Add a user turn, asking for the model to summarize
    3. Build a bridge that contains all the previous user message + the
    summary. Rendered from a template
    4. Start sampling again from a clean conversation with only that bridge
  • feat: reasoning effort as optional (#3527)
    Allow the reasoning effort to be optional
  • bug: fix model save (#3525)
    Fix those 2 behaviors:
    1. The model does not get saved if we don't CTRL + S
    2. The reasoning effort get saved
  • feat: include reasoning_effort in NewConversationResponse (#3506)
    `ClientRequest::NewConversation` picks up the reasoning level from the user's defaults in `config.toml`, so it should be reported in `NewConversationResponse`.
  • fix: improve handle_sandbox_error timeouts (#3435)
    ## Summary
    Handle timeouts the same way, regardless of approval mode. There's more
    to do here, but this is simple and should be zero-regret
    
    ## Testing
    - [x] existing tests pass
    - [x] test locally and verify rollout
  • chore: enable clippy::redundant_clone (#3489)
    Created this PR by:
    
    - adding `redundant_clone` to `[workspace.lints.clippy]` in
    `cargo-rs/Cargol.toml`
    - running `cargo clippy --tests --fix`
    - running `just fmt`
    
    Though I had to clean up one instance of the following that resulted:
    
    ```rust
    let codex = codex;
    ```
  • Add Compact and Turn Context to the rollout items (#3444)
    Adding compact and turn context to the rollout items
    
    based on #3440
  • Change forking to read the rollout from file (#3440)
    This PR changes get history op to get path. Then, forking will use a
    path. This will help us have one unified codepath for resuming/forking
    conversations. Will also help in having rollout history in order. It
    also fixes a bug where you won't see the UI when resuming after forking.
  • Unified execution (#3288)
    ## Unified PTY-Based Exec Tool
    
    Note: this requires to have this flag in the config:
    `use_experimental_unified_exec_tool=true`
    
    - Adds a PTY-backed interactive exec feature (“unified_exec”) with
    session reuse via
      session_id, bounded output (128 KiB), and timeout clamping (≤ 60 s).
    - Protocol: introduces ResponseItem::UnifiedExec { session_id,
    arguments, timeout_ms }.
    - Tools: exposes unified_exec as a function tool (Responses API);
    excluded from Chat
      Completions payload while still supported in tool lists.
    - Path handling: resolves commands via PATH (or explicit paths), with
    UTF‑8/newline‑aware
      truncation (truncate_middle).
    - Tests: cover command parsing, path resolution, session
    persistence/cleanup, multi‑session
      isolation, timeouts, and truncation behavior.
  • Persist model & reasoning changes (#2799)
    Persists `/model` changes across both general and profile-specific
    sessions.
  • Back out "feat: POSIX unification and snapshot sessions (#3179)" (#3430)
    This reverts https://github.com/openai/codex/pull/3179.
    
    #3179 appears to introduce a regression where sourcing dotfiles causes a
    bunch of activity in the title bar (and potentially slows things down?)
    
    
    https://github.com/user-attachments/assets/a68f7fb3-0749-4e0e-a321-2aa6993e01da
    
    Verified this no longer happens after backing out #3179.
    
    Original commit changeset: 62bd0e3d9d
  • Move initial history to protocol (#3422)
    To fix an edge case of forking then resuming
    
    #3419
  • Introduce rollout items (#3380)
    This PR introduces Rollout items. This enable us to rollout eventmsgs
    and session meta.
    
    This is mostly #3214 with rebase on main
  • fix: include rollout_path in NewConversationResponse (#3352)
    Adding the `rollout_path` to the `NewConversationResponse` makes it so a
    client can perform subsequent operations on a `(ConversationId,
    PathBuf)` pair. #3353 will introduce support for `ArchiveConversation`.
    
    ---
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/3352).
    * #3353
    * __->__ #3352
  • feat: POSIX unification and snapshot sessions (#3179)
    ## Session snapshot
    For POSIX shell, the goal is to take a snapshot of the interactive shell
    environment, store it in a session file located in `.codex/` and only
    source this file for every command that is run.
    As a result, if a snapshot files exist, `bash -lc <CALL>` get replaced
    by `bash -c <CALL>`.
    
    This also fixes the issue that `bash -lc` does not source `.bashrc`,
    resulting in missing env variables and aliases in the codex session.
    ## POSIX unification
    Unify `bash` and `zsh` shell into a POSIX shell. The rational is that
    the tool will not use any `zsh` specific capabilities.
    
    ---------
    
    Co-authored-by: Michael Bolin <mbolin@openai.com>
  • Generate more typescript types and return conversation id with ConversationSummary (#3219)
    This PR does multiple things that are necessary for conversation resume
    to work from the extension. I wanted to make sure everything worked so
    these changes wound up in one PR:
    1. Generate more ts types
    2. Resume rollout history files rather than create a new one every time
    it is resumed so you don't see a duplicate conversation in history for
    every resume. Chatted with @aibrahim-oai to verify this
    3. Return conversation_id in conversation summaries
    4. [Cleanup] Use serde and strong types for a lot of the rollout file
    parsing
  • Use ConversationId instead of raw Uuids (#3282)
    We're trying to migrate from `session_id: Uuid` to `conversation_id:
    ConversationId`. Not only does this give us more type safety but it
    unifies our terminology across Codex and with the implementation of
    session resuming, a conversation (which can span multiple sessions) is
    more appropriate.
    
    I started this impl on https://github.com/openai/codex/pull/3219 as part
    of getting resume working in the extension but it's big enough that it
    should be broken out.
  • Move token usage/context information to session level (#3221)
    Move context information into the main loop so it can be used to
    interrupt the loop or start auto-compaction.
  • Never store requests (#3212)
    When item ids are sent to Responses API it will load them from the
    database ignoring the provided values. This adds extra latency.
    
    Not having the mode to store requests also allows us to simplify the
    code.
    
    ## Breaking change
    
    The `disable_response_storage` configuration option is removed.
  • fix: add callback to map before sending request to fix race condition (#3146)
    Last week, I thought I found the smoking gun in our flaky integration
    tests where holding these locks could have led to potential deadlock:
    
    - https://github.com/openai/codex/pull/2876
    - https://github.com/openai/codex/pull/2878
    
    Yet even after those PRs went in, we continued to see flakinees in our
    integration tests! Though with the additional logging added as part of
    debugging those tests, I now saw things like:
    
    ```
    read message from stdout: Notification(JSONRPCNotification { jsonrpc: "2.0", method: "codex/event/exec_approval_request", params: Some(Object {"id": String("0"), "msg": Object {"type": String("exec_approval_request"), "call_id": String("call1"), "command": Array [String("python3"), String("-c"), String("print(42)")], "cwd": String("/tmp/.tmpFj2zwi/workdir")}, "conversationId": String("c67b32c5-9475-41bf-8680-f4b4834ebcc6")}) })
    notification: Notification(JSONRPCNotification { jsonrpc: "2.0", method: "codex/event/exec_approval_request", params: Some(Object {"id": String("0"), "msg": Object {"type": String("exec_approval_request"), "call_id": String("call1"), "command": Array [String("python3"), String("-c"), String("print(42)")], "cwd": String("/tmp/.tmpFj2zwi/workdir")}, "conversationId": String("c67b32c5-9475-41bf-8680-f4b4834ebcc6")}) })
    read message from stdout: Request(JSONRPCRequest { id: Integer(0), jsonrpc: "2.0", method: "execCommandApproval", params: Some(Object {"conversation_id": String("c67b32c5-9475-41bf-8680-f4b4834ebcc6"), "call_id": String("call1"), "command": Array [String("python3"), String("-c"), String("print(42)")], "cwd": String("/tmp/.tmpFj2zwi/workdir")}) })
    writing message to stdin: Response(JSONRPCResponse { id: Integer(0), jsonrpc: "2.0", result: Object {"decision": String("approved")} })
    in read_stream_until_notification_message(codex/event/task_complete)
    [mcp stderr] 2025-09-04T00:00:59.738585Z  INFO codex_mcp_server::message_processor: <- response: JSONRPCResponse { id: Integer(0), jsonrpc: "2.0", result: Object {"decision": String("approved")} }
    [mcp stderr] 2025-09-04T00:00:59.738740Z DEBUG codex_core::codex: Submission sub=Submission { id: "1", op: ExecApproval { id: "0", decision: Approved } }
    [mcp stderr] 2025-09-04T00:00:59.738832Z  WARN codex_core::codex: No pending approval found for sub_id: 0
    ```
    
    That is, a response was sent for a request, but no callback was in place
    to handle the response!
    
    This time, I think I may have found the underlying issue (though the
    fixes for holding locks for too long may have also been part of it),
    which is I found cases where we were sending the request:
    
    
    https://github.com/openai/codex/blob/234c0a0469db222f05df08d00ae5032312f77427/codex-rs/core/src/codex.rs#L597
    
    before inserting the `Sender` into the `pending_approvals` map (which
    has to wait on acquiring a mutex):
    
    
    https://github.com/openai/codex/blob/234c0a0469db222f05df08d00ae5032312f77427/codex-rs/core/src/codex.rs#L598-L601
    
    so it is possible the request could go out and the client could respond
    before `pending_approvals` was updated!
    
    Note this was happening in both `request_command_approval()` and
    `request_patch_approval()`, which maps to the sorts of errors we have
    been seeing when these integration tests have been flaking on us.
    
    While here, I am also adding some extra logging that prints if inserting
    into `pending_approvals` overwrites an entry as opposed to purely
    inserting one. Today, a conversation can have only one pending request
    at a time, but as we are planning to support parallel tool calls, this
    invariant may not continue to hold, in which case we need to revisit
    this abstraction.
  • Dividing UserMsgs into categories to send it back to the tui (#3127)
    This PR does the following:
    
    - divides user msgs into 3 categories: plain, user instructions, and
    environment context
    - Centralizes adding user instructions and environment context to a
    degree
    - Improve the integration testing
    
    Building on top of #3123
    
    Specifically this
    [comment](https://github.com/openai/codex/pull/3123#discussion_r2319885089).
    We need to send the user message while ignoring the User Instructions
    and Environment Context we attach.
  • Replay EventMsgs from Response Items when resuming a session with history. (#3123)
    ### Overview
    
    This PR introduces the following changes:
    	1.	Adds a unified mechanism to convert ResponseItem into EventMsg.
    2. Ensures that when a session is initialized with initial history, a
    vector of EventMsg is sent along with the session configuration. This
    allows clients to re-render the UI accordingly.
    	3. 	Added integration testing
    
    ### Caveats
    
    This implementation does not send every EventMsg that was previously
    dispatched to clients. The excluded events fall into two categories:
    	•	“Arguably” rolled-out events
    Examples include tool calls and apply-patch calls. While these events
    are conceptually rolled out, we currently only roll out ResponseItems.
    These events are already being handled elsewhere and transformed into
    EventMsg before being sent.
    	•	Non-rolled-out events
    Certain events such as TurnDiff, Error, and TokenCount are not rolled
    out at all.
    
    ### Future Directions
    
    At present, resuming a session involves maintaining two states:
    	•	UI State
    Clients can replay most of the important UI from the provided EventMsg
    history.
    	•	Model State
    The model receives the complete session history to reconstruct its
    internal state.
    
    This design provides a solid foundation. If, in the future, more precise
    UI reconstruction is needed, we have two potential paths:
    1. Introduce a third data structure that allows us to derive both
    ResponseItems and EventMsgs.
    2. Clearly divide responsibilities: the core system ensures the
    integrity of the model state, while clients are responsible for
    reconstructing the UI.
  • Move CodexAuth and AuthManager to the core crate (#3074)
    Fix a long standing layering issue.
  • chore: unify history loading (#2736)
    We have two ways of loading conversation with a previous history. Fork
    conversation and the experimental resume that we had before. In this PR,
    I am unifying their code path. The path is getting the history items and
    recording them in a brand new conversation. This PR also constraint the
    rollout recorder responsibilities to be only recording to the disk and
    loading from the disk.
    
    The PR also fixes a current bug when we have two forking in a row:
    History 1:
    <Environment Context>
    UserMessage_1
    UserMessage_2
    UserMessage_3
    
    **Fork with n = 1 (only remove one element)**
    History 2:
    <Environment Context>
    UserMessage_1
    UserMessage_2
    <Environment Context>
    
    **Fork with n = 1 (only remove one element)**
    History 2:
    <Environment Context>
    UserMessage_1
    UserMessage_2
    **<Environment Context>**
    
    This shouldn't happen but because we were appending the `<Environment
    Context>` after each spawning and it's considered as _user message_.
    Now, we don't add this message if restoring and old conversation.
  • Following up on #2371 post commit feedback (#2852)
    - Introduce websearch end to complement the begin 
    - Moves the logic of adding the sebsearch tool to
    create_tools_json_for_responses_api
    - Making it the client responsibility to toggle the tool on or off 
    - Other misc in #2371 post commit feedback
    - Show the query:
    
    <img width="1392" height="151" alt="image"
    src="https://github.com/user-attachments/assets/8457f1a6-f851-44cf-bcca-0d4fe460ce89"
    />
  • Custom /prompts (#2696)
    Adds custom `/prompts` to `~/.codex/prompts/<command>.md`.
    
    <img width="239" height="107" alt="Screenshot 2025-08-25 at 6 22 42 PM"
    src="https://github.com/user-attachments/assets/fe6ebbaa-1bf6-49d3-95f9-fdc53b752679"
    />
    
    ---
    
    Details:
    
    1. Adds `Op::ListCustomPrompts` to core.
    2. Returns `ListCustomPromptsResponse` with list of `CustomPrompt`
    (name, content).
    3. TUI calls the operation on load, and populates the custom prompts
    (excluding prompts that collide with builtins).
    4. Selecting the custom prompt automatically sends the prompt to the
    agent.