Commit Graph

307 Commits

  • Add remote models feature flag (#7648)
    # External (non-OpenAI) Pull Request Requirements
    
    Before opening this Pull Request, please read the dedicated
    "Contributing" markdown file or your PR may be closed:
    https://github.com/openai/codex/blob/main/docs/contributing.md
    
    If your PR conforms to our contribution guidelines, replace this text
    with a detailed and high quality description of your changes.
    
    Include a link to a bug report or enhancement request.
  • Wire with_remote_overrides to construct model families (#7621)
    - This PR wires `with_remote_overrides` and make the
    `construct_model_families` an async function
    - Moves getting model family a level above to keep the function `sync`
    - Updates the tests to local, offline, and `sync` helper for model
    families
  • fix: taking plan type from usage endpoint instead of thru auth token (#7610)
    pull plan type from the usage endpoint, persist it in session state /
    tui state, and propagate through rate limit snapshots
  • Call models endpoint in models manager (#7616)
    - Introduce `with_remote_overrides` and update
    `refresh_available_models`
    - Put `auth_manager` instead of `auth_mode` on `models_manager`
    - Remove `ShellType` and `ReasoningLevel` to use already existing
    structs
  • chore: refactor to move Arc<RwLock> concern outside exec_policy_for (#7615)
    The caller should decide whether wrapping the policy in `Arc<RwLock>` is
    necessary. This should make https://github.com/openai/codex/pull/7609 a
    bit smoother.
    
    - `exec_policy_for()` -> `load_exec_policy_for_features()`
    - introduce `load_exec_policy()` that does not take `Features` as an arg
    - both return `Result<Policy, ExecPolicyError>` instead of
    Result<Arc<RwLock<Policy>>, ExecPolicyError>`
    
    This simplifies the tests as they have no need for `Arc<RwLock>`.
  • Inline response recording and remove process_items indirection (#7310)
    - Inline response recording during streaming: `run_turn` now records
    items as they arrive instead of building a `ProcessedResponseItem` list
    and post‑processing via `process_items`.
    - Simplify turn handling: `handle_output_item_done` returns the
    follow‑up signal + optional tool future; `needs_follow_up` is set only
    there, and in‑flight tool futures are drained once at the end (errors
    logged, no extra state writes).
    - Flattened stream loop: removed `process_items` indirection and the
    extra output queue
    - - Tests: relaxed `tool_parallelism::tool_results_grouped` to allow any
    completion order while still requiring matching call/output IDs.
  • remove model_family from `config (#7571)
    - Remove `model_family` from `config`
    - Make sure to still override config elements related to `model_family`
    like supporting reasoning
  • Refactor execpolicy fallback evaluation (#7544)
    ## Refactor of the `execpolicy` crate
    
    To illustrate why we need this refactor, consider an agent attempting to
    run `apple | rm -rf ./`. Suppose `apple` is allowed by `execpolicy`.
    Before this PR, `execpolicy` would consider `apple` and `pear` and only
    render one rule match: `Allow`. We would skip any heuristics checks on
    `rm -rf ./` and immediately approve `apple | rm -rf ./` to run.
    
    To fix this, we now thread a `fallback` evaluation function into
    `execpolicy` that runs when no `execpolicy` rules match a given command.
    In our example, we would run `fallback` on `rm -rf ./` and prevent
    `apple | rm -rf ./` from being run without approval.
  • whitelist command prefix integration in core and tui (#7033)
    this PR enables TUI to approve commands and add their prefixes to an
    allowlist:
    <img width="708" height="605" alt="Screenshot 2025-11-21 at 4 18 07 PM"
    src="https://github.com/user-attachments/assets/56a19893-4553-4770-a881-becf79eeda32"
    />
    
    note: we only show the option to whitelist the command when 
    1) command is not multi-part (e.g `git add -A && git commit -m 'hello
    world'`)
    2) command is not already matched by an existing rule
  • Migrate model family to models manager (#7565)
    This PR moves `ModelsFamily` to `openai_models`. It also propagates
    `ModelsManager` to session services and use it to drive model family. We
    also make `derive_default_model_family` private because it's a step
    towards what we want: one place that gives model configuration.
    
    This is a second step at having one source of truth for models
    information and config: `ModelsManager`.
    
    Next steps would be to remove `ModelsFamily` from config. That's massive
    because it's being used in 41 occasions mostly pre launching `codex`.
    Also, we need to make `find_family_for_model` private. It's also big
    because it's being used in 21 occasions ~ all tests.
  • fix: Features should be immutable over the lifetime of a session/thread (#7540)
    I noticed that `features: Features` was defined on `struct
    SessionConfiguration`, which is commonly owned by `SessionState`, which
    is in turn owned by `Session`.
    
    Though I do not believe that `Features` should be allowed to be modified
    over the course of a session (if the feature state is not invariant, it
    makes it harder to reason about), which argues that it should live on
    `Session` rather than `SessionState` or `SessionConfiguration`.
    
    This PR moves `Features` to `Session` and updates all call sites. It
    appears the only place we were mutating `Features` was:
    
    - in tests
    - the sub-agent config for a review task:
    
    
    https://github.com/openai/codex/blob/3ef76ff29d5eed258fb6b8550e0e2b973d0dca21/codex-rs/core/src/tasks/review.rs#L86-L89
    
    Note this change also means it is no longer an `async` call to check the
    state of a feature, eliminating the possibility of a
    [TOCTTOU](https://en.wikipedia.org/wiki/Time-of-check_to_time-of-use)
    error between checking the state of a feature and acting on it:
    
    
    https://github.com/openai/codex/blob/3ef76ff29d5eed258fb6b8550e0e2b973d0dca21/codex-rs/core/src/codex.rs#L1069-L1076
  • Migrate model preset (#7542)
    - Introduce `openai_models` in `/core`
    - Move `PRESETS` under it
    - Move `ModelPreset`, `ModelUpgrade`, `ReasoningEffortPreset`,
    `ReasoningEffortPreset`, and `ReasoningEffortPreset` to `protocol`
    - Introduce `Op::ListModels` and `EventMsg::AvailableModels`
    
    Next steps:
    - migrate `app-server` and `tui` to use the introduced Operation
  • feat: support list mcp servers in app server (#7505)
    ### Summary
    Added `mcp/servers/list` which is equivalent to `/mcp` slash command in
    CLI for response. This will be used in VSCE MCP settings to show log in
    status, available tools etc.
  • feat: retroactive image placeholder to prevent poisoning (#6774)
    If an image can't be read by the API, it will poison the entire history,
    preventing any new turn on the conversation.
    This detect such cases and replace the image by a placeholder
  • persisting credits if new snapshot does not contain credit info (#7490)
    in response to incoming changes to responses headers where the header
    may sometimes not contain credits info (no longer forcing a credit
    check)
  • feat: add warning message for the model (#7445)
    Add a warning message as a user turn to the model if the model does not
    behave as expected (here, for example, if the model opens too many
    `unified_exec` sessions)
  • chore: improve rollout session init errors (#7336)
    Title: Improve rollout session initialization error messages
    
    Issue: https://github.com/openai/codex/issues/7283
    
    What: add targeted mapping for rollout/session initialization errors so
    users get actionable messages when Codex cannot access session files.
    
    Why: session creation previously returned a generic internal error,
    hiding permissions/FS issues and making support harder.
    
    How:
    - Added rollout::error::map_session_init_error to translate the more
    common io::Error kinds into user-facing hints (permission, missing dir,
    file blocking, corruption). Others are passed through directly with
    `CodexErr::Fatal`.
    - Reused the mapper in Codex session creation to preserve root causes
    instead of returning InternalAgentDied.
  • fix: Correct the stream error message (#7266)
    Fixes a copy paste bug with the error handling in  `try_run_turn`
    
    I have read the CLA Document and I hereby sign the CLA
  • Account for encrypted reasoning for auto compaction (#7113)
    - The total token used returned from the api doesn't account for the
    reasoning items before the assistant message
    - Account for those for auto compaction
    - Add the encrypted reasoning effort in the common tests utils
    - Add a test to make sure it works as expected
  • feat: declare server capability in shell-tool-mcp (#7112)
    This introduces a new feature to Codex when it operates as an MCP
    _client_ where if an MCP _server_ replies that it has an entry named
    `"codex/sandbox-state"` in its _server capabilities_, then Codex will
    send it an MCP notification with the following structure:
    
    ```json
    {
      "method": "codex/sandbox-state/update",
      "params": {
        "sandboxPolicy": {
          "type": "workspace-write",
          "network-access": false,
          "exclude-tmpdir-env-var": false
          "exclude-slash-tmp": false
        },
        "codexLinuxSandboxExe": null,
        "sandboxCwd": "/Users/mbolin/code/codex2"
      }
    }
    ```
    
    or with whatever values are appropriate for the initial `sandboxPolicy`.
    
    **NOTE:** Codex _should_ continue to send the MCP server notifications
    of the same format if these things change over the lifetime of the
    thread, but that isn't wired up yet.
    
    The result is that `shell-tool-mcp` can consume these values so that
    when it calls `codex_core::exec::process_exec_tool_call()` in
    `codex-rs/exec-server/src/posix/escalate_server.rs`, it is now sure to
    call it with the correct values (whereas previously we relied on
    hardcoded values).
    
    While I would argue this is a supported use case within the MCP
    protocol, the `rmcp` crate that we are using today does not support
    custom notifications. As such, I had to patch it and I submitted it for
    review, so hopefully it will be accepted in some form:
    
    https://github.com/modelcontextprotocol/rust-sdk/pull/556
    
    To test out this change from end-to-end:
    
    - I ran `cargo build` in `~/code/codex2/codex-rs/exec-server`
    - I built the fork of Bash in `~/code/bash/bash`
    - I added the following to my `~/.codex/config.toml`:
    
    ```toml
    # Use with `codex --disable shell_tool`.
    [mcp_servers.execshell]
    args = ["--bash", "/Users/mbolin/code/bash/bash"]
    command = "/Users/mbolin/code/codex2/codex-rs/target/debug/codex-exec-mcp-server"
    ```
    
    - From `~/code/codex2/codex-rs`, I ran `just codex --disable shell_tool`
    - When the TUI started up, I verified that the sandbox mode is
    `workspace-write`
    - I ran `/mcp` to verify that the shell tool from the MCP is there:
    
    <img width="1387" height="1400" alt="image"
    src="https://github.com/user-attachments/assets/1a8addcc-5005-4e16-b59f-95cfd06fd4ab"
    />
    
    - Then I asked it:
    
    > what is the output of `gh issue list`
    
    because this should be auto-approved with our existing dummy policy:
    
    
    https://github.com/openai/codex/blob/af63e6eccc35783f1bf4dca3c61adb090efb6b8a/codex-rs/exec-server/src/posix.rs#L157-L164
    
    And it worked:
    
    <img width="1387" height="1400" alt="image"
    src="https://github.com/user-attachments/assets/7568d2f7-80da-4d68-86d0-c265a6f5e6c1"
    />
  • support MCP elicitations (#6947)
    No support for request schema yet, but we'll at least show the message
    and allow accept/decline.
    
    <img width="823" height="551" alt="Screenshot 2025-11-21 at 2 44 05 PM"
    src="https://github.com/user-attachments/assets/6fbb892d-ca12-4765-921e-9ac4b217534d"
    />
  • [app-server] update doc with codex error info (#6941)
    Document new codex error info. Also fixed the name from
    `codex_error_code` to `codex_error_info`.
  • feat: update process_exec_tool_call() to take a cancellation token (#6972)
    This updates `ExecParams` so that instead of taking `timeout_ms:
    Option<u64>`, it now takes a more general cancellation mechanism,
    `ExecExpiration`, which is an enum that includes a
    `Cancellation(tokio_util::sync::CancellationToken)` variant.
    
    If the cancellation token is fired, then `process_exec_tool_call()`
    returns in the same way as if a timeout was exceeded.
    
    This is necessary so that in #6973, we can manage the timeout logic
    external to the `process_exec_tool_call()` because we want to "suspend"
    the timeout when an elicitation from a human user is pending.
    
    
    
    
    
    
    
    
    ---
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/6972).
    * #7005
    * #6973
    * __->__ #6972
  • [app-server & core] introduce new codex error code and v2 app-server error events (#6938)
    This PR does two things:
    1. populate a new `codex_error_code` protocol in error events sent from
    core to client;
    2. old v1 core events `codex/event/stream_error` and `codex/event/error`
    will now both become `error`. We also show codex error code for
    turncompleted -> error status.
    
    new events in app server test:
    ```
    < {
    <   "method": "codex/event/stream_error",
    <   "params": {
    <     "conversationId": "019aa34c-0c14-70e0-9706-98520a760d67",
    <     "id": "0",
    <     "msg": {
    <       "codex_error_code": {
    <         "response_stream_disconnected": {
    <           "http_status_code": 401
    <         }
    <       },
    <       "message": "Reconnecting... 2/5",
    <       "type": "stream_error"
    <     }
    <   }
    < }
    
     {
    <   "method": "error",
    <   "params": {
    <     "error": {
    <       "codexErrorCode": {
    <         "responseStreamDisconnected": {
    <           "httpStatusCode": 401
    <         }
    <       },
    <       "message": "Reconnecting... 2/5"
    <     }
    <   }
    < }
    
    < {
    <   "method": "turn/completed",
    <   "params": {
    <     "turn": {
    <       "error": {
    <         "codexErrorCode": {
    <           "responseTooManyFailedAttempts": {
    <             "httpStatusCode": 401
    <           }
    <         },
    <         "message": "exceeded retry limit, last status: 401 Unauthorized, request id: 9a1b495a1a97ed3e-SJC"
    <       },
    <       "id": "0",
    <       "items": [],
    <       "status": "failed"
    <     }
    <   }
    < }
    ```
  • Always fallback to real shell (#6953)
    Either cmd.exe or `/bin/sh`.
  • [app-server] feat: v2 apply_patch approval flow (#6760)
    This PR adds the API V2 version of the apply_patch approval flow, which
    centers around `ThreadItem::FileChange`.
    
    This PR wires the new RPC (`item/fileChange/requestApproval`, V2 only)
    and related events (`item/started`, `item/completed` for
    `ThreadItem::FileChange`, which are emitted in both V1 and V2) through
    the app-server
    protocol. The new approval RPC is only sent when the user initiates a
    turn with the new `turn/start` API so we don't break backwards
    compatibility with VSCE.
    
    Similar to https://github.com/openai/codex/pull/6758, the approach I
    took was to make as few changes to the Codex core as possible,
    leveraging existing `EventMsg` core events, and translating those in
    app-server. I did have to add a few additional fields to
    `EventMsg::PatchApplyBegin` and `EventMsg::PatchApplyEnd`, but those
    were fairly lightweight.
    
    However, the `EventMsg`s emitted by core are the following:
    ```
    1) Auto-approved (no request for approval)

    - EventMsg::PatchApplyBegin
    - EventMsg::PatchApplyEnd
    
    2) Approved by user
    - EventMsg::ApplyPatchApprovalRequest
    - EventMsg::PatchApplyBegin
    - EventMsg::PatchApplyEnd
    
    3) Declined by user
    - EventMsg::ApplyPatchApprovalRequest
    - EventMsg::PatchApplyBegin
    - EventMsg::PatchApplyEnd
    ```
    
    For a request triggering an approval, this would result in:
    ```
    item/fileChange/requestApproval
    item/started
    item/completed
    ```
    
    which is different from the `ThreadItem::CommandExecution` flow
    introduced in https://github.com/openai/codex/pull/6758, which does the
    below and is preferable:
    ```
    item/started
    item/commandExecution/requestApproval
    item/completed
    ```
    
    To fix this, we leverage `TurnSummaryStore` on codex_message_processor
    to store a little bit of state, allowing us to fire `item/started` and
    `item/fileChange/requestApproval` whenever we receive the underlying
    `EventMsg::ApplyPatchApprovalRequest`, and no-oping when we receive the
    `EventMsg::PatchApplyBegin` later.
    
    This is much less invasive than modifying the order of EventMsg within
    core (I tried).
    
    The resulting payloads:
    ```
    {
      "method": "item/started",
      "params": {
        "item": {
          "changes": [
            {
              "diff": "Hello from Codex!\n",
              "kind": "add",
              "path": "/Users/owen/repos/codex/codex-rs/APPROVAL_DEMO.txt"
            }
          ],
          "id": "call_Nxnwj7B3YXigfV6Mwh03d686",
          "status": "inProgress",
          "type": "fileChange"
        }
      }
    }
    ```
    
    ```
    {
      "id": 0,
      "method": "item/fileChange/requestApproval",
      "params": {
        "grantRoot": null,
        "itemId": "call_Nxnwj7B3YXigfV6Mwh03d686",
        "reason": null,
        "threadId": "019a9e11-8295-7883-a283-779e06502c6f",
        "turnId": "1"
      }
    }
    ```
    
    ```
    {
      "id": 0,
      "result": {
        "decision": "accept"
      }
    }
    ```
    
    ```
    {
      "method": "item/completed",
      "params": {
        "item": {
          "changes": [
            {
              "diff": "Hello from Codex!\n",
              "kind": "add",
              "path": "/Users/owen/repos/codex/codex-rs/APPROVAL_DEMO.txt"
            }
          ],
          "id": "call_Nxnwj7B3YXigfV6Mwh03d686",
          "status": "completed",
          "type": "fileChange"
        }
      }
    }
    ```
  • Revert "[core] add optional status_code to error events (#6865)" (#6955)
    This reverts commit c2ec477d93.
    
    # External (non-OpenAI) Pull Request Requirements
    
    Before opening this Pull Request, please read the dedicated
    "Contributing" markdown file or your PR may be closed:
    https://github.com/openai/codex/blob/main/docs/contributing.md
    
    If your PR conforms to our contribution guidelines, replace this text
    with a detailed and high quality description of your changes.
    
    Include a link to a bug report or enhancement request.
  • execpolicy2 core integration (#6641)
    This PR threads execpolicy2 into codex-core.
    
    activated via feature flag: exec_policy (on by default)
    
    reads and parses all .codexpolicy files in `codex_home/codex`
    
    refactored tool runtime API to integrate execpolicy logic
    
    ---------
    
    Co-authored-by: Michael Bolin <mbolin@openai.com>
  • [core] add optional status_code to error events (#6865)
    We want to better uncover error status code for clients. Add an optional
    status_code to error events (thread error, error, stream error) so app
    server could uncover the status code from the client side later.
    
    in event log:
    ```
    < {
    <   "method": "codex/event/stream_error",
    <   "params": {
    <     "conversationId": "019a9a32-f576-7292-9711-8e57e8063536",
    <     "id": "0",
    <     "msg": {
    <       "message": "Reconnecting... 5/5",
    <       "status_code": 401,
    <       "type": "stream_error"
    <     }
    <   }
    < }
    < {
    <   "method": "codex/event/error",
    <   "params": {
    <     "conversationId": "019a9a32-f576-7292-9711-8e57e8063536",
    <     "id": "0",
    <     "msg": {
    <       "message": "exceeded retry limit, last status: 401 Unauthorized, request id: 9a0cb03a485067f7-SJC",
    <       "status_code": 401,
    <       "type": "error"
    <     }
    <   }
    < }
    ```
  • fix(core) Support changing /approvals before conversation (#6836)
    ## Summary
    Setting `/approvals` before the start of a conversation was not updating
    the environment_context for a conversation. Not sure exactly when this
    problem was introduced, but this should reduce model confusion
    dramatically.
    
    ## Testing
    - [x] Added unit test to reproduce bug, confirmed fix with update
    - [x] Tested locally
  • Move shell to use truncate_text (#6842)
    Move shell to use the configurable `truncate_text`
    
    ---------
    
    Co-authored-by: pakrym-oai <pakrym@openai.com>
  • fix: add more fields to ThreadStartResponse and ThreadResumeResponse (#6847)
    This adds the following fields to `ThreadStartResponse` and
    `ThreadResumeResponse`:
    
    ```rust
        pub model: String,
        pub model_provider: String,
        pub cwd: PathBuf,
        pub approval_policy: AskForApproval,
        pub sandbox: SandboxPolicy,
        pub reasoning_effort: Option<ReasoningEffort>,
    ```
    
    This is important because these fields are optional in
    `ThreadStartParams` and `ThreadResumeParams`, so the caller needs to be
    able to determine what values were ultimately used to start/resume the
    conversation. (Though note that any of these could be changed later
    between turns in the conversation.)
    
    Though to get this information reliably, it must be read from the
    internal `SessionConfiguredEvent` that is created in response to the
    start of a conversation. Because `SessionConfiguredEvent` (as defined in
    `codex-rs/protocol/src/protocol.rs`) did not have all of these fields, a
    number of them had to be added as part of this PR.
    
    Because `SessionConfiguredEvent` is referenced in many tests, test
    instances of `SessionConfiguredEvent` had to be updated, as well, which
    is why this PR touches so many files.
  • Add the utility to truncate by tokens (#6746)
    - This PR is to make it on path for truncating by tokens. This path will
    be initially used by unified exec and context manager (responsible for
    MCP calls mainly).
    - We are exposing new config `calls_output_max_tokens`
    - Use `tokens` as the main budget unit but truncate based on the model
    family by Introducing `TruncationPolicy`.
    - Introduce `truncate_text` as a router for truncation based on the
    mode.
    
    In next PRs:
    - remove truncate_with_line_bytes_budget
    - Add the ability to the model to override the token budget.
  • feat: remote compaction (#6795)
    Co-authored-by: pakrym-oai <pakrym@openai.com>
  • [app-server] feat: add v2 command execution approval flow (#6758)
    This PR adds the API V2 version of the command‑execution approval flow
    for the shell tool.
    
    This PR wires the new RPC (`item/commandExecution/requestApproval`, V2
    only) and related events (`item/started`, `item/completed`, and
    `item/commandExecution/delta`, which are emitted in both V1 and V2)
    through the app-server
    protocol. The new approval RPC is only sent when the user initiates a
    turn with the new `turn/start` API so we don't break backwards
    compatibility with VSCE.
    
    The approach I took was to make as few changes to the Codex core as
    possible, leveraging existing `EventMsg` core events, and translating
    those in app-server. I did have to add additional fields to
    `EventMsg::ExecCommandEndEvent` to capture the command's input so that
    app-server can statelessly transform these events to a
    `ThreadItem::CommandExecution` item for the `item/completed` event.
    
    Once we stabilize the API and it's complete enough for our partners, we
    can work on migrating the core to be aware of command execution items as
    a first-class concept.
    
    **Note**: We'll need followup work to make sure these APIs work for the
    unified exec tool, but will wait til that's stable and landed before
    doing a pass on app-server.
    
    Example payloads below:
    ```
    {
      "method": "item/started",
      "params": {
        "item": {
          "aggregatedOutput": null,
          "command": "/bin/zsh -lc 'touch /tmp/should-trigger-approval'",
          "cwd": "/Users/owen/repos/codex/codex-rs",
          "durationMs": null,
          "exitCode": null,
          "id": "call_lNWWsbXl1e47qNaYjFRs0dyU",
          "parsedCmd": [
            {
              "cmd": "touch /tmp/should-trigger-approval",
              "type": "unknown"
            }
          ],
          "status": "inProgress",
          "type": "commandExecution"
        }
      }
    }
    ```
    
    ```
    {
      "id": 0,
      "method": "item/commandExecution/requestApproval",
      "params": {
        "itemId": "call_lNWWsbXl1e47qNaYjFRs0dyU",
        "parsedCmd": [
          {
            "cmd": "touch /tmp/should-trigger-approval",
            "type": "unknown"
          }
        ],
        "reason": "Need to create file in /tmp which is outside workspace sandbox",
        "risk": null,
        "threadId": "019a93e8-0a52-7fe3-9808-b6bc40c0989a",
        "turnId": "1"
      }
    }
    ```
    
    ```
    {
      "id": 0,
      "result": {
        "acceptSettings": {
          "forSession": false
        },
        "decision": "accept"
      }
    }
    ```
    
    ```
    {
      "params": {
        "item": {
          "aggregatedOutput": null,
          "command": "/bin/zsh -lc 'touch /tmp/should-trigger-approval'",
          "cwd": "/Users/owen/repos/codex/codex-rs",
          "durationMs": 224,
          "exitCode": 0,
          "id": "call_lNWWsbXl1e47qNaYjFRs0dyU",
          "parsedCmd": [
            {
              "cmd": "touch /tmp/should-trigger-approval",
              "type": "unknown"
            }
          ],
          "status": "completed",
          "type": "commandExecution"
        }
      }
    }
    ```