Commit Graph

858 Commits

  • fix: Correct the stream error message (#7266)
    Fixes a copy paste bug with the error handling in  `try_run_turn`
    
    I have read the CLA Document and I hereby sign the CLA
  • chore: add cargo-deny configuration (#7119)
    - add GitHub workflow running cargo-deny on push/PR
    - document cargo-deny allowlist with workspace-dep notes and advisory
    ignores
    - align workspace crates to inherit version/edition/license for
    consistent checks
  • Allow enterprises to skip upgrade checks and messages (#7213)
    This is a feature primarily for enterprises who centrally manage Codex
    updates.
  • fix(windows) support apply_patch parsing in powershell (#7221)
    ## Summary
    Support powershell parsing of apply_patch
    
    ## Testing
    - [x] Enable apply_patch unit tests
    
    ---------
    
    Co-authored-by: jif-oai <jif@openai.com>
  • [feedback] Add source info into feedback metadata. (#7140)
    Verified the source info is correctly attached based on whether it's cli
    or vscode.
  • Account for encrypted reasoning for auto compaction (#7113)
    - The total token used returned from the api doesn't account for the
    reasoning items before the assistant message
    - Account for those for auto compaction
    - Add the encrypted reasoning effort in the common tests utils
    - Add a test to make sure it works as expected
  • feat: declare server capability in shell-tool-mcp (#7112)
    This introduces a new feature to Codex when it operates as an MCP
    _client_ where if an MCP _server_ replies that it has an entry named
    `"codex/sandbox-state"` in its _server capabilities_, then Codex will
    send it an MCP notification with the following structure:
    
    ```json
    {
      "method": "codex/sandbox-state/update",
      "params": {
        "sandboxPolicy": {
          "type": "workspace-write",
          "network-access": false,
          "exclude-tmpdir-env-var": false
          "exclude-slash-tmp": false
        },
        "codexLinuxSandboxExe": null,
        "sandboxCwd": "/Users/mbolin/code/codex2"
      }
    }
    ```
    
    or with whatever values are appropriate for the initial `sandboxPolicy`.
    
    **NOTE:** Codex _should_ continue to send the MCP server notifications
    of the same format if these things change over the lifetime of the
    thread, but that isn't wired up yet.
    
    The result is that `shell-tool-mcp` can consume these values so that
    when it calls `codex_core::exec::process_exec_tool_call()` in
    `codex-rs/exec-server/src/posix/escalate_server.rs`, it is now sure to
    call it with the correct values (whereas previously we relied on
    hardcoded values).
    
    While I would argue this is a supported use case within the MCP
    protocol, the `rmcp` crate that we are using today does not support
    custom notifications. As such, I had to patch it and I submitted it for
    review, so hopefully it will be accepted in some form:
    
    https://github.com/modelcontextprotocol/rust-sdk/pull/556
    
    To test out this change from end-to-end:
    
    - I ran `cargo build` in `~/code/codex2/codex-rs/exec-server`
    - I built the fork of Bash in `~/code/bash/bash`
    - I added the following to my `~/.codex/config.toml`:
    
    ```toml
    # Use with `codex --disable shell_tool`.
    [mcp_servers.execshell]
    args = ["--bash", "/Users/mbolin/code/bash/bash"]
    command = "/Users/mbolin/code/codex2/codex-rs/target/debug/codex-exec-mcp-server"
    ```
    
    - From `~/code/codex2/codex-rs`, I ran `just codex --disable shell_tool`
    - When the TUI started up, I verified that the sandbox mode is
    `workspace-write`
    - I ran `/mcp` to verify that the shell tool from the MCP is there:
    
    <img width="1387" height="1400" alt="image"
    src="https://github.com/user-attachments/assets/1a8addcc-5005-4e16-b59f-95cfd06fd4ab"
    />
    
    - Then I asked it:
    
    > what is the output of `gh issue list`
    
    because this should be auto-approved with our existing dummy policy:
    
    
    https://github.com/openai/codex/blob/af63e6eccc35783f1bf4dca3c61adb090efb6b8a/codex-rs/exec-server/src/posix.rs#L157-L164
    
    And it worked:
    
    <img width="1387" height="1400" alt="image"
    src="https://github.com/user-attachments/assets/7568d2f7-80da-4d68-86d0-c265a6f5e6c1"
    />
  • bypass sandbox for policy approved commands (#7110)
    allowing cmds greenlit by execpolicy to bypass sandbox + minor refactor
    for a world where we have execpolicy rules with specific sandbox
    requirements
  • refactor: inline sandbox type lookup in process_exec_tool_call (#7122)
    `process_exec_tool_call()` was taking `SandboxType` as a param, but in
    practice, the only place it was constructed was in
    `codex_message_processor.rs` where it was derived from the other
    `sandbox_policy` param, so this PR inlines the logic that decides the
    `SandboxType` into `process_exec_tool_call()`.
    
    
    
    ---
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/7122).
    * #7112
    * __->__ #7122
  • support MCP elicitations (#6947)
    No support for request schema yet, but we'll at least show the message
    and allow accept/decline.
    
    <img width="823" height="551" alt="Screenshot 2025-11-21 at 2 44 05 PM"
    src="https://github.com/user-attachments/assets/6fbb892d-ca12-4765-921e-9ac4b217534d"
    />
  • Windows: flag some invocations that launch browsers/URLs as dangerous (#7111)
    Prevent certain Powershell/cmd invocations from reaching the sandbox
    when they are trying to launch a browser, or run a command with a URL,
    etc.
  • fix: read max_output_tokens param from config (#4139)
    Request param `max_output_tokens` is documented in
    `https://github.com/openai/codex/blob/main/docs/config.md`,
    but nowhere uses the item in config, this commit read it from config for
    GPT responses API.
    
    see https://github.com/openai/codex/issues/4138 for issue report.
    
    Signed-off-by: Yorling <shallowcloud@yeah.net>
  • Support all types of search actions (#7061)
    Fixes the 
    
    ```
    {
      "error": {
        "message": "Invalid value: 'other'. Supported values are: 'search', 'open_page', and 'find_in_page'.",
        "type": "invalid_request_error",
        "param": "input[150].action.type",
        "code": "invalid_value"
      }
    ```
    error.
    
    
    The actual-actual fix here is supporting absent `query` parameter.
  • Support full powershell paths in is_safe_command (#7055)
    New shell implementation always uses full paths.
  • [app-server] update doc with codex error info (#6941)
    Document new codex error info. Also fixed the name from
    `codex_error_code` to `codex_error_info`.
  • feat: waiting for an elicitation should not count against a shell tool timeout (#6973)
    Previously, we were running into an issue where we would run the `shell`
    tool call with a timeout of 10s, but it fired an elicitation asking for
    user approval, the time the user took to respond to the elicitation was
    counted agains the 10s timeout, so the `shell` tool call would fail with
    a timeout error unless the user is very fast!
    
    This PR addresses this issue by introducing a "stopwatch" abstraction
    that is used to manage the timeout. The idea is:
    
    - `Stopwatch::new()` is called with the _real_ timeout of the `shell`
    tool call.
    - `process_exec_tool_call()` is called with the `Cancellation` variant
    of `ExecExpiration` because it should not manage its own timeout in this
    case
    - the `Stopwatch` expiration is wired up to the `cancel_rx` passed to
    `process_exec_tool_call()`
    - when an elicitation for the `shell` tool call is received, the
    `Stopwatch` pauses
    - because it is possible for multiple elicitations to arrive
    concurrently, it keeps track of the number of "active pauses" and does
    not resume until that counter goes down to zero
    
    I verified that I can test the MCP server using
    `@modelcontextprotocol/inspector` and specify `git status` as the
    `command` with a timeout of 500ms and that the elicitation pops up and I
    have all the time in the world to respond whereas previous to this PR,
    that would not have been possible.
    
    ---
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/6973).
    * #7005
    * __->__ #6973
    * #6972
  • fix flaky tool_call_output_exceeds_limit_truncated_chars_limit (#7043)
    I am suspecting this is flaky because of the wall time can become 0,
    0.1, or 1.
  • feat: update process_exec_tool_call() to take a cancellation token (#6972)
    This updates `ExecParams` so that instead of taking `timeout_ms:
    Option<u64>`, it now takes a more general cancellation mechanism,
    `ExecExpiration`, which is an enum that includes a
    `Cancellation(tokio_util::sync::CancellationToken)` variant.
    
    If the cancellation token is fired, then `process_exec_tool_call()`
    returns in the same way as if a timeout was exceeded.
    
    This is necessary so that in #6973, we can manage the timeout logic
    external to the `process_exec_tool_call()` because we want to "suspend"
    the timeout when an elicitation from a human user is pending.
    
    
    
    
    
    
    
    
    ---
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/6972).
    * #7005
    * #6973
    * __->__ #6972
  • core: make shell behavior portable on FreeBSD (#7039)
    - Use /bin/sh instead of /bin/bash on FreeBSD/OpenBSD in the process
    group timeout test to avoid command-not-found failures.
    
    - Accept /usr/local/bin/bash as a valid SHELL path to match common
    FreeBSD installations.
    
    - Switch the shell serialization duration test to /bin/sh for improved
    portability across Unix platforms.
    
    With this change, `cargo test -p codex-core --lib` runs and passes on
    FreeBSD.
  • [app-server & core] introduce new codex error code and v2 app-server error events (#6938)
    This PR does two things:
    1. populate a new `codex_error_code` protocol in error events sent from
    core to client;
    2. old v1 core events `codex/event/stream_error` and `codex/event/error`
    will now both become `error`. We also show codex error code for
    turncompleted -> error status.
    
    new events in app server test:
    ```
    < {
    <   "method": "codex/event/stream_error",
    <   "params": {
    <     "conversationId": "019aa34c-0c14-70e0-9706-98520a760d67",
    <     "id": "0",
    <     "msg": {
    <       "codex_error_code": {
    <         "response_stream_disconnected": {
    <           "http_status_code": 401
    <         }
    <       },
    <       "message": "Reconnecting... 2/5",
    <       "type": "stream_error"
    <     }
    <   }
    < }
    
     {
    <   "method": "error",
    <   "params": {
    <     "error": {
    <       "codexErrorCode": {
    <         "responseStreamDisconnected": {
    <           "httpStatusCode": 401
    <         }
    <       },
    <       "message": "Reconnecting... 2/5"
    <     }
    <   }
    < }
    
    < {
    <   "method": "turn/completed",
    <   "params": {
    <     "turn": {
    <       "error": {
    <         "codexErrorCode": {
    <           "responseTooManyFailedAttempts": {
    <             "httpStatusCode": 401
    <           }
    <         },
    <         "message": "exceeded retry limit, last status: 401 Unauthorized, request id: 9a1b495a1a97ed3e-SJC"
    <       },
    <       "id": "0",
    <       "items": [],
    <       "status": "failed"
    <     }
    <   }
    < }
    ```
  • Attempt to fix unified_exec_formats_large_output_summary flakiness (#7029)
    second attempt to fix this test after
    https://github.com/openai/codex/pull/6884. I think this flakiness is
    happening because yield_time is too small for a 10,000 step loop in
    python.
  • execpolicycheck command in codex cli (#7012)
    adding execpolicycheck tool onto codex cli
    
    this is useful for validating policies (can be multiple) against
    commands.
    
    it will also surface errors in policy syntax:
    <img width="1150" height="281" alt="Screenshot 2025-11-19 at 12 46
    21 PM"
    src="https://github.com/user-attachments/assets/8f99b403-564c-4172-acc9-6574a8d13dc3"
    />
    
    this PR also changes output format when there's no match in the CLI.
    instead of returning the raw string `noMatch`, we return
    `{"noMatch":{}}`
    
    this PR is a rewrite of: https://github.com/openai/codex/pull/6932 (due
    to the numerous merge conflicts present in the original PR)
    
    ---------
    
    Co-authored-by: Michael Bolin <mbolin@openai.com>
  • increasing user shell timeout to 1 hour (#7025)
    setting user shell timeout to an unreasonably high value since there
    isn't an easy way to have a command run without timeouts
    
    currently, user shell commands timeout is 10 seconds
  • Fix: Improve text encoding for shell output in VSCode preview (#6178) (#6182)
    ## 🐛 Problem
    
    Users running commands with non-ASCII characters (like Russian text
    "пример") in Windows/WSL environments experience garbled text in
    VSCode's shell preview window, with Unicode replacement characters (�)
    appearing instead of the actual text.
    
    **Issue**: https://github.com/openai/codex/issues/6178
    
    ## 🔧 Root Cause
    
    The issue was in `StreamOutput<Vec<u8>>::from_utf8_lossy()` method in
    `codex-rs/core/src/exec.rs`, which used `String::from_utf8_lossy()` to
    convert shell output bytes to strings. This function immediately
    replaces any invalid UTF-8 byte sequences with replacement characters,
    without attempting to decode using other common encodings.
    
    In Windows/WSL environments, shell output often uses encodings like:
    
    - Windows-1252 (common Windows encoding)
    - Latin-1/ISO-8859-1 (extended ASCII)
    
    ## 🛠️ Solution
    
    Replaced the simple `String::from_utf8_lossy()` call with intelligent
    encoding detection via a new `bytes_to_string_smart()` function that
    tries multiple encoding strategies:
    
    1. **UTF-8** (fast path for valid UTF-8)
    2. **Windows-1252** (handles Windows-specific characters in 0x80-0x9F
    range)
    3. **Latin-1** (fallback for extended ASCII)
    4. **Lossy UTF-8** (final fallback, same as before)
    
    ## 📁 Changes
    
    ### New Files
    
    - `codex-rs/core/src/text_encoding.rs` - Smart encoding detection module
    - `codex-rs/core/tests/suite/text_encoding_fix.rs` - Integration tests
    
    ### Modified Files
    
    - `codex-rs/core/src/lib.rs` - Added text_encoding module
    - `codex-rs/core/src/exec.rs` - Updated StreamOutput::from_utf8_lossy()
    - `codex-rs/core/tests/suite/mod.rs` - Registered new test module
    
    ##  Testing
    
    - **5 unit tests** covering UTF-8, Windows-1252, Latin-1, and fallback
    scenarios
    - **2 integration tests** simulating the exact Issue #6178 scenario
    - **Demonstrates improvement** over the previous
    `String::from_utf8_lossy()` approach
    
    All tests pass:
    
    ```bash
    cargo test -p codex-core text_encoding
    cargo test -p codex-core test_shell_output_encoding_issue_6178
    ```
    
    ## 🎯 Impact
    
    -  **Eliminates garbled text** in VSCode shell preview for non-ASCII
    content
    -  **Supports Windows/WSL environments** with proper encoding detection
    -  **Zero performance impact** for UTF-8 text (fast path)
    -  **Backward compatible** - UTF-8 content works exactly as before
    -  **Handles edge cases** with robust fallback mechanism
    
    ## 🧪 Test Scenarios
    
    The fix has been tested with:
    
    - Russian text ("пример")
    - Windows-1252 quotation marks (""test")
    - Latin-1 accented characters ("café")
    - Mixed encoding content
    - Invalid byte sequences (graceful fallback)
    
    ## 📋 Checklist
    
    - [X] Addresses the reported issue
    - [X] Includes comprehensive tests
    - [X] Maintains backward compatibility
    - [X] Follows project coding conventions
    - [X] No breaking changes
    
    ---------
    
    Co-authored-by: Josh McKinney <joshka@openai.com>
  • Always fallback to real shell (#6953)
    Either cmd.exe or `/bin/sh`.
  • Added feature switch to disable animations in TUI (#6870)
    This PR adds support for a new feature flag `tui.animations`. By
    default, the TUI uses animations in its welcome screen, "working"
    spinners, and "shimmer" effects. This animations can interfere with
    screen readers, so it's good to provide a way to disable them.
    
    This change is inspired by [a
    PR](https://github.com/openai/codex/pull/4014) contributed by @Orinks.
    That PR has faltered a bit, but I think the core idea is sound. This
    version incorporates feedback from @aibrahim-oai. In particular:
    1. It uses a feature flag (`tui.animations`) rather than the unqualified
    CLI key `no-animations`. Feature flags are the preferred way to expose
    boolean switches. They are also exposed via CLI command switches.
    2. It includes more complete documentation.
    3. It disables a few animations that the other PR omitted.
  • Allow unified_exec to early exit (if the process terminates before yield_time_ms) (#6867)
    Thread through an `exit_notify` tokio `Notify` through to the
    `UnifiedExecSession` so that we can return early if the command
    terminates before `yield_time_ms`.
    
    As Codex review correctly pointed out below 🙌 we also need a
    `exit_signaled` flag so that commands which finish before we start
    waiting can also exit early.
    
    Since the default `yield_time_ms` is now 10s, this means that we don't
    have to wait 10s for trivial commands like ls, sed, etc (which are the
    majority of agent commands 😅)
    
    ---------
    
    Co-authored-by: jif-oai <jif@openai.com>
  • [app-server] feat: v2 apply_patch approval flow (#6760)
    This PR adds the API V2 version of the apply_patch approval flow, which
    centers around `ThreadItem::FileChange`.
    
    This PR wires the new RPC (`item/fileChange/requestApproval`, V2 only)
    and related events (`item/started`, `item/completed` for
    `ThreadItem::FileChange`, which are emitted in both V1 and V2) through
    the app-server
    protocol. The new approval RPC is only sent when the user initiates a
    turn with the new `turn/start` API so we don't break backwards
    compatibility with VSCE.
    
    Similar to https://github.com/openai/codex/pull/6758, the approach I
    took was to make as few changes to the Codex core as possible,
    leveraging existing `EventMsg` core events, and translating those in
    app-server. I did have to add a few additional fields to
    `EventMsg::PatchApplyBegin` and `EventMsg::PatchApplyEnd`, but those
    were fairly lightweight.
    
    However, the `EventMsg`s emitted by core are the following:
    ```
    1) Auto-approved (no request for approval)

    - EventMsg::PatchApplyBegin
    - EventMsg::PatchApplyEnd
    
    2) Approved by user
    - EventMsg::ApplyPatchApprovalRequest
    - EventMsg::PatchApplyBegin
    - EventMsg::PatchApplyEnd
    
    3) Declined by user
    - EventMsg::ApplyPatchApprovalRequest
    - EventMsg::PatchApplyBegin
    - EventMsg::PatchApplyEnd
    ```
    
    For a request triggering an approval, this would result in:
    ```
    item/fileChange/requestApproval
    item/started
    item/completed
    ```
    
    which is different from the `ThreadItem::CommandExecution` flow
    introduced in https://github.com/openai/codex/pull/6758, which does the
    below and is preferable:
    ```
    item/started
    item/commandExecution/requestApproval
    item/completed
    ```
    
    To fix this, we leverage `TurnSummaryStore` on codex_message_processor
    to store a little bit of state, allowing us to fire `item/started` and
    `item/fileChange/requestApproval` whenever we receive the underlying
    `EventMsg::ApplyPatchApprovalRequest`, and no-oping when we receive the
    `EventMsg::PatchApplyBegin` later.
    
    This is much less invasive than modifying the order of EventMsg within
    core (I tried).
    
    The resulting payloads:
    ```
    {
      "method": "item/started",
      "params": {
        "item": {
          "changes": [
            {
              "diff": "Hello from Codex!\n",
              "kind": "add",
              "path": "/Users/owen/repos/codex/codex-rs/APPROVAL_DEMO.txt"
            }
          ],
          "id": "call_Nxnwj7B3YXigfV6Mwh03d686",
          "status": "inProgress",
          "type": "fileChange"
        }
      }
    }
    ```
    
    ```
    {
      "id": 0,
      "method": "item/fileChange/requestApproval",
      "params": {
        "grantRoot": null,
        "itemId": "call_Nxnwj7B3YXigfV6Mwh03d686",
        "reason": null,
        "threadId": "019a9e11-8295-7883-a283-779e06502c6f",
        "turnId": "1"
      }
    }
    ```
    
    ```
    {
      "id": 0,
      "result": {
        "decision": "accept"
      }
    }
    ```
    
    ```
    {
      "method": "item/completed",
      "params": {
        "item": {
          "changes": [
            {
              "diff": "Hello from Codex!\n",
              "kind": "add",
              "path": "/Users/owen/repos/codex/codex-rs/APPROVAL_DEMO.txt"
            }
          ],
          "id": "call_Nxnwj7B3YXigfV6Mwh03d686",
          "status": "completed",
          "type": "fileChange"
        }
      }
    }
    ```
  • Revert "[core] add optional status_code to error events (#6865)" (#6955)
    This reverts commit c2ec477d93.
    
    # External (non-OpenAI) Pull Request Requirements
    
    Before opening this Pull Request, please read the dedicated
    "Contributing" markdown file or your PR may be closed:
    https://github.com/openai/codex/blob/main/docs/contributing.md
    
    If your PR conforms to our contribution guidelines, replace this text
    with a detailed and high quality description of your changes.
    
    Include a link to a bug report or enhancement request.
  • execpolicy2 core integration (#6641)
    This PR threads execpolicy2 into codex-core.
    
    activated via feature flag: exec_policy (on by default)
    
    reads and parses all .codexpolicy files in `codex_home/codex`
    
    refactored tool runtime API to integrate execpolicy logic
    
    ---------
    
    Co-authored-by: Michael Bolin <mbolin@openai.com>