Commit Graph

66 Commits

  • feat(core): zsh exec bridge (#12052)
    zsh fork PR stack:
    - https://github.com/openai/codex/pull/12051 
    - https://github.com/openai/codex/pull/12052 👈 
    
    ### Summary
    This PR introduces a feature-gated native shell runtime path that routes
    shell execution through a patched zsh exec bridge, removing MCP-specific
    behavior from the shell hot path while preserving existing
    CommandExecution lifecycle semantics.
    
    When shell_zsh_fork is enabled, shell commands run via patched zsh with
    per-`execve` interception through EXEC_WRAPPER. Core receives wrapper
    IPC requests over a Unix socket, applies existing approval policy, and
    returns allow/deny before the subcommand executes.
    
    ### What’s included
    **1) New zsh exec bridge runtime in core**
    - Wrapper-mode entrypoint (maybe_run_zsh_exec_wrapper_mode) for
    EXEC_WRAPPER invocations.
    - Per-execution Unix-socket IPC handling for wrapper requests/responses.
    - Approval callback integration using existing core approval
    orchestration.
    - Streaming stdout/stderr deltas to existing command output event
    pipeline.
    - Error handling for malformed IPC, denial/abort, and execution
    failures.
    
    **2) Session lifecycle integration**
    SessionServices now owns a `ZshExecBridge`.
    Session startup initializes bridge state; shutdown tears it down
    cleanly.
    
    **3) Shell runtime routing (feature-gated)**
    When `shell_zsh_fork` is enabled:
    - Build execution env/spec as usual.
    - Add wrapper socket env wiring.
    - Execute via `zsh_exec_bridge.execute_shell_request(...)` instead of
    the regular shell path.
    - Non-zsh-fork behavior remains unchanged.
    
    **4) Config + feature wiring**
    - Added `Feature::ShellZshFork` (under development).
    - Added config support for `zsh_path` (optional absolute path to patched
    zsh):
    - `Config`, `ConfigToml`, `ConfigProfile`, overrides, and schema.
    - Session startup validates that `zsh_path` exists/usable when zsh-fork
    is enabled.
    - Added startup test for missing `zsh_path` failure mode.
    
    **5) Seatbelt/sandbox updates for wrapper IPC**
    - Extended seatbelt policy generation to optionally allow outbound
    connection to explicitly permitted Unix sockets.
    - Wired sandboxing path to pass wrapper socket path through to seatbelt
    policy generation.
    - Added/updated seatbelt tests for explicit socket allow rule and
    argument emission.
    
    **6) Runtime entrypoint hooks**
    - This allows the same binary to act as the zsh wrapper subprocess when
    invoked via `EXEC_WRAPPER`.
    
    **7) Tool selection behavior**
    - ToolsConfig now prefers ShellCommand type when shell_zsh_fork is
    enabled.
    - Added test coverage for precedence with unified-exec enabled.
  • feat(core): add structured network approval plumbing and policy decision model (#11672)
    ### Description
    #### Summary
    Introduces the core plumbing required for structured network approvals
    
    #### What changed
    - Added structured network policy decision modeling in core.
    - Added approval payload/context types needed for network approval
    semantics.
    - Wired shell/unified-exec runtime plumbing to consume structured
    decisions.
    - Updated related core error/event surfaces for structured handling.
    - Updated protocol plumbing used by core approval flow.
    - Included small CLI debug sandbox compatibility updates needed by this
    layer.
    
    #### Why
    establishes the minimal backend foundation for network approvals without
    yet changing high-level orchestration or TUI behavior.
    
    #### Notes
    - Behavior remains constrained by existing requirements/config gating.
    - Follow-up PRs in the stack handle orchestration, UX, and app-server
    integration.
    
    ---------
    
    Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
  • feat: make sandbox read access configurable with ReadOnlyAccess (#11387)
    `SandboxPolicy::ReadOnly` previously implied broad read access and could
    not express a narrower read surface.
    This change introduces an explicit read-access model so we can support
    user-configurable read restrictions in follow-up work, while preserving
    current behavior today.
    
    It also ensures unsupported backends fail closed for restricted-read
    policies instead of silently granting broader access than intended.
    
    ## What
    
    - Added `ReadOnlyAccess` in protocol with:
      - `Restricted { include_platform_defaults, readable_roots }`
      - `FullAccess`
    - Updated `SandboxPolicy` to carry read-access configuration:
      - `ReadOnly { access: ReadOnlyAccess }`
      - `WorkspaceWrite { ..., read_only_access: ReadOnlyAccess }`
    - Preserved existing behavior by defaulting current construction paths
    to `ReadOnlyAccess::FullAccess`.
    - Threaded the new fields through sandbox policy consumers and call
    sites across `core`, `tui`, `linux-sandbox`, `windows-sandbox`, and
    related tests.
    - Updated Seatbelt policy generation to honor restricted read roots by
    emitting scoped read rules when full read access is not granted.
    - Added fail-closed behavior on Linux and Windows backends when
    restricted read access is requested but not yet implemented there
    (`UnsupportedOperation`).
    - Regenerated app-server protocol schema and TypeScript artifacts,
    including `ReadOnlyAccess`.
    
    ## Compatibility / rollout
    
    - Runtime behavior remains unchanged by default (`FullAccess`).
    - API/schema changes are in place so future config wiring can enable
    restricted read access without another policy-shape migration.
  • feat(sandbox): enforce proxy-aware network routing in sandbox (#11113)
    ## Summary
    - expand proxy env injection to cover common tool env vars
    (`HTTP_PROXY`/`HTTPS_PROXY`/`ALL_PROXY`/`NO_PROXY` families +
    tool-specific variants)
    - harden macOS Seatbelt network policy generation to route through
    inferred loopback proxy endpoints and fail closed when proxy env is
    malformed
    - thread proxy-aware Linux sandbox flags and add minimal bwrap netns
    isolation hook for restricted non-proxy runs
    - add/refresh tests for proxy env wiring, Seatbelt policy generation,
    and Linux sandbox argument wiring
  • feat: include NetworkConfig through ExecParams (#11105)
    This PR adds the following field to `Config`:
    
    ```rust
    pub network: Option<NetworkProxy>,
    ```
    
    Though for the moment, it will always be initialized as `None` (this
    will be addressed in a subsequent PR).
    
    This PR does the work to thread `network` through to `execute_exec_env()`, `process_exec_tool_call()`, and `UnifiedExecRuntime.run()` to ensure it is available whenever we span a process.
  • add sandbox policy and sandbox name to codex.tool.call metrics (#10711)
    This will give visibility into the comparative success rate of the
    Windows sandbox implementations compared to other platforms.
  • feat(linux-sandbox): add bwrap support (#9938)
    ## Summary
    This PR introduces a gated Bubblewrap (bwrap) Linux sandbox path. The
    curent Linux sandbox path relies on in-process restrictions (including
    Landlock). Bubblewrap gives us a more uniform filesystem isolation
    model, especially explicit writable roots with the option to make some
    directories read-only and granular network controls.
    
    This is behind a feature flag so we can validate behavior safely before
    making it the default.
    
    - Added temporary rollout flag:
      - `features.use_linux_sandbox_bwrap`
    - Preserved existing default path when the flag is off.
    - In Bubblewrap mode:
    - Added internal retry without /proc when /proc mount is not permitted
    by the host/container.
  • emit a metric when we can't spawn powershell (#10125)
    This will help diagnose and measure the impact of a user-reported bug
    with the elevated sandbox and powershell
  • remove sandbox globals. (#9797)
    Threads sandbox updates through OverrideTurnContext for active turn
    Passes computed sandbox type into safety/exec
  • Fix: cap aggregated exec output consistently (#9759)
    ## WHAT?
    - Bias aggregated output toward stderr under contention (2/3 stderr, 1/3
    stdout) while keeping the 1 MiB cap.
    - Rebalance unused stderr share back to stdout when stderr is tiny to
    avoid underfilling.
    - Add tests for contention, small-stderr rebalance, and under-cap
    ordering (stdout then stderr).
    
    ## WHY?
    - Review feedback requested stderr priority under contention.
    - Avoid underfilled aggregated output when stderr is small while
    preserving a consistent cap across exec paths.
    
    ## HOW?
    - Update `aggregate_output` to compute stdout/stderr shares, then
    reassign unused capacity to the other stream.
    - Use the helper in both Windows and async exec paths.
    - Add regression tests for contention/rebalance and under-cap ordering.
    
    ## BEFORE
    ```rust
    // Best-effort aggregate: stdout then stderr (capped).
    let mut aggregated = Vec::with_capacity(
        stdout
            .text
            .len()
            .saturating_add(stderr.text.len())
            .min(EXEC_OUTPUT_MAX_BYTES),
    );
    append_capped(&mut aggregated, &stdout.text, EXEC_OUTPUT_MAX_BYTES);
    append_capped(&mut aggregated, &stderr.text, EXEC_OUTPUT_MAX_BYTES);
    let aggregated_output = StreamOutput {
        text: aggregated,
        truncated_after_lines: None,
    };
    ```
    
    ## AFTER
    ```rust
    fn aggregate_output(
        stdout: &StreamOutput<Vec<u8>>,
        stderr: &StreamOutput<Vec<u8>>,
    ) -> StreamOutput<Vec<u8>> {
        let total_len = stdout.text.len().saturating_add(stderr.text.len());
        let max_bytes = EXEC_OUTPUT_MAX_BYTES;
        let mut aggregated = Vec::with_capacity(total_len.min(max_bytes));
    
        if total_len <= max_bytes {
            aggregated.extend_from_slice(&stdout.text);
            aggregated.extend_from_slice(&stderr.text);
            return StreamOutput {
                text: aggregated,
                truncated_after_lines: None,
            };
        }
    
        // Under contention, reserve 1/3 for stdout and 2/3 for stderr; rebalance unused stderr to stdout.
        let want_stdout = stdout.text.len().min(max_bytes / 3);
        let want_stderr = stderr.text.len();
        let stderr_take = want_stderr.min(max_bytes.saturating_sub(want_stdout));
        let remaining = max_bytes.saturating_sub(want_stdout + stderr_take);
        let stdout_take = want_stdout + remaining.min(stdout.text.len().saturating_sub(want_stdout));
    
        aggregated.extend_from_slice(&stdout.text[..stdout_take]);
        aggregated.extend_from_slice(&stderr.text[..stderr_take]);
    
        StreamOutput {
            text: aggregated,
            truncated_after_lines: None,
        }
    }
    ```
    
    ## TESTS
    - [x] `just fmt`
    - [x] `just fix -p codex-core`
    - [x] `cargo test -p codex-core aggregate_output_`
    - [x] `cargo test -p codex-core`
    - [x] `cargo test --all-features`
    
    ## FIXES
    Fixes #9758
  • fix: memory leak issue (#9543)
    Co-authored-by: Josh McKinney <joshka@openai.com>
  • feat: introduce ExternalSandbox policy (#8290)
    ## Description
    
    Introduced `ExternalSandbox` policy to cover use case when sandbox
    defined by outside environment, effectively it translates to
    `SandboxMode#DangerFullAccess` for file system (since sandbox configured
    on container level) and configurable `network_access` (either Restricted
    or Enabled by outside environment).
    
    as example you can configure `ExternalSandbox` policy as part of
    `sendUserTurn` v1 app_server API:
    
    ```
     {
                "conversationId": <id>,
                "cwd": <cwd>,
                "approvalPolicy": "never",
                "sandboxPolicy": {
                      "type": ""external-sandbox",
                      "network_access": "enabled"/"restricted"
                },
                "model": <model>,
                "effort": <effort>,
                ....
            }
    ```
  • refactoring with_escalated_permissions to use SandboxPermissions instead (#7750)
    helpful in the future if we want more granularity for requesting
    escalated permissions:
    e.g when running in readonly sandbox, model can request to escalate to a
    sandbox that allows writes
  • chore: update unified exec sandboxing detection (#7541)
    No integration test for now because it would make them flaky. Tracking
    it in my todos to add some once we have a clock based system for
    integration tests
  • refactor: inline sandbox type lookup in process_exec_tool_call (#7122)
    `process_exec_tool_call()` was taking `SandboxType` as a param, but in
    practice, the only place it was constructed was in
    `codex_message_processor.rs` where it was derived from the other
    `sandbox_policy` param, so this PR inlines the logic that decides the
    `SandboxType` into `process_exec_tool_call()`.
    
    
    
    ---
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/7122).
    * #7112
    * __->__ #7122
  • feat: waiting for an elicitation should not count against a shell tool timeout (#6973)
    Previously, we were running into an issue where we would run the `shell`
    tool call with a timeout of 10s, but it fired an elicitation asking for
    user approval, the time the user took to respond to the elicitation was
    counted agains the 10s timeout, so the `shell` tool call would fail with
    a timeout error unless the user is very fast!
    
    This PR addresses this issue by introducing a "stopwatch" abstraction
    that is used to manage the timeout. The idea is:
    
    - `Stopwatch::new()` is called with the _real_ timeout of the `shell`
    tool call.
    - `process_exec_tool_call()` is called with the `Cancellation` variant
    of `ExecExpiration` because it should not manage its own timeout in this
    case
    - the `Stopwatch` expiration is wired up to the `cancel_rx` passed to
    `process_exec_tool_call()`
    - when an elicitation for the `shell` tool call is received, the
    `Stopwatch` pauses
    - because it is possible for multiple elicitations to arrive
    concurrently, it keeps track of the number of "active pauses" and does
    not resume until that counter goes down to zero
    
    I verified that I can test the MCP server using
    `@modelcontextprotocol/inspector` and specify `git status` as the
    `command` with a timeout of 500ms and that the elicitation pops up and I
    have all the time in the world to respond whereas previous to this PR,
    that would not have been possible.
    
    ---
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/6973).
    * #7005
    * __->__ #6973
    * #6972
  • feat: update process_exec_tool_call() to take a cancellation token (#6972)
    This updates `ExecParams` so that instead of taking `timeout_ms:
    Option<u64>`, it now takes a more general cancellation mechanism,
    `ExecExpiration`, which is an enum that includes a
    `Cancellation(tokio_util::sync::CancellationToken)` variant.
    
    If the cancellation token is fired, then `process_exec_tool_call()`
    returns in the same way as if a timeout was exceeded.
    
    This is necessary so that in #6973, we can manage the timeout logic
    external to the `process_exec_tool_call()` because we want to "suspend"
    the timeout when an elicitation from a human user is pending.
    
    
    
    
    
    
    
    
    ---
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/6972).
    * #7005
    * #6973
    * __->__ #6972
  • core: make shell behavior portable on FreeBSD (#7039)
    - Use /bin/sh instead of /bin/bash on FreeBSD/OpenBSD in the process
    group timeout test to avoid command-not-found failures.
    
    - Accept /usr/local/bin/bash as a valid SHELL path to match common
    FreeBSD installations.
    
    - Switch the shell serialization duration test to /bin/sh for improved
    portability across Unix platforms.
    
    With this change, `cargo test -p codex-core --lib` runs and passes on
    FreeBSD.
  • Fix: Improve text encoding for shell output in VSCode preview (#6178) (#6182)
    ## 🐛 Problem
    
    Users running commands with non-ASCII characters (like Russian text
    "пример") in Windows/WSL environments experience garbled text in
    VSCode's shell preview window, with Unicode replacement characters (�)
    appearing instead of the actual text.
    
    **Issue**: https://github.com/openai/codex/issues/6178
    
    ## 🔧 Root Cause
    
    The issue was in `StreamOutput<Vec<u8>>::from_utf8_lossy()` method in
    `codex-rs/core/src/exec.rs`, which used `String::from_utf8_lossy()` to
    convert shell output bytes to strings. This function immediately
    replaces any invalid UTF-8 byte sequences with replacement characters,
    without attempting to decode using other common encodings.
    
    In Windows/WSL environments, shell output often uses encodings like:
    
    - Windows-1252 (common Windows encoding)
    - Latin-1/ISO-8859-1 (extended ASCII)
    
    ## 🛠️ Solution
    
    Replaced the simple `String::from_utf8_lossy()` call with intelligent
    encoding detection via a new `bytes_to_string_smart()` function that
    tries multiple encoding strategies:
    
    1. **UTF-8** (fast path for valid UTF-8)
    2. **Windows-1252** (handles Windows-specific characters in 0x80-0x9F
    range)
    3. **Latin-1** (fallback for extended ASCII)
    4. **Lossy UTF-8** (final fallback, same as before)
    
    ## 📁 Changes
    
    ### New Files
    
    - `codex-rs/core/src/text_encoding.rs` - Smart encoding detection module
    - `codex-rs/core/tests/suite/text_encoding_fix.rs` - Integration tests
    
    ### Modified Files
    
    - `codex-rs/core/src/lib.rs` - Added text_encoding module
    - `codex-rs/core/src/exec.rs` - Updated StreamOutput::from_utf8_lossy()
    - `codex-rs/core/tests/suite/mod.rs` - Registered new test module
    
    ##  Testing
    
    - **5 unit tests** covering UTF-8, Windows-1252, Latin-1, and fallback
    scenarios
    - **2 integration tests** simulating the exact Issue #6178 scenario
    - **Demonstrates improvement** over the previous
    `String::from_utf8_lossy()` approach
    
    All tests pass:
    
    ```bash
    cargo test -p codex-core text_encoding
    cargo test -p codex-core test_shell_output_encoding_issue_6178
    ```
    
    ## 🎯 Impact
    
    -  **Eliminates garbled text** in VSCode shell preview for non-ASCII
    content
    -  **Supports Windows/WSL environments** with proper encoding detection
    -  **Zero performance impact** for UTF-8 text (fast path)
    -  **Backward compatible** - UTF-8 content works exactly as before
    -  **Handles edge cases** with robust fallback mechanism
    
    ## 🧪 Test Scenarios
    
    The fix has been tested with:
    
    - Russian text ("пример")
    - Windows-1252 quotation marks (""test")
    - Latin-1 accented characters ("café")
    - Mixed encoding content
    - Invalid byte sequences (graceful fallback)
    
    ## 📋 Checklist
    
    - [X] Addresses the reported issue
    - [X] Includes comprehensive tests
    - [X] Maintains backward compatibility
    - [X] Follows project coding conventions
    - [X] No breaking changes
    
    ---------
    
    Co-authored-by: Josh McKinney <joshka@openai.com>
  • windows sandbox: support multiple workspace roots (#6854)
    The Windows sandbox did not previously support multiple workspace roots
    via config. Now it does
  • move cap_sid file into ~/.codex so the sandbox cannot overwrite it (#6798)
    The `cap_sid` file contains the IDs of the two custom SIDs that the
    Windows sandbox creates/manages to implement read-only and
    workspace-write sandbox policies.
    
    It previously lived in `<cwd>/.codex` which means that the sandbox could
    write to it, which could degrade the efficacy of the sandbox. This
    change moves it to `~/.codex/` (or wherever `CODEX_HOME` points to) so
    that it is outside the workspace.
  • Avoid hang when tool's process spawns grandchild that shares stderr/stdout (#6575)
    We've received many reports of codex hanging when calling certain tools.
    [Here](https://github.com/openai/codex/issues/3204) is one example. This
    is likely a major cause. The problem occurs when
    `consume_truncated_output` waits for `stdout` and `stderr` to be closed
    once the child process terminates. This normally works fine, but it
    doesn't handle the case where the child has spawned grandchild processes
    that inherits `stdout` and `stderr`.
    
    The fix was originally written by @md-oai in [this
    PR](https://github.com/openai/codex/pull/1852), which has gone stale.
    I've copied the original fix (which looks sound to me) and added an
    integration test to prevent future regressions.
  • Kill shell tool process groups on timeout (#5258)
    ## Summary
    - launch shell tool processes in their own process group so Codex owns
    the full tree
    - on timeout or ctrl-c, send SIGKILL to the process group before
    terminating the tracked child
    - document that the default shell/unified_exec timeout remains 1000 ms
    
    ## Original Bug
    Long-lived shell tool commands hang indefinitely because the timeout
    handler only terminated the direct child process; any grandchildren it
    spawned kept running and held the PTY open, preventing Codex from
    regaining control.
    
    ## Repro Original Bug
    Install next.js and run `next dev` (which is a long-running shell
    process with children). On openai:main, it will cause the agent to
    permanently get stuck here until human intervention. On this branch,
    this command will be terminated successfully after timeout_ms which will
    unblock the agent. This is a critical fix for unmonitored / lightly
    monitored agents that don't have immediate human observation to unblock
    them.
    
    ---------
    
    Co-authored-by: Michael Bolin <mbolin@openai.com>
    Co-authored-by: Michael Bolin <bolinfest@gmail.com>
  • core: widen sandbox to allow certificate ops when network is enabled (#5980)
    This allows `gh api` to work in the workspace-write sandbox w/ network
    enabled. Without this we see e.g.
    
    ```
    $ codex debug seatbelt --full-auto gh api repos/openai/codex/pulls --paginate -X GET -F state=all
    Get "https://api.github.com/repos/openai/codex/pulls?per_page=100&state=all": tls: failed to verify certificate: x509: OSStatus -26276
    ```
  • log sandbox commands to $CODEX_HOME instead of cwd (#6171)
    Logging commands in the Windows Sandbox is temporary, but while we are
    doing it, let's always write to CODEX_HOME instead of dirtying the cwd.
  • Windows Sandbox - Alpha version (#4905)
    - Added the new codex-windows-sandbox crate that builds both a library
    entry point (run_windows_sandbox_capture) and a CLI executable to launch
    commands inside a Windows restricted-token sandbox, including ACL
    management, capability SID provisioning, network lockdown, and output
    capture
    (windows-sandbox-rs/src/lib.rs:167, windows-sandbox-rs/src/main.rs:54).
    - Introduced the experimental WindowsSandbox feature flag and wiring so
    Windows builds can opt into the sandbox:
    SandboxType::WindowsRestrictedToken, the in-process execution path, and
    platform sandbox selection now honor the flag (core/src/features.rs:47,
    core/src/config.rs:1224, core/src/safety.rs:19,
    core/src/sandboxing/mod.rs:69, core/src/exec.rs:79,
    core/src/exec.rs:172).
    - Updated workspace metadata to include the new crate and its
    Windows-specific dependencies so the core crate can link against it
    (codex-rs/
        Cargo.toml:91, core/Cargo.toml:86).
    - Added a PowerShell bootstrap script that installs the Windows
    toolchain, required CLI utilities, and builds the workspace to ease
    development
        on the platform (scripts/setup-windows.ps1:1).
    - Landed a Python smoke-test suite that exercises
    read-only/workspace-write policies, ACL behavior, and network denial for
    the Windows sandbox
        binary (windows-sandbox-rs/sandbox_smoketests.py:1).
  • chore: rework tools execution workflow (#5278)
    Re-work the tool execution flow. Read `orchestrator.rs` to understand
    the structure
  • bug: sandbox denied error logs (#4874)
    Check on STDOUT / STDERR or aggregated output for some logs when sanbox
    is denied
  • Use a unified shell tell to not break cache (#3814)
    Currently, we change the tool description according to the sandbox
    policy and approval policy. This breaks the cache when the user hits
    `/approvals`. This PR does the following:
    - Always use the shell with escalation parameter:
    - removes `create_shell_tool_for_sandbox` and always uses unified tool
    via `create_shell_tool`
    - Reject the func call when the model uses escalation parameter when it
    cannot.
  • fix: ensure cwd for conversation and sandbox are separate concerns (#3874)
    Previous to this PR, both of these functions take a single `cwd`:
    
    
    https://github.com/openai/codex/blob/71038381aa0f51aa62e1a2bcc7cbf26a05b141f3/codex-rs/core/src/seatbelt.rs#L19-L25
    
    
    https://github.com/openai/codex/blob/71038381aa0f51aa62e1a2bcc7cbf26a05b141f3/codex-rs/core/src/landlock.rs#L16-L23
    
    whereas `cwd` and `sandbox_cwd` should be set independently (fixed in
    this PR).
    
    Added `sandbox_distinguishes_command_and_policy_cwds()` to
    `codex-rs/exec/tests/suite/sandbox.rs` to verify this.
  • Include command output when sending timeout to model (#3576)
    Being able to see the output helps the model decide how to handle the
    timeout.
  • fix: use a more efficient wire format for ExecCommandOutputDeltaEvent.chunk (#3163)
    When serializing to JSON, the existing solution created an enormous
    array of ints, which is far more bytes on the wire than a base64-encoded
    string would be.
  • fix: for now, limit the number of deltas sent back to the UI (#2776)
    This is a stopgap solution, but today, we are seeing the client get
    flooded with events. Since we already truncate the output we send to the
    model, it feels reasonable to limit how many deltas we send to the
    client.
  • send-aggregated output (#2364)
    We want to send an aggregated output of stderr and stdout so we don't
    have to aggregate it stderr+stdout as we lose order sometimes.
    
    ---------
    
    Co-authored-by: Gabriel Peal <gpeal@users.noreply.github.com>
  • fix: run python_multiprocessing_lock_works integration test on Mac and Linux (#2318)
    The high-order bit on this PR is that it makes it so `sandbox.rs` tests
    both Mac and Linux, as we introduce a general
    `spawn_command_under_sandbox()` function with platform-specific
    implementations for testing.
    
    An important, and interesting, discovery in porting the test to Linux is
    that (for reasons cited in the code comments), `/dev/shm` has to be
    added to `writable_roots` on Linux in order for `multiprocessing.Lock`
    to work there. Granting write access to `/dev/shm` comes with some
    degree of risk, so we do not make this the default for Codex CLI.
    
    Piggybacking on top of #2317, this moves the
    `python_multiprocessing_lock_works` test yet again, moving
    `codex-rs/core/tests/sandbox.rs` to `codex-rs/exec/tests/sandbox.rs`
    because in `codex-rs/exec/tests` we can use `cargo_bin()` like so:
    
    ```
    let codex_linux_sandbox_exe = assert_cmd::cargo::cargo_bin("codex-exec");
    ```
    
    which is necessary so we can use `codex_linux_sandbox_exe` and therefore
    `spawn_command_under_linux_sandbox` in an integration test.
    
    This also moves `spawn_command_under_linux_sandbox()` out of `exec.rs`
    and into `landlock.rs`, which makes things more consistent with
    `seatbelt.rs` in `codex-core`.
    
    For reference, https://github.com/openai/codex/pull/1808 is the PR that
    made the change to Seatbelt to get this test to pass on Mac.
  • chore: introduce ConversationManager as a clearinghouse for all conversations (#2240)
    This PR does two things because after I got deep into the first one I
    started pulling on the thread to the second:
    
    - Makes `ConversationManager` the place where all in-memory
    conversations are created and stored. Previously, `MessageProcessor` in
    the `codex-mcp-server` crate was doing this via its `session_map`, but
    this is something that should be done in `codex-core`.
    - It unwinds the `ctrl_c: tokio::sync::Notify` that was threaded
    throughout our code. I think this made sense at one time, but now that
    we handle Ctrl-C within the TUI and have a proper `Op::Interrupt` event,
    I don't think this was quite right, so I removed it. For `codex exec`
    and `codex proto`, we now use `tokio::signal::ctrl_c()` directly, but we
    no longer make `Notify` a field of `Codex` or `CodexConversation`.
    
    Changes of note:
    
    - Adds the files `conversation_manager.rs` and `codex_conversation.rs`
    to `codex-core`.
    - `Codex` and `CodexSpawnOk` are no longer exported from `codex-core`:
    other crates must use `CodexConversation` instead (which is created via
    `ConversationManager`).
    - `core/src/codex_wrapper.rs` has been deleted in favor of
    `ConversationManager`.
    - `ConversationManager::new_conversation()` returns `NewConversation`,
    which is in line with the `new_conversation` tool we want to add to the
    MCP server. Note `NewConversation` includes `SessionConfiguredEvent`, so
    we eliminate checks in cases like `codex-rs/core/tests/client.rs` to
    verify `SessionConfiguredEvent` is the first event because that is now
    internal to `ConversationManager`.
    - Quite a bit of code was deleted from
    `codex-rs/mcp-server/src/message_processor.rs` since it no longer has to
    manage multiple conversations itself: it goes through
    `ConversationManager` instead.
    - `core/tests/live_agent.rs` has been deleted because I had to update a
    bunch of tests and all the tests in here were ignored, and I don't think
    anyone ever ran them, so this was just technical debt, at this point.
    - Removed `notify_on_sigint()` from `util.rs` (and in a follow-up, I
    hope to refactor the blandly-named `util.rs` into more descriptive
    files).
    - In general, I started replacing local variables named `codex` as
    `conversation`, where appropriate, though admittedly I didn't do it
    through all the integration tests because that would have added a lot of
    noise to this PR.
    
    
    
    
    ---
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/2240).
    * #2264
    * #2263
    * __->__ #2240
  • Include output truncation message in tool call results (#2183)
    To avoid model being confused about incomplete output.
  • [approval_policy] Add OnRequest approval_policy (#1865)
    ## Summary
    A split-up PR of #1763 , stacked on top of a tools refactor #1858 to
    make the change clearer. From the previous summary:
    
    > Let's try something new: tell the model about the sandbox, and let it
    decide when it will need to break the sandbox. Some local testing
    suggests that it works pretty well with zero iteration on the prompt!
    
    ## Testing
    - [x] Added unit tests
    - [x] Tested locally and it appears to work smoothly!
  • [core] Stop escalating timeouts (#1853)
    ## Summary
    Escalating out of sandbox is (almost always) not going to fix
    long-running commands timing out - therefore we should just pass the
    failure back to the model instead of asking the user to re-run a command
    that took a long time anyway.
    
    ## Testing
    - [x] Ran locally with a timeout and confirmed this worked as expected
  • [sandbox] Filter out certain non-sandbox errors (#1804)
    ## Summary
    Users frequently complain about re-approving commands that have failed
    for non-sandbox reasons. We can't diagnose with complete accuracy which
    errors happened because of a sandbox failure, but we can start to
    eliminate some common simple cases.
    
    This PR captures the most common case I've seen, which is a `command not
    found` error.
    
    ## Testing
    - [x] Added unit tests
    - [x] Ran a few cases locally
  • feat: stream exec stdout events (#1786)
    ## Summary
    - stream command stdout as `ExecCommandStdout` events
    - forward streamed stdout to clients and ignore in human output
    processor
    - adjust call sites for new streaming API
  • chore: refactor exec.rs: create separate seatbelt.rs and spawn.rs files (#1762)
    At 550 lines, `exec.rs` was a bit large. In particular, I found it hard
    to locate the Seatbelt-related code quickly without a file with
    `seatbelt` in the name, so this refactors things so:
    
    - `spawn_command_under_seatbelt()` and dependent code moves to a new
    `seatbelt.rs` file
    - `spawn_child_async()` and dependent code moves to a new `spawn.rs`
    file
  • fix: use PR_SET_PDEATHSIG so to ensure child processes are killed in a timely manner (#1626)
    Some users have reported issues where child processes are not cleaned up
    after Codex exits (e.g., https://github.com/openai/codex/issues/1570).
    
    This is generally a tricky issue on operating systems: if a parent
    process receives `SIGKILL`, then it terminates immediately and cannot
    communicate with the child.
    
    **It only helps on Linux**, but this PR introduces the use of `prctl(2)`
    so that if the parent process dies, `SIGTERM` will be delivered to the
    child process. Whereas previously, I believe that if Codex spawned a
    long-running process (like `tsc --watch`) and the Codex process received
    `SIGKILL`, the `tsc --watch` process would be reparented to the init
    process and would never be killed. Now with the use of `prctl(2)`, the
    `tsc --watch` process should receive `SIGTERM` in that scenario.
    
    We still need to come up with a solution for macOS. I've started to look
    at `launchd`, but I'm researching a number of options.
  • feat: redesign sandbox config (#1373)
    This is a major redesign of how sandbox configuration works and aims to
    fix https://github.com/openai/codex/issues/1248. Specifically, it
    replaces `sandbox_permissions` in `config.toml` (and the
    `-s`/`--sandbox-permission` CLI flags) with a "table" with effectively
    three variants:
    
    ```toml
    # Safest option: full disk is read-only, but writes and network access are disallowed.
    [sandbox]
    mode = "read-only"
    
    # The cwd of the Codex task is writable, as well as $TMPDIR on macOS.
    # writable_roots can be used to specify additional writable folders.
    [sandbox]
    mode = "workspace-write"
    writable_roots = []  # Optional, defaults to the empty list.
    network_access = false  # Optional, defaults to false.
    
    # Disable sandboxing: use at your own risk!!!
    [sandbox]
    mode = "danger-full-access"
    ```
    
    This should make sandboxing easier to reason about. While we have
    dropped support for `-s`, the way it works now is:
    
    - no flags => `read-only`
    - `--full-auto` => `workspace-write`
    - currently, there is no way to specify `danger-full-access` via a CLI
    flag, but we will revisit that as part of
    https://github.com/openai/codex/issues/1254
    
    Outstanding issue:
    
    - As noted in the `TODO` on `SandboxPolicy::is_unrestricted()`, we are
    still conflating sandbox preferences with approval preferences in that
    case, which needs to be cleaned up.
  • fix: overhaul how we spawn commands under seccomp/landlock on Linux (#1086)
    Historically, we spawned the Seatbelt and Landlock sandboxes in
    substantially different ways:
    
    For **Seatbelt**, we would run `/usr/bin/sandbox-exec` with our policy
    specified as an arg followed by the original command:
    
    
    https://github.com/openai/codex/blob/d1de7bb383552e8fadd94be79d65d188e00fd562/codex-rs/core/src/exec.rs#L147-L219
    
    For **Landlock/Seccomp**, we would do
    `tokio::runtime::Builder::new_current_thread()`, _invoke
    Landlock/Seccomp APIs to modify the permissions of that new thread_, and
    then spawn the command:
    
    
    https://github.com/openai/codex/blob/d1de7bb383552e8fadd94be79d65d188e00fd562/codex-rs/core/src/exec_linux.rs#L28-L49
    
    While it is neat that Landlock/Seccomp supports applying a policy to
    only one thread without having to apply it to the entire process, it
    requires us to maintain two different codepaths and is a bit harder to
    reason about. The tipping point was
    https://github.com/openai/codex/pull/1061, in which we had to start
    building up the `env` in an unexpected way for the existing
    Landlock/Seccomp approach to continue to work.
    
    This PR overhauls things so that we do similar things for Mac and Linux.
    It turned out that we were already building our own "helper binary"
    comparable to Mac's `sandbox-exec` as part of the `cli` crate:
    
    
    https://github.com/openai/codex/blob/d1de7bb383552e8fadd94be79d65d188e00fd562/codex-rs/cli/Cargo.toml#L10-L12
    
    We originally created this to build a small binary to include with the
    Node.js version of the Codex CLI to provide support for Linux
    sandboxing.
    
    Though the sticky bit is that, at this point, we still want to deploy
    the Rust version of Codex as a single, standalone binary rather than a
    CLI and a supporting sandboxing binary. To satisfy this goal, we use
    "the arg0 trick," in which we:
    
    * use `std::env::current_exe()` to get the path to the CLI that is
    currently running
    * use the CLI as the `program` for the `Command`
    * set `"codex-linux-sandbox"` as arg0 for the `Command`
    
    A CLI that supports sandboxing should check arg0 at the start of the
    program. If it is `"codex-linux-sandbox"`, it must invoke
    `codex_linux_sandbox::run_main()`, which runs the CLI as if it were
    `codex-linux-sandbox`. When acting as `codex-linux-sandbox`, we make the
    appropriate Landlock/Seccomp API calls and then use `execvp(3)` to spawn
    the original command, so do _replace_ the process rather than spawn a
    subprocess. Incidentally, we do this before starting the Tokio runtime,
    so the process should only have one thread when `execvp(3)` is called.
    
    Because the `core` crate that needs to spawn the Linux sandboxing is not
    a CLI in its own right, this means that every CLI that includes `core`
    and relies on this behavior has to (1) implement it and (2) provide the
    path to the sandboxing executable. While the path is almost always
    `std::env::current_exe()`, we needed to make this configurable for
    integration tests, so `Config` now has a `codex_linux_sandbox_exe:
    Option<PathBuf>` property to facilitate threading this through,
    introduced in https://github.com/openai/codex/pull/1089.
    
    This common pattern is now captured in
    `codex_linux_sandbox::run_with_sandbox()` and all of the `main.rs`
    functions that should use it have been updated as part of this PR.
    
    The `codex-linux-sandbox` crate added to the Cargo workspace as part of
    this PR now has the bulk of the Landlock/Seccomp logic, which makes
    `core` a bit simpler. Indeed, `core/src/exec_linux.rs` and
    `core/src/landlock.rs` were removed/ported as part of this PR. I also
    moved the unit tests for this code into an integration test,
    `linux-sandbox/tests/landlock.rs`, in which I use
    `env!("CARGO_BIN_EXE_codex-linux-sandbox")` as the value for
    `codex_linux_sandbox_exe` since `std::env::current_exe()` is not
    appropriate in that case.
  • feat: introduce support for shell_environment_policy in config.toml (#1061)
    To date, when handling `shell` and `local_shell` tool calls, we were
    spawning new processes using the environment inherited from the Codex
    process itself. This means that the sensitive `OPENAI_API_KEY` that
    Codex needs to talk to OpenAI models was made available to everything
    run by `shell` and `local_shell`. While there are cases where that might
    be useful, it does not seem like a good default.
    
    This PR introduces a complex `shell_environment_policy` config option to
    control the `env` used with these tool calls. It is inevitably a bit
    complex so that it is possible to override individual components of the
    policy so without having to restate the entire thing.
    
    Details are in the updated `README.md` in this PR, but here is the
    relevant bit that explains the individual fields of
    `shell_environment_policy`:
    
    | Field | Type | Default | Description |
    | ------------------------- | -------------------------- | ------- |
    -----------------------------------------------------------------------------------------------------------------------------------------------
    |
    | `inherit` | string | `core` | Starting template for the
    environment:<br>`core` (`HOME`, `PATH`, `USER`, …), `all` (clone full
    parent env), or `none` (start empty). |
    | `ignore_default_excludes` | boolean | `false` | When `false`, Codex
    removes any var whose **name** contains `KEY`, `SECRET`, or `TOKEN`
    (case-insensitive) before other rules run. |
    | `exclude` | array&lt;string&gt; | `[]` | Case-insensitive glob
    patterns to drop after the default filter.<br>Examples: `"AWS_*"`,
    `"AZURE_*"`. |
    | `set` | table&lt;string,string&gt; | `{}` | Explicit key/value
    overrides or additions – always win over inherited values. |
    | `include_only` | array&lt;string&gt; | `[]` | If non-empty, a
    whitelist of patterns; only variables that match _one_ pattern survive
    the final step. (Generally used with `inherit = "all"`.) |
    
    
    In particular, note that the default is `inherit = "core"`, so:
    
    * if you have extra env variables that you want to inherit from the
    parent process, use `inherit = "all"` and then specify `include_only`
    * if you have extra env variables where you want to hardcode the values,
    the default `inherit = "core"` will work fine, but then you need to
    specify `set`
    
    This configuration is not battle-tested, so we will probably still have
    to play with it a bit. `core/src/exec_env.rs` has the critical business
    logic as well as unit tests.
    
    Though if nothing else, previous to this change:
    
    ```
    $ cargo run --bin codex -- debug seatbelt -- printenv OPENAI_API_KEY
    # ...prints OPENAI_API_KEY...
    ```
    
    But after this change it does not print anything (as desired).
    
    One final thing to call out about this PR is that the
    `configure_command!` macro we use in `core/src/exec.rs` has to do some
    complex logic with respect to how it builds up the `env` for the process
    being spawned under Landlock/seccomp. Specifically, doing
    `cmd.env_clear()` followed by `cmd.envs(&$env_map)` (which is arguably
    the most intuitive way to do it) caused the Landlock unit tests to fail
    because the processes spawned by the unit tests started failing in
    unexpected ways! If we forgo `env_clear()` in favor of updating env vars
    one at a time, the tests still pass. The comment in the code talks about
    this a bit, and while I would like to investigate this more, I need to
    move on for the moment, but I do plan to come back to it to fully
    understand what is going on. For example, this suggests that we might
    not be able to spawn a C program that calls `env_clear()`, which would
    be...weird. We may still have to fiddle with our Landlock config if that
    is the case.