Commit Graph

373 Commits

  • Restrict MCP servers from requirements.toml (#9101)
    Enterprises want to restrict the MCP servers their users can use.
    
    Admins can now specify an allowlist of MCPs in `requirements.toml`. The
    MCP servers are matched on both Name and Transport (local path or HTTP
    URL) -- both must match to allow the MCP server. This prevents
    circumventing the allowlist by renaming MCP servers in user config. (It
    is still possible to replace the local path e.g. rewrite say
    `/usr/local/github-mcp` with a nefarious MCP. We could allow hash
    pinning in the future, but that would break updates. I also think this
    represents a broader, out-of-scope problem.)
    
    We introduce a new field to Constrained: "normalizer". In general, it is
    a fn(T) -> T and applies when `Constrained<T>.set()` is called. In this
    particular case, it disables MCP servers which do not match the
    allowlist. An alternative solution would remove this and instead throw a
    ConstraintError. That would stop Codex launching if any MCP server was
    configured which didn't match. I think this is bad.
    
    We currently reuse the enabled flag on MCP servers to disable them, but
    don't propagate any information about why they are disabled. I'd like to
    add that in a follow up PR, possibly by switching out enabled with an
    enum.
    
    In action:
    
    ```
    # MCP server config has two MCPs. We are going to allowlist one of them.
    ➜  codex git:(gt/restrict-mcps) ✗ cat ~/.codex/config.toml | grep mcp_servers -A1
    [mcp_servers.hello_world]
    command = "hello-world-mcp"
    --
    [mcp_servers.docs]
    command = "docs-mcp"
    
    # Restrict the MCPs to the hello_world MCP.
    ➜  codex git:(gt/restrict-mcps) ✗ defaults read com.openai.codex requirements_toml_base64 | base64 -d
    [mcp_server_allowlist.hello_world]
    command = "hello-world-mcp"
    
    # List the MCPs, observe hello_world is enabled and docs is disabled.
    ➜  codex git:(gt/restrict-mcps) ✗ just codex mcp list
    cargo run --bin codex -- "$@"
        Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.25s
         Running `target/debug/codex mcp list`
    Name         Command          Args  Env  Cwd  Status    Auth
    docs         docs-mcp         -     -    -    disabled  Unsupported
    hello_world  hello-world-mcp  -     -    -    enabled   Unsupported
    
    # Remove the restrictions.
    ➜  codex git:(gt/restrict-mcps) ✗ defaults delete com.openai.codex requirements_toml_base64
    
    # Observe both MCPs are enabled.
    ➜  codex git:(gt/restrict-mcps) ✗ just codex mcp list
    cargo run --bin codex -- "$@"
        Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.25s
         Running `target/debug/codex mcp list`
    Name         Command          Args  Env  Cwd  Status   Auth
    docs         docs-mcp         -     -    -    enabled  Unsupported
    hello_world  hello-world-mcp  -     -    -    enabled  Unsupported
    
    # A new requirements that updates the command to one that does not match.
    ➜  codex git:(gt/restrict-mcps) ✗ cat ~/requirements.toml
    [mcp_server_allowlist.hello_world]
    command = "hello-world-mcp-v2"
    
    # Use those requirements.
    ➜  codex git:(gt/restrict-mcps) ✗ defaults write com.openai.codex requirements_toml_base64 "$(base64 -i /Users/gt/requirements.toml)"
    
    # Observe both MCPs are disabled.
    ➜  codex git:(gt/restrict-mcps) ✗ just codex mcp list
    cargo run --bin codex -- "$@"
        Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.75s
         Running `target/debug/codex mcp list`
    Name         Command          Args  Env  Cwd  Status    Auth
    docs         docs-mcp         -     -    -    disabled  Unsupported
    hello_world  hello-world-mcp  -     -    -    disabled  Unsupported
    ```
  • Support response.done and add integration tests (#9129)
    The agent loop using a persistent incremental web socket connection.
  • Reuse websocket connection (#9127)
    Reuses the connection but still sends full requests.
  • Add model client sessions (#9102)
    Maintain a long-running session.
  • Assemble sandbox/approval/network prompts dynamically (#8961)
    - Add a single builder for developer permissions messaging that accepts
    SandboxPolicy and approval policy. This builder now drives the developer
    “permissions” message that’s injected at session start and any time
    sandbox/approval settings change.
    - Trim EnvironmentContext to only include cwd, writable roots, and
    shell; removed sandbox/approval/network duplication and adjusted XML
    serialization and tests accordingly.
    
    Follow-up: adding a config value to replace the developer permissions
    message for custom sandboxes.
  • feat: hot reload mcp servers (#8957)
    ### Summary
    * Added `mcpServer/refresh` command to inform app servers and active
    threads to refresh mcpServer on next turn event.
    * Added `pending_mcp_server_refresh_config` to codex core so that if the
    value is populated, we reinitialize the mcp server manager on the thread
    level.
    * The config is updated on `mcpServer/refresh` command which we iterate
    through threads and provide with the latest config value after last
    write.
  • feat: add wait tool implementation for collab (#9088)
    Add implementation for the `wait` tool.
    
    For this we consider all status different from `PendingInit` and
    `Running` as terminal. The `wait` tool call will return either after a
    given timeout or when the tool reaches a non-terminal status.
    
    A few points to note:
    * The usage of a channel is preferred to prevent some races (just
    looping on `get_status()` could "miss" a terminal status)
    * The order of operations is very important, we need to first subscribe
    and then check the last known status to prevent race conditions
    * If the channel gets dropped, we return an error on purpose
  • Label attached images so agent can understand in-message labels (#8950)
    Agent wouldn't "see" attached images and would instead try to use the
    view_file tool:
    <img width="1516" height="504" alt="image"
    src="https://github.com/user-attachments/assets/68a705bb-f962-4fc1-9087-e932a6859b12"
    />
    
    In this PR, we wrap image content items in XML tags with the name of
    each image (now just a numbered name like `[Image #1]`), so that the
    model can understand inline image references (based on name). We also
    put the image content items above the user message which the model seems
    to prefer (maybe it's more used to definitions being before references).
    
    We also tweak the view_file tool description which seemed to help a bit
    
    Results on a simple eval set of images:
    
    Before
    <img width="980" height="310" alt="image"
    src="https://github.com/user-attachments/assets/ba838651-2565-4684-a12e-81a36641bf86"
    />
    
    After
    <img width="918" height="322" alt="image"
    src="https://github.com/user-attachments/assets/10a81951-7ee6-415e-a27e-e7a3fd0aee6f"
    />
    
    ```json
    [
      {
        "id": "single_describe",
        "prompt": "Describe the attached image in one sentence.",
        "images": ["image_a.png"]
      },
      {
        "id": "single_color",
        "prompt": "What is the dominant color in the image? Answer with a single color word.",
        "images": ["image_b.png"]
      },
      {
        "id": "orientation_check",
        "prompt": "Is the image portrait or landscape? Answer in one sentence.",
        "images": ["image_c.png"]
      },
      {
        "id": "detail_request",
        "prompt": "Look closely at the image and call out any small details you notice.",
        "images": ["image_d.png"]
      },
      {
        "id": "two_images_compare",
        "prompt": "I attached two images. Are they the same or different? Briefly explain.",
        "images": ["image_a.png", "image_b.png"]
      },
      {
        "id": "two_images_captions",
        "prompt": "Provide a short caption for each image (Image 1, Image 2).",
        "images": ["image_c.png", "image_d.png"]
      },
      {
        "id": "multi_image_rank",
        "prompt": "Rank the attached images from most colorful to least colorful.",
        "images": ["image_a.png", "image_b.png", "image_c.png"]
      },
      {
        "id": "multi_image_choice",
        "prompt": "Which image looks more vibrant? Answer with 'Image 1' or 'Image 2'.",
        "images": ["image_b.png", "image_d.png"]
      }
    ]
    ```
  • Immutable CodexAuth (#8857)
    Historically we started with a CodexAuth that knew how to refresh it's
    own tokens and then added AuthManager that did a different kind of
    refresh (re-reading from disk).
    
    I don't think it makes sense for both `CodexAuth` and `AuthManager` to
    be mutable and contain behaviors.
    
    Move all refresh logic into `AuthManager` and keep `CodexAuth` as a data
    object.
  • feat: metrics capabilities (#8318)
    Add metrics capabilities to Codex. The `README.md` is up to date.
    
    This will not be merged with the metrics before this PR of course:
    https://github.com/openai/codex/pull/8350
  • Override truncation policy at model info level (#8856)
    We used to override truncation policy by comparing model info vs config
    value in context manager. A better way to do it is to construct model
    info using the config value
  • remove unnecessary todos (#8842)
    > // todo(aibrahim): why are we passing model here while it can change?
    
    we update it on each turn with `.with_model`
    
    > //TODO(aibrahim): run CI in release mode.
    
    although it's good to have, release builds take double the time tests
    take.
    
    > // todo(aibrahim): make this async function
    
    we figured out another way of doing this sync
  • Merge Modelfamily into modelinfo (#8763)
    - Merge ModelFamily into ModelInfo
    - Remove logic for adding instructions to apply patch
    - Add compaction limit and visible context window to `ModelInfo`
  • chore: unify conversation with thread name (#8830)
    Done and verified by Codex + refactor feature of RustRover
  • add web_search_cached flag (#8795)
    Add `web_search_cached` feature to config. Enables `web_search` tool
    with access only to cached/indexed results (see
    [docs](https://platform.openai.com/docs/guides/tools-web-search#live-internet-access)).
    
    This takes precedence over the existing `web_search_request`, which
    continues to enable `web_search` over live results as it did before.
    
    `web_search_cached` is disabled for review mode, as `web_search_request`
    is.
  • feat(app-server): thread/rollback API (#8454)
    Add `thread/rollback` to app-server to support IDEs undo-ing the last N
    turns of a thread.
    
    For context, an IDE partner will be supporting an "undo" capability
    where the IDE (the app-server client) will be responsible for reverting
    the local changes made during the last turn. To support this well, we
    also need a way to drop the last turn (or more generally, the last N
    turns) from the agent's context. This is what `thread/rollback` does.
    
    **Core idea**: A Thread rollback is represented as a persisted event
    message (EventMsg::ThreadRollback) in the rollout JSONL file, not by
    rewriting history. On resume, both the model's context (core replay) and
    the UI turn list (app-server v2's thread history builder) apply these
    markers so the pruned history is consistent across live conversations
    and `thread/resume`.
    
    Implementation notes:
    - Rollback only affects agent context and appends to the rollout file;
    clients are responsible for reverting files on disk.
    - If a thread rollback is currently in progress, subsequent
    `thread/rollback` calls are rejected.
    - Because we use `CodexConversation::submit` and codex core tracks
    active turns, returning an error on concurrent rollbacks is communicated
    via an `EventMsg::Error` with a new variant
    `CodexErrorInfo::ThreadRollbackFailed`. app-server watches for that and
    sends the BAD_REQUEST RPC response.
    
    Tests cover thread rollbacks in both core and app-server, including when
    `num_turns` > existing turns (which clears all turns).
    
    **Note**: this explicitly does **not** behave like `/undo` which we just
    removed from the CLI, which does the opposite of what `thread/rollback`
    does. `/undo` reverts local changes via ghost commits/snapshots and does
    not modify the agent's context / conversation history.
  • feat: agent controller (#8783)
    Added an agent control plane that lets sessions spawn or message other
    conversations via `AgentControl`.
    
    `AgentBus` (core/src/agent/bus.rs) keeps track of the last known status
    of a conversation.
    
    ConversationManager now holds shared state behind an Arc so AgentControl
    keeps only a weak back-reference, the goal is just to avoid explicit
    cycle reference.
    
    Follow-ups:
    * Build a small tool in the TUI to be able to see every agent and send
    manual message to each of them
    * Handle approval requests in this TUI
    * Add tools to spawn/communicate between agents (see related design)
    * Define agent types
  • Use ConfigLayerStack for skills discovery. (#8497)
    Use ConfigLayerStack to get all folders while loading skills.
  • feat: expose outputSchema to user_turn/turn_start app_server API (#8377)
    What changed
    - Added `outputSchema` support to the app-server APIs, mirroring `codex
    exec --output-schema` behavior.
    - V1 `sendUserTurn` now accepts `outputSchema` and constrains the final
    assistant message for that turn.
    - V2 `turn/start` now accepts `outputSchema` and constrains the final
    assistant message for that turn (explicitly per-turn only).
    
    Core behavior
    - `Op::UserTurn` already supported `final_output_json_schema`; now V1
    `sendUserTurn` forwards `outputSchema` into that field.
    - `Op::UserInput` now carries `final_output_json_schema` for per-turn
    settings updates; core maps it into
    `SessionSettingsUpdate.final_output_json_schema` so it applies to the
    created turn context.
    - V2 `turn/start` does NOT persist the schema via `OverrideTurnContext`
    (it’s applied only for the current turn). Other overrides
    (cwd/model/etc) keep their existing persistent behavior.
    
    API / docs
    - `codex-rs/app-server-protocol/src/protocol/v1.rs`: add `output_schema:
    Option<serde_json::Value>` to `SendUserTurnParams` (serialized as
    `outputSchema`).
    - `codex-rs/app-server-protocol/src/protocol/v2.rs`: add `output_schema:
    Option<JsonValue>` to `TurnStartParams` (serialized as `outputSchema`).
    - `codex-rs/app-server/README.md`: document `outputSchema` for
    `turn/start` and clarify it applies only to the current turn.
    - `codex-rs/docs/codex_mcp_interface.md`: document `outputSchema` for v1
    `sendUserTurn` and v2 `turn/start`.
    
    Tests added/updated
    - New app-server integration tests asserting `outputSchema` is forwarded
    into outbound `/responses` requests as `text.format`:
      - `codex-rs/app-server/tests/suite/output_schema.rs`
      - `codex-rs/app-server/tests/suite/v2/output_schema.rs`
    - Added per-turn semantics tests (schema does not leak to the next
    turn):
      - `send_user_turn_output_schema_is_per_turn_v1`
      - `turn_start_output_schema_is_per_turn_v2`
    - Added protocol wire-compat tests for the merged op:
      - serialize omits `final_output_json_schema` when `None`
      - deserialize works when field is missing
      - serialize includes `final_output_json_schema` when `Some(schema)`
    
    Call site updates (high level)
    - Updated all `Op::UserInput { .. }` constructions to include
    `final_output_json_schema`:
      - `codex-rs/app-server/src/codex_message_processor.rs`
      - `codex-rs/core/src/codex_delegate.rs`
      - `codex-rs/mcp-server/src/codex_tool_runner.rs`
      - `codex-rs/tui/src/chatwidget.rs`
      - `codex-rs/tui2/src/chatwidget.rs`
      - plus impacted core tests.
    
    Validation
    - `just fmt`
    - `cargo test -p codex-core`
    - `cargo test -p codex-app-server`
    - `cargo test -p codex-mcp-server`
    - `cargo test -p codex-tui`
    - `cargo test -p codex-tui2`
    - `cargo test -p codex-protocol`
    - `cargo clippy --all-features --tests --profile dev --fix -- -D
    warnings`
  • fix: /review to respect session cwd (#8738)
    Fixes /review base-branch prompt resolution to use the session/turn cwd
    (respecting runtime cwd overrides) so merge-base/diff guidance is
    computed from the intended repo; adds a regression test for cwd
    overrides; tested with cargo test -p codex-core --test all
    review_uses_overridden_cwd_for_base_branch_merge_base.
  • Attach more tags to feedback submissions (#8688)
    Attach more tags to sentry feedback so it's easier to classify and debug
    without having to scan through logs.
    
    Formatting isn't amazing but it's a start.
    <img width="1234" height="276" alt="image"
    src="https://github.com/user-attachments/assets/521a349d-f627-4051-b511-9811cd5cd933"
    />
  • hard disable remote models refresh when feature is disabled (#8687)
    We were refreshing on tag mismatch even with feature is disabled
  • Account for last token count on resume (#8677)
    last token count in context manager is initialized to 0. Gets populated
    only on events from server.
    
    This PR populates it on resume so we can decide if we need to compact or
    not.
  • Refresh on models etag mismatch (#8491)
    - Send models etag
    - Refresh models on 412
    - This wires `ModelsManager` to `ModelFamily` so we don't mutate it
    mid-turn
  • [chore] add additional_details to StreamErrorEvent + wire through (#8307)
    ### What
    
    Builds on #8293.
    
    Add `additional_details`, which contains the upstream error message, to
    relevant structures used to pass along retryable `StreamError`s.
    
    Uses the new TUI status indicator's `details` field (shows under the
    status header) to display the `additional_details` error to the user on
    retryable `Reconnecting...` errors. This adds clarity for users for
    retryable errors.
    
    Will make corresponding change to VSCode extension to show
    `additional_details` as expandable from the `Reconnecting...` cell.
    
    Examples:
    <img width="1012" height="326" alt="image"
    src="https://github.com/user-attachments/assets/f35e7e6a-8f5e-4a2f-a764-358101776996"
    />
    
    <img width="1526" height="358" alt="image"
    src="https://github.com/user-attachments/assets/0029cbc0-f062-4233-8650-cc216c7808f0"
    />
  • chore: save more about turn context in rollout log file (#8458)
    ### Motivation
    - Persist richer per-turn configuration in rollouts so resumed/forked
    sessions and tooling can reason about the exact instruction inputs and
    output constraints used for a turn.
    
    ### Description
    - Extend `TurnContextItem` to include optional `base_instructions`,
    `user_instructions`, and `developer_instructions`.
    - Record the optional `final_output_json_schema` associated with a turn.
    - Add an optional `truncation_policy` to `TurnContextItem` and populate
    it when writing turn-context rollout items.
    - Introduce a protocol-level `TruncationPolicy` representation and
    convert from core truncation policy when recording.
    
    ### Testing
    - `cargo test -p codex-protocol` (pass)
  • feat: load ExecPolicyManager from ConfigLayerStack (#8453)
    https://github.com/openai/codex/pull/8354 added support for in-repo
    `.config/` files, so this PR updates the logic for loading `*.rules`
    files to load `*.rules` files from all relevant layers. The main change
    to the business logic is `load_exec_policy()` in
    `codex-rs/core/src/exec_policy.rs`.
    
    Note this adds a `config_folder()` method to `ConfigLayerSource` that
    returns `Option<AbsolutePathBuf>` so that it is straightforward to
    iterate over the sources and get the associated config folder, if any.
  • Add ExecPolicyManager (#8349)
    Move exec policy management into services to keep turn context
    immutable.
  • Rename OpenAI models to models manager (#8346)
    # External (non-OpenAI) Pull Request Requirements
    
    Before opening this Pull Request, please read the dedicated
    "Contributing" markdown file or your PR may be closed:
    https://github.com/openai/codex/blob/main/docs/contributing.md
    
    If your PR conforms to our contribution guidelines, replace this text
    with a detailed and high quality description of your changes.
    
    Include a link to a bug report or enhancement request.
  • feat: support allowed_sandbox_modes in requirements.toml (#8298)
    This adds support for `allowed_sandbox_modes` in `requirements.toml` and
    provides legacy support for constraining sandbox modes in
    `managed_config.toml`. This is converted to `Constrained<SandboxPolicy>`
    in `ConfigRequirements` and applied to `Config` such that constraints
    are enforced throughout the harness.
    
    Note that, because `managed_config.toml` is deprecated, we do not add
    support for the new `external-sandbox` variant recently introduced in
    https://github.com/openai/codex/pull/8290. As noted, that variant is not
    supported in `config.toml` today, but can be configured programmatically
    via app server.
  • feat: make ConstraintError an enum (#8330)
    This will make it easier to test for expected errors in unit tests since
    we can compare based on the field values rather than the message (which
    might change over time). See https://github.com/openai/codex/pull/8298
    for an example.
    
    It also ensures more consistency in the way a `ConstraintError` is
    constructed.
  • chore: migrate from Config::load_from_base_config_with_overrides to ConfigBuilder (#8276)
    https://github.com/openai/codex/pull/8235 introduced `ConfigBuilder` and
    this PR updates all call non-test call sites to use it instead of
    `Config::load_from_base_config_with_overrides()`.
    
    This is important because `load_from_base_config_with_overrides()` uses
    an empty `ConfigRequirements`, which is a reasonable default for testing
    so the tests are not influenced by the settings on the host. This method
    is now guarded by `#[cfg(test)]` so it cannot be used by business logic.
    
    Because `ConfigBuilder::build()` is `async`, many of the test methods
    had to be migrated to be `async`, as well. On the bright side, this made
    it possible to eliminate a bunch of `block_on_future()` stuff.
  • Support skills shortDescription. (#8278)
    Allow SKILL.md to specify a more human-readable short description as
    skill metadata.
  • Reintroduce feature flags for skills. (#8244)
    1. Reintroduce feature flags for skills;
    2. UI tweaks (truncate descriptions, better validation error display).
  • Support SYSTEM skills. (#8220)
    1. Remove PUBLIC skills and introduce SYSTEM skills embedded in the
    binary and installed into $CODEX_HOME/skills/.system at startup.
    2. Skills are now always enabled (feature flag removed).
    3. Update skills/list to accept forceReload and plumb it through (not
    used by clients yet).
  • fix: session downgrade (#8196)
    The problem is that the `tokio` task own an `Arc` reference of the
    session and that this task only exit with the broadcast channel get
    closed. But this never get closed if the session is not dropped. So it's
    a snake biting his tail basically
    
    The most notable result was that non of the `Drop` implementation were
    triggered (temporary files, shell snapshots, session cleaning etc etc)
    when closing the session (through a `/new` for example)
    
    The fix is just to weaken the `Arc` and upgrade it on the fly
  • feat: Constrain values for approval_policy (#7778)
    Constrain `approval_policy` through new `admin_policy` config.
    
    This PR will:
    1. Add a `admin_policy` section to config, with a single field (for now)
    `allowed_approval_policies`. This list constrains the set of
    user-settable `approval_policy`s.
    2. Introduce a new `Constrained<T>` type, which combines a current value
    and a validator function. The validator function ensures disallowed
    values are not set.
    3. Change the type of `approval_policy` on `Config` and
    `SessionConfiguration` from `AskForApproval` to
    `Constrained<AskForApproval>`. The validator function is set by the
    values passed into `allowed_approval_policies`.
    4. `GenericDisplayRow`: add a `disabled_reason: Option<String>`. When
    set, it disables selection of the value and indicates as such in the
    menu. This also makes it unselectable with arrow keys or numbers. This
    is used in the `/approvals` menu.
    
    Follow ups are:
    1. Do the same thing to `sandbox_policy`.
    2. Propagate the allowed set of values through app-server for the
    extension (though already this should prevent app-server from setting
    this values, it's just that we want to disable UI elements that are
    unsettable).
    
    Happy to split this PR up if you prefer, into the logical numbered areas
    above. Especially if there are parts we want to gavel on separately
    (e.g. admin_policy).
    
    Disabled full access:
    <img width="1680" height="380" alt="image"
    src="https://github.com/user-attachments/assets/1fb61c8c-1fcb-4dc4-8355-2293edb52ba0"
    />
    
    Disabled `--yolo` on startup:
    <img width="749" height="76" alt="image"
    src="https://github.com/user-attachments/assets/0a1211a0-6eb1-40d6-a1d7-439c41e94ddb"
    />
    
    CODEX-4087
  • Add public skills + improve repo skill discovery and error UX (#8098)
    1. Adds SkillScope::Public end-to-end (core + protocol) and loads skills
    from the public cache directory
    2. Improves repo skill discovery by searching upward for the nearest
    .codex/skills within a git repo
    3. Deduplicates skills by name with deterministic ordering to avoid
    duplicates across sources
    4. Fixes garbled “Skill errors” overlay rendering by preventing pending
    history lines from being injected during the modal
    5. Updates the project docs “Skills” intro wording to avoid hardcoded
    paths
  • nit: trace span for regular task (#8053)
    Logs are too spammy
    
    ---------
    
    Co-authored-by: Anton Panasenko <apanasenko@openai.com>