Commit Graph

32 Commits

  • Reapply "Add app-server transport layer with websocket support" (#11370)
    Reapply "Add app-server transport layer with websocket support" with
    additional fixes from https://github.com/openai/codex/pull/11313/changes
    to avoid deadlocking.
    
    This reverts commit 47356ff83c.
    
    ## Summary
    
    To avoid deadlocking when queues are full, we maintain separate tokio
    tasks dedicated to incoming vs outgoing event handling
    - split the app-server main loop into two tasks in
    `run_main_with_transport`
       - inbound handling (`transport_event_rx`)
       - outbound handling (`outgoing_rx` + `thread_created_rx`)
    - separate incoming and outgoing websocket tasks
    
    ## Validation
    
    Integration tests, testing thoroughly e2e in codex app w/ >10 concurrent
    requests
    
    <img width="1365" height="979" alt="Screenshot 2026-02-10 at 2 54 22 PM"
    src="https://github.com/user-attachments/assets/47ca2c13-f322-4e5c-bedd-25859cbdc45f"
    />
    
    ---------
    
    Co-authored-by: jif-oai <jif@openai.com>
  • feat: split codex-common into smaller utils crates (#11422)
    We are removing feature-gated shared crates from the `codex-rs`
    workspace. `codex-common` grouped several unrelated utilities behind
    `[features]`, which made dependency boundaries harder to reason about
    and worked against the ongoing effort to eliminate feature flags from
    workspace crates.
    
    Splitting these utilities into dedicated crates under `utils/` aligns
    this area with existing workspace structure and keeps each dependency
    explicit at the crate boundary.
    
    ## What changed
    
    - Removed `codex-rs/common` (`codex-common`) from workspace members and
    workspace dependencies.
    - Added six new utility crates under `codex-rs/utils/`:
      - `codex-utils-cli`
      - `codex-utils-elapsed`
      - `codex-utils-sandbox-summary`
      - `codex-utils-approval-presets`
      - `codex-utils-oss`
      - `codex-utils-fuzzy-match`
    - Migrated the corresponding modules out of `codex-common` into these
    crates (with tests), and added matching `BUILD.bazel` targets.
    - Updated direct consumers to use the new crates instead of
    `codex-common`:
      - `codex-rs/cli`
      - `codex-rs/tui`
      - `codex-rs/exec`
      - `codex-rs/app-server`
      - `codex-rs/mcp-server`
      - `codex-rs/chatgpt`
      - `codex-rs/cloud-tasks`
    - Updated workspace lockfile entries to reflect the new dependency graph
    and removal of `codex-common`.
  • Add app-server transport layer with websocket support (#10693)
    - Adds --listen <URL> to codex app-server with two listen modes:
          - stdio:// (default, existing behavior)
          - ws://IP:PORT (new websocket transport)
      - Refactors message routing to be connection-aware:
    - Tracks per-connection session state (initialize/experimental
    capability)
          - Routes responses/errors to the originating connection
    - Broadcasts server notifications/requests to initialized connections
    - Updates initialization semantics to be per connection (not
    process-global), and updates app-server docs accordingly.
    - Adds websocket accept/read/write handling (JSON-RPC per text frame,
    ping/pong handling, connection lifecycle events).
    
    Testing
    
    - Unit tests for transport URL parsing and targeted response/error
    routing.
      - New websocket integration test validating:
          - per-connection initialization requirements
          - no cross-connection response leakage
          - same request IDs on different connections route independently.
  • feat: replace custom mcp-types crate with equivalents from rmcp (#10349)
    We started working with MCP in Codex before
    https://crates.io/crates/rmcp was mature, so we had our own crate for
    MCP types that was generated from the MCP schema:
    
    
    https://github.com/openai/codex/blob/8b95d3e082376f4cb23e92641705a22afb28a9da/codex-rs/mcp-types/README.md
    
    Now that `rmcp` is more mature, it makes more sense to use their MCP
    types in Rust, as they handle details (like the `_meta` field) that our
    custom version ignored. Though one advantage that our custom types had
    is that our generated types implemented `JsonSchema` and `ts_rs::TS`,
    whereas the types in `rmcp` do not. As such, part of the work of this PR
    is leveraging the adapters between `rmcp` types and the serializable
    types that are API for us (app server and MCP) introduced in #10356.
    
    Note this PR results in a number of changes to
    `codex-rs/app-server-protocol/schema`, which merit special attention
    during review. We must ensure that these changes are still
    backwards-compatible, which is possible because we have:
    
    ```diff
    - export type CallToolResult = { content: Array<ContentBlock>, isError?: boolean, structuredContent?: JsonValue, };
    + export type CallToolResult = { content: Array<JsonValue>, structuredContent?: JsonValue, isError?: boolean, _meta?: JsonValue, };
    ```
    
    so `ContentBlock` has been replaced with the more general `JsonValue`.
    Note that `ContentBlock` was defined as:
    
    ```typescript
    export type ContentBlock = TextContent | ImageContent | AudioContent | ResourceLink | EmbeddedResource;
    ```
    
    so the deletion of those individual variants should not be a cause of
    great concern.
    
    Similarly, we have the following change in
    `codex-rs/app-server-protocol/schema/typescript/Tool.ts`:
    
    ```
    - export type Tool = { annotations?: ToolAnnotations, description?: string, inputSchema: ToolInputSchema, name: string, outputSchema?: ToolOutputSchema, title?: string, };
    + export type Tool = { name: string, title?: string, description?: string, inputSchema: JsonValue, outputSchema?: JsonValue, annotations?: JsonValue, icons?: Array<JsonValue>, _meta?: JsonValue, };
    ```
    
    so:
    
    - `annotations?: ToolAnnotations` ➡️ `JsonValue`
    - `inputSchema: ToolInputSchema` ➡️ `JsonValue`
    - `outputSchema?: ToolOutputSchema` ➡️ `JsonValue`
    
    and two new fields: `icons?: Array<JsonValue>, _meta?: JsonValue`
    
    ---
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/10349).
    * #10357
    * __->__ #10349
    * #10356
  • Wire up cloud reqs in exec, app-server (#10241)
    We're fetching cloud requirements in TUI in
    https://github.com/openai/codex/pull/10167.
    
    This adds the same fetching in exec and app-server binaries also.
  • feat(app-server): support external auth mode (#10012)
    This enables a new use case where `codex app-server` is embedded into a
    parent application that will directly own the user's ChatGPT auth
    lifecycle, which means it owns the user’s auth tokens and refreshes it
    when necessary. The parent application would just want a way to pass in
    the auth tokens for codex to use directly.
    
    The idea is that we are introducing a new "auth mode" currently only
    exposed via app server: **`chatgptAuthTokens`** which consist of the
    `id_token` (stores account metadata) and `access_token` (the bearer
    token used directly for backend API calls). These auth tokens are only
    stored in-memory. This new mode is in addition to the existing `apiKey`
    and `chatgpt` auth modes.
    
    This PR reuses the shape of our existing app-server account APIs as much
    as possible:
    - Update `account/login/start` with a new `chatgptAuthTokens` variant,
    which will allow the client to pass in the tokens and have codex
    app-server use them directly. Upon success, the server emits
    `account/login/completed` and `account/updated` notifications.
    - A new server->client request called
    `account/chatgptAuthTokens/refresh` which the server can use whenever
    the access token previously passed in has expired and it needs a new one
    from the parent application.
    
    I leveraged the core 401 retry loop which typically triggers auth token
    refreshes automatically, but made it pluggable:
    - **chatgpt** mode refreshes internally, as usual.
    - **chatgptAuthTokens** mode calls the client via
    `account/chatgptAuthTokens/refresh`, the client responds with updated
    tokens, codex updates its in-memory auth, then retries. This RPC has a
    10s timeout and handles JSON-RPC errors from the client.
    
    Also some additional things:
    - chatgpt logins are blocked while external auth is active (have to log
    out first. typically clients will pick one OR the other, not support
    both)
    - `account/logout` clears external auth in memory
    - Ensures that if `forced_chatgpt_workspace_id` is set via the user's
    config, we respect it in both:
    - `account/login/start` with `chatgptAuthTokens` (returns a JSON-RPC
    error back to the client)
    - `account/chatgptAuthTokens/refresh` (fails the turn, and on next
    request app-server will send another `account/chatgptAuthTokens/refresh`
    request to the client).
  • feat(core) RequestRule (#9489)
    ## Summary
    Instead of trying to derive the prefix_rule for a command mechanically,
    let's let the model decide for us.
    
    ## Testing
    - [x] tested locally
  • feat: ephemeral threads (#9765)
    Add ephemeral threads capabilities. Only exposed through the
    `app-server` v2
    
    The idea is to disable the rollout recorder for those threads.
  • [connectors] Support connectors part 1 - App server & MCP (#9667)
    In order to make Codex work with connectors, we add a built-in gateway
    MCP that acts as a transparent proxy between the client and the
    connectors. The gateway MCP collects actions that are accessible to the
    user and sends them down to the user, when a connector action is chosen
    to be called, the client invokes the action through the gateway MCP as
    well.
    
     - [x] Add the system built-in gateway MCP to list and run connectors.
     - [x] Add the app server methods and protocol
  • feat: introduce codex-utils-cargo-bin as an alternative to assert_cmd::Command (#8496)
    This PR introduces a `codex-utils-cargo-bin` utility crate that
    wraps/replaces our use of `assert_cmd::Command` and
    `escargot::CargoBuild`.
    
    As you can infer from the introduction of `buck_project_root()` in this
    PR, I am attempting to make it possible to build Codex under
    [Buck2](https://buck2.build) as well as `cargo`. With Buck2, I hope to
    achieve faster incremental local builds (largely due to Buck2's
    [dice](https://buck2.build/docs/insights_and_knowledge/modern_dice/)
    build strategy, as well as benefits from its local build daemon) as well
    as faster CI builds if we invest in remote execution and caching.
    
    See
    https://buck2.build/docs/getting_started/what_is_buck2/#why-use-buck2-key-advantages
    for more details about the performance advantages of Buck2.
    
    Buck2 enforces stronger requirements in terms of build and test
    isolation. It discourages assumptions about absolute paths (which is key
    to enabling remote execution). Because the `CARGO_BIN_EXE_*` environment
    variables that Cargo provides are absolute paths (which
    `assert_cmd::Command` reads), this is a problem for Buck2, which is why
    we need this `codex-utils-cargo-bin` utility.
    
    My WIP-Buck2 setup sets the `CARGO_BIN_EXE_*` environment variables
    passed to a `rust_test()` build rule as relative paths.
    `codex-utils-cargo-bin` will resolve these values to absolute paths,
    when necessary.
    
    
    ---
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/8496).
    * #8498
    * __->__ #8496
  • feat: change ConfigLayerName into a disjoint union rather than a simple enum (#8095)
    This attempts to tighten up the types related to "config layers."
    Currently, `ConfigLayerEntry` is defined as follows:
    
    
    https://github.com/openai/codex/blob/bef36f4ae765f471d7cd69372fcf1b92c8f0367a/codex-rs/core/src/config_loader/state.rs#L19-L25
    
    but the `source` field is a bit of a lie, as:
    
    - for `ConfigLayerName::Mdm`, it is
    `"com.openai.codex/config_toml_base64"`
    - for `ConfigLayerName::SessionFlags`, it is `"--config"`
    - for `ConfigLayerName::User`, it is `"config.toml"` (just the file
    name, not the path to the `config.toml` on disk that was read)
    - for `ConfigLayerName::System`, it seems like it is usually
    `/etc/codex/managed_config.toml` in practice, though on Windows, it is
    `%CODEX_HOME%/managed_config.toml`:
    
    
    https://github.com/openai/codex/blob/bef36f4ae765f471d7cd69372fcf1b92c8f0367a/codex-rs/core/src/config_loader/layer_io.rs#L84-L101
    
    All that is to say, in three out of the four `ConfigLayerName`, `source`
    is a `PathBuf` that is not an absolute path (or even a true path).
    
    This PR tries to uplevel things by eliminating `source` from
    `ConfigLayerEntry` and turning `ConfigLayerName` into a disjoint union
    named `ConfigLayerSource` that has the appropriate metadata for each
    variant, favoring the use of `AbsolutePathBuf` where appropriate:
    
    ```rust
    pub enum ConfigLayerSource {
        /// Managed preferences layer delivered by MDM (macOS only).
        #[serde(rename_all = "camelCase")]
        #[ts(rename_all = "camelCase")]
        Mdm { domain: String, key: String },
        /// Managed config layer from a file (usually `managed_config.toml`).
        #[serde(rename_all = "camelCase")]
        #[ts(rename_all = "camelCase")]
        System { file: AbsolutePathBuf },
        /// Session-layer overrides supplied via `-c`/`--config`.
        SessionFlags,
        /// User config layer from a file (usually `config.toml`).
        #[serde(rename_all = "camelCase")]
        #[ts(rename_all = "camelCase")]
        User { file: AbsolutePathBuf },
    }
    ```
  • fix: introduce AbsolutePathBuf as part of sandbox config (#7856)
    Changes the `writable_roots` field of the `WorkspaceWrite` variant of
    the `SandboxPolicy` enum from `Vec<PathBuf>` to `Vec<AbsolutePathBuf>`.
    This is helpful because now callers can be sure the value is an absolute
    path rather than a relative one. (Though when using an absolute path in
    a Seatbelt config policy, we still have to _canonicalize_ it first.)
    
    Because `writable_roots` can be read from a config file, it is important
    that we are able to resolve relative paths properly using the parent
    folder of the config file as the base path.
  • feat: clean config loading and config api (#7924)
    Check the README of the `config_loader` for details
  • [app-server] Make sure that config writes preserve comments & order or configs (#7789)
    Make sure that config writes preserve comments and order of configs by
    utilizing the ConfigEditsBuilder in core.
    
    Tested by running a real example and made sure that nothing in the
    config file changes other than the configs to edit.
  • feat: support mcp in-session login (#7751)
    ### Summary
    * Added `mcpServer/oauthLogin` in app server for supporting in session
    MCP server login
    * Added `McpServerOauthLoginParams` and `McpServerOauthLoginResponse` to
    support above method with response returning the auth URL for consumer
    to open browser or display accordingly.
    * Added `McpServerOauthLoginCompletedNotification` which the app server
    would emit on MCP server login success or failure (i.e. timeout).
    * Refactored rmcp-client oath_login to have the ability on starting a
    auth server which the codex_message_processor uses for in-session auth.
  • feat: support list mcp servers in app server (#7505)
    ### Summary
    Added `mcp/servers/list` which is equivalent to `/mcp` slash command in
    CLI for response. This will be used in VSCE MCP settings to show log in
    status, available tools etc.
  • chore: add cargo-deny configuration (#7119)
    - add GitHub workflow running cargo-deny on push/PR
    - document cargo-deny allowlist with workspace-dep notes and advisory
    ignores
    - align workspace crates to inherit version/edition/license for
    consistent checks
  • fix(app-server) remove www warning (#7046)
    ### Summary
    After #7022, we no longer need this warning. We should also clean up the
    schema for the notification, but this is a quick fix to just stop the
    behavior in the VSCE
    
    ## Testing
    - [x] Ran locally
  • chore(app-server) world-writable windows notification (#6880)
    ## Summary
    On app-server startup, detect whether the experimental sandbox is
    enabled, and send a notification .
    
    **Note**
    New conversations will not respect the feature because we [ignore cli
    overrides in
    NewConversation](https://github.com/openai/codex/blob/a75321a64c990275ed4368bf26a5334c9ddfa0a7/codex-rs/app-server/src/codex_message_processor.rs#L1237-L1252).
    However, this should be okay, since we don't actually use config for
    this, we use a [global
    variable](https://github.com/openai/codex/blob/87cce88f4865685a863e143e0fad4cf5ea542e62/codex-rs/core/src/safety.rs#L105-L110).
    We should carefully unwind this setup at some point.
    
    
    ## Testing
    - [ ] In progress: testing locally
    
    ---------
    
    Co-authored-by: jif-oai <jif@openai.com>
  • [App server] add mcp tool call item started/completed events (#6642)
    this PR does two things:
    1. refactor `apply_bespoke_event_handling` into a separate file as it's
    getting kind of long;
    2. add mcp tool call `item/started` and `item/completed` events. To roll
    out app server events asap we didn't properly migrate mcp core events to
    use TurnItem for mcp tool calls - this will be a follow-up PR.
    
    real events generated in log:
    ```
    {
      "method": "codex/event/mcp_tool_call_end",
      "params": {
        "conversationId": "019a8021-26af-7c20-83db-21ca81e44d68",
        "id": "0",
        "msg": {
          "call_id": "call_7EjRQkD9HnfyMWf7tGrT9FKA",
          "duration": {
            "nanos": 92708,
            "secs": 0
          },
          "invocation": {
            "arguments": {
              "server": ""
            },
            "server": "codex",
            "tool": "list_mcp_resources"
          },
          "result": {
            "Ok": {
              "content": [
                {
                  "text": "{\"resources\":[]}",
                  "type": "text"
                }
              ],
              "isError": false
            }
          },
          "type": "mcp_tool_call_end"
        }
      }
    }
    
    {
      "method": "item/completed",
      "params": {
        "item": {
          "arguments": {
            "server": ""
          },
          "error": null,
          "id": "call_7EjRQkD9HnfyMWf7tGrT9FKA",
          "result": {
            "content": [
              {
                "text": "{\"resources\":[]}",
                "type": "text"
              }
            ],
            "structuredContent": null
          },
          "server": "codex",
          "status": "completed",
          "tool": "list_mcp_resources",
          "type": "mcpToolCall"
        }
      }
    }
    ```
  • Fixed flaky unit test (#5654)
    This PR fixes a test that is sporadically failing in CI.
    
    The problem is that two unit tests (the older `login_and_cancel_chatgpt`
    and a recently added
    `login_chatgpt_includes_forced_workspace_query_param`) exercise code
    paths that start the login server. The server binds to a hard-coded
    localhost port number, so attempts to start more than one server at the
    same time will fail. If these two tests happen to run concurrently, one
    of them will fail.
    
    To fix this, I've added a simple mutex. We can use this same mutex for
    future tests that use the same pattern.
  • [app-server] read rate limits API (#5302)
    Adds a `GET account/rateLimits/read` API to app-server. This calls the
    codex backend to fetch the user's current rate limits.
    
    This would be helpful in checking rate limits without having to send a
    message.
    
    For calling the codex backend usage API, I generated the types and
    manually copied the relevant ones into `codex-backend-openapi-types`.
    It'll be nice to extend our internal openapi generator to support Rust
    so we don't have to run these manual steps.
    
    # External (non-OpenAI) Pull Request Requirements
    
    Before opening this Pull Request, please read the dedicated
    "Contributing" markdown file or your PR may be closed:
    https://github.com/openai/codex/blob/main/docs/contributing.md
    
    If your PR conforms to our contribution guidelines, replace this text
    with a detailed and high quality description of your changes.
  • fix: remove mcp-types from app server protocol (#4537)
    We continue the separation between `codex app-server` and `codex
    mcp-server`.
    
    In particular, we introduce a new crate, `codex-app-server-protocol`,
    and migrate `codex-rs/protocol/src/mcp_protocol.rs` into it, renaming it
    `codex-rs/app-server-protocol/src/protocol.rs`.
    
    Because `ConversationId` was defined in `mcp_protocol.rs`, we move it
    into its own file, `codex-rs/protocol/src/conversation_id.rs`, and
    because it is referenced in a ton of places, we have to touch a lot of
    files as part of this PR.
    
    We also decide to get away from proper JSON-RPC 2.0 semantics, so we
    also introduce `codex-rs/app-server-protocol/src/jsonrpc_lite.rs`, which
    is basically the same `JSONRPCMessage` type defined in `mcp-types`
    except with all of the `"jsonrpc": "2.0"` removed.
    
    Getting rid of `"jsonrpc": "2.0"` makes our serialization logic
    considerably simpler, as we can lean heavier on serde to serialize
    directly into the wire format that we use now.
  • fix: separate codex mcp into codex mcp-server and codex app-server (#4471)
    This is a very large PR with some non-backwards-compatible changes.
    
    Historically, `codex mcp` (or `codex mcp serve`) started a JSON-RPC-ish
    server that had two overlapping responsibilities:
    
    - Running an MCP server, providing some basic tool calls.
    - Running the app server used to power experiences such as the VS Code
    extension.
    
    This PR aims to separate these into distinct concepts:
    
    - `codex mcp-server` for the MCP server
    - `codex app-server` for the "application server"
    
    Note `codex mcp` still exists because it already has its own subcommands
    for MCP management (`list`, `add`, etc.)
    
    The MCP logic continues to live in `codex-rs/mcp-server` whereas the
    refactored app server logic is in the new `codex-rs/app-server` folder.
    Note that most of the existing integration tests in
    `codex-rs/mcp-server/tests/suite` were actually for the app server, so
    all the tests have been moved with the exception of
    `codex-rs/mcp-server/tests/suite/mod.rs`.
    
    Because this is already a large diff, I tried not to change more than I
    had to, so `codex-rs/app-server/tests/common/mcp_process.rs` still uses
    the name `McpProcess` for now, but I will do some mechanical renamings
    to things like `AppServer` in subsequent PRs.
    
    While `mcp-server` and `app-server` share some overlapping functionality
    (like reading streams of JSONL and dispatching based on message types)
    and some differences (completely different message types), I ended up
    doing a bit of copypasta between the two crates, as both have somewhat
    similar `message_processor.rs` and `outgoing_message.rs` files for now,
    though I expect them to diverge more in the near future.
    
    One material change is that of the initialize handshake for `codex
    app-server`, as we no longer use the MCP types for that handshake.
    Instead, we update `codex-rs/protocol/src/mcp_protocol.rs` to add an
    `Initialize` variant to `ClientRequest`, which takes the `ClientInfo`
    object we need to update the `USER_AGENT_SUFFIX` in
    `codex-rs/app-server/src/message_processor.rs`.
    
    One other material change is in
    `codex-rs/app-server/src/codex_message_processor.rs` where I eliminated
    a use of the `send_event_as_notification()` method I am generally trying
    to deprecate (because it blindly maps an `EventMsg` into a
    `JSONNotification`) in favor of `send_server_notification()`, which
    takes a `ServerNotification`, as that is intended to be a custom enum of
    all notification types supported by the app server. So to make this
    update, I had to introduce a new variant of `ServerNotification`,
    `SessionConfigured`, which is a non-backwards compatible change with the
    old `codex mcp`, and clients will have to be updated after the next
    release that contains this PR. Note that
    `codex-rs/app-server/tests/suite/list_resume.rs` also had to be update
    to reflect this change.
    
    I introduced `codex-rs/utils/json-to-toml/src/lib.rs` as a small utility
    crate to avoid some of the copying between `mcp-server` and
    `app-server`.