Commit Graph

43 Commits

  • fix(app-server): set originator header from initialize JSON-RPC request (#8873)
    **Motivation**
    The `originator` header is important for codex-backend’s Responses API
    proxy because it identifies the real end client (codex cli, codex vscode
    extension, codex exec, future IDEs) and is used to categorize requests
    by client for our enterprise compliance API.
    
    Today the `originator` header is set by either:
    - the `CODEX_INTERNAL_ORIGINATOR_OVERRIDE` env var (our VSCode extension
    does this)
    - calling `set_default_originator()` which sets a global immutable
    singleton (`codex exec` does this)
    
    For `codex app-server`, we want the `initialize` JSON-RPC request to set
    that header because it is a natural place to do so. Example:
    ```json
    {
      "method": "initialize",
      "id": 0,
      "params": {
        "clientInfo": {
          "name": "codex_vscode",
          "title": "Codex VS Code Extension",
          "version": "0.1.0"
        }
      }
    }
    ```
    and when app-server receives that request, it can call
    `set_default_originator()`. This is a much more natural interface than
    asking third party developers to set an env var.
    
    One hiccup is that `originator()` reads the global singleton and locks
    in the value, preventing a later `set_default_originator()` call from
    setting it. This would be fine but is brittle, since any codepath that
    calls `originator()` before app-server can process an `initialize`
    JSON-RPC call would prevent app-server from setting it. This was
    actually the case with OTEL initialization which runs on boot, but I
    also saw this behavior in certain tests.
    
    Instead, what we now do is:
    - [unchanged] If `CODEX_INTERNAL_ORIGINATOR_OVERRIDE` env var is set,
    `originator()` would return that value and `set_default_originator()`
    with some other value does NOT override it.
    - [new] If no env var is set, `originator()` would return the default
    value which is `codex_cli_rs` UNTIL `set_default_originator()` is called
    once, in which case it is set to the new value and becomes immutable.
    Later calls to `set_default_originator()` returns
    `SetOriginatorError::AlreadyInitialized`.
    
    **Other notes**
    - I updated `codex_core::otel_init::build_provider` to accepts a service
    name override, and app-server sends a hardcoded `codex_app_server`
    service name to distinguish it from `codex_cli_rs` used by default (e.g.
    TUI).
    
    **Next steps**
    - Update VSCE to set the proper value for `clientInfo.name` on
    `initialize` and drop the `CODEX_INTERNAL_ORIGINATOR_OVERRIDE` env var.
    - Delete support for `CODEX_INTERNAL_ORIGINATOR_OVERRIDE` in codex-rs.
  • [chore] move app server tests from chat completion to responses (#8939)
    We are deprecating chat completions. Move all app server tests from chat
    completion to responses.
  • feat: fork conversation/thread (#8866)
    ## Summary
    - add thread/conversation fork endpoints to the protocol (v1 + v2)
    - implement fork handling in app-server using thread manager and config
    overrides
    - add fork coverage in app-server tests and document `thread/fork` usage
  • [fix] app server flaky send_messages test (#8874)
    Fix flakiness of CI test:
    https://github.com/openai/codex/actions/runs/20350530276/job/58473691434?pr=8282
    
    This PR does two things:
    1. move the flakiness test to use responses API instead of chat
    completion API
    2. make mcp_process agnostic to the order of
    responses/notifications/requests that come in, by buffering messages not
    read
  • [fix] app server flaky thread/resume tests (#8870)
    Fix flakiness of CI tests:
    https://github.com/openai/codex/actions/runs/20350530276/job/58473691443?pr=8282
    
    This PR does two things:
    1. test with responses API instead of chat completions API in
    thread_resume tests;
    2. have a new responses API fixture that mocks out arbitrary numbers of
    responses API calls (including no calls) and have the same repeated
    response.
    
    Tested by CI
  • Fix app-server write_models_cache to treat models with less priority number as higher priority. (#8844)
    Rank models with p0 higher than p1. This shouldn't result in any
    behavioral changes. Just reordering.
  • Merge Modelfamily into modelinfo (#8763)
    - Merge ModelFamily into ModelInfo
    - Remove logic for adding instructions to apply patch
    - Add compaction limit and visible context window to `ModelInfo`
  • chore: unify conversation with thread name (#8830)
    Done and verified by Codex + refactor feature of RustRover
  • feat(app-server): thread/rollback API (#8454)
    Add `thread/rollback` to app-server to support IDEs undo-ing the last N
    turns of a thread.
    
    For context, an IDE partner will be supporting an "undo" capability
    where the IDE (the app-server client) will be responsible for reverting
    the local changes made during the last turn. To support this well, we
    also need a way to drop the last turn (or more generally, the last N
    turns) from the agent's context. This is what `thread/rollback` does.
    
    **Core idea**: A Thread rollback is represented as a persisted event
    message (EventMsg::ThreadRollback) in the rollout JSONL file, not by
    rewriting history. On resume, both the model's context (core replay) and
    the UI turn list (app-server v2's thread history builder) apply these
    markers so the pruned history is consistent across live conversations
    and `thread/resume`.
    
    Implementation notes:
    - Rollback only affects agent context and appends to the rollout file;
    clients are responsible for reverting files on disk.
    - If a thread rollback is currently in progress, subsequent
    `thread/rollback` calls are rejected.
    - Because we use `CodexConversation::submit` and codex core tracks
    active turns, returning an error on concurrent rollbacks is communicated
    via an `EventMsg::Error` with a new variant
    `CodexErrorInfo::ThreadRollbackFailed`. app-server watches for that and
    sends the BAD_REQUEST RPC response.
    
    Tests cover thread rollbacks in both core and app-server, including when
    `num_turns` > existing turns (which clears all turns).
    
    **Note**: this explicitly does **not** behave like `/undo` which we just
    removed from the CLI, which does the opposite of what `thread/rollback`
    does. `/undo` reverts local changes via ghost commits/snapshots and does
    not modify the agent's context / conversation history.
  • feat: introduce codex-utils-cargo-bin as an alternative to assert_cmd::Command (#8496)
    This PR introduces a `codex-utils-cargo-bin` utility crate that
    wraps/replaces our use of `assert_cmd::Command` and
    `escargot::CargoBuild`.
    
    As you can infer from the introduction of `buck_project_root()` in this
    PR, I am attempting to make it possible to build Codex under
    [Buck2](https://buck2.build) as well as `cargo`. With Buck2, I hope to
    achieve faster incremental local builds (largely due to Buck2's
    [dice](https://buck2.build/docs/insights_and_knowledge/modern_dice/)
    build strategy, as well as benefits from its local build daemon) as well
    as faster CI builds if we invest in remote execution and caching.
    
    See
    https://buck2.build/docs/getting_started/what_is_buck2/#why-use-buck2-key-advantages
    for more details about the performance advantages of Buck2.
    
    Buck2 enforces stronger requirements in terms of build and test
    isolation. It discourages assumptions about absolute paths (which is key
    to enabling remote execution). Because the `CARGO_BIN_EXE_*` environment
    variables that Cargo provides are absolute paths (which
    `assert_cmd::Command` reads), this is a problem for Buck2, which is why
    we need this `codex-utils-cargo-bin` utility.
    
    My WIP-Buck2 setup sets the `CARGO_BIN_EXE_*` environment variables
    passed to a `rust_test()` build rule as relative paths.
    `codex-utils-cargo-bin` will resolve these values to absolute paths,
    when necessary.
    
    
    ---
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/8496).
    * #8498
    * __->__ #8496
  • Remove reasoning format (#8484)
    This isn't very useful parameter. 
    
    logic:
    ```
    if model puts `**` in their reasoning, trim it and visualize the header.
    if couldn't trim: don't render
    if model doesn't support: don't render
    ```
    
    We can simplify to:
    ```
    if could trim, visualize header.
    if not, don't render
    ```
  • remove minimal client version (#8447)
    This isn't needed value by client
  • Rename OpenAI models to models manager (#8346)
    # External (non-OpenAI) Pull Request Requirements
    
    Before opening this Pull Request, please read the dedicated
    "Contributing" markdown file or your PR may be closed:
    https://github.com/openai/codex/blob/main/docs/contributing.md
    
    If your PR conforms to our contribution guidelines, replace this text
    with a detailed and high quality description of your changes.
    
    Include a link to a bug report or enhancement request.
  • feat: model picker (#8209)
    # External (non-OpenAI) Pull Request Requirements
    
    Before opening this Pull Request, please read the dedicated
    "Contributing" markdown file or your PR may be closed:
    https://github.com/openai/codex/blob/main/docs/contributing.md
    
    If your PR conforms to our contribution guidelines, replace this text
    with a detailed and high quality description of your changes.
    
    Include a link to a bug report or enhancement request.
  • make model optional in config (#7769)
    - Make Config.model optional and centralize default-selection logic in
    ModelsManager, including a default_model helper (with
    codex-auto-balanced when available) so sessions now carry an explicit
    chosen model separate from the base config.
    - Resolve `model` once in `core` and `tui` from config. Then store the
    state of it on other structs.
    - Move refreshing models to be before resolving the default model
  • chore: add cargo-deny configuration (#7119)
    - add GitHub workflow running cargo-deny on push/PR
    - document cargo-deny allowlist with workspace-dep notes and advisory
    ignores
    - align workspace crates to inherit version/edition/license for
    consistent checks
  • [app-server] feat: expose gitInfo/cwd/etc. on Thread (#7060)
    Port the new additions from https://github.com/openai/codex/pull/6337 on
    the legacy API to v2. Mainly need `gitInfo` and `cwd` for VSCE.
  • [App-server] Implement account/read endpoint (#6336)
    This PR does two things:
    1. add a new function in core that maps the core-internal plan type to
    the external plan type;
    2. implement account/read that get account status (v2 of
    `getAuthStatus`).
  • [app-server] feat: v2 Turn APIs (#6216)
    Implements:
    ```
    turn/start
    turn/interrupt
    ```
    
    along with their integration tests. These are relatively light wrappers
    around the existing core logic, and changes to core logic are minimal.
    
    However, an improvement made for developer ergonomics:
    - `turn/start` replaces both `SendUserMessage` (no turn overrides) and
    `SendUserTurn` (can override model, approval policy, etc.)
  • [App-server] Add account/login/cancel v2 endpoint (#6288)
    Add `account/login/cancel` v2 endpoint for auth. this is similar
    implementation to `cancelLoginChatgpt` v1 endpoint.
  • [App-server] Implement v2 for account/login/start and account/login/completed (#6183)
    This PR implements `account/login/start` and `account/login/completed`.
    Instead of having separate endpoints for login with chatgpt and api, we
    have a single enum handling different login methods. For sync auth
    methods like sign in with api key, we still send a `completed`
    notification back to be compatible with the async login flow.
  • [app-server] feat: v2 Thread APIs (#6214)
    Implements:
    ```
    thread/list
    thread/start
    thread/resume
    thread/archive
    ```
    
    along with their integration tests. These are relatively light wrappers
    around the existing core logic, and changes to core logic are minimal.
    
    However, an improvement made for developer ergonomics:
    - `thread/start` and `thread/resume` automatically attaches a
    conversation listener internally, so clients don't have to make a
    separate `AddConversationListener` call like they do today.
    
    For consistency, also updated `model/list` and `feedback/upload` (naming
    conventions, list API params).
  • [App-server] v2 for account/updated and account/logout (#6175)
    V2 for `account/updated` and `account/logout` for app server. correspond
    to old `authStatusChange` and `LogoutChatGpt` respectively. Followup PRs
    will make other v2 endpoints call `account/updated` instead of
    `authStatusChange` too.
  • [Auth] Choose which auth storage to use based on config (#5792)
    This PR is a follow-up to #5591. It allows users to choose which auth
    storage mode they want by using the new
    `cli_auth_credentials_store_mode` config.
  • [Auth] Introduce New Auth Storage Abstraction for Codex CLI (#5569)
    This PR introduces a new `Auth Storage` abstraction layer that takes
    care of read, write, and load of auth tokens based on the
    AuthCredentialsStoreMode. It is similar to how we handle MCP client
    oauth
    [here](https://github.com/openai/codex/blob/main/codex-rs/rmcp-client/src/oauth.rs).
    Instead of reading and writing directly from disk for auth tokens, Codex
    CLI workflows now should instead use this auth storage using the public
    helper functions.
    
    This PR is just a refactor of the current code so the behavior stays the
    same. We will add support for keyring and hybrid mode in follow-up PRs.
    
    I have read the CLA Document and I hereby sign the CLA
  • [app-server] model/list API (#5382)
    Adds a `model/list` paginated API that returns the list of models
    supported by Codex.
  • [app-server] read rate limits API (#5302)
    Adds a `GET account/rateLimits/read` API to app-server. This calls the
    codex backend to fetch the user's current rate limits.
    
    This would be helpful in checking rate limits without having to send a
    message.
    
    For calling the codex backend usage API, I generated the types and
    manually copied the relevant ones into `codex-backend-openapi-types`.
    It'll be nice to extend our internal openapi generator to support Rust
    so we don't have to run these manual steps.
    
    # External (non-OpenAI) Pull Request Requirements
    
    Before opening this Pull Request, please read the dedicated
    "Contributing" markdown file or your PR may be closed:
    https://github.com/openai/codex/blob/main/docs/contributing.md
    
    If your PR conforms to our contribution guidelines, replace this text
    with a detailed and high quality description of your changes.
  • chore: sandbox refactor 2 (#4653)
    Revert the revert and fix the UI issue
  • chore: sanbox extraction (#4286)
    # Extract and Centralize Sandboxing
    - Goal: Improve safety and clarity by centralizing sandbox planning and
    execution.
      - Approach:
    - Add planner (ExecPlan) and backend registry (Direct/Seatbelt/Linux)
    with run_with_plan.
    - Refactor codex.rs to plan-then-execute; handle failures/escalation via
    the plan.
    - Delegate apply_patch to the codex binary and run it with an empty env
    for determinism.
  • fix: remove mcp-types from app server protocol (#4537)
    We continue the separation between `codex app-server` and `codex
    mcp-server`.
    
    In particular, we introduce a new crate, `codex-app-server-protocol`,
    and migrate `codex-rs/protocol/src/mcp_protocol.rs` into it, renaming it
    `codex-rs/app-server-protocol/src/protocol.rs`.
    
    Because `ConversationId` was defined in `mcp_protocol.rs`, we move it
    into its own file, `codex-rs/protocol/src/conversation_id.rs`, and
    because it is referenced in a ton of places, we have to touch a lot of
    files as part of this PR.
    
    We also decide to get away from proper JSON-RPC 2.0 semantics, so we
    also introduce `codex-rs/app-server-protocol/src/jsonrpc_lite.rs`, which
    is basically the same `JSONRPCMessage` type defined in `mcp-types`
    except with all of the `"jsonrpc": "2.0"` removed.
    
    Getting rid of `"jsonrpc": "2.0"` makes our serialization logic
    considerably simpler, as we can lean heavier on serde to serialize
    directly into the wire format that we use now.
  • fix: use macros to ensure request/response symmetry (#4529)
    Manually curating `protocol-ts/src/lib.rs` was error-prone, as expected.
    I finally asked Codex to write some Rust macros so we can ensure that:
    
    - For every variant of `ClientRequest` and `ServerRequest`, there is an
    associated `params` and `response` type.
    - All response types are included automatically in the output of `codex
    generate-ts`.
  • fix: separate codex mcp into codex mcp-server and codex app-server (#4471)
    This is a very large PR with some non-backwards-compatible changes.
    
    Historically, `codex mcp` (or `codex mcp serve`) started a JSON-RPC-ish
    server that had two overlapping responsibilities:
    
    - Running an MCP server, providing some basic tool calls.
    - Running the app server used to power experiences such as the VS Code
    extension.
    
    This PR aims to separate these into distinct concepts:
    
    - `codex mcp-server` for the MCP server
    - `codex app-server` for the "application server"
    
    Note `codex mcp` still exists because it already has its own subcommands
    for MCP management (`list`, `add`, etc.)
    
    The MCP logic continues to live in `codex-rs/mcp-server` whereas the
    refactored app server logic is in the new `codex-rs/app-server` folder.
    Note that most of the existing integration tests in
    `codex-rs/mcp-server/tests/suite` were actually for the app server, so
    all the tests have been moved with the exception of
    `codex-rs/mcp-server/tests/suite/mod.rs`.
    
    Because this is already a large diff, I tried not to change more than I
    had to, so `codex-rs/app-server/tests/common/mcp_process.rs` still uses
    the name `McpProcess` for now, but I will do some mechanical renamings
    to things like `AppServer` in subsequent PRs.
    
    While `mcp-server` and `app-server` share some overlapping functionality
    (like reading streams of JSONL and dispatching based on message types)
    and some differences (completely different message types), I ended up
    doing a bit of copypasta between the two crates, as both have somewhat
    similar `message_processor.rs` and `outgoing_message.rs` files for now,
    though I expect them to diverge more in the near future.
    
    One material change is that of the initialize handshake for `codex
    app-server`, as we no longer use the MCP types for that handshake.
    Instead, we update `codex-rs/protocol/src/mcp_protocol.rs` to add an
    `Initialize` variant to `ClientRequest`, which takes the `ClientInfo`
    object we need to update the `USER_AGENT_SUFFIX` in
    `codex-rs/app-server/src/message_processor.rs`.
    
    One other material change is in
    `codex-rs/app-server/src/codex_message_processor.rs` where I eliminated
    a use of the `send_event_as_notification()` method I am generally trying
    to deprecate (because it blindly maps an `EventMsg` into a
    `JSONNotification`) in favor of `send_server_notification()`, which
    takes a `ServerNotification`, as that is intended to be a custom enum of
    all notification types supported by the app server. So to make this
    update, I had to introduce a new variant of `ServerNotification`,
    `SessionConfigured`, which is a non-backwards compatible change with the
    old `codex mcp`, and clients will have to be updated after the next
    release that contains this PR. Note that
    `codex-rs/app-server/tests/suite/list_resume.rs` also had to be update
    to reflect this change.
    
    I introduced `codex-rs/utils/json-to-toml/src/lib.rs` as a small utility
    crate to avoid some of the copying between `mcp-server` and
    `app-server`.