Commit Graph

102 Commits

  • test: faster test execution in codex-core (#2633)
    this dramatically improves time to run `cargo test -p codex-core` (~25x
    speedup).
    
    before:
    ```
    cargo test -p codex-core  35.96s user 68.63s system 19% cpu 8:49.80 total
    ```
    
    after:
    ```
    cargo test -p codex-core  5.51s user 8.16s system 63% cpu 21.407 total
    ```
    
    both tests measured "hot", i.e. on a 2nd run with no filesystem changes,
    to exclude compile times.
    
    approach inspired by [Delete Cargo Integration
    Tests](https://matklad.github.io/2021/02/27/delete-cargo-integration-tests.html),
    we move all test cases in tests/ into a single suite in order to have a
    single binary, as there is significant overhead for each test binary
    executed, and because test execution is only parallelized with a single
    binary.
  • Add web search tool (#2371)
    Adds web_search tool, enabling the model to use Responses API web_search
    tool.
    - Disabled by default, enabled by --search flag
    - When --search is passed, exposes web_search_request function tool to
    the model, which triggers user approval. When approved, the model can
    use the web_search tool for the remainder of the turn
    <img width="1033" height="294" alt="image"
    src="https://github.com/user-attachments/assets/62ac6563-b946-465c-ba5d-9325af28b28f"
    />
    
    ---------
    
    Co-authored-by: easong-openai <easong@openai.com>
  • fork conversation from a previous message (#2575)
    This can be the underlying logic in order to start a conversation from a
    previous message. will need some love in the UI.
    
    Base for building this: #2588
  • Fix flakiness in shell command approval test (#2547)
    ## Summary
    - read the shell exec approval request's actual id instead of assuming
    it is always 0
    - use that id when validating and responding in the test
    
    ## Testing
    - `cargo test -p codex-mcp-server
    test_shell_command_approval_triggers_elicitation`
    
    ------
    https://chatgpt.com/codex/tasks/task_i_68a6ab9c732c832c81522cbf11812be0
  • Add AuthManager and enhance GetAuthStatus command (#2577)
    This PR adds a central `AuthManager` struct that manages the auth
    information used across conversations and the MCP server. Prior to this,
    each conversation and the MCP server got their own private snapshots of
    the auth information, and changes to one (such as a logout or token
    refresh) were not seen by others.
    
    This is especially problematic when multiple instances of the CLI are
    run. For example, consider the case where you start CLI 1 and log in to
    ChatGPT account X and then start CLI 2 and log out and then log in to
    ChatGPT account Y. The conversation in CLI 1 is still using account X,
    but if you create a new conversation, it will suddenly (and
    unexpectedly) switch to account Y.
    
    With the `AuthManager`, auth information is read from disk at the time
    the `ConversationManager` is constructed, and it is cached in memory.
    All new conversations use this same auth information, as do any token
    refreshes.
    
    The `AuthManager` is also used by the MCP server's GetAuthStatus
    command, which now returns the auth method currently used by the MCP
    server.
    
    This PR also includes an enhancement to the GetAuthStatus command. It
    now accepts two new (optional) input parameters: `include_token` and
    `refresh_token`. Callers can use this to request the in-use auth token
    and can optionally request to refresh the token.
    
    The PR also adds tests for the login and auth APIs that I recently added
    to the MCP server.
  • Added new auth-related methods and events to mcp server (#2496)
    This PR adds the following:
    * A getAuthStatus method on the mcp server. This returns the auth method
    currently in use (chatgpt or apikey) or none if the user is not
    authenticated. It also returns the "preferred auth method" which
    reflects the `preferred_auth_method` value in the config.
    * A logout method on the mcp server. If called, it logs out the user and
    deletes the `auth.json` file — the same behavior in the cli's `/logout`
    command.
    * An `authStatusChange` event notification that is sent when the auth
    status changes due to successful login or logout operations.
    * Logic to pass command-line config overrides to the mcp server at
    startup time. This allows use cases like `codex mcp -c
    preferred_auth_method=apikey`.
  • [tui] Support /mcp command (#2430)
    ## Summary
    Adds a `/mcp` command to list active tools. We can extend this command
    to allow configuration of MCP tools, but for now a simple list command
    will help debug if your config.toml and your tools are working as
    expected.
  • fix: remove shutdown_flag param to run_login_server() (#2399)
    In practice, this was always passed in as `None`, so eliminated the
    param and updated all the call sites.
    
    ---
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/2399).
    * __->__ #2399
    * #2398
    * #2396
    * #2395
    * #2394
    * #2393
    * #2389
  • fix: make ShutdownHandle a private field of LoginServer (#2396)
    Folds the top-level `shutdown()` function into a method of
    `ShutdownHandle` and then simply stores `ShutdownHandle` on
    `LoginServer` since the two fields it contains were always being used
    together, anyway.
    
    ---
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/2396).
    * #2399
    * #2398
    * __->__ #2396
    * #2395
    * #2394
    * #2393
    * #2389
  • fix: eliminate ServerOptions.login_timeout and have caller use tokio::time::timeout() instead (#2395)
    https://github.com/openai/codex/pull/2373 introduced
    `ServerOptions.login_timeout` and `spawn_timeout_watcher()` to use an
    extra thread to manage the timeout for the login server. Now that we
    have asyncified the login stack, we can use `tokio::time::timeout()`
    from "outside" the login library to manage the timeout rather than
    having to a commit to a specific "timeout" concept from within.
    
    ---
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/2395).
    * #2399
    * #2398
    * #2396
    * __->__ #2395
    * #2394
    * #2393
    * #2389
  • fix: async-ify login flow (#2393)
    This replaces blocking I/O with async/non-blocking I/O in a number of
    cases. This facilitates the use of `tokio::sync::Notify` and
    `tokio::select!` in #2394.
    
    
    
    
    
    
    
    
    
    ---
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/2393).
    * #2399
    * #2398
    * #2396
    * #2395
    * #2394
    * __->__ #2393
    * #2389
  • chore: move mcp-server/src/wire_format.rs to protocol/src/mcp_protocol.rs (#2423)
    The existing `wire_format.rs` should share more types with the
    `codex-protocol` crate (like `AskForApproval` instead of maintaining a
    parallel `CodexToolCallApprovalPolicy` enum), so this PR moves
    `wire_format.rs` into `codex-protocol`, renaming it as
    `mcp-protocol.rs`. We also de-dupe types, where appropriate.
    
    ---
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/2423).
    * #2424
    * __->__ #2423
  • fix: introduce EventMsg::TurnAborted (#2365)
    Introduces `EventMsg::TurnAborted` that should be sent in response to
    `Op::Interrupt`.
    
    In the MCP server, updates the handling of a
    `ClientRequest::InterruptConversation` request such that it sends the
    `Op::Interrupt` but does not respond to the request until it sees an
    `EventMsg::TurnAborted`.
  • Added MCP server command to enable authentication using ChatGPT (#2373)
    This PR adds two new APIs for the MCP server: 1) loginChatGpt, and 2)
    cancelLoginChatGpt. The first starts a login server and returns a local
    URL that allows for browser-based authentication, and the second
    provides a way to cancel the login attempt. If the login attempt
    succeeds, a notification (in the form of an event) is sent to a
    subscriber.
    
    I also added a timeout mechanism for the existing login server. The
    loginChatGpt code path uses a 10-minute timeout by default, so if the
    user fails to complete the login flow in that timeframe, the login
    server automatically shuts down. I tested the timeout code by manually
    setting the timeout to a much lower number and confirming that it works
    as expected when used e2e.
  • feat: introduce ClientRequest::SendUserTurn (#2345)
    This adds a new request type, `SendUserTurn`, that makes it possible to
    submit a `Op::UserTurn` operation (introduced in #2329) to a
    conversation. This PR also adds a new integration test that verifies
    that changing from `AskForApproval::UnlessTrusted` to
    `AskForApproval::Never` mid-conversation ensures that an elicitation is
    no longer sent for running `python3 -c print(42)`.
    
    ---
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/2345).
    * __->__ #2345
    * #2329
    * #2343
    * #2340
    * #2338
  • fix: try to fix flakiness in test_shell_command_approval_triggers_elicitation (#2344)
    I still see flakiness in
    `test_shell_command_approval_triggers_elicitation()` on occasion where
    `MockServer` claims it has not received all of its expected requests.
    
    I recently introduced a similar type of test in #2264,
    `test_codex_jsonrpc_conversation_flow()`, which I have not seen flake
    (yet!), so this PR pulls over two things I did in that test:
    
    - increased `worker_threads` from `2` to `4`
    - added an assertion to make sure the `task_complete` notification is
    received
    
    Honestly, I'm still not sure why `MockServer` claims it sometimes does
    not receive all its expected requests given that we assert that the
    final `JSONRPCResponse` is read on the stream, but let's give this a
    shot.
    
    Assuming this fixes things, my hypothesis is that the increase in
    `worker_threads` helps because perhaps there are async tasks in
    `MockServer` that do not reliably complete fully when there are not
    enough threads available? If that is correct, it seems like the test
    would still be flaky, though perhaps with lower frequency?
  • [tools] Add apply_patch tool (#2303)
    ## Summary
    We've been seeing a number of issues and reports with our synthetic
    `apply_patch` tool, e.g. #802. Let's make this a real tool - in my
    anecdotal testing, it's critical for GPT-OSS models, but I'd like to
    make it the standard across GPT-5 and codex models as well.
    
    ## Testing
    - [x] Tested locally
    - [x] Integration test
  • Added allow-expect-in-tests / allow-unwrap-in-tests (#2328)
    This PR:
    * Added the clippy.toml to configure allowable expect / unwrap usage in
    tests
    * Removed as many expect/allow lines as possible from tests
    * moved a bunch of allows to expects where possible
    
    Note: in integration tests, non `#[test]` helper functions are not
    covered by this so we had to leave a few lingering `expect(expect_used`
    checks around
  • Tag InputItem (#2304)
    Instead of:
    ```
    { Text: { text: string } }
    ```
    
    It is now:
    ```
    { type: "text", data: { text: string } }
    ```
    which makes for cleaner discriminated unions
  • feat: add support for an InterruptConversation request (#2287)
    This adds `ClientRequest::InterruptConversation`, which effectively maps
    directly to `Op::Interrupt`.
    
    ---
    
    * __->__  #2287
    * #2286
    * #2285
  • fix: add support for exec and apply_patch approvals in the new wire format (#2286)
    Now when `CodexMessageProcessor` receives either a
    `EventMsg::ApplyPatchApprovalRequest` or a
    `EventMsg::ExecApprovalRequest`, it sends the appropriate request from
    the server to the client. When it gets a response, it forwards it on to
    the `CodexConversation`.
    
    Note this takes a lot of code from:
    
    
    https://github.com/openai/codex/blob/main/codex-rs/mcp-server/src/conversation_loop.rs
    
    https://github.com/openai/codex/blob/main/codex-rs/mcp-server/src/exec_approval.rs
    
    https://github.com/openai/codex/blob/main/codex-rs/mcp-server/src/patch_approval.rs
    
    I am copy/pasting for now because I am trying to consolidate around the
    new `wire_format.rs`, so I plan to delete these other files soon.
    
    Now that we have requests going both from client-to-server and
    server-to-client, I renamed `CodexRequest` to `ClientRequest`.
    
    ---
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/2286).
    * #2287
    * __->__ #2286
    * #2285
  • fix: verify notifications are sent with the conversationId set (#2278)
    This updates `CodexMessageProcessor` so that each notification it sends
    for a `EventMsg` from a `CodexConversation` such that:
    
    - The `params` always has an appropriate `conversationId` field.
    - The `method` is now includes the name of the `EventMsg` type rather
    than using `codex/event` as the `method` type for all notifications. (We
    currently prefix the method name with `codex/event/`, but I think that
    should go away once we formalize the notification schema in
    `wire_format.rs`.)
    
    As part of this, we update `test_codex_jsonrpc_conversation_flow()` to
    verify that the `task_finished` notification has made it through the
    system instead of sleeping for 5s and "hoping" the server finished
    processing the task. Note we have seen some flakiness in some of our
    other, similar integration tests, and I expect adding a similar check
    would help in those cases, as well.
  • feat: support traditional JSON-RPC request/response in MCP server (#2264)
    This introduces a new set of request types that our `codex mcp`
    supports. Note that these do not conform to MCP tool calls so that
    instead of having to send something like this:
    
    ```json
    {
      "jsonrpc": "2.0",
      "method": "tools/call",
      "id": 42,
      "params": {
        "name": "newConversation",
        "arguments": {
          "model": "gpt-5",
          "approvalPolicy": "on-request"
        }
      }
    }
    ```
    
    we can send something like this:
    
    
    ```json
    {
      "jsonrpc": "2.0",
      "method": "newConversation",
      "id": 42,
      "params": {
        "model": "gpt-5",
        "approvalPolicy": "on-request"
      }
    }
    ```
    
    Admittedly, this new format is not a valid MCP tool call, but we are OK
    with that right now. (That is, not everything we might want to request
    of `codex mcp` is something that is appropriate for an autonomous agent
    to do.)
    
    To start, this introduces four request types:
    
    - `newConversation`
    - `sendUserMessage`
    - `addConversationListener`
    - `removeConversationListener`
    
    The new `mcp-server/tests/codex_message_processor_flow.rs` shows how
    these can be used.
    
    The types are defined on the `CodexRequest` enum, so we introduce a new
    `CodexMessageProcessor` that is responsible for dealing with requests
    from this enum. The top-level `MessageProcessor` has been updated so
    that when `process_request()` is called, it first checks whether the
    request conforms to `CodexRequest` and dispatches it to
    `CodexMessageProcessor` if so.
    
    Note that I also decided to use `camelCase` for the on-the-wire format,
    as that seems to be the convention for MCP.
    
    For the moment, the new protocol is defined in `wire_format.rs` within
    the `mcp-server` crate, but in a subsequent PR, I will probably move it
    to its own crate to ensure the protocol has minimal dependencies and
    that we can codegen a schema from it.
    
    
    
    ---
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/2264).
    * #2278
    * __->__ #2264
  • fix: update OutgoingMessageSender::send_response() to take Serialize (#2263)
    This makes `send_response()` easier to work with.
  • chore: introduce ConversationManager as a clearinghouse for all conversations (#2240)
    This PR does two things because after I got deep into the first one I
    started pulling on the thread to the second:
    
    - Makes `ConversationManager` the place where all in-memory
    conversations are created and stored. Previously, `MessageProcessor` in
    the `codex-mcp-server` crate was doing this via its `session_map`, but
    this is something that should be done in `codex-core`.
    - It unwinds the `ctrl_c: tokio::sync::Notify` that was threaded
    throughout our code. I think this made sense at one time, but now that
    we handle Ctrl-C within the TUI and have a proper `Op::Interrupt` event,
    I don't think this was quite right, so I removed it. For `codex exec`
    and `codex proto`, we now use `tokio::signal::ctrl_c()` directly, but we
    no longer make `Notify` a field of `Codex` or `CodexConversation`.
    
    Changes of note:
    
    - Adds the files `conversation_manager.rs` and `codex_conversation.rs`
    to `codex-core`.
    - `Codex` and `CodexSpawnOk` are no longer exported from `codex-core`:
    other crates must use `CodexConversation` instead (which is created via
    `ConversationManager`).
    - `core/src/codex_wrapper.rs` has been deleted in favor of
    `ConversationManager`.
    - `ConversationManager::new_conversation()` returns `NewConversation`,
    which is in line with the `new_conversation` tool we want to add to the
    MCP server. Note `NewConversation` includes `SessionConfiguredEvent`, so
    we eliminate checks in cases like `codex-rs/core/tests/client.rs` to
    verify `SessionConfiguredEvent` is the first event because that is now
    internal to `ConversationManager`.
    - Quite a bit of code was deleted from
    `codex-rs/mcp-server/src/message_processor.rs` since it no longer has to
    manage multiple conversations itself: it goes through
    `ConversationManager` instead.
    - `core/tests/live_agent.rs` has been deleted because I had to update a
    bunch of tests and all the tests in here were ignored, and I don't think
    anyone ever ran them, so this was just technical debt, at this point.
    - Removed `notify_on_sigint()` from `util.rs` (and in a follow-up, I
    hope to refactor the blandly-named `util.rs` into more descriptive
    files).
    - In general, I started replacing local variables named `codex` as
    `conversation`, where appropriate, though admittedly I didn't do it
    through all the integration tests because that would have added a lot of
    noise to this PR.
    
    
    
    
    ---
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/2240).
    * #2264
    * #2263
    * __->__ #2240
  • Re-add markdown streaming (#2029)
    Wait for newlines, then render markdown on a line by line basis. Word wrap it for the current terminal size and then spit it out line by line into the UI. Also adds tests and fixes some UI regressions.
  • [mcp-server] Support CodexToolCallApprovalPolicy::OnRequest (#2187)
    ## Summary
    #1865 added `AskForApproval::OnRequest`, but missed adding it to our
    custom struct in `mcp-server`. This adds the missing configuration
    
    ## Testing
    - [x] confirmed locally
  • [1/3] Parse exec commands and format them more nicely in the UI (#2095)
    # Note for reviewers
    The bulk of this PR is in in the new file, `parse_command.rs`. This file
    is designed to be written TDD and implemented with Codex. Do not worry
    about reviewing the code, just review the unit tests (if you want). If
    any cases are missing, we'll add more tests and have Codex fix them.
    
    I think the best approach will be to land and iterate. I have some
    follow-ups I want to do after this lands. The next PR after this will
    let us merge (and dedupe) multiple sequential cells of the same such as
    multiple read commands. The deduping will also be important because the
    model often reads the same file multiple times in a row in chunks
    
    ===
    
    This PR formats common commands like reading, formatting, testing, etc
    more nicely:
    
    It tries to extract things like file names, tests and falls back to the
    cmd if it doesn't. It also only shows stdout/err if the command failed.
    
    <img width="770" height="238" alt="CleanShot 2025-08-09 at 16 05 15"
    src="https://github.com/user-attachments/assets/0ead179a-8910-486b-aa3d-7d26264d751e"
    />
    <img width="348" height="158" alt="CleanShot 2025-08-09 at 16 05 32"
    src="https://github.com/user-attachments/assets/4302681b-5e87-4ff3-85b4-0252c6c485a9"
    />
    <img width="834" height="324" alt="CleanShot 2025-08-09 at 16 05 56 2"
    src="https://github.com/user-attachments/assets/09fb3517-7bd6-40f6-a126-4172106b700f"
    />
    
    Part 2: https://github.com/openai/codex/pull/2097
    Part 3: https://github.com/openai/codex/pull/2110
  • [env] Remove git config for now (#1884)
    ## Summary
    Forgot to remove this in #1869 last night! Too much of a performance hit
    on the main thread. We can bring it back via an async thread on startup.
  • [prompts] Add <environment_context> (#1869)
    ## Summary
    Includes a new user message in the api payload which provides useful
    environment context for the model, so it knows about things like the
    current working directory and the sandbox.
    
    ## Testing
    Updated unit tests
  • [tests] Investigate flakey mcp-server test (#1877)
    ## Summary
    Have seen these tests flaking over the course of today on different
    boxes. `wiremock` seems to be generally written with tokio/threads in
    mind but based on the weird panics from the tests, let's see if this
    helps.
  • chore: remove unnecessary default_ prefix (#1854)
    This prefix is not inline with the other fields on the `ConfigOverrides`
    struct.
  • Introduce --oss flag to use gpt-oss models (#1848)
    This adds support for easily running Codex backed by a local Ollama
    instance running our new open source models. See
    https://github.com/openai/gpt-oss for details.
    
    If you pass in `--oss` you'll be prompted to install/launch ollama, and
    it will automatically download the 20b model and attempt to use it.
    
    We'll likely want to expand this with some options later to make the
    experience smoother for users who can't run the 20b or want to run the
    120b.
    
    Co-authored-by: Michael Bolin <mbolin@openai.com>
  • Rescue chat completion changes (#1846)
    https://github.com/openai/codex/pull/1835 has some messed up history.
    
    This adds support for streaming chat completions, which is useful for ollama. We should probably take a very skeptical eye to the code introduced in this PR.
    
    ---------
    
    Co-authored-by: Ahmed Ibrahim <aibrahim@openai.com>
  • Add a TurnDiffTracker to create a unified diff for an entire turn (#1770)
    This lets us show an accumulating diff across all patches in a turn.
    Refer to the docs for TurnDiffTracker for implementation details.
    
    There are multiple ways this could have been done and this felt like the
    right tradeoff between reliability and completeness:
    *Pros*
    * It will pick up all changes to files that the model touched including
    if they prettier or another command that updates them.
    * It will not pick up changes made by the user or other agents to files
    it didn't modify.
    
    *Cons*
    * It will pick up changes that the user made to a file that the model
    also touched
    * It will not pick up changes to codegen or files that were not modified
    with apply_patch
  • Fix flaky test_shell_command_approval_triggers_elicitation test (#1802)
    This doesn't flake very often but this should fix it.
  • MCP: add conversation.create tool [Stack 2/2] (#1783)
    Introduce conversation.create handler (handle_create_conversation) and
    wire it in MessageProcessor.
    
    Stack:
    Top: #1783 
    Bottom: #1784
    
    ---------
    
    Co-authored-by: Gabriel Peal <gpeal@users.noreply.github.com>
  • Add Error variant to ConversationCreateResult [Stack 1/2] (#1784)
    Switch ConversationCreateResult from a struct to a tagged enum (Ok |
    Error)
    
    Stack:
    Top: #1783 
    Bottom: #1784
  • feat: stream exec stdout events (#1786)
    ## Summary
    - stream command stdout as `ExecCommandStdout` events
    - forward streamed stdout to clients and ignore in human output
    processor
    - adjust call sites for new streaming API
  • Introduce a new function to just send user message [Stack 3/3] (#1686)
    - MCP server: add send-user-message tool to send user input to a running
    Codex session
    - Added an integration tests for the happy and sad paths
    
    Changes:
    •	Add tool definition and schema.
    •	Expose tool in capabilities.
    •	Route and handle tool requests with validation.
    •	Tests for success, bad UUID, and missing session.
    
    
    follow‑ups
    • Listen path not implemented yet; the tool is present but marked “don’t
    use yet” in code comments.
    • Session run flag reset: clear running_session_id_set appropriately
    after turn completion/errors.
    
    This is the third PR in a stack.
    Stack:
    Final: #1686
    Intermediate: #1751
    First: #1750
  • MCP server: route structured tool-call requests and expose mcp_protocol [Stack 2/3] (#1751)
    - Expose mcp_protocol from mcp-server for reuse in tests and callers.
    - In MessageProcessor, detect structured ToolCallRequestParams in
    tools/call and forward to a new handler.
    - Add handle_new_tool_calls scaffold (returns error for now).
    - Test helper: add send_send_user_message_tool_call to McpProcess to
    send ConversationSendMessage requests;
    
    This is the second PR in a stack.
    Stack:
    Final: #1686
    Intermediate: #1751
    First: #1750