Commit Graph

9 Commits

  • Upgrade rmcp to 0.14 (#10718)
    - [x] Upgrade rmcp to 0.14
  • Fix: Render MCP image outputs regardless of ordering (#9815)
    ## What?
    - Render an MCP image output cell whenever a decodable image block
    exists in `CallToolResult.content` (including text-before-image or
    malformed image before valid image).
    
    ## Why?
    - Tool results that include caption text before the image currently drop
    the image output cell.
    - A malformed image block can also suppress later valid image output.
    
    ## How?
    - Iterate `content` and return the first successfully decoded image
    instead of only checking the first block.
    - Add unit tests that cover text-before-image ordering and
    invalid-image-before-valid.
    
    ## Before
    ```rust
    let image = match result {
        Ok(mcp_types::CallToolResult { content, .. }) => {
            if let Some(mcp_types::ContentBlock::ImageContent(image)) = content.first() {
                // decode image (fails -> None)
            } else {
                None
            }
        }
        _ => None,
    }?;
    ```
    ## After
    ```rust
    let image = result
        .as_ref()
        .ok()?
        .content
        .iter()
        .find_map(decode_mcp_image)?;
    ```
    
    ## Risk / Impact
    - Low: only affects image cell creation for MCP tool results; no change
    for non-image outputs.
    
    ## Tests
    - [x] `just fmt`
    - [x] `cargo test -p codex-tui`
    - [x] Rerun after branch update (2026-01-27): `just fmt`, `cargo test -p
    codex-tui`
    
    Manual testing
    
    # Manual testing: MCP image tool result rendering (Codex TUI)
    
    # Build the rmcp stdio test server binary:
    cd codex-rs
    cargo build -p codex-rmcp-client --bin test_stdio_server
    
    # Register the server as an MCP server (absolute path to the built binary):
    codex mcp add mcpimg -- /Users/joshka/code/codex-pr-review/codex-rs/target/debug/test_stdio_server
    
    # Then in Codex TUI, ask it to call:
    - mcpimg.image_scenario({"scenario":"image_only"})
    - mcpimg.image_scenario({"scenario":"text_then_image","caption":"Here is the image:"})
    - mcpimg.image_scenario({"scenario":"invalid_base64_then_image"})
    - mcpimg.image_scenario({"scenario":"invalid_image_bytes_then_image"})
    - mcpimg.image_scenario({"scenario":"multiple_valid_images"})
    - mcpimg.image_scenario({"scenario":"image_then_text","caption":"Here is the image:"})
    - mcpimg.image_scenario({"scenario":"text_only","caption":"Here is the image:"})
    
    # Expected:
    # - You should see an extra history cell: "tool result (image output)" when the
    #   tool result contains at least one decodable image block (even if earlier
    #   blocks are text or invalid images).
    
    
    Fixes #9814
    
    ---------
    
    Co-authored-by: Josh McKinney <joshka@openai.com>
  • chore: upgrade rmcp crate from 0.10.0 to 0.12.0 (#8288)
    Version `0.12.0` includes
    https://github.com/modelcontextprotocol/rust-sdk/pull/590, which I will
    use in https://github.com/openai/codex/pull/8142.
    
    Changes:
    
    - `rmcp::model::CustomClientNotification` was renamed to
    `rmcp::model::CustomNotification`
    - a bunch of types have a `meta` field now, but it is `Option`, so I
    added `meta: None` to a bunch of things
  • [MCP] Render MCP tool call result images to the model (#5600)
    It's pretty amazing we have gotten here without the ability for the
    model to see image content from MCP tool calls.
    
    This PR builds off of 4391 and fixes #4819. I would like @KKcorps to get
    adequete credit here but I also want to get this fix in ASAP so I gave
    him a week to update it and haven't gotten a response so I'm going to
    take it across the finish line.
    
    
    This test highlights how absured the current situation is. I asked the
    model to read this image using the Chrome MCP
    <img width="2378" height="674" alt="image"
    src="https://github.com/user-attachments/assets/9ef52608-72a2-4423-9f5e-7ae36b2b56e0"
    />
    
    After this change, it correctly outputs:
    > Captured the page: image dhows a dark terminal-style UI labeled
    `OpenAI Codex (v0.0.0)` with prompt `model: gpt-5-codex medium` and
    working directory `/codex/codex-rs`
    (and more)  
    
    Before this change, it said:
    > Took the full-page screenshot you asked for. It shows a long,
    horizontally repeating pattern of stylized people in orange, light-blue,
    and mustard clothing, holding hands in alternating poses against a white
    background. No text or other graphics-just rows of flat illustration
    stretching off to the right.
    
    Without this change, the Figma, Playwright, Chrome, and other visual MCP
    servers are pretty much entirely useless.
    
    I tested this change with the openai respones api as well as a third
    party completions api
  • [MCP] Add support for resources (#5239)
    This PR adds support for [MCP
    resources](https://modelcontextprotocol.io/specification/2025-06-18/server/resources)
    by adding three new tools for the model:
    1. `list_resources`
    2. `list_resource_templates`
    3. `read_resource`
    
    These 3 tools correspond to the [three primary MCP resource protocol
    messages](https://modelcontextprotocol.io/specification/2025-06-18/server/resources#protocol-messages).
    
    Example of listing and reading a GitHub resource tempalte
    <img width="2984" height="804" alt="CleanShot 2025-10-15 at 17 31 10"
    src="https://github.com/user-attachments/assets/89b7f215-2e2a-41c5-90dd-b932ac84a585"
    />
    
    `/mcp` with Figma configured
    <img width="2984" height="442" alt="CleanShot 2025-10-15 at 18 29 35"
    src="https://github.com/user-attachments/assets/a7578080-2ed2-4c59-b9b4-d8461f90d8ee"
    />
    
    Fixes #4956
  • [MCP] Add support for MCP Oauth credentials (#4517)
    This PR adds oauth login support to streamable http servers when
    `experimental_use_rmcp_client` is enabled.
    
    This PR is large but represents the minimal amount of work required for
    this to work. To keep this PR smaller, login can only be done with
    `codex mcp login` and `codex mcp logout` but it doesn't appear in `/mcp`
    or `codex mcp list` yet. Fingers crossed that this is the last large MCP
    PR and that subsequent PRs can be smaller.
    
    Under the hood, credentials are stored using platform credential
    managers using the [keyring crate](https://crates.io/crates/keyring).
    When the keyring isn't available, it falls back to storing credentials
    in `CODEX_HOME/.credentials.json` which is consistent with how other
    coding agents handle authentication.
    
    I tested this on macOS, Windows, WSL (ubuntu), and Linux. I wasn't able
    to test the dbus store on linux but did verify that the fallback works.
    
    One quirk is that if you have credentials, during development, every
    build will have its own ad-hoc binary so the keyring won't recognize the
    reader as being the same as the write so it may ask for the user's
    password. I may add an override to disable this or allow
    users/enterprises to opt-out of the keyring storage if it causes issues.
    
    <img width="5064" height="686" alt="CleanShot 2025-09-30 at 19 31 40"
    src="https://github.com/user-attachments/assets/9573f9b4-07f1-4160-83b8-2920db287e2d"
    />
    <img width="745" height="486" alt="image"
    src="https://github.com/user-attachments/assets/9562649b-ea5f-4f22-ace2-d0cb438b143e"
    />
  • [MCP] Add experimental support for streamable HTTP MCP servers (#4317)
    This PR adds support for streamable HTTP MCP servers when the
    `experimental_use_rmcp_client` is enabled.
    
    To set one up, simply add a new mcp server config with the url:
    ```
    [mcp_servers.figma]
    url = "http://127.0.0.1:3845/mcp"
    ```
    
    It also supports an optional `bearer_token` which will be provided in an
    authorization header. The full oauth flow is not supported yet.
    
    The config parsing will throw if it detects that the user mixed and
    matched config fields (like command + bearer token or url + env).
    
    The best way to review it is to review `core/src` and then
    `rmcp-client/src/rmcp_client.rs` first. The rest is tests and
    propagating the `Transport` struct around the codebase.
    
    Example with the Figma MCP:
    <img width="5084" height="1614" alt="CleanShot 2025-09-26 at 13 35 40"
    src="https://github.com/user-attachments/assets/eaf2771e-df3e-4300-816b-184d7dec5a28"
    />
  • [MCP] Introduce an experimental official rust sdk based mcp client (#4252)
    The [official Rust
    SDK](https://github.com/modelcontextprotocol/rust-sdk/tree/57fc428c578a1a3fe851ee0838bf068bda120eb3)
    has come a long way since we first started our mcp client implementation
    5 months ago and, today, it is much more complete than our own
    stdio-only implementation.
    
    This PR introduces a new config flag `experimental_use_rmcp_client`
    which will use a new mcp client powered by the sdk instead of our own.
    
    To keep this PR simple, I've only implemented the same stdio MCP
    functionality that we had but will expand on it with future PRs.
    
    ---------
    
    Co-authored-by: pakrym-oai <pakrym@openai.com>