Commit Graph

263 Commits

  • Add the utility to truncate by tokens (#6746)
    - This PR is to make it on path for truncating by tokens. This path will
    be initially used by unified exec and context manager (responsible for
    MCP calls mainly).
    - We are exposing new config `calls_output_max_tokens`
    - Use `tokens` as the main budget unit but truncate based on the model
    family by Introducing `TruncationPolicy`.
    - Introduce `truncate_text` as a router for truncation based on the
    mode.
    
    In next PRs:
    - remove truncate_with_line_bytes_budget
    - Add the ability to the model to override the token budget.
  • feat: remote compaction (#6795)
    Co-authored-by: pakrym-oai <pakrym@openai.com>
  • [app-server] feat: add v2 command execution approval flow (#6758)
    This PR adds the API V2 version of the command‑execution approval flow
    for the shell tool.
    
    This PR wires the new RPC (`item/commandExecution/requestApproval`, V2
    only) and related events (`item/started`, `item/completed`, and
    `item/commandExecution/delta`, which are emitted in both V1 and V2)
    through the app-server
    protocol. The new approval RPC is only sent when the user initiates a
    turn with the new `turn/start` API so we don't break backwards
    compatibility with VSCE.
    
    The approach I took was to make as few changes to the Codex core as
    possible, leveraging existing `EventMsg` core events, and translating
    those in app-server. I did have to add additional fields to
    `EventMsg::ExecCommandEndEvent` to capture the command's input so that
    app-server can statelessly transform these events to a
    `ThreadItem::CommandExecution` item for the `item/completed` event.
    
    Once we stabilize the API and it's complete enough for our partners, we
    can work on migrating the core to be aware of command execution items as
    a first-class concept.
    
    **Note**: We'll need followup work to make sure these APIs work for the
    unified exec tool, but will wait til that's stable and landed before
    doing a pass on app-server.
    
    Example payloads below:
    ```
    {
      "method": "item/started",
      "params": {
        "item": {
          "aggregatedOutput": null,
          "command": "/bin/zsh -lc 'touch /tmp/should-trigger-approval'",
          "cwd": "/Users/owen/repos/codex/codex-rs",
          "durationMs": null,
          "exitCode": null,
          "id": "call_lNWWsbXl1e47qNaYjFRs0dyU",
          "parsedCmd": [
            {
              "cmd": "touch /tmp/should-trigger-approval",
              "type": "unknown"
            }
          ],
          "status": "inProgress",
          "type": "commandExecution"
        }
      }
    }
    ```
    
    ```
    {
      "id": 0,
      "method": "item/commandExecution/requestApproval",
      "params": {
        "itemId": "call_lNWWsbXl1e47qNaYjFRs0dyU",
        "parsedCmd": [
          {
            "cmd": "touch /tmp/should-trigger-approval",
            "type": "unknown"
          }
        ],
        "reason": "Need to create file in /tmp which is outside workspace sandbox",
        "risk": null,
        "threadId": "019a93e8-0a52-7fe3-9808-b6bc40c0989a",
        "turnId": "1"
      }
    }
    ```
    
    ```
    {
      "id": 0,
      "result": {
        "acceptSettings": {
          "forSession": false
        },
        "decision": "accept"
      }
    }
    ```
    
    ```
    {
      "params": {
        "item": {
          "aggregatedOutput": null,
          "command": "/bin/zsh -lc 'touch /tmp/should-trigger-approval'",
          "cwd": "/Users/owen/repos/codex/codex-rs",
          "durationMs": 224,
          "exitCode": 0,
          "id": "call_lNWWsbXl1e47qNaYjFRs0dyU",
          "parsedCmd": [
            {
              "cmd": "touch /tmp/should-trigger-approval",
              "type": "unknown"
            }
          ],
          "status": "completed",
          "type": "commandExecution"
        }
      }
    }
    ```
  • core/tui: non-blocking MCP startup (#6334)
    This makes MCP startup not block TUI startup. Messages sent while MCPs
    are booting will be queued.
    
    
    https://github.com/user-attachments/assets/96e1d234-5d8f-4932-a935-a675d35c05e0
    
    
    Fixes #6317
    
    ---------
    
    Co-authored-by: pakrym-oai <pakrym@openai.com>
  • Improve compact (#6692)
    This PR does the following:
    - Add compact prefix to the summary
    - Change the compaction prompt
    - Allow multiple compaction for long running tasks
    - Filter out summary messages on the following compaction
    
    Considerations:
    - Filtering out the summary message isn't the most clean
    - Theoretically, we can end up in infinite compaction loop if the user
    messages > compaction limit . However, that's not possible in today's
    code because we have hard cap on user messages.
    - We need to address having multiple user messages because it confuses
    the model.
    
    Testing:
    - Making sure that after compact we always end up with one user message
    (task) and one summary, even on multiple compaction.
  • [App-server] add new v2 events:item/reasoning/delta, item/agentMessage/delta & item/reasoning/summaryPartAdded (#6559)
    core event to app server event mapping:
    1. `codex/event/reasoning_content_delta` ->
    `item/reasoning/summaryTextDelta`.
    2. `codex/event/reasoning_raw_content_delta` ->
    `item/reasoning/textDelta`
    3. `codex/event/agent_message_content_delta` →
    `item/agentMessage/delta`.
    4. `codex/event/agent_reasoning_section_break` ->
    `item/reasoning/summaryPartAdded`.
    
    Also added a change in core to pass down content index, summary index
    and item id from events.
    
    Tested with the `git checkout owen/app_server_test_client && cargo run
    -p codex-app-server-test-client -- send-message-v2 "hello"` and verified
    that new events are emitted correctly.
  • feat: warning switch model on resume (#6507)
    <img width="1259" height="40" alt="Screenshot 2025-11-11 at 14 01 41"
    src="https://github.com/user-attachments/assets/48ead3d2-d89c-4d8a-a578-82d9663dbd88"
    />
  • Add unified exec escalation handling and tests (#6492)
    Similar implementation to the shell tool
  • Remove shell tool when unified exec is enabled (#6345)
    Also drop streameable shell that's just an alias for unified exec.
  • Don't retry "insufficient_quota" errors (#6340)
    This PR makes an "insufficient quota" error fatal so we don't attempt to
    retry it multiple times in the agent loop.
    
    We have multiple bug reports from users about intermittent retry
    behaviors, and this could explain some of them. With this change, we'll
    eliminate the retries and surface a clear error message.
    
    The PR is a nearly identical copy of [this
    PR](https://github.com/openai/codex/pull/4837) contributed by
    @abimaelmartell. The original PR has gone stale. Rather than wait for
    the contributor to resolve merge conflicts, I wanted to get this change
    in.
  • Improved token refresh handling to address "Re-connecting" behavior (#6231)
    Currently, when the access token expires, we attempt to use the refresh
    token to acquire a new access token. This works most of the time.
    However, there are situations where the refresh token is expired,
    exhausted (already used to perform a refresh), or revoked. In those
    cases, the current logic treats the error as transient and attempts to
    retry it repeatedly.
    
    This PR changes the token refresh logic to differentiate between
    permanent and transient errors. It also changes callers to treat the
    permanent errors as fatal rather than retrying them. And it provides
    better error messages to users so they understand how to address the
    problem. These error messages should also help us further understand why
    we're seeing examples of refresh token exhaustion.
    
    Here is the error message in the CLI. The same text appears within the
    extension.
    
    <img width="863" height="38" alt="image"
    src="https://github.com/user-attachments/assets/7ffc0d08-ebf0-4900-b9a9-265064202f4f"
    />
    
    I also correct the spelling of "Re-connecting", which shouldn't have a
    hyphen in it.
    
    Testing: I manually tested these code paths by adding temporary code to
    programmatically cause my refresh token to be exhausted (by calling the
    token refresh endpoint in a tight loop more than 50 times). I then
    simulated an access token expiration, which caused the token refresh
    logic to be invoked. I confirmed that the updated logic properly handled
    the error condition.
    
    Note: We earlier discussed the idea of forcefully logging out the user
    at the point where token refresh failed. I made several attempts to do
    this, and all of them resulted in a bad UX. It's important to surface
    this error to users in a way that explains the problem and tells them
    that they need to log in again. We also previously discussed deleting
    the auth.json file when this condition is detected. That also creates
    problems because it effectively changes the auth status from logged in
    to logged out, and this causes odd failures and inconsistent UX. I think
    it's therefore better not to delete auth.json in this case. If the user
    closes the CLI or VSCE and starts it again, we properly detect that the
    access token is expired and the refresh token is "dead", and we force
    the user to go through the login flow at that time.
    
    This should address aspects of #6191, #5679, and #5505
  • refactor Conversation history file into its own directory (#6229)
    This is just a refactor of `conversation_history` file by breaking it up
    into multiple smaller ones with helper. This refactor will help us move
    more functionality related to context management here. in a clean way.
  • fix: Update the deprecation message to link to the docs (#6211)
    The deprecation message is currently a bit confusing. Users may not
    understand what is `[features].x`. I updated the docs and the
    deprecation message for more guidance.
    
    ---------
    
    Co-authored-by: Gabriel Peal <gpeal@users.noreply.github.com>
  • Fixed notify handler so it passes correct input_messages details (#6143)
    This fixes bug #6121. 
    
    The `input_messages` field passed to the notify handler is currently
    empty because the logic is incorrectly including the OutputText rather
    than InputText. I've fixed that and added proper filtering to remove
    messages associated with AGENTS.md and other context injected by the
    harness.
    
    Testing: I wrote a notify handler and verified that the user prompt is
    correctly passed through to the handler.
  • [codex] add developer instructions (#5897)
    we are using developer instructions for code reviews, we need to pass
    them in cli as well.
  • feat: compaction prompt configurable (#5959)
    ```
     codex -c compact_prompt="Summarize in bullet points"
     ```
  • Add item streaming events (#5546)
    Adds AgentMessageContentDelta, ReasoningContentDelta,
    ReasoningRawContentDelta item streaming events while maintaining
    compatibility for old events.
    
    ---------
    
    Co-authored-by: Owen Lin <owen@openai.com>
  • Delegate review to codex instance (#5572)
    In this PR, I am exploring migrating task kind to an invocation of
    Codex. The main reason would be getting rid off multiple
    `ConversationHistory` state and streamlining our context/history
    management.
    
    This approach depends on opening a channel between the sub-codex and
    codex. This channel is responsible for forwarding `interactive`
    (`approvals`) and `non-interactive` events. The `task` is responsible
    for handling those events.
    
    This opens the door for implementing `codex as a tool`, replacing
    `compact` and `review`, and potentially subagents.
    
    One consideration is this code is very similar to `app-server` specially
    in the approval part. If in the future we wanted an interactive
    `sub-codex` we should consider using `codex-mcp`
  • Add a wrapper around raw response items (#5923)
    We currently have nested enums when sending raw response items in the
    app-server protocol. This makes downstream schemas confusing because we
    need to embed `type`-discriminated enums within each other.
    
    This PR adds a small wrapper around the response item so we can keep the
    schemas separate
  • feat: deprecation warning (#5825)
    <img width="955" height="311" alt="Screenshot 2025-10-28 at 14 26 25"
    src="https://github.com/user-attachments/assets/99729b3d-3bc9-4503-aab3-8dc919220ab4"
    />
  • feature: Add "!cmd" user shell execution (#2471)
    feature: Add "!cmd" user shell execution
    
    This change lets users run local shell commands directly from the TUI by
    prefixing their input with ! (e.g. !ls). Output is truncated to keep the
    exec cell usable, and Ctrl-C cleanly
      interrupts long-running commands (e.g. !sleep 10000).
    
    **Summary of changes**
    
    - Route Op::RunUserShellCommand through a dedicated UserShellCommandTask
    (core/src/tasks/user_shell.rs), keeping the task logic out of codex.rs.
    - Reuse the existing tool router: the task constructs a ToolCall for the
    local_shell tool and relies on ShellHandler, so no manual MCP tool
    lookup is required.
    - Emit exec lifecycle events (ExecCommandBegin/ExecCommandEnd) so the
    TUI can show command metadata, live output, and exit status.
    
    **End-to-end flow**
    
      **TUI handling**
    
    1. ChatWidget::submit_user_message (TUI) intercepts messages starting
    with !.
    2. Non-empty commands dispatch Op::RunUserShellCommand { command };
    empty commands surface a help hint.
    3. No UserInput items are created, so nothing is enqueued for the model.
    
      **Core submission loop**
    4. The submission loop routes the op to handlers::run_user_shell_command
    (core/src/codex.rs).
    5. A fresh TurnContext is created and Session::spawn_user_shell_command
    enqueues UserShellCommandTask.
    
      **Task execution**
    6. UserShellCommandTask::run emits TaskStartedEvent, formats the
    command, and prepares a ToolCall targeting local_shell.
      7. ToolCallRuntime::handle_tool_call dispatches to ShellHandler.
    
      **Shell tool runtime**
    8. ShellHandler::run_exec_like launches the process via the unified exec
    runtime, honoring sandbox and shell policies, and emits
    ExecCommandBegin/End.
    9. Stdout/stderr are captured for the UI, but the task does not turn the
    resulting ToolOutput into a model response.
    
      **Completion**
    10. After ExecCommandEnd, the task finishes without an assistant
    message; the session marks it complete and the exec cell displays the
    final output.
    
      **Conversation context**
    
    - The command and its output never enter the conversation history or the
    model prompt; the flow is local-only.
      - Only exec/task events are emitted for UI rendering.
    
    **Demo video**
    
    
    https://github.com/user-attachments/assets/fcd114b0-4304-4448-a367-a04c43e0b996
  • Filter out reasoning items from previous turns (#5857)
    Reduces request size and prevents 400 errors when switching between API
    orgs.
    
    Based on Responses API behavior described in
    https://cookbook.openai.com/examples/responses_api/reasoning_items#caching
  • [Auth] Choose which auth storage to use based on config (#5792)
    This PR is a follow-up to #5591. It allows users to choose which auth
    storage mode they want by using the new
    `cli_auth_credentials_store_mode` config.
  • [MCP] Render MCP tool call result images to the model (#5600)
    It's pretty amazing we have gotten here without the ability for the
    model to see image content from MCP tool calls.
    
    This PR builds off of 4391 and fixes #4819. I would like @KKcorps to get
    adequete credit here but I also want to get this fix in ASAP so I gave
    him a week to update it and haven't gotten a response so I'm going to
    take it across the finish line.
    
    
    This test highlights how absured the current situation is. I asked the
    model to read this image using the Chrome MCP
    <img width="2378" height="674" alt="image"
    src="https://github.com/user-attachments/assets/9ef52608-72a2-4423-9f5e-7ae36b2b56e0"
    />
    
    After this change, it correctly outputs:
    > Captured the page: image dhows a dark terminal-style UI labeled
    `OpenAI Codex (v0.0.0)` with prompt `model: gpt-5-codex medium` and
    working directory `/codex/codex-rs`
    (and more)  
    
    Before this change, it said:
    > Took the full-page screenshot you asked for. It shows a long,
    horizontally repeating pattern of stylized people in orange, light-blue,
    and mustard clothing, holding hands in alternating poses against a white
    background. No text or other graphics-just rows of flat illustration
    stretching off to the right.
    
    Without this change, the Figma, Playwright, Chrome, and other visual MCP
    servers are pretty much entirely useless.
    
    I tested this change with the openai respones api as well as a third
    party completions api
  • Centralize truncation in conversation history (#5652)
    move the truncation logic to conversation history to use on any tool
    output. This will help us in avoiding edge cases while truncating the
    tool calls and mcp calls.
  • Made token refresh code resilient to missing id_token (#5782)
    This PR does the following:
    1. Changes `try_refresh_token` to handle the case where the endpoint
    returns a response without an `id_token`. The OpenID spec indicates that
    this field is optional and clients should not assume it's present.
    2. Changes the `attempt_stream_responses` to propagate token refresh
    errors rather than silently ignoring them.
    3. Fixes a typo in a couple of error messages (unrelated to the above,
    but something I noticed in passing) - "reconnect" should be spelled
    without a hyphen.
    
    This PR does not implement the additional suggestion from @pakrym-oai
    that we should sign out when receiving `refresh_token_expired` from the
    refresh endpoint. Leaving this as a follow-on because I'm undecided on
    whether this should be implemented in `try_refresh_token` or its
    callers.
  • Added model summary and risk assessment for commands that violate sandbox policy (#5536)
    This PR adds support for a model-based summary and risk assessment for
    commands that violate the sandbox policy and require user approval. This
    aids the user in evaluating whether the command should be approved.
    
    The feature works by taking a failed command and passing it back to the
    model and asking it to summarize the command, give it a risk level (low,
    medium, high) and a risk category (e.g. "data deletion" or "data
    exfiltration"). It uses a new conversation thread so the context in the
    existing thread doesn't influence the answer. If the call to the model
    fails or takes longer than 5 seconds, it falls back to the current
    behavior.
    
    For now, this is an experimental feature and is gated by a config key
    `experimental_sandbox_command_assessment`.
    
    Here is a screen shot of the approval prompt showing the risk assessment
    and summary.
    
    <img width="723" height="282" alt="image"
    src="https://github.com/user-attachments/assets/4597dd7c-d5a0-4e9f-9d13-414bd082fd6b"
    />
  • [MCP] Improve startup errors for timeouts and github (#5595)
    1. I have seen too many reports of people hitting startup timeout errors
    and thinking Codex is broken. Hopefully this will help people
    self-serve. We may also want to consider raising the timeout to ~15s.
    2. Make it more clear what PAT is (personal access token) in the GitHub
    error
    
    <img width="2378" height="674" alt="CleanShot 2025-10-23 at 22 05 06"
    src="https://github.com/user-attachments/assets/d148ce1d-ade3-4511-84a4-c164aefdb5c5"
    />
  • Handle cancelling/aborting while processing a turn (#5543)
    Currently we collect all all turn items in a vector, then we add it to
    the history on success. This result in losing those items on errors
    including aborting `ctrl+c`.
    
    This PR:
    - Adds the ability for the tool call to handle cancellation
    - bubble the turn items up to where we are recording this info
    
    Admittedly, this logic is an ad-hoc logic that doesn't handle a lot of
    error edge cases. The right thing to do is recording to the history on
    the spot as `items`/`tool calls output` come. However, this isn't
    possible because of having different `task_kind` that has different
    `conversation_histories`. The `try_run_turn` has no idea what thread are
    we using. We cannot also pass an `arc` to the `conversation_histories`
    because it's a private element of `state`.
    
    That's said, `abort` is the most common case and we should cover it
    until we remove `task kind`
  • chore: clean handle_container_exec_with_params (#5516)
    Drop `handle_container_exec_with_params` to have simpler and more
    straight forward execution path
  • Move changing turn input functionalities to ConversationHistory (#5473)
    We are doing some ad-hoc logic while dealing with conversation history.
    Ideally, we shouldn't mutate `vec[responseitem]` manually at all and
    should depend on `ConversationHistory` for those changes.
    
    Those changes are:
    - Adding input to the history
    - Removing items from the history
    - Correcting history
    
    I am also adding some `error` logs for cases we shouldn't ideally face.
    For example, we shouldn't be missing `toolcalls` or `outputs`. We
    shouldn't hit `ContextWindowExceeded` while performing `compact`
    
    This refactor will give us granular control over our context management.
  • [MCP] Remove the legacy stdio client in favor of rmcp (#5529)
    I haven't heard of any issues with the studio rmcp client so let's
    remove the legacy one and default to the new one.
    
    Any code changes are moving code from the adapter inline but there
    should be no meaningful functionality changes.
  • Add new thread items and rewire event parsing to use them (#5418)
    1. Adds AgentMessage,  Reasoning,  WebSearch items.
    2. Switches the ResponseItem parsing to use new items and then also emit
    3. Removes user-item kind and filters out "special" (environment) user
    items when returning to clients.
  • Enable plan tool by default (#5384)
    ## Summary
    - make the plan tool available by default by removing the feature flag
    and always registering the handler
    - drop plan-tool CLI and API toggles across the exec, TUI, MCP server,
    and app server code paths
    - update tests and configs to reflect the always-on plan tool and guard
    workspace restriction tests against env leakage
    
    ## Testing
    Manually tested the extension. 
    ------
    https://chatgpt.com/codex/tasks/task_i_68f67a3ff2d083209562a773f814c1f9
  • Pass TurnContext around instead of sub_id (#5421)
    Today `sub_id` is an ID of a single incoming Codex Op submition. We then
    associate all events triggered by this operation using the same
    `sub_id`.
    
    At the same time we are also creating a TurnContext per submission and
    we'd like to start associating some events (item added/item completed)
    with an entire turn instead of just the operation that started it.
    
    Using turn context when sending events give us flexibility to change
    notification scheme.
  • feat: include cwd in notify payload (#5415)
    Expose the session cwd in the notify payload and update docs so scripts
    and extensions receive the real project path; users get accurate
    project-aware notifications in CLI and VS Code.
    
    Fixes #5387
  • [MCP] Dedicated error message for GitHub MCPs missing a personal access token (#5393)
    Because the GitHub MCP is one of the most popular MCPs and it
    confusingly doesn't support OAuth, we should make it more clear how to
    make it work so people don't think Codex is broken.