Commit Graph

3066 Commits

  • feat: add memory tool (#10637)
    Add a tool for memory to retrieve a full memory based on the memory ID
  • feat: resumable backfill (#10745)
    ## Summary
    
    This PR makes SQLite rollout backfill resumable and repeatable instead
    of one-shot-on-db-create.
    
    ## What changed
    
    - Added a persisted backfill state table:
      - state/migrations/0008_backfill_state.sql
    - Tracks status (pending|running|complete), last_watermark, and
    last_success_at.
    - Added backfill state model/types in codex-state:
      - BackfillState, BackfillStatus (state/src/model/backfill_state.rs)
    - Added runtime APIs to manage backfill lifecycle/progress:
      - get_backfill_state
      - mark_backfill_running
      - checkpoint_backfill
      - mark_backfill_complete
    - Updated core startup behavior:
    - Backfill now runs whenever state is not Complete (not only when DB
    file is newly created).
    - Reworked backfill execution:
    - Collect rollout files, derive deterministic watermark per path, sort,
    resume from last_watermark.
    - Process in batches (BACKFILL_BATCH_SIZE = 200), checkpoint after each
    batch.
      - Mark complete with last_success_at at the end.
    
    ## Why
    
    Previous behavior could leave users permanently partially backfilled if
    the process exited during initial async backfill. This change allows
    safe continuation across restarts and avoids restarting from scratch.
  • Include real OS info in metrics. (#10425)
    calculated a hashed user ID from either auth user id or API key
    Also correctly populates OS.
    
    These will make our metrics more useful and powerful for analysis.
  • Update explorer role default model (#10748)
    Summary
    - switch the explorer role in core agent configuration to use
    `gpt-5.1-codex-mini` as the default model override
    - leave other role defaults untouched
    
    Testing
    - Not run (not requested)
  • adding fork information (UI) when forking (#10246)
    - shows `/fork` command that ran in prev session
    - shows `session forked from name (uuid) || uuid (if name is not set)` as an event in new session
  • Allow user shell commands to run alongside active turns (#10513)
    Summary
    - refactor user shell command execution into a shared helper and add
    modes for standalone vs active-turn execution
    - run user shell commands asynchronously when a turn is already active
    so they don’t replace or abort the current turn
    - extend the tests to cover the new behavior and add the generated Codex
    environment manifest
    
    Testing
    - Not run (not requested)
  • fix(tui): flush input buffer on init to prevent early exit on Windows (#10729)
    Fixes #10661.
    
    ### Problem
    On Windows, the sign-in menu can exit immediately if the OS-level input
    buffer contains trailing characters (like the Enter key from running the
    command).
    
    ### Solution
    **Flush Input Buffer on Init**: Use FlushConsoleInputBuffer on Windows
    (and cflush on Unix) in ui::init() to discard any input captured before
    the TUI was ready.
    
    Verified by @CodebyAmbrose in #10661.
  • fix(core,app-server) resume with different model (#10719)
    ## Summary
    When resuming with a different model, we should also append a developer
    message with the model instructions
    
    ## Testing
    - [x] Added unit tests
  • Reload cloud requirements after user login (#10725)
    Reload cloud requirements after user login so it could take effect
    immediately.
  • Fix remote compaction estimator/payload instruction small mismatch (#10692)
    ## Summary
    This PR fixes a deterministic mismatch in remote compaction where
    pre-trim estimation and the `/v1/responses/compact` payload could use
    different base instructions.
    
    Before this change:
    - pre-trim estimation used model-derived instructions
    (`model_info.get_model_instructions(...)`)
    - compact payload used session base instructions
    (`sess.get_base_instructions()`)
    
    After this change:
    - remote pre-trim estimation and compact payload both use the same
    `BaseInstructions` instance from session state.
    
    ## Changes
    - Added a shared estimator entry point in `ContextManager`:
    - `estimate_token_count_with_base_instructions(&self, base_instructions:
    &BaseInstructions) -> Option<i64>`
    - Kept `estimate_token_count(&TurnContext)` as a thin wrapper that
    resolves model/personality instructions and delegates to the new helper.
    - Updated remote compaction flow to fetch base instructions once and
    reuse it for both:
      - trim preflight estimation
      - compact request payload construction
    - Added regression coverage for parity and behavior:
      - unit test verifying explicit-base estimator behavior
    - integration test proving remote compaction uses session override
    instructions and trims accordingly
    
    ## Why this matters
    This removes a deterministic divergence source where pre-trim could
    think the request fits while the actual compact request exceeded context
    because its instructions were longer/different.
    
    ## Scope
    In scope:
    - estimator/payload base-instructions parity in remote compaction
    
    Out of scope:
    - retry-on-`context_length_exceeded`
    - compaction threshold/headroom policy changes
    - broader trimming policy changes
    
    ## Codex author:
    `codex fork 019c2b24-c2df-7b31-a482-fb8cf7a28559`
  • Make steer stable by default (#10690)
    Promotes the Steer feature from Experimental to Stable and enables it by
    default.
    
    ## What is Steer mode?
    
    Steer mode changes how message submission works in the TUI:
    
    - **With Steer enabled (new default)**: 
      - `Enter` submits messages immediately, even when a task is running
    - `Tab` queues messages when a task is running (allows building up a
    queue)
      
    - **With Steer disabled (old behavior)**:
      - `Enter` queues messages when a task is running
      - This preserves the previous "queue while a task is running" behavior
    
    ## How Steer vs Queue work
    
    The key difference is in the submission behavior:
    
    1. **Steer mode** (`steer_enabled = true`):
    - Enter → `InputResult::Submitted` → sends immediately via
    `submit_user_message()`
    - Tab → `InputResult::Queued` → queues via `queue_user_message()` if a
    task is running
    - This gives users direct control: Enter for immediate submission, Tab
    for queuing
    
    2. **Queue mode** (`steer_enabled = false`, previous default):
    - Enter → `InputResult::Queued` → always queues when a task is running
       - Tab → `InputResult::Queued` → queues when a task is running
    - This preserves the original behavior where Enter respects the running
    task queue
    
    ## Implementation details
    
    The behavior is controlled in
    `ChatComposer::handle_key_event_without_popup()`:
    - When `steer_enabled` is true, Enter calls `handle_submission(false)`
    (submit immediately)
    - When `steer_enabled` is false, Enter calls `handle_submission(true)`
    (queue)
    
    See `codex-rs/tui/src/bottom_pane/chat_composer.rs` for the
    implementation.
    
    ## Documentation
    
    For more details on the chat composer behavior, see:
    - [TUI Chat Composer documentation](docs/tui-chat-composer.md)
    - Feature flag definition: `codex-rs/core/src/features.rs`
  • Sync collaboration mode naming across Default prompt, tools, and TUI (#10666)
    ## Summary
    - add shared `ModeKind` helpers for display names, TUI visibility, and
    `request_user_input` availability
    - derive TUI mode filtering/labels from shared `ModeKind` metadata
    instead of local hardcoded matches
    - derive `request_user_input` availability text and unavailable error
    mode names from shared mode metadata
    - replace hardcoded known mode names in the Default collaboration-mode
    template with `{{KNOWN_MODE_NAMES}}` and fill it from
    `TUI_VISIBLE_COLLABORATION_MODES`
    - add regression tests for mode metadata sync and placeholder
    replacement
    
    ## Notes
    - `cargo test -p codex-core` integration target (`tests/all`) still
    shows pre-existing env-specific failures in this environment due missing
    `test_stdio_server` binary resolution; core unit tests are green.
    
    ## Codex author
    `codex resume 019c26ff-dfe7-7173-bc04-c9e1fff1e447`
  • fix(core) switching model appends model instructions (#10651)
    ## Summary
    When switching models, we should append the instructions of the new
    model to the conversation as a developer message.
    
    ## Test
    - [x] Adds a unit test
  • chore(config) Default Personality Pragmatic (#10705)
    ## Summary
    Switch back to Pragmatic personality
    
    ## Testing
    - [x] Updated unit tests
  • fix: ensure status indicator present earlier in exec path (#10700)
    ensure status indicator present in all classifications of exec tool.
    fixes indicator disappearing after preambles, will look into using
    `phase` to avoid this class of error in a few hours.
    
    commands parsed as unknown faced this issue
    
    tested locally, added test for specific failure flow
  • fix(tui): restore working shimmer after preamble output (#10701)
    ## Problem
    When a turn streamed a preamble line before any tool activity,
    `ChatWidget` hid the status row while committing streamed lines and did
    not restore it until a later event (commonly `ExecCommandBegin`). During
    that idle gap, the UI looked finished even though the turn was still
    active.
    
    ## Mental model
    The bottom status row and transcript stream are separate progress
    affordances:
    - transcript stream shows committed output
    - status row (spinner/shimmer + header) shows liveness of an active turn
    
    While stream output is actively committing, hiding the status row is
    acceptable to avoid redundant visual noise. Once stream controllers go
    idle, an active turn must restore the status row immediately so liveness
    remains visible across preamble-to-tool gaps.
    
    ## Non-goals
    - No changes to streaming chunking policy or pacing.
    - No changes to final completion behavior (status still hides when task
    actually ends).
    - No refactor of status lifecycle ownership between `ChatWidget` and
    `BottomPane`.
    
    ## Tradeoffs
    - We keep the existing behavior of hiding the status row during active
    stream commits.
    - We add explicit restoration on the idle boundary when the task is
    still running.
    - This introduces one extra status update on idle transitions, which is
    small overhead but makes liveness semantics consistent.
    
    ## Architecture
    `run_commit_tick_with_scope` in `chatwidget.rs` now documents and
    enforces a two-phase contract:
    1. For each committed streamed cell, hide status and append transcript
    output.
    2. If controllers are present and all idle, restore status iff task is
    still running, preserving the current header.
    
    This keeps status ownership in `ChatWidget` while relying on
    `BottomPane` helpers:
    - `hide_status_indicator()` during active stream commits
    - `ensure_status_indicator()` +
    `set_status_header(current_status_header)` at stream-idle boundary
    
    Documentation pass additions:
    - Clarified the function-level contract and lifecycle intent in
    `run_commit_tick_with_scope`.
    - Added an explicit regression snapshot test comment describing the
    failing sequence.
    
    ## Observability
    Signal that the fix is present:
    - In the preamble-idle state, rendered output still includes `• Working
    (… esc to interrupt)`.
    - New snapshot:
    `codex_tui__chatwidget__tests__preamble_keeps_working_status.snap`.
    
    Debug path for future regressions:
    - Start at `run_commit_tick_with_scope` for hide/restore transitions.
    - Verify `bottom_pane.is_task_running()` at idle transition.
    - Confirm `current_status_header` continuity when status is recreated.
    - Use the new snapshot and targeted test sequence to reproduce
    deterministic preamble-idle behavior.
    
    ## Tests
    - Updated regression assertion:
    - `streaming_final_answer_keeps_task_running_state` now expects status
    widget to remain present while turn is running.
    - Renamed/updated behavioral regression:
      - `preamble_keeps_status_indicator_visible_until_exec_begin`.
    - Added snapshot regression coverage:
      - `preamble_keeps_working_status_snapshot`.
    - Snapshot file:
    `tui/src/chatwidget/snapshots/codex_tui__chatwidget__tests__preamble_keeps_working_status.snap`.
    
    Commands run:
    - `just fmt`
    - `cargo test -p codex-tui
    preamble_keeps_status_indicator_visible_until_exec_begin`
    - `cargo test -p codex-tui preamble_keeps_working_status_snapshot`
    
    ## Risks / Inconsistencies
    - Status visibility policy is still split across multiple event paths
    (`commit tick`, `turn complete`, `exec begin`), so future regressions
    can reintroduce ordering gaps.
    - Restoration depends on `is_task_running()` correctness; if task
    lifecycle flags drift, status behavior will drift too.
    - Snapshot proves rendered state, not animation cadence; cadence still
    relies on frame scheduling behavior elsewhere.
  • chore(core) personality migration tests (#10650)
    ## Summary
    Adds additional tests for personality edge cases
    
    ## Testing
    - [x] These are tests
  • Cloud Requirements: increase timeout and retries (#10631)
    Add retries and an increased-length timeout for loading Cloud
    Requirements.
    
    Co-authored-by: alexsong-oai <alexsong@openai.com>
  • feat(core): add configurable log_dir (#10678)
    Adds a top-level `log_dir` config key (defaults to `$CODEX_HOME/log`) so
    one-off runs can redirect `codex-tui.log` via `-c`, e.g.:
    
      codex -c log_dir=./.codex-log
    
    Also resolves relative paths in CLI `-c/--config` overrides for
    `AbsolutePathBuf` values against the effective cwd (when available).
    
    Tests:
    - cargo test -p codex-core
  • Session-level model client (#10664)
    Make ModelClient a session-scoped object.
    Move state that is session level onto the client, and make state that is
    per-turn explicit on corresponding methods.
    Stop taking a huge Config object, instead only pass in values that are
    actually needed.
    
    ---------
    
    Co-authored-by: Josh McKinney <joshka@openai.com>
  • chore(app-server): document experimental API opt-in (#10667)
    Add a section on how to opt in to the experimental API.
  • feat(app-server, core): allow text + image content items for dynamic tool outputs (#10567)
    Took over the work that @aaronl-openai started here:
    https://github.com/openai/codex/pull/10397
    
    Now that app-server clients are able to set up custom tools (called
    `dynamic_tools` in app-server), we should expose a way for clients to
    pass in not just text, but also image outputs. This is something the
    Responses API already supports for function call outputs, where you can
    pass in either a string or an array of content outputs (text, image,
    file):
    https://platform.openai.com/docs/api-reference/responses/create#responses_create-input-input_item_list-item-function_tool_call_output-output-array-input_image
    
    So let's just plumb it through in Codex (with the caveat that we only
    support text and image for now). This is implemented end-to-end across
    app-server v2 protocol types and core tool handling.
    
    ## Breaking API change
    NOTE: This introduces a breaking change with dynamic tools, but I think
    it's ok since this concept was only recently introduced
    (https://github.com/openai/codex/pull/9539) and it's better to get the
    API contract correct. I don't think there are any real consumers of this
    yet (not even the Codex App).
    
    Old shape:
    `{ "output": "dynamic-ok", "success": true }`
    
    New shape:
    ```
    {
        "contentItems": [
          { "type": "inputText", "text": "dynamic-ok" },
          { "type": "inputImage", "imageUrl": "data:image/png;base64,AAA" }
        ]
      "success": true
    }
    ```
  • add none personality option (#10688)
    - add none personality enum value and empty placeholder behavior\n- add
    docs/schema updates and e2e coverage
  • Added support for live updates to skills (#10478)
    Add a centralized FileWatcher in codex-core (using notify) that watches
    skill roots from the config layer stack (recursive)
    
    Send `SkillsChanged` events when relevant file system changes are
    detected
    
    On `SkillsChanged`:
    * Invalidate the skills cache immediately in ThreadManager
    * Emit EventMsg::SkillsUpdateAvailable to active sessions
    ~~* Broadcast a new app-server notification:
    SkillsListUpdatedNotification~~
    
    This change does not inject new items into the event stream. That means
    the agent will not know about new skills, so it won't be able to
    implicitly invoke new skills. It also won't know about changes to
    existing skills, so if it has already read the contents of a modified
    skill, it will not honor the new behavior.
    
    This change also does not detect modifications to AGENTS.md.
    
    I plan to address these limitations in a follow-on PR modeled after
    #9985. Injection of new skills and AGENTS was deemed to risky, hence the
    need to split the feature into two stages. The changes in this PR were
    designed to easily accommodate the second stage once we have some other
    foundational changes in place.
    
    Testing: In addition to automated tests, I did manual testing to confirm
    that newly-created skills, deleted skills, and renamed skills are
    reflected in the TUI skill picker menu. Also confirmed that
    modifications to behaviors for explicitly-invoked skills are honored.
    
    ---------
    
    Co-authored-by: Xin Lin <xl@openai.com>
  • Fix test_shell_command_interruption flake (#10649)
    ## Human summary
    Sandboxing (specifically `LandlockRestrict`) is means that e.g. `sleep
    10` fails immediately. Therefore it cannot be interrupted.
    
    In suite::interrupt::test_shell_command_interruption, sleep 10 is issued
    at 17:28:16.554 (ToolCall: shell_command {"command":"sleep 10"...}),
    then fails at 17:28:16.589 with duration_ms=34, success=false,
    exit_code=101, and
        Sandbox(LandlockRestrict).
    
    ## Codex summary
    - set `sandbox_mode = "danger-full-access"` in `interrupt` and
    `v2/turn_interrupt` integration tests
    - set `sandbox: Some(SandboxMode::DangerFullAccess)` in
    `test_codex_jsonrpc_conversation_flow`
    - set `sandbox_policy: Some(SandboxPolicy::DangerFullAccess)` in
    `command_execution_notifications_include_process_id`
    
    ## Why
    On some Linux CI environments, command execution fails immediately with
    `LandlockRestrict` when sandboxed. These tests are intended to validate
    JSON-RPC/task lifecycle behavior (interrupt semantics, command
    notification shape/process id, request flow), but early sandbox startup
    failure changes turn flow and can trigger extra follow-up requests,
    causing flakes.
    
    This change removes environment-specific sandbox startup dependency from
    these tests while preserving their primary intent.
    
    ## Testing
    - not run in this environment (per request)
  • [apps] Cache MCP actions from apps. (#10662)
    MCP actions take a long time to load for users with lots of apps
    installed. Adding a cache for these actions with 1hr expiration, given
    that they are almost always aren't going to change unless people install
    another app, which means they also need to restart codex to pick it up.
  • Fix jitter in TUI apps/connectors picker (#10593)
    This PR fixes jitter in the TUI apps menu by making the description
    column stable during rendering and height measurement.
    Added a `stable_desc_col` option to
    `SelectionViewParams`/`ListSelectionView`, introduced stable variants of
    the shared row render/measure helpers in `selection_popup_common`, and
    enabled the stable mode for the apps/connectors picker in `chatwidget`.
    With these changes, only the apps/connectors picker uses this new
    option, though it could be used elsewhere in the future.
    
    Why: previously, the description column was computed from only currently
    visible rows, so as you scrolled or filtered, the column could shift and
    cause wrapping/height changes that looked jumpy. Computing it from all
    rows in this popup keeps alignment and layout consistent as users scroll
    through avaialble apps.
    
    
    
    **Before:**
    
    https://github.com/user-attachments/assets/3856cb72-5465-4b90-a993-65a2ffb09113
    
    
    
    
    
    **After:**
    
    https://github.com/user-attachments/assets/37b9d626-0b21-4c0f-8bb8-244c9ef971ff
  • feat: add phase 1 mem db (#10634)
    - Schema: thread_id (PK, FK to threads.id with cascade delete),
    trace_summary, memory_summary, updated_at.
    - Migration: creates the table and an index on (updated_at DESC,
    thread_id DESC) for efficient recent-first reads.
      - Runtime API (DB-only):
          - `get_thread_memory(thread_id)`: fetch one memory row.
    - `upsert_thread_memory(thread_id, trace_summary, memory_summary)`:
    insert/update by thread id and always advance updated_at.
    - `get_last_n_thread_memories_for_cwd(cwd, n)`: join thread_memory with
    threads and return newest n rows for an exact cwd match.
    - Model layer: introduced ThreadMemory and row conversion types to keep
    query decoding typed and consistent with existing state models.
  • Persist pending input user events (#10656)
    - Persist user-message events for mid-turn injected input by emitting
    user message turn items when pending input is recorded.
  • feat(linux-sandbox): add bwrap support (#9938)
    ## Summary
    This PR introduces a gated Bubblewrap (bwrap) Linux sandbox path. The
    curent Linux sandbox path relies on in-process restrictions (including
    Landlock). Bubblewrap gives us a more uniform filesystem isolation
    model, especially explicit writable roots with the option to make some
    directories read-only and granular network controls.
    
    This is behind a feature flag so we can validate behavior safely before
    making it the default.
    
    - Added temporary rollout flag:
      - `features.use_linux_sandbox_bwrap`
    - Preserved existing default path when the flag is off.
    - In Bubblewrap mode:
    - Added internal retry without /proc when /proc mount is not permitted
    by the host/container.
  • Cloud Requirements: take precedence over MDM (#10633)
    Cloud Requirements should be applied before MDM requirements.
  • Add option to approve and remember MCP/Apps tool usage (#10584)
    This PR adds a new approval option for app/MCP tool calls: “Allow and
    remember” (session-scoped).
    When selected, Codex stores a temporary approval and auto-approves
    matching future calls for the rest of the session.
    
    Added a session-scoped approval key (`server`, `connector_id`,
    `tool_name`) and persisted it in `tool_approvals` as
    `ApprovedForSession`.
    On subsequent matching calls, approval is skipped and treated as
    accepted.
    - Updated the approval question options to conditionally include:
    - Accept
    - Allow and remember (conditional)
    - Decline
    - Cancel
    
    The new “Allow and remember” option is only shown when all of these are
    true:
    
    1. The call is routed through the Codex Apps MCP server (codex_apps).
    2. The tool requires approval based on annotations:
    - read_only_hint == false, and
    - destructive_hint == true or open_world_hint == true.
    3. The tool includes a connector_id in metadata (used to build the
    remembered approval key).
    
    If no `connector_id` is present, the prompt still appears (when approval
    is required), but only with the existing choices (Accept / Decline /
    Cancel). Approval prompting in this path has an explicit early return
    unless server == `codex_apps`.
  • Stop client from being state carrier (#10595)
    I'd like to make client session wide. This requires shedding all random
    state it has to carry.
  • feat: land unified_exec (#10641)
    Land `unified_exec` for all non-windows OS
  • Update tests to stop using sse_completed fixture (#10638)
    Summary:
    - replace the `sse_completed` fixture and related JSON template with
    direct `responses::ev_completed` payload builders
    - cascade the new SSE helpers through all affected core tests for
    consistency and clarity
    - remove legacy fixtures that were no longer needed once the helpers are
    in place
    
    Testing:
    - Not run (not requested)
  • Migrate state DB path helpers to versioned filename (#10623)
    Summary
    - add versioned state sqlite filename helpers and re-export them from
    the state crate
    - remove legacy state files when initializing the runtime and update
    consumers/tests to use the new helpers
    - tweak logs client description and database resolution to match the new
    path
  • Add a codex.rate_limits event for websockets (#10324)
    When communicating over websockets, we can't rely on headers to deliver
    rate limit information. This PR adds a `codex.rate_limits` event that
    the server can pass to the client to inform them about rate limit usage.
    The client parses this data the same way we parse rate limit headers in
    HTTP mode.
    
    This PR also wires up the etag and reasoning headers for websockets
  • chore: simplify user message detection (#10611)
    We don't check anymore the response item with `user` role as they may be
    instructions etc
  • Requirements: add source to constrained requirement values (#10568)
    If we want to build `/debug-config`, we'll need to know the requirements
    sources that supplied the values.
    
    This PR adds those sources such that we can render them in the UI.