Commit Graph

21 Commits

  • feat(app-server): list descendant threads by ancestor (#29591)
    ## Why
    
    `thread/list` can filter direct children with `parentThreadId`, but
    clients cannot request an entire spawned subtree. Discovering every
    descendant requires repeated client-side requests and gives up the
    database's existing filtering and pagination path.
    
    ## What changed
    
    Experimental clients can use `ancestorThreadId` to return strict
    descendants at any depth while `parentThreadId` retains its direct-child
    meaning. The filters are mutually exclusive, the ancestor is excluded,
    and every result preserves its immediate `parentThreadId` so callers can
    reconstruct the tree.
    
    ## How it works
    
    - **Explicit relationship:** Internal list parameters distinguish direct
    children from transitive descendants without changing the meaning of
    `parentThreadId`.
    - **Existing graph:** Persisted parent-child spawn edges remain the
    source of truth, so descendant lookup needs no schema migration or
    ancestry cache.
    - **Indexed traversal:** A recursive SQLite query starts from the
    parent-edge index, walks each generation, and applies thread filters,
    sorting, and cursor pagination in the same database request.
    - **Reconstructable results:** The response stays flat and normally
    ordered while carrying each descendant's immediate parent.
    
    ## Verification
    
    Ran 550 tests across the protocol, state, rollout, and thread-store
    crates, then reran the four focused state, store, and app-server
    descendant-listing tests after the final diff reduction. Scoped Clippy
    and formatting checks passed. Stable and experimental schema generation
    was checked; the stable fixtures remain unchanged while the experimental
    schema includes the new field.
  • feat: render typed envelopes for multi-agent v2 messages (#28368)
    ## Why
    
    Multi-agent v2 messages need a consistent, model-visible envelope that
    identifies what kind of interaction occurred, who sent it, and which
    agent it targets. Previously, encrypted deliveries exposed only
    `encrypted_content`, while child completion used the legacy
    `<subagent_notification>` shape. That meant the client could not
    consistently present `NEW_TASK`, `MESSAGE`, and `FINAL_ANSWER` using the
    same format.
    
    This change adds the routing envelope as plaintext while keeping task
    and message payloads encrypted. No new Responses API field is required:
    an encrypted delivery is represented as an `input_text` header
    immediately followed by its existing `encrypted_content` item.
    
    Every envelope now follows this shape:
    
    ```text
    Message Type: <NEW_TASK | MESSAGE | FINAL_ANSWER>
    Task name: <recipient agent path>
    Sender: <author agent path>
    Payload:
    <message payload>
    ```
    
    ## Message types
    
    ### `NEW_TASK`
    
    `NEW_TASK` is used when the recipient should begin a new turn, including
    an initial `spawn_agent` task and a later `followup_task`.
    
    For a root agent spawning `/root/worker`, the request contains a
    plaintext envelope followed by the encrypted task:
    
    ```json
    {
      "type": "agent_message",
      "author": "/root",
      "recipient": "/root/worker",
      "content": [
        {
          "type": "input_text",
          "text": "Message Type: NEW_TASK\nTask name: /root/worker\nSender: /root\nPayload:\n"
        },
        {
          "type": "encrypted_content",
          "encrypted_content": "<encrypted task payload>"
        }
      ]
    }
    ```
    
    Conceptually, the model receives:
    
    ```text
    Message Type: NEW_TASK
    Task name: /root/worker
    Sender: /root
    Payload:
    Review the authentication changes and report any regressions.
    ```
    
    ### `MESSAGE`
    
    `MESSAGE` is used for a queued `send_message` delivery. It communicates
    with an existing agent without starting a new turn.
    
    For `/root/worker` reporting progress to the root agent, the request
    contains:
    
    ```json
    {
      "type": "agent_message",
      "author": "/root/worker",
      "recipient": "/root",
      "content": [
        {
          "type": "input_text",
          "text": "Message Type: MESSAGE\nTask name: /root\nSender: /root/worker\nPayload:\n"
        },
        {
          "type": "encrypted_content",
          "encrypted_content": "<encrypted message payload>"
        }
      ]
    }
    ```
    
    Conceptually, the model receives:
    
    ```text
    Message Type: MESSAGE
    Task name: /root
    Sender: /root/worker
    Payload:
    The protocol tests pass; I am checking the resume path now.
    ```
    
    ### `FINAL_ANSWER`
    
    `FINAL_ANSWER` is emitted when a child agent reaches a terminal state
    and reports its result to its parent. Completion payloads are already
    available locally, so the complete envelope is represented as plaintext
    rather than as a plaintext header plus encrypted content.
    
    For `/root/worker` completing work for the root agent, the request
    contains:
    
    ```json
    {
      "type": "agent_message",
      "author": "/root/worker",
      "recipient": "/root",
      "content": [
        {
          "type": "input_text",
          "text": "Message Type: FINAL_ANSWER\nTask name: /root\nSender: /root/worker\nPayload:\nNo regressions found."
        }
      ]
    }
    ```
    
    The model-visible form is:
    
    ```text
    Message Type: FINAL_ANSWER
    Task name: /root
    Sender: /root/worker
    Payload:
    No regressions found.
    ```
    
    Errored, shut down, and missing agents also use `FINAL_ANSWER`, with a
    terminal-status description in the payload.
    
    ## What changed
    
    - Render `NEW_TASK` or `MESSAGE` in
    `InterAgentCommunication::to_model_input_item`, based on whether the
    encrypted delivery starts a turn.
    - Replace the multi-agent v2 `<subagent_notification>` completion
    payload with a model-visible `FINAL_ANSWER` envelope.
    - Document `Task name`, `Sender`, and `Payload` consistently in the
    multi-agent developer instructions.
    - Prevent local-only history projections from treating an encrypted
    message's plaintext header as the complete assistant message.
    - Preserve rollout-trace interaction edges when an agent message
    contains both plaintext and encrypted content.
    
    Legacy multi-agent behavior remains unchanged.
    
    ## Verification
    
    - `just test -p codex-protocol`
    - `just test -p codex-rollout-trace`
    - `just test -p codex-web-search-extension`
    - `just test -p codex-core
    encrypted_multi_agent_v2_spawn_sends_agent_message_to_child`
    - `just test -p codex-core
    plaintext_multi_agent_v2_completion_sends_agent_message`
    - `just test -p codex-core
    multi_agent_v2_followup_task_completion_notifies_parent_on_every_turn`
    - `just test -p codex-core
    multi_agent_v2_completion_queues_message_for_direct_parent`
  • feat(app-server): filter threads by parent (#26662)
    ## Why
    
    Clients that display or coordinate spawned subagents need an
    authoritative snapshot of a thread's immediate spawned children when
    they connect to app-server or recover after missing live events.
    `thread/list` cannot query by parent, so clients must otherwise scan
    unrelated threads or reconstruct relationships from rollout history and
    transient events.
    
    The direct spawn relationship already exists in persisted
    `thread_spawn_edges` state. Review and Guardian threads do not
    participate in that lifecycle and are intentionally outside this
    filter's scope.
    
    ## What changed
    
    This adds an experimental `parentThreadId` filter to `thread/list`.
    Parent-filtered requests return direct spawned children from persisted
    state while preserving the existing response shape, explicit filters,
    sorting, and timestamp-only cursor behavior. The lookup does not read
    rollout transcripts or recursively return descendants.
    
    Supersedes #25112 with the narrower `thread/list` filter approach.
    
    ## How it works
    
    1. An experimental client passes a valid thread ID as `parentThreadId`.
    2. App-server routes the list through the existing thread-store and
    state-database boundaries.
    3. SQLite selects threads whose IDs have a direct persisted spawn edge
    from that parent.
    4. Omitted provider and source filters include all values; explicit
    filters keep ordinary `thread/list` semantics.
    5. Grandchildren, Review threads, and Guardian threads are excluded.
    
    ## Verification
    
    State (144 tests), rollout (69 tests), and focused app-server
    thread-list (31 tests) suites passed. Scoped Clippy checks and
    repository formatting also passed. Coverage includes direct spawned
    children, omitted grandchildren, pagination, malformed IDs, mixed source
    kinds, explicit filters, and operation without rollout files.
  • Support plaintext agent messages (#27830)
    ## Why
    
    Multi-agent v2 `send_message` deliveries already reach the receiving
    model as typed `agent_message` items with encrypted content.
    Child-completion notifications are generated by Codex itself, so their
    content is plaintext and previously fell back to a serialized JSON
    envelope inside an assistant message.
    
    With plaintext `input_text` supported for `agent_message`, both delivery
    paths can use the same model-visible type while preserving explicit
    author and recipient metadata.
    
    ## What changed
    
    - add plaintext `input_text` support to `AgentMessageInputContent` and
    regenerate the affected app-server schemas
    - preserve `InterAgentCommunication` as structured mailbox input instead
    of converting it to assistant text
    - record delivered communications as typed `agent_message` history items
    - persist a dedicated rollout item so local delivery metadata such as
    `trigger_turn` remains available without leaking into the Responses
    request
    - reconstruct typed agent messages on resume and preserve fork-turn
    truncation behavior
    - remove request-time assistant-content parsing
    - preserve plaintext and encrypted inter-agent deliveries in stage-one
    memory inputs
    - normalize and link plaintext and encrypted agent messages in rollout
    traces without treating inbound messages as child results
    - cover the real MultiAgent V2 child-completion path end to end with
    deterministic mailbox synchronization
    
    ## Verification
    
    - `just test -p codex-core
    plaintext_multi_agent_v2_completion_sends_agent_message`
    - `just test -p codex-core input_queue_drains_mailbox_in_delivery_order
    record_initial_history_reconstructs_typed_inter_agent_message
    fork_turn_positions_use_inter_agent_delivery_metadata`
    - `just test -p codex-memories-write
    serializes_inter_agent_communications_for_memory`
    - `just test -p codex-rollout-trace
    agent_messages_preserve_routing_and_content
    sub_agent_started_activity_creates_spawn_edge`
    - `just test -p codex-rollout-trace
    agent_result_edge_falls_back_to_child_thread_without_result_message`
    - `just test -p codex-protocol -p codex-rollout -p
    codex-app-server-protocol`
  • [codex] Route live thread writes through ThreadStore (#18882)
    Begin migrating the thread write codepaths to ThreadStore.
    
    This starts using ThreadStore inside of core session code, not only in
    the app server code.
    
    Rework the interfaces around thread recording/persistence. We're left
    with the following:
    
    * `ThreadManager`: owns the process-level registry of loaded threads and
    handles cross-thread orchestration: start, resume, fork, lookup, remove,
    and route ops to running CodexThreads.
    * `CodexThread`: represents one loaded/running thread from the outside.
    It is the handle app-server and callers use to submit ops, inspect
    session metadata, and shut the thread down.
    * `LiveThread`: session-owned persistence lifecycle handle for one
    active thread. Core session code uses it to append rollout items,
    materialize lazy persistence, flush, shutdown, discard init-failed
    writers, and load that thread’s persisted history.
    * `ThreadStore`: storage backend abstraction. It answers “how are
    threads persisted, read, listed, updated, archived?” Local and remote
    implementations live behind this trait.
    * `LocalThreadStore`: local ThreadStore implementation. It owns the
    file/sqlite-specific details and keeps RolloutRecorder as a local
    implementation detail.
    
    This is a few too many Thread abstractions for my liking, but they do
    all represent different concepts / needs / layers.
    
    Migration note: in places where the core code explicitly requires a
    path, rather than a thread ID, throw an error if we're running with a
    remote store.
    
    Cover the new local live-writer lifecycle with focused tests and
    preserve app-server thread-start behavior, including ephemeral pathless
    sessions.
  • Support multiple cwd filters for thread list (#18502)
    ## Summary
    
    - Teach app-server `thread/list` to accept either a single `cwd` or an
    array of cwd filters, returning threads whose recorded session cwd
    matches any requested path
    - Add `useStateDbOnly` as an explicit opt-in fast path for callers that
    want to answer `thread/list` from SQLite without scanning JSONL rollout
    files
    - Preserve backwards compatibility: by default, `thread/list` still
    scans JSONL rollouts and repairs SQLite state
    - Wire the new cwd array and SQLite-only options through app-server,
    local/remote thread-store, rollout listing, generated TypeScript/schema
    fixtures, proto output, and docs
    
    ## Test Plan
    
    - `cargo test -p codex-app-server-protocol`
    - `cargo test -p codex-rollout`
    - `cargo test -p codex-thread-store`
    - `cargo test -p codex-app-server thread_list`
    - `just fmt`
    - `just fix -p codex-app-server-protocol -p codex-rollout -p
    codex-thread-store -p codex-app-server`
    - `cargo build -p codex-cli --bin codex`
  • Add sorting/backwardsCursor to thread/list and new thread/turns/list api (#17305)
    To improve performance of UI loads from the app, add two main
    improvements:
    1. The `thread/list` api now gets a `sortDirection` request field and a
    `backwardsCursor` to the response, which lets you paginate forwards and
    backwards from a window. This lets you fetch the first few items to
    display immediately while you paginate to fill in history, then can
    paginate "backwards" on future loads to catch up with any changes since
    the last UI load without a full reload of the entire data set.
    2. Added a new `thread/turns/list` api which also has sortDirection and
    backwardsCursor for the same behavior as `thread/list`, allowing you the
    same small-fetch for immediate display followed by background fill-in
    and resync catchup.
  • Move codex module under session (#18249)
    ## Summary
    - rename the core codex module root to session/mod.rs without using
    #[path]
    - move the codex module directory and tests under core/src/session
    - remove session/mod.rs reexports so call sites use explicit child
    module paths
    
    ## Testing
    - cargo test -p codex-core --lib
    - cargo check -p codex-core --tests
    - just fmt
    - just fix -p codex-core
    - git diff --check
  • Refactor config loading to use filesystem abstraction (#18209)
    Initial pass propagating FileSystem through config loading.
  • [codex] Make realtime startup context truncation deterministic (#18172)
    ## Summary
    
    - remove the final whole-blob truncation pass from realtime
    startup-context assembly
    - enforce fixed per-section budgets, including each section heading
    - keep the existing per-section caps and raise the overall realtime
    startup-context budget to `5300`, matching the sum of those section
    budgets
    - add focused tests for the new wrapping and section-budget behavior
    
    ## Why
    
    The previous flow truncated each section and then middle-truncated the
    final combined startup-context blob again. Small input changes could
    shift that combined cut point, which made retained context unstable and
    caused nondeterministic tests.
    
    ## Impact
    
    Startup context now preserves section boundaries and ordering
    deterministically. Each section is still budgeted independently, but the
    final assembled blob is no longer truncated again as a single opaque
    string. To match that design, the overall startup-context token budget
    is updated to the sum of the existing section budgets rather than
    lowering the section caps.
    
    ## Validation
    
    - `cargo +1.93.0 test -p codex-core realtime_context`
    - `cargo +1.93.0 test -p codex-core --test all
    suite::realtime_conversation::conversation_start_injects_startup_context_from_thread_history
    -- --exact`
    - `cargo +1.93.0 test -p codex-core --test all
    suite::realtime_conversation::conversation_startup_context_current_thread_selects_many_turns_by_budget
    -- --exact`
    - `cargo +1.93.0 test -p codex-core --test all
    suite::realtime_conversation::conversation_startup_context_falls_back_to_workspace_map
    -- --exact`
    - `cargo +1.93.0 test -p codex-core --test all
    suite::realtime_conversation::conversation_startup_context_is_truncated_and_sent_once_per_start
    -- --exact`
  • Async config loading (#18022)
    Parts of config will come from executor. Prepare for that by making
    config loading methods async.
  • Clarify realtime v2 context and handoff messages (#17896)
    ## Summary
    - wrap realtime startup context in
    `<startup_context>...</startup_context>` tags
    - prefix V2 mirrored user text and relayed backend text with `[USER]` /
    `[BACKEND]`
    - remove the V2 progress suffix and replace the final V2 handoff output
    with a short completion acknowledgement while preserving the existing V1
    wrapper
    
    ## Testing
    - cargo test -p codex-api
    realtime_v2_session_update_includes_background_agent_tool_and_handoff_output_item
    -- --exact
    - cargo test -p codex-app-server webrtc_v2_background_agent_
    - cargo test -p codex-app-server webrtc_v2_text_input_is_
    - cargo test -p codex-core conversation_user_text_turn_is_
  • [codex] Add local thread store listing (#17824)
    Builds on top of #17659 
    
    Move the filesystem + sqlite thread listing-related operations inside of
    a local ThreadStore implementation and call ThreadStore from the places
    that used to perform these filesystem/sqlite operations.
    
    This is the first of a series of PRs that will implement the rest of the
    local ThreadStore.
    
    Testing:
    - added unit tests for the thread store implementation
    - adjusted some unit tests in the realtime + personality packages whose
    callsites changed. Specifically I'm trying to hide ThreadMetadata inside
    of the local implementation and make ThreadMetadata a sqlite
    implementation detail concern rather than a public interface, preferring
    the more generate StoredThread interface instead
    - added a corner case test for the personality migration package that
    wasn't covered by the existing test suite
    - adjust the behavior of searched thread listing to run the existing
    local rollout repair/backfill pass _before_ querying SQLite results, so
    callers using ThreadStore::list_threads do not miss matches after a
    partial metadata warm-up
  • Cap realtime mirrored user turns (#17685)
    Cap mirrored user text sent to realtime with the existing 300-token turn
    budget while preserving the full model turn.
    
    Adds integration coverage for capped realtime mirror payloads.
    
    ---------
    
    Co-authored-by: Codex <noreply@openai.com>
  • Budget realtime current thread context (#17519)
    Select Current Thread startup context by budget from newest turns, cap
    each rendered turn at 300 approximate tokens, and add formatter plus
    integration snapshot coverage.
  • Move string truncation helpers into codex-utils-string (#15572)
    - move the shared byte-based middle truncation logic from `core` into
    `codex-utils-string`
    - keep token-specific truncation in `codex-core` so rollout can reuse
    the shared helper in the next stacked PR
    
    ---------
    
    Co-authored-by: Codex <noreply@openai.com>
  • Move git utilities into a dedicated crate (#15564)
    - create `codex-git-utils` and move the shared git helpers into it with
    file moves preserved for diff readability
    - move the `GitInfo` helpers out of `core` so stacked rollout work can
    depend on the shared crate without carrying its own git info module
    
    ---------
    
    Co-authored-by: Ahmed Ibrahim <219906144+aibrahim-oai@users.noreply.github.com>
    Co-authored-by: Codex <noreply@openai.com>
  • [stack 3/4] Add current thread context to realtime startup (#14829)
    ## Stack Position
    3/4. Top-of-stack sibling built on #14830.
    
    ## Base
    - #14830
    
    ## Sibling
    - #14827
    
    ## Scope
    - Extend the realtime startup context with a bounded summary of the
    latest thread turns for continuity.
    
    ---------
    
    Co-authored-by: Codex <noreply@openai.com>
  • Apply argument comment lint across codex-rs (#14652)
    ## Why
    
    Once the repo-local lint exists, `codex-rs` needs to follow the
    checked-in convention and CI needs to keep it from drifting. This commit
    applies the fallback `/*param*/` style consistently across existing
    positional literal call sites without changing those APIs.
    
    The longer-term preference is still to avoid APIs that require comments
    by choosing clearer parameter types and call shapes. This PR is
    intentionally the mechanical follow-through for the places where the
    existing signatures stay in place.
    
    After rebasing onto newer `main`, the rollout also had to cover newly
    introduced `tui_app_server` call sites. That made it clear the first cut
    of the CI job was too expensive for the common path: it was spending
    almost as much time installing `cargo-dylint` and re-testing the lint
    crate as a representative test job spends running product tests. The CI
    update keeps the full workspace enforcement but trims that extra
    overhead from ordinary `codex-rs` PRs.
    
    ## What changed
    
    - keep a dedicated `argument_comment_lint` job in `rust-ci`
    - mechanically annotate remaining opaque positional literals across
    `codex-rs` with exact `/*param*/` comments, including the rebased
    `tui_app_server` call sites that now fall under the lint
    - keep the checked-in style aligned with the lint policy by using
    `/*param*/` and leaving string and char literals uncommented
    - cache `cargo-dylint`, `dylint-link`, and the relevant Cargo
    registry/git metadata in the lint job
    - split changed-path detection so the lint crate's own `cargo test` step
    runs only when `tools/argument-comment-lint/*` or `rust-ci.yml` changes
    - continue to run the repo wrapper over the `codex-rs` workspace, so
    product-code enforcement is unchanged
    
    Most of the code changes in this commit are intentionally mechanical
    comment rewrites or insertions driven by the lint itself.
    
    ## Verification
    
    - `./tools/argument-comment-lint/run.sh --workspace`
    - `cargo test -p codex-tui-app-server -p codex-tui`
    - parsed `.github/workflows/rust-ci.yml` locally with PyYAML
    
    ---
    
    * -> #14652
    * #14651
  • fix: move inline codex-rs/core unit tests into sibling files (#14444)
    ## Why
    PR #13783 moved the `codex.rs` unit tests into `codex_tests.rs`. This
    applies the same extraction pattern across the rest of `codex-rs/core`
    so the production modules stay focused on runtime code instead of large
    inline test blocks.
    
    Keeping the tests in sibling files also makes follow-up edits easier to
    review because product changes no longer have to share a file with
    hundreds or thousands of lines of test scaffolding.
    
    ## What changed
    - replaced each inline `mod tests { ... }` in `codex-rs/core/src/**`
    with a path-based module declaration
    - moved each extracted unit test module into a sibling `*_tests.rs`
    file, using `mod_tests.rs` for `mod.rs` modules
    - preserved the existing `cfg(...)` guards and module-local structure so
    the refactor remains structural rather than behavioral
    
    ## Testing
    - `cargo test -p codex-core --lib` (`1653 passed; 0 failed; 5 ignored`)
    - `just fix -p codex-core`
    - `cargo fmt --check`
    - `cargo shear`
  • Refine realtime startup context formatting (#13560)
    ## Summary
    - group recent work by git repo when available, otherwise by directory
    - render recent work as bounded user asks with per-thread cwd context
    - exclude hidden files and directories from workspace trees