Commit Graph

40 Commits

  • Uprev Rust toolchain pins to 1.95.0 (#24684)
    ## Summary
    - Bump the workspace Rust toolchain from `1.93.0` to `1.95.0` across
    Cargo, Bazel, CI, release workflows, devcontainers, and the Codex
    environment config.
    - Refresh `MODULE.bazel.lock` so the Bazel Rust toolchain artifacts
    match the new version.
    - Leave purpose-specific toolchains unchanged, including the
    `argument-comment-lint` nightly and the upstream `rusty_v8` `1.91.0`
    build pin.
    - Includes fixes for new lints from `just fix` and a few codex-authored
    fixes for lints without a suggestion.
  • Move memory state to a dedicated SQLite DB (#24591)
    ## Summary
    
    Generated memory rows and their stage-one/stage-two job state currently
    live in `state_5.sqlite` alongside thread metadata. That makes memory
    cleanup and regeneration share the main state schema even though those
    rows are memory-pipeline data and can be rebuilt independently from the
    durable thread records.
    
    This PR moves the memory-owned tables into a dedicated
    `memories_1.sqlite` runtime database while keeping thread metadata in
    `state_5.sqlite`.
    
    ## Changes
    
    - Adds a separate memories DB runtime, migrator, path helpers, telemetry
    kind, and Bazel compile data for `state/memory_migrations`.
    - Introduces `MemoryStore` behind `StateRuntime::memories()` and moves
    memory table/job operations onto that store.
    - Drops the old memory tables from the state DB and recreates their
    schema in `state/memory_migrations/0001_memories.sql`.
    - Updates memory startup, citation usage tracking, rollout pollution
    handling, `debug clear-memories`, and app-server `memory/reset` to
    operate through the memories DB.
    - Preserves cross-DB behavior by hydrating thread metadata from the
    state DB when selecting visible memory outputs and checking stage-one
    staleness.
    
    ## Verification
    
    - Added/updated `codex-state` tests for deleted-thread memory visibility
    and already-polluted phase-two enqueue behavior.
    - Updated `debug clear-memories`, app-server `memory/reset`, and
    memories startup tests to seed and assert memory rows through
    `memories_1.sqlite`.
  • Use goal preview metadata for goal-first threads (#21981)
    Fixes #20792
    
    ## Why
    
    `/goal`-first threads are valid resumable threads, but they can be
    missing from `codex resume` and app recents because discovery depends on
    metadata derived from a normal first user message.
    
    PR #21489 attempted to fix this by using the goal objective as
    `first_user_message`. Review feedback pointed out that
    `first_user_message` does more than provide visible text today: it gates
    listing, supplies preview text, and participates in deciding whether a
    later title should surface as a distinct thread name. Reusing it for the
    goal objective could leave a `/goal`-first thread with
    `first_user_message=<goal>` and `title=<later prompt>`, even though the
    goal should only provide the initial visible preview.
    
    This PR follows that feedback by and keeps the `first_user_message` as
    is but introduces a new `preview` field to separate concerns. The
    `preview` field is populated from the first user message or the goal
    objective. We can extend it in the future to include other sources.
    
    ## What Changed
    
    - Added internal thread `preview` metadata in `codex-state`, including a
    SQLite migration that backfills from `first_user_message` and from
    existing `thread_goals` objectives when needed.
    - Treated `ThreadGoalUpdated` as preview-bearing metadata so goal-first
    threads can be listed and searched without mutating
    `first_user_message`.
    - Updated rollout listing, state queries, thread-store conversion, and
    app-server mapping to use preview metadata while continuing to expose
    the existing public `preview` field.
    - Preserved title/name distinctness behavior around literal
    `first_user_message`, so a later normal prompt after `/goal` does not
    surface as a separate name just because the goal supplied the initial
    preview.
    - Preserved compatibility for older/internal metadata writes by deriving
    preview from `first_user_message` when explicit preview metadata is
    absent.
    
    ## Verification
    
    - Manually verified that a thread that starts with a `/goal <objective>`
    shows up in the resume picker.
  • [codex-analytics] rework thread_source for thread analytics (#20949)
    ## Summary
    - make `thread_source` an explicit optional thread-level field on
    `thread/start`, `thread/fork`, and returned thread payloads
    - persist `thread_source` in rollout/session metadata so resumed live
    threads retain the original value
    - replace the old best-effort `session_source` -> `thread_source`
    mapping with an explicit caller-supplied analytics classification
    
    ## Why
    Before this change, analytics `thread_source` was populated by a
    best-effort mapping from `session_source`. `session_source` describes
    the runtime/client surface, not the actual thread-level origin, so that
    projection was not accurate enough to distinguish cases such as `user`,
    `subagent`, `memory_consolidation`, and future thread origins reliably.
    
    Making `thread_source` explicit keeps one thread-level analytics field
    while letting callers provide the real classification directly instead
    of recovering it indirectly from `session_source`.
    
    ## Impact
    For new analytics events, `thread_source` now reflects the explicit
    thread-level classification supplied by the caller rather than an
    inferred value derived from `session_source`. Existing protocol fields
    remain optional; callers that omit `threadSource` now produce `null`
    instead of a best-effort inferred value.
    
    ## Validation
    - `just write-app-server-schema`
    - `cargo test -p codex-analytics -p codex-core -p
    codex-app-server-protocol --no-run`
    - `cargo test -p codex-app-server-protocol
    generated_ts_optional_nullable_fields_only_in_params`
    - `cargo test -p codex-analytics
    thread_initialized_event_serializes_expected_shape`
    - `cargo test -p codex-core
    resume_stopped_thread_from_rollout_preserves_thread_source`
  • feat: trigger memories from user turns with cooldown (#19970)
    ## Why
    
    Memory startup was tied to thread lifecycle events such as create, load,
    and fork. That can run memory work before a thread receives real user
    input, and it makes startup cost scale with thread management instead of
    actual turns. Moving the trigger to `thread/sendInput` keeps memory
    startup aligned with the first real user turn and lets it use the
    current thread config at turn time.
    
    The idea is to prevent ghost cost due to pre-warm triggered by the app
    
    Turn-based startup can also make global phase-2 consolidation easier to
    request repeatedly, so this adds a success cooldown and tightens the
    default startup scan window.
    
    ## What Changed
    
    - Start `codex_memories_write::start_memories_startup_task` after a
    non-empty `thread/sendInput` turn is submitted, instead of from thread
    create/load/fork paths:
    https://github.com/openai/codex/blob/d4a6885b7829e2fd2ec7a09355e4f75ebe1d1fe3/codex-rs/app-server/src/codex_message_processor.rs#L6477-L6487
    - Expose `CodexThread::config()` so app-server can pass the live config
    into memory startup at turn time.
    - Add a six-hour successful-run cooldown for global phase-2
    consolidation via `SkippedCooldown`:
    https://github.com/openai/codex/blob/d4a6885b7829e2fd2ec7a09355e4f75ebe1d1fe3/codex-rs/state/src/runtime/memories.rs#L963-L966
    - Reduce memory startup defaults to at most 2 rollouts over 10 days:
    https://github.com/openai/codex/blob/d4a6885b7829e2fd2ec7a09355e4f75ebe1d1fe3/codex-rs/config/src/types.rs#L31-L34
    
    ## Verification
    
    Updated the memory runtime coverage around phase-2 reclaim behavior,
    including `phase2_global_lock_respects_success_cooldown`.
    
    ---------
    
    Co-authored-by: Codex <noreply@openai.com>
  • Stabilize memory Phase 2 input ordering (#19967)
    ## Why
    
    Phase 2 still needs to choose the most relevant stage-1 memory outputs
    by usage and recency, but exposing that ranking as the rendered
    `raw_memories.md` order creates unnecessary large diff. Usage-count or
    timestamp changes can reshuffle otherwise unchanged memories, making the
    workspace diff noisy and giving the consolidation prompt a misleading
    recency signal from file position.
    This fix will reduce token consumption
    
    ## What Changed
    
    - Keep the existing top-N Phase 2 selection ranking by `usage_count`,
    `last_usage`, `source_updated_at`, and `thread_id`.
    - Return the selected rows in stable ascending `thread_id` order before
    syncing Phase 2 filesystem inputs.
    - Update the memory README, raw memories header, and consolidation
    prompt so they describe the stable order and tell the prompt to use
    metadata and workspace diffs instead of file order as the recency
    signal.
    - Adjust the memory runtime tests to use deterministic thread IDs and
    assert the stable return order separately from the ranked selection
    semantics.
    
    ## Test Coverage
    
    - Existing memory runtime tests in
    `codex-rs/state/src/runtime/memories.rs` now cover the stable returned
    ordering for Phase 2 inputs.
    
    ---------
    
    Co-authored-by: Codex <noreply@openai.com>
  • Avoid rewriting Phase 2 selection on clean workspace (#19812)
    ## Why
    
    Phase 2 can now claim the global consolidation lock on startup even when
    the git-backed memory workspace is already clean. The clean-workspace
    path still finalized through the normal Phase 2 success path, which
    clears and re-marks `selected_for_phase2` rows. That made no-op startups
    perform avoidable writes to `stage1_outputs`, creating unnecessary DB
    I/O and contention when no memory files changed.
    
    ## What Changed
    
    - Added a preserving-selection Phase 2 finalizer in `codex-state` that
    only marks the global job row as succeeded.
    - Kept the existing `mark_global_phase2_job_succeeded` behavior for real
    consolidation runs, where the selected Phase 2 snapshot must be
    rewritten.
    - Switched the `succeeded_no_workspace_changes` branch in
    `core/src/memories/phase2.rs` to use the preserving-selection finalizer.
    - Added a regression test that installs a SQLite trigger on
    `stage1_outputs` and verifies the clean finalizer performs zero updates
    there.
    
    ## Testing
    
    - `cargo test -p codex-state`
    - `cargo test -p codex-core memories::tests::phase2`
  • Allow Phase 2 memory claims after retry exhaustion (#19809)
    ## Why
    
    The Phase 2 memories job row is only the global lock for the git-backed
    memory workspace. Manual memory edits do not enqueue new Stage 1 work,
    so a Phase 2 row with `retry_remaining = 0` could be skipped before the
    worker ever claimed the lock and generated `phase2_workspace_diff.md`.
    
    That left workspace-only changes unconsolidated after repeated failures,
    even when retry backoff had elapsed and the filesystem had real diffable
    work.
    
    ## What Changed
    
    - Allow `try_claim_global_phase2_job` to claim the Phase 2 lock after
    the retry budget is exhausted, while still respecting active `retry_at`
    backoff and fresh running leases.
    - Treat `SkippedRetryUnavailable` for Phase 2 as backoff-only, and
    update the outcome docs to match.
    - Clamp Phase 2 retry bookkeeping at zero when failed attempts are
    recorded.
    
    ## Verification
    
    - Added
    `phase2_global_lock_can_be_claimed_after_retry_budget_is_exhausted` to
    cover the exhausted-budget lock claim path.
    - Ran `cargo test -p codex-state`.
  • feat: use git-backed workspace diffs for memory consolidation (#18982)
    ## Why
    
    This PR make the `morpheus` agent (memory phase 2) use a git diff to
    start it's consolidation. The workflow is the following:
    1. The agent acquire a lock
    2. If `.codex/memories` does not exist or is not a git root, initialize
    everything (and make a first empty commit)
    3. Update `raw_memories.md` and `rollout_summaries/` as before.
    Basically we select max N phase 1 memories based on a given policy
    4. We use git (`gix`) to get a diff between the current state of
    `.codex/memories` and the last commit.
    5. Dump the diff in `phase2_workspace_diff.md`
    6. Spawn `morpheus` and point it to `phase2_workspace_diff.md`
    7. Wait for `morpheus` to be done
    8. Re-create a new `.git` and make one single commit on it. We do this
    because we don't want to preserve history through `.git` and this is
    cheap anyway
    9. We release the lock
    On top of this, we keep the retry policies etc etc
    
    The goals of this new workflow are:
    * Better support of any memory extensions such as `chronicle`
    * Allow the user to manually edit memories and this will be considered
    by the phase 2 agent
     
    As a follow-up we will need to add support for user's edition while
    `morpheus` is running
    
    ## What Changed
    
    - Added memory workspace helpers that prepare the git baseline, compute
    the diff, write `phase2_workspace_diff.md`, and reset the baseline after
    successful consolidation.
    - Updated Phase 2 to sync current inputs into `raw_memories.md` and
    `rollout_summaries/`, prune old extension resources, skip clean
    workspaces, and run the consolidation subagent only when the workspace
    has changes.
    - Tightened Phase 2 job ownership around long-running consolidation with
    heartbeats and an ownership check before resetting the baseline.
    - Simplified the prompt and state APIs so DB watermarks are bookkeeping,
    while workspace dirtiness decides whether consolidation work exists.
    - Updated the memory pipeline README and tests for workspace diffs,
    extension-resource cleanup, pollution-driven forgetting, selection
    ranking, and baseline persistence.
    
    ## Verification
    
    - Added/updated coverage in `core/src/memories/tests.rs`,
    `core/src/memories/workspace_tests.rs`, `state/src/runtime/memories.rs`,
    and `core/tests/suite/memories.rs`.
    
    ---------
    
    Co-authored-by: Codex <noreply@openai.com>
  • Support multiple cwd filters for thread list (#18502)
    ## Summary
    
    - Teach app-server `thread/list` to accept either a single `cwd` or an
    array of cwd filters, returning threads whose recorded session cwd
    matches any requested path
    - Add `useStateDbOnly` as an explicit opt-in fast path for callers that
    want to answer `thread/list` from SQLite without scanning JSONL rollout
    files
    - Preserve backwards compatibility: by default, `thread/list` still
    scans JSONL rollouts and repairs SQLite state
    - Wire the new cwd array and SQLite-only options through app-server,
    local/remote thread-store, rollout listing, generated TypeScript/schema
    fixtures, proto output, and docs
    
    ## Test Plan
    
    - `cargo test -p codex-app-server-protocol`
    - `cargo test -p codex-rollout`
    - `cargo test -p codex-thread-store`
    - `cargo test -p codex-app-server thread_list`
    - `just fmt`
    - `just fix -p codex-app-server-protocol -p codex-rollout -p
    codex-thread-store -p codex-app-server`
    - `cargo build -p codex-cli --bin codex`
  • Add sorting/backwardsCursor to thread/list and new thread/turns/list api (#17305)
    To improve performance of UI loads from the app, add two main
    improvements:
    1. The `thread/list` api now gets a `sortDirection` request field and a
    `backwardsCursor` to the response, which lets you paginate forwards and
    backwards from a window. This lets you fetch the first few items to
    display immediately while you paginate to fill in history, then can
    paginate "backwards" on future loads to catch up with any changes since
    the last UI load without a full reload of the entire data set.
    2. Added a new `thread/turns/list` api which also has sortDirection and
    backwardsCursor for the same behavior as `thread/list`, allowing you the
    same small-fetch for immediate display followed by background fill-in
    and resync catchup.
  • Moving updated-at timestamps to unique millisecond times (#17489)
    To allow the ability to have guaranteed-unique cursors, we make two
    important updates:
    * Add new updated_at_ms and created_at_ms columns that are in
    millisecond precision
    * Guarantee uniqueness -- if multiple items are inserted at the same
    millisecond, bump the new one by one millisecond until it becomes unique
    
    This lets us use single-number cursors for forwards and backwards paging
    through resultsets and guarantee that the cursor is a fixed point to do
    (timestamp > cursor) and get new items only.
    
    This updated implementation is backwards-compatible since multiple
    appservers can be running and won't handle the previous method well.
  • chore: clean up argument-comment lint and roll out all-target CI on macOS (#16054)
    ## Why
    
    `argument-comment-lint` was green in CI even though the repo still had
    many uncommented literal arguments. The main gap was target coverage:
    the repo wrapper did not force Cargo to inspect test-only call sites, so
    examples like the `latest_session_lookup_params(true, ...)` tests in
    `codex-rs/tui_app_server/src/lib.rs` never entered the blocking CI path.
    
    This change cleans up the existing backlog, makes the default repo lint
    path cover all Cargo targets, and starts rolling that stricter CI
    enforcement out on the platform where it is currently validated.
    
    ## What changed
    
    - mechanically fixed existing `argument-comment-lint` violations across
    the `codex-rs` workspace, including tests, examples, and benches
    - updated `tools/argument-comment-lint/run-prebuilt-linter.sh` and
    `tools/argument-comment-lint/run.sh` so non-`--fix` runs default to
    `--all-targets` unless the caller explicitly narrows the target set
    - fixed both wrappers so forwarded cargo arguments after `--` are
    preserved with a single separator
    - documented the new default behavior in
    `tools/argument-comment-lint/README.md`
    - updated `rust-ci` so the macOS lint lane keeps the plain wrapper
    invocation and therefore enforces `--all-targets`, while Linux and
    Windows temporarily pass `-- --lib --bins`
    
    That temporary CI split keeps the stricter all-targets check where it is
    already cleaned up, while leaving room to finish the remaining Linux-
    and Windows-specific target-gated cleanup before enabling
    `--all-targets` on those runners. The Linux and Windows failures on the
    intermediate revision were caused by the wrapper forwarding bug, not by
    additional lint findings in those lanes.
    
    ## Validation
    
    - `bash -n tools/argument-comment-lint/run.sh`
    - `bash -n tools/argument-comment-lint/run-prebuilt-linter.sh`
    - shell-level wrapper forwarding check for `-- --lib --bins`
    - shell-level wrapper forwarding check for `-- --tests`
    - `just argument-comment-lint`
    - `cargo test` in `tools/argument-comment-lint`
    - `cargo test -p codex-terminal-detection`
    
    ## Follow-up
    
    - Clean up remaining Linux-only target-gated callsites, then switch the
    Linux lint lane back to the plain wrapper invocation.
    - Clean up remaining Windows-only target-gated callsites, then switch
    the Windows lint lane back to the plain wrapper invocation.
  • feat: change multi-agent to use path-like system instead of uuids (#15313)
    This PR add an URI-based system to reference agents within a tree. This
    comes from a sync between research and engineering.
    
    The main agent (the one manually spawned by a user) is always called
    `/root`. Any sub-agent spawned by it will be `/root/agent_1` for example
    where `agent_1` is chosen by the model.
    
    Any agent can contact any agents using the path.
    
    Paths can be used either in absolute or relative to the calling agents
    
    Resume is not supported for now on this new path
  • Feat: CXA-1831 Persist latest model and reasoning effort in sqlite (#14859)
    ### Summary
    The goal is for us to get the latest turn model and reasoning effort on
    thread/resume is no override is provided on the thread/resume func call.
    This is the part 1 which we write the model and reasoning effort for a
    thread to the sqlite db and there will be a followup PR to consume the
    two new fields on thread/resume.
    
    [part 2 PR is currently WIP](https://github.com/openai/codex/pull/14888)
    and this one can be merged independently.
  • Apply argument comment lint across codex-rs (#14652)
    ## Why
    
    Once the repo-local lint exists, `codex-rs` needs to follow the
    checked-in convention and CI needs to keep it from drifting. This commit
    applies the fallback `/*param*/` style consistently across existing
    positional literal call sites without changing those APIs.
    
    The longer-term preference is still to avoid APIs that require comments
    by choosing clearer parameter types and call shapes. This PR is
    intentionally the mechanical follow-through for the places where the
    existing signatures stay in place.
    
    After rebasing onto newer `main`, the rollout also had to cover newly
    introduced `tui_app_server` call sites. That made it clear the first cut
    of the CI job was too expensive for the common path: it was spending
    almost as much time installing `cargo-dylint` and re-testing the lint
    crate as a representative test job spends running product tests. The CI
    update keeps the full workspace enforcement but trims that extra
    overhead from ordinary `codex-rs` PRs.
    
    ## What changed
    
    - keep a dedicated `argument_comment_lint` job in `rust-ci`
    - mechanically annotate remaining opaque positional literals across
    `codex-rs` with exact `/*param*/` comments, including the rebased
    `tui_app_server` call sites that now fall under the lint
    - keep the checked-in style aligned with the lint policy by using
    `/*param*/` and leaving string and char literals uncommented
    - cache `cargo-dylint`, `dylint-link`, and the relevant Cargo
    registry/git metadata in the lint job
    - split changed-path detection so the lint crate's own `cargo test` step
    runs only when `tools/argument-comment-lint/*` or `rust-ci.yml` changes
    - continue to run the repo wrapper over the `codex-rs` workspace, so
    product-code enforcement is unchanged
    
    Most of the code changes in this commit are intentionally mechanical
    comment rewrites or insertions driven by the lint itself.
    
    ## Verification
    
    - `./tools/argument-comment-lint/run.sh --workspace`
    - `cargo test -p codex-tui-app-server -p codex-tui`
    - parsed `.github/workflows/rust-ci.yml` locally with PyYAML
    
    ---
    
    * -> #14652
    * #14651
  • feat: polluted memories (#13008)
    Add a feature flag to disable memory creation for "polluted"
  • feat: add debug clear-memories command to hard-wipe memories state (#13085)
    #### what
    adds a `codex debug clear-memories` command to help with clearing all
    memories state from disk, sqlite db, and marking threads as
    `memory_mode=disabled` so they don't get resummarized when the
    `memories` feature is re-enabled.
    
    #### tests
    add tests
  • feat: memories forgetting (#12900)
    Add diff based memory forgetting
  • feat: add search term to thread list (#12578)
    Add `searchTerm` to `thread/list` that will search for a match in the
    titles (the condition being `searchTerm` $$\in$$ `title`)
  • feat: add nick name to sub-agents (#12320)
    Adding random nick name to sub-agents. Used for UX
    
    At the same time, also storing and wiring the role of the sub-agent
  • Add cwd to memory files (#11591)
    Add cwd to memory files so that model can deal with multi cwd memory
    better.
    
    ---------
    
    Co-authored-by: jif-oai <jif@openai.com>
  • fix: db stuff mem (#11575)
    * Documenting DB functions
    * Fixing 1 nit where stage-2 was sorting the stage 1 in the wrong
    direction
    * Added some tests
  • feat: mem slash commands (#11569)
    Add 2 slash commands for memories:
    * `/m_drop` delete all the memories
    * `/m_update` update the memories with phase 1 and 2
  • feat: new memory prompts (#11439)
    * Update prompt
    * Wire CWD in the prompt
    * Handle the no-output case
  • fix: flaky test (#11428)
    stage1_concurrent_claims_respect_running_cap was flaky due to SQLite
    lock contention, not cap logic correctness. The claim flow used deferred
    transactions (BEGIN) with read-then-write behavior, which can fail under
    concurrency with SQLITE_BUSY_SNAPSHOT/database is locked when upgrading
    a read transaction to a write transaction. We fixed this by using BEGIN
    IMMEDIATE for stage1 and phase2 claim paths, so lock acquisition happens
    up front and contenders serialize cleanly instead of failing during
    upgrade. After the change, codex-state tests pass and stress reruns of
    the flaky path no longer reproduced the failure.