31 Commits

• Persist selected capability roots and resolve availability per model step (#29856)

  ## Why
  
  `selectedCapabilityRoots` is durable thread intent: “use this capability
  root from environment `worker`.”
  
  The important product assumption is:
  
  > One environment ID always names the same logical executor and stable
  contents.
  
  `worker` does not silently change from executor A to an unrelated
  executor B. The process-local connection handle for `worker` can still
  be replaced while Codex is running, though, for example when
  `environment/add` registers a fresh handle for the same logical
  environment.
  
  The thread should persist only the stable selection. Each model step
  should pair that selection with the exact ready handle captured for that
  step.
  
  ## The boundary
  
  ```text
  persisted thread intent
    plugin@1 -> environment "worker"
                  |
                  | capture the current step
                  v
  model-step view
    unavailable, or
    plugin@1 + worker's exact captured ready handle
  ```
  
  The environment ID is the stable identity and cache key. The
  `Arc<Environment>` is only a process-local handle retained so consumers
  of one model step use the same captured environment. It is never
  persisted and it does not imply different environment contents.
  
  ## What changes
  
  ### Persist the stable selection
  
  Selected roots are written into `SessionMeta` and restored with the
  thread. Forked subagents inherit the same selections, including
  bounded-history forks.
  
  Only stable data is persisted: root ID, environment ID, and root path.
  
  ### Capture readiness together with the exact handle
  
  The environment snapshot records:
  
  ```rust
  environment_id -> Some(Arc<Environment>) // ready in this step
  environment_id -> None                   // still starting in this step
  ```
  
  This prevents readiness and execution from coming from different
  registry snapshots.
  
  For example:
  
  ```text
  step snapshot: worker -> handle A, ready
  environment/add: worker -> fresh handle B for the same logical environment
  current step: plugin@1 still uses captured handle A
  ```
  
  Without carrying handle A in the snapshot, the resolver could combine “A
  was ready” with handle B and treat B as ready before it had finished
  starting.
  
  This does not change cache invalidation. Stable capability metadata
  remains identified by environment ID and capability root. Replacing a
  process-local handle under the same stable environment ID does not
  invalidate or rediscover that metadata.
  
  ### Resolve availability per model step
  
  - A ready captured environment produces resolved roots using its
  captured handle.
  - A starting, missing, or failed environment is omitted from that step.
  - A selected lazy environment that is outside the turn's captured
  environment set is asked to start, and a later step can observe it as
  ready.
  - No capability files are scanned here.
  
  Transient transport disconnects remain the remote client's reconnect
  concern. This PR models initial attachment/readiness; it does not add
  live socket-connectivity state.
  
  ## Example
  
  ```text
  thread selection: plugin@1 -> environment "worker"
  
  step 1: worker is starting -> plugin@1 unavailable
  step 2: worker is ready    -> plugin@1 resolves through worker's captured handle
  step 3: fresh local handle -> current step remains pinned; a later step captures its own view
  ```
  
  Temporary unavailability does not discard the durable selection. Later
  PRs can retain stable metadata caches while projecting only currently
  available capabilities into model-visible World State.
  
  ## Compatibility
  
  The app-server request shape does not change. Older rollouts without
  `selected_capability_roots` deserialize to an empty list.
  
  ## Stack
  
  1. **This PR:** persist stable selected roots and resolve them through
  an exact model-step handle.
  2. #29960: cache stable skill metadata and project available skills into
  World State.
  3. #29946: cache stable plugin declarations and manage the separate live
  MCP runtime.

  jif · 2026-06-25 17:49:43 +00:00
  8f02973d25
• Support thread-level originator overrides (#29477)

  ## Why
  
  Work(TPP) threads can be launched from the Desktop app, but if they all
  keep the Desktop app's default originator then downstream attribution
  cannot distinguish local Work launches from cloud-backed Work launches.
  `thread/start.serviceName` already carries that launch signal, while
  `SessionMeta.originator` is the durable thread-level value that survives
  resume and fork.
  
  This change converts the Desktop Work service names into an effective
  originator at thread creation time, persists that originator with the
  thread, and keeps using it for later model requests and memory writes.
  
  ## What changed
  
  - Map `CODEX_WORK_LOCAL` and `CODEX_WORK_CLOUD` service names to
  per-thread originators, while preserving
  `CODEX_INTERNAL_ORIGINATOR_OVERRIDE` as the highest-precedence override.
  - Persist the effective originator in `SessionMeta.originator`, read it
  back on resume/fork, and inherit the parent originator for subagent
  spawns when there is no persisted session metadata.
  - Handle truncated `SpawnAgentForkMode::LastNTurns` forks by falling
  back to the live parent originator when the forked history no longer
  includes `SessionMeta`.
  - Thread the per-thread originator through Responses headers,
  websocket/compaction request paths, thread-store creation, rollout
  metadata, and memory stage-one telemetry.
  
  ## Verification
  
  - `just test -p codex-core
  agent::control::tests::spawn_thread_subagent_inherits_parent_originator_without_fork
  agent::control::tests::spawn_thread_subagent_fork_last_n_turns_inherits_parent_originator_without_session_meta
  thread_manager::tests::originator_override_precedes_service_name_remapping`
  - `just test -p codex-core
  agent::control::tests::resume_thread_subagent_restores_stored_metadata_and_effective_multi_agent_mode`
  - `just test -p codex-memories-write`
  - `just fix -p codex-core -p codex-memories-write`
  - `git diff --check`

  alexsong-oai · 2026-06-23 17:23:38 -07:00
  1acb722e8a
• core: persist initial context window metadata (#29519)

  ## Why
  
  PR #29494 made context-window IDs visible to the model by wrapping the
  token-budget window payload in `<context_window>`, but rollout JSONL
  consumers still could not see the initial window identity by tailing the
  session file. Compacted rollout items carry window IDs only after
  compaction has happened, so a session with no compaction had no durable
  JSONL record for window 0.
  
  This change gives tailing consumers a stable initial-window record at
  session creation time.
  
  ## What Changed
  
  - Added `session_meta.context_window.window_id` for the initial
  context-window identity.
  - `CreateThreadParams` now requires `initial_window_id: String`, so
  thread-store callers cannot accidentally create new threads without
  window-0 metadata.
  - Live thread creation derives the persisted initial window ID from the
  same `AutoCompactWindowIds` used to initialize `SessionState`, keeping
  runtime state and JSONL metadata aligned.
  - Rollout reconstruction uses `session_meta.context_window.window_id` as
  the initial-window fallback and derives `window_number = 0`,
  `first_window_id = window_id`, and `previous_window_id = None`
  internally.
  - Fork reconstruction intentionally uses the same rollout reconstruction
  path; consumers that need to distinguish copied initial-window metadata
  can use the rollout `thread_id`.
  - Legacy compactions without `window_number` still use compaction-count
  fallback accounting instead of being reset to window 0 by the
  initial-window fallback.
  - Compacted rollout metadata still takes precedence once compaction
  records exist, preserving the richer chain fields there.
  
  ## JSONL Shape
  
  Real rollout JSONL is one object per line. This example is expanded for
  readability, but shows the new initial `session_meta.context_window`
  record followed by the existing compacted rollout item shape that also
  carries window IDs:
  
  ```jsonl
  {
    "timestamp": "2026-06-22T12:00:00.000Z",
    "type": "session_meta",
    "payload": {
      "session_id": "<THREAD_ID>",
      "id": "<THREAD_ID>",
      "timestamp": "2026-06-22T12:00:00.000Z",
      "cwd": "/repo",
      "originator": "codex",
      "cli_version": "0.0.0",
      "source": "cli",
      "model_provider": "<MODEL_PROVIDER>",
      "context_window": {
        "window_id": "<INITIAL_WINDOW_ID>"
      }
    }
  }
  ...
  {
    "timestamp": "2026-06-22T12:34:56.000Z",
    "type": "compacted",
    "payload": {
      "message": "<COMPACTION_SUMMARY>",
      "replacement_history": [
        "..."
      ],
      "window_number": 1,
      "first_window_id": "<INITIAL_WINDOW_ID>",
      "previous_window_id": "<INITIAL_WINDOW_ID>",
      "window_id": "<NEXT_WINDOW_ID>"
    }
  }
  ```
  
  The nested `context_window` object is intentional: it gives rollout
  consumers a stable namespace for context-window metadata while only
  writing the non-derivable initial `window_id`. For the initial window,
  `window_number`, `first_window_id`, and `previous_window_id` are derived
  internally instead of being written to the rollout.
  
  ## Verification
  
  - `just test -p codex-protocol`
  - `just test -p codex-rollout
  recorder_materializes_on_flush_with_pending_items`
  - `just test -p codex-core reconstruct_history`
  - `just test -p codex-core
  record_initial_history_reconstructs_forked_transcript`
  - `just test -p codex-thread-store`
  - `just test -p codex-state`
  - `just test -p codex-app-server
  thread_read_returns_summary_without_turns`
  - `just test -p codex-rollout persistence_metrics`

  Michael Bolin · 2026-06-23 21:50:50 +00:00
  01f89c8c59
• core: rename metadata -> internal_chat_message_metadata_passthrough (#28968)

  ## Description
  This PR cuts Codex over from generic `ResponseItem.metadata` (introduced
  here: https://github.com/openai/codex/pull/28355) to
  `ResponseItem.internal_chat_message_metadata_passthrough`, which is the
  blessed path and has strongly-typed keys.
  
  For now we have to drop this MAv2 usage of `metadata`:
  https://github.com/openai/codex/pull/28561 until we figure out where
  that should live.

  Owen Lin · 2026-06-22 11:11:25 -07:00
  5b95745eae
• Persist session IDs across thread resume (#29327)

  ## Summary
  
  A cold-resumed subagent kept its durable thread ID but could receive a
  new session ID, splitting one agent tree across multiple sessions after
  a restart.
  
  Persist the root session ID in every rollout `SessionMeta`, carry it
  through thread creation, and restore it before initializing the resumed
  `Session` and `AgentControl`.
  
  ## Behavior
  
  For a nested agent tree:
  
  ```text
  root session R
    parent thread P
      child thread C
  ```
  
  The child rollout stores:
  
  ```text
  session_id:       R
  parent_thread_id: P
  id:               C
  ```
  
  After a cold resume, the child still belongs to root session `R` while
  its immediate parent remains `P`. The integration coverage uses distinct
  values for all three IDs so it catches restoring the session from
  `parent_thread_id`.
  
  ## Legacy rollouts
  
  Previous rollouts have `id` but no `session_id`. `SessionMetaLine`
  deserialization treats a missing `session_id` as `id`, keeping those
  files readable, listable, and resumable. When a legacy subagent is
  resumed through its root, that synthesized child ID no longer overrides
  the inherited root-scoped `AgentControl`. New rollouts always persist
  the explicit root session ID.

  jif · 2026-06-22 09:36:08 +02:00
  6d15bb3d17
• [codex] Add optional IDs to response items (#28812)

  ## Why
  
  `ResponseItem` variants do not have a consistent internal ID shape: some
  variants carry required IDs, some carry optional IDs, and some cannot
  represent an ID at all. The existing fields also use inconsistent serde,
  TypeScript, and JSON-schema annotations. A single enum-level access path
  is needed before history recording can assign and retain IDs.
  
  This PR establishes that internal model only. It intentionally does not
  generate or serialize IDs; allocation and wire persistence are isolated
  in the stacked follow-up.
  
  ## What changed
  
  - Give every concrete `ResponseItem` variant an `Option<String>` ID
  field.
  - Apply the same internal-only annotations to every ID field:
  `#[serde(default, skip_serializing)]`, `#[ts(skip)]`, and
  `#[schemars(skip)]`.
  - Add `ResponseItem::id()` and `ResponseItem::set_id()` as the shared
  accessors.
  - Preserve IDs when history items are rewritten for truncation.
  - Adapt consumers that previously assumed reasoning and image-generation
  IDs were required.
  - Regenerate app-server schemas so the hidden fields are represented
  consistently.
  
  The serde catch-all `ResponseItem::Other` remains ID-less because it
  must remain a unit variant.
  
  ## Test plan
  
  - `cargo check --tests -p codex-core -p codex-api -p codex-rollout-trace
  -p codex-image-generation-extension`
  - `just test -p codex-protocol`
  - `just test -p codex-app-server-protocol`
  - `just test -p codex-api -p codex-rollout-trace -p
  codex-image-generation-extension`
  - `just test -p codex-core event_mapping`

  pakrym-oai · 2026-06-17 18:27:43 -07:00
  dbd2857f4b
• [codex] Restore thread recency with compatible migration history (#28671)

  ## Summary
  
  - Revert #28655, restoring the thread `recencyAt` behavior introduced by
  #27910.
  - Move `threads_recency_at` to migration 0039 so it no longer collides
  with `external_agent_config_imports` at version 0038.
  - Repair databases that already applied the recency migration as version
  38 by moving the matching migration-history row to version 39 before
  SQLx validation. The current version-38 migration can then apply
  normally.
  
  ## Validation
  
  - `just test -p codex-state
  migrations::tests::repairs_recency_migration_that_was_applied_as_version_38`
  - `just test -p codex-state -p codex-rollout -p codex-thread-store -p
  codex-app-server-protocol -p codex-tui`: 3,439 passed; six TUI tests
  could not open the machine's existing read-only incident database at
  `~/.codex/sqlite/state_5.sqlite`.
  - `just fix -p codex-state`
  - `just fmt`
  - Verified that state migration versions are unique.

  Jeremy Rose · 2026-06-17 18:52:18 +00:00
  7dc7096ae1
• Revert thread recencyAt for sidebar ordering (#28655)

  ## Why
  
  Revert #27910 to remove the newly introduced thread `recencyAt`
  persistence and API behavior from `main`.
  
  ## What changed
  
  This reverts commit `fac3158c2a783095768076489815f361fa9b0db4`,
  including the state migration, thread-store propagation, app-server API
  surface, generated schemas, and related tests.
  
  ## Validation
  
  Not run before opening; relying on CI for the initial fast signal.

  pakrym-oai · 2026-06-16 21:39:30 -07:00
  cb15c64760
• thread-store: fix response fixture compilation (#28642)

  ## Why
  
  A `codex-thread-store` test fixture still constructs
  `ResponseItem::FunctionCallOutput` without its required `metadata`
  field, preventing the crate's test targets from compiling on `main`.
  
  ## What changed
  
  - Set the fixture's response-item metadata to `None`.
  
  ## Testing
  
  - `cargo check -p codex-thread-store --tests`

  pakrym-oai · 2026-06-16 19:16:16 -07:00
  6f77491e95
• Add thread recencyAt for sidebar ordering (#27910)

  ## Summary
  
  Add a server-owned `recencyAt` timestamp and `recency_at` thread-list
  sort key for product recency ordering while preserving the existing
  meaning of `updatedAt` as the latest persisted thread mutation.
  
  This is the server-side alternative to #27697. Rather than narrowing
  `updatedAt`, clients can sort the sidebar by `recency_at` and continue
  treating `updatedAt` as mutation time.
  
  Paired Codex Apps PR:
  [openai/openai#1024599](https://github.com/openai/openai/pull/1024599)
  
  ## Contract
  
  - `recencyAt` initializes when a thread is created.
  - A turn start advances `recencyAt` monotonically.
  - Commentary, agent output, tool results, token/accounting updates, turn
  completion, archive, unarchive, resume, and generic metadata writes do
  not advance it.
  - `updatedAt` retains its existing behavior and continues to advance for
  persisted thread mutations.
  - Current servers populate `recencyAt`; the response field is optional
  in generated TypeScript so clients connected to older servers can fall
  back to `updatedAt`.
  - Filesystem-only fallback uses existing updated/mtime ordering when
  SQLite is unavailable.
  
  ## Persistence and compatibility
  
  Migration 0038 adds second- and millisecond-precision recency columns,
  backfills them from the existing updated timestamp, creates list
  indexes, and includes an insert trigger so older binaries writing to a
  migrated database seed recency without causing later mutations to
  advance it.
  
  Generic metadata upserts preserve existing recency values. Turn-start
  updates use a dedicated monotonic touch, and process-local allocation
  keeps millisecond cursor values unique. State DB list, search, read,
  filtered-list repair, rollout fallback propagation, and app-server
  conversions all carry the new field.
  
  ## API
  
  `Thread` responses include:
  
  ```ts
  recencyAt?: number
  ```
  
  `thread/list` and `thread/search` accept:
  
  ```json
  { "sortKey": "recency_at" }
  ```
  
  Generated TypeScript and JSON schemas are included.
  
  ## Validation
  
  - `just test -p codex-state` — 146 passed
  - `just test -p codex-rollout` — 69 passed
  - `just test -p codex-thread-store` — 81 passed
  - `just test -p codex-app-server-protocol` — 231 passed
  - Focused app-server list ordering, response mapping, archive/unarchive,
  and resume lifecycle tests passed
  - Scoped `just fix` for state, rollout, thread-store,
  app-server-protocol, and app-server
  - `just fmt`
  - `git diff --check`
  - Independent correctness, simplicity, elegance, security, and
  test-quality reviews; actionable ordering, lifecycle, query-projection,
  and timestamp-uniqueness findings were addressed

  Jeremy Rose · 2026-06-16 17:06:22 -07:00
  fac3158c2a
• [codex] Remove async_trait from first-party code (#27475)

  ## Why
  
  First-party async traits should expose their `Send` contracts explicitly
  without requiring `async_trait`. This completes the migration pattern
  established in #27303 and #27304.
  
  ## What changed
  
  - Replaced the remaining first-party `async_trait` traits with native
  return-position `impl Future + Send` where statically dispatched and
  explicit boxed `Send` futures where object safety is required.
  - Kept implementations behavior-preserving, outlining existing async
  bodies into inherent methods where that keeps the diff reviewable.
  - Removed all direct first-party `async-trait` dependencies and the
  workspace dependency declaration.
  - Added a cargo-deny policy that permits `async-trait` only through the
  remaining transitive wrapper crates.
  - Updated `rand` from 0.8.5 to 0.8.6 to resolve RUSTSEC-2026-0097 and
  keep the full cargo-deny check passing.
  
  ## Validation
  
  - `just test -p codex-exec-server`: 216 passed, 2 skipped.
  - `just test -p codex-model-provider`: 39 passed.
  - `just test -p codex-core` and `just test`: changed tests passed;
  remaining failures are environment-sensitive suites unrelated to this
  migration.
  - `cargo deny check`
  - `just fix`
  - `just fmt`
  - `cargo shear`
  - `just bazel-lock-check`

  Adam Perry @ OpenAI · 2026-06-11 18:16:39 -07:00
  5a56caf18c
• Add app-server thread/delete API (#25018)

  ## Why
  
  Clients can archive and unarchive threads today, but there is no
  app-server API for permanently removing a thread. Deletion also needs to
  cover the full session tree: deleting a main thread should remove
  spawned subagent threads and the related local metadata instead of
  leaving orphaned rollout files, goals, or subagent state behind.
  
  ## What
  
  - Adds the v2 `thread/delete` request and `thread/deleted` notification,
  with the response shape kept consistent with `thread/archive`.
  - Implements local hard delete for active and archived rollout files.
  - Deletes the requested thread's state DB row as the commit point, then
  best-effort cleans associated state including spawned descendants,
  goals, spawn edges, logs, dynamic tools, and agent job assignments.
  - Updates app-server API docs and generated protocol schema/TypeScript
  fixtures.

  Eric Traut · 2026-06-10 11:22:12 -07:00
  a19d43a40a
• Add extra config to StoredThread, leave empty for now (#27092)
  Boyang Niu · 2026-06-09 00:38:35 +00:00
  e127a0cf99
• Persist multi-agent runtime metadata (#25721)

  Stack split from #25708. Original PR intentionally left open. This
  second PR persists multi-agent runtime metadata through thread creation,
  rollout recording, and thread storage.

  jif-oai · 2026-06-02 13:05:20 +02:00
  0c5ccd18ab
• app-server: remove experimental persist_extended_history bool flag (#25712)

  ## Summary
  
  Remove the dead experimental `persistExtendedHistory` app-server flag
  and collapse rollout persistence to the single policy app-server already
  used.
  
  ## What Changed
  
  - Removed `persistExtendedHistory` from v2 thread start/resume/fork
  params and deleted its deprecation notice path.
  - Removed the persistence-mode enums and plumbing through core, rollout,
  and thread-store.
  - Made rollout filtering mode-free, keeping the existing limited
  persisted-history behavior.
  
  ## Test Plan
  
  - `just write-app-server-schema`
  - `cargo nextest run --no-fail-fast -p codex-app-server-protocol
  schema_fixtures`
  - `cargo nextest run --no-fail-fast -p codex-app-server
  thread_shell_command_history_responses_exclude_persisted_command_executions`
  - `cargo nextest run --no-fail-fast -p codex-rollout -p
  codex-thread-store`
  - final `rg` for removed flag/type names

  Owen Lin · 2026-06-01 23:33:42 +00:00
  11e0f3d3ae
• store and expose parent_thread_id on Threads (#25113)

  ## Why
  
  This PR
  https://github.com/openai/codex/pull/24161#discussion_r3325692763
  revealed a subagent data modeling issue, where we overloaded
  `forked_from_id` to also mean `parent_thread_id`. That's incorrect since
  guardian and review subagents can be a subagent and NOT fork the main
  thread's history.
  
  The solution here is to explicitly store a new `parent_thread_id` on
  `SessionMeta`, alongside `forked_from_id` which already exists. While
  we're at it, also expose it in the app-server protocol on the `Thread`
  object.
  
  A thread->subagent relationship and a fork of thread history are
  orthogonal concepts.
  
  ## What Changed
  
  - Added top-level `parent_thread_id` persistence on `SessionMeta` and
  runtime/session plumbing through `SessionConfiguredEvent`,
  `CodexSpawnArgs`, `SessionConfiguration`, `ThreadConfigSnapshot`,
  `TurnContext`, and `ModelClient`.
  - Made turn metadata, request headers, analytics, and subagent-start
  events read the separate runtime/top-level parent field instead of
  deriving general parent lineage from `SessionSource` or
  `forked_from_thread_id`.
  - Passed parent lineage separately at delegated subagent, review,
  guardian, agent-job, and multi-agent spawn construction sites;
  copied-history fork lineage remains derived only from `InitialHistory`.
  - Persisted and exposed parent lineage through rollout/thread-store
  projections and app-server v2 `Thread.parentThreadId`.
  - Updated app-server README text and regenerated app-server schema
  fixtures for the additive `parentThreadId` response field.

  Owen Lin · 2026-06-01 04:33:20 +00:00
  cf0911076f
• [codex] Add user input client ids (#24653)

  ## Summary
  
  Adds an optional `clientId` field to app-server v2 `UserInput` and
  carries it through the core `UserInput` model so clients can correlate
  echoed user input items without relying on payload equality.
  
  ## Details
  
  - Adds `client_id: Option<String>` to core `UserInput` variants.
  - Exposes the v2 app-server field as `clientId` on the wire and in
  generated TypeScript.
  - Preserves the id when converting between app-server v2 and core
  protocol types.
  - Regenerates app-server schema fixtures.
  
  ## Validation
  
  - `just fmt`
  - `just write-app-server-schema`
  - `cargo test -p codex-app-server-protocol`
  - `cargo test -p codex-protocol`
  - `just fix -p codex-app-server-protocol`
  - `just fix -p codex-protocol`
  - `git diff --check`

  Alexi Christakis · 2026-05-28 14:54:39 -07:00
  e92c952b2e
• [codex] Add rollout-backed thread content search (#23519)

  ## Summary
  - add experimental `thread/search` for local rollout-backed thread
  search using `rg` over JSONL rollouts
  - return search-specific result rows with optional previews instead of
  storing preview data on `StoredThread` or ordinary `Thread` responses
  - keep `thread/list` separate from full-content search and document the
  new app-server surface
  
  ## Testing
  - `cargo test -p codex-app-server-protocol`
  - `cargo test -p codex-app-server
  thread_search_returns_content_and_title_matches -- --nocapture`

  Francis Chalissery · 2026-05-21 11:52:24 -07:00
  ac0bff27e7
• Preserve image detail in app-server inputs (#20693)

  ## Summary
  
  - Add optional image detail to user image inputs across core, app-server
  v2, thread history/event mapping, and the generated app-server
  schemas/types.
  - Preserve requested detail when serializing Responses image inputs:
  omitted detail stays on the existing `high` default, while explicit
  `original` keeps local images on the original-resolution path.
  - Support `high`/`original` consistently for tool image outputs,
  including MCP `codex/imageDetail`, code-mode image helpers, and
  `view_image`.

  Curtis 'Fjord' Hawthorne · 2026-05-15 15:04:04 -07:00
  8543e39885
• Unify thread metadata updates above store (#22236)

  - make ThreadStore::update_thread_metadata accept a broad range of
  metadata patches
  - keep ThreadStore::append_items as raw canonical history append (no
  metadata side effects)
  - in the local store, write these metadata updates to a combination of
  sqlite and rollout jsonl files for backwards-compat. It special cases
  which fields need to go into jsonl vs sqlite vs whatever, confining the
  awkwardness to just this implementation
  - in remote stores we can simply persist the metadata directly to a
  database, no special casing required.
  - move the "implicit metadata updates triggered by appending rollout
  items" from the RolloutRecorder (which is local-threadstore-specific) to
  the LiveThread layer above the ThreadStore, inside of a private helper
  utility called ThreadMetadataSync. LiveThread calls ThreadStore
  append_items and update_metadata separately.
  - Add a generic update metadata method to ThreadManager that works on
  both live threads and "cold" threads
  - Call that ThreadManager method from app server code, so app server
  doesn't need to worry about whether the thread is live or not

  Tom · 2026-05-13 00:28:15 +00:00
  c51c65ad09
• Revert state DB injection and agent graph store (#21481)

  ## Why
  
  Reverts #20689 to restore the previous optional state DB plumbing. The
  conflict resolution keeps the newer installation ID and session/thread
  identity changes that landed after #20689, while removing the mandatory
  state DB and agent graph store dependency from ThreadManager
  construction.
  
  ## What changed
  
  - Restored `Option<StateDbHandle>` through app-server, MCP server,
  prompt debug, and test entry points.
  - Removed the `codex-core` dependency on `codex-agent-graph-store` and
  reverted descendant lookup back to the existing state DB path when
  available.
  - Kept newer `installation_id` forwarding by passing it beside the
  optional DB handle.
  - Kept local thread-name updates working when the optional state DB
  handle is absent.
  
  ## Validation
  
  - `git diff --check`
  - `cargo test -p codex-thread-store`
  - `cargo test -p codex-state -p codex-rollout -p
  codex-app-server-protocol`
  - Attempted `env CARGO_INCREMENTAL=0 cargo test -p codex-core -p
  codex-app-server -p codex-app-server-client -p codex-mcp-server -p
  codex-thread-manager-sample -p codex-tui`; blocked locally by a rustc
  ICE while compiling `v8 v146.4.0` with `rustc 1.93.0 (254b59607
  2026-01-19)` on `aarch64-apple-darwin`.

  pakrym-oai · 2026-05-06 22:48:29 -07:00
  a8488fec5e
• [codex-analytics] rework thread_source for thread analytics (#20949)

  ## Summary
  - make `thread_source` an explicit optional thread-level field on
  `thread/start`, `thread/fork`, and returned thread payloads
  - persist `thread_source` in rollout/session metadata so resumed live
  threads retain the original value
  - replace the old best-effort `session_source` -> `thread_source`
  mapping with an explicit caller-supplied analytics classification
  
  ## Why
  Before this change, analytics `thread_source` was populated by a
  best-effort mapping from `session_source`. `session_source` describes
  the runtime/client surface, not the actual thread-level origin, so that
  projection was not accurate enough to distinguish cases such as `user`,
  `subagent`, `memory_consolidation`, and future thread origins reliably.
  
  Making `thread_source` explicit keeps one thread-level analytics field
  while letting callers provide the real classification directly instead
  of recovering it indirectly from `session_source`.
  
  ## Impact
  For new analytics events, `thread_source` now reflects the explicit
  thread-level classification supplied by the caller rather than an
  inferred value derived from `session_source`. Existing protocol fields
  remain optional; callers that omit `threadSource` now produce `null`
  instead of a best-effort inferred value.
  
  ## Validation
  - `just write-app-server-schema`
  - `cargo test -p codex-analytics -p codex-core -p
  codex-app-server-protocol --no-run`
  - `cargo test -p codex-app-server-protocol
  generated_ts_optional_nullable_fields_only_in_params`
  - `cargo test -p codex-analytics
  thread_initialized_event_serializes_expected_shape`
  - `cargo test -p codex-core
  resume_stopped_thread_from_rollout_preserves_thread_source`

  rhan-oai · 2026-05-06 02:12:31 +00:00
  b3d4f1a9f0
• Inject state DB, agent graph store (#20689)

  ## Why
  
  We want the agent graph store to be passed down the stack as a real
  dependency, the same way we already treat the thread store.
  
  This will let us inject the agent graph store as a real dependency and
  support implementations other than the local SQLite-backed one. Right
  now most code instantiates a state DB and an agent graph store
  just-in-time. Ideally, we would not depend on the state DB directly but
  only read through the higher-level interfaces.
  
  This change makes the dependency boundaries explicit and moves state DB
  initialization to process bootstrap instead of hiding it inside local
  store implementations.
  
  ## What changed
  
  - `ThreadManager` now requires a `StateDbHandle` and an
  `AgentGraphStore` at construction time instead of treating them as
  optional internals.
  - The local store constructors no longer lazily initialize SQLite.
  Callers now initialize the state DB once per process and use that shared
  handle to build:
    - `LocalThreadStore`
    - `LocalAgentGraphStore`
  - App bootstraps (`app-server`, `mcp-server`, `prompt_debug`, and the
  thread-manager sample) now initialize the state DB up front and inject
  the resulting handle down the stack.
  - `app-server` now consistently uses its process-scoped state DB handle
  instead of reopening SQLite or trying to recover it from loaded threads.
  - Device-key storage now reuses the shared state DB handle instead of
  maintaining its own lazy opener.
  - The thread archive / descendant traversal paths now use the injected
  `AgentGraphStore` instead of reaching through local
  thread-store-specific state.
  
  ## Verification
  
  - `cargo check -p codex-core -p codex-thread-store -p codex-app-server
  -p codex-mcp-server -p codex-thread-manager-sample --tests`
  - `cargo test -p codex-thread-store`
  - `cargo test -p codex-core
  thread_manager_accepts_separate_agent_graph_store_and_thread_store --
  --nocapture`
  - `cargo test -p codex-app-server
  thread_archive_archives_spawned_descendants -- --nocapture`

  Rasmus Rygaard · 2026-05-05 21:45:29 +00:00
  7e310bc7f3
• state: pass state db handles through consumers (#20561)

  ## Why
  
  SQLite state was still being opened from consumer paths, including lazy
  `OnceCell`-backed thread-store call sites. That let one process
  construct multiple state DB connections for the same Codex home, which
  makes SQLite lock contention and `database is locked` failures much
  easier to hit.
  
  State DB lifetime should be chosen by main-like entrypoints and tests,
  then passed through explicitly. Consumers should use the supplied
  `Option<StateDbHandle>` or `StateDbHandle` and keep their existing
  filesystem fallback or error behavior when no handle is available.
  
  The startup path also needs to keep the rollout crate in charge of
  SQLite state initialization. Opening `codex_state::StateRuntime`
  directly bypasses rollout metadata backfill, so entrypoints should
  initialize through `codex_rollout::state_db` and receive a handle only
  after required rollout backfills have completed.
  
  ## What Changed
  
  - Initialize the state DB in main-like entrypoints for CLI, TUI,
  app-server, exec, MCP server, and the thread-manager sample.
  - Pass `Option<StateDbHandle>` through `ThreadManager`,
  `LocalThreadStore`, app-server processors, TUI app wiring, rollout
  listing/recording, personality migration, shell snapshot cleanup,
  session-name lookup, and memory/device-key consumers.
  - Remove the lazy local state DB wrapper from the thread store so
  non-test consumers use only the supplied handle or their existing
  fallback path.
  - Make `codex_rollout::state_db::init` the local state startup path: it
  opens/migrates SQLite, runs rollout metadata backfill when needed, waits
  for concurrent backfill workers up to a bounded timeout, verifies
  completion, and then returns the initialized handle.
  - Keep optional/non-owning SQLite helpers, such as remote TUI local
  reads, as open-only paths that do not run startup backfill.
  - Switch app-server startup from direct
  `codex_state::StateRuntime::init` to the rollout state initializer so
  app-server cannot skip rollout backfill.
  - Collapse split rollout lookup/list APIs so callers use the normal
  methods with an optional state handle instead of `_with_state_db`
  variants.
  - Restore `getConversationSummary(ThreadId)` to delegate through
  `ThreadStore::read_thread` instead of a LocalThreadStore-specific
  rollout path special case.
  - Keep DB-backed rollout path lookup keyed on the DB row and file
  existence, without imposing the filesystem filename convention on
  existing DB rows.
  - Verify readable DB-backed rollout paths against `session_meta.id`
  before returning them, so a stale SQLite row that points at another
  thread's JSONL falls back to filesystem search and read-repairs the DB
  row.
  - Keep `debug prompt-input` filesystem-only so a one-off debug command
  does not initialize or backfill SQLite state just to print prompt input.
  - Keep goal-session test Codex homes alive only in the goal-specific
  helper, rather than leaking tempdirs from the shared session test
  helper.
  - Update tests and call sites to pass explicit state handles where DB
  behavior is expected and explicit `None` where filesystem-only behavior
  is intended.
  
  ## Validation
  
  - `CARGO_TARGET_DIR=/tmp/codex-target-state-db cargo check -p
  codex-rollout -p codex-thread-store -p codex-app-server -p codex-core -p
  codex-tui -p codex-exec -p codex-cli --tests`
  - `CARGO_TARGET_DIR=/tmp/codex-target-state-db cargo test -p
  codex-rollout state_db_`
  - `CARGO_TARGET_DIR=/tmp/codex-target-state-db cargo test -p
  codex-rollout find_thread_path`
  - `CARGO_TARGET_DIR=/tmp/codex-target-state-db cargo test -p
  codex-rollout find_thread_path -- --nocapture`
  - `CARGO_TARGET_DIR=/tmp/codex-target-state-db cargo test -p
  codex-rollout try_init_ -- --nocapture`
  - `CARGO_TARGET_DIR=/tmp/codex-target-state-db cargo test -p
  codex-rollout`
  - `CARGO_TARGET_DIR=/tmp/codex-target-state-db cargo clippy -p
  codex-rollout --lib -- -D warnings`
  - `CARGO_TARGET_DIR=/tmp/codex-target-state-db cargo test -p
  codex-thread-store
  read_thread_falls_back_when_sqlite_path_points_to_another_thread --
  --nocapture`
  - `CARGO_TARGET_DIR=/tmp/codex-target-state-db cargo test -p
  codex-thread-store`
  - `CARGO_TARGET_DIR=/tmp/codex-target-state-db cargo test -p codex-core
  shell_snapshot`
  - `CARGO_TARGET_DIR=/tmp/codex-target-state-db cargo test -p codex-core
  --test all personality_migration`
  - `CARGO_TARGET_DIR=/tmp/codex-target-state-db cargo test -p codex-core
  --test all rollout_list_find`
  - `RUST_MIN_STACK=8388608 CODEX_SKIP_VENDORED_BWRAP=1
  CARGO_TARGET_DIR=/tmp/codex-target-state-db cargo test -p codex-core
  --test all rollout_list_find::find_prefers_sqlite_path_by_id --
  --nocapture`
  - `RUST_MIN_STACK=8388608 CODEX_SKIP_VENDORED_BWRAP=1
  CARGO_TARGET_DIR=/tmp/codex-target-state-db cargo test -p codex-core
  --test all rollout_list_find -- --nocapture`
  - `CARGO_TARGET_DIR=/tmp/codex-target-state-db cargo test -p codex-core
  interrupt_accounts_active_goal_before_pausing`
  - `CARGO_TARGET_DIR=/tmp/codex-target-state-db cargo test -p
  codex-app-server get_auth_status -- --test-threads=1`
  - `CODEX_SKIP_VENDORED_BWRAP=1
  CARGO_TARGET_DIR=/tmp/codex-target-state-db cargo test -p
  codex-app-server --lib`
  - `CODEX_SKIP_VENDORED_BWRAP=1
  CARGO_TARGET_DIR=/tmp/codex-target-state-db cargo check -p codex-rollout
  -p codex-app-server --tests`
  - `CARGO_TARGET_DIR=/tmp/codex-target-state-db just fix -p codex-rollout
  -p codex-thread-store -p codex-core -p codex-app-server -p codex-tui -p
  codex-exec -p codex-cli`
  - `CODEX_SKIP_VENDORED_BWRAP=1
  CARGO_TARGET_DIR=/tmp/codex-target-state-db just fix -p codex-rollout -p
  codex-app-server`
  - `CARGO_TARGET_DIR=/tmp/codex-target-state-db just fix -p
  codex-rollout`
  - `CODEX_SKIP_VENDORED_BWRAP=1
  CARGO_TARGET_DIR=/tmp/codex-target-state-db just fix -p codex-core`
  - `just argument-comment-lint -p codex-core`
  - `just argument-comment-lint -p codex-rollout`
  
  Focused coverage added in `codex-rollout`:
  
  - `recorder::tests::state_db_init_backfills_before_returning` verifies
  the rollout metadata row exists before startup init returns.
  - `state_db::tests::try_init_waits_for_concurrent_startup_backfill`
  verifies startup waits for another worker to finish backfill instead of
  disabling the handle for the process.
  -
  `state_db::tests::try_init_times_out_waiting_for_stuck_startup_backfill`
  verifies startup does not hang indefinitely on a stuck backfill lease.
  -
  `tests::find_thread_path_accepts_existing_state_db_path_without_canonical_filename`
  verifies DB-backed lookup accepts valid existing rollout paths even when
  the filename does not include the thread UUID.
  -
  `tests::find_thread_path_falls_back_when_db_path_points_to_another_thread`
  verifies DB-backed lookup ignores a stale row whose existing path
  belongs to another thread and read-repairs the row after filesystem
  fallback.
  
  Focused coverage updated in `codex-core`:
  
  - `rollout_list_find::find_prefers_sqlite_path_by_id` now uses a
  DB-preferred rollout file with matching `session_meta.id`, so it still
  verifies that valid SQLite paths win without depending on stale/empty
  rollout contents.
  
  `cargo test -p codex-app-server thread_list_respects_search_term_filter
  -- --test-threads=1 --nocapture` was attempted locally but timed out
  waiting for the app-server test harness `initialize` response before
  reaching the changed thread-list code path.
  
  `bazel test //codex-rs/thread-store:thread-store-unit-tests
  --test_output=errors` was attempted locally after the thread-store fix,
  but this container failed before target analysis while fetching `v8+`
  through BuildBuddy/direct GitHub. The equivalent local crate coverage,
  including `cargo test -p codex-thread-store`, passes.
  
  A plain local `cargo check -p codex-rollout -p codex-app-server --tests`
  also requires system `libcap.pc` for `codex-linux-sandbox`; the
  follow-up app-server check above used `CODEX_SKIP_VENDORED_BWRAP=1` in
  this container.

  Ruslan Nigmatullin · 2026-05-04 11:46:03 -07:00
  4d201e340e
• Make thread store process-scoped (#19474)

  - Build one app-server process ThreadStore from startup config and share
  it with ThreadManager and CodexMessageProcessor.
  - Remove per-thread/fork store reconstruction so effective thread config
  cannot switch the persistence backend.
  - Add params to ThreadStore create/resume for specifying thread
  metadata, since otherwise the metadata from store creation would be used
  (incorrectly).

  Tom · 2026-04-30 21:24:59 -07:00
  fe05acad23
• Migrate fork and resume reads to thread store (#18900)

  - Route cold thread/resume and thread/fork source loading through
  ThreadStore reads instead of direct rollout path operations
  - Keep lookups that explicitly specify a rollout-path using the local
  thread store methods but return an invalid-request error for remote
  ThreadStore configurations
  - Add some additional unit tests for code path coverage

  Tom · 2026-04-24 13:51:37 -07:00
  0a9b559c0b
• [codex] Route live thread writes through ThreadStore (#18882)

  Begin migrating the thread write codepaths to ThreadStore.
  
  This starts using ThreadStore inside of core session code, not only in
  the app server code.
  
  Rework the interfaces around thread recording/persistence. We're left
  with the following:
  
  * `ThreadManager`: owns the process-level registry of loaded threads and
  handles cross-thread orchestration: start, resume, fork, lookup, remove,
  and route ops to running CodexThreads.
  * `CodexThread`: represents one loaded/running thread from the outside.
  It is the handle app-server and callers use to submit ops, inspect
  session metadata, and shut the thread down.
  * `LiveThread`: session-owned persistence lifecycle handle for one
  active thread. Core session code uses it to append rollout items,
  materialize lazy persistence, flush, shutdown, discard init-failed
  writers, and load that thread’s persisted history.
  * `ThreadStore`: storage backend abstraction. It answers “how are
  threads persisted, read, listed, updated, archived?” Local and remote
  implementations live behind this trait.
  * `LocalThreadStore`: local ThreadStore implementation. It owns the
  file/sqlite-specific details and keeps RolloutRecorder as a local
  implementation detail.
  
  This is a few too many Thread abstractions for my liking, but they do
  all represent different concepts / needs / layers.
  
  Migration note: in places where the core code explicitly requires a
  path, rather than a thread ID, throw an error if we're running with a
  remote store.
  
  Cover the new local live-writer lifecycle with focused tests and
  preserve app-server thread-start behavior, including ephemeral pathless
  sessions.

  Tom · 2026-04-23 10:17:09 -07:00
  f1923a38b1
• Read conversation summaries through thread store (#18716)

  Migrate the conversation summary App Server methods to ThreadStore
  
  Because this app server api allows explicitly fetching the thread by
  rollout path, intercept that case in the app server code and (a) route
  directly to underlying local thread store methods if we're using a local
  thread store, or (b) throw an unsupported error if we're using a remote
  thread store. This keeps the thread store API clean and all filesystem
  operations inside of the local thread store, which pushing the
  "fundamental incompatibility" check as early as possible.

  Tom · 2026-04-20 22:39:10 +00:00
  a718b6fd47
• codex: move unloaded thread writes into store (#18361)

  - Migrates unloaded `thread/name/set` and `thread/memoryModeSet`
  app-server writes behind the generic
  `ThreadStore::update_thread_metadata` API rather than adding one-off
  store methods for setting thread name or memory mode.
  - Implements the local ThreadStore metadata patch path for thread name
  and memory mode, including rollout append, legacy name index updates,
  SessionMeta validation/update, SQLite reconciliation, and re-reading the
  stored thread.
  - Adds focused local thread-store unit coverage plus app-server
  integration coverage for the migrated unloaded write paths.

  Tom · 2026-04-20 09:50:01 -07:00
  ac7c9a685f
• Migrate archive/unarchive to local ThreadStore (#17892)

  # Summary
  - implement local ThreadStore archive/unarchive operations
  - implement local ThreadStore read_thread operation
  - break up the various ThreadStore local method implementations into
  separate files
  - migrate app-server archive/unarchive and core archive fixture to use
  ThreadStore (but not all read operations yet!)
  - use the ThreadStore's read operation as a proxy check for thread
  persistence/existence in the app server code
  - move all other filesystem operations related to archive (path
  validation etc) into the local thread store.
  
  # Tests
  - add dedicated local store archive/unarchive tests

  Tom · 2026-04-15 20:48:09 +00:00
  50d3128269
• [codex] Add local thread store listing (#17824)

  Builds on top of #17659 
  
  Move the filesystem + sqlite thread listing-related operations inside of
  a local ThreadStore implementation and call ThreadStore from the places
  that used to perform these filesystem/sqlite operations.
  
  This is the first of a series of PRs that will implement the rest of the
  local ThreadStore.
  
  Testing:
  - added unit tests for the thread store implementation
  - adjusted some unit tests in the realtime + personality packages whose
  callsites changed. Specifically I'm trying to hide ThreadMetadata inside
  of the local implementation and make ThreadMetadata a sqlite
  implementation detail concern rather than a public interface, preferring
  the more generate StoredThread interface instead
  - added a corner case test for the personality migration package that
  wasn't covered by the existing test suite
  - adjust the behavior of searched thread listing to run the existing
  local rollout repair/backfill pass _before_ querying SQLite results, so
  callers using ThreadStore::list_threads do not miss matches after a
  partial metadata warm-up

  Tom · 2026-04-15 11:34:27 -07:00
  cdfcd2ca92