70 Commits

• Add thread metadata update endpoint to app server (#13280)

  ## Summary
  - add the v2 `thread/metadata/update` API, including
  protocol/schema/TypeScript exports and app-server docs
  - patch stored thread `gitInfo` in sqlite without resuming the thread,
  with validation plus support for explicit `null` clears
  - repair missing sqlite thread rows from rollout data before patching,
  and make those repairs safe by inserting only when absent and updating
  only git columns so newer metadata is not clobbered
  - keep sqlite authoritative for mutable thread git metadata by
  preserving existing sqlite git fields during reconcile/backfill and only
  using rollout `SessionMeta` git fields to fill gaps
  - add regression coverage for the endpoint, repair paths, concurrent
  sqlite writes, clearing git fields, and rollout/backfill reconciliation
  - fix the login server shutdown race so cancelling before the waiter
  starts still terminates `block_until_done()` correctly
  
  ## Testing
  - `cargo test -p codex-state
  apply_rollout_items_preserves_existing_git_branch_and_fills_missing_git_fields`
  - `cargo test -p codex-state
  update_thread_git_info_preserves_newer_non_git_metadata`
  - `cargo test -p codex-core
  backfill_sessions_preserves_existing_git_branch_and_fills_missing_git_fields`
  - `cargo test -p codex-app-server thread_metadata_update`
  - `cargo test`
  - currently fails in existing `codex-core` grep-files tests with
  `unsupported call: grep_files`:
      - `suite::grep_files::grep_files_tool_collects_matches`
      - `suite::grep_files::grep_files_tool_reports_empty_results`

  joeytrasatti-openai · 2026-03-03 15:56:11 -08:00
  935754baa3
• app-server: source /feedback logs from sqlite at trace level (#12969)

  ## Summary
  - write app-server SQLite logs at TRACE level when SQLite is enabled
  - source app-server `/feedback` log attachments from SQLite for the
  requested thread when available
  - flush buffered SQLite log writes before `/feedback` queries them so
  newly emitted events are not lost behind the async inserter
  - include same-process threadless SQLite rows in those `/feedback` logs
  so the attachment matches the process-wide feedback buffer more closely
  - keep the existing in-memory ring buffer fallback unchanged, including
  when the SQLite query returns no rows
  
  ## Details
  - add a byte-bounded `query_feedback_logs` helper in `codex-state` so
  `/feedback` does not fetch all rows before truncating
  - scope SQLite feedback logs to the requested thread plus threadless
  rows from the same `process_uuid`
  - format exported SQLite feedback lines with the log level prefix to
  better match the in-memory feedback formatter
  - add an explicit `LogDbLayer::flush()` control path and await it in
  app-server before querying SQLite for feedback logs
  - pass optional SQLite log bytes through `codex-feedback` as the
  `codex-logs.log` attachment override
  - leave TUI behavior unchanged apart from the updated `upload_feedback`
  call signature
  - add regression coverage for:
    - newest-within-budget ordering
    - excluding oversized newest rows
    - including same-process threadless rows
    - keeping the newest suffix across mixed thread and threadless rows
    - matching the feedback formatter shape aside from span prefixes
    - falling back to the in-memory snapshot when SQLite returns no logs
    - flushing buffered SQLite rows before querying
  
  ## Follow-up
  - SQLite feedback exports still do not reproduce span prefixes like
  `feedback-thread{thread_id=...}:`; there is a `TODO(ccunningham)` in
  `codex-rs/state/src/log_db.rs` for that follow-up.
  
  ## Testing
  - `cd codex-rs && cargo test -p codex-state`
  - `cd codex-rs && cargo test -p codex-app-server`
  - `cd codex-rs && just fmt`

  Charley Cunningham · 2026-03-03 11:17:06 -08:00
  c4bd0aa3b9
• feat: polluted memories (#13008)

  Add a feature flag to disable memory creation for "polluted"

  jif-oai · 2026-03-02 11:57:32 +00:00
  b649953845
• Record realtime close marker on replacement (#13058)

  ## Summary
  - record a realtime close developer message when a new realtime session
  replaces an active one
  - assert the replacement marker through the mocked responses request
  path
  
  ---------
  
  Co-authored-by: Codex <noreply@openai.com>
  Co-authored-by: Charles Cunningham <ccunningham@openai.com>

  Ahmed Ibrahim · 2026-03-01 13:54:12 -08:00
  0aeb55bf08
• feat: add debug clear-memories command to hard-wipe memories state (#13085)

  #### what
  adds a `codex debug clear-memories` command to help with clearing all
  memories state from disk, sqlite db, and marking threads as
  `memory_mode=disabled` so they don't get resummarized when the
  `memories` feature is re-enabled.
  
  #### tests
  add tests

  sayan-oai · 2026-02-27 17:45:55 -08:00
  033ef9cb9d
• feat: use the memory mode for phase 1 extraction (#13002)
  jif-oai · 2026-02-27 12:49:03 +01:00
  c76bc8d1ce
• feat: gen memories config (#12999)
  jif-oai · 2026-02-27 12:38:47 +01:00
  bbd237348d
• feat: add local date/timezone to turn environment context (#12947)

  ## Summary
  
  This PR includes the session's local date and timezone in the
  model-visible environment context and persists that data in
  `TurnContextItem`.
  
    ## What changed
  - captures the current local date and IANA timezone when building a turn
  context, with a UTC fallback if the timezone lookup fails
  - includes current_date and timezone in the serialized
  <environment_context> payload
  - stores those fields on TurnContextItem so they survive rollout/history
  handling, subagent review threads, and resume flows
  - treats date/timezone changes as environment updates, so prompt caching
  and context refresh logic do not silently reuse stale time context
  - updates tests to validate the new environment fields without depending
  on a single hardcoded environment-context string
  
  ## test
  
  built a local build and saw it in the rollout file:
  ```
  {"timestamp":"2026-02-26T21:39:50.737Z","type":"response_item","payload":{"type":"message","role":"user","content":[{"type":"input_text","text":"<environment_context>\n  <shell>zsh</shell>\n  <current_date>2026-02-26</current_date>\n  <timezone>America/Los_Angeles</timezone>\n</environment_context>"}]}}
  ```

  Celia Chen · 2026-02-26 23:17:35 +00:00
  90cc4e79a2
• feat: add git info to memories (#12940)
  jif-oai · 2026-02-26 20:14:13 +00:00
  a6065d30f4
• fix: do not apply turn cwd to metadata (#12887)

  Details here:
  https://openai.slack.com/archives/C09NZ54M4KY/p1772056758227339

  jif-oai · 2026-02-26 17:05:58 +00:00
  739d4b52de
• feat: use memory usage for selection (#12909)
  jif-oai · 2026-02-26 16:44:02 +00:00
  c528f32acb
• chore: clean DB runtime (#12905)
  jif-oai · 2026-02-26 14:11:10 +00:00
  79d6f80e41
• feat: memories forgetting (#12900)

  Add diff based memory forgetting

  jif-oai · 2026-02-26 13:19:57 +00:00
  382fa338b3
• Add rollout path to memory files and search for them during read (#12684)

  Co-authored-by: jif-oai <jif@openai.com>

  Wendy Jiao · 2026-02-26 10:57:01 +00:00
  52aa49db1b
• nit: migration (#12772)
  jif-oai · 2026-02-25 13:56:52 +00:00
  8d49e0d0c4
• feat: record memory usage (#12761)
  jif-oai · 2026-02-25 13:48:40 +00:00
  e4bfa763f6
• feat: add search term to thread list (#12578)

  Add `searchTerm` to `thread/list` that will search for a match in the
  titles (the condition being `searchTerm` $$\in$$ `title`)

  jif-oai · 2026-02-25 09:59:41 +00:00
  f46b767b7e
• Agent jobs (spawn_agents_on_csv) + progress UI (#10935)

  ## Summary
  - Add agent job support: spawn a batch of sub-agents from CSV, auto-run,
  auto-export, and store results in SQLite.
  - Simplify workflow: remove run/resume/get-status/export tools; spawn is
  deterministic and completes in one call.
  - Improve exec UX: stable, single-line progress bar with ETA; suppress
  sub-agent chatter in exec.
  
  ## Why
  Enables map-reduce style workflows over arbitrarily large repos using
  the existing Codex orchestrator. This addresses review feedback about
  overly complex job controls and non-deterministic monitoring.
  
  ## Demo (progress bar)
  ```
  ./codex-rs/target/debug/codex exec \
    --enable collab \
    --enable sqlite \
    --full-auto \
    --progress-cursor \
    -c agents.max_threads=16 \
    -C /Users/daveaitel/code/codex \
    - <<'PROMPT'
  Create /tmp/agent_job_progress_demo.csv with columns: path,area and 30 rows:
  path = item-01..item-30, area = test.
  
  Then call spawn_agents_on_csv with:
  - csv_path: /tmp/agent_job_progress_demo.csv
  - instruction: "Run `python - <<'PY'` to sleep a random 0.3–1.2s, then output JSON with keys: path, score (int). Set score = 1."
  - output_csv_path: /tmp/agent_job_progress_demo_out.csv
  PROMPT
  ```
  
  ## Review feedback addressed
  - Auto-start jobs on spawn; removed run/resume/status/export tools.
  - Auto-export on success.
  - More descriptive tool spec + clearer prompts.
  - Avoid deadlocks on spawn failure; pending/running handled safely.
  - Progress bar no longer scrolls; stable single-line redraw.
  
  ## Tests
  - `cd codex-rs && cargo test -p codex-exec`
  - `cd codex-rs && cargo build -p codex-cli`

  daveaitel-openai · 2026-02-24 21:00:19 +00:00
  dcab40123f
• feat: do not enqueue phase 2 if not necessary (#12344)
  jif-oai · 2026-02-20 17:21:45 +00:00
  fd67aba114
• feat: add nick name to sub-agents (#12320)

  Adding random nick name to sub-agents. Used for UX
  
  At the same time, also storing and wiring the role of the sub-agent

  jif-oai · 2026-02-20 14:39:49 +00:00
  0f9eed3a6f
• state: enforce 10 MiB log caps for thread and threadless process logs (#12038)

  ## Summary
  - enforce a 10 MiB cap per `thread_id` in state log storage
  - enforce a 10 MiB cap per `process_uuid` for threadless (`thread_id IS
  NULL`) logs
  - scope pruning to only keys affected by the current insert batch
  - add a cheap per-key `SUM(...)` precheck so windowed prune queries only
  run for keys that are currently over the cap
  - add SQLite indexes used by the pruning queries
  - add focused runtime tests covering both pruning behaviors
  
  ## Why
  This keeps log growth bounded by the intended partition semantics while
  preserving a small, readable implementation localized to the existing
  insert path.
  
  ## Local Latency Snapshot (No Truncation-Pressure Run)
  Collected from session `019c734f-1d16-7002-9e00-c966c9fbbcae` using
  local-only (uncommitted) instrumentation, while not specifically
  benchmarking the truncation-heavy regime.
  
  ### Percentiles By Query (ms)
  | query | count | p50 | p90 | p95 | p99 | max |
  |---|---:|---:|---:|---:|---:|---:|
  | `insert_logs.insert_batch` | 110 | 0.332 | 0.999 | 1.811 | 2.978 |
  3.493 |
  | `insert_logs.precheck.process` | 106 | 0.074 | 0.152 | 0.206 | 0.258 |
  0.426 |
  | `insert_logs.precheck.thread` | 73 | 0.118 | 0.206 | 0.253 | 1.025 |
  1.025 |
  | `insert_logs.prune.process` | 58 | 0.291 | 0.576 | 0.607 | 1.088 |
  1.088 |
  | `insert_logs.prune.thread` | 44 | 0.318 | 0.467 | 0.728 | 0.797 |
  0.797 |
  | `insert_logs.prune_total` | 110 | 0.488 | 0.976 | 1.237 | 1.593 |
  1.684 |
  | `insert_logs.total` | 110 | 1.315 | 2.889 | 3.623 | 5.739 | 5.961 |
  | `insert_logs.tx_begin` | 110 | 0.133 | 0.235 | 0.282 | 0.412 | 0.546 |
  | `insert_logs.tx_commit` | 110 | 0.259 | 0.689 | 0.772 | 1.065 | 1.080
  |
  
  ### `insert_logs.total` Histogram (ms)
  | bucket | count |
  |---|---:|
  | `<= 0.100` | 0 |
  | `<= 0.250` | 0 |
  | `<= 0.500` | 7 |
  | `<= 1.000` | 33 |
  | `<= 2.000` | 40 |
  | `<= 5.000` | 28 |
  | `<= 10.000` | 2 |
  | `<= 20.000` | 0 |
  | `<= 50.000` | 0 |
  | `<= 100.000` | 0 |
  | `> 100.000` | 0 |
  
  ## Local Latency Snapshot (Truncation-Heavy / Cap-Hit Regime)
  Collected from a run where cap-hit behavior was frequent (`135/180`
  insert calls), using local-only (uncommitted) instrumentation and a
  temporary local cap of `10_000` bytes for stress testing (not the merged
  `10 MiB` cap).
  
  ### Percentiles By Query (ms)
  | query | count | p50 | p90 | p95 | p99 | max |
  |---|---:|---:|---:|---:|---:|---:|
  | `insert_logs.insert_batch` | 180 | 0.524 | 1.645 | 2.163 | 3.424 |
  3.777 |
  | `insert_logs.precheck.process` | 171 | 0.086 | 0.235 | 0.373 | 0.758 |
  1.147 |
  | `insert_logs.precheck.thread` | 100 | 0.105 | 0.251 | 0.291 | 1.176 |
  1.622 |
  | `insert_logs.prune.process` | 109 | 0.386 | 0.839 | 1.146 | 1.548 |
  2.588 |
  | `insert_logs.prune.thread` | 56 | 0.253 | 0.550 | 1.148 | 2.484 |
  2.484 |
  | `insert_logs.prune_total` | 180 | 0.511 | 1.221 | 1.695 | 4.548 |
  5.512 |
  | `insert_logs.total` | 180 | 1.631 | 3.902 | 5.103 | 8.901 | 9.095 |
  | `insert_logs.total_cap_hit` | 135 | 1.876 | 4.501 | 5.547 | 8.902 |
  9.096 |
  | `insert_logs.total_no_cap_hit` | 45 | 0.520 | 1.700 | 2.079 | 3.294 |
  3.294 |
  | `insert_logs.tx_begin` | 180 | 0.109 | 0.253 | 0.287 | 1.088 | 1.406 |
  | `insert_logs.tx_commit` | 180 | 0.267 | 0.813 | 1.170 | 2.497 | 2.574
  |
  
  ### `insert_logs.total` Histogram (ms)
  | bucket | count |
  |---|---:|
  | `<= 0.100` | 0 |
  | `<= 0.250` | 0 |
  | `<= 0.500` | 16 |
  | `<= 1.000` | 39 |
  | `<= 2.000` | 60 |
  | `<= 5.000` | 54 |
  | `<= 10.000` | 11 |
  | `<= 20.000` | 0 |
  | `<= 50.000` | 0 |
  | `<= 100.000` | 0 |
  | `> 100.000` | 0 |
  
  ### `insert_logs.total` Histogram When Cap Was Hit (ms)
  | bucket | count |
  |---|---:|
  | `<= 0.100` | 0 |
  | `<= 0.250` | 0 |
  | `<= 0.500` | 0 |
  | `<= 1.000` | 22 |
  | `<= 2.000` | 51 |
  | `<= 5.000` | 51 |
  | `<= 10.000` | 11 |
  | `<= 20.000` | 0 |
  | `<= 50.000` | 0 |
  | `<= 100.000` | 0 |
  | `> 100.000` | 0 |
  
  ### Performance Takeaways
  - Even in a cap-hit-heavy run (`75%` cap-hit calls), `insert_logs.total`
  stays sub-10ms at p99 (`8.901ms`) and max (`9.095ms`).
  - Calls that did **not** hit the cap are materially cheaper
  (`insert_logs.total_no_cap_hit` p95 `2.079ms`) than cap-hit calls
  (`insert_logs.total_cap_hit` p95 `5.547ms`).
  - Compared to the earlier non-truncation-pressure run, overall
  `insert_logs.total` rose from p95 `3.623ms` to p95 `5.103ms`
  (+`1.48ms`), indicating bounded overhead when pruning is active.
  - This truncation-heavy run used an intentionally low local cap for
  stress testing; with the real 10 MiB cap, cap-hit frequency should be
  much lower in normal sessions.
  
  ## Testing
  - `just fmt` (in `codex-rs`)
  - `cargo test -p codex-state` (in `codex-rs`)

  Charley Cunningham · 2026-02-18 17:08:08 -08:00
  7f3dbaeb25
• feat: add --compact mode to just log (#11994)

  Summary:
  - add a `--compact` flag to the logs client to suppress thread/target
  info
  - format rows and timestamps differently when compact mode is enabled so
  only hour time, level, and message remain

  jif-oai · 2026-02-17 14:21:26 +00:00
  4ab44e2c5c
• feat: add --search to just log (#11995)

  Summary
  - extend the log client to accept an optional `--search` substring
  filter when querying codex-state logs
  - propagate the filter through `LogQuery` and apply it in
  `push_log_filters` via `INSTR(message, ...)`
  - add an integration test that exercises the new search filtering
  behavior
  
  Testing
  - Not run (not requested)

  jif-oai · 2026-02-17 14:19:52 +00:00
  31d4bfdde0
• Add process_uuid to sqlite logs (#11534)

  ## Summary
  This PR is the first slice of the per-session `/feedback` logging work:
  it adds a process-unique identifier to SQLite log rows.
  
  It does **not** change `/feedback` sourcing behavior yet.
  
  ## Changes
  - Add migration `0009_logs_process_id.sql` to extend `logs` with:
    - `process_uuid TEXT`
    - `idx_logs_process_uuid` index
  - Extend state log models:
    - `LogEntry.process_uuid: Option<String>`
    - `LogRow.process_uuid: Option<String>`
  - Stamp each log row with a stable per-process UUID in the sqlite log
  layer:
    - generated once per process as `pid:<pid>:<uuid>`
  - Update sqlite log insert/query paths to persist and read
  `process_uuid`:
    - `INSERT INTO logs (..., process_uuid, ...)`
    - `SELECT ..., process_uuid, ... FROM logs`
  
  ## Why
  App-server runs many sessions in one process. This change provides a
  process-scoping primitive we need for follow-up `/feedback` work, so
  threadless/process-level logs can be associated with the emitting
  process without mixing across processes.
  
  ## Non-goals in this PR
  - No `/feedback` transport/source changes
  - No attachment size changes
  - No sqlite retention/trim policy changes
  
  ## Testing
  - `just fmt`
  - CI will run the full checks

  Charley Cunningham · 2026-02-14 17:27:22 -08:00
  fce4ad9cf4
• feat: add slug in name (#11739)
  jif-oai · 2026-02-13 15:24:03 +00:00
  db66d827be
• Add cwd to memory files (#11591)

  Add cwd to memory files so that model can deal with multi cwd memory
  better.
  
  ---------
  
  Co-authored-by: jif-oai <jif@openai.com>

  Wendy Jiao · 2026-02-12 17:46:49 +00:00
  88c5ca2573
• nit: upgrade DB version (#11581)
  jif-oai · 2026-02-12 13:16:28 +00:00
  2a409ca67c
• fix: db stuff mem (#11575)

  * Documenting DB functions
  * Fixing 1 nit where stage-2 was sorting the stage 1 in the wrong
  direction
  * Added some tests

  jif-oai · 2026-02-12 12:53:47 +00:00
  19ab038488
• Ensure list_threads drops stale rollout files (#11572)

  Summary
  - trim `state_db::list_threads_db` results to entries whose rollout
  files still exist, logging and recording a discrepancy for dropped rows
  - delete stale metadata rows from the SQLite store so future calls don’t
  surface invalid paths
  - add regression coverage in `recorder.rs` to verify stale DB paths are
  dropped when the file is missing

  jif-oai · 2026-02-12 12:49:31 +00:00
  adad23f743
• feat: mem slash commands (#11569)

  Add 2 slash commands for memories:
  * `/m_drop` delete all the memories
  * `/m_update` update the memories with phase 1 and 2

  jif-oai · 2026-02-12 10:39:43 +00:00
  a0dab25c68
• feat: make sandbox read access configurable with ReadOnlyAccess (#11387)

  `SandboxPolicy::ReadOnly` previously implied broad read access and could
  not express a narrower read surface.
  This change introduces an explicit read-access model so we can support
  user-configurable read restrictions in follow-up work, while preserving
  current behavior today.
  
  It also ensures unsupported backends fail closed for restricted-read
  policies instead of silently granting broader access than intended.
  
  ## What
  
  - Added `ReadOnlyAccess` in protocol with:
    - `Restricted { include_platform_defaults, readable_roots }`
    - `FullAccess`
  - Updated `SandboxPolicy` to carry read-access configuration:
    - `ReadOnly { access: ReadOnlyAccess }`
    - `WorkspaceWrite { ..., read_only_access: ReadOnlyAccess }`
  - Preserved existing behavior by defaulting current construction paths
  to `ReadOnlyAccess::FullAccess`.
  - Threaded the new fields through sandbox policy consumers and call
  sites across `core`, `tui`, `linux-sandbox`, `windows-sandbox`, and
  related tests.
  - Updated Seatbelt policy generation to honor restricted read roots by
  emitting scoped read rules when full read access is not granted.
  - Added fail-closed behavior on Linux and Windows backends when
  restricted read access is requested but not yet implemented there
  (`UnsupportedOperation`).
  - Regenerated app-server protocol schema and TypeScript artifacts,
  including `ReadOnlyAccess`.
  
  ## Compatibility / rollout
  
  - Runtime behavior remains unchanged by default (`FullAccess`).
  - API/schema changes are in place so future config wiring can enable
  restricted read access without another policy-shape migration.

  Michael Bolin · 2026-02-11 18:31:14 -08:00
  abbd74e2be
• feat: new memory prompts (#11439)

  * Update prompt
  * Wire CWD in the prompt
  * Handle the no-output case

  jif-oai · 2026-02-11 13:57:52 +00:00
  f5d4a21098
• fix: flaky test (#11428)

  stage1_concurrent_claims_respect_running_cap was flaky due to SQLite
  lock contention, not cap logic correctness. The claim flow used deferred
  transactions (BEGIN) with read-then-write behavior, which can fail under
  concurrency with SQLITE_BUSY_SNAPSHOT/database is locked when upgrading
  a read transaction to a write transaction. We fixed this by using BEGIN
  IMMEDIATE for stage1 and phase2 claim paths, so lock acquisition happens
  up front and contenders serialize cleanly instead of failing during
  upgrade. After the change, codex-state tests pass and stress reruns of
  the flaky path no longer reproduced the failure.

  jif-oai · 2026-02-11 10:23:18 +00:00
  2c5eeb6b1f
• Disable very flaky tests (#11394)

  Collected from last 20 builds of main in
  https://github.com/openai/codex/commits/main/.

  pakrym-oai · 2026-02-10 18:50:11 -08:00
  bfd4e2112c
• feat: prevent double backfill (#11377)

  ## Summary
  
  Add a DB-backed lease to prevent duplicate `.sqlite` backfill workers
  from running concurrently.
  
  ### What changed
  - Added StateRuntime::try_claim_backfill(lease_seconds) that atomically
  claims backfill only when:
    - backfill is not complete, and
    - no fresh running worker currently owns it.
  - Updated backfill_sessions to use the claim API and exit early when
  another worker already holds the lease.
  - Added runtime tests covering:
    - singleton claim behavior,
    - stale lease takeover,
    - claim blocked after complete.
  - Set backfill lease to 900s in production and 1s in tests.
  
  ### Why
  
  This avoids duplicate backfill work and reduces backfill status churn
  under concurrent startup, while preserving
  current best-effort fallback behavior.

  jif-oai · 2026-02-11 00:24:20 +00:00
  87bbfc50a1
• feat: mem v2 - PR6 (consolidation) (#11374)
  jif-oai · 2026-02-11 00:02:57 +00:00
  674799d356
• feat: mem v2 - PR5 (#11372)
  jif-oai · 2026-02-10 23:22:55 +00:00
  2c9be54c9a
• feat: mem v2 - PR4 (#11369)

  # Memories migration plan (simplified global workflow)
  
  ## Target behavior
  
  - One shared memory root only: `~/.codex/memories/`.
  - No per-cwd memory buckets, no cwd hash handling.
  - Phase 1 candidate rules:
  - Not currently being processed unless the job lease is stale.
  - Rollout updated within the max-age window (currently 30 days).
  - Rollout idle for at least 12 hours (new constant).
  - Global cap: at most 64 stage-1 jobs in `running` state at any time
  (new invariant).
  - Stage-1 model output shape (new):
  - `rollout_slug` (accepted but ignored for now).
  - `rollout_summary`.
  - `raw_memory`.
  - Phase-1 artifacts written under the shared root:
  - `rollout_summaries/<thread_id>.md` for each rollout summary.
  - `raw_memories.md` containing appended/merged raw memory paragraphs.
  - Phase 2 runs one consolidation agent for the shared `memories/`
  directory.
  - Phase-2 lock is DB-backed with 1 hour lease and heartbeat/expiry.
  
  ## Current code map
  
  - Core startup pipeline: `core/src/memories/startup/mod.rs`.
  - Stage-1 request+parse: `core/src/memories/startup/extract.rs`,
  `core/src/memories/stage_one.rs`, templates in
  `core/templates/memories/`.
  - File materialization: `core/src/memories/storage.rs`,
  `core/src/memories/layout.rs`.
  - Scope routing (cwd/user): `core/src/memories/scope.rs`,
  `core/src/memories/startup/mod.rs`.
  - DB job lifecycle and scope queueing: `state/src/runtime/memory.rs`.
  
  ## PR plan
  
  ## PR 1: Correct phase-1 selection invariants (no behavior-breaking
  layout changes yet)
  
  - Add `PHASE_ONE_MIN_ROLLOUT_IDLE_HOURS: i64 = 12` in
  `core/src/memories/mod.rs`.
  - Thread this into `state::claim_stage1_jobs_for_startup(...)`.
  - Enforce idle-time filter in DB selection logic (not only in-memory
  filtering after `scan_limit`) so eligible threads are not starved by
  very recent threads.
  - Enforce global running cap of 64 at claim time in DB logic:
  - Count fresh `memory_stage1` running jobs.
  - Only allow new claims while count < cap.
  - Keep stale-lease takeover behavior intact.
  - Add/adjust tests in `state/src/runtime.rs`:
  - Idle filter inclusion/exclusion around 12h boundary.
  - Global running-cap guarantee.
  - Existing stale/fresh ownership behavior still passes.
  
  Acceptance criteria:
  - Startup never creates more than 64 fresh `memory_stage1` running jobs.
  - Threads updated <12h ago are skipped.
  - Threads older than 30d are skipped.
  
  ## PR 2: Stage-1 output contract + storage artifacts
  (forward-compatible)
  
  - Update parser/types to accept the new structured output while keeping
  backward compatibility:
  - Add `rollout_slug` (optional for now).
  - Add `rollout_summary`.
  - Keep alias support for legacy `summary` and `rawMemory` until prompt
  swap completes.
  - Update stage-1 schema generator in `core/src/memories/stage_one.rs` to
  include the new keys.
  - Update prompt templates:
  - `core/templates/memories/stage_one_system.md`.
  - `core/templates/memories/stage_one_input.md`.
  - Replace storage model in `core/src/memories/storage.rs`:
  - Introduce `rollout_summaries/` directory writer (`<thread_id>.md`
  files).
  - Introduce `raw_memories.md` aggregator writer from DB rows.
  - Keep deterministic rebuild behavior from DB outputs so files can
  always be regenerated.
  - Update consolidation prompt template to reference `rollout_summaries/`
  + `raw_memories.md` inputs.
  
  Acceptance criteria:
  - Stage-1 accepts both old and new output keys during migration.
  - Phase-1 artifacts are generated in new format from DB state.
  - No dependence on per-thread files in `raw_memories/`.
  
  ## PR 3: Remove per-cwd memories and move to one global memory root
  
  - Simplify layout in `core/src/memories/layout.rs`:
  - Single root: `codex_home/memories`.
  - Remove cwd-hash bucket helpers and normalization logic used only for
  memory pathing.
  - Remove scope branching from startup phase-2 dispatch path:
  - No cwd/user mapping in `core/src/memories/startup/mod.rs`.
  - One target root for consolidation.
  - In `state/src/runtime/memory.rs`, stop enqueueing/handling cwd
  consolidation scope.
  - Keep one logical consolidation scope/job key (global/user) to avoid a
  risky schema rewrite in same PR.
  - Add one-time migration helper (core side) to preserve current shared
  memory output:
  - If `~/.codex/memories/user/memory` exists and new root is empty,
  move/copy contents into `~/.codex/memories`.
  - Leave old hashed cwd buckets untouched for now (safe/no-destructive
  migration).
  
  Acceptance criteria:
  - New runs only read/write `~/.codex/memories`.
  - No new cwd-scoped consolidation jobs are enqueued.
  - Existing user-shared memory content is preserved.
  
  ## PR 4: Phase-2 global lock simplification and cleanup
  
  - Replace multi-scope dispatch with a single global consolidation claim
  path:
  - Either reuse jobs table with one fixed key, or add a tiny dedicated
  lock helper; keep 1h lease.
  - Ensure at most one consolidation agent can run at once.
  - Keep heartbeat + stale lock recovery semantics in
  `core/src/memories/startup/watch.rs`.
  - Remove dead scope code and legacy constants no longer used.
  - Update tests:
  - One-agent-at-a-time behavior.
  - Lock expiry allows takeover after stale lease.
  
  Acceptance criteria:
  - Exactly one phase-2 consolidation agent can be active cluster-wide
  (per local DB).
  - Stale lock recovers automatically.
  
  ## PR 5: Final cleanup and docs
  
  - Remove legacy artifacts and references:
  - `raw_memories/` and `memory_summary.md` assumptions from
  prompts/comments/tests.
  - Scope constants for cwd memory pathing in core/state if fully unused.
  - Update docs under `docs/` for memory workflow and directory layout.
  - Add a brief operator note for rollout: compatibility window for old
  stage-1 JSON keys and when to remove aliases.
  
  Acceptance criteria:
  - Code and docs reflect only the simplified global workflow.
  - No stale references to per-cwd memory buckets.
  
  ## Notes on sequencing
  
  - PR 1 is safest first because it improves correctness without changing
  external artifact layout.
  - PR 2 keeps parser compatibility so prompt deployment can happen
  independently.
  - PR 3 and PR 4 split filesystem/scope simplification from locking
  simplification to reduce blast radius.
  - PR 5 is intentionally cleanup-only.

  jif-oai · 2026-02-10 23:10:35 +00:00
  623d3f4071
• feat: mem v2 - PR3 (#11366)

  # Memories migration plan (simplified global workflow)
  
  ## Target behavior
  
  - One shared memory root only: `~/.codex/memories/`.
  - No per-cwd memory buckets, no cwd hash handling.
  - Phase 1 candidate rules:
  - Not currently being processed unless the job lease is stale.
  - Rollout updated within the max-age window (currently 30 days).
  - Rollout idle for at least 12 hours (new constant).
  - Global cap: at most 64 stage-1 jobs in `running` state at any time
  (new invariant).
  - Stage-1 model output shape (new):
  - `rollout_slug` (accepted but ignored for now).
  - `rollout_summary`.
  - `raw_memory`.
  - Phase-1 artifacts written under the shared root:
  - `rollout_summaries/<thread_id>.md` for each rollout summary.
  - `raw_memories.md` containing appended/merged raw memory paragraphs.
  - Phase 2 runs one consolidation agent for the shared `memories/`
  directory.
  - Phase-2 lock is DB-backed with 1 hour lease and heartbeat/expiry.
  
  ## Current code map
  
  - Core startup pipeline: `core/src/memories/startup/mod.rs`.
  - Stage-1 request+parse: `core/src/memories/startup/extract.rs`,
  `core/src/memories/stage_one.rs`, templates in
  `core/templates/memories/`.
  - File materialization: `core/src/memories/storage.rs`,
  `core/src/memories/layout.rs`.
  - Scope routing (cwd/user): `core/src/memories/scope.rs`,
  `core/src/memories/startup/mod.rs`.
  - DB job lifecycle and scope queueing: `state/src/runtime/memory.rs`.
  
  ## PR plan
  
  ## PR 1: Correct phase-1 selection invariants (no behavior-breaking
  layout changes yet)
  
  - Add `PHASE_ONE_MIN_ROLLOUT_IDLE_HOURS: i64 = 12` in
  `core/src/memories/mod.rs`.
  - Thread this into `state::claim_stage1_jobs_for_startup(...)`.
  - Enforce idle-time filter in DB selection logic (not only in-memory
  filtering after `scan_limit`) so eligible threads are not starved by
  very recent threads.
  - Enforce global running cap of 64 at claim time in DB logic:
  - Count fresh `memory_stage1` running jobs.
  - Only allow new claims while count < cap.
  - Keep stale-lease takeover behavior intact.
  - Add/adjust tests in `state/src/runtime.rs`:
  - Idle filter inclusion/exclusion around 12h boundary.
  - Global running-cap guarantee.
  - Existing stale/fresh ownership behavior still passes.
  
  Acceptance criteria:
  - Startup never creates more than 64 fresh `memory_stage1` running jobs.
  - Threads updated <12h ago are skipped.
  - Threads older than 30d are skipped.
  
  ## PR 2: Stage-1 output contract + storage artifacts
  (forward-compatible)
  
  - Update parser/types to accept the new structured output while keeping
  backward compatibility:
  - Add `rollout_slug` (optional for now).
  - Add `rollout_summary`.
  - Keep alias support for legacy `summary` and `rawMemory` until prompt
  swap completes.
  - Update stage-1 schema generator in `core/src/memories/stage_one.rs` to
  include the new keys.
  - Update prompt templates:
  - `core/templates/memories/stage_one_system.md`.
  - `core/templates/memories/stage_one_input.md`.
  - Replace storage model in `core/src/memories/storage.rs`:
  - Introduce `rollout_summaries/` directory writer (`<thread_id>.md`
  files).
  - Introduce `raw_memories.md` aggregator writer from DB rows.
  - Keep deterministic rebuild behavior from DB outputs so files can
  always be regenerated.
  - Update consolidation prompt template to reference `rollout_summaries/`
  + `raw_memories.md` inputs.
  
  Acceptance criteria:
  - Stage-1 accepts both old and new output keys during migration.
  - Phase-1 artifacts are generated in new format from DB state.
  - No dependence on per-thread files in `raw_memories/`.
  
  ## PR 3: Remove per-cwd memories and move to one global memory root
  
  - Simplify layout in `core/src/memories/layout.rs`:
  - Single root: `codex_home/memories`.
  - Remove cwd-hash bucket helpers and normalization logic used only for
  memory pathing.
  - Remove scope branching from startup phase-2 dispatch path:
  - No cwd/user mapping in `core/src/memories/startup/mod.rs`.
  - One target root for consolidation.
  - In `state/src/runtime/memory.rs`, stop enqueueing/handling cwd
  consolidation scope.
  - Keep one logical consolidation scope/job key (global/user) to avoid a
  risky schema rewrite in same PR.
  - Add one-time migration helper (core side) to preserve current shared
  memory output:
  - If `~/.codex/memories/user/memory` exists and new root is empty,
  move/copy contents into `~/.codex/memories`.
  - Leave old hashed cwd buckets untouched for now (safe/no-destructive
  migration).
  
  Acceptance criteria:
  - New runs only read/write `~/.codex/memories`.
  - No new cwd-scoped consolidation jobs are enqueued.
  - Existing user-shared memory content is preserved.
  
  ## PR 4: Phase-2 global lock simplification and cleanup
  
  - Replace multi-scope dispatch with a single global consolidation claim
  path:
  - Either reuse jobs table with one fixed key, or add a tiny dedicated
  lock helper; keep 1h lease.
  - Ensure at most one consolidation agent can run at once.
  - Keep heartbeat + stale lock recovery semantics in
  `core/src/memories/startup/watch.rs`.
  - Remove dead scope code and legacy constants no longer used.
  - Update tests:
  - One-agent-at-a-time behavior.
  - Lock expiry allows takeover after stale lease.
  
  Acceptance criteria:
  - Exactly one phase-2 consolidation agent can be active cluster-wide
  (per local DB).
  - Stale lock recovers automatically.
  
  ## PR 5: Final cleanup and docs
  
  - Remove legacy artifacts and references:
  - `raw_memories/` and `memory_summary.md` assumptions from
  prompts/comments/tests.
  - Scope constants for cwd memory pathing in core/state if fully unused.
  - Update docs under `docs/` for memory workflow and directory layout.
  - Add a brief operator note for rollout: compatibility window for old
  stage-1 JSON keys and when to remove aliases.
  
  Acceptance criteria:
  - Code and docs reflect only the simplified global workflow.
  - No stale references to per-cwd memory buckets.
  
  ## Notes on sequencing
  
  - PR 1 is safest first because it improves correctness without changing
  external artifact layout.
  - PR 2 keeps parser compatibility so prompt deployment can happen
  independently.
  - PR 3 and PR 4 split filesystem/scope simplification from locking
  simplification to reduce blast radius.
  - PR 5 is intentionally cleanup-only.

  jif-oai · 2026-02-10 22:12:50 +00:00
  3419660767
• feat: mem v2 - PR1 (#11364)

  # Memories migration plan (simplified global workflow)
  
  ## Target behavior
  
  - One shared memory root only: `~/.codex/memories/`.
  - No per-cwd memory buckets, no cwd hash handling.
  - Phase 1 candidate rules:
  - Not currently being processed unless the job lease is stale.
  - Rollout updated within the max-age window (currently 30 days).
  - Rollout idle for at least 12 hours (new constant).
  - Global cap: at most 64 stage-1 jobs in `running` state at any time
  (new invariant).
  - Stage-1 model output shape (new):
  - `rollout_slug` (accepted but ignored for now).
  - `rollout_summary`.
  - `raw_memory`.
  - Phase-1 artifacts written under the shared root:
  - `rollout_summaries/<thread_id>.md` for each rollout summary.
  - `raw_memories.md` containing appended/merged raw memory paragraphs.
  - Phase 2 runs one consolidation agent for the shared `memories/`
  directory.
  - Phase-2 lock is DB-backed with 1 hour lease and heartbeat/expiry.
  
  ## Current code map
  
  - Core startup pipeline: `core/src/memories/startup/mod.rs`.
  - Stage-1 request+parse: `core/src/memories/startup/extract.rs`,
  `core/src/memories/stage_one.rs`, templates in
  `core/templates/memories/`.
  - File materialization: `core/src/memories/storage.rs`,
  `core/src/memories/layout.rs`.
  - Scope routing (cwd/user): `core/src/memories/scope.rs`,
  `core/src/memories/startup/mod.rs`.
  - DB job lifecycle and scope queueing: `state/src/runtime/memory.rs`.
  
  ## PR plan
  
  ## PR 1: Correct phase-1 selection invariants (no behavior-breaking
  layout changes yet)
  
  - Add `PHASE_ONE_MIN_ROLLOUT_IDLE_HOURS: i64 = 12` in
  `core/src/memories/mod.rs`.
  - Thread this into `state::claim_stage1_jobs_for_startup(...)`.
  - Enforce idle-time filter in DB selection logic (not only in-memory
  filtering after `scan_limit`) so eligible threads are not starved by
  very recent threads.
  - Enforce global running cap of 64 at claim time in DB logic:
  - Count fresh `memory_stage1` running jobs.
  - Only allow new claims while count < cap.
  - Keep stale-lease takeover behavior intact.
  - Add/adjust tests in `state/src/runtime.rs`:
  - Idle filter inclusion/exclusion around 12h boundary.
  - Global running-cap guarantee.
  - Existing stale/fresh ownership behavior still passes.
  
  Acceptance criteria:
  - Startup never creates more than 64 fresh `memory_stage1` running jobs.
  - Threads updated <12h ago are skipped.
  - Threads older than 30d are skipped.
  
  ## PR 2: Stage-1 output contract + storage artifacts
  (forward-compatible)
  
  - Update parser/types to accept the new structured output while keeping
  backward compatibility:
  - Add `rollout_slug` (optional for now).
  - Add `rollout_summary`.
  - Keep alias support for legacy `summary` and `rawMemory` until prompt
  swap completes.
  - Update stage-1 schema generator in `core/src/memories/stage_one.rs` to
  include the new keys.
  - Update prompt templates:
  - `core/templates/memories/stage_one_system.md`.
  - `core/templates/memories/stage_one_input.md`.
  - Replace storage model in `core/src/memories/storage.rs`:
  - Introduce `rollout_summaries/` directory writer (`<thread_id>.md`
  files).
  - Introduce `raw_memories.md` aggregator writer from DB rows.
  - Keep deterministic rebuild behavior from DB outputs so files can
  always be regenerated.
  - Update consolidation prompt template to reference `rollout_summaries/`
  + `raw_memories.md` inputs.
  
  Acceptance criteria:
  - Stage-1 accepts both old and new output keys during migration.
  - Phase-1 artifacts are generated in new format from DB state.
  - No dependence on per-thread files in `raw_memories/`.
  
  ## PR 3: Remove per-cwd memories and move to one global memory root
  
  - Simplify layout in `core/src/memories/layout.rs`:
  - Single root: `codex_home/memories`.
  - Remove cwd-hash bucket helpers and normalization logic used only for
  memory pathing.
  - Remove scope branching from startup phase-2 dispatch path:
  - No cwd/user mapping in `core/src/memories/startup/mod.rs`.
  - One target root for consolidation.
  - In `state/src/runtime/memory.rs`, stop enqueueing/handling cwd
  consolidation scope.
  - Keep one logical consolidation scope/job key (global/user) to avoid a
  risky schema rewrite in same PR.
  - Add one-time migration helper (core side) to preserve current shared
  memory output:
  - If `~/.codex/memories/user/memory` exists and new root is empty,
  move/copy contents into `~/.codex/memories`.
  - Leave old hashed cwd buckets untouched for now (safe/no-destructive
  migration).
  
  Acceptance criteria:
  - New runs only read/write `~/.codex/memories`.
  - No new cwd-scoped consolidation jobs are enqueued.
  - Existing user-shared memory content is preserved.
  
  ## PR 4: Phase-2 global lock simplification and cleanup
  
  - Replace multi-scope dispatch with a single global consolidation claim
  path:
  - Either reuse jobs table with one fixed key, or add a tiny dedicated
  lock helper; keep 1h lease.
  - Ensure at most one consolidation agent can run at once.
  - Keep heartbeat + stale lock recovery semantics in
  `core/src/memories/startup/watch.rs`.
  - Remove dead scope code and legacy constants no longer used.
  - Update tests:
  - One-agent-at-a-time behavior.
  - Lock expiry allows takeover after stale lease.
  
  Acceptance criteria:
  - Exactly one phase-2 consolidation agent can be active cluster-wide
  (per local DB).
  - Stale lock recovers automatically.
  
  ## PR 5: Final cleanup and docs
  
  - Remove legacy artifacts and references:
  - `raw_memories/` and `memory_summary.md` assumptions from
  prompts/comments/tests.
  - Scope constants for cwd memory pathing in core/state if fully unused.
  - Update docs under `docs/` for memory workflow and directory layout.
  - Add a brief operator note for rollout: compatibility window for old
  stage-1 JSON keys and when to remove aliases.
  
  Acceptance criteria:
  - Code and docs reflect only the simplified global workflow.
  - No stale references to per-cwd memory buckets.
  
  ## Notes on sequencing
  
  - PR 1 is safest first because it improves correctness without changing
  external artifact layout.
  - PR 2 keeps parser compatibility so prompt deployment can happen
  independently.
  - PR 3 and PR 4 split filesystem/scope simplification from locking
  simplification to reduce blast radius.
  - PR 5 is intentionally cleanup-only.

  jif-oai · 2026-02-10 21:29:06 +00:00
  07da740c8a
• chore: unify memory job flow (#11334)
  jif-oai · 2026-02-10 20:26:39 +00:00
  a6e9469fa4
• feat: phase 2 consolidation (#11306)

  Consolidation phase of memories
  
  Cleaning and better handling of concurrency

  jif-oai · 2026-02-10 14:31:16 +00:00
  e57892b211
• feat: align memory phase 1 and make it stronger (#11300)

  ## Align with the new phase-1 design
  
  Basically we know run phase 1 in parallel by considering:
  * Max 64 rollouts
  * Max 1 month old
  * Consider the most recent first
  
  This PR also adds stronger parallelization capabilities by detecting
  stale jobs, retry policies, ownership of computation to prevent double
  computations etc etc

  jif-oai · 2026-02-10 13:42:09 +00:00
  1d5eba0090
• memories: add extraction and prompt module foundation (#11200)

  ## Summary
  - add the new `core/src/memories` module (phase-one parsing, rollout
  filtering, storage, selection, prompts)
  - add Askama-backed memory templates for stage-one input/system and
  consolidation prompts
  - add module tests for parsing, filtering, path bucketing, and summary
  maintenance
  
  ## Testing
  - just fmt
  - cargo test -p codex-core --lib memories::

  jif-oai · 2026-02-10 10:10:24 +00:00
  6049ff02a0
• state: add memory consolidation lock primitives (#11199)

  ## Summary
  - add a migration for memory_consolidation_locks
  - add acquire/release lock primitives to codex-state runtime
  - add core/state_db wrappers and cwd normalization for memory queries
  and lock keys
  
  ## Testing
  - cargo test -p codex-state memory_consolidation_lock_
  - cargo test -p codex-core --lib state_db::

  jif-oai · 2026-02-09 21:04:20 +00:00
  74ecd6e3b2
• feat: backfill async again (#10894)
  jif-oai · 2026-02-06 15:41:52 +01:00
  3800173459
• Leverage state DB metadata for thread summaries (#10621)

  Summary:
  - read conversation summaries and cwd info from the state DB when
  possible so we no longer rely on rollout files for metadata and avoid
  extra I/O
  - persist CLI version in thread metadata, surface it through summary
  builders, and add the necessary DB migration hooks
  - simplify thread listing by using enriched state DB data directly
  rather than reading rollout heads
  
  Testing:
  - Not run (not requested)

  jif-oai · 2026-02-05 16:39:11 +00:00
  9ee746afd6
• nit: add DB version is discrepancy recording (#10762)
  jif-oai · 2026-02-05 16:24:18 +00:00
  68e82e5dc9
• feat: resumable backfill (#10745)

  ## Summary
  
  This PR makes SQLite rollout backfill resumable and repeatable instead
  of one-shot-on-db-create.
  
  ## What changed
  
  - Added a persisted backfill state table:
    - state/migrations/0008_backfill_state.sql
  - Tracks status (pending|running|complete), last_watermark, and
  last_success_at.
  - Added backfill state model/types in codex-state:
    - BackfillState, BackfillStatus (state/src/model/backfill_state.rs)
  - Added runtime APIs to manage backfill lifecycle/progress:
    - get_backfill_state
    - mark_backfill_running
    - checkpoint_backfill
    - mark_backfill_complete
  - Updated core startup behavior:
  - Backfill now runs whenever state is not Complete (not only when DB
  file is newly created).
  - Reworked backfill execution:
  - Collect rollout files, derive deterministic watermark per path, sort,
  resume from last_watermark.
  - Process in batches (BACKFILL_BATCH_SIZE = 200), checkpoint after each
  batch.
    - Mark complete with last_success_at at the end.
  
  ## Why
  
  Previous behavior could leave users permanently partially backfilled if
  the process exited during initial async backfill. This change allows
  safe continuation across restarts and avoids restarting from scratch.

  jif-oai · 2026-02-05 14:34:34 +00:00
  4033f905c6
• feat: add phase 1 mem db (#10634)

  - Schema: thread_id (PK, FK to threads.id with cascade delete),
  trace_summary, memory_summary, updated_at.
  - Migration: creates the table and an index on (updated_at DESC,
  thread_id DESC) for efficient recent-first reads.
    - Runtime API (DB-only):
        - `get_thread_memory(thread_id)`: fetch one memory row.
  - `upsert_thread_memory(thread_id, trace_summary, memory_summary)`:
  insert/update by thread id and always advance updated_at.
  - `get_last_n_thread_memories_for_cwd(cwd, n)`: join thread_memory with
  threads and return newest n rows for an exact cwd match.
  - Model layer: introduced ThreadMemory and row conversion types to keep
  query decoding typed and consistent with existing state models.

  jif-oai · 2026-02-04 21:38:39 +00:00
  4922b3e571