27 Commits

• Add thread metadata update endpoint to app server (#13280)

  ## Summary
  - add the v2 `thread/metadata/update` API, including
  protocol/schema/TypeScript exports and app-server docs
  - patch stored thread `gitInfo` in sqlite without resuming the thread,
  with validation plus support for explicit `null` clears
  - repair missing sqlite thread rows from rollout data before patching,
  and make those repairs safe by inserting only when absent and updating
  only git columns so newer metadata is not clobbered
  - keep sqlite authoritative for mutable thread git metadata by
  preserving existing sqlite git fields during reconcile/backfill and only
  using rollout `SessionMeta` git fields to fill gaps
  - add regression coverage for the endpoint, repair paths, concurrent
  sqlite writes, clearing git fields, and rollout/backfill reconciliation
  - fix the login server shutdown race so cancelling before the waiter
  starts still terminates `block_until_done()` correctly
  
  ## Testing
  - `cargo test -p codex-state
  apply_rollout_items_preserves_existing_git_branch_and_fills_missing_git_fields`
  - `cargo test -p codex-state
  update_thread_git_info_preserves_newer_non_git_metadata`
  - `cargo test -p codex-core
  backfill_sessions_preserves_existing_git_branch_and_fills_missing_git_fields`
  - `cargo test -p codex-app-server thread_metadata_update`
  - `cargo test`
  - currently fails in existing `codex-core` grep-files tests with
  `unsupported call: grep_files`:
      - `suite::grep_files::grep_files_tool_collects_matches`
      - `suite::grep_files::grep_files_tool_reports_empty_results`

  joeytrasatti-openai · 2026-03-03 15:56:11 -08:00
  935754baa3
• fix: db windows path (#13336)
  jif-oai · 2026-03-03 09:50:52 +00:00
  938c6dd388
• feat: polluted memories (#13008)

  Add a feature flag to disable memory creation for "polluted"

  jif-oai · 2026-03-02 11:57:32 +00:00
  b649953845
• feat: gen memories config (#12999)
  jif-oai · 2026-02-27 12:38:47 +01:00
  bbd237348d
• feat: add search term to thread list (#12578)

  Add `searchTerm` to `thread/list` that will search for a match in the
  titles (the condition being `searchTerm` $$\in$$ `title`)

  jif-oai · 2026-02-25 09:59:41 +00:00
  f46b767b7e
• Agent jobs (spawn_agents_on_csv) + progress UI (#10935)

  ## Summary
  - Add agent job support: spawn a batch of sub-agents from CSV, auto-run,
  auto-export, and store results in SQLite.
  - Simplify workflow: remove run/resume/get-status/export tools; spawn is
  deterministic and completes in one call.
  - Improve exec UX: stable, single-line progress bar with ETA; suppress
  sub-agent chatter in exec.
  
  ## Why
  Enables map-reduce style workflows over arbitrarily large repos using
  the existing Codex orchestrator. This addresses review feedback about
  overly complex job controls and non-deterministic monitoring.
  
  ## Demo (progress bar)
  ```
  ./codex-rs/target/debug/codex exec \
    --enable collab \
    --enable sqlite \
    --full-auto \
    --progress-cursor \
    -c agents.max_threads=16 \
    -C /Users/daveaitel/code/codex \
    - <<'PROMPT'
  Create /tmp/agent_job_progress_demo.csv with columns: path,area and 30 rows:
  path = item-01..item-30, area = test.
  
  Then call spawn_agents_on_csv with:
  - csv_path: /tmp/agent_job_progress_demo.csv
  - instruction: "Run `python - <<'PY'` to sleep a random 0.3–1.2s, then output JSON with keys: path, score (int). Set score = 1."
  - output_csv_path: /tmp/agent_job_progress_demo_out.csv
  PROMPT
  ```
  
  ## Review feedback addressed
  - Auto-start jobs on spawn; removed run/resume/status/export tools.
  - Auto-export on success.
  - More descriptive tool spec + clearer prompts.
  - Avoid deadlocks on spawn failure; pending/running handled safely.
  - Progress bar no longer scrolls; stable single-line redraw.
  
  ## Tests
  - `cd codex-rs && cargo test -p codex-exec`
  - `cd codex-rs && cargo build -p codex-cli`

  daveaitel-openai · 2026-02-24 21:00:19 +00:00
  dcab40123f
• Ensure list_threads drops stale rollout files (#11572)

  Summary
  - trim `state_db::list_threads_db` results to entries whose rollout
  files still exist, logging and recording a discrepancy for dropped rows
  - delete stale metadata rows from the SQLite store so future calls don’t
  surface invalid paths
  - add regression coverage in `recorder.rs` to verify stale DB paths are
  dropped when the file is missing

  jif-oai · 2026-02-12 12:49:31 +00:00
  adad23f743
• feat: improve thread listing (#11429)

  Improve listing by doing:
  1. List using the rollout file system
  2. Upsert the result in the DB (if present)
  3. Return the result of a DB listing
  4. Fallback on the result of 1 
  
  + some metrics on top of this

  jif-oai · 2026-02-11 11:22:05 +00:00
  3d0ead8db8
• chore: unify memory job flow (#11334)
  jif-oai · 2026-02-10 20:26:39 +00:00
  a6e9469fa4
• feat: phase 2 consolidation (#11306)

  Consolidation phase of memories
  
  Cleaning and better handling of concurrency

  jif-oai · 2026-02-10 14:31:16 +00:00
  e57892b211
• memories: add extraction and prompt module foundation (#11200)

  ## Summary
  - add the new `core/src/memories` module (phase-one parsing, rollout
  filtering, storage, selection, prompts)
  - add Askama-backed memory templates for stage-one input/system and
  consolidation prompts
  - add module tests for parsing, filtering, path bucketing, and summary
  maintenance
  
  ## Testing
  - just fmt
  - cargo test -p codex-core --lib memories::

  jif-oai · 2026-02-10 10:10:24 +00:00
  6049ff02a0
• state: add memory consolidation lock primitives (#11199)

  ## Summary
  - add a migration for memory_consolidation_locks
  - add acquire/release lock primitives to codex-state runtime
  - add core/state_db wrappers and cwd normalization for memory queries
  and lock keys
  
  ## Testing
  - cargo test -p codex-state memory_consolidation_lock_
  - cargo test -p codex-core --lib state_db::

  jif-oai · 2026-02-09 21:04:20 +00:00
  74ecd6e3b2
• feat: backfill async again (#10894)
  jif-oai · 2026-02-06 15:41:52 +01:00
  3800173459
• feat: wait for backfill to be ready (#10790)
  jif-oai · 2026-02-05 20:45:16 +00:00
  428a9f6035
• nit: add DB version is discrepancy recording (#10762)
  jif-oai · 2026-02-05 16:24:18 +00:00
  68e82e5dc9
• feat: repair DB in case of missing lines (#10751)
  jif-oai · 2026-02-05 16:21:49 +00:00
  901215e310
• feat: resumable backfill (#10745)

  ## Summary
  
  This PR makes SQLite rollout backfill resumable and repeatable instead
  of one-shot-on-db-create.
  
  ## What changed
  
  - Added a persisted backfill state table:
    - state/migrations/0008_backfill_state.sql
  - Tracks status (pending|running|complete), last_watermark, and
  last_success_at.
  - Added backfill state model/types in codex-state:
    - BackfillState, BackfillStatus (state/src/model/backfill_state.rs)
  - Added runtime APIs to manage backfill lifecycle/progress:
    - get_backfill_state
    - mark_backfill_running
    - checkpoint_backfill
    - mark_backfill_complete
  - Updated core startup behavior:
  - Backfill now runs whenever state is not Complete (not only when DB
  file is newly created).
  - Reworked backfill execution:
  - Collect rollout files, derive deterministic watermark per path, sort,
  resume from last_watermark.
  - Process in batches (BACKFILL_BATCH_SIZE = 200), checkpoint after each
  batch.
    - Mark complete with last_success_at at the end.
  
  ## Why
  
  Previous behavior could leave users permanently partially backfilled if
  the process exited during initial async backfill. This change allows
  safe continuation across restarts and avoids restarting from scratch.

  jif-oai · 2026-02-05 14:34:34 +00:00
  4033f905c6
• feat: add phase 1 mem db (#10634)

  - Schema: thread_id (PK, FK to threads.id with cascade delete),
  trace_summary, memory_summary, updated_at.
  - Migration: creates the table and an index on (updated_at DESC,
  thread_id DESC) for efficient recent-first reads.
    - Runtime API (DB-only):
        - `get_thread_memory(thread_id)`: fetch one memory row.
  - `upsert_thread_memory(thread_id, trace_summary, memory_summary)`:
  insert/update by thread id and always advance updated_at.
  - `get_last_n_thread_memories_for_cwd(cwd, n)`: join thread_memory with
  threads and return newest n rows for an exact cwd match.
  - Model layer: introduced ThreadMemory and row conversion types to keep
  query decoding typed and consistent with existing state models.

  jif-oai · 2026-02-04 21:38:39 +00:00
  4922b3e571
• Migrate state DB path helpers to versioned filename (#10623)

  Summary
  - add versioned state sqlite filename helpers and re-export them from
  the state crate
  - remove legacy state files when initializing the runtime and update
  consumers/tests to use the new helpers
  - tweak logs client description and database resolution to match the new
  path

  jif-oai · 2026-02-04 14:31:12 +00:00
  583e5d4f41
• Prefer state DB thread listings before filesystem (#10544)

  Summary
  - add Cursor/ThreadsPage conversions so state DB listings can be mapped
  back into the rollout list model
  - make recorder list helpers query the state DB first (archived flag
  included) and only fall back to file traversal if needed, along with
  populating head bytes lazily
  - add extensive tests to ensure the DB path is honored for active and
  archived threads and that the fallback works
  
  Testing
  - Not run (not requested)
  
  <img width="1196" height="693" alt="Screenshot 2026-02-03 at 20 42 33"
  src="https://github.com/user-attachments/assets/826b3c7a-ef11-4b27-802a-3c343695794a"
  />

  jif-oai · 2026-02-04 09:27:24 +00:00
  100eb6e6f0
• [feat] persist thread_dynamic_tools in db (#10252)

  Persist thread_dynamic_tools in sqlite and read first from it. Fall back
  to rollout files if it's not found. Persist dynamic tools to both sqlite
  and rollout files.
  
  Saw that new sessions get populated to db correctly & old sessions get
  backfilled correctly at startup:
  ```
  celia@com-92114 codex-rs % sqlite3 ~/.codex/state.sqlite \      "select thread_id, position,name,description,input_schema from thread_dynamic_tools;"
  019c0cad-ec0d-74b2-a787-e8b33a349117|0|geo_lookup|lookup a city|{"properties":{"city":{"type":"string"}},"required":["city"],"type":"object"}
  ....
  019c10ca-aa4b-7620-ae40-c0919fbd7ea7|0|geo_lookup|lookup a city|{"properties":{"city":{"type":"string"}},"required":["city"],"type":"object"}
  ```

  Celia Chen · 2026-02-03 00:06:44 +00:00
  fb2df99cf1
• fix: thread listing (#10383)
  jif-oai · 2026-02-02 12:52:49 +00:00
  e9a774e7ae
• chore: do not clean the DB anymore (#10232)
  jif-oai · 2026-01-30 18:23:00 +01:00
  887bec0dee
• chore: unify metric (#10220)
  jif-oai · 2026-01-30 11:13:43 +01:00
  25ad414680
• feat: async backfill (#10089)
  jif-oai · 2026-01-29 09:57:50 +00:00
  714dc8d8bd
• feat: add log db (#10086)

  Add a log DB. The goal is just to store our logs in a `.sqlite` DB to
  make it easier to crawl them and drop the oldest ones.

  jif-oai · 2026-01-29 10:23:03 +01:00
  780482da84
• feat: sqlite 1 (#10004)

  Add a `.sqlite` database to be used to store rollout metatdata (and
  later logs)
  This PR is phase 1:
  * Add the database and the required infrastructure
  * Add a backfill of the database
  * Persist the newly created rollout both in files and in the DB
  * When we need to get metadata or a rollout, consider the `JSONL` as the
  source of truth but compare the results with the DB and show any errors

  jif-oai · 2026-01-28 15:29:14 +01:00
  3878c3dc7c