25 Commits

• feat: change multi-agent to use path-like system instead of uuids (#15313)

  This PR add an URI-based system to reference agents within a tree. This
  comes from a sync between research and engineering.
  
  The main agent (the one manually spawned by a user) is always called
  `/root`. Any sub-agent spawned by it will be `/root/agent_1` for example
  where `agent_1` is chosen by the model.
  
  Any agent can contact any agents using the path.
  
  Paths can be used either in absolute or relative to the calling agents
  
  Resume is not supported for now on this new path

  jif-oai · 2026-03-20 18:23:48 +00:00
  79ad7b247b
• feat: add graph representation of agent network (#15056)

  Add a representation of the agent graph. This is now used for:
  * Cascade close agents (when I close a parent, it close the kids)
  * Cascade resume (oposite)
  
  Later, this will also be used for post-compaction stuffing of the
  context
  
  Direct fix for: https://github.com/openai/codex/issues/14458

  jif-oai · 2026-03-19 10:21:25 +00:00
  70cdb17703
• Feat: CXA-1831 Persist latest model and reasoning effort in sqlite (#14859)

  ### Summary
  The goal is for us to get the latest turn model and reasoning effort on
  thread/resume is no override is provided on the thread/resume func call.
  This is the part 1 which we write the model and reasoning effort for a
  thread to the sqlite db and there will be a followup PR to consume the
  two new fields on thread/resume.
  
  [part 2 PR is currently WIP](https://github.com/openai/codex/pull/14888)
  and this one can be merged independently.

  Shijie Rao · 2026-03-17 10:14:34 -07:00
  8e258eb3f5
• dynamic tool calls: add param exposeToContext to optionally hide tool (#14501)

  This extends dynamic_tool_calls to allow us to hide a tool from the
  model context but still use it as part of the general tool calling
  runtime (for ex from js_repl/code_mode)

  Channing Conger · 2026-03-14 01:58:43 -07:00
  70eddad6b0
• feat: memories forgetting (#12900)

  Add diff based memory forgetting

  jif-oai · 2026-02-26 13:19:57 +00:00
  382fa338b3
• nit: migration (#12772)
  jif-oai · 2026-02-25 13:56:52 +00:00
  8d49e0d0c4
• feat: record memory usage (#12761)
  jif-oai · 2026-02-25 13:48:40 +00:00
  e4bfa763f6
• Agent jobs (spawn_agents_on_csv) + progress UI (#10935)

  ## Summary
  - Add agent job support: spawn a batch of sub-agents from CSV, auto-run,
  auto-export, and store results in SQLite.
  - Simplify workflow: remove run/resume/get-status/export tools; spawn is
  deterministic and completes in one call.
  - Improve exec UX: stable, single-line progress bar with ETA; suppress
  sub-agent chatter in exec.
  
  ## Why
  Enables map-reduce style workflows over arbitrarily large repos using
  the existing Codex orchestrator. This addresses review feedback about
  overly complex job controls and non-deterministic monitoring.
  
  ## Demo (progress bar)
  ```
  ./codex-rs/target/debug/codex exec \
    --enable collab \
    --enable sqlite \
    --full-auto \
    --progress-cursor \
    -c agents.max_threads=16 \
    -C /Users/daveaitel/code/codex \
    - <<'PROMPT'
  Create /tmp/agent_job_progress_demo.csv with columns: path,area and 30 rows:
  path = item-01..item-30, area = test.
  
  Then call spawn_agents_on_csv with:
  - csv_path: /tmp/agent_job_progress_demo.csv
  - instruction: "Run `python - <<'PY'` to sleep a random 0.3–1.2s, then output JSON with keys: path, score (int). Set score = 1."
  - output_csv_path: /tmp/agent_job_progress_demo_out.csv
  PROMPT
  ```
  
  ## Review feedback addressed
  - Auto-start jobs on spawn; removed run/resume/status/export tools.
  - Auto-export on success.
  - More descriptive tool spec + clearer prompts.
  - Avoid deadlocks on spawn failure; pending/running handled safely.
  - Progress bar no longer scrolls; stable single-line redraw.
  
  ## Tests
  - `cd codex-rs && cargo test -p codex-exec`
  - `cd codex-rs && cargo build -p codex-cli`

  daveaitel-openai · 2026-02-24 21:00:19 +00:00
  dcab40123f
• feat: add nick name to sub-agents (#12320)

  Adding random nick name to sub-agents. Used for UX
  
  At the same time, also storing and wiring the role of the sub-agent

  jif-oai · 2026-02-20 14:39:49 +00:00
  0f9eed3a6f
• state: enforce 10 MiB log caps for thread and threadless process logs (#12038)

  ## Summary
  - enforce a 10 MiB cap per `thread_id` in state log storage
  - enforce a 10 MiB cap per `process_uuid` for threadless (`thread_id IS
  NULL`) logs
  - scope pruning to only keys affected by the current insert batch
  - add a cheap per-key `SUM(...)` precheck so windowed prune queries only
  run for keys that are currently over the cap
  - add SQLite indexes used by the pruning queries
  - add focused runtime tests covering both pruning behaviors
  
  ## Why
  This keeps log growth bounded by the intended partition semantics while
  preserving a small, readable implementation localized to the existing
  insert path.
  
  ## Local Latency Snapshot (No Truncation-Pressure Run)
  Collected from session `019c734f-1d16-7002-9e00-c966c9fbbcae` using
  local-only (uncommitted) instrumentation, while not specifically
  benchmarking the truncation-heavy regime.
  
  ### Percentiles By Query (ms)
  | query | count | p50 | p90 | p95 | p99 | max |
  |---|---:|---:|---:|---:|---:|---:|
  | `insert_logs.insert_batch` | 110 | 0.332 | 0.999 | 1.811 | 2.978 |
  3.493 |
  | `insert_logs.precheck.process` | 106 | 0.074 | 0.152 | 0.206 | 0.258 |
  0.426 |
  | `insert_logs.precheck.thread` | 73 | 0.118 | 0.206 | 0.253 | 1.025 |
  1.025 |
  | `insert_logs.prune.process` | 58 | 0.291 | 0.576 | 0.607 | 1.088 |
  1.088 |
  | `insert_logs.prune.thread` | 44 | 0.318 | 0.467 | 0.728 | 0.797 |
  0.797 |
  | `insert_logs.prune_total` | 110 | 0.488 | 0.976 | 1.237 | 1.593 |
  1.684 |
  | `insert_logs.total` | 110 | 1.315 | 2.889 | 3.623 | 5.739 | 5.961 |
  | `insert_logs.tx_begin` | 110 | 0.133 | 0.235 | 0.282 | 0.412 | 0.546 |
  | `insert_logs.tx_commit` | 110 | 0.259 | 0.689 | 0.772 | 1.065 | 1.080
  |
  
  ### `insert_logs.total` Histogram (ms)
  | bucket | count |
  |---|---:|
  | `<= 0.100` | 0 |
  | `<= 0.250` | 0 |
  | `<= 0.500` | 7 |
  | `<= 1.000` | 33 |
  | `<= 2.000` | 40 |
  | `<= 5.000` | 28 |
  | `<= 10.000` | 2 |
  | `<= 20.000` | 0 |
  | `<= 50.000` | 0 |
  | `<= 100.000` | 0 |
  | `> 100.000` | 0 |
  
  ## Local Latency Snapshot (Truncation-Heavy / Cap-Hit Regime)
  Collected from a run where cap-hit behavior was frequent (`135/180`
  insert calls), using local-only (uncommitted) instrumentation and a
  temporary local cap of `10_000` bytes for stress testing (not the merged
  `10 MiB` cap).
  
  ### Percentiles By Query (ms)
  | query | count | p50 | p90 | p95 | p99 | max |
  |---|---:|---:|---:|---:|---:|---:|
  | `insert_logs.insert_batch` | 180 | 0.524 | 1.645 | 2.163 | 3.424 |
  3.777 |
  | `insert_logs.precheck.process` | 171 | 0.086 | 0.235 | 0.373 | 0.758 |
  1.147 |
  | `insert_logs.precheck.thread` | 100 | 0.105 | 0.251 | 0.291 | 1.176 |
  1.622 |
  | `insert_logs.prune.process` | 109 | 0.386 | 0.839 | 1.146 | 1.548 |
  2.588 |
  | `insert_logs.prune.thread` | 56 | 0.253 | 0.550 | 1.148 | 2.484 |
  2.484 |
  | `insert_logs.prune_total` | 180 | 0.511 | 1.221 | 1.695 | 4.548 |
  5.512 |
  | `insert_logs.total` | 180 | 1.631 | 3.902 | 5.103 | 8.901 | 9.095 |
  | `insert_logs.total_cap_hit` | 135 | 1.876 | 4.501 | 5.547 | 8.902 |
  9.096 |
  | `insert_logs.total_no_cap_hit` | 45 | 0.520 | 1.700 | 2.079 | 3.294 |
  3.294 |
  | `insert_logs.tx_begin` | 180 | 0.109 | 0.253 | 0.287 | 1.088 | 1.406 |
  | `insert_logs.tx_commit` | 180 | 0.267 | 0.813 | 1.170 | 2.497 | 2.574
  |
  
  ### `insert_logs.total` Histogram (ms)
  | bucket | count |
  |---|---:|
  | `<= 0.100` | 0 |
  | `<= 0.250` | 0 |
  | `<= 0.500` | 16 |
  | `<= 1.000` | 39 |
  | `<= 2.000` | 60 |
  | `<= 5.000` | 54 |
  | `<= 10.000` | 11 |
  | `<= 20.000` | 0 |
  | `<= 50.000` | 0 |
  | `<= 100.000` | 0 |
  | `> 100.000` | 0 |
  
  ### `insert_logs.total` Histogram When Cap Was Hit (ms)
  | bucket | count |
  |---|---:|
  | `<= 0.100` | 0 |
  | `<= 0.250` | 0 |
  | `<= 0.500` | 0 |
  | `<= 1.000` | 22 |
  | `<= 2.000` | 51 |
  | `<= 5.000` | 51 |
  | `<= 10.000` | 11 |
  | `<= 20.000` | 0 |
  | `<= 50.000` | 0 |
  | `<= 100.000` | 0 |
  | `> 100.000` | 0 |
  
  ### Performance Takeaways
  - Even in a cap-hit-heavy run (`75%` cap-hit calls), `insert_logs.total`
  stays sub-10ms at p99 (`8.901ms`) and max (`9.095ms`).
  - Calls that did **not** hit the cap are materially cheaper
  (`insert_logs.total_no_cap_hit` p95 `2.079ms`) than cap-hit calls
  (`insert_logs.total_cap_hit` p95 `5.547ms`).
  - Compared to the earlier non-truncation-pressure run, overall
  `insert_logs.total` rose from p95 `3.623ms` to p95 `5.103ms`
  (+`1.48ms`), indicating bounded overhead when pruning is active.
  - This truncation-heavy run used an intentionally low local cap for
  stress testing; with the real 10 MiB cap, cap-hit frequency should be
  much lower in normal sessions.
  
  ## Testing
  - `just fmt` (in `codex-rs`)
  - `cargo test -p codex-state` (in `codex-rs`)

  Charley Cunningham · 2026-02-18 17:08:08 -08:00
  7f3dbaeb25
• Add process_uuid to sqlite logs (#11534)

  ## Summary
  This PR is the first slice of the per-session `/feedback` logging work:
  it adds a process-unique identifier to SQLite log rows.
  
  It does **not** change `/feedback` sourcing behavior yet.
  
  ## Changes
  - Add migration `0009_logs_process_id.sql` to extend `logs` with:
    - `process_uuid TEXT`
    - `idx_logs_process_uuid` index
  - Extend state log models:
    - `LogEntry.process_uuid: Option<String>`
    - `LogRow.process_uuid: Option<String>`
  - Stamp each log row with a stable per-process UUID in the sqlite log
  layer:
    - generated once per process as `pid:<pid>:<uuid>`
  - Update sqlite log insert/query paths to persist and read
  `process_uuid`:
    - `INSERT INTO logs (..., process_uuid, ...)`
    - `SELECT ..., process_uuid, ... FROM logs`
  
  ## Why
  App-server runs many sessions in one process. This change provides a
  process-scoping primitive we need for follow-up `/feedback` work, so
  threadless/process-level logs can be associated with the emitting
  process without mixing across processes.
  
  ## Non-goals in this PR
  - No `/feedback` transport/source changes
  - No attachment size changes
  - No sqlite retention/trim policy changes
  
  ## Testing
  - `just fmt`
  - CI will run the full checks

  Charley Cunningham · 2026-02-14 17:27:22 -08:00
  fce4ad9cf4
• feat: add slug in name (#11739)
  jif-oai · 2026-02-13 15:24:03 +00:00
  db66d827be
• feat: mem v2 - PR5 (#11372)
  jif-oai · 2026-02-10 23:22:55 +00:00
  2c9be54c9a
• chore: unify memory job flow (#11334)
  jif-oai · 2026-02-10 20:26:39 +00:00
  a6e9469fa4
• feat: align memory phase 1 and make it stronger (#11300)

  ## Align with the new phase-1 design
  
  Basically we know run phase 1 in parallel by considering:
  * Max 64 rollouts
  * Max 1 month old
  * Consider the most recent first
  
  This PR also adds stronger parallelization capabilities by detecting
  stale jobs, retry policies, ownership of computation to prevent double
  computations etc etc

  jif-oai · 2026-02-10 13:42:09 +00:00
  1d5eba0090
• state: add memory consolidation lock primitives (#11199)

  ## Summary
  - add a migration for memory_consolidation_locks
  - add acquire/release lock primitives to codex-state runtime
  - add core/state_db wrappers and cwd normalization for memory queries
  and lock keys
  
  ## Testing
  - cargo test -p codex-state memory_consolidation_lock_
  - cargo test -p codex-core --lib state_db::

  jif-oai · 2026-02-09 21:04:20 +00:00
  74ecd6e3b2
• Leverage state DB metadata for thread summaries (#10621)

  Summary:
  - read conversation summaries and cwd info from the state DB when
  possible so we no longer rely on rollout files for metadata and avoid
  extra I/O
  - persist CLI version in thread metadata, surface it through summary
  builders, and add the necessary DB migration hooks
  - simplify thread listing by using enriched state DB data directly
  rather than reading rollout heads
  
  Testing:
  - Not run (not requested)

  jif-oai · 2026-02-05 16:39:11 +00:00
  9ee746afd6
• feat: resumable backfill (#10745)

  ## Summary
  
  This PR makes SQLite rollout backfill resumable and repeatable instead
  of one-shot-on-db-create.
  
  ## What changed
  
  - Added a persisted backfill state table:
    - state/migrations/0008_backfill_state.sql
  - Tracks status (pending|running|complete), last_watermark, and
  last_success_at.
  - Added backfill state model/types in codex-state:
    - BackfillState, BackfillStatus (state/src/model/backfill_state.rs)
  - Added runtime APIs to manage backfill lifecycle/progress:
    - get_backfill_state
    - mark_backfill_running
    - checkpoint_backfill
    - mark_backfill_complete
  - Updated core startup behavior:
  - Backfill now runs whenever state is not Complete (not only when DB
  file is newly created).
  - Reworked backfill execution:
  - Collect rollout files, derive deterministic watermark per path, sort,
  resume from last_watermark.
  - Process in batches (BACKFILL_BATCH_SIZE = 200), checkpoint after each
  batch.
    - Mark complete with last_success_at at the end.
  
  ## Why
  
  Previous behavior could leave users permanently partially backfilled if
  the process exited during initial async backfill. This change allows
  safe continuation across restarts and avoids restarting from scratch.

  jif-oai · 2026-02-05 14:34:34 +00:00
  4033f905c6
• feat: add phase 1 mem db (#10634)

  - Schema: thread_id (PK, FK to threads.id with cascade delete),
  trace_summary, memory_summary, updated_at.
  - Migration: creates the table and an index on (updated_at DESC,
  thread_id DESC) for efficient recent-first reads.
    - Runtime API (DB-only):
        - `get_thread_memory(thread_id)`: fetch one memory row.
  - `upsert_thread_memory(thread_id, trace_summary, memory_summary)`:
  insert/update by thread id and always advance updated_at.
  - `get_last_n_thread_memories_for_cwd(cwd, n)`: join thread_memory with
  threads and return newest n rows for an exact cwd match.
  - Model layer: introduced ThreadMemory and row conversion types to keep
  query decoding typed and consistent with existing state models.

  jif-oai · 2026-02-04 21:38:39 +00:00
  4922b3e571
• [feat] persist thread_dynamic_tools in db (#10252)

  Persist thread_dynamic_tools in sqlite and read first from it. Fall back
  to rollout files if it's not found. Persist dynamic tools to both sqlite
  and rollout files.
  
  Saw that new sessions get populated to db correctly & old sessions get
  backfilled correctly at startup:
  ```
  celia@com-92114 codex-rs % sqlite3 ~/.codex/state.sqlite \      "select thread_id, position,name,description,input_schema from thread_dynamic_tools;"
  019c0cad-ec0d-74b2-a787-e8b33a349117|0|geo_lookup|lookup a city|{"properties":{"city":{"type":"string"}},"required":["city"],"type":"object"}
  ....
  019c10ca-aa4b-7620-ae40-c0919fbd7ea7|0|geo_lookup|lookup a city|{"properties":{"city":{"type":"string"}},"required":["city"],"type":"object"}
  ```

  Celia Chen · 2026-02-03 00:06:44 +00:00
  fb2df99cf1
• chore: unify log queries (#10152)

  Unify log queries to only have SQLX code in the runtime and use it for
  both the log client and for tests

  jif-oai · 2026-01-29 16:28:15 +00:00
  e6c4f548ab
• feat: adding thread ID to logs + filter in the client (#10150)
  jif-oai · 2026-01-29 16:53:30 +01:00
  89c5f3c4d4
• chore: improve client (#10149)

  <img width="883" height="84" alt="Screenshot 2026-01-29 at 11 13 12"
  src="https://github.com/user-attachments/assets/090a2fec-94ed-4c0f-aee5-1653ed8b1439"
  />

  jif-oai · 2026-01-29 11:25:22 +01:00
  4ba911d48c
• feat: add log db (#10086)

  Add a log DB. The goal is just to store our logs in a `.sqlite` DB to
  make it easier to crawl them and drop the oldest ones.

  jif-oai · 2026-01-29 10:23:03 +01:00
  780482da84
• feat: sqlite 1 (#10004)

  Add a `.sqlite` database to be used to store rollout metatdata (and
  later logs)
  This PR is phase 1:
  * Add the database and the required infrastructure
  * Add a backfill of the database
  * Persist the newly created rollout both in files and in the DB
  * When we need to get metadata or a rollout, consider the `JSONL` as the
  source of truth but compare the results with the DB and show any errors

  jif-oai · 2026-01-28 15:29:14 +01:00
  3878c3dc7c