27 Commits

• feat: use git-backed workspace diffs for memory consolidation (#18982)

  ## Why
  
  This PR make the `morpheus` agent (memory phase 2) use a git diff to
  start it's consolidation. The workflow is the following:
  1. The agent acquire a lock
  2. If `.codex/memories` does not exist or is not a git root, initialize
  everything (and make a first empty commit)
  3. Update `raw_memories.md` and `rollout_summaries/` as before.
  Basically we select max N phase 1 memories based on a given policy
  4. We use git (`gix`) to get a diff between the current state of
  `.codex/memories` and the last commit.
  5. Dump the diff in `phase2_workspace_diff.md`
  6. Spawn `morpheus` and point it to `phase2_workspace_diff.md`
  7. Wait for `morpheus` to be done
  8. Re-create a new `.git` and make one single commit on it. We do this
  because we don't want to preserve history through `.git` and this is
  cheap anyway
  9. We release the lock
  On top of this, we keep the retry policies etc etc
  
  The goals of this new workflow are:
  * Better support of any memory extensions such as `chronicle`
  * Allow the user to manually edit memories and this will be considered
  by the phase 2 agent
   
  As a follow-up we will need to add support for user's edition while
  `morpheus` is running
  
  ## What Changed
  
  - Added memory workspace helpers that prepare the git baseline, compute
  the diff, write `phase2_workspace_diff.md`, and reset the baseline after
  successful consolidation.
  - Updated Phase 2 to sync current inputs into `raw_memories.md` and
  `rollout_summaries/`, prune old extension resources, skip clean
  workspaces, and run the consolidation subagent only when the workspace
  has changes.
  - Tightened Phase 2 job ownership around long-running consolidation with
  heartbeats and an ownership check before resetting the baseline.
  - Simplified the prompt and state APIs so DB watermarks are bookkeeping,
  while workspace dirtiness decides whether consolidation work exists.
  - Updated the memory pipeline README and tests for workspace diffs,
  extension-resource cleanup, pollution-driven forgetting, selection
  ranking, and baseline persistence.
  
  ## Verification
  
  - Added/updated coverage in `core/src/memories/tests.rs`,
  `core/src/memories/workspace_tests.rs`, `state/src/runtime/memories.rs`,
  and `core/tests/suite/memories.rs`.
  
  ---------
  
  Co-authored-by: Codex <noreply@openai.com>

  jif-oai · 2026-04-27 14:32:44 +02:00
  01ab25dbb5
• Add goal persistence foundation (1 / 5) (#18073)

  Adds the persisted goal foundation for the rest of the stack. This PR is
  intentionally limited to feature flag and state-layer behavior;
  app-server APIs, model tools, runtime continuation, and TUI UX are
  layered in later PRs.
  
  ## Why
  
  Goal mode needs durable thread-level state before clients or model tools
  can safely build on it. The state layer needs to know whether a goal
  exists, what objective it tracks, whether it is active, paused,
  budget-limited, or complete, and how much time/token usage has already
  been accounted.
  
  ## What changed
  
  - Added the `goals` feature flag and generated config schema entry.
  - Added the `thread_goals` state table and Rust model for persisted
  thread goals.
  - Added state runtime APIs for creating, replacing, updating, deleting,
  and accounting goal usage.
  - Added `goal_id`-based stale update protection so an old goal update
  cannot overwrite a replacement.
  - Kept this PR scoped to persistence and state runtime behavior, with no
  app-server, model-facing, continuation, or TUI behavior yet.
  
  ## Verification
  
  - Added state runtime coverage for goal creation, replacement, stale
  update protection, status transitions, token-budget behavior, and usage
  accounting.

  Eric Traut · 2026-04-24 20:51:38 -07:00
  0ee737cea6
• app-server: persist device key bindings in sqlite (#19206)

  ## Why
  
  Device-key providers should only own platform key material. The
  account/client binding used to authorize a signing payload is app-server
  state, and keeping that state in provider-specific metadata makes the
  same check harder to audit and harder to share across platform
  implementations.
  
  Persisting the binding in the shared state database gives the device-key
  crate a platform-neutral source of truth before it asks a provider to
  sign. It also lets app-server move potentially blocking key operations
  off the main message processor path, which matters once providers may
  wait for OS authentication prompts.
  
  ## What changed
  
  - Add a `device_key_bindings` state migration plus `StateRuntime`
  helpers keyed by `key_id`.
  - Add an async `DeviceKeyBindingStore` abstraction to `codex-device-key`
  and use it from `DeviceKeyStore::create` and `DeviceKeyStore::sign`.
  - Keep provider calls behind async store methods and run the synchronous
  provider work through `spawn_blocking`.
  - Wire app-server device-key RPC handling to the SQLite-backed binding
  store and spawn response/error delivery tasks for device-key requests.
  - Run the turn-start tracing test on the existing larger current-thread
  test harness after the larger async surface made the default test stack
  too small locally.
  
  ## Validation
  
  - `cargo test -p codex-device-key`
  - `cargo test -p codex-state device_key`
  - `cargo test -p codex-state`
  - `cargo test -p codex-app-server device_key`
  - `cargo test -p codex-app-server
  message_processor::tracing_tests::turn_start_jsonrpc_span_parents_core_turn_spans`
  - `cargo test -p codex-app-server`
  - `just fix -p codex-device-key`
  - `just fix -p codex-state`
  - `just fix -p codex-app-server`
  - `just bazel-lock-update`
  - `just bazel-lock-check`
  - `git diff --check`

  Ruslan Nigmatullin · 2026-04-23 21:55:56 -07:00
  19badb0be2
• Add sorting/backwardsCursor to thread/list and new thread/turns/list api (#17305)

  To improve performance of UI loads from the app, add two main
  improvements:
  1. The `thread/list` api now gets a `sortDirection` request field and a
  `backwardsCursor` to the response, which lets you paginate forwards and
  backwards from a window. This lets you fetch the first few items to
  display immediately while you paginate to fill in history, then can
  paginate "backwards" on future loads to catch up with any changes since
  the last UI load without a full reload of the entire data set.
  2. Added a new `thread/turns/list` api which also has sortDirection and
  backwardsCursor for the same behavior as `thread/list`, allowing you the
  same small-fetch for immediate display followed by background fill-in
  and resync catchup.

  David de Regt · 2026-04-17 11:49:02 -07:00
  eaf78e43f2
• app-server: Add transport for remote control (#15951)
  Ruslan Nigmatullin · 2026-04-06 14:55:59 -07:00
  73dab2046f
• feat: log db better maintenance (#16330)

  Run a DB clean-up more frequently with an incremental `VACCUM` in it

  jif-oai · 2026-03-31 19:15:44 +02:00
  868ac158d7
• feat: add graph representation of agent network (#15056)

  Add a representation of the agent graph. This is now used for:
  * Cascade close agents (when I close a parent, it close the kids)
  * Cascade resume (oposite)
  
  Later, this will also be used for post-compaction stuffing of the
  context
  
  Direct fix for: https://github.com/openai/codex/issues/14458

  jif-oai · 2026-03-19 10:21:25 +00:00
  70cdb17703
• feat: simplify DB further (#13771)
  jif-oai · 2026-03-07 03:48:36 -08:00
  cf143bf71e
• chore: improve DB flushing (#13620)

  This branch:
  * Avoid flushing DB when not necessary
  * Filter events for which we perfom an `upsert` into the DB
  * Add a dedicated update function of the `thread:updated_at` that is
  lighter
  
  This should significantly reduce the DB lock contention. If it is not
  sufficient, we can de-sync the flush of the DB for `updated_at`

  jif-oai · 2026-03-06 19:58:14 +01:00
  0e41a5c4a8
• Move sqlite logs to a dedicated database (#13772)

  ## Summary
  - move sqlite log reads and writes onto a dedicated `logs_1.sqlite`
  database to reduce lock contention with the main state DB
  - add a dedicated logs migrator and route `codex-state-logs` to the new
  database path
  - leave the old `logs` table in the existing state DB untouched for now
  
  ## Testing
  - just fmt
  - cargo test -p codex-state
  
  ---------
  
  Co-authored-by: Codex <noreply@openai.com>

  Charley Cunningham · 2026-03-06 10:54:20 -08:00
  4e6c6193a1
• feat: drop discrepency metrics (#13753)
  jif-oai · 2026-03-06 18:32:25 +01:00
  f891f516a5
• feat: memories forgetting (#12900)

  Add diff based memory forgetting

  jif-oai · 2026-02-26 13:19:57 +00:00
  382fa338b3
• Agent jobs (spawn_agents_on_csv) + progress UI (#10935)

  ## Summary
  - Add agent job support: spawn a batch of sub-agents from CSV, auto-run,
  auto-export, and store results in SQLite.
  - Simplify workflow: remove run/resume/get-status/export tools; spawn is
  deterministic and completes in one call.
  - Improve exec UX: stable, single-line progress bar with ETA; suppress
  sub-agent chatter in exec.
  
  ## Why
  Enables map-reduce style workflows over arbitrarily large repos using
  the existing Codex orchestrator. This addresses review feedback about
  overly complex job controls and non-deterministic monitoring.
  
  ## Demo (progress bar)
  ```
  ./codex-rs/target/debug/codex exec \
    --enable collab \
    --enable sqlite \
    --full-auto \
    --progress-cursor \
    -c agents.max_threads=16 \
    -C /Users/daveaitel/code/codex \
    - <<'PROMPT'
  Create /tmp/agent_job_progress_demo.csv with columns: path,area and 30 rows:
  path = item-01..item-30, area = test.
  
  Then call spawn_agents_on_csv with:
  - csv_path: /tmp/agent_job_progress_demo.csv
  - instruction: "Run `python - <<'PY'` to sleep a random 0.3–1.2s, then output JSON with keys: path, score (int). Set score = 1."
  - output_csv_path: /tmp/agent_job_progress_demo_out.csv
  PROMPT
  ```
  
  ## Review feedback addressed
  - Auto-start jobs on spawn; removed run/resume/status/export tools.
  - Auto-export on success.
  - More descriptive tool spec + clearer prompts.
  - Avoid deadlocks on spawn failure; pending/running handled safely.
  - Progress bar no longer scrolls; stable single-line redraw.
  
  ## Tests
  - `cd codex-rs && cargo test -p codex-exec`
  - `cd codex-rs && cargo build -p codex-cli`

  daveaitel-openai · 2026-02-24 21:00:19 +00:00
  dcab40123f
• nit: upgrade DB version (#11581)
  jif-oai · 2026-02-12 13:16:28 +00:00
  2a409ca67c
• feat: mem v2 - PR5 (#11372)
  jif-oai · 2026-02-10 23:22:55 +00:00
  2c9be54c9a
• feat: mem v2 - PR4 (#11369)

  # Memories migration plan (simplified global workflow)
  
  ## Target behavior
  
  - One shared memory root only: `~/.codex/memories/`.
  - No per-cwd memory buckets, no cwd hash handling.
  - Phase 1 candidate rules:
  - Not currently being processed unless the job lease is stale.
  - Rollout updated within the max-age window (currently 30 days).
  - Rollout idle for at least 12 hours (new constant).
  - Global cap: at most 64 stage-1 jobs in `running` state at any time
  (new invariant).
  - Stage-1 model output shape (new):
  - `rollout_slug` (accepted but ignored for now).
  - `rollout_summary`.
  - `raw_memory`.
  - Phase-1 artifacts written under the shared root:
  - `rollout_summaries/<thread_id>.md` for each rollout summary.
  - `raw_memories.md` containing appended/merged raw memory paragraphs.
  - Phase 2 runs one consolidation agent for the shared `memories/`
  directory.
  - Phase-2 lock is DB-backed with 1 hour lease and heartbeat/expiry.
  
  ## Current code map
  
  - Core startup pipeline: `core/src/memories/startup/mod.rs`.
  - Stage-1 request+parse: `core/src/memories/startup/extract.rs`,
  `core/src/memories/stage_one.rs`, templates in
  `core/templates/memories/`.
  - File materialization: `core/src/memories/storage.rs`,
  `core/src/memories/layout.rs`.
  - Scope routing (cwd/user): `core/src/memories/scope.rs`,
  `core/src/memories/startup/mod.rs`.
  - DB job lifecycle and scope queueing: `state/src/runtime/memory.rs`.
  
  ## PR plan
  
  ## PR 1: Correct phase-1 selection invariants (no behavior-breaking
  layout changes yet)
  
  - Add `PHASE_ONE_MIN_ROLLOUT_IDLE_HOURS: i64 = 12` in
  `core/src/memories/mod.rs`.
  - Thread this into `state::claim_stage1_jobs_for_startup(...)`.
  - Enforce idle-time filter in DB selection logic (not only in-memory
  filtering after `scan_limit`) so eligible threads are not starved by
  very recent threads.
  - Enforce global running cap of 64 at claim time in DB logic:
  - Count fresh `memory_stage1` running jobs.
  - Only allow new claims while count < cap.
  - Keep stale-lease takeover behavior intact.
  - Add/adjust tests in `state/src/runtime.rs`:
  - Idle filter inclusion/exclusion around 12h boundary.
  - Global running-cap guarantee.
  - Existing stale/fresh ownership behavior still passes.
  
  Acceptance criteria:
  - Startup never creates more than 64 fresh `memory_stage1` running jobs.
  - Threads updated <12h ago are skipped.
  - Threads older than 30d are skipped.
  
  ## PR 2: Stage-1 output contract + storage artifacts
  (forward-compatible)
  
  - Update parser/types to accept the new structured output while keeping
  backward compatibility:
  - Add `rollout_slug` (optional for now).
  - Add `rollout_summary`.
  - Keep alias support for legacy `summary` and `rawMemory` until prompt
  swap completes.
  - Update stage-1 schema generator in `core/src/memories/stage_one.rs` to
  include the new keys.
  - Update prompt templates:
  - `core/templates/memories/stage_one_system.md`.
  - `core/templates/memories/stage_one_input.md`.
  - Replace storage model in `core/src/memories/storage.rs`:
  - Introduce `rollout_summaries/` directory writer (`<thread_id>.md`
  files).
  - Introduce `raw_memories.md` aggregator writer from DB rows.
  - Keep deterministic rebuild behavior from DB outputs so files can
  always be regenerated.
  - Update consolidation prompt template to reference `rollout_summaries/`
  + `raw_memories.md` inputs.
  
  Acceptance criteria:
  - Stage-1 accepts both old and new output keys during migration.
  - Phase-1 artifacts are generated in new format from DB state.
  - No dependence on per-thread files in `raw_memories/`.
  
  ## PR 3: Remove per-cwd memories and move to one global memory root
  
  - Simplify layout in `core/src/memories/layout.rs`:
  - Single root: `codex_home/memories`.
  - Remove cwd-hash bucket helpers and normalization logic used only for
  memory pathing.
  - Remove scope branching from startup phase-2 dispatch path:
  - No cwd/user mapping in `core/src/memories/startup/mod.rs`.
  - One target root for consolidation.
  - In `state/src/runtime/memory.rs`, stop enqueueing/handling cwd
  consolidation scope.
  - Keep one logical consolidation scope/job key (global/user) to avoid a
  risky schema rewrite in same PR.
  - Add one-time migration helper (core side) to preserve current shared
  memory output:
  - If `~/.codex/memories/user/memory` exists and new root is empty,
  move/copy contents into `~/.codex/memories`.
  - Leave old hashed cwd buckets untouched for now (safe/no-destructive
  migration).
  
  Acceptance criteria:
  - New runs only read/write `~/.codex/memories`.
  - No new cwd-scoped consolidation jobs are enqueued.
  - Existing user-shared memory content is preserved.
  
  ## PR 4: Phase-2 global lock simplification and cleanup
  
  - Replace multi-scope dispatch with a single global consolidation claim
  path:
  - Either reuse jobs table with one fixed key, or add a tiny dedicated
  lock helper; keep 1h lease.
  - Ensure at most one consolidation agent can run at once.
  - Keep heartbeat + stale lock recovery semantics in
  `core/src/memories/startup/watch.rs`.
  - Remove dead scope code and legacy constants no longer used.
  - Update tests:
  - One-agent-at-a-time behavior.
  - Lock expiry allows takeover after stale lease.
  
  Acceptance criteria:
  - Exactly one phase-2 consolidation agent can be active cluster-wide
  (per local DB).
  - Stale lock recovers automatically.
  
  ## PR 5: Final cleanup and docs
  
  - Remove legacy artifacts and references:
  - `raw_memories/` and `memory_summary.md` assumptions from
  prompts/comments/tests.
  - Scope constants for cwd memory pathing in core/state if fully unused.
  - Update docs under `docs/` for memory workflow and directory layout.
  - Add a brief operator note for rollout: compatibility window for old
  stage-1 JSON keys and when to remove aliases.
  
  Acceptance criteria:
  - Code and docs reflect only the simplified global workflow.
  - No stale references to per-cwd memory buckets.
  
  ## Notes on sequencing
  
  - PR 1 is safest first because it improves correctness without changing
  external artifact layout.
  - PR 2 keeps parser compatibility so prompt deployment can happen
  independently.
  - PR 3 and PR 4 split filesystem/scope simplification from locking
  simplification to reduce blast radius.
  - PR 5 is intentionally cleanup-only.

  jif-oai · 2026-02-10 23:10:35 +00:00
  623d3f4071
• feat: mem v2 - PR1 (#11364)

  # Memories migration plan (simplified global workflow)
  
  ## Target behavior
  
  - One shared memory root only: `~/.codex/memories/`.
  - No per-cwd memory buckets, no cwd hash handling.
  - Phase 1 candidate rules:
  - Not currently being processed unless the job lease is stale.
  - Rollout updated within the max-age window (currently 30 days).
  - Rollout idle for at least 12 hours (new constant).
  - Global cap: at most 64 stage-1 jobs in `running` state at any time
  (new invariant).
  - Stage-1 model output shape (new):
  - `rollout_slug` (accepted but ignored for now).
  - `rollout_summary`.
  - `raw_memory`.
  - Phase-1 artifacts written under the shared root:
  - `rollout_summaries/<thread_id>.md` for each rollout summary.
  - `raw_memories.md` containing appended/merged raw memory paragraphs.
  - Phase 2 runs one consolidation agent for the shared `memories/`
  directory.
  - Phase-2 lock is DB-backed with 1 hour lease and heartbeat/expiry.
  
  ## Current code map
  
  - Core startup pipeline: `core/src/memories/startup/mod.rs`.
  - Stage-1 request+parse: `core/src/memories/startup/extract.rs`,
  `core/src/memories/stage_one.rs`, templates in
  `core/templates/memories/`.
  - File materialization: `core/src/memories/storage.rs`,
  `core/src/memories/layout.rs`.
  - Scope routing (cwd/user): `core/src/memories/scope.rs`,
  `core/src/memories/startup/mod.rs`.
  - DB job lifecycle and scope queueing: `state/src/runtime/memory.rs`.
  
  ## PR plan
  
  ## PR 1: Correct phase-1 selection invariants (no behavior-breaking
  layout changes yet)
  
  - Add `PHASE_ONE_MIN_ROLLOUT_IDLE_HOURS: i64 = 12` in
  `core/src/memories/mod.rs`.
  - Thread this into `state::claim_stage1_jobs_for_startup(...)`.
  - Enforce idle-time filter in DB selection logic (not only in-memory
  filtering after `scan_limit`) so eligible threads are not starved by
  very recent threads.
  - Enforce global running cap of 64 at claim time in DB logic:
  - Count fresh `memory_stage1` running jobs.
  - Only allow new claims while count < cap.
  - Keep stale-lease takeover behavior intact.
  - Add/adjust tests in `state/src/runtime.rs`:
  - Idle filter inclusion/exclusion around 12h boundary.
  - Global running-cap guarantee.
  - Existing stale/fresh ownership behavior still passes.
  
  Acceptance criteria:
  - Startup never creates more than 64 fresh `memory_stage1` running jobs.
  - Threads updated <12h ago are skipped.
  - Threads older than 30d are skipped.
  
  ## PR 2: Stage-1 output contract + storage artifacts
  (forward-compatible)
  
  - Update parser/types to accept the new structured output while keeping
  backward compatibility:
  - Add `rollout_slug` (optional for now).
  - Add `rollout_summary`.
  - Keep alias support for legacy `summary` and `rawMemory` until prompt
  swap completes.
  - Update stage-1 schema generator in `core/src/memories/stage_one.rs` to
  include the new keys.
  - Update prompt templates:
  - `core/templates/memories/stage_one_system.md`.
  - `core/templates/memories/stage_one_input.md`.
  - Replace storage model in `core/src/memories/storage.rs`:
  - Introduce `rollout_summaries/` directory writer (`<thread_id>.md`
  files).
  - Introduce `raw_memories.md` aggregator writer from DB rows.
  - Keep deterministic rebuild behavior from DB outputs so files can
  always be regenerated.
  - Update consolidation prompt template to reference `rollout_summaries/`
  + `raw_memories.md` inputs.
  
  Acceptance criteria:
  - Stage-1 accepts both old and new output keys during migration.
  - Phase-1 artifacts are generated in new format from DB state.
  - No dependence on per-thread files in `raw_memories/`.
  
  ## PR 3: Remove per-cwd memories and move to one global memory root
  
  - Simplify layout in `core/src/memories/layout.rs`:
  - Single root: `codex_home/memories`.
  - Remove cwd-hash bucket helpers and normalization logic used only for
  memory pathing.
  - Remove scope branching from startup phase-2 dispatch path:
  - No cwd/user mapping in `core/src/memories/startup/mod.rs`.
  - One target root for consolidation.
  - In `state/src/runtime/memory.rs`, stop enqueueing/handling cwd
  consolidation scope.
  - Keep one logical consolidation scope/job key (global/user) to avoid a
  risky schema rewrite in same PR.
  - Add one-time migration helper (core side) to preserve current shared
  memory output:
  - If `~/.codex/memories/user/memory` exists and new root is empty,
  move/copy contents into `~/.codex/memories`.
  - Leave old hashed cwd buckets untouched for now (safe/no-destructive
  migration).
  
  Acceptance criteria:
  - New runs only read/write `~/.codex/memories`.
  - No new cwd-scoped consolidation jobs are enqueued.
  - Existing user-shared memory content is preserved.
  
  ## PR 4: Phase-2 global lock simplification and cleanup
  
  - Replace multi-scope dispatch with a single global consolidation claim
  path:
  - Either reuse jobs table with one fixed key, or add a tiny dedicated
  lock helper; keep 1h lease.
  - Ensure at most one consolidation agent can run at once.
  - Keep heartbeat + stale lock recovery semantics in
  `core/src/memories/startup/watch.rs`.
  - Remove dead scope code and legacy constants no longer used.
  - Update tests:
  - One-agent-at-a-time behavior.
  - Lock expiry allows takeover after stale lease.
  
  Acceptance criteria:
  - Exactly one phase-2 consolidation agent can be active cluster-wide
  (per local DB).
  - Stale lock recovers automatically.
  
  ## PR 5: Final cleanup and docs
  
  - Remove legacy artifacts and references:
  - `raw_memories/` and `memory_summary.md` assumptions from
  prompts/comments/tests.
  - Scope constants for cwd memory pathing in core/state if fully unused.
  - Update docs under `docs/` for memory workflow and directory layout.
  - Add a brief operator note for rollout: compatibility window for old
  stage-1 JSON keys and when to remove aliases.
  
  Acceptance criteria:
  - Code and docs reflect only the simplified global workflow.
  - No stale references to per-cwd memory buckets.
  
  ## Notes on sequencing
  
  - PR 1 is safest first because it improves correctness without changing
  external artifact layout.
  - PR 2 keeps parser compatibility so prompt deployment can happen
  independently.
  - PR 3 and PR 4 split filesystem/scope simplification from locking
  simplification to reduce blast radius.
  - PR 5 is intentionally cleanup-only.

  jif-oai · 2026-02-10 21:29:06 +00:00
  07da740c8a
• chore: unify memory job flow (#11334)
  jif-oai · 2026-02-10 20:26:39 +00:00
  a6e9469fa4
• feat: phase 2 consolidation (#11306)

  Consolidation phase of memories
  
  Cleaning and better handling of concurrency

  jif-oai · 2026-02-10 14:31:16 +00:00
  e57892b211
• feat: align memory phase 1 and make it stronger (#11300)

  ## Align with the new phase-1 design
  
  Basically we know run phase 1 in parallel by considering:
  * Max 64 rollouts
  * Max 1 month old
  * Consider the most recent first
  
  This PR also adds stronger parallelization capabilities by detecting
  stale jobs, retry policies, ownership of computation to prevent double
  computations etc etc

  jif-oai · 2026-02-10 13:42:09 +00:00
  1d5eba0090
• feat: resumable backfill (#10745)

  ## Summary
  
  This PR makes SQLite rollout backfill resumable and repeatable instead
  of one-shot-on-db-create.
  
  ## What changed
  
  - Added a persisted backfill state table:
    - state/migrations/0008_backfill_state.sql
  - Tracks status (pending|running|complete), last_watermark, and
  last_success_at.
  - Added backfill state model/types in codex-state:
    - BackfillState, BackfillStatus (state/src/model/backfill_state.rs)
  - Added runtime APIs to manage backfill lifecycle/progress:
    - get_backfill_state
    - mark_backfill_running
    - checkpoint_backfill
    - mark_backfill_complete
  - Updated core startup behavior:
  - Backfill now runs whenever state is not Complete (not only when DB
  file is newly created).
  - Reworked backfill execution:
  - Collect rollout files, derive deterministic watermark per path, sort,
  resume from last_watermark.
  - Process in batches (BACKFILL_BATCH_SIZE = 200), checkpoint after each
  batch.
    - Mark complete with last_success_at at the end.
  
  ## Why
  
  Previous behavior could leave users permanently partially backfilled if
  the process exited during initial async backfill. This change allows
  safe continuation across restarts and avoids restarting from scratch.

  jif-oai · 2026-02-05 14:34:34 +00:00
  4033f905c6
• feat: add phase 1 mem db (#10634)

  - Schema: thread_id (PK, FK to threads.id with cascade delete),
  trace_summary, memory_summary, updated_at.
  - Migration: creates the table and an index on (updated_at DESC,
  thread_id DESC) for efficient recent-first reads.
    - Runtime API (DB-only):
        - `get_thread_memory(thread_id)`: fetch one memory row.
  - `upsert_thread_memory(thread_id, trace_summary, memory_summary)`:
  insert/update by thread id and always advance updated_at.
  - `get_last_n_thread_memories_for_cwd(cwd, n)`: join thread_memory with
  threads and return newest n rows for an exact cwd match.
  - Model layer: introduced ThreadMemory and row conversion types to keep
  query decoding typed and consistent with existing state models.

  jif-oai · 2026-02-04 21:38:39 +00:00
  4922b3e571
• Migrate state DB path helpers to versioned filename (#10623)

  Summary
  - add versioned state sqlite filename helpers and re-export them from
  the state crate
  - remove legacy state files when initializing the runtime and update
  consumers/tests to use the new helpers
  - tweak logs client description and database resolution to match the new
  path

  jif-oai · 2026-02-04 14:31:12 +00:00
  583e5d4f41
• feat: backfill timing metric (#10218)

  1. Add a metric to measure the backfill time
  2. Add a unit to the timing histogram

  jif-oai · 2026-01-30 10:19:41 +01:00
  129787493f
• chore: unify log queries (#10152)

  Unify log queries to only have SQLX code in the runtime and use it for
  both the log client and for tests

  jif-oai · 2026-01-29 16:28:15 +00:00
  e6c4f548ab
• feat: add log db (#10086)

  Add a log DB. The goal is just to store our logs in a `.sqlite` DB to
  make it easier to crawl them and drop the oldest ones.

  jif-oai · 2026-01-29 10:23:03 +01:00
  780482da84
• feat: sqlite 1 (#10004)

  Add a `.sqlite` database to be used to store rollout metatdata (and
  later logs)
  This PR is phase 1:
  * Add the database and the required infrastructure
  * Add a backfill of the database
  * Persist the newly created rollout both in files and in the DB
  * When we need to get metadata or a rollout, consider the `JSONL` as the
  source of truth but compare the results with the DB and show any errors

  jif-oai · 2026-01-28 15:29:14 +01:00
  3878c3dc7c