21 Commits

• feat: add phase 1 mem db (#10634)

  - Schema: thread_id (PK, FK to threads.id with cascade delete),
  trace_summary, memory_summary, updated_at.
  - Migration: creates the table and an index on (updated_at DESC,
  thread_id DESC) for efficient recent-first reads.
    - Runtime API (DB-only):
        - `get_thread_memory(thread_id)`: fetch one memory row.
  - `upsert_thread_memory(thread_id, trace_summary, memory_summary)`:
  insert/update by thread id and always advance updated_at.
  - `get_last_n_thread_memories_for_cwd(cwd, n)`: join thread_memory with
  threads and return newest n rows for an exact cwd match.
  - Model layer: introduced ThreadMemory and row conversion types to keep
  query decoding typed and consistent with existing state models.

  jif-oai · 2026-02-04 21:38:39 +00:00
  4922b3e571
• Migrate state DB path helpers to versioned filename (#10623)

  Summary
  - add versioned state sqlite filename helpers and re-export them from
  the state crate
  - remove legacy state files when initializing the runtime and update
  consumers/tests to use the new helpers
  - tweak logs client description and database resolution to match the new
  path

  jif-oai · 2026-02-04 14:31:12 +00:00
  583e5d4f41
• nit: cleaning (#10619)
  jif-oai · 2026-02-04 13:01:24 +00:00
  aab60a55f1
• chore: simplify user message detection (#10611)

  We don't check anymore the response item with `user` role as they may be
  instructions etc

  jif-oai · 2026-02-04 11:14:53 +00:00
  38f6c6b114
• fix: single transaction for dyn tools injection (#10614)
  jif-oai · 2026-02-04 10:57:58 +00:00
  3d8deeea4b
• Prefer state DB thread listings before filesystem (#10544)

  Summary
  - add Cursor/ThreadsPage conversions so state DB listings can be mapped
  back into the rollout list model
  - make recorder list helpers query the state DB first (archived flag
  included) and only fall back to file traversal if needed, along with
  populating head bytes lazily
  - add extensive tests to ensure the DB path is honored for active and
  archived threads and that the fallback works
  
  Testing
  - Not run (not requested)
  
  <img width="1196" height="693" alt="Screenshot 2026-02-03 at 20 42 33"
  src="https://github.com/user-attachments/assets/826b3c7a-ef11-4b27-802a-3c343695794a"
  />

  jif-oai · 2026-02-04 09:27:24 +00:00
  100eb6e6f0
• Avoid redundant transactional check before inserting dynamic tools (#10521)

  Summary
  - remove the extra transaction guard that checked for existing dynamic
  tools per thread before inserting new ones
  - insert each tool record with `ON CONFLICT(thread_id, position) DO
  NOTHING` to ignore duplicates instead of pre-querying
  - simplify execution to use the shared pool directly and avoid unneeded
  commits
  
  Testing
  - Not run (not requested)

  jif-oai · 2026-02-03 15:34:28 +00:00
  ed778f9017
• chore: add phase to message responseitem (#10455)

  ### What
  
  add wiring for `phase` field on `ResponseItem::Message` to lay
  groundwork for differentiating model preambles and final messages.
  currently optional.
  
  follows pattern in #9698.
  
  updated schemas with `just write-app-server-schema` so we can see type
  changes.
  
  ### Tests
  Updated existing tests for SSE parsing and hydrating from history

  sayan-oai · 2026-02-03 02:52:26 +00:00
  fc05374344
• [feat] persist thread_dynamic_tools in db (#10252)

  Persist thread_dynamic_tools in sqlite and read first from it. Fall back
  to rollout files if it's not found. Persist dynamic tools to both sqlite
  and rollout files.
  
  Saw that new sessions get populated to db correctly & old sessions get
  backfilled correctly at startup:
  ```
  celia@com-92114 codex-rs % sqlite3 ~/.codex/state.sqlite \      "select thread_id, position,name,description,input_schema from thread_dynamic_tools;"
  019c0cad-ec0d-74b2-a787-e8b33a349117|0|geo_lookup|lookup a city|{"properties":{"city":{"type":"string"}},"required":["city"],"type":"object"}
  ....
  019c10ca-aa4b-7620-ae40-c0919fbd7ea7|0|geo_lookup|lookup a city|{"properties":{"city":{"type":"string"}},"required":["city"],"type":"object"}
  ```

  Celia Chen · 2026-02-03 00:06:44 +00:00
  fb2df99cf1
• feat: drop sqlx logging (#10398)
  jif-oai · 2026-02-02 19:26:58 +00:00
  34c0534f6e
• nit: fix db with multiple metadata lines (#10237)
  jif-oai · 2026-01-30 17:32:10 +00:00
  0212f4010e
• feat: improve logs client (#10229)
  jif-oai · 2026-01-30 18:23:18 +01:00
  eff11f792b
• feat: backfill timing metric (#10218)

  1. Add a metric to measure the backfill time
  2. Add a unit to the timing histogram

  jif-oai · 2026-01-30 10:19:41 +01:00
  129787493f
• feat: reduce span exposition (#10171)

  This only avoids the creation of duplicates spans

  jif-oai · 2026-01-29 18:15:22 +00:00
  798c4b3260
• chore: unify log queries (#10152)

  Unify log queries to only have SQLX code in the runtime and use it for
  both the log client and for tests

  jif-oai · 2026-01-29 16:28:15 +00:00
  e6c4f548ab
• feat: add log retention and delete them after 90 days (#10151)
  jif-oai · 2026-01-29 16:55:01 +01:00
  d6631fb5a9
• feat: adding thread ID to logs + filter in the client (#10150)
  jif-oai · 2026-01-29 16:53:30 +01:00
  89c5f3c4d4
• chore: improve client (#10149)

  <img width="883" height="84" alt="Screenshot 2026-01-29 at 11 13 12"
  src="https://github.com/user-attachments/assets/090a2fec-94ed-4c0f-aee5-1653ed8b1439"
  />

  jif-oai · 2026-01-29 11:25:22 +01:00
  4ba911d48c
• feat: log db client (#10087)

  ```
  just log -h
  if [ "${1:-}" = "--" ]; then shift; fi; cargo run -p codex-state --bin logs_client -- "$@"
      Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.21s
       Running `target/debug/logs_client -h`
  Tail Codex logs from state.sqlite with simple filters
  
  Usage: logs_client [OPTIONS]
  
  Options:
        --codex-home <CODEX_HOME>  Path to CODEX_HOME. Defaults to $CODEX_HOME or ~/.codex [env: CODEX_HOME=]
        --db <DB>                  Direct path to the SQLite database. Overrides --codex-home
        --level <LEVEL>            Log level to match exactly (case-insensitive)
        --from <RFC3339|UNIX>      Start timestamp (RFC3339 or unix seconds)
        --to <RFC3339|UNIX>        End timestamp (RFC3339 or unix seconds)
        --module <MODULE>          Substring match on module_path
        --file <FILE>              Substring match on file path
        --backfill <BACKFILL>      Number of matching rows to show before tailing [default: 200]
        --poll-ms <POLL_MS>        Poll interval in milliseconds [default: 500]
    -h, --help                     Print help
    ```

  jif-oai · 2026-01-29 11:11:47 +01:00
  6a06726af2
• feat: add log db (#10086)

  Add a log DB. The goal is just to store our logs in a `.sqlite` DB to
  make it easier to crawl them and drop the oldest ones.

  jif-oai · 2026-01-29 10:23:03 +01:00
  780482da84
• feat: sqlite 1 (#10004)

  Add a `.sqlite` database to be used to store rollout metatdata (and
  later logs)
  This PR is phase 1:
  * Add the database and the required infrastructure
  * Add a backfill of the database
  * Persist the newly created rollout both in files and in the DB
  * When we need to get metadata or a rollout, consider the `JSONL` as the
  source of truth but compare the results with the DB and show any errors

  jif-oai · 2026-01-28 15:29:14 +01:00
  3878c3dc7c