12 Commits

• Align SQLite feedback logs with feedback formatter (#13494)

  ## Summary
  - store a pre-rendered `feedback_log_body` in SQLite so `/feedback`
  exports keep span prefixes and structured event fields
  - render SQLite feedback exports with timestamps and level prefixes to
  match the old in-memory feedback formatter, while preserving existing
  trailing newlines
  - count `feedback_log_body` in the SQLite retention budget so structured
  or span-prefixed rows still prune correctly
  - bound `/feedback` row loading in SQL with the retention estimate, then
  apply exact whole-line truncation in Rust so uploads stay capped without
  splitting lines
  
  ## Details
  - add a `feedback_log_body` column to `logs` and backfill it from
  `message` for existing rows
  - capture span names plus formatted span and event fields at write time,
  since SQLite does not retain enough structure to reconstruct the old
  formatter later
  - keep SQLite feedback queries scoped to the requested thread plus
  same-process threadless rows
  - restore a SQL-side cumulative `estimated_bytes` cap for feedback
  export queries so over-retained partitions do not load every matching
  row before truncation
  - add focused formatting coverage for exported feedback lines and parity
  coverage against `tracing_subscriber`
  
  ## Testing
  - cargo test -p codex-state
  - just fix -p codex-state
  - just fmt
  
  codex author: `codex resume 019ca1b0-0ecc-78b1-85eb-6befdd7e4f1f`
  
  ---------
  
  Co-authored-by: Codex <noreply@openai.com>

  Charley Cunningham · 2026-03-18 22:44:31 +00:00
  ebbbc52ce4
• feat: simplify DB further (#13771)
  jif-oai · 2026-03-07 03:48:36 -08:00
  cf143bf71e
• Reduce SQLite log retention to 10 days (#13781)

  ## Summary
  - reduce the SQLite-backed log retention window from 90 days to 10 days
  
  ## Testing
  - just fmt
  - cargo test -p codex-state
  
  Co-authored-by: Codex <noreply@openai.com>

  Charley Cunningham · 2026-03-06 11:15:28 -08:00
  ad98504d74
• Move sqlite logs to a dedicated database (#13772)

  ## Summary
  - move sqlite log reads and writes onto a dedicated `logs_1.sqlite`
  database to reduce lock contention with the main state DB
  - add a dedicated logs migrator and route `codex-state-logs` to the new
  database path
  - leave the old `logs` table in the existing state DB untouched for now
  
  ## Testing
  - just fmt
  - cargo test -p codex-state
  
  ---------
  
  Co-authored-by: Codex <noreply@openai.com>

  Charley Cunningham · 2026-03-06 10:54:20 -08:00
  4e6c6193a1
• feat: limit number of rows per log (#13763)

  avoid DB explosion. This is a temp solution

  jif-oai · 2026-03-06 18:51:42 +01:00
  c8f4b5bc1e
• Add timestamped SQLite /feedback logs without schema changes (#13645)

  ## Summary
  - keep the SQLite schema unchanged (no migrations)
  - add timestamps to SQLite-backed `/feedback` log exports
  - keep the existing SQL-side byte cap behavior and newline handling
  - document the remaining fidelity gap (span prefixes + structured
  fields) with TODOs
  
  ## Details
  - update `query_feedback_logs` to format each exported line as:
    - `YYYY-MM-DDTHH:MM:SS.ffffffZ {level} {message}`
  - continue scoping rows to requested-thread + same-process threadless
  logs
  - continue capping in SQL before returning rows
  - keep the existing fallback behavior unchanged when SQLite returns no
  rows
  - update parity tests to normalize away the new timestamp prefix while
  we still only store `message`
  
  ## Follow-up
  - TODO already in code: persist enough span/event metadata in SQLite to
  reproduce span prefixes and structured fields in `/feedback` exports
  
  ## Testing
  - `cargo test -p codex-state`
  - `just fmt`
  
  ---------
  
  Co-authored-by: Codex <noreply@openai.com>

  Charley Cunningham · 2026-03-05 16:53:37 -08:00
  9f91c7f90f
• app-server: source /feedback logs from sqlite at trace level (#12969)

  ## Summary
  - write app-server SQLite logs at TRACE level when SQLite is enabled
  - source app-server `/feedback` log attachments from SQLite for the
  requested thread when available
  - flush buffered SQLite log writes before `/feedback` queries them so
  newly emitted events are not lost behind the async inserter
  - include same-process threadless SQLite rows in those `/feedback` logs
  so the attachment matches the process-wide feedback buffer more closely
  - keep the existing in-memory ring buffer fallback unchanged, including
  when the SQLite query returns no rows
  
  ## Details
  - add a byte-bounded `query_feedback_logs` helper in `codex-state` so
  `/feedback` does not fetch all rows before truncating
  - scope SQLite feedback logs to the requested thread plus threadless
  rows from the same `process_uuid`
  - format exported SQLite feedback lines with the log level prefix to
  better match the in-memory feedback formatter
  - add an explicit `LogDbLayer::flush()` control path and await it in
  app-server before querying SQLite for feedback logs
  - pass optional SQLite log bytes through `codex-feedback` as the
  `codex-logs.log` attachment override
  - leave TUI behavior unchanged apart from the updated `upload_feedback`
  call signature
  - add regression coverage for:
    - newest-within-budget ordering
    - excluding oversized newest rows
    - including same-process threadless rows
    - keeping the newest suffix across mixed thread and threadless rows
    - matching the feedback formatter shape aside from span prefixes
    - falling back to the in-memory snapshot when SQLite returns no logs
    - flushing buffered SQLite rows before querying
  
  ## Follow-up
  - SQLite feedback exports still do not reproduce span prefixes like
  `feedback-thread{thread_id=...}:`; there is a `TODO(ccunningham)` in
  `codex-rs/state/src/log_db.rs` for that follow-up.
  
  ## Testing
  - `cd codex-rs && cargo test -p codex-state`
  - `cd codex-rs && cargo test -p codex-app-server`
  - `cd codex-rs && just fmt`

  Charley Cunningham · 2026-03-03 11:17:06 -08:00
  c4bd0aa3b9
• Add process_uuid to sqlite logs (#11534)

  ## Summary
  This PR is the first slice of the per-session `/feedback` logging work:
  it adds a process-unique identifier to SQLite log rows.
  
  It does **not** change `/feedback` sourcing behavior yet.
  
  ## Changes
  - Add migration `0009_logs_process_id.sql` to extend `logs` with:
    - `process_uuid TEXT`
    - `idx_logs_process_uuid` index
  - Extend state log models:
    - `LogEntry.process_uuid: Option<String>`
    - `LogRow.process_uuid: Option<String>`
  - Stamp each log row with a stable per-process UUID in the sqlite log
  layer:
    - generated once per process as `pid:<pid>:<uuid>`
  - Update sqlite log insert/query paths to persist and read
  `process_uuid`:
    - `INSERT INTO logs (..., process_uuid, ...)`
    - `SELECT ..., process_uuid, ... FROM logs`
  
  ## Why
  App-server runs many sessions in one process. This change provides a
  process-scoping primitive we need for follow-up `/feedback` work, so
  threadless/process-level logs can be associated with the emitting
  process without mixing across processes.
  
  ## Non-goals in this PR
  - No `/feedback` transport/source changes
  - No attachment size changes
  - No sqlite retention/trim policy changes
  
  ## Testing
  - `just fmt`
  - CI will run the full checks

  Charley Cunningham · 2026-02-14 17:27:22 -08:00
  fce4ad9cf4
• feat: add log retention and delete them after 90 days (#10151)
  jif-oai · 2026-01-29 16:55:01 +01:00
  d6631fb5a9
• feat: adding thread ID to logs + filter in the client (#10150)
  jif-oai · 2026-01-29 16:53:30 +01:00
  89c5f3c4d4
• chore: improve client (#10149)

  <img width="883" height="84" alt="Screenshot 2026-01-29 at 11 13 12"
  src="https://github.com/user-attachments/assets/090a2fec-94ed-4c0f-aee5-1653ed8b1439"
  />

  jif-oai · 2026-01-29 11:25:22 +01:00
  4ba911d48c
• feat: add log db (#10086)

  Add a log DB. The goal is just to store our logs in a `.sqlite` DB to
  make it easier to crawl them and drop the oldest ones.

  jif-oai · 2026-01-29 10:23:03 +01:00
  780482da84