Commit Graph

1 Commits

  • feat: resumable backfill (#10745)
    ## Summary
    
    This PR makes SQLite rollout backfill resumable and repeatable instead
    of one-shot-on-db-create.
    
    ## What changed
    
    - Added a persisted backfill state table:
      - state/migrations/0008_backfill_state.sql
    - Tracks status (pending|running|complete), last_watermark, and
    last_success_at.
    - Added backfill state model/types in codex-state:
      - BackfillState, BackfillStatus (state/src/model/backfill_state.rs)
    - Added runtime APIs to manage backfill lifecycle/progress:
      - get_backfill_state
      - mark_backfill_running
      - checkpoint_backfill
      - mark_backfill_complete
    - Updated core startup behavior:
    - Backfill now runs whenever state is not Complete (not only when DB
    file is newly created).
    - Reworked backfill execution:
    - Collect rollout files, derive deterministic watermark per path, sort,
    resume from last_watermark.
    - Process in batches (BACKFILL_BATCH_SIZE = 200), checkpoint after each
    batch.
      - Mark complete with last_success_at at the end.
    
    ## Why
    
    Previous behavior could leave users permanently partially backfilled if
    the process exited during initial async backfill. This change allows
    safe continuation across restarts and avoids restarting from scratch.