mirror of
https://github.com/pchuan98/codex.git
synced 2026-07-01 00:31:56 +08:00
jif-oai
4033f905c6
## Summary This PR makes SQLite rollout backfill resumable and repeatable instead of one-shot-on-db-create. ## What changed - Added a persisted backfill state table: - state/migrations/0008_backfill_state.sql - Tracks status (pending|running|complete), last_watermark, and last_success_at. - Added backfill state model/types in codex-state: - BackfillState, BackfillStatus (state/src/model/backfill_state.rs) - Added runtime APIs to manage backfill lifecycle/progress: - get_backfill_state - mark_backfill_running - checkpoint_backfill - mark_backfill_complete - Updated core startup behavior: - Backfill now runs whenever state is not Complete (not only when DB file is newly created). - Reworked backfill execution: - Collect rollout files, derive deterministic watermark per path, sort, resume from last_watermark. - Process in batches (BACKFILL_BATCH_SIZE = 200), checkpoint after each batch. - Mark complete with last_success_at at the end. ## Why Previous behavior could leave users permanently partially backfilled if the process exited during initial async backfill. This change allows safe continuation across restarts and avoids restarting from scratch.
47 lines
1.6 KiB
Rust
47 lines
1.6 KiB
Rust
//! SQLite-backed state for rollout metadata.
|
|
//!
|
|
//! This crate is intentionally small and focused: it extracts rollout metadata
|
|
//! from JSONL rollouts and mirrors it into a local SQLite database. Backfill
|
|
//! orchestration and rollout scanning live in `codex-core`.
|
|
|
|
mod extract;
|
|
pub mod log_db;
|
|
mod migrations;
|
|
mod model;
|
|
mod paths;
|
|
mod runtime;
|
|
|
|
pub use model::LogEntry;
|
|
pub use model::LogQuery;
|
|
pub use model::LogRow;
|
|
/// Preferred entrypoint: owns configuration and metrics.
|
|
pub use runtime::StateRuntime;
|
|
|
|
/// Low-level storage engine: useful for focused tests.
|
|
///
|
|
/// Most consumers should prefer [`StateRuntime`].
|
|
pub use extract::apply_rollout_item;
|
|
pub use model::Anchor;
|
|
pub use model::BackfillState;
|
|
pub use model::BackfillStats;
|
|
pub use model::BackfillStatus;
|
|
pub use model::ExtractionOutcome;
|
|
pub use model::SortKey;
|
|
pub use model::ThreadMemory;
|
|
pub use model::ThreadMetadata;
|
|
pub use model::ThreadMetadataBuilder;
|
|
pub use model::ThreadsPage;
|
|
pub use runtime::STATE_DB_FILENAME;
|
|
pub use runtime::STATE_DB_VERSION;
|
|
pub use runtime::state_db_filename;
|
|
pub use runtime::state_db_path;
|
|
|
|
/// Errors encountered during DB operations. Tags: [stage]
|
|
pub const DB_ERROR_METRIC: &str = "codex.db.error";
|
|
/// Metrics on backfill process. Tags: [status]
|
|
pub const DB_METRIC_BACKFILL: &str = "codex.db.backfill";
|
|
/// Metrics on backfill duration. Tags: [status]
|
|
pub const DB_METRIC_BACKFILL_DURATION_MS: &str = "codex.db.backfill.duration_ms";
|
|
/// Metrics on errors during comparison between DB and rollout file. Tags: [stage]
|
|
pub const DB_METRIC_COMPARE_ERROR: &str = "codex.db.compare_error";
|