jif-oai 01cb97851b Compress cold local rollouts (#25089) ...

## Rollout compression stack

This stack splits #24941 into reviewable steps for local rollout
compression. The design is intentionally staged:

1. Teach readers, listing, search, and lookup to understand compressed
rollouts.
2. Make append and resume paths materialize compressed rollouts back to
plain JSONL before writing.
3. Add a disabled-by-default worker that can compress cold archived
rollouts behind `local_thread_store_compression`.

The key invariant is that writers append to plain `.jsonl`. A
`.jsonl.zst` file is a cold/read representation; if a write is needed,
the compressed file is materialized back to plain JSONL first. Readers
prefer plain `.jsonl` when both forms exist and can fall back to the
compressed sibling during transitions.

The worker is deliberately the last PR and remains behind an
under-development feature flag. It currently scans only
`archived_sessions`, not active `sessions`, because active sessions have
the highest resume/append race risk. That means this stack does not yet
compress most unarchived local history.

## Known race / follow-up

The remaining unresolved design question is writer/compressor
coordination. Even for archived rollouts, a resume or metadata update
can append while the worker is replacing the plain file with
`.jsonl.zst`; the current double-stat checks narrow but do not fully
eliminate the window where a writer has opened the plain file before
unlink. Do not treat the worker PR as production-ready until we either:

- prevent append/resume paths from racing archived compression, or
- introduce a shared representation/append lock or equivalent
coordination.

The first two PRs are useful independently: they make compressed
rollouts readable and make append paths safely recover back to plain
JSONL. The third PR isolates the worker behavior so that coordination
issue is reviewable separately.

## Validation

Focused local validation for the stack includes:

- `just test -p codex-rollout`
- `just test -p codex-thread-store` where thread-store paths were
touched
- `just test -p codex-features` for the feature flag slice
- `just bazel-lock-check` after dependency graph changes
- scoped `just fix -p ...` passes for changed crates

CI is still the source of truth for the full platform matrix.

## This PR in the stack

This is PR 3/3, based on #25088. It adds the under-development feature
flag and starts the best-effort background worker when enabled. The
worker currently compresses only cold archived rollouts, skips active
sessions, verifies compressed output, preserves mtime and permissions,
keeps a store-level lock heartbeat, and cleans stale temp files.

Stack order:

1. #25087: read compressed local rollouts.
2. #25088: materialize compressed rollouts before append.
3. This PR: add the disabled local compression worker.

01cb97851b · 2026-06-01 18:35:58 +02:00

History

..

src

Compress cold local rollouts (#25089)

2026-06-01 18:35:58 +02:00

BUILD.bazel

Remove js_repl feature (#19410)

2026-04-24 17:49:29 -07:00

Cargo.toml

Refactor config types into a separate crate (#16962)

2026-04-07 00:32:41 +00:00