Stack split from #25708. Original PR intentionally left open. This first
PR adds the multi-agent runtime metadata types and catalog plumbing used
by the rest of the stack.
## Summary
- teach rollout search to return precomputed snippets for compressed
rollouts
- reuse those snippets in local thread search instead of reopening
matching compressed files
- keep the no-`rg` fallback single-pass and add regression coverage for
the compressed path
## Why
`thread/search` currently decodes matching compressed rollouts twice:
once to discover the matching path and again to extract the snippet
shown in results. That defeats a meaningful part of the compressed-read
optimization work.
## Impact
Compressed rollout hits now pay one decode pass on the search path while
plain `.jsonl` hits keep the existing ripgrep-driven flow.
## Validation
- `just test -p codex-rollout`
- `just test -p codex-thread-store`
- `just fix -p codex-rollout`
- `just fix -p codex-thread-store`
- `just fmt`
## Why
[#25089](https://github.com/openai/codex/pull/25089) introduced the
background worker that compresses cold archived rollouts, and
[#25654](https://github.com/openai/codex/pull/25654) made that pass
faster once it starts. But the worker still deleted
`rollout-compression.lock` on successful exit, so the existing six-hour
staleness window only helped with overlapping or crashed workers. Each
new local thread-store initialization could immediately rescan archived
rollouts even if a full pass had just finished.
This change keeps the existing marker around long enough to throttle
redundant reruns. The worker is still best-effort, but it no longer does
repeated startup scans when nothing new is eligible for compression.
## What Changed
- Replace the drop-scoped `CompressionLock` with a
`CompressionRunMarker` that claims the existing
`.tmp/rollout-compression.lock` path and leaves it in place after
success.
- Reuse the existing six-hour staleness window to block both overlapping
starts and immediate reruns, while still letting a stale marker be
reclaimed.
- Update the worker docs and debug logging to describe the new "already
running or recently ran" behavior.
- Extend the rollout compression tests to assert that a successful run
leaves the marker behind and that a fresh marker suppresses a new run.
## Validation
- `just test -p codex-rollout`
## Rollout compression stack
This stack splits #24941 into reviewable steps for local rollout
compression. The design is intentionally staged:
1. Teach readers, listing, search, and lookup to understand compressed
rollouts.
2. Make append and resume paths materialize compressed rollouts back to
plain JSONL before writing.
3. Add a disabled-by-default worker that can compress cold archived
rollouts behind `local_thread_store_compression`.
The key invariant is that writers append to plain `.jsonl`. A
`.jsonl.zst` file is a cold/read representation; if a write is needed,
the compressed file is materialized back to plain JSONL first. Readers
prefer plain `.jsonl` when both forms exist and can fall back to the
compressed sibling during transitions.
The worker is deliberately the last PR and remains behind an
under-development feature flag. It currently scans only
`archived_sessions`, not active `sessions`, because active sessions have
the highest resume/append race risk. That means this stack does not yet
compress most unarchived local history.
## Known race / follow-up
The remaining unresolved design question is writer/compressor
coordination. Even for archived rollouts, a resume or metadata update
can append while the worker is replacing the plain file with
`.jsonl.zst`; the current double-stat checks narrow but do not fully
eliminate the window where a writer has opened the plain file before
unlink. Do not treat the worker PR as production-ready until we either:
- prevent append/resume paths from racing archived compression, or
- introduce a shared representation/append lock or equivalent
coordination.
The first two PRs are useful independently: they make compressed
rollouts readable and make append paths safely recover back to plain
JSONL. The third PR isolates the worker behavior so that coordination
issue is reviewable separately.
## Validation
Focused local validation for the stack includes:
- `just test -p codex-rollout`
- `just test -p codex-thread-store` where thread-store paths were
touched
- `just test -p codex-features` for the feature flag slice
- `just bazel-lock-check` after dependency graph changes
- scoped `just fix -p ...` passes for changed crates
CI is still the source of truth for the full platform matrix.
## This PR in the stack
This is PR 3/3, based on #25088. It adds the under-development feature
flag and starts the best-effort background worker when enabled. The
worker currently compresses only cold archived rollouts, skips active
sessions, verifies compressed output, preserves mtime and permissions,
keeps a store-level lock heartbeat, and cleans stale temp files.
Stack order:
1. #25087: read compressed local rollouts.
2. #25088: materialize compressed rollouts before append.
3. This PR: add the disabled local compression worker.
## Summary
- initialize `parent_thread_id` in the compressed rollout test fixture's
`SessionMeta`
- restore rollout test compilation across Bazel test, clippy,
release-build, and argument-comment-lint jobs
## Root cause
PR #25087 (`Read compressed rollouts and materialize before append`)
added `codex-rs/rollout/src/compression_tests.rs` in merge commit
`a8a6071279b6f3112fcc5fc3fee69c48473d7149`. Its `write_rollout` fixture
constructs `SessionMeta` without the required `parent_thread_id` field,
causing `error[E0063]` when Bazel compiles `rollout-unit-tests-bin` on
`main` and downstream PRs.
## Validation
- `UV_CACHE_DIR=/private/tmp/codex-uv-cache just fmt`
- `just test -p codex-rollout` (`59` tests passed; bench smoke passed)
- `git diff --check`
- manually audited the touched Rust diff for positional literal argument
comments; the change adds no positional callsite
## Local lint blocker
- `just argument-comment-lint` could not reach source inspection locally
because Bazel's LLVM dependency fails analysis:
`compiler-rt/BUILD.bazel` glob `include/sanitizer/*.h` matched no files.
## Why
Local rollout compression needs a cold `.jsonl.zst` representation
without letting compressed physical paths leak into append-mode writers.
The unsafe case is resume or metadata update code successfully reading a
compressed rollout and then appending raw JSONL bytes to the zstd file.
This PR folds the former #25088 materialization slice into the
read-support PR so the reader changes and append-safety invariant land
together.
## What Changed
- Teach rollout readers, discovery, listing, search, and ID lookup to
understand compressed `.jsonl.zst` rollouts.
- Keep `.jsonl` as the logical/stored rollout path while allowing read
paths to open either plain or compressed storage.
- Materialize compressed rollouts back to plain `.jsonl` before
append-mode writes, including resume and direct metadata append paths.
- Preserve compressed-file permissions when materializing back to plain
JSONL.
- Refresh thread-store resolved rollout paths after compatibility
metadata writes so reconciliation follows the materialized file.
- Avoid treating transient compression temp files as real rollout lookup
results.
## Remaining Stack
#25089 remains the separate worker PR. It is based directly on this PR
and stays behind the disabled `local_thread_store_compression` feature
flag.
The worker still has a broader coordination question: a resume or
metadata update can race with background compression while a plain file
is being replaced by `.jsonl.zst`. This PR handles the read and
materialize-before-append primitives; it does not make the worker
production-ready.
## Validation
- `just test -p codex-rollout`
- `just test -p codex-thread-store`
- `just fix -p codex-rollout`
- `just fix -p codex-thread-store`
- `just bazel-lock-check`
jif-oaiandGitHub
Felipe CouryandGitHub