Read compressed rollouts and materialize before append (#25087)

## Why

Local rollout compression needs a cold `.jsonl.zst` representation
without letting compressed physical paths leak into append-mode writers.
The unsafe case is resume or metadata update code successfully reading a
compressed rollout and then appending raw JSONL bytes to the zstd file.

This PR folds the former #25088 materialization slice into the
read-support PR so the reader changes and append-safety invariant land
together.

## What Changed

- Teach rollout readers, discovery, listing, search, and ID lookup to
understand compressed `.jsonl.zst` rollouts.
- Keep `.jsonl` as the logical/stored rollout path while allowing read
paths to open either plain or compressed storage.
- Materialize compressed rollouts back to plain `.jsonl` before
append-mode writes, including resume and direct metadata append paths.
- Preserve compressed-file permissions when materializing back to plain
JSONL.
- Refresh thread-store resolved rollout paths after compatibility
metadata writes so reconciliation follows the materialized file.
- Avoid treating transient compression temp files as real rollout lookup
results.

## Remaining Stack

#25089 remains the separate worker PR. It is based directly on this PR
and stays behind the disabled `local_thread_store_compression` feature
flag.

The worker still has a broader coordination question: a resume or
metadata update can race with background compression while a plain file
is being replaced by `.jsonl.zst`. This PR handles the read and
materialize-before-append primitives; it does not make the worker
production-ready.

## Validation

- `just test -p codex-rollout`
- `just test -p codex-thread-store`
- `just fix -p codex-rollout`
- `just fix -p codex-thread-store`
- `just bazel-lock-check`
This commit is contained in:
jif-oai
2026-06-01 15:14:19 +02:00
committed by GitHub
parent f27bbbd49c
commit a8a6071279
16 changed files with 924 additions and 131 deletions
+10 -11
View File
@@ -1105,16 +1105,15 @@ See the Codex keymap documentation for supported actions and examples."
#[cfg(not(debug_assertions))]
let pre_loop_exit_reason = if let Some(latest_version) = upgrade_version {
let control = app
.handle_event(
tui,
&mut app_server,
AppEvent::InsertHistoryCell(Box::new(UpdateAvailableHistoryCell::new(
latest_version,
crate::update_action::get_update_action(),
))),
)
.await?;
let control = Box::pin(app.handle_event(
tui,
&mut app_server,
AppEvent::InsertHistoryCell(Box::new(UpdateAvailableHistoryCell::new(
latest_version,
crate::update_action::get_update_action(),
))),
))
.await?;
match control {
AppRunControl::Continue => None,
AppRunControl::Exit(exit_reason) => Some(exit_reason),
@@ -1131,7 +1130,7 @@ See the Codex keymap documentation for supported actions and examples."
loop {
let control = select! {
Some(event) = app_event_rx.recv() => {
match app.handle_event(tui, &mut app_server, event).await {
match Box::pin(app.handle_event(tui, &mut app_server, event)).await {
Ok(control) => control,
Err(err) => break Err(err),
}
+3 -5
View File
@@ -14,11 +14,11 @@ use crate::cwd_prompt::CwdPromptOutcome;
use crate::cwd_prompt::CwdSelection;
use crate::tui::Tui;
use codex_protocol::ThreadId;
use codex_rollout::open_rollout_line_reader;
use codex_state::StateRuntime;
use codex_utils_path as path_utils;
use serde::Deserialize;
use serde_json::Value;
use tokio::io::AsyncBufReadExt;
#[derive(Default)]
struct RolloutResumeState {
@@ -142,13 +142,11 @@ pub(crate) fn cwds_differ(current_cwd: &Path, session_cwd: &Path) -> bool {
}
async fn read_rollout_resume_state(path: &Path) -> io::Result<RolloutResumeState> {
let file = tokio::fs::File::open(path).await?;
let reader = tokio::io::BufReader::new(file);
let mut lines = reader.lines();
let mut reader = open_rollout_line_reader(path).await?;
let mut state = RolloutResumeState::default();
let mut saw_record = false;
while let Some(line) = lines.next_line().await? {
while let Some(line) = reader.next_line().await? {
let trimmed = line.trim();
if trimmed.is_empty() {
continue;