kbazzi
ccf1a18518
## Background Codex can use **Auto Review** for permission requests. Instead of asking the user immediately, Codex starts a separate locked-down reviewer session called **Guardian**, which returns a structured `allow` or `deny` assessment. The Guardian reviewer is itself a Codex session, so its model request can fail for transient infrastructure reasons such as model overload, HTTP connection failure, or response-stream disconnect. Today, any such failure immediately ends the Auto Review attempt and blocks the action. This PR adds bounded retries for failures that the existing protocol explicitly identifies as transient. Linear context: [CA-539](https://linear.app/openai/issue/CA-539/retry-auto-review-infrastructure-failures-and-fall-back-to-manual) ## What changes A Guardian review can now make at most **three total attempts**: 1. Run the review normally. 2. Retry after a jittered delay of roughly 180–220 ms if the first attempt fails with an eligible error. 3. Retry after a jittered delay of roughly 360–440 ms if the second attempt also fails with an eligible error. All attempts share the original review deadline. Jitter spreads retries from concurrent clients to reduce synchronized load during broader outages. The retries do not reset the user's maximum wait time, and the backoff waits terminate early if the review is cancelled or the deadline expires. Before retrying, the existing Guardian session lifecycle decides whether the session remains usable. Healthy trunks are reused, broken trunks are removed by the existing cleanup path, and ephemeral sessions continue to clean themselves up. The review still emits one logical lifecycle to clients. Recoverable intermediate failures do not produce warnings or terminal events. ## Retry policy ### Retried up to twice - model/server overload - HTTP connection failure - response-stream connection failure - response-stream disconnect - internal server error - a final reviewer message that cannot be parsed as the required Guardian assessment ### Not retried - bad or invalid requests - authentication failures - usage limits - cyber-policy failures - errors without a structured category - a request that already exhausted the lower-level Responses retry budget - a completed Guardian turn with no assessment payload - prompt-construction failures - Guardian review timeout - cancellation or abort - a valid `deny` assessment The session-error classification uses `ErrorEvent.codex_error_info`; it does not inspect error-message strings. ## Implementation notes - `wait_for_guardian_review` preserves the complete `ErrorEvent`, including structured `codex_error_info`. - Guardian session failures preserve the original message and optional structured `CodexErrorInfo`. - The retry policy classifies the explicitly transient `CodexErrorInfo` variants; unknown, absent, and deterministic categories are not retried. - The Guardian session manager receives the caller's deadline rather than creating a new timeout per attempt. - Analytics record the final `attempt_count`. - Retry orchestration does not add a separate session-cleanup protocol; it relies on the existing trunk and ephemeral lifecycle decisions. ## Automated testing Focused Guardian coverage verifies: - every supported transient `CodexErrorInfo` is classified as retryable, while absent and non-transient categories are not; - structured transient session failure -> retry -> approval with the healthy trunk reused; - two invalid Guardian responses -> third attempt -> approval, with exactly three requests; - three invalid responses -> existing fail-closed result, with exactly three requests and one terminal lifecycle; - valid denial, missing payload, invalid request, timeout, cancellation, and prompt/session construction failures are not retried; - retry eligibility ends after the third attempt; - retry delays use the shared exponential backoff helper and remain within the expected jitter bounds; - cancellation and deadline expiry interrupt the backoff wait; - healthy trunks are reused across retryable failures; - broken event streams remove the trunk through the existing lifecycle cleanup; - an ephemeral retry does not disturb a concurrent trunk review. Validation performed: - `just test -p codex-core guardian_review_ guardian_ephemeral_retry_preserves_parallel_trunk_and_fork_history run_review_removes_trunk_when_event_stream_is_broken` — **42 passed**; - `just test -p codex-analytics` — **71 passed**; - scoped Clippy fixes for `codex-core` and `codex-analytics` passed. A prior full `codex-core` run had unrelated environment-sensitive failures outside Guardian coverage. ## Manual QA The focused integration tests use the local mock Responses server to inspect exact request counts and emitted lifecycle events. They confirm that retries are internal, a successful later attempt supplies the final decision, non-retryable failures issue only one request, and exhausted retries emit only one terminal result.
codex-core
This crate implements the business logic for Codex. It is designed to be used by the various Codex UIs written in Rust.
Dependencies
Note that codex-core makes some assumptions about certain helper utilities being available in the environment. Currently, this support matrix is:
macOS
Expects /usr/bin/sandbox-exec to be present.
When using the workspace-write sandbox policy, the Seatbelt profile allows
writes under the configured writable roots while keeping .git (directory or
pointer file), the resolved gitdir: target, and .codex read-only.
Network access and filesystem read/write roots are controlled by
SandboxPolicy. Seatbelt consumes the resolved policy and enforces it.
Seatbelt also keeps the legacy default preferences read access
(user-preference-read) needed for cfprefs-backed macOS behavior.
Linux
Expects the binary containing codex-core to run the equivalent of codex sandbox when arg0 is codex-linux-sandbox. See the codex-arg0 crate for details.
Legacy SandboxPolicy / sandbox_mode configs are still supported on Linux.
They can continue to use the legacy Landlock path when the split filesystem
policy is sandbox-equivalent to the legacy model after cwd resolution.
Split filesystem policies that need direct FileSystemSandboxPolicy
enforcement, such as read-only or denied carveouts under a broader writable
root, automatically route through bubblewrap. The legacy Landlock path is used
only when the split filesystem policy round-trips through the legacy
SandboxPolicy model without changing semantics. That includes overlapping
cases like /repo = write, /repo/a = none, /repo/a/b = write, where the
more specific writable child must reopen under a denied parent.
The Linux sandbox helper prefers the first bwrap found on PATH outside the
current working directory whenever it is available. If bwrap is present but
too old to support --argv0, the helper keeps using system bubblewrap and
switches to a no---argv0 compatibility path for the inner re-exec. If
bwrap is missing, it falls back to the bundled codex-resources/bwrap
binary shipped with Codex and Codex surfaces a startup warning through its
normal notification path instead of printing directly from the sandbox helper.
Codex also surfaces a startup warning when bubblewrap cannot create user
namespaces. WSL2 uses the normal Linux bubblewrap path. WSL1 is not supported
for bubblewrap sandboxing because it cannot create the required user
namespaces, so Codex rejects sandboxed shell commands that would enter the
bubblewrap path before invoking bwrap.
Windows
Legacy SandboxPolicy / sandbox_mode configs are still supported on
Windows. Legacy read-only and workspace-write policies imply full
filesystem read access; exact readable roots are represented by split
filesystem policies instead.
The elevated Windows sandbox also supports:
- legacy
ReadOnlyandWorkspaceWritebehavior - split filesystem policies that need exact readable roots, exact writable roots, or extra read-only carveouts under writable roots
- backend-managed system read roots required for basic execution, such as
C:\Windows,C:\Program Files,C:\Program Files (x86), andC:\ProgramData, when a split filesystem policy requests platform defaults
The unelevated restricted-token backend still supports the legacy full-read
Windows model for legacy ReadOnly and WorkspaceWrite behavior. It also
supports a narrow split-filesystem subset: full-read split policies whose
writable roots still match the legacy WorkspaceWrite root set, but add extra
read-only carveouts under those writable roots.
New [permissions] / split filesystem policies remain supported on Windows
only when they can be enforced directly by the selected Windows backend or
round-trip through the legacy SandboxPolicy model without changing semantics.
Policies that would require direct explicit unreadable carveouts (none) or
reopened writable descendants under read-only carveouts still fail closed
instead of running with weaker enforcement.
All Platforms
Expects the binary containing codex-core to simulate the virtual
apply_patch CLI when arg1 is --codex-run-as-apply-patch. See the
codex-arg0 crate for details.