Commit Graph
3 Commits
Author SHA1 Message Date
jifandGitHub 9f06cf1a09 Report remote sandbox denials semantically (#29424)
## Why

#29113 moved remote sandbox setup and enforcement to the exec server.
That gives the executor ownership of the platform-specific work: a Linux
executor chooses and runs a Linux sandbox even when the Codex
orchestrator is running on macOS or Windows.

It also means the orchestrator no longer knows which concrete sandbox
the executor selected. When that sandbox blocks a remote command, the
orchestrator currently sees only a failed process and can treat the
denial as an ordinary command failure. The existing sandbox approval and
retry path is then skipped.

This PR lets the executor report one portable fact:

> This command probably failed because the executor sandbox blocked it.

The executor keeps its concrete sandbox type private. The protocol sends
only the semantic result.

## Example

Suppose a local macOS Codex session asks a Linux devbox to write outside
the allowed workspace.

Before this PR:

```text
Linux sandbox blocks the write
    -> remote process exits with "Permission denied"
    -> local orchestrator sees an ordinary command failure
    -> the normal sandbox approval and retry path can be skipped
```

With this PR:

```text
Linux sandbox blocks the write
    -> executor reports sandboxDenied: true
    -> unified exec returns UnifiedExecError::SandboxDenied
    -> the existing approval prompt is shown
    -> an approved retry runs through the existing unsandboxed retry path
```

## What changes

### The executor remembers its selected sandbox

The prepared remote process now retains the executor-selected
`SandboxType`. This value never crosses the executor boundary.

Commands started without a sandbox retain `SandboxType::None` and are
never reported as sandbox denials.

### The executor uses the existing denial heuristic

The existing local denial heuristic moves from `codex-core` into the
shared `codex-sandboxing` crate.

When a sandboxed remote process exits, the executor:

1. waits the same short output grace period used by local unified exec;
2. reads the output currently available in the existing retained output
buffer;
3. runs the existing heuristic using the exit code and common denial
messages;
4. stores the yes/no result before publishing the process exit.

This deliberately matches the old local unified-exec behavior. It does
not add a new streaming classifier, another output buffer, or stronger
output-retention guarantees.

### The protocol reports a portable boolean

`process/read` gains `sandboxDenied`:

```json
{
  "exited": true,
  "exitCode": 1,
  "closed": false,
  "sandboxDenied": true
}
```

The field defaults to `false` when an older executor omits it. The
response does not expose the executor sandbox implementation or
executor-native paths.

### Unified exec uses the existing error path

The exec-server client carries `sandboxDenied` into the unified process
state. If it is true, unified exec returns the existing `SandboxDenied`
error instead of trying to classify remote output using an
orchestrator-side sandbox type.

Remote process exit remains visible as soon as the process exits. This
PR does not wait for stdout or stderr to close and does not change the
existing process lifecycle.

## Scope

This PR is intentionally limited to matching the existing local
unified-exec behavior for the initial command execution path.

It does not add:

- incremental denial tracking across the full output stream;
- new denial handling for commands completed later through
`write_stdin`;
- new guarantees for preserving the semantic flag during the narrow
reconnect-recovery race.

Those can be considered separately if the same behavior is added for
local execution.

## Test coverage

One remote end-to-end integration test covers the complete intended
flow:

```text
remote read-only sandbox
    -> denied write
    -> executor reports the denial
    -> Codex requests approval
    -> user approves
    -> retry succeeds on the remote executor
```

Existing lifecycle coverage continues to verify that remote process exit
is reported before late output streams close.
2026-06-22 19:33:28 +02:00
jifandGitHub a5229e0686 Back off registry retries during exec recovery (#28546)
## Why

PR #28512 retries a failed session recovery every 100 ms. Every Noise
recovery attempt first asks the environment registry for a fresh
connection bundle, even when the eventual failure comes from the
WebSocket or initialize handshake. During an outage, that could make
each disconnected client call the registry about 250 times during the
25-second recovery window.

## What changes

All retryable Noise recovery failures now use a separate backoff
schedule:

```text
base:    500 ms -> 1 s -> 2 s -> 4 s -> 5 s maximum
actual:  500-750 ms, 1-1.5 s, 2-3 s, 4-6 s, 5-7.5 s
```

The extra 0-50% is deterministic per-session jitter so disconnected
clients do not retry together. Direct WebSocket recovery keeps the
existing 100 ms retry because it does not re-enter the registry.
2026-06-17 11:52:23 +02:00
jifandGitHub cf17e1bc20 Resume exec-server sessions after disconnect (#28512)
Supersedes #28288 (closed).

## Why

A short WebSocket interruption currently ends every client-side process
handle, even though exec-server keeps the server session and its
processes alive for a short time.

This is especially visible for executor-backed stdio MCP servers: a
temporary connection loss becomes a permanent `Transport closed` error.
The server already has the information needed to resume the session, but
the client opens a fresh session instead of using it.

This change reconnects below the process and MCP layers. Existing
process handles stay valid, missed output is recovered, and the same
server-side processes continue running.

## State machine

One logical `ExecServerClient` stays alive while its underlying RPC
connection changes generations.

```text
                         transport closes
       +------------------------------------------------+
       |                                                v
+-------------+                                  +-------------+
|  Connected  |                                  | Recovering  |
+-------------+                                  +-------------+
       ^                                                |
       | session resumed, processes caught up           | retryable error
       +------------------------------------------------+ loops until deadline
                                                        |
                                                        | deadline or permanent error
                                                        v
                                                  +-------------+
                                                  |   Failed    |
                                                  +-------------+
```

### `Connected`

- New RPC calls use the current connection.
- Process notifications are published in sequence order.
- A disconnect only starts recovery if it came from the current
connection generation. Late events from older generations cannot replace
the active connection.

### `Recovering`

- New calls wait instead of choosing a half-connected RPC client.
- Existing process handles, wake subscriptions, and event subscriptions
stay open.
- Streaming HTTP response bodies fail immediately because their byte
streams cannot be resumed safely.
- Recovery first waits for process starts that were already in flight. A
start whose result became ambiguous is cleaned up after reconnection
instead of being silently adopted.
- The client reconnects with the learned `session_id`. The server may
briefly report that the old connection is still attached, so that error
is retried until the detach finishes.
- The notification consumer starts before the resume handshake
completes. This prevents a busy process from filling the notification
queue and blocking the initialize response.
- Before installing the new connection, the client catches up every
recoverable process with `process/read`.

### `Failed`

- Recovery stops after 25 seconds or after a permanent error.
- Waiting calls are released with one stable disconnect error.
- Existing process sessions receive a terminal failure instead of
waiting forever.

## Recovering process events

Output, exit, and close events share one sequence. During normal
operation, the client buffers early events until every lower sequence
has been published.

After reconnection, the client reads each process starting after its
last published sequence:

1. Retained output chunks are inserted by sequence number.
2. Exit and close state are reconstructed in their sequence positions.
3. Events already received as live notifications are ignored as
duplicates.
4. Newly contiguous events are published in order.
5. If the server no longer retains enough output to fill a sequence gap,
only that process is terminated and failed. The recovered connection
remains usable for other processes.

The server reports its full next event sequence for unbounded reads,
including exit and close events. Closed processes remain readable for
the same 30-second window used to retain detached sessions.

## Other details

- Detached server sessions are retained for 30 seconds, leaving margin
around the client's 25-second recovery deadline.
- Session attach and detach update the active notification sender under
the same attachment lock, so an old connection cannot clear a newly
attached sender.
- A dedicated error code distinguishes the temporary "session is still
attached" race from permanent initialization errors.
- Process starts are identity-checked on both client and server. Cleanup
from an older start cannot remove a newer process that reused the same
ID.
- Mutating requests that were already in flight when the transport
closed are not replayed, because the client cannot know whether the
server applied them. Requests started after recovery is known wait for
the replacement connection.
- We assume the server/client version stays in sync (on the before/after
this PR)

## User impact

Long-running commands and stdio MCP servers can survive a temporary
exec-server WebSocket interruption without changing process IDs or
losing output produced during the outage.
2026-06-17 10:20:39 +02:00