Files
codex/codex-rs/exec-server/src/server/session_registry.rs
T
jif cf17e1bc20 Resume exec-server sessions after disconnect (#28512)
Supersedes #28288 (closed).

## Why

A short WebSocket interruption currently ends every client-side process
handle, even though exec-server keeps the server session and its
processes alive for a short time.

This is especially visible for executor-backed stdio MCP servers: a
temporary connection loss becomes a permanent `Transport closed` error.
The server already has the information needed to resume the session, but
the client opens a fresh session instead of using it.

This change reconnects below the process and MCP layers. Existing
process handles stay valid, missed output is recovered, and the same
server-side processes continue running.

## State machine

One logical `ExecServerClient` stays alive while its underlying RPC
connection changes generations.

```text
                         transport closes
       +------------------------------------------------+
       |                                                v
+-------------+                                  +-------------+
|  Connected  |                                  | Recovering  |
+-------------+                                  +-------------+
       ^                                                |
       | session resumed, processes caught up           | retryable error
       +------------------------------------------------+ loops until deadline
                                                        |
                                                        | deadline or permanent error
                                                        v
                                                  +-------------+
                                                  |   Failed    |
                                                  +-------------+
```

### `Connected`

- New RPC calls use the current connection.
- Process notifications are published in sequence order.
- A disconnect only starts recovery if it came from the current
connection generation. Late events from older generations cannot replace
the active connection.

### `Recovering`

- New calls wait instead of choosing a half-connected RPC client.
- Existing process handles, wake subscriptions, and event subscriptions
stay open.
- Streaming HTTP response bodies fail immediately because their byte
streams cannot be resumed safely.
- Recovery first waits for process starts that were already in flight. A
start whose result became ambiguous is cleaned up after reconnection
instead of being silently adopted.
- The client reconnects with the learned `session_id`. The server may
briefly report that the old connection is still attached, so that error
is retried until the detach finishes.
- The notification consumer starts before the resume handshake
completes. This prevents a busy process from filling the notification
queue and blocking the initialize response.
- Before installing the new connection, the client catches up every
recoverable process with `process/read`.

### `Failed`

- Recovery stops after 25 seconds or after a permanent error.
- Waiting calls are released with one stable disconnect error.
- Existing process sessions receive a terminal failure instead of
waiting forever.

## Recovering process events

Output, exit, and close events share one sequence. During normal
operation, the client buffers early events until every lower sequence
has been published.

After reconnection, the client reads each process starting after its
last published sequence:

1. Retained output chunks are inserted by sequence number.
2. Exit and close state are reconstructed in their sequence positions.
3. Events already received as live notifications are ignored as
duplicates.
4. Newly contiguous events are published in order.
5. If the server no longer retains enough output to fill a sequence gap,
only that process is terminated and failed. The recovered connection
remains usable for other processes.

The server reports its full next event sequence for unbounded reads,
including exit and close events. Closed processes remain readable for
the same 30-second window used to retain detached sessions.

## Other details

- Detached server sessions are retained for 30 seconds, leaving margin
around the client's 25-second recovery deadline.
- Session attach and detach update the active notification sender under
the same attachment lock, so an old connection cannot clear a newly
attached sender.
- A dedicated error code distinguishes the temporary "session is still
attached" race from permanent initialization errors.
- Process starts are identity-checked on both client and server. Cleanup
from an older start cannot remove a newer process that reused the same
ID.
- Mutating requests that were already in flight when the transport
closed are not replayed, because the client cannot know whether the
server applied them. Requests started after recovery is known wait for
the replacement connection.
- We assume the server/client version stays in sync (on the before/after
this PR)

## User impact

Long-running commands and stdio MCP servers can survive a temporary
exec-server WebSocket interruption without changing process IDs or
losing output produced during the outage.
2026-06-17 10:20:39 +02:00

258 lines
8.0 KiB
Rust

use std::collections::HashMap;
use std::sync::Arc;
use std::sync::Mutex as StdMutex;
use std::time::Duration;
use codex_app_server_protocol::JSONRPCErrorError;
use tokio::sync::Mutex;
use uuid::Uuid;
use crate::rpc::RpcNotificationSender;
use crate::rpc::invalid_request;
use crate::rpc::session_already_attached;
use crate::server::process_handler::ProcessHandler;
#[cfg(test)]
const DETACHED_SESSION_TTL: Duration = Duration::from_millis(200);
#[cfg(not(test))]
const DETACHED_SESSION_TTL: Duration = Duration::from_secs(30);
pub(crate) struct SessionRegistry {
sessions: Mutex<HashMap<String, Arc<SessionEntry>>>,
}
struct SessionEntry {
session_id: String,
process: ProcessHandler,
attachment: StdMutex<AttachmentState>,
}
struct AttachmentState {
current_connection_id: Option<ConnectionId>,
detached_connection_id: Option<ConnectionId>,
detached_expires_at: Option<tokio::time::Instant>,
}
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
struct ConnectionId(Uuid);
impl std::fmt::Display for ConnectionId {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
self.0.fmt(f)
}
}
#[derive(Clone)]
pub(crate) struct SessionHandle {
registry: Arc<SessionRegistry>,
entry: Arc<SessionEntry>,
connection_id: ConnectionId,
}
impl SessionRegistry {
pub(crate) fn new() -> Arc<Self> {
Arc::new(Self {
sessions: Mutex::new(HashMap::new()),
})
}
pub(crate) async fn attach(
self: &Arc<Self>,
resume_session_id: Option<String>,
notifications: RpcNotificationSender,
) -> Result<SessionHandle, JSONRPCErrorError> {
enum AttachOutcome {
Attached(Arc<SessionEntry>),
Expired {
session_id: String,
entry: Arc<SessionEntry>,
},
}
let connection_id = ConnectionId(Uuid::new_v4());
let outcome = {
let mut sessions = self.sessions.lock().await;
if let Some(session_id) = resume_session_id {
let entry = sessions
.get(&session_id)
.cloned()
.ok_or_else(|| invalid_request(format!("unknown session id {session_id}")))?;
if entry.is_expired(tokio::time::Instant::now()) {
let entry = sessions.remove(&session_id).ok_or_else(|| {
invalid_request(format!("unknown session id {session_id}"))
})?;
Ok(AttachOutcome::Expired { session_id, entry })
} else if entry.has_active_connection() {
Err(session_already_attached(format!(
"session {session_id} is already attached to another connection"
)))
} else {
entry.process.set_notification_sender(Some(notifications));
entry.attach(connection_id);
Ok(AttachOutcome::Attached(entry))
}
} else {
let session_id = Uuid::new_v4().to_string();
let entry = Arc::new(SessionEntry::new(
session_id.clone(),
ProcessHandler::new(notifications),
connection_id,
));
sessions.insert(session_id, Arc::clone(&entry));
Ok(AttachOutcome::Attached(entry))
}
};
let entry = match outcome? {
AttachOutcome::Attached(entry) => entry,
AttachOutcome::Expired { session_id, entry } => {
entry.process.shutdown().await;
return Err(invalid_request(format!("unknown session id {session_id}")));
}
};
Ok(SessionHandle {
registry: Arc::clone(self),
entry,
connection_id,
})
}
async fn expire_if_detached(&self, session_id: String, connection_id: ConnectionId) {
tokio::time::sleep(DETACHED_SESSION_TTL).await;
let removed = {
let mut sessions = self.sessions.lock().await;
let Some(entry) = sessions.get(&session_id) else {
return;
};
if !entry.is_detached_connection_expired(connection_id, tokio::time::Instant::now()) {
return;
}
sessions.remove(&session_id)
};
if let Some(entry) = removed {
entry.process.shutdown().await;
}
}
}
impl Default for SessionRegistry {
fn default() -> Self {
Self {
sessions: Mutex::new(HashMap::new()),
}
}
}
impl SessionEntry {
fn new(session_id: String, process: ProcessHandler, connection_id: ConnectionId) -> Self {
Self {
session_id,
process,
attachment: StdMutex::new(AttachmentState {
current_connection_id: Some(connection_id),
detached_connection_id: None,
detached_expires_at: None,
}),
}
}
fn attach(&self, connection_id: ConnectionId) {
let mut attachment = self
.attachment
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
attachment.current_connection_id = Some(connection_id);
attachment.detached_connection_id = None;
attachment.detached_expires_at = None;
}
fn detach(&self, connection_id: ConnectionId) -> bool {
let mut attachment = self
.attachment
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
if attachment.current_connection_id != Some(connection_id) {
return false;
}
self.process.set_notification_sender(/*notifications*/ None);
attachment.current_connection_id = None;
attachment.detached_connection_id = Some(connection_id);
attachment.detached_expires_at = Some(tokio::time::Instant::now() + DETACHED_SESSION_TTL);
true
}
fn has_active_connection(&self) -> bool {
self.attachment
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.current_connection_id
.is_some()
}
fn is_attached_to(&self, connection_id: ConnectionId) -> bool {
self.attachment
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.current_connection_id
== Some(connection_id)
}
fn is_expired(&self, now: tokio::time::Instant) -> bool {
self.attachment
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.detached_expires_at
.is_some_and(|deadline| now >= deadline)
}
fn is_detached_connection_expired(
&self,
connection_id: ConnectionId,
now: tokio::time::Instant,
) -> bool {
let attachment = self
.attachment
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
attachment.current_connection_id.is_none()
&& attachment.detached_connection_id == Some(connection_id)
&& attachment
.detached_expires_at
.is_some_and(|deadline| now >= deadline)
}
}
impl SessionHandle {
pub(crate) fn session_id(&self) -> &str {
&self.entry.session_id
}
pub(crate) fn connection_id(&self) -> String {
self.connection_id.to_string()
}
pub(crate) fn is_session_attached(&self) -> bool {
self.entry.is_attached_to(self.connection_id)
}
pub(crate) fn process(&self) -> &ProcessHandler {
&self.entry.process
}
pub(crate) async fn detach(&self) {
if !self.entry.detach(self.connection_id) {
return;
}
let registry = Arc::clone(&self.registry);
let session_id = self.entry.session_id.clone();
let connection_id = self.connection_id;
tokio::spawn(async move {
registry.expire_if_detached(session_id, connection_id).await;
});
}
}