mirror of
https://github.com/pchuan98/codex.git
synced 2026-07-01 00:31:56 +08:00
cf17e1bc20
Supersedes #28288 (closed). ## Why A short WebSocket interruption currently ends every client-side process handle, even though exec-server keeps the server session and its processes alive for a short time. This is especially visible for executor-backed stdio MCP servers: a temporary connection loss becomes a permanent `Transport closed` error. The server already has the information needed to resume the session, but the client opens a fresh session instead of using it. This change reconnects below the process and MCP layers. Existing process handles stay valid, missed output is recovered, and the same server-side processes continue running. ## State machine One logical `ExecServerClient` stays alive while its underlying RPC connection changes generations. ```text transport closes +------------------------------------------------+ | v +-------------+ +-------------+ | Connected | | Recovering | +-------------+ +-------------+ ^ | | session resumed, processes caught up | retryable error +------------------------------------------------+ loops until deadline | | deadline or permanent error v +-------------+ | Failed | +-------------+ ``` ### `Connected` - New RPC calls use the current connection. - Process notifications are published in sequence order. - A disconnect only starts recovery if it came from the current connection generation. Late events from older generations cannot replace the active connection. ### `Recovering` - New calls wait instead of choosing a half-connected RPC client. - Existing process handles, wake subscriptions, and event subscriptions stay open. - Streaming HTTP response bodies fail immediately because their byte streams cannot be resumed safely. - Recovery first waits for process starts that were already in flight. A start whose result became ambiguous is cleaned up after reconnection instead of being silently adopted. - The client reconnects with the learned `session_id`. The server may briefly report that the old connection is still attached, so that error is retried until the detach finishes. - The notification consumer starts before the resume handshake completes. This prevents a busy process from filling the notification queue and blocking the initialize response. - Before installing the new connection, the client catches up every recoverable process with `process/read`. ### `Failed` - Recovery stops after 25 seconds or after a permanent error. - Waiting calls are released with one stable disconnect error. - Existing process sessions receive a terminal failure instead of waiting forever. ## Recovering process events Output, exit, and close events share one sequence. During normal operation, the client buffers early events until every lower sequence has been published. After reconnection, the client reads each process starting after its last published sequence: 1. Retained output chunks are inserted by sequence number. 2. Exit and close state are reconstructed in their sequence positions. 3. Events already received as live notifications are ignored as duplicates. 4. Newly contiguous events are published in order. 5. If the server no longer retains enough output to fill a sequence gap, only that process is terminated and failed. The recovered connection remains usable for other processes. The server reports its full next event sequence for unbounded reads, including exit and close events. Closed processes remain readable for the same 30-second window used to retain detached sessions. ## Other details - Detached server sessions are retained for 30 seconds, leaving margin around the client's 25-second recovery deadline. - Session attach and detach update the active notification sender under the same attachment lock, so an old connection cannot clear a newly attached sender. - A dedicated error code distinguishes the temporary "session is still attached" race from permanent initialization errors. - Process starts are identity-checked on both client and server. Cleanup from an older start cannot remove a newer process that reused the same ID. - Mutating requests that were already in flight when the transport closed are not replayed, because the client cannot know whether the server applied them. Requests started after recovery is known wait for the replacement connection. - We assume the server/client version stays in sync (on the before/after this PR) ## User impact Long-running commands and stdio MCP servers can survive a temporary exec-server WebSocket interruption without changing process IDs or losing output produced during the outage.
258 lines
8.0 KiB
Rust
258 lines
8.0 KiB
Rust
use std::collections::HashMap;
|
|
use std::sync::Arc;
|
|
use std::sync::Mutex as StdMutex;
|
|
use std::time::Duration;
|
|
|
|
use codex_app_server_protocol::JSONRPCErrorError;
|
|
use tokio::sync::Mutex;
|
|
use uuid::Uuid;
|
|
|
|
use crate::rpc::RpcNotificationSender;
|
|
use crate::rpc::invalid_request;
|
|
use crate::rpc::session_already_attached;
|
|
use crate::server::process_handler::ProcessHandler;
|
|
|
|
#[cfg(test)]
|
|
const DETACHED_SESSION_TTL: Duration = Duration::from_millis(200);
|
|
#[cfg(not(test))]
|
|
const DETACHED_SESSION_TTL: Duration = Duration::from_secs(30);
|
|
|
|
pub(crate) struct SessionRegistry {
|
|
sessions: Mutex<HashMap<String, Arc<SessionEntry>>>,
|
|
}
|
|
|
|
struct SessionEntry {
|
|
session_id: String,
|
|
process: ProcessHandler,
|
|
attachment: StdMutex<AttachmentState>,
|
|
}
|
|
|
|
struct AttachmentState {
|
|
current_connection_id: Option<ConnectionId>,
|
|
detached_connection_id: Option<ConnectionId>,
|
|
detached_expires_at: Option<tokio::time::Instant>,
|
|
}
|
|
|
|
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
|
struct ConnectionId(Uuid);
|
|
|
|
impl std::fmt::Display for ConnectionId {
|
|
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
|
self.0.fmt(f)
|
|
}
|
|
}
|
|
|
|
#[derive(Clone)]
|
|
pub(crate) struct SessionHandle {
|
|
registry: Arc<SessionRegistry>,
|
|
entry: Arc<SessionEntry>,
|
|
connection_id: ConnectionId,
|
|
}
|
|
|
|
impl SessionRegistry {
|
|
pub(crate) fn new() -> Arc<Self> {
|
|
Arc::new(Self {
|
|
sessions: Mutex::new(HashMap::new()),
|
|
})
|
|
}
|
|
|
|
pub(crate) async fn attach(
|
|
self: &Arc<Self>,
|
|
resume_session_id: Option<String>,
|
|
notifications: RpcNotificationSender,
|
|
) -> Result<SessionHandle, JSONRPCErrorError> {
|
|
enum AttachOutcome {
|
|
Attached(Arc<SessionEntry>),
|
|
Expired {
|
|
session_id: String,
|
|
entry: Arc<SessionEntry>,
|
|
},
|
|
}
|
|
|
|
let connection_id = ConnectionId(Uuid::new_v4());
|
|
let outcome = {
|
|
let mut sessions = self.sessions.lock().await;
|
|
if let Some(session_id) = resume_session_id {
|
|
let entry = sessions
|
|
.get(&session_id)
|
|
.cloned()
|
|
.ok_or_else(|| invalid_request(format!("unknown session id {session_id}")))?;
|
|
if entry.is_expired(tokio::time::Instant::now()) {
|
|
let entry = sessions.remove(&session_id).ok_or_else(|| {
|
|
invalid_request(format!("unknown session id {session_id}"))
|
|
})?;
|
|
Ok(AttachOutcome::Expired { session_id, entry })
|
|
} else if entry.has_active_connection() {
|
|
Err(session_already_attached(format!(
|
|
"session {session_id} is already attached to another connection"
|
|
)))
|
|
} else {
|
|
entry.process.set_notification_sender(Some(notifications));
|
|
entry.attach(connection_id);
|
|
Ok(AttachOutcome::Attached(entry))
|
|
}
|
|
} else {
|
|
let session_id = Uuid::new_v4().to_string();
|
|
let entry = Arc::new(SessionEntry::new(
|
|
session_id.clone(),
|
|
ProcessHandler::new(notifications),
|
|
connection_id,
|
|
));
|
|
sessions.insert(session_id, Arc::clone(&entry));
|
|
Ok(AttachOutcome::Attached(entry))
|
|
}
|
|
};
|
|
let entry = match outcome? {
|
|
AttachOutcome::Attached(entry) => entry,
|
|
AttachOutcome::Expired { session_id, entry } => {
|
|
entry.process.shutdown().await;
|
|
return Err(invalid_request(format!("unknown session id {session_id}")));
|
|
}
|
|
};
|
|
|
|
Ok(SessionHandle {
|
|
registry: Arc::clone(self),
|
|
entry,
|
|
connection_id,
|
|
})
|
|
}
|
|
|
|
async fn expire_if_detached(&self, session_id: String, connection_id: ConnectionId) {
|
|
tokio::time::sleep(DETACHED_SESSION_TTL).await;
|
|
|
|
let removed = {
|
|
let mut sessions = self.sessions.lock().await;
|
|
let Some(entry) = sessions.get(&session_id) else {
|
|
return;
|
|
};
|
|
if !entry.is_detached_connection_expired(connection_id, tokio::time::Instant::now()) {
|
|
return;
|
|
}
|
|
sessions.remove(&session_id)
|
|
};
|
|
|
|
if let Some(entry) = removed {
|
|
entry.process.shutdown().await;
|
|
}
|
|
}
|
|
}
|
|
|
|
impl Default for SessionRegistry {
|
|
fn default() -> Self {
|
|
Self {
|
|
sessions: Mutex::new(HashMap::new()),
|
|
}
|
|
}
|
|
}
|
|
|
|
impl SessionEntry {
|
|
fn new(session_id: String, process: ProcessHandler, connection_id: ConnectionId) -> Self {
|
|
Self {
|
|
session_id,
|
|
process,
|
|
attachment: StdMutex::new(AttachmentState {
|
|
current_connection_id: Some(connection_id),
|
|
detached_connection_id: None,
|
|
detached_expires_at: None,
|
|
}),
|
|
}
|
|
}
|
|
|
|
fn attach(&self, connection_id: ConnectionId) {
|
|
let mut attachment = self
|
|
.attachment
|
|
.lock()
|
|
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
|
attachment.current_connection_id = Some(connection_id);
|
|
attachment.detached_connection_id = None;
|
|
attachment.detached_expires_at = None;
|
|
}
|
|
|
|
fn detach(&self, connection_id: ConnectionId) -> bool {
|
|
let mut attachment = self
|
|
.attachment
|
|
.lock()
|
|
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
|
if attachment.current_connection_id != Some(connection_id) {
|
|
return false;
|
|
}
|
|
|
|
self.process.set_notification_sender(/*notifications*/ None);
|
|
attachment.current_connection_id = None;
|
|
attachment.detached_connection_id = Some(connection_id);
|
|
attachment.detached_expires_at = Some(tokio::time::Instant::now() + DETACHED_SESSION_TTL);
|
|
true
|
|
}
|
|
|
|
fn has_active_connection(&self) -> bool {
|
|
self.attachment
|
|
.lock()
|
|
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
|
.current_connection_id
|
|
.is_some()
|
|
}
|
|
|
|
fn is_attached_to(&self, connection_id: ConnectionId) -> bool {
|
|
self.attachment
|
|
.lock()
|
|
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
|
.current_connection_id
|
|
== Some(connection_id)
|
|
}
|
|
|
|
fn is_expired(&self, now: tokio::time::Instant) -> bool {
|
|
self.attachment
|
|
.lock()
|
|
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
|
.detached_expires_at
|
|
.is_some_and(|deadline| now >= deadline)
|
|
}
|
|
|
|
fn is_detached_connection_expired(
|
|
&self,
|
|
connection_id: ConnectionId,
|
|
now: tokio::time::Instant,
|
|
) -> bool {
|
|
let attachment = self
|
|
.attachment
|
|
.lock()
|
|
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
|
attachment.current_connection_id.is_none()
|
|
&& attachment.detached_connection_id == Some(connection_id)
|
|
&& attachment
|
|
.detached_expires_at
|
|
.is_some_and(|deadline| now >= deadline)
|
|
}
|
|
}
|
|
|
|
impl SessionHandle {
|
|
pub(crate) fn session_id(&self) -> &str {
|
|
&self.entry.session_id
|
|
}
|
|
|
|
pub(crate) fn connection_id(&self) -> String {
|
|
self.connection_id.to_string()
|
|
}
|
|
|
|
pub(crate) fn is_session_attached(&self) -> bool {
|
|
self.entry.is_attached_to(self.connection_id)
|
|
}
|
|
|
|
pub(crate) fn process(&self) -> &ProcessHandler {
|
|
&self.entry.process
|
|
}
|
|
|
|
pub(crate) async fn detach(&self) {
|
|
if !self.entry.detach(self.connection_id) {
|
|
return;
|
|
}
|
|
|
|
let registry = Arc::clone(&self.registry);
|
|
let session_id = self.entry.session_id.clone();
|
|
let connection_id = self.connection_id;
|
|
tokio::spawn(async move {
|
|
registry.expire_if_detached(session_id, connection_id).await;
|
|
});
|
|
}
|
|
}
|