mirror of
https://github.com/pchuan98/codex.git
synced 2026-07-01 00:31:56 +08:00
fix(app-server): avoid blocking connection cleanup (#26852)
## Why Remote-control app-server sessions can reconnect every 5-7 seconds when the shared transport-event queue fills. The queue's consumer handled `ConnectionClosed` by awaiting all in-flight RPCs for the disconnected connection. A stuck RPC therefore blocked processing of replacement connection and initialize events until remote-control forwarding hit its five-second timeout and reconnected again. Related issue: N/A (internal remote-control incident investigation). ## What Changed - Split fast RPC admission closure from draining: `ConnectionRpcGate::close()` rejects queued and future RPCs, while `shutdown()` continues waiting for RPCs that already started. - Close a disconnected connection's RPC gate before spawning the existing RPC drain and resource cleanup in a tracked background task, so the transport-event consumer remains available without waiting for active RPCs. - Reap completed cleanup tasks during normal operation, drain them during graceful shutdown, and abort them during forced shutdown. - Add regression coverage for closing with an active RPC, rejecting post-close requests without polling them, and preserving the existing shutdown wait behavior. ## Verification `just test -p codex-app-server --lib connection_rpc_gate` passes all 6 tests, including the new close-versus-drain regression coverage.
This commit is contained in:
committed by
GitHub
Unverified
parent
6d0e313e23
commit
b128da272e
@@ -91,6 +91,7 @@ use tokio_util::sync::CancellationToken;
|
||||
use tracing::Instrument;
|
||||
|
||||
const EXTERNAL_AUTH_REFRESH_TIMEOUT: Duration = Duration::from_secs(10);
|
||||
const CONNECTION_RPC_DRAIN_TIMEOUT: Duration = Duration::from_secs(/*secs*/ 30);
|
||||
|
||||
#[derive(Clone)]
|
||||
struct ExternalAuthRefreshBridge {
|
||||
@@ -723,7 +724,19 @@ impl MessageProcessor {
|
||||
connection_id: ConnectionId,
|
||||
session_state: &ConnectionSessionState,
|
||||
) {
|
||||
session_state.rpc_gate.shutdown().await;
|
||||
if timeout(
|
||||
CONNECTION_RPC_DRAIN_TIMEOUT,
|
||||
session_state.rpc_gate.shutdown(),
|
||||
)
|
||||
.await
|
||||
.is_err()
|
||||
{
|
||||
tracing::warn!(
|
||||
?connection_id,
|
||||
timeout_seconds = CONNECTION_RPC_DRAIN_TIMEOUT.as_secs(),
|
||||
"timed out waiting for connection RPCs to drain"
|
||||
);
|
||||
}
|
||||
self.outgoing.connection_closed(connection_id).await;
|
||||
self.fs_processor.connection_closed(connection_id).await;
|
||||
self.command_exec_processor
|
||||
|
||||
Reference in New Issue
Block a user