fix(app-server): avoid blocking connection cleanup (#26852)

## Why

Remote-control app-server sessions can reconnect every 5-7 seconds when
the shared transport-event queue fills. The queue's consumer handled
`ConnectionClosed` by awaiting all in-flight RPCs for the disconnected
connection. A stuck RPC therefore blocked processing of replacement
connection and initialize events until remote-control forwarding hit its
five-second timeout and reconnected again.

Related issue: N/A (internal remote-control incident investigation).

## What Changed

- Split fast RPC admission closure from draining:
`ConnectionRpcGate::close()` rejects queued and future RPCs, while
`shutdown()` continues waiting for RPCs that already started.
- Close a disconnected connection's RPC gate before spawning the
existing RPC drain and resource cleanup in a tracked background task, so
the transport-event consumer remains available without waiting for
active RPCs.
- Reap completed cleanup tasks during normal operation, drain them
during graceful shutdown, and abort them during forced shutdown.
- Add regression coverage for closing with an active RPC, rejecting
post-close requests without polling them, and preserving the existing
shutdown wait behavior.

## Verification

`just test -p codex-app-server --lib connection_rpc_gate` passes all 6
tests, including the new close-versus-drain regression coverage.
This commit is contained in:
Anton Panasenko
2026-06-08 10:20:54 -07:00
committed by GitHub
Unverified
parent 6d0e313e23
commit b128da272e
5 changed files with 117 additions and 13 deletions
+14 -1
View File
@@ -91,6 +91,7 @@ use tokio_util::sync::CancellationToken;
use tracing::Instrument;
const EXTERNAL_AUTH_REFRESH_TIMEOUT: Duration = Duration::from_secs(10);
const CONNECTION_RPC_DRAIN_TIMEOUT: Duration = Duration::from_secs(/*secs*/ 30);
#[derive(Clone)]
struct ExternalAuthRefreshBridge {
@@ -723,7 +724,19 @@ impl MessageProcessor {
connection_id: ConnectionId,
session_state: &ConnectionSessionState,
) {
session_state.rpc_gate.shutdown().await;
if timeout(
CONNECTION_RPC_DRAIN_TIMEOUT,
session_state.rpc_gate.shutdown(),
)
.await
.is_err()
{
tracing::warn!(
?connection_id,
timeout_seconds = CONNECTION_RPC_DRAIN_TIMEOUT.as_secs(),
"timed out waiting for connection RPCs to drain"
);
}
self.outgoing.connection_closed(connection_id).await;
self.fs_processor.connection_closed(connection_id).await;
self.command_exec_processor