[codex] control automatic realtime handoff delivery (#27986)

## What Built on the realtime speech-control plumbing merged in #27917. - Add optional `codexResponseHandoffPrefix` to `thread/realtime/start`. - Apply that prefix only to automatic V1 commentary sent through `conversation.handoff.append`; final answers remain unprefixed. - Add opt-in `clientManagedHandoffs`. When true, core suppresses automatic response handoffs and completion output so delivery is controlled by explicit client append APIs. - Preserve existing automatic behavior by default. `codexResponsesAsItems: true` continues to select item routing when client-managed mode is disabled. ## Why Voice clients need two delivery policies: automatic background context with silent commentary instructions and fully client-owned handoffs. Phase-aware prefixing keeps routine commentary silent without suppressing the final answer, while client-managed mode lets an app decide exactly which updates to append. ## Validation - `just fmt` - `cargo test -p codex-app-server-protocol serialize_thread_realtime_start` - `RUST_MIN_STACK=16777216 cargo test -p codex-core --test all conversation_handoff_persists_across_item_done_until_turn_complete` - `RUST_MIN_STACK=16777216 cargo test -p codex-app-server --test all webrtc_v1_client_managed_handoffs_disable_automatic_output` - `RUST_MIN_STACK=16777216 cargo test -p codex-app-server --test all webrtc_v1_final_automatic_handoff_omits_silent_prefix` - `cargo build -p codex-cli --bin codex` - Local Codex Apps compatibility check: 43 focused webview tests passed, and a live voice session routed through the source-built app-server. The explicit `RUST_MIN_STACK` avoids a macOS Tokio test-worker stack overflow seen with the default test environment.
2026-07-01 00:31:56 +08:00 · 2026-06-18 02:22:29 +00:00
parent a306ac4ee3
commit 683bd170dc
13 changed files with 355 additions and 24 deletions
@@ -165,7 +165,7 @@ Example with notification opt-out:
 - `thread/inject_items` — append raw Responses API items to a loaded thread’s model-visible history without starting a user turn; returns `{}` on success.
 - `turn/steer` — add user input to an already in-flight regular turn without starting a new turn; returns the active `turnId` that accepted the input. `clientUserMessageId` is optional; when supplied, the corresponding `userMessage` item echoes it as `clientId`. Review and manual compaction turns reject `turn/steer`.
 - `turn/interrupt` — request cancellation of an in-flight turn by `(thread_id, turn_id)`; success is an empty `{}` response and the turn finishes with `status: "interrupted"`.
- `thread/realtime/start` — start a thread-scoped realtime session (experimental); pass `outputModality: "text"` or `outputModality: "audio"` to choose model output, optionally pass `model` and `version` to override configured realtime selection for this session only, and pass `includeStartupContext: false` to omit Codex's generated startup context. By default, automatic Codex text follows the protocol's speakable output path. Pass `codexResponsesAsItems: true` to send automatic Codex responses as realtime conversation items instead, and optionally pass `codexResponseItemPrefix` to prepend experiment instructions to those items. Returns `{}` and streams `thread/realtime/*` notifications. Omit `transport` for the websocket transport, or pass `{ "type": "webrtc", "sdp": "..." }` to create a WebRTC session from a browser-generated SDP offer; the remote answer SDP is emitted as `thread/realtime/sdp`.
+- `thread/realtime/start` — start a thread-scoped realtime session (experimental); pass `outputModality: "text"` or `outputModality: "audio"` to choose model output, optionally pass `model` and `version` to override configured realtime selection for this session only, and pass `includeStartupContext: false` to omit Codex's generated startup context. By default, automatic Codex text follows the protocol's speakable output path. Pass `clientManagedHandoffs: true` to disable automatic Codex response delivery so only the client's explicit append calls produce handoffs. Pass `codexResponsesAsItems: true` to send automatic Codex responses as realtime conversation items instead, and optionally pass `codexResponseItemPrefix` to prepend experiment instructions to those items. For V1 sessions, pass `codexResponseHandoffPrefix` while item mode is disabled to route automatic Codex commentary through `conversation.handoff.append` with that prefix; final answers remain unprefixed. Returns `{}` and streams `thread/realtime/*` notifications. Omit `transport` for the websocket transport, or pass `{ "type": "webrtc", "sdp": "..." }` to create a WebRTC session from a browser-generated SDP offer; the remote answer SDP is emitted as `thread/realtime/sdp`.
 - `thread/realtime/appendAudio` — append an input audio chunk to the active realtime session (experimental); returns `{}`.
 - `thread/realtime/appendText` — append text input to the active realtime session with a required `role` of `user` or `developer` (experimental); returns `{}`. Older clients that omit `role` default to `user`.
 - `thread/realtime/appendSpeech` — append text that the realtime model should speak to the user (experimental); returns `{}`.
@@ -882,12 +882,19 @@ Clients may also pass `model` and `version` on `thread/realtime/start` to select
 different realtime session configuration without changing thread or user config.
 Pass `includeStartupContext: false` to skip Codex's startup context for this
 session while still using the selected backend prompt.
+Pass `clientManagedHandoffs: true` to suppress automatic Codex response handoffs
+and items. The client can then choose which updates to deliver with
+`thread/realtime/appendText` or `thread/realtime/appendSpeech`.
 Pass `codexResponsesAsItems: true` to inject automatic Codex responses with
 `conversation.item.create` instead of the protocol's default speakable output
 path. When using that mode, `codexResponseItemPrefix` can prepend short
 experiment instructions to each automatic Codex response item. Omit
 `codexResponsesAsItems`, or pass `false`, to preserve the default speakable
-behavior. Call
+behavior. For V1 sessions, `codexResponseHandoffPrefix` instead routes automatic
+Codex commentary through `conversation.handoff.append` and prepends the provided
+text. Final answers remain unprefixed. Item mode takes precedence when
+`codexResponsesAsItems` is true.
+Call
 `thread/realtime/appendText` to append app-provided realtime text items, or
 `thread/realtime/appendSpeech` when the app decides a realtime update should be
 spoken.
@@ -931,8 +931,10 @@ impl TurnRequestProcessor {
            thread.as_ref(),
            Op::RealtimeConversationStart(ConversationStartParams {
                architecture: params.architecture,
+                client_managed_handoffs: params.client_managed_handoffs.unwrap_or(false),
                codex_responses_as_items: params.codex_responses_as_items.unwrap_or(false),
                codex_response_item_prefix: params.codex_response_item_prefix,
+                codex_response_handoff_prefix: params.codex_response_handoff_prefix,
                model: params.model,
                output_modality: params.output_modality,
                include_startup_context: params.include_startup_context.unwrap_or(true),
@@ -80,8 +80,10 @@ async fn realtime_conversation_start_requires_experimental_api_capability() -> R
    let request_id = mcp
        .send_thread_realtime_start_request(ThreadRealtimeStartParams {
            architecture: None,
+            client_managed_handoffs: None,
            codex_responses_as_items: None,
            codex_response_item_prefix: None,
+            codex_response_handoff_prefix: None,
            thread_id: "thr_123".to_string(),
            model: None,
            output_modality: RealtimeOutputModality::Audio,
@@ -192,8 +194,10 @@ async fn realtime_webrtc_start_requires_experimental_api_capability() -> Result<
    let request_id = mcp
        .send_thread_realtime_start_request(ThreadRealtimeStartParams {
            architecture: None,
+            client_managed_handoffs: None,
            codex_responses_as_items: None,
            codex_response_item_prefix: None,
+            codex_response_handoff_prefix: None,
            thread_id: "thr_123".to_string(),
            model: None,
            output_modality: RealtimeOutputModality::Audio,
@@ -86,6 +86,8 @@ const V2_HANDOFF_COMPLETE_ACKNOWLEDGEMENT: &str =
    "Background agent finished. Use the preceding [BACKEND] messages as the result.";
 const RESPONSE_ITEM_PREFIX: &str =
    "Use the following context to inform future responses, but do not speak it to the user.";
+const RESPONSE_HANDOFF_PREFIX: &str =
+    "Silent Codex context. Do not speak, acknowledge, or summarize this item.";

 #[derive(Debug, Clone, Copy)]
 enum StartupContextConfig<'a> {
@@ -313,8 +315,9 @@ impl RealtimeE2eHarness {
    }

    async fn start_webrtc_realtime(&mut self, offer_sdp: &str) -> Result<StartedWebrtcRealtime> {
-        self.start_webrtc_realtime_with_codex_responses_as_items(
-            offer_sdp, /*codex_responses_as_items*/ None,
+        self.start_webrtc_realtime_with_codex_response_routing(
+            offer_sdp, /*client_managed_handoffs*/ None,
+            /*codex_responses_as_items*/ None, /*codex_response_handoff_prefix*/ None,
        )
        .await
    }
@@ -323,17 +326,21 @@ impl RealtimeE2eHarness {
        &mut self,
        offer_sdp: &str,
    ) -> Result<StartedWebrtcRealtime> {
-        self.start_webrtc_realtime_with_codex_responses_as_items(
+        self.start_webrtc_realtime_with_codex_response_routing(
            offer_sdp,
+            /*client_managed_handoffs*/ None,
            /*codex_responses_as_items*/ Some(true),
+            /*codex_response_handoff_prefix*/ None,
        )
        .await
    }

-    async fn start_webrtc_realtime_with_codex_responses_as_items(
+    async fn start_webrtc_realtime_with_codex_response_routing(
        &mut self,
        offer_sdp: &str,
+        client_managed_handoffs: Option<bool>,
        codex_responses_as_items: Option<bool>,
+        codex_response_handoff_prefix: Option<&str>,
    ) -> Result<StartedWebrtcRealtime> {
        // Starts realtime through the public JSON-RPC method, then waits for the same client-visible
        // notifications a desktop app needs: started first, SDP answer second.
@@ -341,10 +348,12 @@ impl RealtimeE2eHarness {
            .mcp
            .send_thread_realtime_start_request(ThreadRealtimeStartParams {
                architecture: None,
+                client_managed_handoffs,
                thread_id: self.thread_id.clone(),
                codex_response_item_prefix: codex_responses_as_items
                    .unwrap_or(false)
                    .then(|| RESPONSE_ITEM_PREFIX.to_string()),
+                codex_response_handoff_prefix: codex_response_handoff_prefix.map(str::to_string),
                codex_responses_as_items,
                model: None,
                output_modality: RealtimeOutputModality::Audio,
@@ -611,8 +620,10 @@ async fn realtime_conversation_streams_v2_notifications() -> Result<()> {
    let start_request_id = mcp
        .send_thread_realtime_start_request(ThreadRealtimeStartParams {
            architecture: None,
+            client_managed_handoffs: None,
            codex_responses_as_items: None,
            codex_response_item_prefix: None,
+            codex_response_handoff_prefix: None,
            thread_id: thread_start.thread.id.clone(),
            model: Some("realtime-treatment-model".to_string()),
            output_modality: RealtimeOutputModality::Audio,
@@ -867,8 +878,10 @@ async fn realtime_start_can_skip_startup_context() -> Result<()> {
    let start_request_id = mcp
        .send_thread_realtime_start_request(ThreadRealtimeStartParams {
            architecture: None,
+            client_managed_handoffs: None,
            codex_responses_as_items: None,
            codex_response_item_prefix: None,
+            codex_response_handoff_prefix: None,
            thread_id: thread_start.thread.id.clone(),
            model: None,
            output_modality: RealtimeOutputModality::Audio,
@@ -964,8 +977,10 @@ async fn realtime_text_output_modality_requests_text_output_and_final_transcript
    let start_request_id = mcp
        .send_thread_realtime_start_request(ThreadRealtimeStartParams {
            architecture: None,
+            client_managed_handoffs: None,
            codex_responses_as_items: None,
            codex_response_item_prefix: None,
+            codex_response_handoff_prefix: None,
            thread_id: thread_start.thread.id.clone(),
            model: None,
            output_modality: RealtimeOutputModality::Text,
@@ -1144,8 +1159,10 @@ async fn realtime_conversation_stop_emits_closed_notification() -> Result<()> {
    let start_request_id = mcp
        .send_thread_realtime_start_request(ThreadRealtimeStartParams {
            architecture: None,
+            client_managed_handoffs: None,
            codex_responses_as_items: None,
            codex_response_item_prefix: None,
+            codex_response_handoff_prefix: None,
            thread_id: thread_start.thread.id.clone(),
            model: None,
            output_modality: RealtimeOutputModality::Audio,
@@ -1247,8 +1264,10 @@ async fn realtime_webrtc_start_emits_sdp_notification() -> Result<()> {
    let start_request_id = mcp
        .send_thread_realtime_start_request(ThreadRealtimeStartParams {
            architecture: None,
+            client_managed_handoffs: None,
            codex_responses_as_items: None,
            codex_response_item_prefix: None,
+            codex_response_handoff_prefix: None,
            thread_id: thread_id.clone(),
            model: None,
            output_modality: RealtimeOutputModality::Audio,
@@ -1480,6 +1499,135 @@ async fn webrtc_v1_default_automatic_output_uses_handoff_append() -> Result<()>
    Ok(())
 }

+#[tokio::test]
+async fn webrtc_v1_client_managed_handoffs_disable_automatic_output() -> Result<()> {
+    skip_if_no_network!(Ok(()));
+
+    let mut harness = RealtimeE2eHarness::new(
+        RealtimeTestVersion::V1,
+        main_loop_responses(vec![create_final_assistant_message_sse_response(
+            "client-managed output",
+        )?]),
+        realtime_sideband(vec![realtime_sideband_connection(vec![
+            vec![session_updated("sess_v1_client_managed_handoffs")],
+            vec![],
+        ])]),
+    )
+    .await?;
+
+    let started = harness
+        .start_webrtc_realtime_with_codex_response_routing(
+            "v=offer\r\n",
+            /*client_managed_handoffs*/ Some(true),
+            /*codex_responses_as_items*/ None,
+            /*codex_response_handoff_prefix*/ None,
+        )
+        .await?;
+    assert_eq!(started.started.version, RealtimeConversationVersion::V1);
+    assert_v1_session_update(&harness.sideband_outbound_request(/*request_index*/ 0).await)?;
+
+    let turn_request_id = harness
+        .mcp
+        .send_turn_start_request(TurnStartParams {
+            thread_id: harness.thread_id.clone(),
+            input: vec![V2UserInput::Text {
+                text: "leave realtime delivery to the client".to_string(),
+                text_elements: Vec::new(),
+            }],
+            ..Default::default()
+        })
+        .await?;
+    let turn_response: JSONRPCResponse = timeout(
+        DEFAULT_TIMEOUT,
+        harness
+            .mcp
+            .read_stream_until_response_message(RequestId::Integer(turn_request_id)),
+    )
+    .await??;
+    let _: TurnStartResponse = to_response(turn_response)?;
+    let _ = harness
+        .read_notification::<TurnCompletedNotification>("turn/completed")
+        .await?;
+
+    let automatic_handoff = timeout(
+        Duration::from_millis(200),
+        harness
+            .realtime_server
+            .wait_for_request(/*connection_index*/ 0, /*request_index*/ 1),
+    )
+    .await;
+    assert!(
+        automatic_handoff.is_err(),
+        "automatic Codex output should not reach realtime in client-managed handoff mode"
+    );
+
+    harness
+        .append_speech(harness.thread_id.clone(), "client-selected speech")
+        .await?;
+    assert_eq!(
+        harness.sideband_outbound_request(/*request_index*/ 1).await,
+        json!({
+            "type": "conversation.handoff.append",
+            "handoff_id": "codex",
+            "output_text": "client-selected speech",
+        })
+    );
+
+    harness.shutdown().await;
+    Ok(())
+}
+
+#[tokio::test]
+async fn webrtc_v1_final_automatic_handoff_omits_silent_prefix() -> Result<()> {
+    skip_if_no_network!(Ok(()));
+
+    let mut harness = RealtimeE2eHarness::new(
+        RealtimeTestVersion::V1,
+        main_loop_responses(vec![create_final_assistant_message_sse_response(
+            "background progress",
+        )?]),
+        realtime_sideband(vec![realtime_sideband_connection(vec![
+            vec![
+                session_updated("sess_v1_prefixed_handoff"),
+                json!({
+                    "type": "conversation.handoff.requested",
+                    "handoff_id": "handoff_prefixed",
+                    "item_id": "item_prefixed",
+                    "input_transcript": "run the background task"
+                }),
+            ],
+            vec![],
+            vec![],
+        ])]),
+    )
+    .await?;
+
+    let started = harness
+        .start_webrtc_realtime_with_codex_response_routing(
+            "v=offer\r\n",
+            /*client_managed_handoffs*/ None,
+            /*codex_responses_as_items*/ None,
+            Some(RESPONSE_HANDOFF_PREFIX),
+        )
+        .await?;
+    assert_eq!(started.started.version, RealtimeConversationVersion::V1);
+    let _ = harness
+        .read_notification::<TurnCompletedNotification>("turn/completed")
+        .await?;
+
+    assert_eq!(
+        harness.sideband_outbound_request(/*request_index*/ 1).await,
+        json!({
+            "type": "conversation.handoff.append",
+            "handoff_id": "handoff_prefixed",
+            "output_text": "background progress",
+        })
+    );
+
+    harness.shutdown().await;
+    Ok(())
+}
+
 #[tokio::test]
 async fn webrtc_v1_handoff_request_delegates_context_and_manual_append_speaks() -> Result<()> {
    skip_if_no_network!(Ok(()));
@@ -2468,8 +2616,10 @@ async fn realtime_webrtc_start_surfaces_backend_error() -> Result<()> {
    let start_request_id = mcp
        .send_thread_realtime_start_request(ThreadRealtimeStartParams {
            architecture: None,
+            client_managed_handoffs: None,
            codex_responses_as_items: None,
            codex_response_item_prefix: None,
+            codex_response_handoff_prefix: None,
            thread_id: thread_start.thread.id,
            model: None,
            output_modality: RealtimeOutputModality::Audio,
@@ -2533,8 +2683,10 @@ async fn realtime_conversation_requires_feature_flag() -> Result<()> {
    let start_request_id = mcp
        .send_thread_realtime_start_request(ThreadRealtimeStartParams {
            architecture: None,
+            client_managed_handoffs: None,
            codex_responses_as_items: None,
            codex_response_item_prefix: None,
+            codex_response_handoff_prefix: None,
            thread_id: thread_start.thread.id.clone(),
            model: None,
            output_modality: RealtimeOutputModality::Audio,