mirror of
https://github.com/pchuan98/codex.git
synced 2026-07-01 00:31:56 +08:00
Add realtime speech append control (#27917)
## Why Realtime voice harness tuning needs app-side control over what backend Codex text is spoken. Backend orchestrator text is written for a reading UI, so automatically speaking every preamble, progress update, or final assistant message can make the realtime voice model too chatty. For experimentation, clients need two simple controls: keep app/client text-item injection on the existing item-create path, and add an explicit speakable path that app code can call only when it wants realtime to speak. Automatic Codex output also needs an opt-in way to switch from the protocol's default speakable path to regular realtime items, with a caller-provided prefix so prompt wording can be tuned outside core. The default remains unchanged: if a client omits the new start fields and never calls `appendSpeech`, automatic backend output continues down the existing speakable path for the selected realtime protocol. ## What Changed - Adds experimental `thread/realtime/appendSpeech` for app-provided speakable text. - Keeps existing `thread/realtime/appendText` as the item-create API for app-provided realtime text items. - Adds `codexResponsesAsItems` / `codex_responses_as_items` on `thread/realtime/start` to send automatic Codex responses with `conversation.item.create` instead of the protocol's default speakable output path. - Adds `codexResponseItemPrefix` / `codex_response_item_prefix` so clients can prepend experiment instructions to those automatic Codex response items. - Keeps literal `conversation.handoff.append` routing scoped to the v1 speakable path; v2 default speech uses its item/function-output plus `response.create` behavior. - Removes the earlier public silent-context API and hardcoded silent-context prefix. - Updates realtime tests to cover default automatic speakable behavior, opt-in automatic item-create behavior, and explicit `appendSpeech` behavior. ## Validation - `cargo check -p codex-core -p codex-app-server -p codex-api` - `just test -p codex-app-server realtime_conversation` - `just test -p codex-core realtime_conversation` (50/51 passed in the filtered parallel run; the lone failure passed when rerun in isolation) - `just test -p codex-core conversation_mirrors_assistant_message_text_to_realtime_handoff` - `just test -p codex-api e2e_connect_and_exchange_events_against_mock_ws_server` - `just fix -p codex-core` - `just fix -p codex-app-server` - `cargo build -p codex-cli`
This commit is contained in:
committed by
GitHub
Unverified
parent
9728992fab
commit
1d8ff89aa3
@@ -182,6 +182,16 @@ impl TurnRequestProcessor {
|
||||
.map(|response| response.map(Into::into))
|
||||
}
|
||||
|
||||
pub(crate) async fn thread_realtime_append_speech(
|
||||
&self,
|
||||
request_id: &ConnectionRequestId,
|
||||
params: ThreadRealtimeAppendSpeechParams,
|
||||
) -> Result<Option<ClientResponsePayload>, JSONRPCErrorError> {
|
||||
self.thread_realtime_append_speech_inner(request_id, params)
|
||||
.await
|
||||
.map(|response| response.map(Into::into))
|
||||
}
|
||||
|
||||
pub(crate) async fn thread_realtime_stop(
|
||||
&self,
|
||||
request_id: &ConnectionRequestId,
|
||||
@@ -942,6 +952,8 @@ impl TurnRequestProcessor {
|
||||
thread.as_ref(),
|
||||
Op::RealtimeConversationStart(ConversationStartParams {
|
||||
architecture: params.architecture,
|
||||
codex_responses_as_items: params.codex_responses_as_items.unwrap_or(false),
|
||||
codex_response_item_prefix: params.codex_response_item_prefix,
|
||||
model: params.model,
|
||||
output_modality: params.output_modality,
|
||||
prompt: params.prompt,
|
||||
@@ -1018,6 +1030,31 @@ impl TurnRequestProcessor {
|
||||
Ok(Some(ThreadRealtimeAppendTextResponse::default()))
|
||||
}
|
||||
|
||||
async fn thread_realtime_append_speech_inner(
|
||||
&self,
|
||||
request_id: &ConnectionRequestId,
|
||||
params: ThreadRealtimeAppendSpeechParams,
|
||||
) -> Result<Option<ThreadRealtimeAppendSpeechResponse>, JSONRPCErrorError> {
|
||||
let Some((_, thread)) = self
|
||||
.prepare_realtime_conversation_thread(request_id, ¶ms.thread_id)
|
||||
.await?
|
||||
else {
|
||||
return Ok(None);
|
||||
};
|
||||
self.submit_core_op(
|
||||
request_id,
|
||||
thread.as_ref(),
|
||||
Op::RealtimeConversationSpeech(ConversationSpeechParams { text: params.text }),
|
||||
)
|
||||
.await
|
||||
.map_err(|err| {
|
||||
internal_error(format!(
|
||||
"failed to append realtime conversation speech: {err}"
|
||||
))
|
||||
})?;
|
||||
Ok(Some(ThreadRealtimeAppendSpeechResponse::default()))
|
||||
}
|
||||
|
||||
async fn thread_realtime_stop_inner(
|
||||
&self,
|
||||
request_id: &ConnectionRequestId,
|
||||
|
||||
Reference in New Issue
Block a user