codex

Update realtime handoff transcript handling (#18597 )

## Summary

This PR aims to improve integration between the realtime model and the
codex agent by sharing more context with each other. In particular, we
now share full realtime conversation transcript deltas in addition to
the delegation message.

realtime_conversation.rs now turns a handoff into:
```
<realtime_delegation>
  <input>...</input>
  <transcript_delta>...</transcript_delta>
</realtime_delegation>
```

## Implementation notes

The transcript is accumulated in the realtime websocket layer as parsed
realtime events arrive. When a background-agent handoff is requested,
the current transcript snapshot is copied onto the handoff event and
then serialized by `realtime_conversation.rs` into the hidden realtime
delegation envelope that Codex receives as user-turn context.

For Realtime V2, the session now explicitly enables input audio
transcription, and the parser handles the relevant input/output
transcript completion events so the snapshot includes both user speech
and realtime model responses. The delegation `<input>` remains the
actual handoff request, while `<transcript_delta>` carries the
surrounding conversation history for context.

Reviewers should note that the transcript payload is intended for Codex
context sharing, not UI rendering. The realtime delegation envelope
should stay hidden from the user-facing transcript surface, while still
being included in the background-agent turn so Codex can answer with the
same conversational context the realtime model had.

guinness-oai · 2026-04-20 14:04:09 -07:00

126bd6e7a8

[codex] Update realtime V2 VAD silence delay and 1.5 prompt (#18092 )

## Summary

- set the realtime v2 server VAD silence delay to 500ms
- update the default realtime 1.5 backend prompt to the v4 text
- keep the session payload and prompt rendering tests aligned with those
changes

## Why

- the VAD change gives the voice path a longer pause before ending the
user's turn
- the prompt change makes the default bundled realtime prompt match the
current v4 content

## Validation

- `cargo +1.93.0 test -p codex-core realtime_prompt --manifest-path
/tmp/codex-realtime-v2-vad-prompt-v4/codex-rs/Cargo.toml`
- `CARGO_TARGET_DIR=/tmp/codex-pr-v4-target cargo +1.93.0 test -p
codex-api
realtime_v2_session_update_includes_background_agent_tool_and_handoff_output_item
--manifest-path
/tmp/codex-realtime-v2-vad-prompt-v4/codex-rs/Cargo.toml`
- `CARGO_TARGET_DIR=/tmp/codex-pr-v4-target cargo +1.93.0 test -p
codex-app-server --test all
'suite::v2::realtime_conversation::realtime_webrtc_start_emits_sdp_notification'
--manifest-path /tmp/codex-realtime-v2-vad-prompt-v4/codex-rs/Cargo.toml
-- --exact`

bxie-openai · 2026-04-16 14:30:57 -07:00

6a1ddfc366

Add realtime output modality and transcript events (#17701 )

- Add outputModality to thread/realtime/start and wire text/audio output
selection through app-server, core, API, and TUI.\n- Rename the realtime
transcript delta notification and add a separate transcript done
notification that forwards final text from item done without correlating
it with deltas.

Ahmed Ibrahim · 2026-04-14 00:13:13 -07:00

2f6fc7c137

Add realtime voice selection (#17176 )

- Add realtime voice selection for realtime/start.
- Expose the supported v1/v2 voice lists and cover explicit, configured,
default, and invalid voice paths.

Ahmed Ibrahim · 2026-04-08 20:19:15 -07:00

2f9090be62

Add WebRTC transport to realtime start (#16960 )

Adds WebRTC startup to the experimental app-server
`thread/realtime/start` method with an optional transport enum. The
websocket path remains the default; WebRTC offers create the realtime
session through the shared start flow and emit the answer SDP via
`thread/realtime/sdp`.

---------

Co-authored-by: Codex <noreply@openai.com>

Ahmed Ibrahim · 2026-04-07 15:43:38 -07:00

fb3dcfde1d

[codex] reduce module visibility (#16978 )

## Summary
- reduce public module visibility across Rust crates, preferring private
or crate-private modules with explicit crate-root public exports
- update external call sites and tests to use the intended public crate
APIs instead of reaching through module trees
- add the module visibility guideline to AGENTS.md

## Validation
- `cargo check --workspace --all-targets --message-format=short` passed
before the final fix/format pass
- `just fix` completed successfully
- `just fmt` completed successfully
- `git diff --check` passed

pakrym-oai · 2026-04-07 08:03:35 -07:00

413c1e1fdf

[stack 2/4] Align main realtime v2 wire and runtime flow (#14830 )

## Stack Position
2/4. Built on top of #14828.

## Base
- #14828

## Unblocks
- #14829
- #14827

## Scope
- Port the realtime v2 wire parsing, session, app-server, and
conversation runtime behavior onto the split websocket-method base.
- Branch runtime behavior directly on the current realtime session kind
instead of parser-derived flow flags.
- Keep regression coverage in the existing e2e suites.

---------

Co-authored-by: Codex <noreply@openai.com>

Ahmed Ibrahim · 2026-03-16 21:38:07 -07:00

fbd7f9b986

Use parser-specific realtime voice enum (#14636 )

Model realtime session output voices with an enum and map by parser so
v1 uses fathom and v2 uses alloy.

Co-authored-by: Codex <noreply@openai.com>

Ahmed Ibrahim · 2026-03-13 16:17:13 -07:00

7fa5201365

Add realtime transcription mode for websocket sessions (#14556 )

- add experimental_realtime_ws_mode (conversational/transcription) and
plumb it into realtime conversation session config
- switch realtime websocket intent and session.update payload shape
based on mode
- update config schema and realtime/config tests

---------

Co-authored-by: Codex <noreply@openai.com>

Ahmed Ibrahim · 2026-03-12 23:50:30 -07:00

2253a9d1d7

Add codex tool support for realtime v2 handoff (#14554 )

- Advertise a `codex` function tool in realtime v2 session updates.
- Emit handoff replies as `function_call_output` items while keeping v1
behavior unchanged.
- Split realtime event parsing into explicit v1/v2 modules with shared
common helpers.

---------

Co-authored-by: Codex <noreply@openai.com>

Ahmed Ibrahim · 2026-03-12 23:30:02 -07:00

eaf81d3f6f

Add realtime v2 event parser behind feature flag (#14537 )

- Add a feature-flagged realtime v2 parser on the existing
websocket/session pipeline.
- Wire parser selection from core feature flags and map the codex
handoff tool-call path into existing handoff events.

---------

Co-authored-by: Codex <noreply@openai.com>

Ahmed Ibrahim · 2026-03-12 21:12:40 -07:00

3e8f47169e

Use realtime transcript for handoff context (#14132 )

- collect input/output transcript deltas into active handoff transcript
state
- attach and clear that transcript on each handoff, and regenerate
schema/tests

Ahmed Ibrahim · 2026-03-09 22:30:03 -07:00

2e24be2134

Update realtime websocket API (#13265 )

- migrate the realtime websocket transport to the new session and
handoff flow
- make the realtime model configurable in config.toml and use API-key
auth for the websocket

---------

Co-authored-by: Codex <noreply@openai.com>

Ahmed Ibrahim · 2026-03-02 16:05:40 -08:00

b20b6aa46f

Wire realtime api to core (#12268 )

- Introduce `RealtimeConversationManager` for realtime API management 
- Add `op::conversation` to start conversation, insert audio, insert
text, and close conversation.
- emit conversation lifecycle and realtime events.
- Move shared realtime payload types into codex-protocol and add core
e2e websocket tests for start/replace/transport-close paths.

Things to consider:
- Should we use the same `op::` and `Events` channel to carry audio? I
think we should try this simple approach and later we can create
separate one if the channels got congested.
- Sending text updates to the client: we can start simple and later
restrict that.
- Provider auth isn't wired for now intentionally

Ahmed Ibrahim · 2026-02-20 19:06:35 -08:00

6817f0be8a

codex-api: realtime websocket session.create + typed inbound events (#12036 )

## Summary
- add realtime websocket client transport in codex-api
- send session.create on connect with backend prompt and optional
conversation_id
- keep session.update for prompt changes after connect
- switch inbound event parsing to a tagged enum (typed variants instead
of optional field bag)
- add a websocket e2e integration test in
codex-rs/codex-api/tests/realtime_websocket_e2e.rs

## Why
This moves the realtime transport to an explicit session-create
handshake and improves protocol safety with typed inbound events.

## Testing
- Added e2e integration test coverage for session create + event flow in
the API crate.

Ahmed Ibrahim · 2026-02-17 22:17:01 -08:00

03ce01e71f

15 Commits