19 Commits

• [codex] consume pushed exec-server process events (#30273)

  ## Summary
  
  - complete unified-exec processes from the ordered event stream instead
  of issuing a final zero-wait `process/read`
  - add optional executor sandbox-denial state to `process/exited`
  - retain `process/read` as a retained-output and compatibility fallback
  for receiver lag, sequence gaps, and legacy servers
  - recover sandbox-denial state across transport reconnection
  - cover the real `TestCodex` remote-exec path without adding a public
  test-only event constructor
  
  ## Why
  
  A successful one-shot tool call currently receives its output and
  terminal notifications, then pays another wide-area `process/read` round
  trip before returning. Staging traces showed that remote response wait
  accounted for more than 99.8% of RPC time; local serialization,
  queueing, and deserialization were below 0.6 ms.
  
  ## Measured impact
  
  A direct staging A/B used the same build and route and changed only
  completion mode. Each arm ran three times with 30 one-shot
  `/usr/bin/true` calls per run. The table reports the median of the three
  per-run percentiles.
  
  | Metric | Final `process/read` | Pushed events | Change |
  | --- | ---: | ---: | ---: |
  | End-to-end completion p50 | 159.5 ms | 118.7 ms | -40.8 ms (-25.6%) |
  | End-to-end completion p95 | 182.4 ms | 131.7 ms | -50.6 ms (-27.8%) |
  | Completion-wait p50 | 80.1 ms | 41.5 ms | -38.5 ms (-48.1%) |
  | Final `process/read` RPC p50 | 79.9 ms | eliminated | -79.9 ms |
  
  TCP_NODELAY was enabled in both A/B arms, so its effect cancels out. The
  successful, complete, in-order event path issued zero final
  `process/read` calls.
  
  ## Compatibility and recovery
  
  - new servers send `sandboxDenied` on `process/exited`
  - legacy servers omit it, which triggers one compatibility
  `process/read`
  - broadcast lag or a sequence gap triggers a retained-output read
  - recovery remains bounded by the server's existing 1 MiB
  retained-output window
  - complete, in-order event streams issue no completion read
  - sandbox denial is attached to the exit event before consumers can
  observe process completion
  - server-first and client-first rollouts remain wire-compatible;
  server-first realizes the latency win immediately
  
  ## Integration coverage
  
  The `TestCodex` suite exercises four distinct remote-exec contracts:
  
  - complete pushed output/exit/close with zero reads
  - direct pushed sandbox denial with zero reads
  - legacy missing denial metadata with exactly one compatibility read
  - count-bounded replay eviction recovered from retained output without
  duplication
  
  ## Validation
  
  - `just test -p codex-core
  exec_command_consumes_pushed_remote_process_events`: 4 passed
  - `just test -p codex-core unified_exec::process_tests::`: 4 passed
  - `just test -p codex-exec-server`: 294 passed, 2 skipped
  - `just test -p codex-exec-server-protocol`: 5 passed
  - `just test -p codex-rmcp-client`: 89 passed, 2 skipped
  - focused Bazel `//codex-rs/core:core-all-test`: passed across 16 shards
  - scoped `just fix` passed for core and exec-server
  - `just fmt` passed
  
  The complete workspace suite was not rerun; focused Cargo and Bazel
  coverage passed for the changed behavior.

  richardopenai · 2026-06-26 18:05:52 -07:00
  d4ec08b8f0
• protocol: separate app and exec RPC ownership (#29714)

  ## Why
  
  The app-server and exec-server expose separate JSON-RPC APIs, but
  exec-server currently sources its serialized protocol and envelope types
  through app-server-oriented code. Giving each API an explicit owner
  makes the crate boundary legible without introducing shared generic
  envelopes.
  
  ## What changed
  
  - Added `codex-exec-server-protocol` to own exec DTOs, process IDs, and
  JSON-RPC envelopes.
  - Updated exec-server clients, transports, handlers, and tests to use
  the new crate.
  - Exposed app-server's existing JSON-RPC types through a public `rpc`
  module while retaining root re-exports.
  - Preserved existing wire shapes, including exec `PathUri` behavior.
  
  ## Stack
  
  This is PR 1 of 6. Next: [PR
  #29721](https://github.com/openai/codex/pull/29721), which moves auth
  mode below the app wire boundary.
  
  ## Validation
  
  - Exec-server protocol and server coverage passed in the focused
  protocol test runs.
  - App-server protocol schema fixtures passed.

  Adam Perry @ OpenAI · 2026-06-23 22:37:31 +00:00
  829f5b6b59
• path-uri: remove legacy path deserialization (#29158)

  ## Why
  
  I'd originally added `PathUri` legacy path deserialization thinking we'd
  want it for having `PathUri` in public app-server APIs. Since then we've
  added `LegacyAppPathString` to handle the messy conversions that we need
  for backcompat. It's confusing for `PathUri` to support deserializing
  legacy paths when we don't yet want to actually expose app-server
  callers or rollout storage to the new URI format.
  
  Stacked on top of #29472 to avoid breaking compatibility in case those
  types ended up stored somewhere for someone.
  
  ## What changed
  
  - Parse deserialized `PathUri` values exclusively as valid `file:` URIs.
  - Replace legacy acceptance coverage with rejection coverage for
  top-level filesystem paths and sandbox working directories.
  - Serialize CWDs in hand-built exec-server process requests as `PathUri`
  values.

  Adam Perry @ OpenAI · 2026-06-23 21:47:00 +00:00
  c26f961b85
• [codex] exec-server: stream files in chunks (#28354)

  ## Why
  
  `fs/readFile` buffers the entire file in one response, which makes large
  remote reads expensive and prevents callers from applying backpressure.
  We need an opt-in streaming path with bounded block sizes while
  preserving the existing single-call API for small and sandboxed reads.
  
  ## What changed
  
  - Add `ExecServerClient::stream`, returning a named `FileReadStream`
  that implements `futures::Stream` and yields immutable 1 MiB byte
  blocks.
  - Add internal `fs/open`, `fs/readBlock`, and `fs/close` RPCs.
  `fs/readBlock` accepts an explicit offset and length.
  - Keep unsandboxed files open between block reads, cap open handles per
  connection, and clean them up on EOF, error, stream drop, explicit
  close, or connection shutdown.
  - Reject platform-sandboxed streaming opens instead of turning the
  one-shot sandbox helper into a persistent server. Existing `fs/readFile`
  behavior is unchanged.
  
  ## Testing
  
  - `just test -p codex-exec-server`
  - Integration coverage for 1 MiB chunking, exact block-boundary EOF,
  sandbox rejection, and continued reads from the opened file after path
  replacement.
  - Handle-manager coverage for non-sequential offsets, variable block
  lengths, the 128-handle limit, and capacity release after close.

  pakrym-oai · 2026-06-16 09:50:55 -07:00
  a4711b88dd
• exec-server: default remote transport to Noise (#26245)

  ## Why
  
  The transport in
  [openai/codex#26242](https://github.com/openai/codex/pull/26242) needs
  to be used by every remote orchestrator-to-executor connection before
  JSON-RPC traffic starts.
  
  ## Changes
  
  - Generates one executor Noise identity when remote exec-server starts
  and registers its public key.
  - Creates a harness identity for each physical remote environment
  connection.
  - Fetches a fresh registry bundle before connecting and validates the
  authenticated harness key before completing the executor handshake.
  - Multiplexes encrypted logical streams over the existing executor
  WebSocket.
  - Adds bounded stream, handshake-failure, and reassembly state.
  - Adds safe lifecycle diagnostics without logging keys, authorizations,
  plaintext, or ciphertext.
  - Covers reconnects, replay rejection, validation failure, framing
  limits, and encrypted JSON-RPC tool traffic.
  
  ## Stack
  
  1. [openai/codex#26242](https://github.com/openai/codex/pull/26242):
  Noise channel and relay transport
  2. **[openai/codex#26245](https://github.com/openai/codex/pull/26245)**:
  remote registration and runtime activation
  
  ## Verification
  
  - `just test -p codex-exec-server`
  - `just fix -p codex-exec-server`
  - `just bazel-lock-check`
  - `cargo shear`
  
  ---------
  
  Co-authored-by: Codex <noreply@openai.com>

  viyatb-oai · 2026-06-15 17:39:00 -07:00
  6e50b22e55
• [codex] Carry exec-server cwd as PathUri (#28032)

  ## Why
  
  This is the second-to-last place in the exec-server protocol that needs
  to migrate to URIs to support cross-OS operation.
  
  ## What
  
  - Change `ExecParams.cwd` to `PathUri`.
  - Keep the cwd URI-shaped through core and rmcp producers, converting it
  to `AbsolutePathBuf` only in `LocalProcess::start_process`.
  - Reject non-native cwd URIs before launch and update the affected
  protocol documentation and call sites.

  Adam Perry @ OpenAI · 2026-06-13 20:56:42 +00:00
  0fed4497f5
• [codex] migrate exec-server filesystem protocol to PathUri (#27653)

  Exec-server filesystem calls should preserve cross-platform `file:` URIs
  across the remote boundary instead of converting them through paths
  native to the client host.
  
  This changes the exec-server filesystem protocol DTOs to use `PathUri`,
  carries those values directly through remote and sandbox-helper
  transports, and keeps legacy native absolute-path request strings
  readable for compatibility. It also updates protocol documentation and
  coverage for URI serialization and non-native URI forwarding.

  Adam Perry @ OpenAI · 2026-06-11 15:09:12 -07:00
  d23bb22f25
• Allow API-key auth for remote exec-server registration (#24666)

  ## Overview
  Allow remote `codex exec-server` registration to use existing API-key
  auth while restricting where those credentials can be sent.
  
  - Accept `CodexAuth::ApiKey` for the normal `--remote` registration
  path.
  - Restrict API-key remote registration to HTTPS `openai.com` and
  `openai.org` hosts and subdomains, with explicit HTTP loopback support
  for local development.
  - Disable registry registration redirects so credentials cannot be
  forwarded to an unvalidated destination.
  - Retain `--use-agent-identity-auth` as the explicit Agent Identity
  path.
  - Document remote registration using `CODEX_API_KEY`.
  
  ## Big picture
  Callers can now provide an API key directly to `exec-server`
  registration without first establishing ChatGPT login state:
  
  ```sh
  CODEX_API_KEY="$OPENAI_API_KEY" \
  codex exec-server \
    --remote "https://<host>.openai.org/api" \
    --environment-id "$ENVIRONMENT_ID"
  ```
  
  ## Validation
  - `cargo fmt --all` (`just fmt` is not installed on this host)
  - `cargo test -p codex-cli -p codex-exec-server`

  Steve Coffey · 2026-05-27 21:17:38 +00:00
  c57dee98b7
• Migrate exec-server remote registration to environments (#23633)

  ## Summary
  - migrate exec-server remote registration naming from executor to
  environment
  - align CLI, public Rust exports, registry error messages, and relay
  test fixtures with the environment registry contract
  - keep the live registration path and response model consistent with
  `/cloud/environment/{environment_id}/register`
  
  ## Verification
  - `cargo test -p codex-exec-server
  remote::tests::register_environment_posts_with_auth_provider_headers
  --manifest-path /Users/richardlee/code/codex/codex-rs/Cargo.toml`
  - `cargo test -p codex-exec-server --test relay
  multiplexed_remote_environment_routes_independent_virtual_streams
  --manifest-path /Users/richardlee/code/codex/codex-rs/Cargo.toml`
  - `cargo check -p codex-cli --manifest-path
  /Users/richardlee/code/codex/codex-rs/Cargo.toml` (still running when PR
  opened; will update after completion if needed)

  richardopenai · 2026-05-20 00:25:04 -07:00
  000bf5ce6d
• exec-server: support auth-backed remote executor registration (#22769)

  This updates remote `exec-server` registration to use normal Codex auth
  instead of a registry-issued credential. The registry request is built
  from the existing auth-provider path, which preserves the biscuit-only
  registry contract introduced in
  [openai/openai#924101](https://github.com/openai/openai/pull/924101)
  while removing the old remote registry bearer env var and its direct
  transport assumptions.
  
  The default remote flow uses persisted ChatGPT auth from the normal
  Codex config/storage path. This PR also includes the containerized Agent
  Identity path needed by
  [openai/openai#924260](https://github.com/openai/openai/pull/924260):
  remote `exec-server` accepts `--allow-agent-identity-auth`, permits
  Agent Identity auth loaded from `CODEX_ACCESS_TOKEN` only when that flag
  is present, and reuses the existing Agent task registration plus derived
  `AgentAssertion` header generation. API-key auth remains unsupported,
  and Agent Identity stays opt-in.
  
  Validation performed beyond normal presubmit coverage:
  - `cargo fmt --all --check`
  - `cargo check -p codex-cli`
  - `cargo test -p codex-exec-server`
  - `cargo test -p codex-cli exec_server_agent_identity_auth_flag_`
  - `cargo test -p codex-cli remote_exec_server_auth_mode_`
  
  I also attempted `cargo test -p codex-cli`. The new CLI tests passed
  inside that run, but the suite ended on an unrelated local
  marketplace-state failure in
  `plugin_list_excludes_unconfigured_repo_local_marketplaces`.

  Michael Zeng · 2026-05-16 12:48:28 -07:00
  b200dd1b6f
• feat(exec-server): use protobuf relay frames (#22343)

  ## Why
  
  Remote exec-server now needs one executor websocket to serve multiple
  harness JSON-RPC sessions. Rendezvous routes by `stream_id`, and the
  exec-server side needs to use the same stable relay frame contract
  instead of a hand-rolled JSON shape.
  
  The relay protocol also needs to make ownership boundaries clear:
  harness and executor endpoints own sequencing, acks, retries, duplicate
  suppression, segmentation, and reassembly; rendezvous only routes
  frames.
  
  ## What Changed
  
  - Add the checked-in `codex.exec_server.relay.v1.RelayMessageFrame`
  proto plus generated prost bindings for `codex-exec-server`.
  - Encode remote harness/executor relay traffic as binary protobuf
  websocket frames while keeping local websocket JSON-RPC unchanged.
  - Demux executor-side relay streams into independent
  `ConnectionProcessor` sessions keyed by `stream_id`.
  - Add a programmatic `RemoteExecutorConfig::with_bearer_token(...)`
  constructor for non-CLI callers and integration tests.
  - Add an integration test that starts the remote executor against a fake
  registry/rendezvous websocket and verifies two virtual streams share one
  executor websocket without cross-talk, including per-stream reset
  behavior.
  - Document the remote relay envelope, sequence ranges, `ack`/`ack_bits`,
  and endpoint responsibilities in `exec-server/README.md`.
  
  ## Verification
  
  - `cargo test -p codex-exec-server --test relay
  multiplexed_remote_executor_routes_independent_virtual_streams --
  --exact`
  - `cargo test -p codex-exec-server --test relay`
  - `cargo test -p codex-exec-server` passed outside the sandbox. The
  sandboxed run hit macOS `sandbox-exec: sandbox_apply: Operation not
  permitted` in filesystem sandbox tests.

  Anton Panasenko · 2026-05-12 16:50:45 -07:00
  ac466c0dbd
• Add cloud executor registration to exec-server (#19575)

  ## Summary
  This PR adds the first `codex-rs` milestone for remote-exec e2e: a local
  `codex exec-server` can now register itself with
  `codex-cloud-environments` and attach to the returned rendezvous
  websocket.
  
  At a high level, `codex exec-server --cloud ...` now:
  - loads ChatGPT auth from normal Codex config
  - registers an executor with `codex-cloud-environments`
  - receives a signed rendezvous websocket URL
  - serves the existing exec-server JSON-RPC protocol over that websocket
  
  ## What Changed
  - Added `--cloud`, `--cloud-base-url`, `--cloud-environment-id`, and
  `--cloud-name` to `codex exec-server`
  - Added a new `exec-server/src/cloud.rs` module that handles:
    - registration requests
    - auth/header setup
    - bounded auth retry on `401/403`
    - reconnect/backoff after websocket disconnects
  - Reused the existing `ConnectionProcessor` / `ExecServerHandler` path
  so cloud mode serves the same exec/filesystem RPC surface as local
  websocket mode
  - Added cloud-specific error variants and minimal docs for the new mode
  
  ## Testing
  Manual e2e test that fully goes through exec server flow with our codex
  cloud agent as orchestrator

  Michael Zeng · 2026-05-05 22:01:48 +00:00
  d0f9d5eba2
• [2/8] Support piped stdin in exec process API (#18086)

  ## Summary
  - Add an explicit stdin mode to process/start.
  - Keep normal non-interactive exec stdin closed while allowing
  pipe-backed processes.
  
  ## Stack
  ```text
  o  #18027 [8/8] Fail exec client operations after disconnect
  │
  o  #18025 [7/8] Cover MCP stdio tests with executor placement
  │
  o  #18089 [6/8] Wire remote MCP stdio through executor
  │
  o  #18088 [5/8] Add executor process transport for MCP stdio
  │
  o  #18087 [4/8] Abstract MCP stdio server launching
  │
  o  #18020 [3/8] Add pushed exec process events
  │
  @  #18086 [2/8] Support piped stdin in exec process API
  │
  o  #18085 [1/8] Add MCP server environment config
  │
  o  main
  ```
  
  Co-authored-by: Codex <noreply@openai.com>

  Ahmed Ibrahim · 2026-04-16 10:30:10 -07:00
  2ca270d08d
• Run exec-server fs operations through sandbox helper (#17294)

  ## Summary
  - run exec-server filesystem RPCs requiring sandboxing through a
  `codex-fs` arg0 helper over stdin/stdout
  - keep direct local filesystem execution for `DangerFullAccess` and
  external sandbox policies
  - remove the standalone exec-server binary path in favor of top-level
  arg0 dispatch/runtime paths
  - add sandbox escape regression coverage for local and remote filesystem
  paths
  
  ## Validation
  - `just fmt`
  - `git diff --check`
  - remote devbox: `cd codex-rs && bazel test --bes_backend=
  --bes_results_url= //codex-rs/exec-server:all` (6/6 passed)
  
  ---------
  
  Co-authored-by: Codex <noreply@openai.com>

  starr-openai · 2026-04-12 18:36:03 -07:00
  d626dc3895
• Remove stdio transport from exec server (#15119)

  Summary
  - delete the deprecated stdio transport plumbing from the exec server
  stack
  - add a basic `exec_server()` harness plus test utilities to start a
  server, send requests, and await events
  - refresh exec-server dependencies, configs, and documentation to
  reflect the new flow
  
  Testing
  - Not run (not requested)
  
  ---------
  
  Co-authored-by: starr-openai <starr@openai.com>
  Co-authored-by: Codex <noreply@openai.com>

  pakrym-oai · 2026-03-19 01:00:35 +00:00
  903660edba
• Add exec-server stub server and protocol docs (#15089)

  Stacked PR 1/3.
  
  This is the initialize-only exec-server stub slice: binary/client
  scaffolding and protocol docs, without exec/filesystem implementation.
  
  ---------
  
  Co-authored-by: Codex <noreply@openai.com>

  starr-openai · 2026-03-19 00:30:05 +00:00
  81996fcde6
• refactor: delete exec-server and move execve wrapper into shell-escalation (#12632)

  ## Why
  
  We already plan to remove the shell-tool MCP path, and doing that
  cleanup first makes the follow-on `shell-escalation` work much simpler.
  
  This change removes the last remaining reason to keep
  `codex-rs/exec-server` around by moving the `codex-execve-wrapper`
  binary and shared shell test fixtures to the crates/tests that now own
  that functionality.
  
  ## What Changed
  
  ### Delete `codex-rs/exec-server`
  
  - Remove the `exec-server` crate, including the MCP server binary,
  MCP-specific modules, and its test support/test suite
  - Remove `exec-server` from the `codex-rs` workspace and update
  `Cargo.lock`
  
  ### Move `codex-execve-wrapper` into `codex-rs/shell-escalation`
  
  - Move the wrapper implementation into `shell-escalation`
  (`src/unix/execve_wrapper.rs`)
  - Add the `codex-execve-wrapper` binary entrypoint under
  `shell-escalation/src/bin/`
  - Update `shell-escalation` exports/module layout so the wrapper
  entrypoint is hosted there
  - Move the wrapper README content from `exec-server` to
  `shell-escalation/README.md`
  
  ### Move shared shell test fixtures to `app-server`
  
  - Move the DotSlash `bash`/`zsh` test fixtures from
  `exec-server/tests/suite/` to `app-server/tests/suite/`
  - Update `app-server` zsh-fork tests to reference the new fixture paths
  
  ### Keep `shell-tool-mcp` as a shell-assets package
  
  - Update `.github/workflows/shell-tool-mcp.yml` packaging so the npm
  artifact contains only patched Bash/Zsh payloads (no Rust binaries)
  - Update `shell-tool-mcp/package.json`, `shell-tool-mcp/src/index.ts`,
  and docs to reflect the shell-assets-only package shape
  - `shell-tool-mcp-ci.yml` does not need changes because it is already
  JS-only
  
  ## Verification
  
  - `cargo shear`
  - `cargo clippy -p codex-shell-escalation --tests`
  - `just clippy`

  Michael Bolin · 2026-02-23 20:10:22 -08:00
  38f84b6b29
• feat(shell-tool-mcp): add patched zsh build pipeline (#11668)

  ## Summary
  - add `shell-tool-mcp/patches/zsh-exec-wrapper.patch` against upstream
  zsh `77045ef899e53b9598bebc5a41db93a548a40ca6`
  - add `zsh-linux` and `zsh-darwin` jobs to
  `.github/workflows/shell-tool-mcp.yml`
  - stage zsh binaries under `artifacts/vendor/<target>/zsh/<variant>/zsh`
  - include zsh artifact jobs in `package.needs`
  - mark staged zsh binaries executable during packaging
  
  ## Notes
  - zsh source is cloned from `https://git.code.sf.net/p/zsh/code`
  - workflow pins zsh commit `77045ef899e53b9598bebc5a41db93a548a40ca6`
  - zsh build runs `./Util/preconfig` before `./configure`
  
  ## Validation
  - parsed workflow YAML locally (`yaml-ok`)
  - validated zsh patch applies cleanly with `git apply --check` on a
  fresh zsh clone

  Jeremy Rose · 2026-02-13 01:34:48 +00:00
  9cf7a07281
• docs: update the docs for @openai/codex-shell-tool-mcp (#7962)

  The existing version of `shell-tool-mcp/README.md` was not written in a
  way that was meant to be consumed by end-users. This is now fixed.
  
  Added `codex-rs/exec-server/README.md` for the more technical bits.

  Michael Bolin · 2025-12-13 09:44:26 -08:00
  a2c86e5d88