Commit Graph

27 Commits

  • Support multiple managed environments (#18401)
    ## Summary
    - refactor EnvironmentManager to own keyed environments with
    default/local lookup helpers
    - keep remote exec-server client creation lazy until exec/fs use
    - preserve disabled agent environment access separately from internal
    local environment access
    
    ## Validation
    - not run (per Codex worktree instruction to avoid tests/builds unless
    requested)
    
    ---------
    
    Co-authored-by: Codex <noreply@openai.com>
  • app-server: implement device key v2 methods (#18430)
    ## Why
    
    The device-key protocol needs an app-server implementation that keeps
    local key operations behind the same request-processing boundary as
    other v2 APIs.
    
    app-server owns request dispatch, transport policy, documentation, and
    JSON-RPC error shaping. `codex-device-key` owns key binding, validation,
    platform provider selection, and signing mechanics. Keeping the adapter
    thin makes the boundary easier to review and avoids moving local
    key-management details into thread orchestration code.
    
    ## What changed
    
    - Added `DeviceKeyApi` as the app-server adapter around
    `DeviceKeyStore`.
    - Converted protocol protection policies, payload variants, algorithms,
    and protection classes to and from the device-key crate types.
    - Encoded SPKI public keys and DER signatures as base64 protocol fields.
    - Routed `device/key/create`, `device/key/public`, and `device/key/sign`
    through `MessageProcessor`.
    - Rejected remote transports before provider access while allowing local
    `stdio` and in-process callers to reach the device-key API.
    - Added stdio, in-process, and websocket tests for device-key validation
    and transport policy.
    - Documented the device-key methods in the app-server v2 method list.
    
    ## Test coverage
    
    - `device_key_create_rejects_empty_account_user_id`
    - `in_process_allows_device_key_requests_to_reach_device_key_api`
    - `device_key_methods_are_rejected_over_websocket`
    
    ## Stack
    
    This is PR 3 of 4 in the device-key app-server stack. It is stacked on
    #18429.
    
    ## Validation
    
    - `cargo test -p codex-app-server device_key`
    - `just fix -p codex-app-server`
  • Load app-server config through ConfigManager (#18870)
    ## Summary
    - Load app-server startup config through `ConfigManager` instead of
    direct `ConfigBuilder` calls.
    - Move `ConfigManager` constructor-owned state (`cli_overrides`, runtime
    feature map, cloud requirements loader) behind internal manager fields.
    - Pass `ConfigManager` into `MessageProcessor` directly instead of
    reconstructing it from raw args.
    
    ## Tests
    - `cargo check -p codex-app-server`
    - `cargo test -p codex-app-server`
    - `just fix -p codex-app-server`
    - `just fmt`
  • Add session config loader interface (#18208)
    ## Why
    
    Cloud-hosted sessions need a way for the service that starts or manages
    a thread to provide session-owned config without treating all config as
    if it came from the same user/project/workspace TOML stack.
    
    The important boundary is ownership: some values should be controlled by
    the session/orchestrator, some by the authenticated user, and later some
    may come from the executor. The earlier broad config-store shape made
    that boundary too fuzzy and overlapped heavily with the existing
    filesystem-backed config loader. This PR starts with the smaller piece
    we need now: a typed session config loader that can feed the existing
    config layer stack while preserving the normal precedence and merge
    behavior.
    
    ## What Changed
    
    - Added `ThreadConfigLoader` and related typed payloads in
    `codex-config`.
    - `SessionThreadConfig` currently supports `model_provider`,
    `model_providers`, and feature flags.
    - `UserThreadConfig` is present as an ownership boundary, but does not
    yet add TOML-backed fields.
    - `NoopThreadConfigLoader` preserves existing behavior when no external
    loader is configured.
      - `StaticThreadConfigLoader` supports tests and simple callers.
    
    - Taught thread config sources to produce ordinary `ConfigLayerEntry`
    values so the existing `ConfigLayerStack` remains the place where
    precedence and merging happen.
    
    - Wired the loader through `ConfigBuilder`, the config loader, and
    app-server startup paths so app-server can provide session-owned config
    before deriving a thread config.
    
    - Added coverage for:
      - translating typed thread config into config layers,
    - inserting thread config layers into the stack at the right precedence,
    - applying session-provided model provider and feature settings when
    app-server derives config from thread params.
    
    ## Follow-Ups
    
    This intentionally stops short of adding the remote/service transport.
    The next pieces are expected to be:
    
    1. Define the proto/API shape for this interface.
    2. Add a client implementation that can source session config from the
    service side.
    
    ## Verification
    
    - Added unit coverage in `codex-config` for the loader and layer
    conversion.
    - Added `codex-core` config loader coverage for thread config layer
    precedence.
    - Added app-server coverage that verifies session thread config wins
    over request-provided config for model provider and feature settings.
  • Async config loading (#18022)
    Parts of config will come from executor. Prepare for that by making
    config loading methods async.
  • app-server: prepare to run initialized rpcs concurrently (#17372)
    ## Summary
    
    - Refactors `MessageProcessor` and per-connection session state so
    initialized service RPC handling can be moved into spawned tasks in a
    follow-up PR.
    - Shares the processor and initialized session data with
    `Arc`/`OnceLock` instead of mutable borrowed connection state.
    - Keeps initialized request handling synchronous in this PR; it does
    **not** call `tokio::spawn` for service RPCs yet.
    
    ## Testing
    
    - `just fmt`
    - `cargo test -p codex-app-server` *(fails on existing hardening gaps
    covered by #17375, #17376, and #17377; the pipelined config regression
    passed before the unrelated failures)*
    - `just fix -p codex-app-server`
  • [codex] Support remote exec cwd in TUI startup (#17142)
    When running with remote executor the cwd is the remote path. Today we
    check for existence of a local directory on startup and attempt to load
    config from it.
    
    For remote executors don't do that.
  • app-server: Allow enabling remote control in runtime (#16973)
    Refresh the feature flag on writes to the config.
  • [codex-analytics] add protocol-native turn timestamps (#16638)
    ---
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/16638).
    * #16870
    * #16706
    * #16659
    * #16641
    * #16640
    * __->__ #16638
  • app-server: centralize AuthManager initialization (#16764)
    Extract a shared helper that builds AuthManager from Config and applies
    the forced ChatGPT workspace override in one place.
    
    Create the shared AuthManager at MessageProcessor call sites so that
    upcoming new transport's initialization can reuse the same handle, and
    keep only external auth refresher wiring inside `MessageProcessor`.
    
    Remove the now-unused `AuthManager::shared_with_external_auth` helper.
  • core: remove cross-crate re-exports from lib.rs (#16512)
    ## Why
    
    `codex-core` was re-exporting APIs owned by sibling `codex-*` crates,
    which made downstream crates depend on `codex-core` as a proxy module
    instead of the actual owner crate.
    
    Removing those forwards makes crate boundaries explicit and lets leaf
    crates drop unnecessary `codex-core` dependencies. In this PR, this
    reduces the dependency on `codex-core` to `codex-login` in the following
    files:
    
    ```
    codex-rs/backend-client/Cargo.toml
    codex-rs/mcp-server/tests/common/Cargo.toml
    ```
    
    ## What
    
    - Remove `codex-rs/core/src/lib.rs` re-exports for symbols owned by
    `codex-login`, `codex-mcp`, `codex-rollout`, `codex-analytics`,
    `codex-protocol`, `codex-shell-command`, `codex-sandboxing`,
    `codex-tools`, and `codex-utils-path`.
    - Delete the `default_client` forwarding shim in `codex-rs/core`.
    - Update in-crate and downstream callsites to import directly from the
    owning `codex-*` crate.
    - Add direct Cargo dependencies where callsites now target the owner
    crate, and remove `codex-core` from `codex-rs/backend-client`.
  • [codex-analytics] thread events (#15690)
    - add event for thread initialization
    - thread/start, thread/fork, thread/resume
    - feature flagged behind `FeatureFlag::GeneralAnalytics`
    - does not yet support threads started by subagents
    
    PR stack:
    - --> [[telemetry] thread events
    #15690](https://github.com/openai/codex/pull/15690)
    - [[telemetry] subagent events
    #15915](https://github.com/openai/codex/pull/15915)
    - [[telemetry] turn events
    #15591](https://github.com/openai/codex/pull/15591)
    - [[telemetry] steer events
    #15697](https://github.com/openai/codex/pull/15697)
    - [[telemetry] queued prompt data
    #15804](https://github.com/openai/codex/pull/15804)
    
    
    Sample extracted logs in Codex-backend
    ```
    INFO     | 2026-03-29 16:39:37 | codex_backend.routers.analytics_events | analytics_events.track_analytics_events:398 | Tracked analytics event codex_thread_initialized thread_id=019d3bf7-9f5f-7f82-9877-6d48d1052531 product_surface=codex product_client_id=CODEX_CLI client_name=codex-tui client_version=0.0.0 rpc_transport=in_process experimental_api_enabled=True codex_rs_version=0.0.0 runtime_os=macos runtime_os_version=26.4.0 runtime_arch=aarch64 model=gpt-5.3-codex ephemeral=False thread_source=user initialization_mode=new subagent_source=None parent_thread_id=None created_at=1774827577 | 
    INFO     | 2026-03-29 16:45:46 | codex_backend.routers.analytics_events | analytics_events.track_analytics_events:398 | Tracked analytics event codex_thread_initialized thread_id=019d3b84-5731-79d0-9b3b-9c6efe5f5066 product_surface=codex product_client_id=CODEX_CLI client_name=codex-tui client_version=0.0.0 rpc_transport=in_process experimental_api_enabled=True codex_rs_version=0.0.0 runtime_os=macos runtime_os_version=26.4.0 runtime_arch=aarch64 model=gpt-5.3-codex ephemeral=False thread_source=user initialization_mode=resumed subagent_source=None parent_thread_id=None created_at=1774820022 | 
    INFO     | 2026-03-29 16:45:49 | codex_backend.routers.analytics_events | analytics_events.track_analytics_events:398 | Tracked analytics event codex_thread_initialized thread_id=019d3bfd-4cd6-7c12-a13e-48cef02e8c4d product_surface=codex product_client_id=CODEX_CLI client_name=codex-tui client_version=0.0.0 rpc_transport=in_process experimental_api_enabled=True codex_rs_version=0.0.0 runtime_os=macos runtime_os_version=26.4.0 runtime_arch=aarch64 model=gpt-5.3-codex ephemeral=False thread_source=user initialization_mode=forked subagent_source=None parent_thread_id=None created_at=1774827949 | 
    INFO     | 2026-03-29 17:20:29 | codex_backend.routers.analytics_events | analytics_events.track_analytics_events:398 | Tracked analytics event codex_thread_initialized thread_id=019d3c1d-0412-7ed2-ad24-c9c0881a36b0 product_surface=codex product_client_id=CODEX_SERVICE_EXEC client_name=codex_exec client_version=0.0.0 rpc_transport=in_process experimental_api_enabled=True codex_rs_version=0.0.0 runtime_os=macos runtime_os_version=26.4.0 runtime_arch=aarch64 model=gpt-5.3-codex ephemeral=False thread_source=user initialization_mode=new subagent_source=None parent_thread_id=None created_at=1774830027 | 
    ```
    
    Notes
    - `product_client_id` gets canonicalized in codex-backend
    - subagent threads are addressed in a following pr
  • chore: clean up argument-comment lint and roll out all-target CI on macOS (#16054)
    ## Why
    
    `argument-comment-lint` was green in CI even though the repo still had
    many uncommented literal arguments. The main gap was target coverage:
    the repo wrapper did not force Cargo to inspect test-only call sites, so
    examples like the `latest_session_lookup_params(true, ...)` tests in
    `codex-rs/tui_app_server/src/lib.rs` never entered the blocking CI path.
    
    This change cleans up the existing backlog, makes the default repo lint
    path cover all Cargo targets, and starts rolling that stricter CI
    enforcement out on the platform where it is currently validated.
    
    ## What changed
    
    - mechanically fixed existing `argument-comment-lint` violations across
    the `codex-rs` workspace, including tests, examples, and benches
    - updated `tools/argument-comment-lint/run-prebuilt-linter.sh` and
    `tools/argument-comment-lint/run.sh` so non-`--fix` runs default to
    `--all-targets` unless the caller explicitly narrows the target set
    - fixed both wrappers so forwarded cargo arguments after `--` are
    preserved with a single separator
    - documented the new default behavior in
    `tools/argument-comment-lint/README.md`
    - updated `rust-ci` so the macOS lint lane keeps the plain wrapper
    invocation and therefore enforces `--all-targets`, while Linux and
    Windows temporarily pass `-- --lib --bins`
    
    That temporary CI split keeps the stricter all-targets check where it is
    already cleaned up, while leaving room to finish the remaining Linux-
    and Windows-specific target-gated cleanup before enabling
    `--all-targets` on those runners. The Linux and Windows failures on the
    intermediate revision were caused by the wrapper forwarding bug, not by
    additional lint findings in those lanes.
    
    ## Validation
    
    - `bash -n tools/argument-comment-lint/run.sh`
    - `bash -n tools/argument-comment-lint/run-prebuilt-linter.sh`
    - shell-level wrapper forwarding check for `-- --lib --bins`
    - shell-level wrapper forwarding check for `-- --tests`
    - `just argument-comment-lint`
    - `cargo test` in `tools/argument-comment-lint`
    - `cargo test -p codex-terminal-detection`
    
    ## Follow-up
    
    - Clean up remaining Linux-only target-gated callsites, then switch the
    Linux lint lane back to the plain wrapper invocation.
    - Clean up remaining Windows-only target-gated callsites, then switch
    the Windows lint lane back to the plain wrapper invocation.
  • Add cached environment manager for exec server URL (#15785)
    Add environment manager that is a singleton and is created early in
    app-server (before skill manager, before config loading).
    
    Use an environment variable to point to a running exec server.
  • tui_app_server: cancel active login before Ctrl+C exit (#15673)
    ## Summary
    
    Fixes slow `Ctrl+C` exit from the ChatGPT browser-login screen in
    `tui_app_server`.
    
    ## Root cause
    
    Onboarding-level `Ctrl+C` quit bypassed the auth widget's cancel path.
    That let the active ChatGPT login keep running, and in-process
    app-server shutdown then waited on the stale login attempt before
    finishing.
    
    ## Changes
    
    - Extract a shared `cancel_active_attempt()` path in the auth widget
    - Use that path from onboarding-level `Ctrl+C` before exiting the TUI
    - Add focused tests for canceling browser-login and device-code attempts
    - Add app-server shutdown cleanup that explicitly drops any active login
    before draining background work
  • app-server: Add back pressure and batching to command/exec (#15547)
    * Add
    `OutgoingMessageSender::send_server_notification_to_connection_and_wait`
    which returns only once message is written to websocket (or failed to do
    so)
    * Use this mechanism to apply back pressure to stdout/stderr streams of
    processes spawned by `command/exec`, to limit them to at most one
    message in-memory at a time
    * Use back pressure signal to also batch smaller chunks into ≈64KiB ones
    
    This should make commands execution more robust over
    high-latency/low-throughput networks
  • Finish moving codex exec to app-server (#15424)
    This PR completes the conversion of non-interactive `codex exec` to use
    app server rather than directly using core events and methods.
    
    ### Summary
    - move `codex-exec` off exec-owned `AuthManager` and `ThreadManager`
    state
    - route exec bootstrap, resume, and auth refresh through existing
    app-server paths
    - replace legacy `codex/event/*` decoding in exec with typed app-server
    notification handling
    - update human and JSONL exec output adapters to translate existing
    app-server notifications only
    - clean up "app server client" layer by eliminating support for legacy
    notifications; this is no longer needed
    - remove exposure of `authManager` and `threadManager` from "app server
    client" layer
    
    ### Testing
    - `exec` has pretty extensive unit and integration tests already, and
    these all pass
    - In addition, I asked Codex to put together a comprehensive manual set
    of tests to cover all of the `codex exec` functionality (including
    command-line options), and it successfully generated and ran these tests
  • Revert "fix: harden plugin feature gating" (#15102)
    Reverts openai/codex#15020
    
    I messed up the commit in my PR and accidentally merged changes that
    were still under review.
  • fix: harden plugin feature gating (#15020)
    1. Use requirement-resolved config.features as the plugin gate.
    2. Guard plugin/list, plugin/read, and related flows behind that gate.
    3. Skip bad marketplace.json files instead of failing the whole list.
    4. Simplify plugin state and caching.
  • Apply argument comment lint across codex-rs (#14652)
    ## Why
    
    Once the repo-local lint exists, `codex-rs` needs to follow the
    checked-in convention and CI needs to keep it from drifting. This commit
    applies the fallback `/*param*/` style consistently across existing
    positional literal call sites without changing those APIs.
    
    The longer-term preference is still to avoid APIs that require comments
    by choosing clearer parameter types and call shapes. This PR is
    intentionally the mechanical follow-through for the places where the
    existing signatures stay in place.
    
    After rebasing onto newer `main`, the rollout also had to cover newly
    introduced `tui_app_server` call sites. That made it clear the first cut
    of the CI job was too expensive for the common path: it was spending
    almost as much time installing `cargo-dylint` and re-testing the lint
    crate as a representative test job spends running product tests. The CI
    update keeps the full workspace enforcement but trims that extra
    overhead from ordinary `codex-rs` PRs.
    
    ## What changed
    
    - keep a dedicated `argument_comment_lint` job in `rust-ci`
    - mechanically annotate remaining opaque positional literals across
    `codex-rs` with exact `/*param*/` comments, including the rebased
    `tui_app_server` call sites that now fall under the lint
    - keep the checked-in style aligned with the lint policy by using
    `/*param*/` and leaving string and char literals uncommented
    - cache `cargo-dylint`, `dylint-link`, and the relevant Cargo
    registry/git metadata in the lint job
    - split changed-path detection so the lint crate's own `cargo test` step
    runs only when `tools/argument-comment-lint/*` or `rust-ci.yml` changes
    - continue to run the repo wrapper over the `codex-rs` workspace, so
    product-code enforcement is unchanged
    
    Most of the code changes in this commit are intentionally mechanical
    comment rewrites or insertions driven by the lint itself.
    
    ## Verification
    
    - `./tools/argument-comment-lint/run.sh --workspace`
    - `cargo test -p codex-tui-app-server -p codex-tui`
    - parsed `.github/workflows/rust-ci.yml` locally with PyYAML
    
    ---
    
    * -> #14652
    * #14651
  • fix: tui freeze when sub-agents are present (#14816)
    The issue was due to a circular `Drop` schema where the embedded
    app-server wait for some listeners that wait for this app-server
    them-selves.
    
    The fix is an explicit cleaning
    
    **Repro:**
    * Start codex
    * Ask it to spawn a sub-agent
    * Close Codex
    * It takes 5s to exit
  • Start TUI on embedded app server (#14512)
    This PR is part of the effort to move the TUI on top of the app server.
    In a previous PR, we introduced an in-process app server and moved
    `exec` on top of it.
    
    For the TUI, we want to do the migration in stages. The app server
    doesn't currently expose all of the functionality required by the TUI,
    so we're going to need to support a hybrid approach as we make the
    transition.
    
    This PR changes the TUI initialization to instantiate an in-process app
    server and access its `AuthManager` and `ThreadManager` rather than
    constructing its own copies. It also adds a placeholder TUI event
    handler that will eventually translate app server events into TUI
    events. App server notifications are accepted but ignored for now. It
    also adds proper shutdown of the app server when the TUI terminates.
  • feat(app-server): propagate traces across tasks and core ops (#14387)
    ## Summary
    
    This PR keeps app-server RPC request trace context alive for the full
    lifetime of the work that request kicks off (e.g. for `thread/start`,
    this is `app-server rpc handler -> tokio background task -> core op
    submissions`). Previously we lose trace lineage once the request handler
    returns or hands work off to background tasks.
    
    This approach is especially relevant for `thread/start` and other RPC
    handlers that run in a non-blocking way. In the near future we'll most
    likely want to make all app-server handlers run in a non-blocking way by
    default, and only queue operations that must operate in order (e.g.
    thread RPCs per thread?), so we want to make sure tracing in app-server
    just generally works.
    
    Depends on https://github.com/openai/codex/pull/14300
    
    **Before**
    <img width="155" height="207" alt="image"
    src="https://github.com/user-attachments/assets/c9487459-36f1-436c-beb7-fafeb40737af"
    />
    
    
    **After**
    <img width="299" height="337" alt="image"
    src="https://github.com/user-attachments/assets/727392b2-d072-4427-9dc4-0502d8652dea"
    />
    
    ## What changed
    
    - Keep request-scoped trace context around until we send the final
    response or error, or the connection closes.
    - Thread that trace context through detached `thread/start` work so
    background startup stays attached to the originating request.
    - Pass request trace context through to downstream core operations,
    including:
      - thread creation
      - resume/fork flows
      - turn submission
      - review
      - interrupt
      - realtime conversation operations
    - Add tracing tests that verify:
      - remote W3C trace context is preserved for `thread/start`
      - remote W3C trace context is preserved for `turn/start`
      - downstream core spans stay under the originating request span
      - request-scoped tracing state is cleaned up correctly
    - Clean up shutdown behavior so detached background tasks and spawned
    threads are drained before process exit.
  • chore(app-server): stop emitting codex/event/ notifications (#14392)
    ## Description
    
    This PR stops emitting legacy `codex/event/*` notifications from the
    public app-server transports.
    
    It's been a long time coming! app-server was still producing a raw
    notification stream from core, alongside the typed app-server
    notifications and server requests, for compatibility reasons. Now,
    external clients should no longer be depending on those legacy
    notifications, so this change removes them from the stdio and websocket
    contract and updates the surrounding docs, examples, and tests to match.
    
    ### Caveat
    I left the "in-process" version of app-server alone for now, since
    `codex exec` was recently based on top of app-server via this in-process
    form here: https://github.com/openai/codex/pull/14005
    
    Seems like `codex exec` still consumes some legacy notifications
    internally, so this branch only removes `codex/event/*` from app-server
    over stdio and websockets.
    
    ## Follow-up
    
    Once `codex exec` is fully migrated off `codex/event/*` notifications,
    we'll be able to stop emitting them entirely entirely instead of just
    filtering it at the external transport boundary.
  • Add in-process app server and wire up exec to use it (#14005)
    This is a subset of PR #13636. See that PR for a full overview of the
    architectural change.
    
    This PR implements the in-process app server and modifies the
    non-interactive "exec" entry point to use the app server.
    
    ---------
    
    Co-authored-by: Felipe Coury <felipe.coury@gmail.com>