Commit Graph

123 Commits

  • [codex] Refactor app-server dispatch result flow (#20897)
    ## Why
    
    App-server request handling had response sending spread across many
    individual handlers, which made it harder to see which requests return
    payloads, which methods send their own delayed response, and which
    branches emit notifications after a response.
    
    ## What changed
    
    - Centralized normal `ClientResponsePayload` sending in the dispatch
    path.
    - Kept explicit-response methods explicit where they need custom
    ordering or delayed delivery.
    - Removed forward-only handler wrappers and immediate `async { ...
    }.await` bodies where they were not needed.
    - Moved branch-specific post-response notifications into the branches
    that own the response ordering.
    - Replaced unreachable delegated request-family error arms with explicit
    `unreachable!` cases.
    
    ## Verification
    
    - `cargo check -p codex-app-server`
    - `cargo test -p codex-app-server thread_goal`
    - `just fix -p codex-app-server`
  • Make thread store process-scoped (#19474)
    - Build one app-server process ThreadStore from startup config and share
    it with ThreadManager and CodexMessageProcessor.
    - Remove per-thread/fork store reconstruction so effective thread config
    cannot switch the persistence backend.
    - Add params to ThreadStore create/resume for specifying thread
    metadata, since otherwise the metadata from store creation would be used
    (incorrectly).
  • Reduce the surface of collaboration modes (#20149)
    Collaboration modes were slightly invasive both into ThreadManager
    construction and ModelProvider
  • Import external agent sessions in background (#20284)
    Summary:
    - Return from external agent import before session history import
    finishes
    - Run session import work in the background and emit the existing
    completion notification when it is done
    - Serialize session imports so duplicate requests do not create
    duplicate imported threads
    
    Verification:
    - cargo test -p codex-app-server external_agent_config_
    - cargo test -p codex-external-agent-sessions
    - just fix -p codex-app-server
    - just fix -p codex-external-agent-sessions
    - git diff --check
  • [app-server] type client response payloads (#20050)
    ## Why
    
    `pr17088` adds typed server-originated request/response plumbing, but
    successful client responses are still erased into bare JSON-RPC `result`
    values before app-server can make any typed decision about them.
    
    This precursor PR keeps successful client responses typed until the
    outgoing response seam. It is intentionally limited to
    protocol/app-server plumbing so the analytics behavior change can review
    separately on top.
    
    ## What changed
    
    - Add `ClientResponsePayload` as the pre-serialization client response
    body type.
    - Route app-server successful response paths through the typed payload
    seam while preserving existing handler-local analytics behavior.
    - Keep `InterruptConversation` JSON-RPC-only because it has no
    `ClientResponse` variant.
    - Move the new payload conversion tests into a dedicated protocol test
    module.
    
    ## Verification
    
    - `cargo check -p codex-app-server`
    - `cargo test -p codex-app-server-protocol`
  • [codex-analytics] ingest server requests and responses (#17088)
    ## Why
    
    Codex analytics needs a typed seam for app-server-originated
    request/response traffic so future tool-approval analytics can consume
    those facts without adding bespoke callsite tracking each time. Server
    responses arrive as JSON-RPC `id + result` payloads, so analytics has to
    reconstruct the matching typed response from the original typed request
    while that request context still exists in app-server.
    
    This also puts analytics on the app-server outbound path, which needs to
    avoid keeping the runtime alive during shutdown. The final ownership fix
    keeps the normal strong auth-manager retention in analytics and makes
    the external-auth refresh bridge hold a weak back-reference to
    `OutgoingMessageSender`, breaking the runtime cycle at the bridge
    boundary instead of exposing retention policy through the analytics
    client API.
    
    ## What changed
    
    - Adds typed `ServerRequest` and `ServerResponse` analytics facts, plus
    `AnalyticsEventsClient::track_server_request` and
    `track_server_response`.
    - Renames the existing client-side facts to `ClientRequest` and
    `ClientResponse` so reducers can distinguish client-to-server traffic
    from server-to-client traffic.
    - Adds `ServerRequest::response_from_result`, allowing a stored typed
    request to decode the matching typed server response from a raw JSON-RPC
    result payload.
    - Threads `AnalyticsEventsClient` through `OutgoingMessageSender` and
    records targeted server requests, replayed targeted requests, and
    matching targeted responses with the responding connection id needed for
    correlation.
    - Intentionally leaves broadcast server requests/responses out of
    analytics for now because the current model is per connection, while
    broadcasts fan one logical request out across multiple connections.
    - Breaks the app-server shutdown cycle by storing
    `Weak<OutgoingMessageSender>` in `ExternalAuthRefreshBridge` and
    upgrading it only when an external-auth refresh is actually requested.
    - Keeps reducer ingestion of the new server-side facts as no-ops for
    now; this PR is plumbing for later tool-approval analytics work.
    
    ## Verification
    
    - `cargo test -p codex-analytics`
    - `cargo test -p codex-app-server outgoing_message::tests::`
    - Covers typed-response reconstruction plus the targeted, replayed,
    broadcast-exclusion, and response-attribution analytics paths.
    
    ## Follow-up
    
    This PR intentionally stops at ingestion plumbing, so `ServerRequest`
    and `ServerResponse` facts are still reducer no-ops. Once a follow-up PR
    adds real downstream analytics output for those facts:
    
    - replace the temporary pre-reducer observation seam with reducer tests
    for the emitted event shape;
    - add end-to-end coverage in `app-server/tests/suite/v2/analytics.rs`
    for the real app-server workflow and captured analytics payload;
    - remove the temporary sender-level observer tests added here in favor
    of the real-output coverage above.
    
    ---
    
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/17088).
    * #18748
    * #18747
    * #17090
    * #17089
    * #20241
    * #20239
    * __->__ #17088
  • feat: Use remote installed plugin cache for skills and MCP (#20096)
    - Fetches and caches remote /installed plugin state
    - Lets skills/list load skills from remote-installed cached plugins
    without requiring a local marketplace entry
    - Routes plugin list/startup/install/uninstall changes through async
    plugin cache invalidation and MCP refresh
  • feat: expose provider capability bounds to app server clients (#20049)
    follow up of #19442. The app server now exposes provider-derived bounds
    through a new v2 `modelProvider/read` method. The response reports the
    configured provider map key as `modelProvider` and returns the effective
    capability booleans so clients can align their UI with the same
    provider-owned limits used by core.
  • Support detect and import MCP, Subagents, hooks, commands from external (#19949)
    ## Why
    This PR expands the migration path so Codex can detect and import MCP
    server config, hooks, commands, and subagents configs in a Codex-native
    shape.
    
    ## What changed
    
    - Added a `codex-external-agent-migration` crate that owns conversion
    logic for external-agent MCP servers, hooks, commands, and subagents.
    - Extended the app-server external-agent config detection/import API
    with migration item types for MCP server config, hooks, commands, and
    subagents.
    
    ## Migration strategy
    
    The migration is intentionally conservative: Codex only imports
    external-agent config that can be represented safely in Codex today.
    Unsupported or ambiguous config is skipped instead of being partially
    translated into behavior that may not match the source system.
    
    - **MCP servers**: import supported stdio and HTTP MCP server
    definitions into `mcp_servers`. Disabled servers and servers filtered
    out by source `enabledMcpjsonServers` / `disabledMcpjsonServers` are
    skipped. Project-scoped MCP entries from `.claude.json` are included
    when they match the repo path.
    - **Hooks**: import only supported command hooks into
    `.codex/hooks.json`. Unsupported hook features such as conditional
    groups, async handlers, prompt/http hooks, or unknown fields are
    skipped. Referenced hook scripts are copied into `.codex/hooks/`,
    preserving any existing target scripts.
    - **Commands**: import supported external commands as Codex skills under
    `.agents/skills/source-command-*`. Commands that rely on source runtime
    expansion such as `$ARGUMENTS`, `$1`, `@file` references, shell
    interpolation, or colliding generated names are skipped.
    - **Subagents**: import valid subagent Markdown files into
    `.codex/agents/*.toml` when they have the minimum Codex agent fields.
    Source model names are not migrated, so imported agents keep the user’s
    Codex default model; compatible reasoning effort and sandbox mode are
    migrated when present.
    - **Skills and project guidance**: copy missing skill directories into
    `.agents/skills` and migrate `CLAUDE.md` guidance into `AGENTS.md`,
    rewriting source-agent terminology to Codex terminology where
    appropriate.
    - **Detection details**: detected migration items include lightweight
    details for UI preview, such as MCP server names, hook event names,
    generated command skill names, and subagent names. Import still
    recomputes from disk instead of trusting details as the source of truth.
    
    - Adds focused coverage for the new migration behavior and app-server
    import flow.
    
    ## Verification
    
    - `cargo test -p codex-external-agent-migration`
    - `cargo test -p codex-hooks`
    - `cargo test -p codex-app-server external_agent_config`
    - `just bazel-lock-check`
  • app-server: run initialized rpcs with keyed serialization (#17373)
    ## Why
    
    Initialized app-server RPCs no longer need to bottleneck behind one
    request processor path. Running them concurrently improves
    responsiveness, but several request families still mutate shared state
    or depend on ordered side effects. Those stateful families need an
    auditable serialization contract so concurrency does not reorder thread,
    config, auth, command, watcher, MCP, or similar state transitions.
    
    This PR keeps that boundary explicit: stateful work is serialized by the
    smallest useful key, while intentionally read-only or externally
    concurrent work remains unkeyed. In particular, `thread/list` and
    `thread/turns/list` explicitly have no serialization because they
    primarily read append-only rollout storage and should continue to be
    served concurrently.
    
    ## What changed
    
    - Adds `ClientRequest::serialization_scope()` in `app-server-protocol`
    and requires every client request definition to declare its
    serialization behavior.
    - Introduces keyed request scopes for thread, thread path, command exec
    process, fuzzy search session, fs watch, MCP OAuth, and global state
    buckets such as config, account auth, memory, and device keys.
    - Routes initialized app-server RPCs through per-key FIFO serialization
    while allowing unkeyed initialized requests to run concurrently.
    - Cancels in-flight initialized RPC work when the connection disconnects
    or the app-server exits so spawned request tasks do not outlive their
    session.
    - Adds focused coverage for representative keyed and unkeyed
    serialization scopes, including explicitly concurrent
    `thread/turns/list` behavior.
    
    ## Validation
    
    - Added protocol tests for representative keyed serialization scopes and
    intentionally unkeyed request families.
    - Added app-server request serialization tests covering per-key FIFO
    behavior, concurrent unkeyed execution, disconnect shutdown, and config
    read-after-write ordering.
    - Local focused protocol validation after the latest rebase is currently
    blocked by packageproxy failing to resolve locked `rustls-webpki
    0.103.13`; CI is expected to provide the full validation signal.
  • External agent session support (#19895)
    ## Summary
    
    This extends external agent detection/import beyond config artifacts so
    Codex can detect recent sessions files from the external agent home and
    import them into Codex rollout history.
    
    ## What changed
    
    - Added a focused `external_agent_sessions` module for:
      - session discovery
      - source-record parsing
      - rollout construction
      - import ledger tracking
    - Wired session detection/import into the app-server external agent
    config API.
    - Added compaction handling so large imported sessions can be resumed
    safely before the first follow-up turn.
    
    ## Testing
    
    Added coverage for:
    - recent-session detection
    - custom-title handling
    - recency filtering
    - dedupe and re-detect-after-source-change behavior
    - visible imported turn construction
    - backward-compatible import payload deserialization
    - end-to-end RPC import flow
    - rejection of undetected session paths
    - repeat-import behavior
    - large-session compaction before first follow-up
    
    Ran:
    - `cargo test -p codex-app-server external_agent_config_import_ --test
    all`
  • Lift app-server JSON-RPC error handling to request boundary (#19484)
    ## Why
    
    App-server request handling had a lot of repeated JSON-RPC error
    construction and one-off `send_error`/`return` branches. This made small
    handlers noisy and pushed error response details into leaf code that
    otherwise only needed to validate input or call the underlying API.
    
    ## What Changed
    
    - Added shared JSON-RPC error constructors in
    `codex-rs/app-server/src/error_code.rs`.
    - Lifted straightforward request result emission into
    `codex-rs/app-server/src/message_processor.rs` so response/error
    dispatch happens at the request boundary.
    - Reused the result helpers across command exec, config, filesystem,
    device-key, external-agent config, fs-watch, and outgoing-message paths.
    - Removed leaf wrapper handlers where the method body was only
    forwarding to a response helper.
    - Returned request validation errors upward in the simple cases instead
    of sending an error locally and immediately returning.
    
    ## Verification
    
    - `cargo test -p codex-app-server --lib command_exec::tests`
    - `cargo test -p codex-app-server --lib outgoing_message::tests`
    - `cargo test -p codex-app-server --lib in_process::tests`
    - `cargo test -p codex-app-server --test all v2::fs`
    - `cargo test -p codex-app-server --test all v2::config_rpc`
    - `cargo test -p codex-app-server --test all v2::external_agent_config`
    - `cargo test -p codex-app-server --test all v2::initialize`
    - `just fix -p codex-app-server`
    - `git diff --check`
    
    Note: full `cargo test -p codex-app-server` was attempted and stopped in
    `message_processor::tracing_tests::turn_start_jsonrpc_span_parents_core_turn_spans`
    with a stack overflow after unrelated tests had already passed.
  • test: harden app-server integration tests (#19683)
    ## Why
    
    Windows Bazel runs in the permissions stack exposed that app-server
    integration tests were launching normal plugin startup warmups in every
    subprocess. Those warmups can call
    `https://chatgpt.com/backend-api/plugins/featured` when a test is not
    specifically exercising plugin startup, which adds slow background work,
    noisy stderr, and dependence on external network state. The relevant
    startup/featured-plugin behavior was introduced across #15042 and
    #15264.
    
    A few app-server tests also had long optional waits or unbounded cleanup
    paths, making failures expensive to diagnose and contributing to slow
    Windows shards. One external-agent config test from #18246 used a
    GitHub-style marketplace source, which was enough to exercise the
    pending remote-import path but also meant the background completion task
    could attempt a real clone.
    
    ## What Changed
    
    - Adds explicit `AppServerRuntimeOptions` / `PluginStartupTasks`
    plumbing and a hidden debug-only
    `--disable-plugin-startup-tasks-for-tests` app-server flag, so
    integration tests can suppress startup plugin warmups without adding a
    production env-var gate.
    - Has the app-server test harness pass that hidden flag by default,
    while opting plugin-startup coverage back in for tests that
    intentionally exercise startup sync and featured-plugin warmup behavior.
    - Lowers normal app-server subprocess logging from `info`/`debug` to
    `warn` to avoid multi-megabyte stderr output in Bazel logs.
    - Prevents the external-agent config test from attempting a real
    marketplace clone by using an invalid non-local source while still
    exercising the pending-import completion path.
    - Bounds optional filesystem/realtime waits and fake WebSocket
    test-server shutdown so failures produce targeted timeouts instead of
    hanging a shard.
    - Fixes the Unix script-resolution test in `rmcp-client` to exercise
    PATH resolution directly and include the actual spawn error in failures.
    
    ## Verification
    
    - `cargo check -p codex-app-server`
    - `cargo clippy -p codex-app-server --tests -- -D warnings`
    - `cargo test -p codex-rmcp-client
    program_resolver::tests::test_unix_executes_script_without_extension`
    - `cargo test -p codex-app-server --test all
    external_agent_config_import_sends_completion_notification_after_pending_plugins_finish
    -- --nocapture`
    - `cargo test -p codex-app-server --test all
    plugin_list_uses_warmed_featured_plugin_ids_cache_on_first_request --
    --nocapture`
    - Windows Local Bazel passed with this test-hardening bundle before it
    was extracted from #19606.
    
    ---
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/19683).
    * #19395
    * #19394
    * #19393
    * #19392
    * #19606
    * __->__ #19683
  • app-server: persist device key bindings in sqlite (#19206)
    ## Why
    
    Device-key providers should only own platform key material. The
    account/client binding used to authorize a signing payload is app-server
    state, and keeping that state in provider-specific metadata makes the
    same check harder to audit and harder to share across platform
    implementations.
    
    Persisting the binding in the shared state database gives the device-key
    crate a platform-neutral source of truth before it asks a provider to
    sign. It also lets app-server move potentially blocking key operations
    off the main message processor path, which matters once providers may
    wait for OS authentication prompts.
    
    ## What changed
    
    - Add a `device_key_bindings` state migration plus `StateRuntime`
    helpers keyed by `key_id`.
    - Add an async `DeviceKeyBindingStore` abstraction to `codex-device-key`
    and use it from `DeviceKeyStore::create` and `DeviceKeyStore::sign`.
    - Keep provider calls behind async store methods and run the synchronous
    provider work through `spawn_blocking`.
    - Wire app-server device-key RPC handling to the SQLite-backed binding
    store and spawn response/error delivery tasks for device-key requests.
    - Run the turn-start tracing test on the existing larger current-thread
    test harness after the larger async surface made the default test stack
    too small locally.
    
    ## Validation
    
    - `cargo test -p codex-device-key`
    - `cargo test -p codex-state device_key`
    - `cargo test -p codex-state`
    - `cargo test -p codex-app-server device_key`
    - `cargo test -p codex-app-server
    message_processor::tracing_tests::turn_start_jsonrpc_span_parents_core_turn_spans`
    - `cargo test -p codex-app-server`
    - `just fix -p codex-device-key`
    - `just fix -p codex-state`
    - `just fix -p codex-app-server`
    - `just bazel-lock-update`
    - `just bazel-lock-check`
    - `git diff --check`
  • refactor: route Codex auth through AuthProvider (#18811)
    ## Summary
    
    This PR moves Codex backend request authentication from direct
    bearer-token handling to `AuthProvider`.
    
    The new `codex-auth-provider` crate defines the shared request-auth
    trait. `CodexAuth::provider()` returns a provider that can apply all
    headers needed for the selected auth mode.
    
    This lets ChatGPT token auth and AgentIdentity auth share the same
    callsite path:
    - ChatGPT token auth applies bearer auth plus account/FedRAMP headers
    where needed.
    - AgentIdentity auth applies AgentAssertion plus account/FedRAMP headers
    where needed.
    
    Reference old stack: https://github.com/openai/codex/pull/17387/changes
    
    ## Callsite Migration
    
    | Area | Change |
    | --- | --- |
    | backend-client | accepts an `AuthProvider` instead of a raw
    token/header |
    | chatgpt client/connectors | applies auth through
    `CodexAuth::provider()` |
    | cloud tasks | keeps Codex-backend gating, applies auth through
    provider |
    | cloud requirements | uses Codex-backend auth checks and provider
    headers |
    | app-server remote control | applies provider headers for backend calls
    |
    | MCP Apps/connectors | gates on `uses_codex_backend()` and keys caches
    from generic account getters |
    | model refresh | treats AgentIdentity as Codex-backend auth |
    | OpenAI file upload path | rejects non-Codex-backend auth before
    applying headers |
    | core client setup | keeps model-provider auth flow and allows
    AgentIdentity through provider-backed OpenAI auth |
    
    ## Stack
    
    1. https://github.com/openai/codex/pull/18757: full revert
    2. https://github.com/openai/codex/pull/18871: isolated Agent Identity
    crate
    3. https://github.com/openai/codex/pull/18785: explicit AgentIdentity
    auth mode and startup task allocation
    4. This PR: migrate Codex backend auth callsites through AuthProvider
    5. https://github.com/openai/codex/pull/18904: accept AgentIdentity JWTs
    and load `CODEX_AGENT_IDENTITY`
    
    ## Testing
    
    Tests: targeted Rust checks, cargo-shear, Bazel lock check, and CI.
  • app-server: add Unix socket transport (#18255)
    ## Summary
    - add unix:// app-server transport backed by the shared codex-uds crate
    - reuse the websocket connection loop for axum and tungstenite-backed
    streams
    - add codex app-server proxy to bridge stdio clients to the control
    socket
    - tolerate Windows UDS backends that report a missing rendezvous path as
    connection refused before binding
    
    ## Tests
    - cargo test -p codex-app-server
    control_socket_acceptor_forwards_websocket_text_messages_and_pings
    - cargo test -p codex-app-server
    - just fmt
    - just fix -p codex-app-server
    - git -c core.fsmonitor=false diff --check
  • Support multiple managed environments (#18401)
    ## Summary
    - refactor EnvironmentManager to own keyed environments with
    default/local lookup helpers
    - keep remote exec-server client creation lazy until exec/fs use
    - preserve disabled agent environment access separately from internal
    local environment access
    
    ## Validation
    - not run (per Codex worktree instruction to avoid tests/builds unless
    requested)
    
    ---------
    
    Co-authored-by: Codex <noreply@openai.com>
  • app-server: implement device key v2 methods (#18430)
    ## Why
    
    The device-key protocol needs an app-server implementation that keeps
    local key operations behind the same request-processing boundary as
    other v2 APIs.
    
    app-server owns request dispatch, transport policy, documentation, and
    JSON-RPC error shaping. `codex-device-key` owns key binding, validation,
    platform provider selection, and signing mechanics. Keeping the adapter
    thin makes the boundary easier to review and avoids moving local
    key-management details into thread orchestration code.
    
    ## What changed
    
    - Added `DeviceKeyApi` as the app-server adapter around
    `DeviceKeyStore`.
    - Converted protocol protection policies, payload variants, algorithms,
    and protection classes to and from the device-key crate types.
    - Encoded SPKI public keys and DER signatures as base64 protocol fields.
    - Routed `device/key/create`, `device/key/public`, and `device/key/sign`
    through `MessageProcessor`.
    - Rejected remote transports before provider access while allowing local
    `stdio` and in-process callers to reach the device-key API.
    - Added stdio, in-process, and websocket tests for device-key validation
    and transport policy.
    - Documented the device-key methods in the app-server v2 method list.
    
    ## Test coverage
    
    - `device_key_create_rejects_empty_account_user_id`
    - `in_process_allows_device_key_requests_to_reach_device_key_api`
    - `device_key_methods_are_rejected_over_websocket`
    
    ## Stack
    
    This is PR 3 of 4 in the device-key app-server stack. It is stacked on
    #18429.
    
    ## Validation
    
    - `cargo test -p codex-app-server device_key`
    - `just fix -p codex-app-server`
  • Load app-server config through ConfigManager (#18870)
    ## Summary
    - Load app-server startup config through `ConfigManager` instead of
    direct `ConfigBuilder` calls.
    - Move `ConfigManager` constructor-owned state (`cli_overrides`, runtime
    feature map, cloud requirements loader) behind internal manager fields.
    - Pass `ConfigManager` into `MessageProcessor` directly instead of
    reconstructing it from raw args.
    
    ## Tests
    - `cargo check -p codex-app-server`
    - `cargo test -p codex-app-server`
    - `just fix -p codex-app-server`
    - `just fmt`
  • Refactor app-server config loading into ConfigManager (#18442)
    Localize app-server configuration loading in one place.
  • Add session config loader interface (#18208)
    ## Why
    
    Cloud-hosted sessions need a way for the service that starts or manages
    a thread to provide session-owned config without treating all config as
    if it came from the same user/project/workspace TOML stack.
    
    The important boundary is ownership: some values should be controlled by
    the session/orchestrator, some by the authenticated user, and later some
    may come from the executor. The earlier broad config-store shape made
    that boundary too fuzzy and overlapped heavily with the existing
    filesystem-backed config loader. This PR starts with the smaller piece
    we need now: a typed session config loader that can feed the existing
    config layer stack while preserving the normal precedence and merge
    behavior.
    
    ## What Changed
    
    - Added `ThreadConfigLoader` and related typed payloads in
    `codex-config`.
    - `SessionThreadConfig` currently supports `model_provider`,
    `model_providers`, and feature flags.
    - `UserThreadConfig` is present as an ownership boundary, but does not
    yet add TOML-backed fields.
    - `NoopThreadConfigLoader` preserves existing behavior when no external
    loader is configured.
      - `StaticThreadConfigLoader` supports tests and simple callers.
    
    - Taught thread config sources to produce ordinary `ConfigLayerEntry`
    values so the existing `ConfigLayerStack` remains the place where
    precedence and merging happen.
    
    - Wired the loader through `ConfigBuilder`, the config loader, and
    app-server startup paths so app-server can provide session-owned config
    before deriving a thread config.
    
    - Added coverage for:
      - translating typed thread config into config layers,
    - inserting thread config layers into the stack at the right precedence,
    - applying session-provided model provider and feature settings when
    app-server derives config from thread params.
    
    ## Follow-Ups
    
    This intentionally stops short of adding the remote/service transport.
    The next pieces are expected to be:
    
    1. Define the proto/API shape for this interface.
    2. Add a client implementation that can source session config from the
    service side.
    
    ## Verification
    
    - Added unit coverage in `codex-config` for the loader and layer
    conversion.
    - Added `codex-core` config loader coverage for thread config layer
    precedence.
    - Added app-server coverage that verifies session thread config wins
    over request-provided config for model provider and feature settings.
  • Sync local plugin imports, async remote imports, refresh caches after… (#18246)
    … import
    
    ## Why
    
    `externalAgentConfig/import` used to spawn plugin imports in the
    background and return immediately. That meant local marketplace imports
    could still be in flight when the caller refreshed plugin state, so
    newly imported plugins would not show up right away.
    
    This change makes local marketplace imports complete before the RPC
    returns, while keeping remote marketplace imports asynchronous so we do
    not block on remote fetches.
    
    ## What changed
    
    - split plugin migration details into local and remote marketplace
    imports based on the external config source
    - import local marketplaces synchronously during
    `externalAgentConfig/import`
    - return pending remote plugin imports to the app-server so it can
    finish them in the background
    - clear the plugin and skills caches before responding to plugin
    imports, and again after background remote imports complete, so the next
    `plugin/list` reloads fresh state
    - keep marketplace source parsing encapsulated behind
    `is_local_marketplace_source(...)` instead of re-exporting the internal
    enum
    - add core and app-server coverage for the synchronous local import path
    and the pending remote import path
    
    ## Verification
    
    - `cargo test -p codex-app-server-protocol`
    - `cargo test -p codex-core` (currently fails an existing unrelated
    test:
    `config_loader::tests::cli_override_can_update_project_local_mcp_server_when_project_is_trusted`)
    - `cargo test` (currently fails existing `codex-app-server` integration
    tests in MCP/skills/thread-start areas, plus the unrelated `codex-core`
    failure above)
  • [codex] Fix app-server initialized request analytics build (#17830)
    Problem: PR #17372 moved initialized request handling into
    `dispatch_initialized_client_request`, leaving analytics code that uses
    `connection_id` without a local binding and breaking `codex-app-server`
    builds.
    
    Solution: Restore the `connection_id` binding from
    `connection_request_id` before initialized request validation and
    analytics tracking.
  • app-server: prepare to run initialized rpcs concurrently (#17372)
    ## Summary
    
    - Refactors `MessageProcessor` and per-connection session state so
    initialized service RPC handling can be moved into spawned tasks in a
    follow-up PR.
    - Shares the processor and initialized session data with
    `Arc`/`OnceLock` instead of mutable borrowed connection state.
    - Keeps initialized request handling synchronous in this PR; it does
    **not** call `tokio::spawn` for service RPCs yet.
    
    ## Testing
    
    - `just fmt`
    - `cargo test -p codex-app-server` *(fails on existing hardening gaps
    covered by #17375, #17376, and #17377; the pipelined config regression
    passed before the unrelated failures)*
    - `just fix -p codex-app-server`
  • [codex-analytics] feature plumbing and emittance (#16640)
    ---
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/16640).
    * #16870
    * #16706
    * #16641
    * __->__ #16640
  • [codex-analytics] add compaction analytics event (#17155)
    - event for compaction analytics
    - introduces thread-connection and thread metadata caches for data
    denormalization, expected to be useful for denormalization onto core
    emitted events in general
    - threads analytics event client into core (mirrors approved
    implementation in #16640)
    - denormalizes key thread metadata: thread_source, subagent_source,
    parent_thread_id, as well as app-server client and runtime metadata)
    - compaction strategy defaults to memento, forward compatible with
    expected prefill_compaction strategy
    
    1. Manual standalone compact, local
    `INFO | 2026-04-09 17:35:50 | codex_backend.routers.analytics_events |
    analytics_events.track_analytics_events:526 | Tracked
    codex_compaction_event event params={'thread_id':
    '019d74d0-5cfb-70c0-bef9-165c3bf9b2df', 'turn_id':
    '019d74d0-d7f6-7c81-acc6-aae2030243d6', 'product_surface': 'codex',
    'app_server_client': {'product_client_id': 'CODEX_CLI', 'client_name':
    'codex-tui', 'client_version': '0.0.0', 'rpc_transport': 'in_process',
    'experimental_api_enabled': True}, 'runtime': {'codex_rs_version':
    '0.0.0', 'runtime_os': 'macos', 'runtime_os_version': '26.4.0',
    'runtime_arch': 'aarch64'}, 'trigger': 'manual', 'reason':
    'user_requested', 'implementation': 'responses', 'phase':
    'standalone_turn', 'strategy': 'memento', 'status': 'completed',
    'active_context_tokens_before': 20170, 'active_context_tokens_after':
    4830, 'started_at': 1775781337, 'completed_at': 1775781350,
    'thread_source': 'user', 'subagent_source': None, 'parent_thread_id':
    None, 'error': None, 'duration_ms': 13524} | `
    
    2. Auto pre-turn compact, local
    `INFO | 2026-04-09 17:37:30 | codex_backend.routers.analytics_events |
    analytics_events.track_analytics_events:526 | Tracked
    codex_compaction_event event params={'thread_id':
    '019d74d2-45ef-71d1-9c93-23cc0c13d988', 'turn_id':
    '019d74d2-7b42-7372-9f0e-c0da3f352328', 'product_surface': 'codex',
    'app_server_client': {'product_client_id': 'CODEX_CLI', 'client_name':
    'codex-tui', 'client_version': '0.0.0', 'rpc_transport': 'in_process',
    'experimental_api_enabled': True}, 'runtime': {'codex_rs_version':
    '0.0.0', 'runtime_os': 'macos', 'runtime_os_version': '26.4.0',
    'runtime_arch': 'aarch64'}, 'trigger': 'auto', 'reason':
    'context_limit', 'implementation': 'responses', 'phase': 'pre_turn',
    'strategy': 'memento', 'status': 'completed',
    'active_context_tokens_before': 20063, 'active_context_tokens_after':
    4822, 'started_at': 1775781444, 'completed_at': 1775781449,
    'thread_source': 'user', 'subagent_source': None, 'parent_thread_id':
    None, 'error': None, 'duration_ms': 5497} | `
    
    3. Auto mid-turn compact, local
    `INFO | 2026-04-09 17:38:28 | codex_backend.routers.analytics_events |
    analytics_events.track_analytics_events:526 | Tracked
    codex_compaction_event event params={'thread_id':
    '019d74d3-212f-7a20-8c0a-4816a978675e', 'turn_id':
    '019d74d3-3ee1-7462-89f6-2ffbeefcd5e3', 'product_surface': 'codex',
    'app_server_client': {'product_client_id': 'CODEX_CLI', 'client_name':
    'codex-tui', 'client_version': '0.0.0', 'rpc_transport': 'in_process',
    'experimental_api_enabled': True}, 'runtime': {'codex_rs_version':
    '0.0.0', 'runtime_os': 'macos', 'runtime_os_version': '26.4.0',
    'runtime_arch': 'aarch64'}, 'trigger': 'auto', 'reason':
    'context_limit', 'implementation': 'responses', 'phase': 'mid_turn',
    'strategy': 'memento', 'status': 'completed',
    'active_context_tokens_before': 20325, 'active_context_tokens_after':
    14641, 'started_at': 1775781500, 'completed_at': 1775781508,
    'thread_source': 'user', 'subagent_source': None, 'parent_thread_id':
    None, 'error': None, 'duration_ms': 7507} | `
    
    4. Remote /responses/compact, manual standalone
    `INFO | 2026-04-09 17:40:20 | codex_backend.routers.analytics_events |
    analytics_events.track_analytics_events:526 | Tracked
    codex_compaction_event event params={'thread_id':
    '019d74d4-7a11-78a1-89f7-0535a1149416', 'turn_id':
    '019d74d4-e087-7183-9c20-b1e40b7578c0', 'product_surface': 'codex',
    'app_server_client': {'product_client_id': 'CODEX_CLI', 'client_name':
    'codex-tui', 'client_version': '0.0.0', 'rpc_transport': 'in_process',
    'experimental_api_enabled': True}, 'runtime': {'codex_rs_version':
    '0.0.0', 'runtime_os': 'macos', 'runtime_os_version': '26.4.0',
    'runtime_arch': 'aarch64'}, 'trigger': 'manual', 'reason':
    'user_requested', 'implementation': 'responses_compact', 'phase':
    'standalone_turn', 'strategy': 'memento', 'status': 'completed',
    'active_context_tokens_before': 23461, 'active_context_tokens_after':
    6171, 'started_at': 1775781601, 'completed_at': 1775781620,
    'thread_source': 'user', 'subagent_source': None, 'parent_thread_id':
    None, 'error': None, 'duration_ms': 18971} | `
  • app-server: Use shared receivers for app-server message processors (#17256)
    We do not rely on the mutability here, so express it in the type system.
  • app-server: Allow enabling remote control in runtime (#16973)
    Refresh the feature flag on writes to the config.
  • Refactor config types into a separate crate (#16962)
    Move config types into a separate crate because their macros expand into
    a lot of new code.
  • feat: refresh non-curated cache from plugin list. (#16191)
    1. Use versions for non-curated plugin (defined in plugin.json) for
    cache refresh
    2. Trigger refresh from plugin/list roots
  • app-server: centralize AuthManager initialization (#16764)
    Extract a shared helper that builds AuthManager from Config and applies
    the forced ChatGPT workspace override in one place.
    
    Create the shared AuthManager at MessageProcessor call sites so that
    upcoming new transport's initialization can reuse the same handle, and
    keep only external auth refresher wiring inside `MessageProcessor`.
    
    Remove the now-unused `AuthManager::shared_with_external_auth` helper.
  • [codex-analytics] subagent analytics (#15915)
    - creates custom event that emits subagent thread analytics from core
    - wires client metadata (`product_client_id, client_name,
    client_version`), through from app-server
    - creates `created_at `timestamp in core
    - subagent analytics are behind `FeatureFlag::GeneralAnalytics`
    
    PR stack
    - [[telemetry] thread events
    #15690](https://github.com/openai/codex/pull/15690)
    - --> [[telemetry] subagent events
    #15915](https://github.com/openai/codex/pull/15915)
    - [[telemetry] turn events
    #15591](https://github.com/openai/codex/pull/15591)
    - [[telemetry] steer events
    #15697](https://github.com/openai/codex/pull/15697)
    - [[telemetry] queued prompt data
    #15804](https://github.com/openai/codex/pull/15804)
    
    Notes:
    - core does not spawn a subagent thread for compact, but represented in
    mapping for consistency
    
    `INFO | 2026-04-01 13:08:12 | codex_backend.routers.analytics_events |
    analytics_events.track_analytics_events:399 | Tracked
    codex_thread_initialized event params={'thread_id':
    '019d4aa9-233b-70f2-a958-c3dbae1e30fa', 'product_surface': 'codex',
    'app_server_client': {'product_client_id': 'CODEX_CLI', 'client_name':
    'codex-tui', 'client_version': '0.0.0', 'rpc_transport': 'in_process',
    'experimental_api_enabled': None}, 'runtime': {'codex_rs_version':
    '0.0.0', 'runtime_os': 'macos', 'runtime_os_version': '26.4.0',
    'runtime_arch': 'aarch64'}, 'model': 'gpt-5.3-codex', 'ephemeral':
    False, 'initialization_mode': 'new', 'created_at': 1775074091,
    'thread_source': 'subagent', 'subagent_source': 'thread_spawn',
    'parent_thread_id': '019d4aa8-51ec-77e3-bafb-2c1b8e29e385'} | `
    
    `INFO | 2026-04-01 13:08:41 | codex_backend.routers.analytics_events |
    analytics_events.track_analytics_events:399 | Tracked
    codex_thread_initialized event params={'thread_id':
    '019d4aa9-94e3-75f1-8864-ff8ad0e55e1e', 'product_surface': 'codex',
    'app_server_client': {'product_client_id': 'CODEX_CLI', 'client_name':
    'codex-tui', 'client_version': '0.0.0', 'rpc_transport': 'in_process',
    'experimental_api_enabled': None}, 'runtime': {'codex_rs_version':
    '0.0.0', 'runtime_os': 'macos', 'runtime_os_version': '26.4.0',
    'runtime_arch': 'aarch64'}, 'model': 'gpt-5.3-codex', 'ephemeral':
    False, 'initialization_mode': 'new', 'created_at': 1775074120,
    'thread_source': 'subagent', 'subagent_source': 'review',
    'parent_thread_id': None} | `
    
    ---------
    
    Co-authored-by: jif-oai <jif@openai.com>
    Co-authored-by: Michael Bolin <mbolin@openai.com>
  • remove temporary ownership re-exports (#16626)
    Stacked on #16508.
    
    This removes the temporary `codex-core` / `codex-login` re-export shims
    from the ownership split and rewrites callsites to import directly from
    `codex-model-provider-info`, `codex-models-manager`, `codex-api`,
    `codex-protocol`, `codex-feedback`, and `codex-response-debug-context`.
    
    No behavior change intended; this is the mechanical import cleanup layer
    split out from the ownership move.
    
    ---------
    
    Co-authored-by: Codex <noreply@openai.com>
  • core: remove cross-crate re-exports from lib.rs (#16512)
    ## Why
    
    `codex-core` was re-exporting APIs owned by sibling `codex-*` crates,
    which made downstream crates depend on `codex-core` as a proxy module
    instead of the actual owner crate.
    
    Removing those forwards makes crate boundaries explicit and lets leaf
    crates drop unnecessary `codex-core` dependencies. In this PR, this
    reduces the dependency on `codex-core` to `codex-login` in the following
    files:
    
    ```
    codex-rs/backend-client/Cargo.toml
    codex-rs/mcp-server/tests/common/Cargo.toml
    ```
    
    ## What
    
    - Remove `codex-rs/core/src/lib.rs` re-exports for symbols owned by
    `codex-login`, `codex-mcp`, `codex-rollout`, `codex-analytics`,
    `codex-protocol`, `codex-shell-command`, `codex-sandboxing`,
    `codex-tools`, and `codex-utils-path`.
    - Delete the `default_client` forwarding shim in `codex-rs/core`.
    - Update in-crate and downstream callsites to import directly from the
    owning `codex-*` crate.
    - Add direct Cargo dependencies where callsites now target the owner
    crate, and remove `codex-core` from `codex-rs/backend-client`.
  • Refactor external auth to use a single trait (#16356)
    ## Summary
    - Replace the separate external auth enum and refresher trait with a
    single `ExternalAuth` trait in login auth flow
    - Move bearer token auth behind `BearerTokenRefresher` and update
    `AuthManager` and app-server wiring to use the generic external auth API
  • [codex-analytics] thread events (#15690)
    - add event for thread initialization
    - thread/start, thread/fork, thread/resume
    - feature flagged behind `FeatureFlag::GeneralAnalytics`
    - does not yet support threads started by subagents
    
    PR stack:
    - --> [[telemetry] thread events
    #15690](https://github.com/openai/codex/pull/15690)
    - [[telemetry] subagent events
    #15915](https://github.com/openai/codex/pull/15915)
    - [[telemetry] turn events
    #15591](https://github.com/openai/codex/pull/15591)
    - [[telemetry] steer events
    #15697](https://github.com/openai/codex/pull/15697)
    - [[telemetry] queued prompt data
    #15804](https://github.com/openai/codex/pull/15804)
    
    
    Sample extracted logs in Codex-backend
    ```
    INFO     | 2026-03-29 16:39:37 | codex_backend.routers.analytics_events | analytics_events.track_analytics_events:398 | Tracked analytics event codex_thread_initialized thread_id=019d3bf7-9f5f-7f82-9877-6d48d1052531 product_surface=codex product_client_id=CODEX_CLI client_name=codex-tui client_version=0.0.0 rpc_transport=in_process experimental_api_enabled=True codex_rs_version=0.0.0 runtime_os=macos runtime_os_version=26.4.0 runtime_arch=aarch64 model=gpt-5.3-codex ephemeral=False thread_source=user initialization_mode=new subagent_source=None parent_thread_id=None created_at=1774827577 | 
    INFO     | 2026-03-29 16:45:46 | codex_backend.routers.analytics_events | analytics_events.track_analytics_events:398 | Tracked analytics event codex_thread_initialized thread_id=019d3b84-5731-79d0-9b3b-9c6efe5f5066 product_surface=codex product_client_id=CODEX_CLI client_name=codex-tui client_version=0.0.0 rpc_transport=in_process experimental_api_enabled=True codex_rs_version=0.0.0 runtime_os=macos runtime_os_version=26.4.0 runtime_arch=aarch64 model=gpt-5.3-codex ephemeral=False thread_source=user initialization_mode=resumed subagent_source=None parent_thread_id=None created_at=1774820022 | 
    INFO     | 2026-03-29 16:45:49 | codex_backend.routers.analytics_events | analytics_events.track_analytics_events:398 | Tracked analytics event codex_thread_initialized thread_id=019d3bfd-4cd6-7c12-a13e-48cef02e8c4d product_surface=codex product_client_id=CODEX_CLI client_name=codex-tui client_version=0.0.0 rpc_transport=in_process experimental_api_enabled=True codex_rs_version=0.0.0 runtime_os=macos runtime_os_version=26.4.0 runtime_arch=aarch64 model=gpt-5.3-codex ephemeral=False thread_source=user initialization_mode=forked subagent_source=None parent_thread_id=None created_at=1774827949 | 
    INFO     | 2026-03-29 17:20:29 | codex_backend.routers.analytics_events | analytics_events.track_analytics_events:398 | Tracked analytics event codex_thread_initialized thread_id=019d3c1d-0412-7ed2-ad24-c9c0881a36b0 product_surface=codex product_client_id=CODEX_SERVICE_EXEC client_name=codex_exec client_version=0.0.0 rpc_transport=in_process experimental_api_enabled=True codex_rs_version=0.0.0 runtime_os=macos runtime_os_version=26.4.0 runtime_arch=aarch64 model=gpt-5.3-codex ephemeral=False thread_source=user initialization_mode=new subagent_source=None parent_thread_id=None created_at=1774830027 | 
    ```
    
    Notes
    - `product_client_id` gets canonicalized in codex-backend
    - subagent threads are addressed in a following pr
  • auth: let AuthManager own external bearer auth (#16287)
    ## Summary
    
    `AuthManager` and `UnauthorizedRecovery` already own token resolution
    and staged `401` recovery. The missing piece for provider auth was a
    bearer-only mode that still fit that design, instead of pushing a second
    auth abstraction into `codex-core`.
    
    This PR keeps the design centered on `AuthManager`: it teaches
    `codex-login` how to own external bearer auth directly so later provider
    work can keep calling `AuthManager.auth()` and `UnauthorizedRecovery`.
    
    ## Motivation
    
    This is the middle layer for #15189.
    
    The intended design is still:
    
    - `AuthManager` encapsulates token storage and refresh
    - `UnauthorizedRecovery` powers staged `401` recovery
    - all request tokens go through `AuthManager.auth()`
    
    This PR makes that possible for provider-backed bearer tokens by adding
    a bearer-only auth mode inside `AuthManager` instead of building
    parallel request-auth plumbing in `core`.
    
    ## What Changed
    
    - move `ModelProviderAuthInfo` into `codex-protocol` so `core` and
    `login` share one config shape
    - add `login/src/auth/external_bearer.rs`, which runs the configured
    command, caches the bearer token in memory, and refreshes it after `401`
    - add `AuthManager::external_bearer_only(...)` for provider-scoped
    request paths that should use command-backed bearer auth without
    mutating the shared OpenAI auth manager
    - add `AuthManager::shared_with_external_chatgpt_auth_refresher(...)`
    and rename the other `AuthManager` helpers that only apply to external
    ChatGPT auth so the ChatGPT-only path is explicit at the call site
    - keep external ChatGPT refresh behavior unchanged while ensuring
    bearer-only external auth never persists to `auth.json`
    
    ## Testing
    
    - `cargo test -p codex-login`
    - `cargo test -p codex-protocol`
    
    
    
    
    
    ---
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/16287).
    * #16288
    * __->__ #16287
  • auth: generalize external auth tokens for bearer-only sources (#16286)
    ## Summary
    
    `ExternalAuthRefresher` was still shaped around external ChatGPT auth:
    `ExternalAuthTokens` always implied ChatGPT account metadata even when a
    caller only needed a bearer token.
    
    This PR generalizes that contract so bearer-only sources are
    first-class, while keeping the existing ChatGPT paths strict anywhere we
    persist or rebuild ChatGPT auth state.
    
    ## Motivation
    
    This is the first step toward #15189.
    
    The follow-on provider-auth work needs one shared external-auth contract
    that can do both of these things:
    
    - resolve the current bearer token before a request is sent
    - return a refreshed bearer token after a `401`
    
    That should not require a second token result type just because there is
    no ChatGPT account metadata attached.
    
    ## What Changed
    
    - change `ExternalAuthTokens` to carry `access_token` plus optional
    `ExternalAuthChatgptMetadata`
    - add helper constructors for bearer-only tokens and ChatGPT-backed
    tokens
    - add `ExternalAuthRefresher::resolve()` with a default no-op
    implementation so refreshers can optionally provide the current token
    before a request is sent
    - keep ChatGPT-only persistence strict by continuing to require ChatGPT
    metadata anywhere the login layer seeds or reloads ChatGPT auth state
    - update the app-server bridge to construct the new token shape for
    external ChatGPT auth refreshes
    
    ## Testing
    
    - `cargo test -p codex-login`
    
    
    ---
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/16286).
    * #16288
    * #16287
    * __->__ #16286
  • Remove the codex-tui app-server originator workaround (#16116)
    ## Summary
    - remove the temporary `codex-tui` special-case when setting the default
    originator during app-server initialization
  • Remove the legacy TUI split (#15922)
    This is the part 1 of 2 PRs that will delete the `tui` /
    `tui_app_server` split. This part simply deletes the existing `tui`
    directory and marks the `tui_app_server` feature flag as removed. I left
    the `tui_app_server` feature flag in place for now so its presence
    doesn't result in an error. It is simply ignored.
    
    Part 2 will rename the `tui_app_server` directory `tui`. I did this as
    two parts to reduce visible code churn.
  • [mcp] Improve custom MCP elicitation (#15800)
    - [x] Support don't ask again for custom MCP tool calls.
    - [x] Don't run arc in yolo mode.
    - [x] Run arc for custom MCP tools in always allow mode.
  • Add cached environment manager for exec server URL (#15785)
    Add environment manager that is a singleton and is created early in
    app-server (before skill manager, before config loading).
    
    Use an environment variable to point to a running exec server.
  • Extract codex-analytics crate (#15748)
    ## Summary
    - move the analytics events client into codex-analytics
    - update codex-core and app-server callsites to use the new crate
    
    ## Testing
    - CI
    
    ---------
    
    Co-authored-by: Codex <noreply@openai.com>
  • [app-server] Add a method to override feature flags. (#15601)
    - [x] Add a method to override feature flags globally and not just
    thread level.