Commit Graph

22 Commits

  • core: add wait_for_environment for starting environments (#29745)
    ## Why
    
    With `DeferredExecutor`, a sampling request can begin while an
    environment is still starting. The model can see that pending state, but
    needs a way to wait for the environment within the same turn before
    continuing.
    
    Environment startup is owned by Core, so the wait tool should use the
    same request-frozen `StepContext` that advertised the starting
    environment. This keeps tool registration and execution tied to the
    exact startup operation the model saw, even if live thread state later
    changes.
    
    Supersedes #29735.
    
    ## What
    
    - register `wait_for_environment` when the current `StepContext`
    contains starting environments
    - wait on the selected `StartingTurnEnvironment` shared resolution and
    return a bounded ready or failed result
    - rebuild the next request normally, removing the wait tool and exposing
    ready environment tools, or reporting the environment as unavailable
    after failure
    
    ## Testing
    
    - `just test -p codex-core deferred_executor_`
    - verifies the wait tool is replaced by environment-backed tools after
    startup
    - verifies startup failure removes both the wait tool and unavailable
    environment tools while notifying the model
  • [3/3] app-server: configure environment connection timeout (#29025)
    ## Why
    
    Remote environments registered through `environment/add` currently use
    the fixed 10-second WebSocket connection timeout. Slow-starting
    executors need a caller-selected connection window, but this should not
    add retry policy or couple exec-server behavior to Core’s
    `deferred_executor` feature.
    
    Make the timeout an optional part of the existing experimental request.
    Existing clients continue using the current default, while callers that
    know an executor may take longer can request a larger window explicitly.
    
    Depends on #28683.
    
    ## What changed
    
    - Add optional `connectTimeoutMs` to `EnvironmentAddParams` and document
    it in the app-server README.
    - Pass the optional timeout through `EnvironmentRequestProcessor` into
    one `EnvironmentManager::upsert_environment()` path; the manager applies
    the existing default when it is omitted.
    - Preserve the existing single-attempt lifecycle. The configured value
    controls WebSocket connection and handshake time for both initial
    connection and later reconnects; initialization retains its separate
    timeout.
    - Add an app-server integration test that sends the real JSON-RPC
    request and verifies a stalled handshake observes the requested timeout.
    
    ## Test plan
    
    - `just test -p codex-app-server-protocol`
    - `just test -p codex-exec-server`
    - `just test -p codex-app-server
    environment_add_applies_connect_timeout`
    
    ## Rollout
    
    This is additive and does not enable `deferred_executor`. Callers should
    send a non-default timeout only after a compatible app-server is
    deployed; omitted or `null` values retain the existing 10-second
    default.
  • [2/3] core: track starting environments in snapshots (#28683)
    ## Why
    
    Remote environments may still be resolving when Codex creates a session
    or turn. Waiting for the existing all-or-nothing environment snapshot
    can hold startup until the selected environment is usable.
    
    Behind the default-off `deferred_executor` feature, let callers take a
    useful snapshot immediately: completed environments remain available
    normally, while unfinished environments are reported without blocking
    startup. With the feature disabled, snapshots preserve the existing
    blocking behavior.
    
    Depends on #28674.
    
    ## What changed
    
    - Store one ordered list of selected environments in
    `ThreadEnvironments`. Each selection owns one shared resolution that
    produces its complete `TurnEnvironment`.
    - Start new resolutions in the background with `remote_handle()`,
    allowing snapshots and the future wait tool to share the same result
    while cancellation follows the retained handles.
    - Make `snapshot()` a read-only operation: nonblocking snapshots collect
    completed resolutions and retain handles for unfinished ones, while
    blocking snapshots await every resolution.
    - Replace completed failed resolutions from the current manager entry
    and log when failed environments are omitted.
    - Return attached and starting environments as a point-in-time view, and
    count starting environments when deciding whether a snapshot is
    local-only.
    - Keep existing consumers attached-only. `to_selections()` derives from
    attached environments, so child threads do not inherit an environment
    that is still starting.
    
    ## Test plan
    
    - `just test -p codex-core environment_selection`
    - `just test -p codex-core
    deferred_executor_reaches_model_before_remote_environment_is_ready`
    
    ## Landing note
    
    Keep `deferred_executor` disabled for slow-starting executors until
    configurable `environment/add` connection timeouts and caller support
    land. When enabled, an environment that attaches after session startup
    may remain absent from environment-derived model context, tools,
    instructions, skills, and related state until follow-up refresh work
    lands.
  • app-server: preserve target-native environment cwd (#28146)
    ## Why
    
    app-server may run on a different OS from the selected exec-server
    environment. Parsing that environment’s cwd with the Codex host’s path
    rules prevents thread startup.
    
    ## What
    
    Carry environment cwd values as `LegacyAppPathString` at the app-server
    boundary and `PathUri` internally. Existing tool-call schemas and
    relative-path behavior stay host-native; remaining local-only consumers
    convert explicitly and leave follow-up TODOs.
    
    The Wine integration test verifies app-server can start a thread and
    complete an ordinary turn with a Windows environment cwd from Linux.
    
    ## Validation
    
    - `bazel test //codex-rs/core/tests/remote_env_windows:smoke-test
    --test_output=errors`
    - focused app-server environment-selection and protocol schema tests
    - scoped Clippy for `codex-core` and `codex-app-server-protocol`
  • [codex] Use local environment for user shell commands (#28163)
    ## Why
    
    User shell commands still read the legacy turn cwd and session shell
    even though execution context is now owned by selected turn
    environments. App-server also defines `thread/shellCommand` as a
    local-host escape hatch, so it must use an available local environment
    even when a remote environment is primary.
    
    ## What changed
    
    - Add `ResolvedTurnEnvironments::local()` to find the selected local
    environment.
    - Resolve the user shell command cwd and shell from that local
    `TurnEnvironment`.
    - Emit the standard `shell is unavailable in this session` error when no
    selected local environment or resolved local shell is available.
    - Add an integration test covering `/shell` without a local environment.
    
    ## Test plan
    
    - `just test -p codex-core
    user_shell_command_without_local_environment_emits_error`
  • [codex] Bind shell snapshots to retained thread environments (#28421)
    ## Why
    
    Shell snapshots are currently session-scoped even though shell and cwd
    are properties of a selected turn environment. That makes snapshot
    refresh depend on separate session-cwd plumbing, prevents retained
    environments from retaining their snapshot work, and can make snapshot
    construction use a different shell than command execution.
    
    This follows #27955 by making the retained thread-environment service
    own environment snapshot lifecycles. Session configuration remains the
    requested selection state, while `ThreadEnvironments` remains the source
    of successfully resolved environments.
    
    ## What changed
    
    - Configure the shell-snapshot builder before initial environment
    resolution.
    - Start each local environment snapshot task when its `TurnEnvironment`
    is built and retain that shared task while environment ID and cwd still
    match.
    - Inherit retained environment snapshots into spawned child threads.
    - Carry the selected `TurnEnvironment` through shell runtimes so
    snapshot construction and command execution use the same
    environment-specific shell and cwd.
    - Load project instructions and warm plugins/skills after initial
    environment resolution.
    - Continue decoding invalid UTF-8 instruction files lossily without
    emitting a startup warning.
    - Keep requested selections in `SessionConfiguration`; failed or
    duplicate resolutions only affect the resolved environment snapshot.
    
    ## Validation
    
    - `cargo check -p codex-core --tests`
    - `just test -p codex-home instructions` (6 passed)
    - Focused environment, instruction, shell-snapshot, and user-shell tests
    (84 passed)
    - Focused shell-snapshot, user-shell, and unified-exec tests (126
    passed; two event-timing tests passed on retry)
  • [codex] retain resolved environments across turns (#27955)
    ## Why
    
    Selected execution environments are thread-scoped resources, but startup
    and turn construction repeatedly resolved their IDs and working
    directories. That discarded existing environment handles and shell
    metadata even when a selection had not changed.
    
    Session configuration updates also need to affect future turns without
    changing the resolved environment set already captured by a running
    turn.
    
    ## What changed
    
    - Create a `ThreadEnvironments` service inside `Codex` from the spawned
    `EnvironmentManager` and raw environment selections, then store it on
    `SessionServices`.
    - Split service construction from `update_selections`, allowing session
    configuration updates to mutate the resolved set in place.
    - Retain an existing `TurnEnvironment` when its environment ID and
    working directory match; resolve only added or changed selections and
    remove selections that are no longer present.
    - Normalize duplicate IDs by keeping the first selection and skip
    individual selections that fail to resolve instead of rejecting the
    entire update.
    - Give each `TurnContext` a cloned `TurnEnvironmentSnapshot`, so later
    session configuration updates affect future turns without rewriting an
    active turn.
    - Reuse the service-owned environment manager and resolved snapshot for
    startup work, MCP initialization, and child-thread spawning instead of
    flowing resolved environments through spawn arguments.
    
    ## Test plan
    
    - `cargo check -p codex-core --tests`
    - `just test -p codex-core environment_selection`
    - `just test -p codex-core turn_environments`
    - `just test -p codex-core
    session_update_settings_does_not_rewrite_sticky_environment_cwds`
    - `just test -p codex-core
    default_turn_does_not_overlay_legacy_fallback_cwd_onto_stored_thread_environments`
  • [codex] exec-server honors remote environment cwd and shell (#28122)
    ## Why
    
    Next slice needed to make progress on the `remote_env_windows` test is
    to support passing a Windows cwd for the remote environment and using
    that environment's native shell. This lets the test run a real Windows
    process instead of only recording an early path or shell mismatch.
    
    ## What
    
    - change `TurnEnvironmentSelection.cwd` from `AbsolutePathBuf` to
    `PathUri`
    - convert local cwd values to URIs when constructing selections
    - preserve a remote primary cwd instead of replacing it with the local
    legacy fallback
    - prefer the selected environment's discovered shell for unified exec,
    falling back to the session shell when unavailable
    - convert back to a host-native absolute path at current native-only
    consumer boundaries
    - reject or deny unsupported foreign cwd values at the existing
    request-permissions boundary, with TODOs for its future migration
    - extend the hermetic Wine test to execute Windows PowerShell in
    `C:\windows` and verify successful process completion
    - record the current app-server rejection against the same Wine-backed
    remote Windows fixture when its cwd is supplied as a native Windows path
  • [codex] make PathUri::from_abs_path infallible (#27976)
    ## Why
    
    `PathUri::from_abs_path` can fail for absolute paths that do not have a
    normal `file:` URI representation, forcing filesystem call sites to
    handle a conversion error even though the original path can be preserved
    losslessly.
    
    ## What
    
    Make `from_abs_path` infallible and migrate its callers. Unrepresentable
    paths use `file:///%00/bad/path/<base64>`, encoding Unix bytes or
    Windows UTF-16LE; `to_abs_path` validates and decodes that fallback. The
    leading encoded null reserves a namespace that cannot collide with a
    real Unix or Windows path, and fallback URIs remain opaque to lexical
    path operations.
    
    ## Validation
    
    Added path-URI coverage for Unix null and non-UTF-8 paths, Windows
    device/verbatim and non-Unicode paths, serialization, malformed
    fallbacks, opaque lexical operations, invalid native payloads, and
    literal `/bad/path` collision resistance.
  • sandboxing: migrate cwd inputs to PathUri (#27816)
    ## Why
    
    Sandbox cwd values can cross app-server and exec-server host boundaries.
    They should retain URI semantics until the receiving host validates them
    instead of being interpreted early as native paths.
    
    ## What
    
    - Carry `PathUri` through filesystem sandbox contexts, sandbox commands,
    and transform inputs.
    - Convert command and policy cwd once in `SandboxManager::transform`,
    then keep launch requests native.
    - Preserve sandbox cwd over remote filesystem transport and reject
    non-native URIs without fallback.
    - Cache paired native/URI turn-environment cwd values during migration,
    with immutable access to keep them synchronized.
    - Extend existing protocol, forwarding, transform, and core runtime
    tests.
  • [codex] Load AGENTS.md from all bound environments (#27696)
    ## Why
    
    We already have the machinery to support multiple environments on a
    single thread, but we only show the model the contents of `AGENTS.md`
    files in the primary environment.
    
    We should show the model all of the relevant project instructions when
    we know there's more than one environment.
    
    ## Known Gaps
    
    As discussed in the RFC, this implementation:
    
    1. doesn't handle environments being added/removed to/from the thread
    after its creation
    2. it doesn't enforce an aggregate context budget across environments,
    and instead applies the configured project maximum independently to each
    environment
    
    ## Implementation
    
    - Discover project instructions in environment order with an independent
    byte budget per environment and preserve source provenance/order.
    - Keep the legacy fragment byte-for-byte when exactly one environment
    contributes project instructions; use environment-labeled sections when
    two or more environments contribute.
    - Freeze the complete rendered fragment in `LoadedAgentsMd`, insert it
    directly into requests, and recognize both layouts in contextual and
    memory filtering.
    - Add exact rendering, independent-budget, source-order,
    creation-snapshot, and consumer coverage without changing app-server
    schemas.
  • [codex] resolve environment shell metadata eagerly (#27709)
    ## Why
    
    Turn construction passed resolved environments through several layers
    while leaving the environment shell unresolved. As a result,
    model-visible environment context could fall back to the session shell
    instead of reporting the selected remote environment's shell.
    
    Resolve environment metadata at the turn-context boundary so each turn
    carries the shell that belongs to its selected environment. Keep request
    validation in app-server, where invalid selections can be returned as
    straightforward JSON-RPC errors without coupling core turn construction
    to that policy.
    
    ## What changed
    
    - resolve environment selections eagerly in
    `new_turn_context_from_configuration`
    - store the full resolved `Shell` on each `TurnEnvironment`
    - simplify the now-redundant resolved-environment constructor plumbing
    - keep duplicate and unknown-environment validation as a small
    app-server preflight
    - add a remote-environment integration test that runs a full
    `test_codex` turn and verifies the model-visible environment message
    reports `bash`
    
    ## Testing
    
    - `cargo check -p codex-core --test all -p codex-app-server`
    - `remote_test_env_exposes_bash_shell_to_model` on the Linux
    remote-executor harness
  • feat: guard git enrichment (#26175)
    Skip turn git metadata enrichment when a turn has remote or multiple
    executors, so we do not report the orchestrator checkout as executor
    workspace metadata.
    
    Test: `just test -p codex-core` (blocked by existing
    `Session::conversation_id` compile error in `close_agent.rs`).
  • Make local environment optional in EnvironmentManager (#23369)
    ## Summary
    - make `EnvironmentManager` local environment/runtime paths optional
    - simplify constructor surface around snapshot materialization
    - rename local env accessors to `require_local_environment` /
    `try_local_environment`
    
    ## Validation
    - devbox Bazel build for touched crate surfaces
    - `//codex-rs/exec-server:exec-server-unit-tests`
    - `//codex-rs/app-server-client:app-server-client-unit-tests`
    - filtered touched `//codex-rs/core:core-unit-tests` cases
  • Make environment provider snapshots path-free (#21794)
    ## Summary
    - make EnvironmentProvider::snapshot path-free and keep providers
    focused on provider-owned remote environments
    - let provider snapshots request local inclusion via include_local, with
    environments.toml including local and CODEX_EXEC_SERVER_URL excluding
    local
    - move reserved local environment construction into EnvironmentManager
    using ExecServerRuntimePaths
    
    Follow-up to https://github.com/openai/codex/pull/20667
    
    ## Testing
    - just fmt
    - git diff --check
    - devbox: bazel build --bes_backend= --bes_results_url=
    //codex-rs/exec-server:exec-server
    - devbox: bazel test --bes_backend= --bes_results_url=
    //codex-rs/exec-server:exec-server-unit-tests
    
    Co-authored-by: Codex <noreply@openai.com>
  • Load configured environments from CODEX_HOME (#20667)
    ## Why
    
    The earlier PRs add stdio transport support and the config-backed
    environment provider, but the feature remains inert until normal Codex
    entrypoints construct `EnvironmentManager` with enough context to
    discover `CODEX_HOME/environments.toml`. This final stack PR activates
    the provider while preserving the legacy `CODEX_EXEC_SERVER_URL`
    fallback when no environments file exists.
    
    **Stack position:** this is PR 5 of 5. It is the product wiring PR that
    activates the configured environment provider added in PR 4.
    
    ## What Changed
    
    - Thread `codex_home` into `EnvironmentManagerArgs`.
    - Change `EnvironmentManager::new(...)` to load the provider from
    `CODEX_HOME`.
    - Preserve legacy behavior by falling back to
    `DefaultEnvironmentProvider::from_env()` when `environments.toml` is
    absent.
    - Make `environments.toml`-backed managers start new threads with all
    configured environments, default first, while keeping the legacy env-var
    path single-default.
    - Update the app-server, TUI, exec, MCP server, connector, prompt-debug,
    and thread-manager-sample callsites to pass `codex_home` and handle
    provider-loading errors.
    
    ## Self-Review Notes
    
    - The multi-environment startup path is intentionally tied to the
    `environments.toml` provider. Using `>1` configured environment as the
    only signal would also expand the legacy `CODEX_EXEC_SERVER_URL`
    provider because it keeps `local` addressable alongside `remote`.
    - The startup environment list is still derived inside
    `EnvironmentManager`; the provider only says whether its snapshot should
    start new threads with all configured environments.
    - The thread-manager sample was updated to pass the current
    `ThreadManager::new(...)` installation id argument so the stack compiles
    under Bazel.
    
    ## Stack
    
    - 1. https://github.com/openai/codex/pull/20663 - Add stdio exec-server
    listener
    - 2. https://github.com/openai/codex/pull/20664 - Add stdio exec-server
    client transport
    - 3. https://github.com/openai/codex/pull/20665 - Make environment
    providers own default selection
    - 4. https://github.com/openai/codex/pull/20666 - Add CODEX_HOME
    environments TOML provider
    - **5. This PR:** https://github.com/openai/codex/pull/20667 - Load
    configured environments from CODEX_HOME
    
    Split from original draft: https://github.com/openai/codex/pull/20508
    
    ## Validation
    
    - `just fmt`
    - `git diff --check`
    - `bazel build --config=remote --strategy=remote
    --remote_download_toplevel
    //codex-rs/thread-manager-sample:codex-thread-manager-sample`
    - `bazel test --config=remote --strategy=remote
    --remote_download_toplevel
    //codex-rs/exec-server:exec-server-unit-tests`
    - `bazel test --config=remote --strategy=remote
    --remote_download_toplevel --test_sharding_strategy=disabled
    --test_arg=default_thread_environment_selections_use_manager_default_id
    //codex-rs/core:core-unit-tests`
    - `bazel test --config=remote --strategy=remote
    --remote_download_toplevel --test_sharding_strategy=disabled
    --test_arg=start_thread_uses_all_default_environments_from_codex_home
    //codex-rs/core:core-unit-tests`
    
    ## Documentation
    
    This activates `CODEX_HOME/environments.toml`; user-facing documentation
    should be added before this stack is treated as a documented public
    workflow.
    
    ---------
    
    Co-authored-by: Codex <noreply@openai.com>
  • Avoid hard-coded environment context shell (#21390)
    ## Summary
    - make resolved turn environment shell metadata optional instead of
    hard-coding bash
    - render environment context shells from explicit environment metadata
    when present, falling back to the existing session shell
    - update environment context tests for inherited PowerShell-style
    fallback and explicit per-environment shell override
    
    ## Testing
    - Not run (not requested; formatted with `just fmt`).
    
    Co-authored-by: Codex <noreply@openai.com>
  • Prepare selected environment plumbing (#20669)
    ## Why
    This is a prep PR in the multi-environment process-tool stack. It
    separates ownership/config cleanup from the behavior change that teaches
    process tools to route by selected environment, so the follow-up PR can
    focus on model-facing `environment_id` behavior.
    
    ## Stack
    1. https://github.com/openai/codex/pull/20646 - `EnvironmentContext`
    rendering for selected environments
    2. https://github.com/openai/codex/pull/20669 - selected-environment
    ownership and tool config prep (this PR)
    3. https://github.com/openai/codex/pull/20647 - process-tool
    `environment_id` routing
    
    ## What Changed
    - keep the resolved turn environment list wrapped in
    `ResolvedTurnEnvironments` through `TurnContext` instead of unwrapping
    it back to a raw `Vec`
    - add `TurnContext::resolve_path_against` so cwd-relative path
    resolution has one shared helper
    - replace the old tool config boolean with `ToolEnvironmentMode::{None,
    Single, Multiple}`
    
    ## Testing
    - Tests not run locally; this prep refactor is covered by GitHub CI for
    the stack.
    
    Co-authored-by: Codex <noreply@openai.com>
  • Surface multi-environment choices in environment context (#20646)
    ## Why
    The model needs a way to see which environments are available during a
    multi-environment turn without changing the legacy single-environment
    prompt surface or pulling replay/persistence changes into the same
    review.
    
    ## Stack
    1. https://github.com/openai/codex/pull/20646 - `EnvironmentContext`
    rendering for selected environments (this PR)
    2. https://github.com/openai/codex/pull/20669 - selected-environment
    ownership and tool config prep
    3. https://github.com/openai/codex/pull/20647 - process-tool
    `environment_id` routing
    
    ## What Changed
    - extend `environment_context` so multi-environment turns render an
    `<environments>` block with the selected environment ids and cwd values
    - keep zero- and single-environment turns on the existing cwd-only
    render path
    - keep replay and persistence paths on the legacy surface for now so
    this PR stays scoped to live prompt rendering
    - add focused coverage in
    `codex-rs/core/src/context/environment_context_tests.rs`
    
    ## Testing
    - CI
    
    ---------
    
    Co-authored-by: Codex <noreply@openai.com>
  • Use selected turn environments for runtime context (#20281)
    ## Summary
    - make selected turn environments the source of truth for session
    runtime cwd and MCP runtime environment selection
    - keep local/no-selection fallback behavior intact
    - add coverage for duplicate selected environments, cwd resolution, and
    MCP runtime environment selection
    
    ## Validation
    - git diff --check
    - rustfmt was run on touched Rust files during the implementation
    workflow
    
    CI should provide the full Bazel/test signal.
    
    ---------
    
    Co-authored-by: Codex <noreply@openai.com>
  • Add environment provider snapshot (#20058)
    ## Summary
    - Change `EnvironmentProvider` to return concrete `Environment`
    instances instead of `EnvironmentConfigurations`.
    - Make `DefaultEnvironmentProvider` provide the provider-visible `local`
    environment plus optional `remote` environment from
    `CODEX_EXEC_SERVER_URL`.
    - Keep `EnvironmentManager` as the concrete cache while exposing its own
    explicit local environment for `local_environment()` fallback paths.
    
    ## Validation
    - `just fmt`
    - `git diff --check`
    
    ---------
    
    Co-authored-by: Codex <noreply@openai.com>
  • Add sticky environment API and thread state (#18897)
    ## Summary
    - add sticky environment selections to app-server v2 thread/start and
    turn/start request flow
    - carry thread-level selections through core session/thread state
    - add app-server coverage for sticky selections and turn overrides
    
    ## Stack
    1. This PR: API and thread persistence
    2. #18898: config.toml named environment loading
    3. #18899: downstream tool/runtime consumers
    
    ## Validation
    - Not run locally; split only.
    
    ---------
    
    Co-authored-by: Codex <noreply@openai.com>