33 Commits

  • [codex] Load user instructions through an injected provider (#27101)
    ## Why
    
    We want to remove implicit use of `$CODEX_HOME` from `codex-core` and
    make embedders responsible for supplying user-level instructions. This
    also ensures user instructions load when no primary environment is
    selected.
    
    ## What changed
    
    Stacked on #27415, which makes `codex exec` surface thread-scoped
    runtime warnings.
    
    - Added `UserInstructionsProvider` to `codex-extension-api`, with
    absolute source attribution and recoverable loading warnings.
    - Added `codex-home` with the filesystem-backed provider for
    `AGENTS.override.md` and `AGENTS.md`, preserving precedence, fallback,
    trimming, lossy UTF-8 handling, and the existing uncapped global
    instruction size.
    - Removed global instruction loading from `Config` and require
    `ThreadManager` callers to inject a provider.
    - Load provider instructions once for each fresh root runtime, including
    runtimes without a primary environment. Running sessions retain their
    snapshot, while child agents inherit the parent snapshot without
    invoking the provider.
    - Keep provider instructions separate while loading project `AGENTS.md`,
    then assemble the model-visible instructions with the existing ordering,
    source attribution, warning, and turn-context behavior.
    - Wired the Codex home provider through the CLI, app server, MCP server,
    core facade, and thread-manager sample.
    
    ## Validation
    
    - `just test -p codex-home -p codex-extension-api`
    - `just test -p codex-core agents_md`
    - `just test -p codex-core guardian`
    - `just test -p codex-app-server
    thread_start_without_selected_environment_includes_only_global_instruction_source`
    - `just test -p codex-exec warning`
    - `just bazel-lock-check`
  • [codex] migrate ExecutorFileSystem paths to PathUri (#27424)
    ## Why
    
    We're moving exec-server to use PathUri for its internal path
    representations.
    
    ## What
    
    Move `ExecutorFileSystem` APIs to use `PathUri` instead of
    `AbsolutePathBuf`. Future changes will convert higher-level parts of
    exec-server.
  • extension: wire extension registries into sessions (#21737)
    ## Why
    
    [#21736](https://github.com/openai/codex/pull/21736) introduces the
    typed extension API, but the runtime does not yet carry a registry
    through thread/session startup or give contributors host-owned stores to
    read from. This PR wires that host-side path so later feature migrations
    can move product-specific behavior behind typed contributions without
    adding another bespoke seam directly to `codex-core`.
    
    ## What changed
    
    - Thread `ExtensionRegistry<Config>` through `ThreadManager`,
    `CodexSpawnArgs`, `Session`, and sub-agent spawn paths.
    - Wire `ThreadStartContributor` and `ContextContributor`
    - Expose the small supporting surface needed by non-core callers that
    construct threads directly, including `empty_extension_registry()`
    through `codex-core-api`.
    
    This PR lands the host plumbing only: the app-server registry is still
    empty, and concrete feature migrations are intended to follow
    separately.
  • Disable empty Cargo test targets (#21584)
    ## Summary
    
    `cargo test` has entails both running standard Rust tests and doctests.
    It turns out that the doctest discovery is fairly slow, and it's a cost
    you pay even for crates that don't include any doctests.
    
    This PR disables doctests with `doctest = false` for crates that lack
    any doctests.
    
    For the collection of crates below, this speeds up test execution by
    >4x.
    
    E.g., before this PR:
    
    ```
    Benchmark 1: cargo test     -p codex-utils-absolute-path     -p codex-utils-cache     -p codex-utils-cli     -p codex-utils-home-dir     -p codex-utils-output-truncation     -p codex-utils-path     -p codex-utils-string     -p codex-utils-template     -p codex-utils-elapsed     -p codex-utils-json-to-toml
      Time (mean ± σ):      1.849 s ±  4.455 s    [User: 0.752 s, System: 1.367 s]
      Range (min … max):    0.418 s … 14.529 s    10 runs
    ```
    
    And after:
    
    ```
    Benchmark 1: cargo test     -p codex-utils-absolute-path     -p codex-utils-cache     -p codex-utils-cli     -p codex-utils-home-dir     -p codex-utils-output-truncation     -p codex-utils-path     -p codex-utils-string     -p codex-utils-template     -p codex-utils-elapsed     -p codex-utils-json-to-toml
      Time (mean ± σ):     428.6 ms ±   6.9 ms    [User: 187.7 ms, System: 219.7 ms]
      Range (min … max):   418.0 ms … 436.8 ms    10 runs
    ```
    
    For a single crate, with >2x speedup, before:
    
    ```
    Benchmark 1: cargo test -p codex-utils-string
      Time (mean ± σ):     491.1 ms ±   9.0 ms    [User: 229.8 ms, System: 234.9 ms]
      Range (min … max):   480.9 ms … 512.0 ms    10 runs
    ```
    
    And after:
    
    ```
    Benchmark 1: cargo test -p codex-utils-string
      Time (mean ± σ):     213.9 ms ±   4.3 ms    [User: 112.8 ms, System: 84.0 ms]
      Range (min … max):   206.8 ms … 221.0 ms    13 runs
    ```
    
    Co-authored-by: Codex <noreply@openai.com>
  • Remove core MCP list tools op (#21281)
    ## Why
    
    The core `Op::ListMcpTools` request path is no longer needed. Keeping it
    around left a dead request/response surface alongside the app-server MCP
    inventory APIs that own current server status listing.
    
    ## What Changed
    
    - Removed `Op::ListMcpTools`, `EventMsg::McpListToolsResponse`, and the
    core handler that built the MCP snapshot response.
    - Removed the now-unused `codex-mcp` snapshot wrapper/export and passive
    event handling arms in rollout and MCP-server consumers.
    - Updated tests that used the old op as a synchronization hook to wait
    on existing startup/skills events, and deleted the plugin test that only
    exercised the removed listing op.
    
    ## Validation
    
    - `cargo test -p codex-protocol`
    - `cargo test -p codex-mcp`
    - `cargo test -p codex-rollout -p codex-rollout-trace -p
    codex-mcp-server`
    - `cargo test -p codex-core --test all
    pending_input::queued_inter_agent_mail`
    - `cargo test -p codex-core --test all
    rmcp_client::stdio_mcp_tool_call_includes_sandbox_state_meta`
    - `cargo test -p codex-core --test all
    rmcp_client::stdio_image_responses`
    - `just fix -p codex-core -p codex-protocol -p codex-mcp -p
    codex-rollout -p codex-rollout-trace -p codex-mcp-server`
  • hook trust metadata and enforcement (#20321)
    # Why
    
    We want shared hook trust that both the app and the TUI can build on,
    but the metadata is only useful if runtime behavior agrees with it. This
    PR adds a single backend trust model for hooks so unmanaged hooks cannot
    run until the current definition has been reviewed, while managed hooks
    remain runnable and non-configurable.
    
    # What
    
    - persist `trusted_hash` alongside hook state in `config.toml`
    - expose `currentHash` and derived `trustStatus` through `hooks/list`
    - derive trust from normalized hook definitions so equivalent hooks from
    `config.toml` and `hooks.json` share the same trust identity
    - gate unmanaged hooks on trust before they enter the runnable handler
    set
    
    # Reviewer Notes
    
    - key file to review is `codex-rs/hooks/src/engine/discovery.rs`
    - the only **core** change is schema related
  • permissions: add built-in default profiles (#19900)
    ## Why
    
    The migration away from `SandboxPolicy` needs new configs to start from
    permissions profiles instead of deriving profiles from legacy sandbox
    modes. Existing users can have empty `config.toml` files, and we should
    not rewrite user-owned config files that may live in shared
    repositories.
    
    This PR introduces built-in profile names so an empty config can resolve
    to a canonical `PermissionProfile`, while explicit named `[permissions]`
    profiles still behave predictably.
    
    ## What changed
    
    - Adds built-in `default_permissions` profile names:
      - `:read-only` maps to `PermissionProfile::read_only()`.
    - `:workspace` maps to the workspace-write profile, including
    project-root metadata carveouts.
    - `:danger-no-sandbox` maps to `PermissionProfile::Disabled`, preserving
    the distinction between no sandbox and a broad managed sandbox.
    - Reserves the `:` prefix for built-in profiles so user-defined
    `[permissions]` profiles cannot collide with future built-ins.
    - Allows `default_permissions` to reference a built-in profile without
    requiring a `[permissions]` table.
    - Makes an otherwise empty config choose a built-in profile by
    trust/platform context: trusted or untrusted project roots use
    `:workspace` when the platform supports that sandbox, while roots
    without a trust decision use `:read-only`.
    - Keeps legacy `sandbox_mode` configs on the legacy path, and still
    rejects user-defined `[permissions]` profiles that omit
    `default_permissions` so we do not silently guess among custom profiles.
    - Preserves compatibility behavior for implicit defaults: bare
    `network.enabled = true` allows runtime network without starting the
    managed proxy, explicit profile proxy policy still starts the proxy, and
    implicit workspace/add-dir roots keep legacy metadata carveouts.
    
    ## Verification
    
    - `cargo test -p codex-core builtin --lib`
    - `cargo test -p codex-core profile_network_proxy_config`
    - `cargo test -p codex-core
    implicit_builtin_workspace_profile_preserves_add_dir_metadata_carveouts`
    - `cargo test -p codex-core
    permissions_profiles_network_enabled_allows_runtime_network_without_proxy`
    - `cargo test -p codex-core
    permissions_profiles_proxy_policy_starts_managed_network_proxy`
    
    ## Documentation
    
    Public Codex config docs should mention these built-in names when the
    `[permissions]` config format is ready to document as stable.
    
    
    
    
    
    
    
    
    
    ---
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/19900).
    * #20041
    * #20040
    * #20037
    * #20035
    * #20034
    * #20033
    * #20032
    * #20030
    * #20028
    * #20027
    * #20026
    * #20024
    * #20021
    * #20018
    * #20016
    * #20015
    * #20013
    * #20011
    * #20010
    * #20008
    * __->__ #19900
  • remove temporary ownership re-exports (#16626)
    Stacked on #16508.
    
    This removes the temporary `codex-core` / `codex-login` re-export shims
    from the ownership split and rewrites callsites to import directly from
    `codex-model-provider-info`, `codex-models-manager`, `codex-api`,
    `codex-protocol`, `codex-feedback`, and `codex-response-debug-context`.
    
    No behavior change intended; this is the mechanical import cleanup layer
    split out from the ownership move.
    
    ---------
    
    Co-authored-by: Codex <noreply@openai.com>
  • extract models manager and related ownership from core (#16508)
    ## Summary
    - split `models-manager` out of `core` and add `ModelsManagerConfig`
    plus `Config::to_models_manager_config()` so model metadata paths stop
    depending on `core::Config`
    - move login-owned/auth-owned code out of `core` into `codex-login`,
    move model provider config into `codex-model-provider-info`, move API
    bridge mapping into `codex-api`, move protocol-owned types/impls into
    `codex-protocol`, and move response debug helpers into a dedicated
    `response-debug-context` crate
    - move feedback tag emission into `codex-feedback`, relocate tests to
    the crates that now own the code, and keep broad temporary re-exports so
    this PR avoids a giant import-only rewrite
    
    ## Major moves and decisions
    - created `codex-models-manager` as the owner for model
    cache/catalog/config/model info logic, including the new
    `ModelsManagerConfig` struct
    - created `codex-model-provider-info` as the owner for provider config
    parsing/defaults and kept temporary `codex-login`/`codex-core`
    re-exports for old import paths
    - moved `api_bridge` error mapping + `CoreAuthProvider` into
    `codex-api`, while `codex-login::api_bridge` temporarily re-exports
    those symbols and keeps the `auth_provider_from_auth` wrapper
    - moved `auth_env_telemetry` and `provider_auth` ownership to
    `codex-login`
    - moved `CodexErr` ownership to `codex-protocol::error`, plus
    `StreamOutput`, `bytes_to_string_smart`, and network policy helpers to
    protocol-owned modules
    - created `codex-response-debug-context` for
    `extract_response_debug_context`, `telemetry_transport_error_message`,
    and related response-debug plumbing instead of leaving that behavior in
    `core`
    - moved `FeedbackRequestTags`, `emit_feedback_request_tags`, and
    `emit_feedback_request_tags_with_auth_env` to `codex-feedback`
    - deferred removal of temporary re-exports and the mechanical import
    rewrites to a stacked follow-up PR so this PR stays reviewable
    
    ## Test moves
    - moved auth refresh coverage from `core/tests/suite/auth_refresh.rs` to
    `login/tests/suite/auth_refresh.rs`
    - moved text encoding coverage from
    `core/tests/suite/text_encoding_fix.rs` to
    `protocol/src/exec_output_tests.rs`
    - moved model info override coverage from
    `core/tests/suite/model_info_overrides.rs` to
    `models-manager/src/model_info_overrides_tests.rs`
    
    ---------
    
    Co-authored-by: Codex <noreply@openai.com>
  • core: remove cross-crate re-exports from lib.rs (#16512)
    ## Why
    
    `codex-core` was re-exporting APIs owned by sibling `codex-*` crates,
    which made downstream crates depend on `codex-core` as a proxy module
    instead of the actual owner crate.
    
    Removing those forwards makes crate boundaries explicit and lets leaf
    crates drop unnecessary `codex-core` dependencies. In this PR, this
    reduces the dependency on `codex-core` to `codex-login` in the following
    files:
    
    ```
    codex-rs/backend-client/Cargo.toml
    codex-rs/mcp-server/tests/common/Cargo.toml
    ```
    
    ## What
    
    - Remove `codex-rs/core/src/lib.rs` re-exports for symbols owned by
    `codex-login`, `codex-mcp`, `codex-rollout`, `codex-analytics`,
    `codex-protocol`, `codex-shell-command`, `codex-sandboxing`,
    `codex-tools`, and `codex-utils-path`.
    - Delete the `default_client` forwarding shim in `codex-rs/core`.
    - Update in-crate and downstream callsites to import directly from the
    owning `codex-*` crate.
    - Add direct Cargo dependencies where callsites now target the owner
    crate, and remove `codex-core` from `codex-rs/backend-client`.
  • ci: verify codex-rs Cargo manifests inherit workspace settings (#16353)
    ## Why
    
    Bazel clippy now catches lints that `cargo clippy` can still miss when a
    crate under `codex-rs` forgets to opt into workspace lints. The concrete
    example here was `codex-rs/app-server/tests/common/Cargo.toml`: Bazel
    flagged a clippy violation in `models_cache.rs`, but Cargo did not
    because that crate inherited workspace package metadata without
    declaring `[lints] workspace = true`.
    
    We already mirror the workspace clippy deny list into Bazel after
    [#15955](https://github.com/openai/codex/pull/15955), so we also need a
    repo-side check that keeps every `codex-rs` manifest opted into the same
    workspace settings.
    
    ## What changed
    
    - add `.github/scripts/verify_cargo_workspace_manifests.py`, which
    parses every `codex-rs/**/Cargo.toml` with `tomllib` and verifies:
      - `version.workspace = true`
      - `edition.workspace = true`
      - `license.workspace = true`
      - `[lints] workspace = true`
    - top-level crate names follow the `codex-*` / `codex-utils-*`
    conventions, with explicit exceptions for `windows-sandbox-rs` and
    `utils/path-utils`
    - run that script in `.github/workflows/ci.yml`
    - update the current outlier manifests so the check is enforceable
    immediately
    - fix the newly exposed clippy violations in the affected crates
    (`app-server/tests/common`, `file-search`, `feedback`,
    `shell-escalation`, and `debug-client`)
    
    
    
    
    
    
    ---
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/16353).
    * #16351
    * __->__ #16353
  • Add remote env CI matrix and integration test (#14869)
    `CODEX_TEST_REMOTE_ENV` will make `test_codex` start the executor
    "remotely" (inside a docker container) turning any integration test into
    remote test.
  • Split features into codex-features crate (#15253)
    - Split the feature system into a new `codex-features` crate.
    - Cut `codex-core` and workspace consumers over to the new config and
    warning APIs.
    
    Co-authored-by: Ahmed Ibrahim <219906144+aibrahim-oai@users.noreply.github.com>
    Co-authored-by: Codex <noreply@openai.com>
  • feat(core, tracing): create turn spans over websockets (#14632)
    ## Description
    
    Dependent on:
    - [responsesapi] https://github.com/openai/openai/pull/760991 
    - [codex-backend] https://github.com/openai/openai/pull/760985
    
    `codex app-server -> codex-backend -> responsesapi` now reuses a
    persistent websocket connection across many turns. This PR updates
    tracing when using websockets so that each `response.create` websocket
    request propagates the current tracing context, so we can get a holistic
    end-to-end trace for each turn.
    
    Tracing is propagated via special keys (`ws_request_header_traceparent`,
    `ws_request_header_tracestate`) set in the `client_metadata` param in
    Responses API.
    
    Currently tracing on websockets is a bit broken because we only set
    tracing context on ws connection time, so it's detached from a
    `turn/start` request.
  • Remove test-support feature from codex-core and replace it with explicit test toggles (#11405)
    ## Why
    
    `codex-core` was being built in multiple feature-resolved permutations
    because test-only behavior was modeled as crate features. For a large
    crate, those permutations increase compile cost and reduce cache reuse.
    
    ## Net Change
    
    - Removed the `test-support` crate feature and related feature wiring so
    `codex-core` no longer needs separate feature shapes for test consumers.
    - Standardized cross-crate test-only access behind
    `codex_core::test_support`.
    - External test code now imports helpers from
    `codex_core::test_support`.
    - Underlying implementation hooks are kept internal (`pub(crate)`)
    instead of broadly public.
    
    ## Outcome
    
    - Fewer `codex-core` build permutations.
    - Better incremental cache reuse across test targets.
    - No intended production behavior change.
  • Remove deterministic_process_ids feature to avoid duplicate codex-core builds (#11393)
    ## Why
    
    `codex-core` enabled `deterministic_process_ids` through a self
    dev-dependency.
    That forced a second feature-resolved build of the same crate, which
    increased
    compile time and test latency.
    
    ## What Changed
    
    - Removed the `deterministic_process_ids` feature from
    `codex-rs/core/Cargo.toml`.
    - Removed the self dev-dependency on `codex-core` that enabled that
    feature.
    - Removed the Bazel `deterministic_process_ids` crate feature for
    `codex-core`.
    - Added a test-only `AtomicBool` override in unified exec process-id
    allocation.
    - Added a test-support setter for that override and re-exported it from
    `codex-core`.
    - Enabled deterministic process IDs in integration tests via
    `core_test_support` ctor.
    
    ## Behavior
    
    - Production behavior remains random process IDs.
    - Unit tests remain deterministic via `cfg(test)`.
    - Integration tests remain deterministic via explicit test-support
    initialization.
    
    ## Validation
    
    - `just fmt`
    - `cargo test -p codex-core unified_exec::`
    - `cargo test -p codex-core --test all unified_exec -- --test-threads=1`
    - `cargo tree -p codex-core -e features` (verified the removed feature
    path)
  • default enable compression, update test helpers (#10102)
    set `enable_request_compression` flag to default-enabled.
    
    update integration test helpers to decompress `zstd` if flag set.
  • Add model client sessions (#9102)
    Maintain a long-running session.
  • feat: introduce codex-utils-cargo-bin as an alternative to assert_cmd::Command (#8496)
    This PR introduces a `codex-utils-cargo-bin` utility crate that
    wraps/replaces our use of `assert_cmd::Command` and
    `escargot::CargoBuild`.
    
    As you can infer from the introduction of `buck_project_root()` in this
    PR, I am attempting to make it possible to build Codex under
    [Buck2](https://buck2.build) as well as `cargo`. With Buck2, I hope to
    achieve faster incremental local builds (largely due to Buck2's
    [dice](https://buck2.build/docs/insights_and_knowledge/modern_dice/)
    build strategy, as well as benefits from its local build daemon) as well
    as faster CI builds if we invest in remote execution and caching.
    
    See
    https://buck2.build/docs/getting_started/what_is_buck2/#why-use-buck2-key-advantages
    for more details about the performance advantages of Buck2.
    
    Buck2 enforces stronger requirements in terms of build and test
    isolation. It discourages assumptions about absolute paths (which is key
    to enabling remote execution). Because the `CARGO_BIN_EXE_*` environment
    variables that Cargo provides are absolute paths (which
    `assert_cmd::Command` reads), this is a problem for Buck2, which is why
    we need this `codex-utils-cargo-bin` utility.
    
    My WIP-Buck2 setup sets the `CARGO_BIN_EXE_*` environment variables
    passed to a `rust_test()` build rule as relative paths.
    `codex-utils-cargo-bin` will resolve these values to absolute paths,
    when necessary.
    
    
    ---
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/8496).
    * #8498
    * __->__ #8496
  • Wire with_remote_overrides to construct model families (#7621)
    - This PR wires `with_remote_overrides` and make the
    `construct_model_families` an async function
    - Moves getting model family a level above to keep the function `sync`
    - Updates the tests to local, offline, and `sync` helper for model
    families
  • chore: add cargo-deny configuration (#7119)
    - add GitHub workflow running cargo-deny on push/PR
    - document cargo-deny allowlist with workspace-dep notes and advisory
    ignores
    - align workspace crates to inherit version/edition/license for
    consistent checks
  • Account for encrypted reasoning for auto compaction (#7113)
    - The total token used returned from the api doesn't account for the
    reasoning items before the assistant message
    - Account for those for auto compaction
    - Add the encrypted reasoning effort in the common tests utils
    - Add a test to make sure it works as expected
  • test: reduce time dependency on test harness (#5053)
    Tightened the CLI integration tests to stop relying on wall-clock
    sleeps—new fs watcher helper waits for session files instead of timing
    out, and SSE mocks/fixtures make the flows deterministic.
  • Make output assertions more explicit (#4784)
    Match using precise regexes.
  • Add codex exec testing helpers (#4254)
    Add a shortcut to create working directories and run codex exec with
    fake server.
  • Add notifier tests (#4064)
    Proposal:
    1. Use anyhow for tests and avoid unwrap
    2. Extract a helper for starting a test instance of codex
  • chore: unify cargo versions (#4044)
    Unify cargo versions at root
  • fix: create separate test_support crates to eliminate #[allow(dead_code)] (#1667)
    Because of a quirk of how implementation tests work in Rust, we had a
    number of `#[allow(dead_code)]` annotations that were misleading because
    the functions _were_ being used, just not by all integration tests in a
    `tests/` folder, so when compiling the test that did not use the
    function, clippy would complain that it was unused.
    
    This fixes things by create a "test_support" crate under the `tests/`
    folder that is imported as a dev dependency for the respective crate.