## Summary
Set `RUST_MIN_STACK=8388608` for Rust test entry points so
libtest-spawned test threads get an 8 MiB stack.
The Windows BuildBuddy failure on #18893 showed
`//codex-rs/tui:tui-unit-tests` exiting with a stack overflow in a
`#[tokio::test]` even though later test binaries in the shard printed
successful summaries. Default `#[tokio::test]` uses a current-thread
Tokio runtime, which means the async test body is driven on libtest's
std-spawned test thread. Increasing the test thread stack addresses that
failure mode directly.
To date, we have been fixing these stack-pressure problems with
localized future-size reductions, such as #13429, and by adding
`Box::pin()` in specific async wrapper chains. This gives us a baseline
test-runner stack size instead of continuing to patch individual tests
only after CI finds another large async future.
## What changed
- Added `common --test_env=RUST_MIN_STACK=8388608` in `.bazelrc` so
Bazel test actions receive the env var through Bazel's cache-keyed test
environment path.
- Set the same `RUST_MIN_STACK` value for Cargo/nextest CI entry points
and `just test`.
- Annotated the existing Windows Bazel linker stack reserve as 8 MiB so
it stays aligned with the libtest thread stack size.
## Testing
- `just --list`
- parsed `.github/workflows/rust-ci.yml` and
`.github/workflows/rust-ci-full.yml` with Ruby's YAML loader
- compared `bazel aquery` `TestRunner` action keys before/after explicit
`--test_env=RUST_MIN_STACK=...` and after moving the Bazel env to
`.bazelrc`
- `bazel test //codex-rs/tui:tui-unit-tests --test_output=errors`
- failed locally on the existing sandbox-specific status snapshot
permission mismatch, but loaded the Starlark changes and ran the TUI
test shards
## What changed
This PR makes the default Cargo dev profile use line-tables-only debug
info:
```toml
[profile.dev]
debug = 1
```
That keeps useful backtraces while avoiding the cost of full variable
debug
info in normal local dev builds.
This also makes the Bazel CI setting explicit with `-Cdebuginfo=0` for
target
and exec-configuration Rust actions. Bazel/rules_rust does not read
Cargo
profiles for this setting, and the current fastbuild action already
emitted
`--codegen=debuginfo=0`; the Bazel part of this PR makes that choice
direct in
our build configuration.
## Why
The slow codex-core rebuilds are dominated by debug-info codegen, not
parsing
or type checking. On a warm-dependency package rebuild, the baseline
codex-core compile was about 39.5s wall / 38.9s rustc total, with
codegen_crate around 14.0s and LLVM_passes around 13.4s. Setting
codex-core
to line-tables-only debug info brought that to about 27.2s wall / 26.7s
rustc
total, with codegen_crate around 3.1s and LLVM_passes around 2.8s.
`debug = 0` was only about another 0.7s faster than `debug = 1` in the
codex-core measurement, so `debug = 1` is the better default dev
tradeoff: it
captures nearly all of the compile-time win while preserving basic
debuggability.
I also sampled other first-party crates instead of keeping a
codex-core-only
package override. codex-app-server showed the same pattern: rustc total
dropped from 15.85s to 10.48s, while codegen_crate plus LLVM_passes
dropped
from about 13.47s to 3.23s. codex-app-server-protocol had a smaller but
still
real improvement, 16.05s to 14.58s total, and smaller crates showed
modest
wins. That points to a workspace dev-profile policy rather than a
hand-maintained list of large crates.
## Relationship to #18612
[#18612](https://github.com/openai/codex/pull/18612) added the
`dev-small`
profile. That remains useful when someone wants a working local build
quickly
and is willing to opt in with `cargo build --profile dev-small`.
This PR is deliberately less aggressive: it changes the common default
dev
profile while preserving line tables/backtraces. `dev-small` remains the
explicit "build quickly, no debuggability concern" path.
## Other investigation
I looked for another structural win comparable to
[#16631](https://github.com/openai/codex/pull/16631) and
[#16630](https://github.com/openai/codex/pull/16630), but did not find
one.
The attempted TOML monomorphization changes were noisy or worse in
measurement, and the async task changes reduced some instantiations but
only
translated to roughly a one-second improvement while being much more
disruptive. The debug-info setting was the one repeatable, material win
that
survived measurement.
## Verification
- `just bazel-lock-update`
- `just bazel-lock-check`
- `cargo check -p codex-core --lib`
- `cargo test -p codex-core --lib`
- Bazel `aquery --config=ci-linux` confirmed `--codegen=debuginfo=0` and
`-Cdebuginfo=0` for `//codex-rs/core:core`
---
[//]: # (BEGIN SAPLING FOOTER)
Stack created with [Sapling](https://sapling-scm.com). Best reviewed
with [ReviewStack](https://reviewstack.dev/openai/codex/pull/18844).
* #18846
* __->__ #18844
Follow-up to https://github.com/openai/codex/pull/18178, where we said
the await-holding clippy rule would be enabled separately.
Enable `await_holding_lock` and `await_holding_invalid_type` after the
preceding commits fixed or explicitly documented the current offenders.
## Why
Bazel CI had two independent Windows issues:
- The workflow saved/restored `~/.cache/bazel-repo-cache`, but
`.bazelrc` configured `common:ci-windows
--repository_cache=D:/a/.cache/bazel-repo-cache`, so `actions/cache` and
Bazel could point at different directories.
- The Windows `Bazel clippy` job passed the full explicit target list
from `//codex-rs/...`, but some of those explicit targets are
intentionally incompatible with `//:local_windows`.
`run-argument-comment-lint-bazel.sh` already handles that with
`--skip_incompatible_explicit_targets`; the clippy workflow path did
not.
I also tried switching the workflow cache path to
`D:\a\.cache\bazel-repo-cache`, but the Windows clippy job repeatedly
failed with `Failed to restore: Cache service responded with 400`, so
the final change standardizes on `$HOME/.cache/bazel-repo-cache` and
makes cache restore non-fatal.
## What Changed
- Expose one repository-cache path from
`.github/actions/setup-bazel-ci/action.yml` and export that path as
`BAZEL_REPOSITORY_CACHE` so `run-bazel-ci.sh` passes it to Bazel after
`--config=ci-*`.
- Move `actions/cache/restore` out of the composite action into
`.github/workflows/bazel.yml`, and make restore failures non-fatal
there.
- Save exactly the exported cache path in `.github/workflows/bazel.yml`.
- Remove `common:ci-windows
--repository_cache=D:/a/.cache/bazel-repo-cache` from `.bazelrc` so the
Windows CI config no longer disagrees with the workflow cache path.
- Pass `--skip_incompatible_explicit_targets` in the Windows `Bazel
clippy` job so incompatible explicit targets do not fail analysis while
the lint aspect still traverses compatible Rust dependencies.
## Verification
- Parsed `.github/actions/setup-bazel-ci/action.yml` and
`.github/workflows/bazel.yml` with Ruby's YAML loader.
- Resubmitted PR `#16740`; CI is rerunning on the amended commit.
## Why
Follow-up to #16345, the Bazel clippy rollout in #15955, and the cleanup
pass in #16353.
`cargo clippy` was enforcing the workspace deny-list from
`codex-rs/Cargo.toml` because the member crates opt into `[lints]
workspace = true`, but Bazel clippy was only using `rules_rust` plus
`clippy.toml`. That left the Bazel lane vulnerable to drift:
`clippy.toml` can tune lint behavior, but it cannot set
allow/warn/deny/forbid levels.
This PR now closes both sides of the follow-up. It keeps `.bazelrc` in
sync with `[workspace.lints.clippy]`, and it fixes the real clippy
violations that the newly-synced Windows Bazel lane surfaced once that
deny-list started matching Cargo.
## What Changed
- added `.github/scripts/verify_bazel_clippy_lints.py`, a Python check
that parses `codex-rs/Cargo.toml` with `tomllib`, reads the Bazel
`build:clippy` `clippy_flag` entries from `.bazelrc`, and reports
missing, extra, or mismatched lint levels
- ran that verifier from the lightweight `ci.yml` workflow so the sync
check does not depend on a Rust toolchain being installed first
- expanded the `.bazelrc` comment to explain the Cargo `workspace =
true` linkage and why Bazel needs the deny-list duplicated explicitly
- fixed the Windows-only `codex-windows-sandbox` violations that Bazel
clippy reported after the sync, using the same style as #16353: inline
`format!` args, method references instead of trivial closures, removed
redundant clones, and replaced SID conversion `unwrap` and `expect`
calls with proper errors
- cleaned up the remaining cross-platform violations the Bazel lane
exposed in `codex-backend-client` and `core_test_support`
## Testing
Key new test introduced by this PR:
`python3 .github/scripts/verify_bazel_clippy_lints.py`
## Why
`argument-comment-lint` had become a PR bottleneck because the repo-wide
lane was still effectively running a `cargo dylint`-style flow across
the workspace instead of reusing Bazel's Rust dependency graph. That
kept the lint enforced, but it threw away the main benefit of moving
this job under Bazel in the first place: metadata reuse and cacheable
per-target analysis in the same shape as Clippy.
This change moves the repo-wide lint onto a native Bazel Rust aspect so
Linux and macOS can lint `codex-rs` without rebuilding the world
crate-by-crate through the wrapper path.
## What Changed
- add a nightly Rust toolchain with `rustc-dev` for Bazel and a
dedicated crate-universe repo for `tools/argument-comment-lint`
- add `tools/argument-comment-lint/driver.rs` and
`tools/argument-comment-lint/lint_aspect.bzl` so Bazel can run the lint
as a custom `rustc_driver`
- switch repo-wide `just argument-comment-lint` and the Linux/macOS
`rust-ci` lanes to `bazel build --config=argument-comment-lint
//codex-rs/...`
- keep the Python/DotSlash wrappers as the package-scoped fallback path
and as the current Windows CI path
- gate the Dylint entrypoint behind a `bazel_native` feature so the
Bazel-native library avoids the `dylint_*` packaging stack
- update the aspect runtime environment so the driver can locate
`rustc_driver` correctly under remote execution
- keep the dedicated `tools/argument-comment-lint` package tests and
wrapper unit tests in CI so the source and packaged entrypoints remain
covered
## Verification
- `python3 -m unittest discover -s tools/argument-comment-lint -p
'test_*.py'`
- `cargo test` in `tools/argument-comment-lint`
- `bazel build
//tools/argument-comment-lint:argument-comment-lint-driver
--@rules_rust//rust/toolchain/channel=nightly`
- `bazel build --config=argument-comment-lint
//codex-rs/utils/path-utils:all`
- `bazel build --config=argument-comment-lint
//codex-rs/rollout:rollout`
---
[//]: # (BEGIN SAPLING FOOTER)
Stack created with [Sapling](https://sapling-scm.com). Best reviewed
with [ReviewStack](https://reviewstack.dev/openai/codex/pull/16106).
* #16120
* __->__ #16106
## Why
This PR is the current, consolidated follow-up to the earlier Windows
Bazel attempt in #11229. The goal is no longer just to get a tiny
Windows smoke job limping along: it is to make the ordinary Bazel CI
path usable on `windows-latest` for `x86_64-pc-windows-gnullvm`, with
the same broad `//...` test shape that macOS and Linux already use.
The earlier smoke-list version of this work was useful as a foothold,
but it was not a good long-term landing point. Windows Bazel kept
surfacing real issues outside that allowlist:
- GitHub's Windows runner exposed runfiles-manifest bugs such as
`FINDSTR: Cannot open D:MANIFEST`, which broke Bazel test launchers even
when the manifest file existed.
- `rules_rs`, `rules_rust`, LLVM extraction, and Abseil still needed
`windows-gnullvm`-specific fixes for our hermetic toolchain.
- the V8 path needed more work than just turning the Windows matrix
entry back on: `rusty_v8` does not ship Windows GNU artifacts in the
same shape we need, and Bazel's in-tree V8 build needed a set of Windows
GNU portability fixes.
Windows performance pressure also pushed this toward a full solution
instead of a permanent smoke suite. During this investigation we hit
targets such as `//codex-rs/shell-command:shell-command-unit-tests` that
were much more expensive on Windows because they repeatedly spawn real
PowerShell parsers (see #16057 for one concrete example of that
pressure). That made it much more valuable to get the real Windows Bazel
path working than to keep iterating on a narrowly curated subset.
The net result is that this PR now aims for the same CI contract on
Windows that we already expect elsewhere: keep standalone
`//third_party/v8:all` out of the ordinary Bazel lane, but allow V8
consumers under `//codex-rs/...` to build and test transitively through
`//...`.
## What Changed
### CI and workflow wiring
- re-enable the `windows-latest` / `x86_64-pc-windows-gnullvm` Bazel
matrix entry in `.github/workflows/bazel.yml`
- move the Windows Bazel output root to `D:\b` and enable `git config
--global core.longpaths true` in
`.github/actions/setup-bazel-ci/action.yml`
- keep the ordinary Bazel target set on Windows aligned with macOS and
Linux by running `//...` while excluding only standalone
`//third_party/v8:all` targets from the normal lane
### Toolchain and module support for `windows-gnullvm`
- patch `rules_rs` so `windows-gnullvm` is modeled as a distinct Windows
exec/toolchain platform instead of collapsing into the generic Windows
shape
- patch `rules_rust` build-script environment handling so llvm-mingw
build-script probes do not inherit unsupported `-fstack-protector*`
flags
- patch the LLVM module archive so it extracts cleanly on Windows and
provides the MinGW libraries this toolchain needs
- patch Abseil so its thread-local identity path matches the hermetic
`windows-gnullvm` toolchain instead of taking an incompatible MinGW
pthread path
- keep both MSVC and GNU Windows targets in the generated Cargo metadata
because the current V8 release-asset story still uses MSVC-shaped names
in some places while the Bazel build targets the GNU ABI
### Windows test-launch and binary-behavior fixes
- update `workspace_root_test_launcher.bat.tpl` to read the runfiles
manifest directly instead of shelling out to `findstr`, which was the
source of the `D:MANIFEST` failures on the GitHub Windows runner
- thread a larger Windows GNU stack reserve through `defs.bzl` so
Bazel-built binaries that pull in V8 behave correctly both under normal
builds and under `bazel test`
- remove the no-longer-needed Windows bootstrap sh-toolchain override
from `.bazelrc`
### V8 / `rusty_v8` Windows GNU support
- export and apply the new Windows GNU patch set from
`patches/BUILD.bazel` / `MODULE.bazel`
- patch the V8 module/rules/source layers so the in-tree V8 build can
produce Windows GNU archives under Bazel
- teach `third_party/v8/BUILD.bazel` to build Windows GNU static
archives in-tree instead of aliasing them to the MSVC prebuilts
- reuse the Linux release binding for the experimental Windows GNU path
where `rusty_v8` does not currently publish a Windows GNU binding
artifact
## Testing
- the primary end-to-end validation for this work is the `Bazel`
workflow plus `v8-canary`, since the hard parts are Windows-specific and
depend on real GitHub runner behavior
- before consolidation back onto this PR, the same net change passed the
full Bazel matrix in [run
23675590471](https://github.com/openai/codex/actions/runs/23675590471)
and passed `v8-canary` in [run
23675590453](https://github.com/openai/codex/actions/runs/23675590453)
- those successful runs included the `windows-latest` /
`x86_64-pc-windows-gnullvm` Bazel job with the ordinary `//...` path,
not the earlier Windows smoke allowlist
---
[//]: # (BEGIN SAPLING FOOTER)
Stack created with [Sapling](https://sapling-scm.com). Best reviewed
with [ReviewStack](https://reviewstack.dev/openai/codex/pull/15952).
* #16067
* __->__ #15952
## Why
`bazel.yml` already builds and tests the Bazel graph, but `rust-ci.yml`
still runs `cargo clippy` separately. This PR starts the transition to a
Bazel-backed lint lane for `codex-rs` so we can eventually replace the
duplicate Rust build, test, and lint work with Bazel while explicitly
keeping the V8 Bazel path out of scope for now.
To make that lane practical, the workflow also needs to look like the
Bazel job we already trust. That means sharing the common Bazel setup
and invocation logic instead of hand-copying it, and covering the arm64
macOS path in addition to Linux.
Landing the workflow green also required fixing the first lint findings
that Bazel surfaced and adding the matching local entrypoint.
## What changed
- add a reusable `build:clippy` config to `.bazelrc` and export
`codex-rs/clippy.toml` from `codex-rs/BUILD.bazel` so Bazel can run the
repository's existing Clippy policy
- add `just bazel-clippy` so the local developer entrypoint matches the
new CI lane
- extend `.github/workflows/bazel.yml` with a dedicated Bazel clippy job
for `codex-rs`, scoped to `//codex-rs/... -//codex-rs/v8-poc:all`
- run that clippy job on Linux x64 and arm64 macOS
- factor the shared Bazel workflow setup into
`.github/actions/setup-bazel-ci/action.yml` and the shared Bazel
invocation logic into `.github/scripts/run-bazel-ci.sh` so the clippy
and build/test jobs stay aligned
- fix the first Bazel-clippy findings needed to keep the lane green,
including the cross-target `cmsghdr::cmsg_len` normalization in
`codex-rs/shell-escalation/src/unix/socket.rs` and the no-`voice-input`
dead-code warnings in `codex-rs/tui` and `codex-rs/tui_app_server`
## Verification
- `just bazel-clippy`
- `RUNNER_OS=macOS ./.github/scripts/run-bazel-ci.sh -- build
--config=clippy --build_metadata=COMMIT_SHA=local-check
--build_metadata=TAG_job=clippy -- //codex-rs/...
-//codex-rs/v8-poc:all`
- `bazel build --config=clippy
//codex-rs/shell-escalation:shell-escalation`
- `CARGO_TARGET_DIR=/tmp/codex4-shell-escalation-test cargo test -p
codex-shell-escalation`
- `ruby -e 'require "yaml";
YAML.load_file(".github/workflows/bazel.yml");
YAML.load_file(".github/actions/setup-bazel-ci/action.yml")'`
## Notes
- `CARGO_TARGET_DIR=/tmp/codex4-tui-app-server-test cargo test -p
codex-tui-app-server` still hits existing guardian-approvals test and
snapshot failures unrelated to this PR's Bazel-clippy changes.
Related: #15954
Replaced ci.bazelrc and v8-ci.bazelrc by custom configs inside the main
.bazelrc file. As a result, github workflows setup is simplified down to
a single '--config=<foo>' flag usage.
Moved the build metadata flags to config=ci.
Added custom tags metadata to help differentiate invocations based on
workflow (bazel vs v8) and os (linux/macos/windows).
Enabled users to override the default values in .bazelrc by using a
user.bazelrc file locally.
Added user.bazelrc to gitignore.
This lets us drop various patches and go all the way to a very clean
setup.
In case folks are curious what was going on... we were depending on the
toolchain finding stdlib headers as sibling files of `clang++`, and for
linking we were providing a `-resource-dir` containing the runtime libs.
However, some users of the cc toolchain (such as rust build scripts) do
the equivalent of `$CC $CCFLAGS $LDFLAGS` so the `-resource-dir` was
being passed when compiling, which suppressed the default stdlib header
location logic. The upstream fix was to swap to using `-isystem` to pass
the stdlib headers, while carefully controlling the ordering to simulate
them coming from the resource-dir.
# External (non-OpenAI) Pull Request Requirements
Before opening this Pull Request, please read the dedicated
"Contributing" markdown file or your PR may be closed:
https://github.com/openai/codex/blob/main/docs/contributing.md
If your PR conforms to our contribution guidelines, replace this text
with a detailed and high quality description of your changes.
Include a link to a bug report or enhancement request.
we can't use runfiles directory on Windows due to path lengths, so swap
to manifest strategy. Parsing the manifest is a bit complex and the
format is changing in Bazel upstream, so pull in the official Rust
library (via a small hack to make it importable...) and cleanup all the
associated logic to work cleanly in both bazel and cargo without extra
confusion
On bazel9 this lets us avoid performing some external repo downloads if
they've been previously uploaded to remote cache, downloads are deferred
until they are actually needed to execute an uncached action
This PR configures Codex CLI so it can be built with
[Bazel](https://bazel.build) in addition to Cargo. The `.bazelrc`
includes configuration so that remote builds can be done using
[BuildBuddy](https://www.buildbuddy.io).
If you are familiar with Bazel, things should work as you expect, e.g.,
run `bazel test //... --keep-going` to run all the tests in the repo,
but we have also added some new aliases in the `justfile` for
convenience:
- `just bazel-test` to run tests locally
- `just bazel-remote-test` to run tests remotely (currently, the remote
build is for x86_64 Linux regardless of your host platform). Note we are
currently seeing the following test failures in the remote build, so we
still need to figure out what is happening here:
```
failures:
suite::compact::manual_compact_twice_preserves_latest_user_messages
suite::compact_resume_fork::compact_resume_after_second_compaction_preserves_history
suite::compact_resume_fork::compact_resume_and_fork_preserve_model_history_view
```
- `just build-for-release` to build release binaries for all
platforms/architectures remotely
To setup remote execution:
- [Create a buildbuddy account](https://app.buildbuddy.io/) (OpenAI
employees should also request org access at
https://openai.buildbuddy.io/join/ with their `@openai.com` email
address.)
- [Copy your API key](https://app.buildbuddy.io/docs/setup/) to
`~/.bazelrc` (add the line `build
--remote_header=x-buildbuddy-api-key=YOUR_KEY`)
- Use `--config=remote` in your `bazel` invocations (or add `common
--config=remote` to your `~/.bazelrc`, or use the `just` commands)
## CI
In terms of CI, this PR introduces `.github/workflows/bazel.yml`, which
uses Bazel to run the tests _locally_ on Mac and Linux GitHub runners
(we are working on supporting Windows, but that is not ready yet). Note
that the failures we are seeing in `just bazel-remote-test` do not occur
on these GitHub CI jobs, so everything in `.github/workflows/bazel.yml`
is green right now.
The `bazel.yml` uses extra config in `.github/workflows/ci.bazelrc` so
that macOS CI jobs build _remotely_ on Linux hosts (using the
`docker://docker.io/mbolin491/codex-bazel` Docker image declared in the
root `BUILD.bazel`) using cross-compilation to build the macOS
artifacts. Then these artifacts are downloaded locally to GitHub's macOS
runner so the tests can be executed natively. This is the relevant
config that enables this:
```
common:macos --config=remote
common:macos --strategy=remote
common:macos --strategy=TestRunner=darwin-sandbox,local
```
Because of the remote caching benefits we get from BuildBuddy, these new
CI jobs can be extremely fast! For example, consider these two jobs that
ran all the tests on Linux x86_64:
- Bazel 1m37s
https://github.com/openai/codex/actions/runs/20861063212/job/59940545209?pr=8875
- Cargo 9m20s
https://github.com/openai/codex/actions/runs/20861063192/job/59940559592?pr=8875
For now, we will continue to run both the Bazel and Cargo jobs for PRs,
but once we add support for Windows and running Clippy, we should be
able to cutover to using Bazel exclusively for PRs, which should still
speed things up considerably. We will probably continue to run the Cargo
jobs post-merge for commits that land on `main` as a sanity check.
Release builds will also continue to be done by Cargo for now.
Earlier attempt at this PR: https://github.com/openai/codex/pull/8832
Earlier attempt to add support for Buck2, now abandoned:
https://github.com/openai/codex/pull/8504
---------
Co-authored-by: David Zbarsky <dzbarsky@gmail.com>
Co-authored-by: Michael Bolin <mbolin@openai.com>
Michael Bolin ·
Ruslan Nigmatullin ·
Son Luong Ngoc ·
zbarsky-openai ·