This adds a dummy v8-poc project that in Cargo links against our
prebuilt binaries and the ones provided by rusty_v8 for non musl
platforms. This demonstrates that we can successfully link and use v8 on
all platforms that we want to target.
In bazel things are slightly more complicated. Since the libraries as
published have libc++ linked in already we end up with a lot of double
linked symbols if we try to use them in bazel land. Instead we fall back
to building rusty_v8 and v8 from source (cached of course) on the
platforms we ship to.
There is likely some compatibility drift in the windows bazel builder
that we'll need to reconcile before we can re-enable them. I'm happy to
be on the hook to unwind that.
Fix Bazel macOS CI failures caused by the llvm module's pinned macOS SDK
URL returning 403 Forbidden from Apple's CDN.
Bump llvm to 0.6.8, switch to the new osx.from_archive(...) /
osx.frameworks(...) API, and refresh MODULE.bazel.lock so Bazel uses the
updated SDK archive configuration.
Alternative approach, we use rusty_v8 for all platforms that its
predefined, but lets build from source a musl v8 version with bazel for
x86 and aarch64 only. We would need to release this on github and then
use the release.
This lets us drop various patches and go all the way to a very clean
setup.
In case folks are curious what was going on... we were depending on the
toolchain finding stdlib headers as sibling files of `clang++`, and for
linking we were providing a `-resource-dir` containing the runtime libs.
However, some users of the cc toolchain (such as rust build scripts) do
the equivalent of `$CC $CCFLAGS $LDFLAGS` so the `-resource-dir` was
being passed when compiling, which suppressed the default stdlib header
location logic. The upstream fix was to swap to using `-isystem` to pass
the stdlib headers, while carefully controlling the ordering to simulate
them coming from the resource-dir.
# External (non-OpenAI) Pull Request Requirements
Before opening this Pull Request, please read the dedicated
"Contributing" markdown file or your PR may be closed:
https://github.com/openai/codex/blob/main/docs/contributing.md
If your PR conforms to our contribution guidelines, replace this text
with a detailed and high quality description of your changes.
Include a link to a bug report or enhancement request.
Summary
- simplify the macOS sleep inhibitor FFI by replacing `dlopen` / `dlsym`
/ `transmute` with normal IOKit extern calls and `SAFETY` comments
- switch to cfg-selected platform implementations
(`imp::SleepInhibitor`) instead of `Box<dyn ...>`
- check in minimal IOKit bindings generated with `bindgen` and include
them from the macOS backend
- enable direct IOKit linkage in Bazel macOS builds by registering
`IOKit` in the Bazel `osx.framework(...)` toolchain extension list
- update `Cargo.lock` and `MODULE.bazel.lock` after removing the
build-time `bindgen` dependency path
Testing
- `just fmt`
- `cargo clippy -p codex-utils-sleep-inhibitor --all-targets -- -D
warnings`
- `cargo test -p codex-utils-sleep-inhibitor`
- `bazel test //codex-rs/utils/sleep-inhibitor:all --test_output=errors`
- `just bazel-lock-update`
- `just bazel-lock-check`
Context
- follow-up to #11711 addressing Ryan's review comments
- `bindgen` is used to generate the checked-in bindings file, but not at
build time
## Summary
This PR removes the temporary `CODEX_BWRAP_ENABLE_FFI` flag and makes
Linux builds always compile vendored bubblewrap support for
`codex-linux-sandbox`.
## Changes
- Removed `CODEX_BWRAP_ENABLE_FFI` gating from
`codex-rs/linux-sandbox/build.rs`.
- Linux builds now fail fast if vendored bubblewrap compilation fails
(instead of warning and continuing).
- Updated fallback/help text in
`codex-rs/linux-sandbox/src/vendored_bwrap.rs` to remove references to
`CODEX_BWRAP_ENABLE_FFI`.
- Removed `CODEX_BWRAP_ENABLE_FFI` env wiring from:
- `.github/workflows/rust-ci.yml`
- `.github/workflows/bazel.yml`
- `.github/workflows/rust-release.yml`
---------
Co-authored-by: David Zbarsky <zbarsky@openai.com>
we can't use runfiles directory on Windows due to path lengths, so swap
to manifest strategy. Parsing the manifest is a bit complex and the
format is changing in Bazel upstream, so pull in the official Rust
library (via a small hack to make it importable...) and cleanup all the
associated logic to work cleanly in both bazel and cargo without extra
confusion
I needed to upgrade bazel one to get gnullvm artifacts and then noticed
monorepo had drifted forward. They should move in lockstep. Also 1.93
already shipped so we can try that instead.
On bazel9 this lets us avoid performing some external repo downloads if
they've been previously uploaded to remote cache, downloads are deferred
until they are actually needed to execute an uncached action
This PR configures Codex CLI so it can be built with
[Bazel](https://bazel.build) in addition to Cargo. The `.bazelrc`
includes configuration so that remote builds can be done using
[BuildBuddy](https://www.buildbuddy.io).
If you are familiar with Bazel, things should work as you expect, e.g.,
run `bazel test //... --keep-going` to run all the tests in the repo,
but we have also added some new aliases in the `justfile` for
convenience:
- `just bazel-test` to run tests locally
- `just bazel-remote-test` to run tests remotely (currently, the remote
build is for x86_64 Linux regardless of your host platform). Note we are
currently seeing the following test failures in the remote build, so we
still need to figure out what is happening here:
```
failures:
suite::compact::manual_compact_twice_preserves_latest_user_messages
suite::compact_resume_fork::compact_resume_after_second_compaction_preserves_history
suite::compact_resume_fork::compact_resume_and_fork_preserve_model_history_view
```
- `just build-for-release` to build release binaries for all
platforms/architectures remotely
To setup remote execution:
- [Create a buildbuddy account](https://app.buildbuddy.io/) (OpenAI
employees should also request org access at
https://openai.buildbuddy.io/join/ with their `@openai.com` email
address.)
- [Copy your API key](https://app.buildbuddy.io/docs/setup/) to
`~/.bazelrc` (add the line `build
--remote_header=x-buildbuddy-api-key=YOUR_KEY`)
- Use `--config=remote` in your `bazel` invocations (or add `common
--config=remote` to your `~/.bazelrc`, or use the `just` commands)
## CI
In terms of CI, this PR introduces `.github/workflows/bazel.yml`, which
uses Bazel to run the tests _locally_ on Mac and Linux GitHub runners
(we are working on supporting Windows, but that is not ready yet). Note
that the failures we are seeing in `just bazel-remote-test` do not occur
on these GitHub CI jobs, so everything in `.github/workflows/bazel.yml`
is green right now.
The `bazel.yml` uses extra config in `.github/workflows/ci.bazelrc` so
that macOS CI jobs build _remotely_ on Linux hosts (using the
`docker://docker.io/mbolin491/codex-bazel` Docker image declared in the
root `BUILD.bazel`) using cross-compilation to build the macOS
artifacts. Then these artifacts are downloaded locally to GitHub's macOS
runner so the tests can be executed natively. This is the relevant
config that enables this:
```
common:macos --config=remote
common:macos --strategy=remote
common:macos --strategy=TestRunner=darwin-sandbox,local
```
Because of the remote caching benefits we get from BuildBuddy, these new
CI jobs can be extremely fast! For example, consider these two jobs that
ran all the tests on Linux x86_64:
- Bazel 1m37s
https://github.com/openai/codex/actions/runs/20861063212/job/59940545209?pr=8875
- Cargo 9m20s
https://github.com/openai/codex/actions/runs/20861063192/job/59940559592?pr=8875
For now, we will continue to run both the Bazel and Cargo jobs for PRs,
but once we add support for Windows and running Clippy, we should be
able to cutover to using Bazel exclusively for PRs, which should still
speed things up considerably. We will probably continue to run the Cargo
jobs post-merge for commits that land on `main` as a sanity check.
Release builds will also continue to be done by Cargo for now.
Earlier attempt at this PR: https://github.com/openai/codex/pull/8832
Earlier attempt to add support for Buck2, now abandoned:
https://github.com/openai/codex/pull/8504
---------
Co-authored-by: David Zbarsky <dzbarsky@gmail.com>
Co-authored-by: Michael Bolin <mbolin@openai.com>
Channing Conger ·
jif-oai ·
zbarsky-openai ·
Jeremy Rose ·
Yaroslav Volovich ·
viyatb-oai ·
Anton Panasenko ·