mirror of
https://github.com/pchuan98/codex.git
synced 2026-07-01 00:31:56 +08:00
343d1af3da
## Why This PR is the current, consolidated follow-up to the earlier Windows Bazel attempt in #11229. The goal is no longer just to get a tiny Windows smoke job limping along: it is to make the ordinary Bazel CI path usable on `windows-latest` for `x86_64-pc-windows-gnullvm`, with the same broad `//...` test shape that macOS and Linux already use. The earlier smoke-list version of this work was useful as a foothold, but it was not a good long-term landing point. Windows Bazel kept surfacing real issues outside that allowlist: - GitHub's Windows runner exposed runfiles-manifest bugs such as `FINDSTR: Cannot open D:MANIFEST`, which broke Bazel test launchers even when the manifest file existed. - `rules_rs`, `rules_rust`, LLVM extraction, and Abseil still needed `windows-gnullvm`-specific fixes for our hermetic toolchain. - the V8 path needed more work than just turning the Windows matrix entry back on: `rusty_v8` does not ship Windows GNU artifacts in the same shape we need, and Bazel's in-tree V8 build needed a set of Windows GNU portability fixes. Windows performance pressure also pushed this toward a full solution instead of a permanent smoke suite. During this investigation we hit targets such as `//codex-rs/shell-command:shell-command-unit-tests` that were much more expensive on Windows because they repeatedly spawn real PowerShell parsers (see #16057 for one concrete example of that pressure). That made it much more valuable to get the real Windows Bazel path working than to keep iterating on a narrowly curated subset. The net result is that this PR now aims for the same CI contract on Windows that we already expect elsewhere: keep standalone `//third_party/v8:all` out of the ordinary Bazel lane, but allow V8 consumers under `//codex-rs/...` to build and test transitively through `//...`. ## What Changed ### CI and workflow wiring - re-enable the `windows-latest` / `x86_64-pc-windows-gnullvm` Bazel matrix entry in `.github/workflows/bazel.yml` - move the Windows Bazel output root to `D:\b` and enable `git config --global core.longpaths true` in `.github/actions/setup-bazel-ci/action.yml` - keep the ordinary Bazel target set on Windows aligned with macOS and Linux by running `//...` while excluding only standalone `//third_party/v8:all` targets from the normal lane ### Toolchain and module support for `windows-gnullvm` - patch `rules_rs` so `windows-gnullvm` is modeled as a distinct Windows exec/toolchain platform instead of collapsing into the generic Windows shape - patch `rules_rust` build-script environment handling so llvm-mingw build-script probes do not inherit unsupported `-fstack-protector*` flags - patch the LLVM module archive so it extracts cleanly on Windows and provides the MinGW libraries this toolchain needs - patch Abseil so its thread-local identity path matches the hermetic `windows-gnullvm` toolchain instead of taking an incompatible MinGW pthread path - keep both MSVC and GNU Windows targets in the generated Cargo metadata because the current V8 release-asset story still uses MSVC-shaped names in some places while the Bazel build targets the GNU ABI ### Windows test-launch and binary-behavior fixes - update `workspace_root_test_launcher.bat.tpl` to read the runfiles manifest directly instead of shelling out to `findstr`, which was the source of the `D:MANIFEST` failures on the GitHub Windows runner - thread a larger Windows GNU stack reserve through `defs.bzl` so Bazel-built binaries that pull in V8 behave correctly both under normal builds and under `bazel test` - remove the no-longer-needed Windows bootstrap sh-toolchain override from `.bazelrc` ### V8 / `rusty_v8` Windows GNU support - export and apply the new Windows GNU patch set from `patches/BUILD.bazel` / `MODULE.bazel` - patch the V8 module/rules/source layers so the in-tree V8 build can produce Windows GNU archives under Bazel - teach `third_party/v8/BUILD.bazel` to build Windows GNU static archives in-tree instead of aliasing them to the MSVC prebuilts - reuse the Linux release binding for the experimental Windows GNU path where `rusty_v8` does not currently publish a Windows GNU binding artifact ## Testing - the primary end-to-end validation for this work is the `Bazel` workflow plus `v8-canary`, since the hard parts are Windows-specific and depend on real GitHub runner behavior - before consolidation back onto this PR, the same net change passed the full Bazel matrix in [run 23675590471](https://github.com/openai/codex/actions/runs/23675590471) and passed `v8-canary` in [run 23675590453](https://github.com/openai/codex/actions/runs/23675590453) - those successful runs included the `windows-latest` / `x86_64-pc-windows-gnullvm` Bazel job with the ordinary `//...` path, not the earlier Windows smoke allowlist --- [//]: # (BEGIN SAPLING FOOTER) Stack created with [Sapling](https://sapling-scm.com). Best reviewed with [ReviewStack](https://reviewstack.dev/openai/codex/pull/15952). * #16067 * __->__ #15952
278 lines
11 KiB
Python
278 lines
11 KiB
Python
load("@crates//:data.bzl", "DEP_DATA")
|
|
load("@crates//:defs.bzl", "all_crate_deps")
|
|
load("@rules_platform//platform_data:defs.bzl", "platform_data")
|
|
load("@rules_rust//rust:defs.bzl", "rust_binary", "rust_library", "rust_proc_macro", "rust_test")
|
|
load("@rules_rust//cargo/private:cargo_build_script_wrapper.bzl", "cargo_build_script")
|
|
|
|
PLATFORMS = [
|
|
"linux_arm64_musl",
|
|
"linux_amd64_musl",
|
|
"macos_amd64",
|
|
"macos_arm64",
|
|
"windows_amd64",
|
|
"windows_arm64",
|
|
]
|
|
|
|
# The Bazel-built windows-gnullvm binaries that pull in V8 need a larger PE
|
|
# stack reserve than the default linker setting. Thread the flag through the
|
|
# executable and test entry points so the final linked artifacts behave the same
|
|
# in normal builds and under `bazel test`.
|
|
WINDOWS_GNULLVM_RUSTC_STACK_FLAGS = select({
|
|
"@rules_rs//rs/experimental/platforms/constraints:windows_gnullvm": [
|
|
"-C",
|
|
"link-arg=-Wl,--stack,8388608",
|
|
],
|
|
"//conditions:default": [],
|
|
})
|
|
|
|
def multiplatform_binaries(name, platforms = PLATFORMS):
|
|
for platform in platforms:
|
|
platform_data(
|
|
name = name + "_" + platform,
|
|
platform = "@llvm//platforms:" + platform,
|
|
target = name,
|
|
tags = ["manual"],
|
|
)
|
|
|
|
native.filegroup(
|
|
name = "release_binaries",
|
|
srcs = [name + "_" + platform for platform in platforms],
|
|
tags = ["manual"],
|
|
)
|
|
|
|
def _workspace_root_test_impl(ctx):
|
|
is_windows = ctx.target_platform_has_constraint(ctx.attr._windows_constraint[platform_common.ConstraintValueInfo])
|
|
launcher = ctx.actions.declare_file(ctx.label.name + ".bat" if is_windows else ctx.label.name)
|
|
test_bin = ctx.executable.test_bin
|
|
workspace_root_marker = ctx.file.workspace_root_marker
|
|
launcher_template = ctx.file._windows_launcher_template if is_windows else ctx.file._bash_launcher_template
|
|
ctx.actions.expand_template(
|
|
template = launcher_template,
|
|
output = launcher,
|
|
is_executable = True,
|
|
substitutions = {
|
|
"__TEST_BIN__": test_bin.short_path,
|
|
"__WORKSPACE_ROOT_MARKER__": workspace_root_marker.short_path,
|
|
},
|
|
)
|
|
|
|
runfiles = ctx.runfiles(files = [test_bin, workspace_root_marker]).merge(ctx.attr.test_bin[DefaultInfo].default_runfiles)
|
|
|
|
return [
|
|
DefaultInfo(
|
|
executable = launcher,
|
|
files = depset([launcher]),
|
|
runfiles = runfiles,
|
|
),
|
|
RunEnvironmentInfo(
|
|
environment = ctx.attr.env,
|
|
),
|
|
]
|
|
|
|
workspace_root_test = rule(
|
|
implementation = _workspace_root_test_impl,
|
|
test = True,
|
|
attrs = {
|
|
"env": attr.string_dict(),
|
|
"test_bin": attr.label(
|
|
cfg = "target",
|
|
executable = True,
|
|
mandatory = True,
|
|
),
|
|
"workspace_root_marker": attr.label(
|
|
allow_single_file = True,
|
|
mandatory = True,
|
|
),
|
|
"_windows_constraint": attr.label(
|
|
default = "@platforms//os:windows",
|
|
providers = [platform_common.ConstraintValueInfo],
|
|
),
|
|
"_bash_launcher_template": attr.label(
|
|
allow_single_file = True,
|
|
default = "//:workspace_root_test_launcher.sh.tpl",
|
|
),
|
|
"_windows_launcher_template": attr.label(
|
|
allow_single_file = True,
|
|
default = "//:workspace_root_test_launcher.bat.tpl",
|
|
),
|
|
},
|
|
)
|
|
|
|
def codex_rust_crate(
|
|
name,
|
|
crate_name,
|
|
crate_features = [],
|
|
crate_srcs = None,
|
|
crate_edition = None,
|
|
proc_macro = False,
|
|
build_script_enabled = True,
|
|
build_script_data = [],
|
|
compile_data = [],
|
|
lib_data_extra = [],
|
|
rustc_flags_extra = [],
|
|
rustc_env = {},
|
|
deps_extra = [],
|
|
integration_compile_data_extra = [],
|
|
test_data_extra = [],
|
|
test_tags = [],
|
|
extra_binaries = []):
|
|
"""Defines a Rust crate with library, binaries, and tests wired for Bazel + Cargo parity.
|
|
|
|
The macro mirrors Cargo conventions: it builds a library when `src/` exists,
|
|
wires build scripts, exports `CARGO_BIN_EXE_*` for integration tests, and
|
|
creates unit + integration test targets. Dependency buckets map to the
|
|
Cargo.lock resolution in `@crates`.
|
|
|
|
Args:
|
|
name: Bazel target name for the library, should be the directory name.
|
|
Example: `app-server`.
|
|
crate_name: Cargo crate name from Cargo.toml
|
|
Example: `codex_app_server`.
|
|
crate_features: Cargo features to enable for this crate.
|
|
Crates are only compiled in a single configuration across the workspace, i.e.
|
|
with all features in this list enabled. So use sparingly, and prefer to refactor
|
|
optional functionality to a separate crate.
|
|
crate_srcs: Optional explicit srcs; defaults to `src/**/*.rs`.
|
|
crate_edition: Rust edition override, if not default.
|
|
You probably don't want this, it's only here for a single caller.
|
|
proc_macro: Whether this crate builds a proc-macro library.
|
|
build_script_data: Data files exposed to the build script at runtime.
|
|
compile_data: Non-Rust compile-time data for the library target.
|
|
lib_data_extra: Extra runtime data for the library target.
|
|
rustc_env: Extra rustc_env entries to merge with defaults.
|
|
deps_extra: Extra normal deps beyond @crates resolution.
|
|
Typically only needed when features add additional deps.
|
|
integration_compile_data_extra: Extra compile_data for integration tests.
|
|
test_data_extra: Extra runtime data for tests.
|
|
test_tags: Tags applied to unit + integration test targets.
|
|
Typically used to disable the sandbox, but see https://bazel.build/reference/be/common-definitions#common.tags
|
|
extra_binaries: Additional binary labels to surface as test data and
|
|
`CARGO_BIN_EXE_*` environment variables. These are only needed for binaries from a different crate.
|
|
"""
|
|
test_env = {
|
|
# The launcher resolves an absolute workspace root at runtime so
|
|
# manifest-only platforms like macOS still point Insta at the real
|
|
# `codex-rs` checkout.
|
|
"INSTA_WORKSPACE_ROOT": ".",
|
|
"INSTA_SNAPSHOT_PATH": "src",
|
|
}
|
|
|
|
rustc_env = {
|
|
"BAZEL_PACKAGE": native.package_name(),
|
|
} | rustc_env
|
|
|
|
binaries = DEP_DATA.get(native.package_name())["binaries"]
|
|
|
|
lib_srcs = crate_srcs or native.glob(["src/**/*.rs"], exclude = binaries.values(), allow_empty = True)
|
|
|
|
maybe_deps = []
|
|
|
|
if build_script_enabled and native.glob(["build.rs"], allow_empty = True):
|
|
cargo_build_script(
|
|
name = name + "-build-script",
|
|
srcs = ["build.rs"],
|
|
deps = all_crate_deps(build = True),
|
|
data = build_script_data,
|
|
# Some build script deps sniff version-related env vars...
|
|
version = "0.0.0",
|
|
)
|
|
|
|
maybe_deps += [name + "-build-script"]
|
|
|
|
if lib_srcs:
|
|
lib_rule = rust_proc_macro if proc_macro else rust_library
|
|
lib_rule(
|
|
name = name,
|
|
crate_name = crate_name,
|
|
crate_features = crate_features,
|
|
deps = all_crate_deps() + maybe_deps + deps_extra,
|
|
compile_data = compile_data,
|
|
data = lib_data_extra,
|
|
srcs = lib_srcs,
|
|
edition = crate_edition,
|
|
rustc_flags = rustc_flags_extra,
|
|
rustc_env = rustc_env,
|
|
visibility = ["//visibility:public"],
|
|
)
|
|
|
|
unit_test_binary = name + "-unit-tests-bin"
|
|
rust_test(
|
|
name = unit_test_binary,
|
|
crate = name,
|
|
deps = all_crate_deps(normal = True, normal_dev = True) + maybe_deps + deps_extra,
|
|
# Bazel has emitted both `codex-rs/<crate>/...` and
|
|
# `../codex-rs/<crate>/...` paths for `file!()`. Strip either
|
|
# prefix so the workspace-root launcher sees Cargo-like metadata
|
|
# such as `tui/src/...`.
|
|
rustc_flags = rustc_flags_extra + [
|
|
"--remap-path-prefix=../codex-rs=",
|
|
"--remap-path-prefix=codex-rs=",
|
|
],
|
|
rustc_env = rustc_env,
|
|
data = test_data_extra,
|
|
tags = test_tags + ["manual"],
|
|
)
|
|
|
|
workspace_root_test(
|
|
name = name + "-unit-tests",
|
|
env = test_env,
|
|
test_bin = ":" + unit_test_binary,
|
|
workspace_root_marker = "//codex-rs/utils/cargo-bin:repo_root.marker",
|
|
tags = test_tags,
|
|
)
|
|
|
|
maybe_deps += [name]
|
|
|
|
sanitized_binaries = []
|
|
cargo_env = {}
|
|
for binary, main in binaries.items():
|
|
#binary = binary.replace("-", "_")
|
|
sanitized_binaries.append(binary)
|
|
cargo_env["CARGO_BIN_EXE_" + binary] = "$(rlocationpath :%s)" % binary
|
|
|
|
rust_binary(
|
|
name = binary,
|
|
crate_name = binary.replace("-", "_"),
|
|
crate_root = main,
|
|
deps = all_crate_deps() + maybe_deps + deps_extra,
|
|
edition = crate_edition,
|
|
rustc_flags = rustc_flags_extra + WINDOWS_GNULLVM_RUSTC_STACK_FLAGS,
|
|
srcs = native.glob(["src/**/*.rs"]),
|
|
visibility = ["//visibility:public"],
|
|
)
|
|
|
|
for binary_label in extra_binaries:
|
|
sanitized_binaries.append(binary_label)
|
|
binary = Label(binary_label).name
|
|
cargo_env["CARGO_BIN_EXE_" + binary] = "$(rlocationpath %s)" % binary_label
|
|
|
|
for test in native.glob(["tests/*.rs"], allow_empty = True):
|
|
test_file_stem = test.removeprefix("tests/").removesuffix(".rs")
|
|
test_crate_name = test_file_stem.replace("-", "_")
|
|
test_name = name + "-" + test_file_stem.replace("/", "-")
|
|
if not test_name.endswith("-test"):
|
|
test_name += "-test"
|
|
|
|
rust_test(
|
|
name = test_name,
|
|
crate_name = test_crate_name,
|
|
crate_root = test,
|
|
srcs = [test],
|
|
data = native.glob(["tests/**"], allow_empty = True) + sanitized_binaries + test_data_extra,
|
|
compile_data = native.glob(["tests/**"], allow_empty = True) + integration_compile_data_extra,
|
|
deps = all_crate_deps(normal = True, normal_dev = True) + maybe_deps + deps_extra,
|
|
# Bazel has emitted both `codex-rs/<crate>/...` and
|
|
# `../codex-rs/<crate>/...` paths for `file!()`. Strip either
|
|
# prefix so Insta records Cargo-like metadata such as `core/tests/...`.
|
|
rustc_flags = rustc_flags_extra + WINDOWS_GNULLVM_RUSTC_STACK_FLAGS + [
|
|
"--remap-path-prefix=../codex-rs=",
|
|
"--remap-path-prefix=codex-rs=",
|
|
],
|
|
rustc_env = rustc_env,
|
|
# Important: do not merge `test_env` here. Its unit-test-only
|
|
# `INSTA_WORKSPACE_ROOT="codex-rs"` is tuned for unit tests that
|
|
# execute from the repo root and can misplace integration snapshots.
|
|
env = cargo_env,
|
|
tags = test_tags,
|
|
)
|