## Why
`just test` currently uses the CI-oriented nextest profile, which
serializes app-server integration tests even on developer machines that
can run several safely. Bounded local parallelism substantially shortens
this common iteration loop without changing CI behavior.
Eight-worker experiments were faster, but keeping them reliable required
relaxing several test deadlines. Four workers for integration tests is a
solid tradeoff that speeds up local testing without needing to change
test logic.
## What changed
- Add a `local` nextest profile that inherits the existing defaults.
- Allow up to four app-server integration tests to run concurrently
under that profile.
- Make `just test` select the local profile on Unix and Windows.
- Keep the default CI profile serialized and leave all test deadlines
unchanged.
The tests use separate processes, randomized temporary `CODEX_HOME`
directories, and ephemeral ports. The remaining shared constraints are
system resources; each app-server also uses a multi-thread Tokio
runtime, and fuzzy-search tests can create additional worker threads, so
the local cap remains intentionally conservative.
## Performance and validation
All measurements below are warm, execution-only app-server runs with
nextest retries disabled.
On the current rebased branch, an AMD EPYC 7763 machine with 16 logical
CPUs and 62 GiB RAM completed three consecutive runs:
| Run | Nextest time | Wall time | Result |
| --- | ---: | ---: | --- |
| 1 | 142.941s | 145.17s | 836/836 passed |
| 2 | 143.402s | 145.59s | 836/836 passed |
| 3 | 142.870s | 145.08s | 836/836 passed |
The mean wall time was 145.28s. The slow-inventory, approval replay, and
zsh-fork tests all passed with their original deadlines.
Earlier measurements on the same Linux machine, before the suite grew,
showed the scaling that motivated the change:
| App-server concurrency | Nextest time | Result |
| --- | ---: | --- |
| 1 | 369.5s | 572/572 passed |
| 2 | 194.5s | 572/572 passed |
| 4 | 111.0s mean over 3 runs | 3/3 clean |
Four workers reduced that execution time by about 70%, a roughly 3.3x
speedup over serialization.
## Why
`rust-ci-full` was paying the full Cargo nextest build-and-run cost once
per platform, with Windows ARM64 as the long pole. This change moves the
heavy work into one reusable per-platform flow: build a nextest archive
once, then replay it across four shards so the platform lane spends less
time running tests serially. For Windows ARM64, the archive is
cross-compiled on Windows x64 and replayed on native Windows ARM64
shards so the slow ARM64 machine is used for execution rather than
compilation.
## What changed
- split the `rust-ci-full` nextest matrix into five explicit
per-platform reusable-workflow calls
- add `.github/workflows/rust-ci-full-nextest-platform.yml` to build one
archive, upload timings/helpers, replay four nextest shards, upload
per-shard JUnit, and roll the shard status back up per platform
- add Windows CI helpers for Dev Drive setup and MSVC ARM64 linker
environment export so the Windows ARM64 archive can be produced on
Windows x64
- keep the existing Cargo git CLI fetch hardening inside the reusable
workflow, since caller workflow-level `env` does not flow through
`workflow_call`
- document the archive-backed shard shape in
`.github/workflows/README.md`
- raise the default nextest slow timeout to 30s so the sharded full-CI
path does not treat every >15s test as stuck
## Verification
- validated the archive/shard flow with live GitHub Actions runs on this
PR branch
- Windows ARM64 cross-compile latency on completed runs:
- https://github.com/openai/codex/actions/runs/26118759651: `34m30s`
lane e2e, `17m16s` archive build, `9m55s` shard phase
- https://github.com/openai/codex/actions/runs/26120777976: `30m36s`
lane e2e, `17m21s` archive build, `6m50s` shard phase
- comparable pre-cross-compile sharded Windows ARM64 runs were `55m01s`,
`50m21s`, and `46m42s`, so the completed cross-compile runs improved the
lane by roughly `12m` to `24m` versus the prior range
- latest corrected cross-compile run:
https://github.com/openai/codex/actions/runs/26120777976
- Windows ARM64 archive built successfully on Windows x64
- native Windows ARM64 shards started immediately after the archive
upload
- 3/4 Windows ARM64 shards passed; the failing shard hit the same
existing `code_mode` test failure seen outside this lane
- downloaded failed-shard JUnit XML from the validation runs and
confirmed the remaining red is from known test failures, not
archive/shard wiring
- no local Codex tests run per repo guidance
## Notes
- this PR does not change developers.openai.com documentation
## Why
Recent `rust-ci-full` failures were dominated by transient Windows
timeout clusters in process-heavy tests such as `suite::resume`,
`suite::cli_stream`, `suite::auth_env`,
`start_thread_uses_all_default_environments_from_codex_home`, and
`connect_stdio_command_initializes_json_rpc_client_on_windows`.
The goal here is to make those known flaky paths less likely to fail
full CI without relaxing the global nextest timeout policy.
## What changed
- Enable one global nextest retry with `retries = 1` so a single
transient failure can recover.
- Add a `windows_process_heavy` test group with `max-threads = 2` for
the recurring Windows subprocess/session-heavy timeout families.
- Add Windows-only slow-timeout overrides for that process-heavy group.
- Add a narrower Windows-only timeout override for
`start_thread_uses_all_default_environments_from_codex_home`, which
still exceeded the broader Windows bucket in both Windows full-CI lanes.
- Increase the `rust-ci-full` nextest job timeout from `45m` to `60m` so
Windows ARM64 still has job-level headroom after retries and targeted
per-test timeout increases.
- Keep the global `slow-timeout` unchanged at `15s`.
## Validation
Validated through `rust-ci-full` GitHub Actions reruns on this PR.
Observed improvement on the tuned Windows lanes:
- Windows x64 went from `5 timed out` to `0 timed out`.
- Windows ARM64 went from `2 timed out` to `0 timed out`.
- `start_thread_uses_all_default_environments_from_codex_home` recovered
as a flaky pass on Windows ARM64 instead of timing out.
The remaining failing tests in those runs were unrelated hard failures
outside this nextest timeout tuning.
## Why
`rust-ci-full` failures currently leave downstream investigation
reconstructing basic test facts from raw logs. `cargo nextest` can emit
standard JUnit XML for each lane, which gives us a small structured
artifact for post-run failure analysis without changing the test
execution model.
## What changed
- enable nextest JUnit output in `codex-rs/.config/nextest.toml`
- upload the lane-scoped JUnit XML artifact from each `rust-ci-full`
test lane
## Verification
- `rust-ci-full` run `26018931531` on head
`52d77c60e79b36859d944ef28a36b014055c5c48` produced JUnit artifacts for
macOS, Linux x64 remote, Windows x64, and Windows ARM64 test lanes
- `rust-ci-full` run `26021241006` on the same head produced the missing
Linux ARM JUnit artifact after the first run lost that runner before
export
- downloaded all five lane JUnit artifacts and verified each contains
non-empty test counters and failure data
## Why
A [Windows Cargo
build](https://github.com/openai/codex/actions/runs/24754807756/job/72425641062)
on `main` timed out in several unrelated-looking suites at the same
time:
- `codex-app-server` account tests failed before account logic, while
`mcp.initialize()` was waiting for the first JSON-RPC response.
- `codex-core` `apply_patch_cli` tests timed out while running full
Codex/apply_patch turns.
- `codex-windows-sandbox` legacy session tests timed out while creating
restricted-token child processes and private desktops.
The app-server log reached the test harness write path in
[`McpProcess::initialize_with_params`](https://github.com/openai/codex/blob/731b54d08fb93aec31fb020999a62447886f3ab3/codex-rs/app-server/tests/common/mcp_process.rs#L244-L263),
but never printed the matching stdout read from
[`read_jsonrpc_message`](https://github.com/openai/codex/blob/731b54d08fb93aec31fb020999a62447886f3ab3/codex-rs/app-server/tests/common/mcp_process.rs#L1123-L1128).
The server initialize handler is a small bookkeeping/response path
([`message_processor.rs`](https://github.com/openai/codex/blob/731b54d08fb93aec31fb020999a62447886f3ab3/codex-rs/app-server/src/message_processor.rs#L601-L728)),
so the failure looks like Windows runner process/pipe scheduling
starvation rather than account-specific behavior.
## What Changed
This updates `.config/nextest.toml` to serialize two process-heavy sets:
- `codex-core` tests matching `package(codex-core) & kind(test) &
test(apply_patch_cli)`
- `codex-windows-sandbox` tests matching `package(codex-windows-sandbox)
& test(legacy_)`
`codex-app-server` integration tests were already serialized inside
their own package; this change reduces overlap with the other suites
that were saturating the runner at the same time.
## Verification
- `cargo nextest list --filterset "package(codex-core) & kind(test) &
test(apply_patch_cli)"`
- `cargo nextest list --filterset "package(codex-windows-sandbox) &
test(legacy_)"`
The Windows sandbox filter naturally lists no tests on macOS, but it
validates the nextest filter/config syntax locally.
## What changed
- TypeScript schema fixture generation now goes through in-memory tree
helpers rather than a heavier on-disk generation path.
- The comparison logic normalizes generated banner and path differences
that are not semantically relevant to the exported schema.
- TypeScript and JSON fixture coverage are split into separate tests,
and the expensive schema-export tests are serialized in `nextest`.
## Why this fixes the flake
- The original fixture coverage mixed several heavy codegen paths into
one monolithic test and then compared generated output that included
incidental banner/path differences.
- On Windows CI, that combination created both runtime pressure and
output variance unrelated to the schema shapes we actually care about.
- Splitting the coverage isolates failures by format, in-memory
generation reduces filesystem churn, normalization strips generator
noise, and serializing the heavy tests removes parallel resource
contention.
## Scope
- Production helper change plus test changes.
Adam Perry @ OpenAI ·
starr-openai ·
Michael Bolin ·
Ahmed Ibrahim ·
pakrym-oai ·