[codex] Speed up local nextest runs (#26479)

## Why

`just test` currently uses the CI-oriented nextest profile, which
serializes app-server integration tests even on developer machines that
can run several safely. Bounded local parallelism substantially shortens
this common iteration loop without changing CI behavior.

Eight-worker experiments were faster, but keeping them reliable required
relaxing several test deadlines. Four workers for integration tests is a
solid tradeoff that speeds up local testing without needing to change
test logic.

## What changed

- Add a `local` nextest profile that inherits the existing defaults.
- Allow up to four app-server integration tests to run concurrently
under that profile.
- Make `just test` select the local profile on Unix and Windows.
- Keep the default CI profile serialized and leave all test deadlines
unchanged.

The tests use separate processes, randomized temporary `CODEX_HOME`
directories, and ephemeral ports. The remaining shared constraints are
system resources; each app-server also uses a multi-thread Tokio
runtime, and fuzzy-search tests can create additional worker threads, so
the local cap remains intentionally conservative.

## Performance and validation

All measurements below are warm, execution-only app-server runs with
nextest retries disabled.

On the current rebased branch, an AMD EPYC 7763 machine with 16 logical
CPUs and 62 GiB RAM completed three consecutive runs:

| Run | Nextest time | Wall time | Result |
| --- | ---: | ---: | --- |
| 1 | 142.941s | 145.17s | 836/836 passed |
| 2 | 143.402s | 145.59s | 836/836 passed |
| 3 | 142.870s | 145.08s | 836/836 passed |

The mean wall time was 145.28s. The slow-inventory, approval replay, and
zsh-fork tests all passed with their original deadlines.

Earlier measurements on the same Linux machine, before the suite grew,
showed the scaling that motivated the change:

| App-server concurrency | Nextest time | Result |
| --- | ---: | --- |
| 1 | 369.5s | 572/572 passed |
| 2 | 194.5s | 572/572 passed |
| 4 | 111.0s mean over 3 runs | 3/3 clean |

Four workers reduced that execution time by about 70%, a roughly 3.3x
speedup over serialization.
This commit is contained in:
Adam Perry @ OpenAI
2026-06-09 12:48:04 -07:00
committed by GitHub
Unverified
parent a71e040df5
commit 6e7ab52929
2 changed files with 16 additions and 2 deletions
+14
View File
@@ -8,12 +8,20 @@ retries = 1
[profile.default.junit]
path = "junit.xml"
[profile.local]
inherits = "default"
[test-groups.app_server_protocol_codegen]
max-threads = 1
[test-groups.app_server_integration]
max-threads = 1
# Higher concurrency causes integration test timeouts under resource contention
# on common developer machines.
[test-groups.app_server_integration_local]
max-threads = 4
[test-groups.core_apply_patch_cli_integration]
max-threads = 1
@@ -42,6 +50,12 @@ test-group = 'app_server_protocol_codegen'
filter = 'package(codex-app-server) & kind(test)'
test-group = 'app_server_integration'
[[profile.local.overrides]]
# Use up to four app-server subprocesses locally. The global nextest pool still
# limits this to the machine's logical CPU count.
filter = 'package(codex-app-server) & kind(test)'
test-group = 'app_server_integration_local'
[[profile.default.overrides]]
# These tests exercise full Codex turns and apply_patch execution, and they are
# sensitive to Windows runner process-startup stalls when many cases launch at once.
+2 -2
View File
@@ -75,11 +75,11 @@ install:
# there should be no need to add `--all-features`.
[unix]
test *args:
RUST_MIN_STACK={{ rust_min_stack }} cargo nextest run --no-fail-fast "$@"
RUST_MIN_STACK={{ rust_min_stack }} NEXTEST_PROFILE=local cargo nextest run --no-fail-fast "$@"
[windows]
test *args:
$env:RUST_MIN_STACK = "{{ rust_min_stack }}"; cargo nextest run --no-fail-fast @($args | Select-Object -Skip 1)
$env:RUST_MIN_STACK = "{{ rust_min_stack }}"; $env:NEXTEST_PROFILE = "local"; cargo nextest run --no-fail-fast @($args | Select-Object -Skip 1)
# Run from the repository root so scripts that resolve paths from `cwd` see
# the same layout they use in GitHub Actions.