Commit Graph

40 Commits

  • Inject SKILL.md when it's explicitly mentioned. (#7763)
    1. Skills load once in core at session start; the cached outcome is
    reused across core and surfaced to TUI via SessionConfigured.
    2. TUI detects explicit skill selections, and core injects the matching
    SKILL.md content into the turn when a selected skill is present.
  • Add remote models feature flag (#7648)
    # External (non-OpenAI) Pull Request Requirements
    
    Before opening this Pull Request, please read the dedicated
    "Contributing" markdown file or your PR may be closed:
    https://github.com/openai/codex/blob/main/docs/contributing.md
    
    If your PR conforms to our contribution guidelines, replace this text
    with a detailed and high quality description of your changes.
    
    Include a link to a bug report or enhancement request.
  • feat(core) Add login to shell_command tool (#6846)
    ## Summary
    Adds the `login` parameter to the `shell_command` tool - optional,
    defaults to true.
    
    ## Testing
    - [x] Tested locally
  • chore(core): test apply_patch_cli on Windows (#7554)
    ## Summary
    These tests pass on windows, let's enable them.
    
    ## Testing
    - [x] These are more tests
  • Migrate model preset (#7542)
    - Introduce `openai_models` in `/core`
    - Move `PRESETS` under it
    - Move `ModelPreset`, `ModelUpgrade`, `ReasoningEffortPreset`,
    `ReasoningEffortPreset`, and `ReasoningEffortPreset` to `protocol`
    - Introduce `Op::ListModels` and `EventMsg::AvailableModels`
    
    Next steps:
    - migrate `app-server` and `tui` to use the introduced Operation
  • Fix: Improve text encoding for shell output in VSCode preview (#6178) (#6182)
    ## 🐛 Problem
    
    Users running commands with non-ASCII characters (like Russian text
    "пример") in Windows/WSL environments experience garbled text in
    VSCode's shell preview window, with Unicode replacement characters (�)
    appearing instead of the actual text.
    
    **Issue**: https://github.com/openai/codex/issues/6178
    
    ## 🔧 Root Cause
    
    The issue was in `StreamOutput<Vec<u8>>::from_utf8_lossy()` method in
    `codex-rs/core/src/exec.rs`, which used `String::from_utf8_lossy()` to
    convert shell output bytes to strings. This function immediately
    replaces any invalid UTF-8 byte sequences with replacement characters,
    without attempting to decode using other common encodings.
    
    In Windows/WSL environments, shell output often uses encodings like:
    
    - Windows-1252 (common Windows encoding)
    - Latin-1/ISO-8859-1 (extended ASCII)
    
    ## 🛠️ Solution
    
    Replaced the simple `String::from_utf8_lossy()` call with intelligent
    encoding detection via a new `bytes_to_string_smart()` function that
    tries multiple encoding strategies:
    
    1. **UTF-8** (fast path for valid UTF-8)
    2. **Windows-1252** (handles Windows-specific characters in 0x80-0x9F
    range)
    3. **Latin-1** (fallback for extended ASCII)
    4. **Lossy UTF-8** (final fallback, same as before)
    
    ## 📁 Changes
    
    ### New Files
    
    - `codex-rs/core/src/text_encoding.rs` - Smart encoding detection module
    - `codex-rs/core/tests/suite/text_encoding_fix.rs` - Integration tests
    
    ### Modified Files
    
    - `codex-rs/core/src/lib.rs` - Added text_encoding module
    - `codex-rs/core/src/exec.rs` - Updated StreamOutput::from_utf8_lossy()
    - `codex-rs/core/tests/suite/mod.rs` - Registered new test module
    
    ##  Testing
    
    - **5 unit tests** covering UTF-8, Windows-1252, Latin-1, and fallback
    scenarios
    - **2 integration tests** simulating the exact Issue #6178 scenario
    - **Demonstrates improvement** over the previous
    `String::from_utf8_lossy()` approach
    
    All tests pass:
    
    ```bash
    cargo test -p codex-core text_encoding
    cargo test -p codex-core test_shell_output_encoding_issue_6178
    ```
    
    ## 🎯 Impact
    
    -  **Eliminates garbled text** in VSCode shell preview for non-ASCII
    content
    -  **Supports Windows/WSL environments** with proper encoding detection
    -  **Zero performance impact** for UTF-8 text (fast path)
    -  **Backward compatible** - UTF-8 content works exactly as before
    -  **Handles edge cases** with robust fallback mechanism
    
    ## 🧪 Test Scenarios
    
    The fix has been tested with:
    
    - Russian text ("пример")
    - Windows-1252 quotation marks (""test")
    - Latin-1 accented characters ("café")
    - Mixed encoding content
    - Invalid byte sequences (graceful fallback)
    
    ## 📋 Checklist
    
    - [X] Addresses the reported issue
    - [X] Includes comprehensive tests
    - [X] Maintains backward compatibility
    - [X] Follows project coding conventions
    - [X] No breaking changes
    
    ---------
    
    Co-authored-by: Josh McKinney <joshka@openai.com>
  • execpolicy2 core integration (#6641)
    This PR threads execpolicy2 into codex-core.
    
    activated via feature flag: exec_policy (on by default)
    
    reads and parses all .codexpolicy files in `codex_home/codex`
    
    refactored tool runtime API to integrate execpolicy logic
    
    ---------
    
    Co-authored-by: Michael Bolin <mbolin@openai.com>
  • feat: remote compaction (#6795)
    Co-authored-by: pakrym-oai <pakrym@openai.com>
  • chore(core) Consolidate apply_patch tests (#6545)
    ## Summary
    Consolidates our apply_patch tests into one suite, and ensures each test
    case tests the various ways the harness supports apply_patch:
    1. Freeform custom tool call
    2. JSON function tool
    3. Simple shell call
    4. Heredoc shell call
    
    There are a few test cases that are specific to a particular variant,
    I've left those alone.
    
    ## Testing
    - [x] This adds a significant number of tests
  • Use codex-linux-sandbox in unified exec (#6480)
    Unified exec isn't working on Linux because we don't provide the correct
    arg0.
    
    The library we use for pty management doesn't allow setting arg0
    separately from executable. Use the same aliasing strategy we use for
    `apply_patch` for `codex-linux-sandbox`.
    
    Use `#[ctor]` hack to dispatch codex-linux-sandbox calls.
    
    
    Addresses https://github.com/openai/codex/issues/6450
  • Don't retry "insufficient_quota" errors (#6340)
    This PR makes an "insufficient quota" error fatal so we don't attempt to
    retry it multiple times in the agent loop.
    
    We have multiple bug reports from users about intermittent retry
    behaviors, and this could explain some of them. With this change, we'll
    eliminate the retries and surface a clear error message.
    
    The PR is a nearly identical copy of [this
    PR](https://github.com/openai/codex/pull/4837) contributed by
    @abimaelmartell. The original PR has gone stale. Rather than wait for
    the contributor to resolve merge conflicts, I wanted to get this change
    in.
  • Improved token refresh handling to address "Re-connecting" behavior (#6231)
    Currently, when the access token expires, we attempt to use the refresh
    token to acquire a new access token. This works most of the time.
    However, there are situations where the refresh token is expired,
    exhausted (already used to perform a refresh), or revoked. In those
    cases, the current logic treats the error as transient and attempts to
    retry it repeatedly.
    
    This PR changes the token refresh logic to differentiate between
    permanent and transient errors. It also changes callers to treat the
    permanent errors as fatal rather than retrying them. And it provides
    better error messages to users so they understand how to address the
    problem. These error messages should also help us further understand why
    we're seeing examples of refresh token exhaustion.
    
    Here is the error message in the CLI. The same text appears within the
    extension.
    
    <img width="863" height="38" alt="image"
    src="https://github.com/user-attachments/assets/7ffc0d08-ebf0-4900-b9a9-265064202f4f"
    />
    
    I also correct the spelling of "Re-connecting", which shouldn't have a
    hyphen in it.
    
    Testing: I manually tested these code paths by adding temporary code to
    programmatically cause my refresh token to be exhausted (by calling the
    token refresh endpoint in a tight loop more than 50 times). I then
    simulated an access token expiration, which caused the token refresh
    logic to be invoked. I confirmed that the updated logic properly handled
    the error condition.
    
    Note: We earlier discussed the idea of forcefully logging out the user
    at the point where token refresh failed. I made several attempts to do
    this, and all of them resulted in a bad UX. It's important to surface
    this error to users in a way that explains the problem and tells them
    that they need to log in again. We also previously discussed deleting
    the auth.json file when this condition is detected. That also creates
    problems because it effectively changes the auth status from logged in
    to logged out, and this causes odd failures and inconsistent UX. I think
    it's therefore better not to delete auth.json in this case. If the user
    closes the CLI or VSCE and starts it again, we properly detect that the
    access token is expired and the refresh token is "dead", and we force
    the user to go through the login flow at that time.
    
    This should address aspects of #6191, #5679, and #5505
  • chore: testing on freeform apply_patch (#5952)
    ## Summary
    Duplicates the tests in `apply_patch_cli.rs`, but tests the freeform
    apply_patch tool as opposed to the function call path. The good news is
    that all the tests pass with zero logical tests, with the exception of
    the heredoc, which doesn't really make sense in the freeform tool
    context anyway.
    
    @jif-oai since you wrote the original tests in #5557, I'd love your
    opinion on the right way to DRY these test cases between the two. Happy
    to set up a more sophisticated harness, but didn't want to go down the
    rabbit hole until we agreed on the right pattern
    
    ## Testing
    - [x] These are tests
  • Delegate review to codex instance (#5572)
    In this PR, I am exploring migrating task kind to an invocation of
    Codex. The main reason would be getting rid off multiple
    `ConversationHistory` state and streamlining our context/history
    management.
    
    This approach depends on opening a channel between the sub-codex and
    codex. This channel is responsible for forwarding `interactive`
    (`approvals`) and `non-interactive` events. The `task` is responsible
    for handling those events.
    
    This opens the door for implementing `codex as a tool`, replacing
    `compact` and `review`, and potentially subagents.
    
    One consideration is this code is very similar to `app-server` specially
    in the approval part. If in the future we wanted an interactive
    `sub-codex` we should consider using `codex-mcp`
  • feat: deprecation warning (#5825)
    <img width="955" height="311" alt="Screenshot 2025-10-28 at 14 26 25"
    src="https://github.com/user-attachments/assets/99729b3d-3bc9-4503-aab3-8dc919220ab4"
    />
  • feature: Add "!cmd" user shell execution (#2471)
    feature: Add "!cmd" user shell execution
    
    This change lets users run local shell commands directly from the TUI by
    prefixing their input with ! (e.g. !ls). Output is truncated to keep the
    exec cell usable, and Ctrl-C cleanly
      interrupts long-running commands (e.g. !sleep 10000).
    
    **Summary of changes**
    
    - Route Op::RunUserShellCommand through a dedicated UserShellCommandTask
    (core/src/tasks/user_shell.rs), keeping the task logic out of codex.rs.
    - Reuse the existing tool router: the task constructs a ToolCall for the
    local_shell tool and relies on ShellHandler, so no manual MCP tool
    lookup is required.
    - Emit exec lifecycle events (ExecCommandBegin/ExecCommandEnd) so the
    TUI can show command metadata, live output, and exit status.
    
    **End-to-end flow**
    
      **TUI handling**
    
    1. ChatWidget::submit_user_message (TUI) intercepts messages starting
    with !.
    2. Non-empty commands dispatch Op::RunUserShellCommand { command };
    empty commands surface a help hint.
    3. No UserInput items are created, so nothing is enqueued for the model.
    
      **Core submission loop**
    4. The submission loop routes the op to handlers::run_user_shell_command
    (core/src/codex.rs).
    5. A fresh TurnContext is created and Session::spawn_user_shell_command
    enqueues UserShellCommandTask.
    
      **Task execution**
    6. UserShellCommandTask::run emits TaskStartedEvent, formats the
    command, and prepares a ToolCall targeting local_shell.
      7. ToolCallRuntime::handle_tool_call dispatches to ShellHandler.
    
      **Shell tool runtime**
    8. ShellHandler::run_exec_like launches the process via the unified exec
    runtime, honoring sandbox and shell policies, and emits
    ExecCommandBegin/End.
    9. Stdout/stderr are captured for the UI, but the task does not turn the
    resulting ToolOutput into a model response.
    
      **Completion**
    10. After ExecCommandEnd, the task finishes without an assistant
    message; the session marks it complete and the exec cell displays the
    final output.
    
      **Conversation context**
    
    - The command and its output never enter the conversation history or the
    model prompt; the flow is local-only.
      - Only exec/task events are emitted for UI rendering.
    
    **Demo video**
    
    
    https://github.com/user-attachments/assets/fcd114b0-4304-4448-a367-a04c43e0b996
  • Centralize truncation in conversation history (#5652)
    move the truncation logic to conversation history to use on any tool
    output. This will help us in avoiding edge cases while truncating the
    tool calls and mcp calls.
  • Add ItemStarted/ItemCompleted events for UserInputItem (#5306)
    Adds a new ItemStarted event and delivers UserMessage as the first item
    type (more to come).
    
    
    Renames `InputItem` to `UserInput` considering we're using the `Item`
    suffix for actual items.
  • chore: rework tools execution workflow (#5278)
    Re-work the tool execution flow. Read `orchestrator.rs` to understand
    the structure
  • feat: grep_files as a tool (#4820)
    Add `grep_files` to be able to perform more action in parallel
  • feat: list_dir tool (#4817)
    Add a tool to list_dir. It is useful because we can mark it as
    non-mutating and so use it in parallel
  • feat: parallel tool calls (#4663)
    Add parallel tool calls. This is configurable at model level and tool
    level
  • feat: Freeform apply_patch with simple shell output (#4718)
    ## Summary
    This PR is an alternative approach to #4711, but instead of changing our
    storage, parses out shell calls in the client and reserializes them on
    the fly before we send them out as part of the request.
    
    What this changes:
    1. Adds additional serialization logic when the
    ApplyPatchToolType::Freeform is in use.
    2. Adds a --custom-apply-patch flag to enable this setting on a
    session-by-session basis.
    
    This change is delicate, but is not meant to be permanent. It is meant
    to be the first step in a migration:
    1. (This PR) Add in-flight serialization with config
    2. Update model_family default
    3. Update serialization logic to store turn outputs in a structured
    format, with logic to serialize based on model_family setting.
    4. Remove this rewrite in-flight logic.
    
    ## Test Plan
    - [x] Additional unit tests added
    - [x] Integration tests added
    - [x] Tested locally
  • chore: refactor tool handling (#4510)
    # Tool System Refactor
    
    - Centralizes tool definitions and execution in `core/src/tools/*`:
    specs (`spec.rs`), handlers (`handlers/*`), router (`router.rs`),
    registry/dispatch (`registry.rs`), and shared context (`context.rs`).
    One registry now builds the model-visible tool list and binds handlers.
    - Router converts model responses to tool calls; Registry dispatches
    with consistent telemetry via `codex-rs/otel` and unified error
    handling. Function, Local Shell, MCP, and experimental `unified_exec`
    all flow through this path; legacy shell aliases still work.
    - Rationale: reduce per‑tool boilerplate, keep spec/handler in sync, and
    make adding tools predictable and testable.
    
    Example: `read_file`
    - Spec: `core/src/tools/spec.rs` (see `create_read_file_tool`,
    registered by `build_specs`).
    - Handler: `core/src/tools/handlers/read_file.rs` (absolute `file_path`,
    1‑indexed `offset`, `limit`, `L#: ` prefixes, safe truncation).
    - E2E test: `core/tests/suite/read_file.rs` validates the tool returns
    the requested lines.
    
    ## Next steps:
    - Decompose `handle_container_exec_with_params` 
    - Add parallel tool calls
  • OpenTelemetry events (#2103)
    ### Title
    
    ## otel
    
    Codex can emit [OpenTelemetry](https://opentelemetry.io/) **log events**
    that
    describe each run: outbound API requests, streamed responses, user
    input,
    tool-approval decisions, and the result of every tool invocation. Export
    is
    **disabled by default** so local runs remain self-contained. Opt in by
    adding an
    `[otel]` table and choosing an exporter.
    
    ```toml
    [otel]
    environment = "staging"   # defaults to "dev"
    exporter = "none"          # defaults to "none"; set to otlp-http or otlp-grpc to send events
    log_user_prompt = false    # defaults to false; redact prompt text unless explicitly enabled
    ```
    
    Codex tags every exported event with `service.name = "codex-cli"`, the
    CLI
    version, and an `env` attribute so downstream collectors can distinguish
    dev/staging/prod traffic. Only telemetry produced inside the
    `codex_otel`
    crate—the events listed below—is forwarded to the exporter.
    
    ### Event catalog
    
    Every event shares a common set of metadata fields: `event.timestamp`,
    `conversation.id`, `app.version`, `auth_mode` (when available),
    `user.account_id` (when available), `terminal.type`, `model`, and
    `slug`.
    
    With OTEL enabled Codex emits the following event types (in addition to
    the
    metadata above):
    
    - `codex.api_request`
      - `cf_ray` (optional)
      - `attempt`
      - `duration_ms`
      - `http.response.status_code` (optional)
      - `error.message` (failures)
    - `codex.sse_event`
      - `event.kind`
      - `duration_ms`
      - `error.message` (failures)
      - `input_token_count` (completion only)
      - `output_token_count` (completion only)
      - `cached_token_count` (completion only, optional)
      - `reasoning_token_count` (completion only, optional)
      - `tool_token_count` (completion only)
    - `codex.user_prompt`
      - `prompt_length`
      - `prompt` (redacted unless `log_user_prompt = true`)
    - `codex.tool_decision`
      - `tool_name`
      - `call_id`
    - `decision` (`approved`, `approved_for_session`, `denied`, or `abort`)
      - `source` (`config` or `user`)
    - `codex.tool_result`
      - `tool_name`
      - `call_id`
      - `arguments`
      - `duration_ms` (execution time for the tool)
      - `success` (`"true"` or `"false"`)
      - `output`
    
    ### Choosing an exporter
    
    Set `otel.exporter` to control where events go:
    
    - `none` – leaves instrumentation active but skips exporting. This is
    the
      default.
    - `otlp-http` – posts OTLP log records to an OTLP/HTTP collector.
    Specify the
      endpoint, protocol, and headers your collector expects:
    
      ```toml
      [otel]
      exporter = { otlp-http = {
        endpoint = "https://otel.example.com/v1/logs",
        protocol = "binary",
        headers = { "x-otlp-api-key" = "${OTLP_TOKEN}" }
      }}
      ```
    
    - `otlp-grpc` – streams OTLP log records over gRPC. Provide the endpoint
    and any
      metadata headers:
    
      ```toml
      [otel]
      exporter = { otlp-grpc = {
        endpoint = "https://otel.example.com:4317",
        headers = { "x-otlp-meta" = "abc123" }
      }}
      ```
    
    If the exporter is `none` nothing is written anywhere; otherwise you
    must run or point to your
    own collector. All exporters run on a background batch worker that is
    flushed on
    shutdown.
    
    If you build Codex from source the OTEL crate is still behind an `otel`
    feature
    flag; the official prebuilt binaries ship with the feature enabled. When
    the
    feature is disabled the telemetry hooks become no-ops so the CLI
    continues to
    function without the extra dependencies.
    
    ---------
    
    Co-authored-by: Anton Panasenko <apanasenko@openai.com>
  • [MCP] Introduce an experimental official rust sdk based mcp client (#4252)
    The [official Rust
    SDK](https://github.com/modelcontextprotocol/rust-sdk/tree/57fc428c578a1a3fe851ee0838bf068bda120eb3)
    has come a long way since we first started our mcp client implementation
    5 months ago and, today, it is much more complete than our own
    stdio-only implementation.
    
    This PR introduces a new config flag `experimental_use_rmcp_client`
    which will use a new mcp client powered by the sdk instead of our own.
    
    To keep this PR simple, I've only implemented the same stdio MCP
    functionality that we had but will expand on it with future PRs.
    
    ---------
    
    Co-authored-by: pakrym-oai <pakrym@openai.com>
  • ref: state - 2 (#4229)
    Extracting tasks in a module and start abstraction behind a Trait (more
    to come on this but each task will be tackled in a dedicated PR)
    The goal was to drop the ActiveTask and to have a (potentially) set of
    tasks during each turn
  • Add exec output-schema parameter (#4079)
    Adds structured output to `exec` via the `--structured-output`
    parameter.
  • Add notifier tests (#4064)
    Proposal:
    1. Use anyhow for tests and avoid unwrap
    2. Extract a helper for starting a test instance of codex
  • enable-resume (#3537)
    Adding the ability to resume conversations.
    we have one verb `resume`. 
    
    Behavior:
    
    `tui`:
    `codex resume`: opens session picker
    `codex resume --last`: continue last message
    `codex resume <session id>`: continue conversation with `session id`
    
    `exec`:
    `codex resume --last`: continue last conversation
    `codex resume <session id>`: continue conversation with `session id`
    
    Implementation:
    - I added a function to find the path in `~/.codex/sessions/` with a
    `UUID`. This is helpful in resuming with session id.
    - Added the above mentioned flags
    - Added lots of testing
  • Handle resuming/forking after compact (#3533)
    We need to construct the history different when compact happens. For
    this, we need to just consider the history after compact and convert
    compact to a response item.
    
    This needs to change and use `build_compact_history` when this #3446 is
    merged.
  • Review Mode (Core) (#3401)
    ## 📝 Review Mode -- Core
    
    This PR introduces the Core implementation for Review mode:
    
    - New op `Op::Review { prompt: String }:` spawns a child review task
    with isolated context, a review‑specific system prompt, and a
    `Config.review_model`.
    - `EnteredReviewMode`: emitted when the child review session starts.
    Every event from this point onwards reflects the review session.
    - `ExitedReviewMode(Option<ReviewOutputEvent>)`: emitted when the review
    finishes or is interrupted, with optional structured findings:
    
    ```json
    {
      "findings": [
        {
          "title": "<≤ 80 chars, imperative>",
          "body": "<valid Markdown explaining *why* this is a problem; cite files/lines/functions>",
          "confidence_score": <float 0.0-1.0>,
          "priority": <int 0-3>,
          "code_location": {
            "absolute_file_path": "<file path>",
            "line_range": {"start": <int>, "end": <int>}
          }
        }
      ],
      "overall_correctness": "patch is correct" | "patch is incorrect",
      "overall_explanation": "<1-3 sentence explanation justifying the overall_correctness verdict>",
      "overall_confidence_score": <float 0.0-1.0>
    }
    ```
    
    ## Questions
    
    ### Why separate out its own message history?
    
    We want the review thread to match the training of our review models as
    much as possible -- that means using a custom prompt, removing user
    instructions, and starting a clean chat history.
    
    We also want to make sure the review thread doesn't leak into the parent
    thread.
    
    ### Why do this as a mode, vs. sub-agents?
    
    1. We want review to be a synchronous task, so it's fine for now to do a
    bespoke implementation.
    2. We're still unclear about the final structure for sub-agents. We'd
    prefer to land this quickly and then refactor into sub-agents without
    rushing that implementation.
  • bug: fix model save (#3525)
    Fix those 2 behaviors:
    1. The model does not get saved if we don't CTRL + S
    2. The reasoning effort get saved
  • chore: unify history loading (#2736)
    We have two ways of loading conversation with a previous history. Fork
    conversation and the experimental resume that we had before. In this PR,
    I am unifying their code path. The path is getting the history items and
    recording them in a brand new conversation. This PR also constraint the
    rollout recorder responsibilities to be only recording to the disk and
    loading from the disk.
    
    The PR also fixes a current bug when we have two forking in a row:
    History 1:
    <Environment Context>
    UserMessage_1
    UserMessage_2
    UserMessage_3
    
    **Fork with n = 1 (only remove one element)**
    History 2:
    <Environment Context>
    UserMessage_1
    UserMessage_2
    <Environment Context>
    
    **Fork with n = 1 (only remove one element)**
    History 2:
    <Environment Context>
    UserMessage_1
    UserMessage_2
    **<Environment Context>**
    
    This shouldn't happen but because we were appending the `<Environment
    Context>` after each spawning and it's considered as _user message_.
    Now, we don't add this message if restoring and old conversation.
  • test: faster test execution in codex-core (#2633)
    this dramatically improves time to run `cargo test -p codex-core` (~25x
    speedup).
    
    before:
    ```
    cargo test -p codex-core  35.96s user 68.63s system 19% cpu 8:49.80 total
    ```
    
    after:
    ```
    cargo test -p codex-core  5.51s user 8.16s system 63% cpu 21.407 total
    ```
    
    both tests measured "hot", i.e. on a 2nd run with no filesystem changes,
    to exclude compile times.
    
    approach inspired by [Delete Cargo Integration
    Tests](https://matklad.github.io/2021/02/27/delete-cargo-integration-tests.html),
    we move all test cases in tests/ into a single suite in order to have a
    single binary, as there is significant overhead for each test binary
    executed, and because test execution is only parallelized with a single
    binary.