Commit Graph

42 Commits

  • Fixed a hole in token refresh logic for app server (#11802)
    We've continued to receive reports from users that they're seeing the
    error message "Your access token could not be refreshed because your
    refresh token was already used. Please log out and sign in again." This
    PR fixes two holes in the token refresh logic that lead to this
    condition.
    
    Background: A previous change in token refresh introduced the
    `UnauthorizedRecovery` object. It implements a state machine in the core
    agent loop that first performs a load of the on-disk auth information
    guarded by a check for matching account ID. If it finds that the on-disk
    version has been updated by another instance of codex, it uses the
    reloaded auth tokens. If the on-disk version hasn't been updated, it
    issues a refresh request from the token authority.
    
    There are two problems that this PR addresses:
    
    Problem 1: We weren't doing the same thing for the code path used by the
    app server interface. This PR effectively replicates the
    `UnauthorizedRecovery` logic for that code path.
    
    Problem 2: The `UnauthorizedRecovery` logic contained a hole in the
    `ReloadOutcome::Skipped` case. Here's the scenario. A user starts two
    instances of the CLI. Instance 1 is active (working on a task), instance
    2 is idle. Both instances have the same in-memory cached tokens. The
    user then runs `codex logout` or `codex login` to log in to a separate
    account, which overwrites the `auth.json` file. Instance 1 receives a
    401 and refreshes its token, but it doesn't write the new token to the
    `auth.json` file because the account ID doesn't match. Instance 2 is
    later activated and presented with a new task. It immediately hits a 401
    and attempts to refresh its token but fails because its cached refresh
    token is now invalid. To avoid this situation, I've changed the logic to
    immediately fail a token refresh if the user has since logged out or
    logged in to another account. This will still be seen as an error by the
    user, but the cause will be clearer.
    
    I also took this opportunity to clean up the names of existing functions
    to make their roles clearer.
    * `try_refresh_token` is renamed `request_chatgpt_token_refresh`
    * the existing `refresh_token` is renamed `refresh_token_from_authority`
    (there's a new higher-level function named `refresh_token` now)
    * `refresh_tokens` is renamed `refresh_and_persist_chatgpt_token`, and
    it now implicitly reloads
    * `update_tokens` is renamed `persist_tokens`
  • Remove test-support feature from codex-core and replace it with explicit test toggles (#11405)
    ## Why
    
    `codex-core` was being built in multiple feature-resolved permutations
    because test-only behavior was modeled as crate features. For a large
    crate, those permutations increase compile cost and reduce cache reuse.
    
    ## Net Change
    
    - Removed the `test-support` crate feature and related feature wiring so
    `codex-core` no longer needs separate feature shapes for test consumers.
    - Standardized cross-crate test-only access behind
    `codex_core::test_support`.
    - External test code now imports helpers from
    `codex_core::test_support`.
    - Underlying implementation hooks are kept internal (`pub(crate)`)
    instead of broadly public.
    
    ## Outcome
    
    - Fewer `codex-core` build permutations.
    - Better incremental cache reuse across test targets.
    - No intended production behavior change.
  • fix(app-server): for external auth, replace id_token with chatgpt_acc… (#11240)
    …ount_id and chatgpt_plan_type
    
    ### Summary
    Following up on external auth mode which was introduced here:
    https://github.com/openai/codex/pull/10012
    
    Turns out some clients have a differently shaped ID token and don't have
    a chosen workspace (aka chatgpt_account_id) encoded in their ID token.
    So, let's replace `id_token` param with `chatgpt_account_id` and
    `chatgpt_plan_type` (optional) when initializing the external ChatGPT
    auth mode (`account/login/start` with `chatgptAuthTokens`).
    
    The client was able to test end-to-end with a Codex build from this
    branch and verified it worked!
  • fix(auth): isolate chatgptAuthTokens concept to auth manager and app-server (#10423)
    So that the rest of the codebase (like TUI) don't need to be concerned
    whether ChatGPT auth was handled by Codex itself or passed in via
    app-server's external auth mode.
  • chore: rename ChatGpt -> Chatgpt in type names (#10244)
    When using ChatGPT in names of types, we should be consistent, so this
    renames some types with `ChatGpt` in the name to `Chatgpt`. From
    https://rust-lang.github.io/api-guidelines/naming.html:
    
    > In `UpperCamelCase`, acronyms and contractions of compound words count
    as one word: use `Uuid` rather than `UUID`, `Usize` rather than `USize`
    or `Stdin` rather than `StdIn`. In `snake_case`, acronyms and
    contractions are lower-cased: `is_xid_start`.
    
    This PR updates existing uses of `ChatGpt` and changes them to
    `Chatgpt`. Though in all cases where it could affect the wire format, I
    visually inspected that we don't change anything there. That said, this
    _will_ change the codegen because it will affect the spelling of type
    names.
    
    For example, this renames `AuthMode::ChatGPT` to `AuthMode::Chatgpt` in
    `app-server-protocol`, but the wire format is still `"chatgpt"`.
    
    This PR also updates a number of types in `codex-rs/core/src/auth.rs`.
  • feat: refactor CodexAuth so invalid state cannot be represented (#10208)
    Previously, `CodexAuth` was defined as follows:
    
    
    https://github.com/openai/codex/blob/d550fbf41afc09d7d7b5ac813aea38de07b2a73f/codex-rs/core/src/auth.rs#L39-L46
    
    But if you looked at its constructors, we had creation for
    `AuthMode::ApiKey` where `storage` was built using a nonsensical path
    (`PathBuf::new()`) and `auth_dot_json` was `None`:
    
    
    https://github.com/openai/codex/blob/d550fbf41afc09d7d7b5ac813aea38de07b2a73f/codex-rs/core/src/auth.rs#L212-L220
    
    By comparison, when `AuthMode::ChatGPT` was used, `api_key` was always
    `None`:
    
    
    https://github.com/openai/codex/blob/d550fbf41afc09d7d7b5ac813aea38de07b2a73f/codex-rs/core/src/auth.rs#L665-L671
    
    https://github.com/openai/codex/pull/10012 took things further because
    it introduced a new `ChatgptAuthTokens` variant to `AuthMode`, which is
    important in when invoking `account/login/start` via the app server, but
    most logic _internal_ to the app server should just reason about two
    `AuthMode` variants: `ApiKey` and `ChatGPT`.
    
    This PR tries to clean things up as follows:
    
    - `LoginAccountParams` and `AuthMode` in `codex-rs/app-server-protocol/`
    both continue to have the `ChatgptAuthTokens` variant, though it is used
    exclusively for the on-the-wire messaging.
    - `codex-rs/core/src/auth.rs` now has its own `AuthMode` enum, which
    only has two variants: `ApiKey` and `ChatGPT`.
    - `CodexAuth` has been changed from a struct to an enum. It is a
    disjoint union where each variant (`ApiKey`, `ChatGpt`, and
    `ChatGptAuthTokens`) have only the associated fields that make sense for
    that variant.
    
    ---
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/10208).
    * #10224
    * __->__ #10208
  • add error messages for the go plan type (#10181)
    Adds support for the Go plan type
    Updates rate limit error messages to point to the usage page
  • feat(app-server): support external auth mode (#10012)
    This enables a new use case where `codex app-server` is embedded into a
    parent application that will directly own the user's ChatGPT auth
    lifecycle, which means it owns the user’s auth tokens and refreshes it
    when necessary. The parent application would just want a way to pass in
    the auth tokens for codex to use directly.
    
    The idea is that we are introducing a new "auth mode" currently only
    exposed via app server: **`chatgptAuthTokens`** which consist of the
    `id_token` (stores account metadata) and `access_token` (the bearer
    token used directly for backend API calls). These auth tokens are only
    stored in-memory. This new mode is in addition to the existing `apiKey`
    and `chatgpt` auth modes.
    
    This PR reuses the shape of our existing app-server account APIs as much
    as possible:
    - Update `account/login/start` with a new `chatgptAuthTokens` variant,
    which will allow the client to pass in the tokens and have codex
    app-server use them directly. Upon success, the server emits
    `account/login/completed` and `account/updated` notifications.
    - A new server->client request called
    `account/chatgptAuthTokens/refresh` which the server can use whenever
    the access token previously passed in has expired and it needs a new one
    from the parent application.
    
    I leveraged the core 401 retry loop which typically triggers auth token
    refreshes automatically, but made it pluggable:
    - **chatgpt** mode refreshes internally, as usual.
    - **chatgptAuthTokens** mode calls the client via
    `account/chatgptAuthTokens/refresh`, the client responds with updated
    tokens, codex updates its in-memory auth, then retries. This RPC has a
    10s timeout and handles JSON-RPC errors from the client.
    
    Also some additional things:
    - chatgpt logins are blocked while external auth is active (have to log
    out first. typically clients will pick one OR the other, not support
    both)
    - `account/logout` clears external auth in memory
    - Ensures that if `forced_chatgpt_workspace_id` is set via the user's
    config, we respect it in both:
    - `account/login/start` with `chatgptAuthTokens` (returns a JSON-RPC
    error back to the client)
    - `account/chatgptAuthTokens/refresh` (fails the turn, and on next
    request app-server will send another `account/chatgptAuthTokens/refresh`
    request to the client).
  • [connectors] Support connectors part 1 - App server & MCP (#9667)
    In order to make Codex work with connectors, we add a built-in gateway
    MCP that acts as a transparent proxy between the client and the
    connectors. The gateway MCP collects actions that are accessible to the
    user and sends them down to the user, when a connector action is chosen
    to be called, the client invokes the action through the gateway MCP as
    well.
    
     - [x] Add the system built-in gateway MCP to list and run connectors.
     - [x] Add the app server methods and protocol
  • Attempt to reload auth as a step in 401 recovery (#8880)
    When authentication fails, first attempt to reload the auth from file
    and then attempt to refresh it.
  • Immutable CodexAuth (#8857)
    Historically we started with a CodexAuth that knew how to refresh it's
    own tokens and then added AuthManager that did a different kind of
    refresh (re-reading from disk).
    
    I don't think it makes sense for both `CodexAuth` and `AuthManager` to
    be mutable and contain behaviors.
    
    Move all refresh logic into `AuthManager` and keep `CodexAuth` as a data
    object.
  • Move tests below auth manager (#8840)
    To simplify future diffs
  • chore: migrate from Config::load_from_base_config_with_overrides to ConfigBuilder (#8276)
    https://github.com/openai/codex/pull/8235 introduced `ConfigBuilder` and
    this PR updates all call non-test call sites to use it instead of
    `Config::load_from_base_config_with_overrides()`.
    
    This is important because `load_from_base_config_with_overrides()` uses
    an empty `ConfigRequirements`, which is a reasonable default for testing
    so the tests are not influenced by the settings on the host. This method
    is now guarded by `#[cfg(test)]` so it cannot be used by business logic.
    
    Because `ConfigBuilder::build()` is `async`, many of the test methods
    had to be migrated to be `async`, as well. On the bright side, this made
    it possible to eliminate a bunch of `block_on_future()` stuff.
  • Reimplement skills loading using SkillsManager + skills/list op. (#7914)
    refactor the way we load and manage skills:
    1. Move skill discovery/caching into SkillsManager and reuse it across
    sessions.
    2. Add the skills/list API (Op::ListSkills/SkillsListResponse) to fetch
    skills for one or more cwds. Also update app-server for VSCE/App;
    3. Trigger skills/list during session startup so UIs preload skills and
    handle errors immediately.
  • load models from disk and set a ttl and etag (#7722)
    # External (non-OpenAI) Pull Request Requirements
    
    Before opening this Pull Request, please read the dedicated
    "Contributing" markdown file or your PR may be closed:
    https://github.com/openai/codex/blob/main/docs/contributing.md
    
    If your PR conforms to our contribution guidelines, replace this text
    with a detailed and high quality description of your changes.
    
    Include a link to a bug report or enhancement request.
  • fix: taking plan type from usage endpoint instead of thru auth token (#7610)
    pull plan type from the usage endpoint, persist it in session state /
    tui state, and propagate through rate limit snapshots
  • Fixed status output to use auth information from AuthManager (#6529)
    This PR addresses https://github.com/openai/codex/issues/6360. The root
    problem is that the TUI was directly loading the `auth.json` file to
    access the auth information. It should instead be using the AuthManager,
    which records the current auth information. The `auth.json` file can be
    overwritten at any time by other instances of the CLI or extension, so
    its information can be out of sync with the current instance. The
    `/status` command should always report the auth information associated
    with the current instance.
    
    An alternative fix for this bug was submitted by @chojs23 in [this
    PR](https://github.com/openai/codex/pull/6495). That approach was only a
    partial fix.
  • [App-server] Implement account/read endpoint (#6336)
    This PR does two things:
    1. add a new function in core that maps the core-internal plan type to
    the external plan type;
    2. implement account/read that get account status (v2 of
    `getAuthStatus`).
  • Improved token refresh handling to address "Re-connecting" behavior (#6231)
    Currently, when the access token expires, we attempt to use the refresh
    token to acquire a new access token. This works most of the time.
    However, there are situations where the refresh token is expired,
    exhausted (already used to perform a refresh), or revoked. In those
    cases, the current logic treats the error as transient and attempts to
    retry it repeatedly.
    
    This PR changes the token refresh logic to differentiate between
    permanent and transient errors. It also changes callers to treat the
    permanent errors as fatal rather than retrying them. And it provides
    better error messages to users so they understand how to address the
    problem. These error messages should also help us further understand why
    we're seeing examples of refresh token exhaustion.
    
    Here is the error message in the CLI. The same text appears within the
    extension.
    
    <img width="863" height="38" alt="image"
    src="https://github.com/user-attachments/assets/7ffc0d08-ebf0-4900-b9a9-265064202f4f"
    />
    
    I also correct the spelling of "Re-connecting", which shouldn't have a
    hyphen in it.
    
    Testing: I manually tested these code paths by adding temporary code to
    programmatically cause my refresh token to be exhausted (by calling the
    token refresh endpoint in a tight loop more than 50 times). I then
    simulated an access token expiration, which caused the token refresh
    logic to be invoked. I confirmed that the updated logic properly handled
    the error condition.
    
    Note: We earlier discussed the idea of forcefully logging out the user
    at the point where token refresh failed. I made several attempts to do
    this, and all of them resulted in a bad UX. It's important to surface
    this error to users in a way that explains the problem and tells them
    that they need to log in again. We also previously discussed deleting
    the auth.json file when this condition is detected. That also creates
    problems because it effectively changes the auth status from logged in
    to logged out, and this causes odd failures and inconsistent UX. I think
    it's therefore better not to delete auth.json in this case. If the user
    closes the CLI or VSCE and starts it again, we properly detect that the
    access token is expired and the refresh token is "dead", and we force
    the user to go through the login flow at that time.
    
    This should address aspects of #6191, #5679, and #5505
  • [Auth] Choose which auth storage to use based on config (#5792)
    This PR is a follow-up to #5591. It allows users to choose which auth
    storage mode they want by using the new
    `cli_auth_credentials_store_mode` config.
  • [Auth] Introduce New Auth Storage Abstraction for Codex CLI (#5569)
    This PR introduces a new `Auth Storage` abstraction layer that takes
    care of read, write, and load of auth tokens based on the
    AuthCredentialsStoreMode. It is similar to how we handle MCP client
    oauth
    [here](https://github.com/openai/codex/blob/main/codex-rs/rmcp-client/src/oauth.rs).
    Instead of reading and writing directly from disk for auth tokens, Codex
    CLI workflows now should instead use this auth storage using the public
    helper functions.
    
    This PR is just a refactor of the current code so the behavior stays the
    same. We will add support for keyring and hybrid mode in follow-up PRs.
    
    I have read the CLA Document and I hereby sign the CLA
  • Made token refresh code resilient to missing id_token (#5782)
    This PR does the following:
    1. Changes `try_refresh_token` to handle the case where the endpoint
    returns a response without an `id_token`. The OpenID spec indicates that
    this field is optional and clients should not assume it's present.
    2. Changes the `attempt_stream_responses` to propagate token refresh
    errors rather than silently ignoring them.
    3. Fixes a typo in a couple of error messages (unrelated to the above,
    but something I noticed in passing) - "reconnect" should be spelled
    without a hyphen.
    
    This PR does not implement the additional suggestion from @pakrym-oai
    that we should sign out when receiving `refresh_token_expired` from the
    refresh endpoint. Leaving this as a follow-on because I'm undecided on
    whether this should be implemented in `try_refresh_token` or its
    callers.
  • Log more types of request IDs (#5645)
    Different services return different sets of IDs, log all of them to
    simplify debugging.
  • Add CodexHttpClient wrapper with request logging (#5564)
    ## Summary
    - wrap the default reqwest::Client inside a new
    CodexHttpClient/CodexRequestBuilder pair and log the HTTP method, URL,
    and status for each request
    - update the auth/model/provider plumbing to use the new builder helpers
    so headers and bearer auth continue to be applied consistently
    - add the shared `http` dependency that backs the header conversion
    helpers
    
    ## Testing
    - `CODEX_SANDBOX=seatbelt CODEX_SANDBOX_NETWORK_DISABLED=1 cargo test -p
    codex-core`
    - `CODEX_SANDBOX=seatbelt CODEX_SANDBOX_NETWORK_DISABLED=1 cargo test -p
    codex-chatgpt`
    - `CODEX_SANDBOX=seatbelt CODEX_SANDBOX_NETWORK_DISABLED=1 cargo test -p
    codex-tui`
    
    ------
    https://chatgpt.com/codex/tasks/task_i_68fa5038c17483208b1148661c5873be
  • Fix flaky auth tests (#5461)
    This #[serial] approach is not ideal. I am tracking a separate issue to
    create an injectable env var provider but I want to fix these tests
    first.
    
    Fixes #5447
  • Add forced_chatgpt_workspace_id and forced_login_method configuration options (#5303)
    This PR adds support for configs to specify a forced login method
    (chatgpt or api) as well as a forced chatgpt account id. This lets
    enterprises uses [managed
    configs](https://developers.openai.com/codex/security#managed-configuration)
    to force all employees to use their company's workspace instead of their
    own or any other.
    
    When a workspace id is set, a query param is sent to the login flow
    which auto-selects the given workspace or errors if the user isn't a
    member of it.
    
    This PR is large but a large % of it is tests, wiring, and required
    formatting changes.
    
    API login with chatgpt forced
    <img width="1592" height="116" alt="CleanShot 2025-10-19 at 22 40 04"
    src="https://github.com/user-attachments/assets/560c6bb4-a20a-4a37-95af-93df39d057dd"
    />
    
    ChatGPT login with api forced
    <img width="1018" height="100" alt="CleanShot 2025-10-19 at 22 40 29"
    src="https://github.com/user-attachments/assets/d010bbbb-9c8d-4227-9eda-e55bf043b4af"
    />
    
    Onboarding with api forced
    <img width="892" height="460" alt="CleanShot 2025-10-19 at 22 41 02"
    src="https://github.com/user-attachments/assets/cc0ed45c-b257-4d62-a32e-6ca7514b5edd"
    />
    
    Onboarding with ChatGPT forced
    <img width="1154" height="426" alt="CleanShot 2025-10-19 at 22 41 27"
    src="https://github.com/user-attachments/assets/41c41417-dc68-4bb4-b3e7-3b7769f7e6a1"
    />
    
    Logging in with the wrong workspace
    <img width="2222" height="84" alt="CleanShot 2025-10-19 at 22 42 31"
    src="https://github.com/user-attachments/assets/0ff4222c-f626-4dd3-b035-0b7fe998a046"
    />
  • [codex][otel] propagate user email in otel events (#5223)
    include user email into otel events for proper user-level attribution in
    case of workspace setup
  • Support CODEX_API_KEY for codex exec (#4615)
    Allows to set API key per invocation of `codex exec`
  • fix: remove mcp-types from app server protocol (#4537)
    We continue the separation between `codex app-server` and `codex
    mcp-server`.
    
    In particular, we introduce a new crate, `codex-app-server-protocol`,
    and migrate `codex-rs/protocol/src/mcp_protocol.rs` into it, renaming it
    `codex-rs/app-server-protocol/src/protocol.rs`.
    
    Because `ConversationId` was defined in `mcp_protocol.rs`, we move it
    into its own file, `codex-rs/protocol/src/conversation_id.rs`, and
    because it is referenced in a ton of places, we have to touch a lot of
    files as part of this PR.
    
    We also decide to get away from proper JSON-RPC 2.0 semantics, so we
    also introduce `codex-rs/app-server-protocol/src/jsonrpc_lite.rs`, which
    is basically the same `JSONRPCMessage` type defined in `mcp-types`
    except with all of the `"jsonrpc": "2.0"` removed.
    
    Getting rid of `"jsonrpc": "2.0"` makes our serialization logic
    considerably simpler, as we can lean heavier on serde to serialize
    directly into the wire format that we use now.
  • Fixed login failure with API key in IDE extension when a .codex directory doesn't exist (#4258)
    This addresses bug #4092
    
    Testing:
    * Confirmed error occurs prior to fix if logging in using API key and no
    `~/.codex` directory exists
    * Confirmed after fix that `~/.codex` directory is properly created and
    error doesn't occur
  • When logging in using ChatGPT, make sure to overwrite API key (#3611)
    When logging in using ChatGPT using the `codex login` command, a
    successful login should write a new `auth.json` file with the ChatGPT
    token information. The old code attempted to retain the API key and
    merge the token information into the existing `auth.json` file. With the
    new simplified login mechanism, `auth.json` should have auth information
    for only ChatGPT or API Key, not both.
    
    The `codex login --api-key <key>` code path was already doing the right
    thing here, but the `codex login` command was incorrect. This PR fixes
    the problem and adds test cases for both commands.
  • Use PlanType enum when formatting usage-limit CTA (#3495)
    - Started using Play type struct
    - Added CTA for team/business 
    - Refactored a bit to unify the logic
  • chore: enable clippy::redundant_clone (#3489)
    Created this PR by:
    
    - adding `redundant_clone` to `[workspace.lints.clippy]` in
    `cargo-rs/Cargol.toml`
    - running `cargo clippy --tests --fix`
    - running `just fmt`
    
    Though I had to clean up one instance of the following that resulted:
    
    ```rust
    let codex = codex;
    ```
  • Simplify auth flow and reconcile differences between ChatGPT and API Key auth (#3189)
    This PR does the following:
    * Adds the ability to paste or type an API key.
    * Removes the `preferred_auth_method` config option. The last login
    method is always persisted in auth.json, so this isn't needed.
    * If OPENAI_API_KEY env variable is defined, the value is used to
    prepopulate the new UI. The env variable is otherwise ignored by the
    CLI.
    * Adds a new MCP server entry point "login_api_key" so we can implement
    this same API key behavior for the VS Code extension.
    <img width="473" height="140" alt="Screenshot 2025-09-04 at 3 51 04 PM"
    src="https://github.com/user-attachments/assets/c11bbd5b-8a4d-4d71-90fd-34130460f9d9"
    />
    <img width="726" height="254" alt="Screenshot 2025-09-04 at 3 51 32 PM"
    src="https://github.com/user-attachments/assets/6cc76b34-309a-4387-acbc-15ee5c756db9"
    />
  • Replace config.responses_originator_header_internal_override with CODEX_INTERNAL_ORIGINATOR_OVERRIDE_ENV_VAR (#3388)
    The previous config approach had a few issues:
    1. It is part of the config but not designed to be used externally
    2. It had to be wired through many places (look at the +/- on this PR
    3. It wasn't guaranteed to be set consistently everywhere because we
    don't have a super well defined way that configs stack. For example, the
    extension would configure during newConversation but anything that
    happened outside of that (like login) wouldn't get it.
    
    This env var approach is cleaner and also creates one less thing we have
    to deal with when coming up with a better holistic story around configs.
    
    One downside is that I removed the unit test testing for the override
    because I don't want to deal with setting the global env or spawning
    child processes and figuring out how to introspect their originator
    header. The new code is sufficiently simple and I tested it e2e that I
    feel as if this is still worth it.
  • Add a common way to create HTTP client (#3110)
    Ensure User-Agent and originator are always sent.
  • Move CodexAuth and AuthManager to the core crate (#3074)
    Fix a long standing layering issue.