Always enable original image detail on supported models (#17665)

## Summary

This PR removes `image_detail_original` as a runtime experiment and
makes original image detail available whenever the selected model
supports it.

Concretely, this change:
- drops the `image_detail_original` feature flag from the feature
registry and generated config schema
- makes tool-emitted image detail depend only on
`ModelInfo.supports_image_detail_original`
- updates `view_image` and `code_mode`/`js_repl` image emission to use
that capability check directly
- removes now-redundant experiment-specific tests and instruction
coverage
- keeps backward compatibility for existing configs by silently ignoring
a stale `features.image_detail_original` entry

The net effect is that `detail: "original"` is always available on
supported models, without requiring an experiment toggle.
This commit is contained in:
Curtis 'Fjord' Hawthorne
2026-04-14 08:15:56 -07:00
committed by GitHub
parent e6947f85f6
commit f030ab62eb
15 changed files with 48 additions and 222 deletions
-19
View File
@@ -228,25 +228,6 @@ async fn js_repl_tools_only_instructions_are_feature_gated() {
assert_eq!(res, expected);
}
#[tokio::test]
async fn js_repl_image_detail_original_does_not_change_instructions() {
let tmp = tempfile::tempdir().expect("tempdir");
let mut cfg = make_config(&tmp, /*limit*/ 4096, /*instructions*/ None).await;
let mut features = cfg.features.get().clone();
features
.enable(Feature::JsRepl)
.enable(Feature::ImageDetailOriginal);
cfg.features
.set(features)
.expect("test config should allow js_repl image detail settings");
let res = get_user_instructions(&cfg)
.await
.expect("js_repl instructions expected");
let expected = "## JavaScript REPL (Node)\n- Use `js_repl` for Node-backed JavaScript with top-level await in a persistent kernel.\n- `js_repl` is a freeform/custom tool. Direct `js_repl` calls must send raw JavaScript tool input (optionally with first-line `// codex-js-repl: timeout_ms=15000`). Do not wrap code in JSON (for example `{\"code\":\"...\"}`), quotes, or markdown code fences.\n- Helpers: `codex.cwd`, `codex.homeDir`, `codex.tmpDir`, `codex.tool(name, args?)`, and `codex.emitImage(imageLike)`.\n- `codex.tool` executes a normal tool call and resolves to the raw tool output object. Use it for shell and non-shell tools alike. Nested tool outputs stay inside JavaScript unless you emit them explicitly.\n- `codex.emitImage(...)` adds one image to the outer `js_repl` function output each time you call it, so you can call it multiple times to emit multiple images. It accepts a data URL, a single `input_image` item, an object like `{ bytes, mimeType }`, or a raw tool response object with exactly one image and no text. It rejects mixed text-and-image content.\n- `codex.tool(...)` and `codex.emitImage(...)` keep stable helper identities across cells. Saved references and persisted objects can reuse them in later cells, but async callbacks that fire after a cell finishes still fail because no exec is active.\n- Request full-resolution image processing with `detail: \"original\"` only when the `view_image` tool schema includes a `detail` argument. The same availability applies to `codex.emitImage(...)`: if `view_image.detail` is present, you may also pass `detail: \"original\"` there. Use this when high-fidelity image perception or precise localization is needed, especially for CUA agents.\n- Example of sharing an in-memory Playwright screenshot: `await codex.emitImage({ bytes: await page.screenshot({ type: \"jpeg\", quality: 85 }), mimeType: \"image/jpeg\", detail: \"original\" })`.\n- Example of sharing a local image tool result: `await codex.emitImage(codex.tool(\"view_image\", { path: \"/absolute/path\", detail: \"original\" }))`.\n- When encoding an image to send with `codex.emitImage(...)` or `view_image`, prefer JPEG at about 85 quality when lossy compression is acceptable; use PNG when transparency or lossless detail matters. Smaller uploads are faster and less likely to hit size limits.\n- Top-level bindings persist across cells. If a cell throws, prior bindings remain available and bindings that finished initializing before the throw often remain usable in later cells. For code you plan to reuse across cells, prefer declaring or assigning it in direct top-level statements before operations that might throw. If you hit `SyntaxError: Identifier 'x' has already been declared`, first reuse the existing binding, reassign a previously declared `let`, or pick a new descriptive name. Use `{ ... }` only for a short temporary block when you specifically need local scratch names; do not wrap an entire cell in block scope if you want those names reusable later. Reset the kernel with `js_repl_reset` only when you need a clean state.\n- Top-level static import declarations (for example `import x from \"./file.js\"`) are currently unsupported in `js_repl`; use dynamic imports with `await import(\"pkg\")`, `await import(\"./file.js\")`, or `await import(\"/abs/path/file.mjs\")` instead. Imported local files must be ESM `.js`/`.mjs` files and run in the same REPL VM context. Bare package imports always resolve from REPL-global search roots (`CODEX_JS_REPL_NODE_MODULE_DIRS`, then cwd), not relative to the imported file location. Local files may statically import only other local relative/absolute/`file://` `.js`/`.mjs` files; package and builtin imports from local files must stay dynamic. `import.meta.resolve()` returns importable strings such as `file://...`, bare package names, and `node:...` specifiers. Local file modules reload between execs, while top-level bindings persist until `js_repl_reset`.\n- Avoid direct access to `process.stdout` / `process.stderr` / `process.stdin`; it can corrupt the JSON line protocol. Use `console.log`, `codex.tool(...)`, and `codex.emitImage(...)`.";
assert_eq!(res, expected);
}
/// When both system instructions *and* a project doc are present the two
/// should be concatenated with the separator.
#[tokio::test]
@@ -125,8 +125,7 @@ impl ToolHandler for ViewImageHandler {
})?;
let event_path = abs_path.to_path_buf();
let can_request_original_detail =
can_request_original_image_detail(turn.features.get(), &turn.model_info);
let can_request_original_detail = can_request_original_image_detail(&turn.model_info);
let use_original_detail =
can_request_original_detail && matches!(detail, Some(ViewImageDetail::Original));
let image_mode = if use_original_detail {
+1 -1
View File
@@ -1714,7 +1714,7 @@ fn emitted_image_content_item(
) -> FunctionCallOutputContentItem {
FunctionCallOutputContentItem::InputImage {
image_url,
detail: normalize_output_image_detail(turn.features.get(), &turn.model_info, detail),
detail: normalize_output_image_detail(&turn.model_info, detail),
}
}
+2 -37
View File
@@ -2,7 +2,6 @@ use super::*;
use crate::codex::make_session_and_context;
use crate::codex::make_session_and_context_with_dynamic_tools_and_rx;
use crate::turn_diff_tracker::TurnDiffTracker;
use codex_features::Feature;
use codex_protocol::dynamic_tools::DynamicToolCallOutputContentItem;
use codex_protocol::dynamic_tools::DynamicToolResponse;
use codex_protocol::dynamic_tools::DynamicToolSpec;
@@ -295,42 +294,8 @@ async fn emitted_image_content_item_drops_unsupported_explicit_detail() {
}
#[tokio::test]
async fn emitted_image_content_item_does_not_force_original_when_enabled() {
async fn emitted_image_content_item_allows_explicit_original_detail_when_supported() {
let (_session, mut turn) = make_session_and_context().await;
Arc::make_mut(&mut turn.config)
.features
.enable(Feature::ImageDetailOriginal)
.expect("test config should allow feature update");
turn.features
.enable(Feature::ImageDetailOriginal)
.expect("test turn features should allow feature update");
turn.model_info.supports_image_detail_original = true;
let content_item = emitted_image_content_item(
&turn,
"data:image/png;base64,AAA".to_string(),
/*detail*/ None,
);
assert_eq!(
content_item,
FunctionCallOutputContentItem::InputImage {
image_url: "data:image/png;base64,AAA".to_string(),
detail: None,
}
);
}
#[tokio::test]
async fn emitted_image_content_item_allows_explicit_original_detail_when_enabled() {
let (_session, mut turn) = make_session_and_context().await;
Arc::make_mut(&mut turn.config)
.features
.enable(Feature::ImageDetailOriginal)
.expect("test config should allow feature update");
turn.features
.enable(Feature::ImageDetailOriginal)
.expect("test turn features should allow feature update");
turn.model_info.supports_image_detail_original = true;
let content_item = emitted_image_content_item(
@@ -349,7 +314,7 @@ async fn emitted_image_content_item_allows_explicit_original_detail_when_enabled
}
#[tokio::test]
async fn emitted_image_content_item_drops_explicit_original_detail_when_disabled() {
async fn emitted_image_content_item_drops_explicit_original_detail_when_unsupported() {
let (_session, turn) = make_session_and_context().await;
let content_item = emitted_image_content_item(