Preserve image detail in app-server inputs (#20693)

## Summary

- Add optional image detail to user image inputs across core, app-server
v2, thread history/event mapping, and the generated app-server
schemas/types.
- Preserve requested detail when serializing Responses image inputs:
omitted detail stays on the existing `high` default, while explicit
`original` keeps local images on the original-resolution path.
- Support `high`/`original` consistently for tool image outputs,
including MCP `codex/imageDetail`, code-mode image helpers, and
`view_image`.
This commit is contained in:
Curtis 'Fjord' Hawthorne
2026-05-15 15:04:04 -07:00
committed by GitHub
Unverified
parent 249d50aafc
commit 8543e39885
81 changed files with 1302 additions and 156 deletions
+1 -1
View File
@@ -24,7 +24,7 @@ const EXEC_DESCRIPTION_TEMPLATE: &str = r#"Run JavaScript code to orchestrate/co
- Global helpers:
- `exit()`: Immediately ends the current script successfully (like an early return from the top level).
- `text(value: string | number | boolean | undefined | null)`: Appends a text item. Non-string values are stringified with `JSON.stringify(...)` when possible.
- `image(imageUrlOrItem: string | { image_url: string; detail?: "auto" | "low" | "high" | "original" | null } | ImageContent, detail?: "auto" | "low" | "high" | "original" | null)`: Appends an image item. `image_url` can be an HTTPS URL or a base64-encoded `data:` URL. To forward an MCP tool image, pass an individual `ImageContent` block from `result.content`, for example `image(result.content[0])`. MCP image blocks may request detail with `_meta: { "codex/imageDetail": "original" }`. When provided, the second `detail` argument overrides any detail embedded in the first argument.
- `image(imageUrlOrItem: string | { image_url: string; detail?: "high" | "original" | null } | ImageContent, detail?: "high" | "original" | null)`: Appends an image item. `image_url` can be an HTTPS URL or a base64-encoded `data:` URL. To forward an MCP tool image, pass an individual `ImageContent` block from `result.content`, for example `image(result.content[0])`. MCP image blocks may request detail with `_meta: { "codex/imageDetail": "original" }`. When provided, the second `detail` argument overrides any detail embedded in the first argument.
- `store(key: string, value: any)`: stores a serializable value under a string key for later `exec` calls in the same session.
- `load(key: string)`: returns the stored value for a string key, or `undefined` if it is missing.
- `notify(value: string | number | boolean | undefined | null)`: immediately injects an extra `custom_tool_call_output` for the current `exec` call. Values are stringified like `text(...)`.
-2
View File
@@ -4,8 +4,6 @@ use serde::Serialize;
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
#[serde(rename_all = "lowercase")]
pub enum ImageDetail {
Auto,
Low,
High,
Original,
}
+2 -6
View File
@@ -71,14 +71,10 @@ pub(super) fn normalize_output_image(
Some(detail) => {
let normalized = detail.to_ascii_lowercase();
Some(match normalized.as_str() {
"auto" => ImageDetail::Auto,
"low" => ImageDetail::Low,
"high" => ImageDetail::High,
"original" => ImageDetail::Original,
_ => {
return Err(
"image detail must be one of: auto, low, high, original".to_string()
);
return Err("image detail must be one of: high, original".to_string());
}
})
}
@@ -160,7 +156,7 @@ fn parse_mcp_output_image(
.and_then(JsonValue::as_object)
.and_then(|meta| meta.get(CODEX_IMAGE_DETAIL_META_KEY))
.and_then(JsonValue::as_str)
.filter(|detail| matches!(*detail, "auto" | "low" | "high" | "original"))
.filter(|detail| matches!(*detail, "high" | "original"))
.map(str::to_string);
Ok((image_url, detail))
}
+33 -3
View File
@@ -1308,7 +1308,7 @@ image({
image(
{
image_url: "https://example.com/image.jpg",
detail: "low",
detail: "high",
},
"original",
);
@@ -1348,7 +1348,7 @@ image(
mimeType: "image/png",
_meta: { "codex/imageDetail": "original" },
},
"low",
"high",
);
"#
.to_string(),
@@ -1364,7 +1364,7 @@ image(
cell_id: "1".to_string(),
content_items: vec![FunctionCallOutputContentItem::InputImage {
image_url: "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR4nGP4z8DwHwAFAAH/iZk9HQAAAABJRU5ErkJggg==".to_string(),
detail: Some(crate::ImageDetail::Low),
detail: Some(crate::ImageDetail::High),
}],
stored_values: HashMap::new(),
error_text: None,
@@ -1372,6 +1372,36 @@ image(
);
}
#[tokio::test]
async fn image_helper_rejects_unsupported_detail() {
let service = CodeModeService::new();
let response = service
.execute(ExecuteRequest {
source: r#"
image({
image_url: "https://example.com/image.jpg",
detail: "low",
});
"#
.to_string(),
yield_time_ms: None,
..execute_request("")
})
.await
.unwrap();
assert_eq!(
response,
RuntimeResponse::Result {
cell_id: "1".to_string(),
content_items: Vec::new(),
stored_values: HashMap::new(),
error_text: Some("image detail must be one of: high, original".to_string()),
}
);
}
#[tokio::test]
async fn image_helper_rejects_raw_mcp_result_container() {
let service = CodeModeService::new();