Took over the work that @aaronl-openai started here:
https://github.com/openai/codex/pull/10397
Now that app-server clients are able to set up custom tools (called
`dynamic_tools` in app-server), we should expose a way for clients to
pass in not just text, but also image outputs. This is something the
Responses API already supports for function call outputs, where you can
pass in either a string or an array of content outputs (text, image,
file):
https://platform.openai.com/docs/api-reference/responses/create#responses_create-input-input_item_list-item-function_tool_call_output-output-array-input_image
So let's just plumb it through in Codex (with the caveat that we only
support text and image for now). This is implemented end-to-end across
app-server v2 protocol types and core tool handling.
## Breaking API change
NOTE: This introduces a breaking change with dynamic tools, but I think
it's ok since this concept was only recently introduced
(https://github.com/openai/codex/pull/9539) and it's better to get the
API contract correct. I don't think there are any real consumers of this
yet (not even the Codex App).
Old shape:
`{ "output": "dynamic-ok", "success": true }`
New shape:
```
{
"contentItems": [
{ "type": "inputText", "text": "dynamic-ok" },
{ "type": "inputImage", "imageUrl": "data:image/png;base64,AAA" }
]
"success": true
}
```
Similar to what @sayan-oai did in openai/codex#8956 for
`config.schema.json`, this PR updates the repo so that it includes the
output of `codex app-server generate-json-schema` and `codex app-server
generate-ts` and adds a test to verify it is in sync with the current
code.
Motivation:
- This makes any schema changes introduced by a PR transparent during
code review.
- In particular, this should help us catch PRs that would introduce a
non-backwards-compatible change to the app schema (eventually, this
should also be enforced by tooling).
- Once https://github.com/openai/codex/pull/10231 is in to formalize the
notion of "experimental" fields, we can work on ensuring the
non-experimental bits are backwards-compatible.
`codex-rs/app-server-protocol/tests/schema_fixtures.rs` was added as the
test and `just write-app-server-schema` can be use to generate the
vendored schema files.
Incidentally, when I run:
```
rg _ codex-rs/app-server-protocol/schema/typescript/v2
```
I see a number of `snake_case` names that should be `camelCase`.
Owen Lin ·
Michael Bolin ·