mirror of
https://github.com/microsoft/agent-framework.git
synced 2026-06-16 21:04:09 +08:00
Python: Unify tool results as Content items with rich content support (#4331)
* feat(python): allow @tool functions to return rich content (images, audio) Add support for tool functions to return Content objects that the model can perceive natively. Closes #4272 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Anthropic logging + mypy fix * Address PR review: fix MCP ordering, fold helper into from_function_result, fix Chat client - Preserve original content order in MCP tool results instead of text-first - Move _build_function_result logic into Content.from_function_result() - Chat Completions: inject user message for rich items (API only supports string tool content) - Update tests for ordering and new from_function_result behavior Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use native Responses API multi-part output, warn+omit for Chat client - Responses client: put rich items directly in function_call_output's output field as list (native API support) instead of user message injection - Chat client: warn and omit rich items (API doesn't support multi-part tool results), matching Ollama/Bedrock pattern - Unify test image: use sample_image.jpg across all integration tests - Add Azure OpenAI Responses integration test - Assert model describes house image to verify perception Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix lint: remove print statement, wrap long line Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback: bug fixes, single-pass MCP, unit tests - Add isinstance guard in from_function_result for non-Content lists - Fix Anthropic empty tool_content fallback to string result - Fix Content(type='text', text=None) edge case in parse_result - Rewrite MCP _parse_tool_result_from_mcp as single-pass (no index counters) - Add Anthropic unit tests: data image, uri image, unsupported media, all-unsupported - Add OpenAI Chat unit test: rich items warning and omission - Add OpenAI Responses unit tests: function_result with/without items - Add test_types tests: only-rich-items list, non-Content list fallback Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix pyright errors: add type ignore comments for Any list iteration Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix mypy/pyright: ensure ToolExecutionException receives str Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix lint: remove duplicate test_prepare_options_excludes_conversation_id Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor: unify all tool results into Content items * addressed copilot comments * pyright fix * small fix * comments * fix: address Copilot review - warnings, blob safety, dedup - Add warning logs when rich content is dropped in Claude agent and MCP server handlers (matching Chat/Bedrock/Ollama pattern) - Defensive blob URI construction: wrap plain base64 in data: prefix - Simplify Chat client _prepare_content_for_openai to use content.result - Simplify Responses client text-only path, remove redundant nesting - Add test for plain base64 blob without data: prefix Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix token double-counting in compaction and address review comments - Exclude items from _serialize_content() to prevent double-counting tokens when items mirrors result in function_result content - Add rich content warning in GitHub Copilot agent tool handler - Replace raw Content debug log with concise item count/type summary - Update stale test comments about FunctionTool.invoke return type Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
Giles Odigwe
Copilot
@@ -716,12 +716,46 @@ class AnthropicClient(
|
||||
"input": content.parse_arguments(),
|
||||
})
|
||||
case "function_result":
|
||||
a_content.append({
|
||||
"type": "tool_result",
|
||||
"tool_use_id": content.call_id,
|
||||
"content": content.result if content.result is not None else "",
|
||||
"is_error": content.exception is not None,
|
||||
})
|
||||
if content.items:
|
||||
tool_content: list[dict[str, Any]] = []
|
||||
for item in content.items:
|
||||
if item.type == "text":
|
||||
tool_content.append({"type": "text", "text": item.text or ""})
|
||||
elif item.type == "data" and item.has_top_level_media_type("image"):
|
||||
tool_content.append({
|
||||
"type": "image",
|
||||
"source": {
|
||||
"data": _get_data_bytes_as_str(item), # type: ignore[attr-defined]
|
||||
"media_type": item.media_type,
|
||||
"type": "base64",
|
||||
},
|
||||
})
|
||||
elif item.type == "uri" and item.has_top_level_media_type("image"):
|
||||
tool_content.append({
|
||||
"type": "image",
|
||||
"source": {"type": "url", "url": item.uri},
|
||||
})
|
||||
else:
|
||||
logger.debug(
|
||||
"Ignoring unsupported rich content media type in tool result: %s",
|
||||
item.media_type,
|
||||
)
|
||||
tool_result_content = (
|
||||
tool_content if tool_content else (content.result if content.result is not None else "")
|
||||
)
|
||||
a_content.append({
|
||||
"type": "tool_result",
|
||||
"tool_use_id": content.call_id,
|
||||
"content": tool_result_content,
|
||||
"is_error": content.exception is not None,
|
||||
})
|
||||
else:
|
||||
a_content.append({
|
||||
"type": "tool_result",
|
||||
"tool_use_id": content.call_id,
|
||||
"content": content.result if content.result is not None else "",
|
||||
"is_error": content.exception is not None,
|
||||
})
|
||||
case "mcp_server_tool_call":
|
||||
mcp_call: dict[str, Any] = {
|
||||
"type": "mcp_tool_use",
|
||||
|
||||
Reference in New Issue
Block a user