agent/agent-framework
1
0
Fork 0
mirror of https://github.com/microsoft/agent-framework.git synced 2026-06-16 21:04:09 +08:00
Code Issues Packages Projects Releases Wiki Activity

Python: Unify tool results as Content items with rich content support (#4331)

Browse Source 
* feat(python): allow @tool functions to return rich content (images, audio)

Add support for tool functions to return Content objects that the model can perceive natively. Closes #4272

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Anthropic logging + mypy fix

* Address PR review: fix MCP ordering, fold helper into from_function_result, fix Chat client

- Preserve original content order in MCP tool results instead of text-first
- Move _build_function_result logic into Content.from_function_result()
- Chat Completions: inject user message for rich items (API only supports string tool content)
- Update tests for ordering and new from_function_result behavior

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Use native Responses API multi-part output, warn+omit for Chat client

- Responses client: put rich items directly in function_call_output's
  output field as list (native API support) instead of user message injection
- Chat client: warn and omit rich items (API doesn't support multi-part
  tool results), matching Ollama/Bedrock pattern
- Unify test image: use sample_image.jpg across all integration tests
- Add Azure OpenAI Responses integration test
- Assert model describes house image to verify perception

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix lint: remove print statement, wrap long line

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review feedback: bug fixes, single-pass MCP, unit tests

- Add isinstance guard in from_function_result for non-Content lists
- Fix Anthropic empty tool_content fallback to string result
- Fix Content(type='text', text=None) edge case in parse_result
- Rewrite MCP _parse_tool_result_from_mcp as single-pass (no index counters)
- Add Anthropic unit tests: data image, uri image, unsupported media, all-unsupported
- Add OpenAI Chat unit test: rich items warning and omission
- Add OpenAI Responses unit tests: function_result with/without items
- Add test_types tests: only-rich-items list, non-Content list fallback

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix pyright errors: add type ignore comments for Any list iteration

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix mypy/pyright: ensure ToolExecutionException receives str

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix lint: remove duplicate test_prepare_options_excludes_conversation_id

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* refactor: unify all tool results into Content items

* addressed copilot comments

* pyright fix

* small fix

* comments

* fix: address Copilot review - warnings, blob safety, dedup

- Add warning logs when rich content is dropped in Claude agent and
  MCP server handlers (matching Chat/Bedrock/Ollama pattern)
- Defensive blob URI construction: wrap plain base64 in data: prefix
- Simplify Chat client _prepare_content_for_openai to use content.result
- Simplify Responses client text-only path, remove redundant nesting
- Add test for plain base64 blob without data: prefix

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix token double-counting in compaction and address review comments

- Exclude items from _serialize_content() to prevent double-counting
  tokens when items mirrors result in function_result content
- Add rich content warning in GitHub Copilot agent tool handler
- Replace raw Content debug log with concise item count/type summary
- Update stale test comments about FunctionTool.invoke return type

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
Giles Odigwe
2026-03-12 15:30:09 -07:00
committed by GitHub
Unverified
parent b6a1315386
commit 5e33deff45
27 changed files with 1337 additions and 332 deletions
Download Patch File Download Diff File Expand all files Collapse all files
python/packages/core/tests/assets/sample_image.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 178 KiB

python/packages/core/tests/azure/test_azure_chat_client.py
+37 -8
View File
@@ -89,18 +89,26 @@ def test_init_endpoint(azure_openai_unit_test_env: dict[str, str]) -> None:
@pytest.mark.parametrize("exclude_list", [["AZURE_OPENAI_CHAT_DEPLOYMENT_NAME"]], indirect=True)
def test_init_with_empty_deployment_name(azure_openai_unit_test_env: dict[str, str]) -> None:
def test_init_with_empty_deployment_name(
azure_openai_unit_test_env: dict[str, str],
) -> None:
with pytest.raises(ValueError):
AzureOpenAIChatClient()
@pytest.mark.parametrize("exclude_list", [["AZURE_OPENAI_ENDPOINT", "AZURE_OPENAI_BASE_URL"]], indirect=True)
def test_init_with_empty_endpoint_and_base_url(azure_openai_unit_test_env: dict[str, str]) -> None:
def test_init_with_empty_endpoint_and_base_url(
azure_openai_unit_test_env: dict[str, str],
) -> None:
with pytest.raises(ValueError):
AzureOpenAIChatClient()
@pytest.mark.parametrize("override_env_param_dict", [{"AZURE_OPENAI_ENDPOINT": "http://test.com"}], indirect=True)
@pytest.mark.parametrize(
"override_env_param_dict",
[{"AZURE_OPENAI_ENDPOINT": "http://test.com"}],
indirect=True,
)
def test_init_with_invalid_endpoint(azure_openai_unit_test_env: dict[str, str]) -> None:
# Note: URL scheme validation was previously handled by pydantic's HTTPsUrl type.
# After migrating to load_settings with TypedDict, endpoint is a plain string and no longer
@@ -147,7 +155,11 @@ def mock_chat_completion_response() -> ChatCompletion:
return ChatCompletion(
id="test_id",
choices=[
Choice(index=0, message=ChatCompletionMessage(content="test", role="assistant"), finish_reason="stop")
Choice(
index=0,
message=ChatCompletionMessage(content="test", role="assistant"),
finish_reason="stop",
)
],
created=0,
model="test",
@@ -159,7 +171,13 @@ def mock_chat_completion_response() -> ChatCompletion:
def mock_streaming_chat_completion_response() -> AsyncStream[ChatCompletionChunk]:
content = ChatCompletionChunk(
id="test_id",
choices=[ChunkChoice(index=0, delta=ChunkChoiceDelta(content="test", role="assistant"), finish_reason="stop")],
choices=[
ChunkChoice(
index=0,
delta=ChunkChoiceDelta(content="test", role="assistant"),
finish_reason="stop",
)
],
created=0,
model="test",
object="chat.completion.chunk",
@@ -546,7 +564,9 @@ async def test_bad_request_non_content_filter(
test_endpoint = os.getenv("AZURE_OPENAI_ENDPOINT")
assert test_endpoint is not None
mock_create.side_effect = openai.BadRequestError(
"The request was bad.", response=Response(400, request=Request("POST", test_endpoint)), body={}
"The request was bad.",
response=Response(400, request=Request("POST", test_endpoint)),
body={},
)
azure_chat_client = AzureOpenAIChatClient()
@@ -605,7 +625,13 @@ async def test_streaming_with_none_delta(
# Second chunk has actual content
chunk_with_content = ChatCompletionChunk(
id="test_id",
choices=[ChunkChoice(index=0, delta=ChunkChoiceDelta(content="test", role="assistant"), finish_reason="stop")],
choices=[
ChunkChoice(
index=0,
delta=ChunkChoiceDelta(content="test", role="assistant"),
finish_reason="stop",
)
],
created=0,
model="test",
object="chat.completion.chunk",
@@ -854,7 +880,10 @@ async def test_azure_openai_chat_client_agent_basic_run_streaming():
) as agent:
# Test streaming run
full_text = ""
async for chunk in agent.run("Please respond with exactly: 'This is a streaming response test.'", stream=True):
async for chunk in agent.run(
"Please respond with exactly: 'This is a streaming response test.'",
stream=True,
):
assert isinstance(chunk, AgentResponseUpdate)
if chunk.text:
full_text += chunk.text
python/packages/core/tests/azure/test_azure_responses_client.py
+65 -6
View File
@@ -3,6 +3,7 @@
import json
import logging
import os
from pathlib import Path
from typing import Annotated, Any
from unittest.mock import MagicMock
@@ -44,10 +45,13 @@ async def get_weather(location: Annotated[str, "The location as a city name"]) -
return f"The weather in {location} is sunny and 72°F."
async def create_vector_store(client: AzureOpenAIResponsesClient) -> tuple[str, Content]:
async def create_vector_store(
client: AzureOpenAIResponsesClient,
) -> tuple[str, Content]:
"""Create a vector store with sample documents for testing."""
file = await client.client.files.create(
file=("todays_weather.txt", b"The weather today is sunny with a high of 75F."), purpose="assistants"
file=("todays_weather.txt", b"The weather today is sunny with a high of 75F."),
purpose="assistants",
)
vector_store = await client.client.vector_stores.create(
name="knowledge_base",
@@ -98,7 +102,9 @@ def test_init_model_id_kwarg(azure_openai_unit_test_env: dict[str, str]) -> None
assert isinstance(azure_responses_client, SupportsChatGetResponse)
def test_init_model_id_kwarg_does_not_override_deployment_name(azure_openai_unit_test_env: dict[str, str]) -> None:
def test_init_model_id_kwarg_does_not_override_deployment_name(
azure_openai_unit_test_env: dict[str, str],
) -> None:
"""Test that deployment_name takes precedence over model_id kwarg (issue #4299)."""
azure_responses_client = AzureOpenAIResponsesClient(deployment_name="my-deployment", model_id="gpt-4o")
@@ -323,7 +329,12 @@ def test_serialize(azure_openai_unit_test_env: dict[str, str]) -> None:
"temperature_c": {"type": "number"},
"advisory": {"type": "string"},
},
"required": ["location", "conditions", "temperature_c", "advisory"],
"required": [
"location",
"conditions",
"temperature_c",
"advisory",
],
"additionalProperties": False,
},
},
@@ -445,7 +456,12 @@ async def test_integration_web_search() -> None:
# Test that the client will use the web search tool with location
content = {
"messages": [Message(role="user", text="What is the current weather? Do not ask for my current location.")],
"messages": [
Message(
role="user",
text="What is the current weather? Do not ask for my current location.",
)
],
"options": {
"tool_choice": "auto",
"tools": [
@@ -556,7 +572,12 @@ async def test_integration_client_agent_hosted_code_interpreter_tool():
client = AzureOpenAIResponsesClient(credential=AzureCliCredential())
response = await client.get_response(
messages=[Message(role="user", text="Calculate the sum of numbers from 1 to 10 using Python code.")],
messages=[
Message(
role="user",
text="Calculate the sum of numbers from 1 to 10 using Python code.",
)
],
options={
"tools": [AzureOpenAIResponsesClient.get_code_interpreter_tool()],
},
@@ -604,6 +625,44 @@ async def test_integration_client_agent_existing_session():
assert "photography" in second_response.text.lower()
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_azure_openai_responses_client_tool_rich_content_image() -> None:
"""Test that Azure OpenAI Responses client can handle tool results containing images."""
image_path = Path(__file__).parent.parent / "assets" / "sample_image.jpg"
image_bytes = image_path.read_bytes()
@tool(approval_mode="never_require")
def get_test_image() -> Content:
"""Return a test image for analysis."""
return Content.from_data(data=image_bytes, media_type="image/jpeg")
client = AzureOpenAIResponsesClient(credential=AzureCliCredential())
client.function_invocation_configuration["max_iterations"] = 2
for streaming in [False, True]:
messages = [
Message(
role="user",
text="Call the get_test_image tool and describe what you see.",
)
]
options: dict[str, Any] = {"tools": [get_test_image], "tool_choice": "auto"}
if streaming:
response = await client.get_response(messages=messages, stream=True, options=options).get_final_response()
else:
response = await client.get_response(messages=messages, options=options)
assert response is not None
assert isinstance(response, ChatResponse)
assert response.text is not None
assert len(response.text) > 0
# sample_image.jpg contains a photo of a house; the model should mention it.
assert "house" in response.text.lower(), f"Model did not describe the house image. Response: {response.text}"
# region Integration with Foundry V2
python/packages/core/tests/core/test_agents.py
+12 -8
View File
@@ -761,9 +761,10 @@ async def test_chat_agent_as_tool_function_execution(
# Test function execution
result = await tool.invoke(arguments=tool.input_model(task="Hello"))
# Should return the agent's response text
assert isinstance(result, str)
assert result == "test response" # From mock chat client
# Should return the agent's response text as a list of Content items
assert isinstance(result, list)
assert len(result) == 1
assert result[0].text == "test response" # From mock chat client
async def test_chat_agent_as_tool_with_stream_callback(
@@ -785,10 +786,11 @@ async def test_chat_agent_as_tool_with_stream_callback(
# Should have collected streaming updates
assert len(collected_updates) > 0
assert isinstance(result, str)
assert isinstance(result, list)
result_text = result[0].text
# Result should be concatenation of all streaming updates
expected_text = "".join(update.text for update in collected_updates)
assert result == expected_text
assert result_text == expected_text
async def test_chat_agent_as_tool_with_custom_arg_name(
@@ -801,7 +803,8 @@ async def test_chat_agent_as_tool_with_custom_arg_name(
# Test that the custom argument name works
result = await tool.invoke(arguments=tool.input_model(prompt="Test prompt"))
assert result == "test response"
assert isinstance(result, list)
assert result[0].text == "test response"
async def test_chat_agent_as_tool_with_async_stream_callback(
@@ -823,10 +826,11 @@ async def test_chat_agent_as_tool_with_async_stream_callback(
# Should have collected streaming updates
assert len(collected_updates) > 0
assert isinstance(result, str)
assert isinstance(result, list)
result_text = result[0].text
# Result should be concatenation of all streaming updates
expected_text = "".join(update.text for update in collected_updates)
assert result == expected_text
assert result_text == expected_text
async def test_chat_agent_as_tool_name_sanitization(
python/packages/core/tests/core/test_mcp.py
+81 -26
View File
@@ -67,30 +67,31 @@ def test_mcp_prompt_message_to_ai_content():
def test_parse_tool_result_from_mcp():
"""Test conversion from MCP tool result to string representation."""
"""Test conversion from MCP tool result with images preserves original order."""
mcp_result = types.CallToolResult(
content=[
types.TextContent(type="text", text="Result text"),
types.ImageContent(type="image", data="eHl6", mimeType="image/png"),
types.TextContent(type="text", text="After image"),
types.ImageContent(type="image", data="YWJj", mimeType="image/webp"),
]
)
result = _parse_tool_result_from_mcp(mcp_result)
# Multiple items produce a JSON array of strings
assert isinstance(result, str)
import json
parsed = json.loads(result)
assert len(parsed) == 3
assert parsed[0] == "Result text"
# Image items are JSON-encoded strings within the array
img1 = json.loads(parsed[1])
assert img1["type"] == "image"
assert img1["data"] == "eHl6"
img2 = json.loads(parsed[2])
assert img2["type"] == "image"
assert img2["data"] == "YWJj"
# Results with images return a list of Content objects in original order
assert isinstance(result, list)
assert len(result) == 4
# Order is preserved: text, image, text, image
assert result[0].type == "text"
assert result[0].text == "Result text"
assert result[1].type == "data"
assert result[1].media_type == "image/png"
assert "eHl6" in result[1].uri
assert result[2].type == "text"
assert result[2].text == "After image"
assert result[3].type == "data"
assert result[3].media_type == "image/webp"
assert "YWJj" in result[3].uri
def test_parse_tool_result_from_mcp_single_text():
@@ -98,26 +99,73 @@ def test_parse_tool_result_from_mcp_single_text():
mcp_result = types.CallToolResult(content=[types.TextContent(type="text", text="Simple result")])
result = _parse_tool_result_from_mcp(mcp_result)
# Single text item returns just the text
assert result == "Simple result"
# Single text item returns list with one text Content
assert isinstance(result, list)
assert len(result) == 1
assert result[0].type == "text"
assert result[0].text == "Simple result"
def test_parse_tool_result_from_mcp_meta_not_in_string():
"""Test that _meta data is not included in the string result (it's tool-level, not content-level)."""
"""Test that _meta data is not included in the result (it's tool-level, not content-level)."""
mcp_result = types.CallToolResult(
content=[types.TextContent(type="text", text="Error occurred")],
_meta={"isError": True, "errorCode": "TOOL_ERROR"},
)
result = _parse_tool_result_from_mcp(mcp_result)
assert result == "Error occurred"
assert isinstance(result, list)
assert len(result) == 1
assert result[0].text == "Error occurred"
def test_parse_tool_result_from_mcp_empty_content():
"""Test that empty content produces empty string."""
"""Test that empty content produces list with empty text Content."""
mcp_result = types.CallToolResult(content=[])
result = _parse_tool_result_from_mcp(mcp_result)
assert result == ""
assert isinstance(result, list)
assert len(result) == 1
assert result[0].type == "text"
assert result[0].text == ""
def test_parse_tool_result_from_mcp_audio_content():
"""Test conversion from MCP tool result with audio returns rich content list."""
mcp_result = types.CallToolResult(
content=[
types.AudioContent(type="audio", data="YXVkaW8=", mimeType="audio/wav"),
]
)
result = _parse_tool_result_from_mcp(mcp_result)
assert isinstance(result, list)
assert len(result) == 1
assert result[0].type == "data"
assert result[0].media_type == "audio/wav"
assert "YXVkaW8=" in result[0].uri
def test_parse_tool_result_from_mcp_blob_plain_base64():
"""Test that plain base64 blob (without data: prefix) is wrapped into a data URI."""
mcp_result = types.CallToolResult(
content=[
types.EmbeddedResource(
type="resource",
resource=types.BlobResourceContents(
uri=AnyUrl("file://test.bin"),
mimeType="application/pdf",
blob="dGVzdCBkYXRh",
),
),
]
)
result = _parse_tool_result_from_mcp(mcp_result)
assert isinstance(result, list)
assert len(result) == 1
assert result[0].type == "data"
assert result[0].media_type == "application/pdf"
assert "dGVzdCBkYXRh" in result[0].uri
def test_mcp_content_types_to_ai_content_text():
@@ -769,7 +817,10 @@ async def test_mcp_tool_call_tool_with_meta_integration():
func = server.functions[0]
result = await func.invoke(param="test_value")
assert result == "Tool executed with metadata"
assert isinstance(result, list)
assert len(result) == 1
assert result[0].type == "text"
assert result[0].text == "Tool executed with metadata"
async def test_local_mcp_server_function_execution():
@@ -808,7 +859,8 @@ async def test_local_mcp_server_function_execution():
func = server.functions[0]
result = await func.invoke(param="test_value")
assert result == "Tool executed successfully"
assert isinstance(result, list)
assert result[0].text == "Tool executed successfully"
async def test_local_mcp_server_function_execution_with_nested_object():
@@ -855,7 +907,8 @@ async def test_local_mcp_server_function_execution_with_nested_object():
# Call with nested object
result = await func.invoke(params={"customer_id": 251})
assert result == '{"name": "John Doe", "id": 251}'
assert isinstance(result, list)
assert result[0].text == '{"name": "John Doe", "id": 251}'
# Verify the session.call_tool was called with the correct nested structure
server.session.call_tool.assert_called_once()
@@ -977,7 +1030,8 @@ async def test_mcp_tool_call_tool_succeeds_when_is_error_false():
await server.load_tools()
func = server.functions[0]
result = await func.invoke(param="test_value")
assert result == "Success"
assert isinstance(result, list)
assert result[0].text == "Success"
async def test_mcp_tool_is_error_propagates_through_function_middleware():
@@ -1080,7 +1134,8 @@ async def test_local_mcp_server_prompt_execution():
prompt = server.functions[0]
result = await prompt.invoke(arg="test_value")
assert result == "Test message"
assert isinstance(result, list)
assert result[0].text == "Test message"
@pytest.mark.parametrize(
python/packages/core/tests/core/test_observability.py
+2 -1
View File
@@ -2385,7 +2385,8 @@ async def test_tool_result_preserves_non_ascii_characters(span_exporter: InMemor
span_exporter.clear()
result = await echo.invoke(text=arabic_text)
assert result == arabic_text
assert isinstance(result, list)
assert result[0].text == arabic_text
spans = span_exporter.get_finished_spans()
assert len(spans) == 1
span = spans[0]
python/packages/core/tests/core/test_tools.py
+22 -11
View File
@@ -124,7 +124,8 @@ async def test_tool_decorator_with_json_schema_invoke_uses_mapping():
return f"{query}:{max_results}"
result = await search.invoke(arguments={"query": "hello", "max_results": 3})
assert result == "hello:3"
assert isinstance(result, list)
assert result[0].text == "hello:3"
async def test_tool_decorator_with_json_schema_invoke_missing_required():
@@ -221,7 +222,8 @@ async def test_tool_decorator_with_schema_invoke():
return a + b
result = await calculate.invoke(arguments=CalcInput(a=3, b=7))
assert result == "10"
assert isinstance(result, list)
assert result[0].text == "10"
def test_tool_decorator_with_schema_overrides_annotations():
@@ -492,11 +494,13 @@ async def test_tool_decorator_shared_state():
# Test with invoke method as well (simulating agent execution)
result6 = await increment_tool.invoke(amount=5)
assert result6 == "Counter incremented by 5. New value: 60"
assert isinstance(result6, list)
assert result6[0].text == "Counter incremented by 5. New value: 60"
assert counter_instance.counter == 60
result7 = await get_value_tool.invoke()
assert result7 == "Current counter value: 60"
assert isinstance(result7, list)
assert result7[0].text == "Current counter value: 60"
assert counter_instance.counter == 60
@@ -519,7 +523,8 @@ async def test_tool_invoke_telemetry_enabled(span_exporter: InMemorySpanExporter
result = await telemetry_test_tool.invoke(x=1, y=2, tool_call_id="test_call_id")
# Verify result
assert result == "3"
assert isinstance(result, list)
assert result[0].text == "3"
# Verify telemetry calls
spans = span_exporter.get_finished_spans()
@@ -563,7 +568,8 @@ async def test_tool_invoke_telemetry_sensitive_disabled(span_exporter: InMemoryS
result = await telemetry_test_tool.invoke(x=1, y=2, tool_call_id="test_call_id")
# Verify result
assert result == "3"
assert isinstance(result, list)
assert result[0].text == "3"
# Verify telemetry calls
spans = span_exporter.get_finished_spans()
@@ -604,7 +610,8 @@ async def test_tool_invoke_ignores_additional_kwargs() -> None:
options={"model_id": "dummy"},
)
assert result == "HELLO WORLD"
assert isinstance(result, list)
assert result[0].text == "HELLO WORLD"
async def test_tool_invoke_telemetry_with_pydantic_args(span_exporter: InMemorySpanExporter):
@@ -628,7 +635,8 @@ async def test_tool_invoke_telemetry_with_pydantic_args(span_exporter: InMemoryS
result = await pydantic_test_tool.invoke(arguments=args_model, tool_call_id="pydantic_call")
# Verify result
assert result == "15"
assert isinstance(result, list)
assert result[0].text == "15"
spans = span_exporter.get_finished_spans()
assert len(spans) == 1
span = spans[0]
@@ -696,7 +704,8 @@ async def test_tool_invoke_telemetry_async_function(span_exporter: InMemorySpanE
result = await async_telemetry_test.invoke(x=3, y=4, tool_call_id="async_call")
# Verify result
assert result == "12"
assert isinstance(result, list)
assert result[0].text == "12"
spans = span_exporter.get_finished_spans()
assert len(spans) == 1
span = spans[0]
@@ -932,13 +941,15 @@ async def test_ai_function_with_kwargs_injection():
arguments=tool_with_kwargs.input_model(x=5),
user_id="user2",
)
assert result == "x=5, user=user2"
assert isinstance(result, list)
assert result[0].text == "x=5, user=user2"
# Verify invoke works without injected args (uses default)
result_default = await tool_with_kwargs.invoke(
arguments=tool_with_kwargs.input_model(x=10),
)
assert result_default == "x=10, user=unknown"
assert isinstance(result_default, list)
assert result_default[0].text == "x=10, user=unknown"
# region _parse_annotation tests
python/packages/core/tests/core/test_types.py
+194 -48
View File
@@ -542,7 +542,12 @@ def test_function_result_content():
# Check the type and content
assert content.type == "function_result"
assert content.result == {"param1": "value1"}
# Dict results are stringified and stored as text items
assert "param1" in content.result
assert "value1" in content.result
assert content.items is not None
assert len(content.items) == 1
assert content.items[0].type == "text"
# Ensure the instance is of type BaseContent
assert isinstance(content, Content)
@@ -2455,12 +2460,13 @@ class NestedModel(BaseModel):
def test_parse_result_pydantic_model():
"""Test that Pydantic BaseModel subclasses are properly serialized using model_dump()."""
result = WeatherResult(temperature=22.5, condition="sunny")
json_result = FunctionTool.parse_result(result)
parsed = FunctionTool.parse_result(result)
# The result should be a valid JSON string
assert isinstance(json_result, str)
assert '"temperature": 22.5' in json_result or '"temperature":22.5' in json_result
assert '"condition": "sunny"' in json_result or '"condition":"sunny"' in json_result
assert isinstance(parsed, list)
assert len(parsed) == 1
assert parsed[0].type == "text"
assert '"temperature": 22.5' in parsed[0].text or '"temperature":22.5' in parsed[0].text
assert '"condition": "sunny"' in parsed[0].text or '"condition":"sunny"' in parsed[0].text
def test_parse_result_pydantic_model_in_list():
@@ -2469,14 +2475,14 @@ def test_parse_result_pydantic_model_in_list():
WeatherResult(temperature=20.0, condition="cloudy"),
WeatherResult(temperature=25.0, condition="sunny"),
]
json_result = FunctionTool.parse_result(results)
parsed = FunctionTool.parse_result(results)
# The result should be a valid JSON string representing a list
assert isinstance(json_result, str)
assert json_result.startswith("[")
assert json_result.endswith("]")
assert "cloudy" in json_result
assert "sunny" in json_result
assert isinstance(parsed, list)
assert len(parsed) == 1
assert parsed[0].type == "text"
assert parsed[0].text.startswith("[")
assert "cloudy" in parsed[0].text
assert "sunny" in parsed[0].text
def test_parse_result_pydantic_model_in_dict():
@@ -2485,26 +2491,28 @@ def test_parse_result_pydantic_model_in_dict():
"current": WeatherResult(temperature=22.0, condition="partly cloudy"),
"forecast": WeatherResult(temperature=24.0, condition="sunny"),
}
json_result = FunctionTool.parse_result(results)
parsed = FunctionTool.parse_result(results)
# The result should be a valid JSON string representing a dict
assert isinstance(json_result, str)
assert "current" in json_result
assert "forecast" in json_result
assert "partly cloudy" in json_result
assert "sunny" in json_result
assert isinstance(parsed, list)
assert len(parsed) == 1
assert parsed[0].type == "text"
assert "current" in parsed[0].text
assert "forecast" in parsed[0].text
assert "partly cloudy" in parsed[0].text
assert "sunny" in parsed[0].text
def test_parse_result_nested_pydantic_model():
"""Test that nested Pydantic models are properly serialized."""
result = NestedModel(name="Seattle", weather=WeatherResult(temperature=18.0, condition="rainy"))
json_result = FunctionTool.parse_result(result)
parsed = FunctionTool.parse_result(result)
# The result should be a valid JSON string
assert isinstance(json_result, str)
assert "Seattle" in json_result
assert "rainy" in json_result
assert "18.0" in json_result or "18" in json_result
assert isinstance(parsed, list)
assert len(parsed) == 1
assert parsed[0].type == "text"
assert "Seattle" in parsed[0].text
assert "rainy" in parsed[0].text
assert "18.0" in parsed[0].text or "18" in parsed[0].text
# region FunctionTool.parse_result with MCP TextContent-like objects
@@ -2518,11 +2526,12 @@ def test_parse_result_text_content_single():
text: str
result = [MockTextContent("Hello from MCP tool!")]
json_result = FunctionTool.parse_result(result)
parsed = FunctionTool.parse_result(result)
# Should extract text and serialize as JSON array of strings
assert isinstance(json_result, str)
assert json_result == '["Hello from MCP tool!"]'
# Non-Content list items are serialized via _make_dumpable
assert isinstance(parsed, list)
assert len(parsed) == 1
assert parsed[0].type == "text"
def test_parse_result_text_content_multiple():
@@ -2533,11 +2542,12 @@ def test_parse_result_text_content_multiple():
text: str
result = [MockTextContent("First result"), MockTextContent("Second result")]
json_result = FunctionTool.parse_result(result)
parsed = FunctionTool.parse_result(result)
# Should extract text from each and serialize as JSON array
assert isinstance(json_result, str)
assert json_result == '["First result", "Second result"]'
# Non-Content list items are serialized via _make_dumpable
assert isinstance(parsed, list)
assert len(parsed) == 1
assert parsed[0].type == "text"
def test_parse_result_text_content_with_non_string_text():
@@ -2548,38 +2558,174 @@ def test_parse_result_text_content_with_non_string_text():
self.text = 12345 # Not a string!
result = [BadTextContent()]
json_result = FunctionTool.parse_result(result)
parsed = FunctionTool.parse_result(result)
# Should not extract text since it's not a string, will serialize the object
assert isinstance(json_result, str)
assert isinstance(parsed, list)
assert len(parsed) == 1
assert parsed[0].type == "text"
def test_parse_result_none_returns_empty_string():
"""Test that None returns an empty string."""
assert FunctionTool.parse_result(None) == ""
"""Test that None returns a list with empty text Content."""
parsed = FunctionTool.parse_result(None)
assert isinstance(parsed, list)
assert len(parsed) == 1
assert parsed[0].type == "text"
assert parsed[0].text == ""
def test_parse_result_string_passthrough():
"""Test that strings are returned as-is."""
assert FunctionTool.parse_result("hello world") == "hello world"
assert FunctionTool.parse_result('{"key": "value"}') == '{"key": "value"}'
"""Test that strings are wrapped in Content."""
parsed = FunctionTool.parse_result("hello world")
assert isinstance(parsed, list)
assert len(parsed) == 1
assert parsed[0].text == "hello world"
parsed2 = FunctionTool.parse_result('{"key": "value"}')
assert isinstance(parsed2, list)
assert len(parsed2) == 1
assert parsed2[0].text == '{"key": "value"}'
def test_parse_result_content_object():
"""Test that Content objects are serialized via to_dict."""
"""Test that text Content objects are wrapped in a list."""
content = Content.from_text("hello")
result = FunctionTool.parse_result(content)
assert isinstance(result, str)
assert "hello" in result
assert isinstance(result, list)
assert len(result) == 1
assert result[0].type == "text"
assert result[0].text == "hello"
def test_parse_result_list_of_content():
"""Test that list[Content] is serialized to JSON."""
"""Test that list[Content] with text-only items is returned as list[Content]."""
contents = [Content.from_text("hello"), Content.from_text("world")]
result = FunctionTool.parse_result(contents)
assert isinstance(result, str)
assert "hello" in result
assert "world" in result
assert isinstance(result, list)
assert len(result) == 2
assert result[0].text == "hello"
assert result[1].text == "world"
def test_parse_result_single_image_content():
"""Test that a single image Content is preserved as list[Content]."""
image_content = Content.from_data(data=b"fake_png_bytes", media_type="image/png")
result = FunctionTool.parse_result(image_content)
assert isinstance(result, list)
assert len(result) == 1
assert result[0].type == "data"
assert result[0].media_type == "image/png"
def test_parse_result_single_text_content():
"""Test that a single text Content returns a list with one text Content."""
text_content = Content.from_text("just text")
result = FunctionTool.parse_result(text_content)
assert isinstance(result, list)
assert len(result) == 1
assert result[0].type == "text"
assert result[0].text == "just text"
def test_parse_result_mixed_content_list():
"""Test that list with text and image Content is preserved."""
contents = [
Content.from_text("Chart rendered."),
Content.from_data(data=b"image_bytes", media_type="image/png"),
]
result = FunctionTool.parse_result(contents)
assert isinstance(result, list)
assert len(result) == 2
assert result[0].type == "text"
assert result[1].type == "data"
def test_from_function_result_with_content_list():
"""Test Content.from_function_result stores all items uniformly."""
content_list = [
Content.from_text("Chart rendered."),
Content.from_data(data=b"image_bytes", media_type="image/png"),
]
result = Content.from_function_result(call_id="test-123", result=content_list)
assert result.type == "function_result"
assert result.call_id == "test-123"
assert result.result == "Chart rendered."
assert result.items is not None
assert len(result.items) == 2
assert result.items[0].type == "text"
assert result.items[0].text == "Chart rendered."
assert result.items[1].type == "data"
assert result.items[1].media_type == "image/png"
def test_from_function_result_with_string():
"""Test Content.from_function_result with plain string result."""
result = Content.from_function_result(call_id="test-123", result="just text")
assert result.type == "function_result"
assert result.call_id == "test-123"
assert result.result == "just text"
assert result.items is not None
assert len(result.items) == 1
assert result.items[0].type == "text"
assert result.items[0].text == "just text"
def test_content_from_function_result_items_in_to_dict():
"""Test that items are included in to_dict serialization."""
content_list = [
Content.from_text("done"),
Content.from_data(data=b"png_data", media_type="image/png"),
]
result = Content.from_function_result(
call_id="call-1",
result=content_list,
)
d = result.to_dict()
assert "items" in d
assert len(d["items"]) == 2
assert d["items"][0]["type"] == "text"
assert d["items"][1]["type"] == "data"
def test_from_function_result_with_only_rich_content_list():
"""Test Content.from_function_result with only image items and no text."""
content_list = [
Content.from_data(data=b"image_bytes", media_type="image/png"),
]
result = Content.from_function_result(call_id="test-456", result=content_list)
assert result.type == "function_result"
assert result.result == ""
assert result.items is not None
assert len(result.items) == 1
assert result.items[0].type == "data"
def test_function_result_items_roundtrip_via_dict():
"""Test that items survive a to_dict/from_dict round-trip as Content objects."""
content_list = [
Content.from_text("done"),
Content.from_data(data=b"png_data", media_type="image/png"),
]
original = Content.from_function_result(call_id="call-rt", result=content_list)
restored = Content.from_dict(original.to_dict())
assert restored.items is not None
assert len(restored.items) == 2
assert isinstance(restored.items[0], Content)
assert restored.items[0].type == "text"
assert restored.items[0].text == "done"
assert isinstance(restored.items[1], Content)
assert restored.items[1].type == "data"
def test_from_function_result_with_non_content_list():
"""Test Content.from_function_result with a list of non-Content objects falls back to str."""
result = Content.from_function_result(call_id="test-789", result=["hello", "world"])
assert result.type == "function_result"
assert result.result == "['hello', 'world']"
assert result.items is not None
assert len(result.items) == 1
assert result.items[0].type == "text"
# endregion
python/packages/core/tests/openai/test_openai_chat_client.py
+161 -43
View File
@@ -142,7 +142,9 @@ def test_serialize_with_org_id(openai_unit_test_env: dict[str, str]) -> None:
assert "User-Agent" not in dumped_settings.get("default_headers", {})
async def test_content_filter_exception_handling(openai_unit_test_env: dict[str, str]) -> None:
async def test_content_filter_exception_handling(
openai_unit_test_env: dict[str, str],
) -> None:
"""Test that content filter errors are properly handled."""
client = OpenAIChatClient()
messages = [Message(role="user", text="test message")]
@@ -150,7 +152,9 @@ async def test_content_filter_exception_handling(openai_unit_test_env: dict[str,
# Create a mock BadRequestError with content_filter code
mock_response = MagicMock()
mock_error = BadRequestError(
message="Content filter error", response=mock_response, body={"error": {"code": "content_filter"}}
message="Content filter error",
response=mock_response,
body={"error": {"code": "content_filter"}},
)
mock_error.code = "content_filter"
@@ -184,7 +188,9 @@ def test_unsupported_tool_handling(openai_unit_test_env: dict[str, str]) -> None
assert result["tools"] == [dict_tool]
def test_prepare_tools_with_single_function_tool(openai_unit_test_env: dict[str, str]) -> None:
def test_prepare_tools_with_single_function_tool(
openai_unit_test_env: dict[str, str],
) -> None:
"""Test that a single FunctionTool is accepted for tool preparation."""
client = OpenAIChatClient()
@@ -241,12 +247,17 @@ async def test_exception_message_includes_original_error_details() -> None:
assert original_error_message in exception_message
def test_chat_response_content_order_text_before_tool_calls(openai_unit_test_env: dict[str, str]):
def test_chat_response_content_order_text_before_tool_calls(
openai_unit_test_env: dict[str, str],
):
"""Test that text content appears before tool calls in ChatResponse contents."""
# Import locally to avoid break other tests when the import changes
from openai.types.chat.chat_completion import ChatCompletion, Choice
from openai.types.chat.chat_completion_message import ChatCompletionMessage
from openai.types.chat.chat_completion_message_tool_call import ChatCompletionMessageToolCall, Function
from openai.types.chat.chat_completion_message_tool_call import (
ChatCompletionMessageToolCall,
Function,
)
# Create a mock OpenAI response with both text and tool calls
mock_response = ChatCompletion(
@@ -296,9 +307,10 @@ def test_function_result_falsy_values_handling(openai_unit_test_env: dict[str, s
"""
client = OpenAIChatClient()
# Test with empty list serialized as JSON string (as FunctionTool.invoke would produce)
# Test with empty list serialized as JSON string (pre-serialized result passed to from_function_result)
message_with_empty_list = Message(
role="tool", contents=[Content.from_function_result(call_id="call-123", result="[]")]
role="tool",
contents=[Content.from_function_result(call_id="call-123", result="[]")],
)
openai_messages = client._prepare_message_for_openai(message_with_empty_list)
@@ -307,16 +319,18 @@ def test_function_result_falsy_values_handling(openai_unit_test_env: dict[str, s
# Test with empty string (falsy but not None)
message_with_empty_string = Message(
role="tool", contents=[Content.from_function_result(call_id="call-456", result="")]
role="tool",
contents=[Content.from_function_result(call_id="call-456", result="")],
)
openai_messages = client._prepare_message_for_openai(message_with_empty_string)
assert len(openai_messages) == 1
assert openai_messages[0]["content"] == "" # Empty string should be preserved
# Test with False serialized as JSON string (as FunctionTool.invoke would produce)
# Test with False serialized as JSON string (pre-serialized result passed to from_function_result)
message_with_false = Message(
role="tool", contents=[Content.from_function_result(call_id="call-789", result="false")]
role="tool",
contents=[Content.from_function_result(call_id="call-789", result="false")],
)
openai_messages = client._prepare_message_for_openai(message_with_false)
@@ -336,7 +350,11 @@ def test_function_result_exception_handling(openai_unit_test_env: dict[str, str]
message_with_exception = Message(
role="tool",
contents=[
Content.from_function_result(call_id="call-123", result="Error: Function failed.", exception=test_exception)
Content.from_function_result(
call_id="call-123",
result="Error: Function failed.",
exception=test_exception,
)
],
)
@@ -346,16 +364,50 @@ def test_function_result_exception_handling(openai_unit_test_env: dict[str, str]
assert openai_messages[0]["tool_call_id"] == "call-123"
def test_function_result_with_rich_items_warns_and_omits(
openai_unit_test_env: dict[str, str],
) -> None:
"""Test that function_result with items logs a warning and omits rich items."""
client = OpenAIChatClient()
image_content = Content.from_data(data=b"image_bytes", media_type="image/png")
message = Message(
role="tool",
contents=[
Content.from_function_result(
call_id="call_rich",
result=[Content.from_text("Result text"), image_content],
)
],
)
with patch("agent_framework.openai._chat_client.logger") as mock_logger:
openai_messages = client._prepare_message_for_openai(message)
# Warning should be logged
mock_logger.warning.assert_called_once()
assert "does not support rich content" in mock_logger.warning.call_args[0][0]
# Tool message should still be emitted with text result
assert len(openai_messages) == 1
assert openai_messages[0]["role"] == "tool"
assert openai_messages[0]["tool_call_id"] == "call_rich"
assert openai_messages[0]["content"] == "Result text"
def test_parse_result_string_passthrough():
"""Test that string values are passed through directly without JSON encoding."""
"""Test that string values are wrapped in Content."""
from agent_framework import FunctionTool
result = FunctionTool.parse_result("simple string")
assert result == "simple string"
assert isinstance(result, str)
assert isinstance(result, list)
assert len(result) == 1
assert result[0].text == "simple string"
def test_prepare_content_for_openai_data_content_image(openai_unit_test_env: dict[str, str]) -> None:
def test_prepare_content_for_openai_data_content_image(
openai_unit_test_env: dict[str, str],
) -> None:
"""Test _prepare_content_for_openai converts DataContent with image media type to OpenAI format."""
client = OpenAIChatClient()
@@ -397,7 +449,8 @@ def test_prepare_content_for_openai_data_content_image(openai_unit_test_env: dic
# Test DataContent with MP3 audio
mp3_data_content = Content.from_uri(
uri="data:audio/mp3;base64,//uQAAAAWGluZwAAAA8AAAACAAACcQ==", media_type="audio/mp3"
uri="data:audio/mp3;base64,//uQAAAAWGluZwAAAA8AAAACAAACcQ==",
media_type="audio/mp3",
)
result = client._prepare_content_for_openai(mp3_data_content) # type: ignore
@@ -409,7 +462,9 @@ def test_prepare_content_for_openai_data_content_image(openai_unit_test_env: dic
assert result["input_audio"]["format"] == "mp3"
def test_prepare_content_for_openai_document_file_mapping(openai_unit_test_env: dict[str, str]) -> None:
def test_prepare_content_for_openai_document_file_mapping(
openai_unit_test_env: dict[str, str],
) -> None:
"""Test _prepare_content_for_openai converts document files (PDF, DOCX, etc.) to OpenAI file format."""
client = OpenAIChatClient()
@@ -515,7 +570,9 @@ def test_prepare_content_for_openai_document_file_mapping(openai_unit_test_env:
assert "filename" not in result["file"] # None filename should be omitted
def test_parse_text_reasoning_content_from_response(openai_unit_test_env: dict[str, str]) -> None:
def test_parse_text_reasoning_content_from_response(
openai_unit_test_env: dict[str, str],
) -> None:
"""Test that TextReasoningContent is correctly parsed from OpenAI response with reasoning_details."""
client = OpenAIChatClient()
@@ -563,7 +620,9 @@ def test_parse_text_reasoning_content_from_response(openai_unit_test_env: dict[s
assert parsed_details == mock_reasoning_details
def test_parse_text_reasoning_content_from_streaming_chunk(openai_unit_test_env: dict[str, str]) -> None:
def test_parse_text_reasoning_content_from_streaming_chunk(
openai_unit_test_env: dict[str, str],
) -> None:
"""Test that TextReasoningContent is correctly parsed from streaming OpenAI chunk with reasoning_details."""
from openai.types.chat.chat_completion_chunk import ChatCompletionChunk
from openai.types.chat.chat_completion_chunk import Choice as ChunkChoice
@@ -611,7 +670,9 @@ def test_parse_text_reasoning_content_from_streaming_chunk(openai_unit_test_env:
assert parsed_details == mock_reasoning_details
def test_prepare_message_with_text_reasoning_content(openai_unit_test_env: dict[str, str]) -> None:
def test_prepare_message_with_text_reasoning_content(
openai_unit_test_env: dict[str, str],
) -> None:
"""Test that TextReasoningContent with protected_data is correctly prepared for OpenAI."""
client = OpenAIChatClient()
@@ -643,7 +704,9 @@ def test_prepare_message_with_text_reasoning_content(openai_unit_test_env: dict[
assert prepared[0]["content"] == "The answer is 42."
def test_prepare_message_with_only_text_reasoning_content(openai_unit_test_env: dict[str, str]) -> None:
def test_prepare_message_with_only_text_reasoning_content(
openai_unit_test_env: dict[str, str],
) -> None:
"""Test that a message with only text_reasoning content does not raise IndexError.
Regression test for https://github.com/microsoft/agent-framework/issues/4384
@@ -677,7 +740,9 @@ def test_prepare_message_with_only_text_reasoning_content(openai_unit_test_env:
assert prepared[0]["content"] == ""
def test_prepare_message_with_text_reasoning_before_text(openai_unit_test_env: dict[str, str]) -> None:
def test_prepare_message_with_text_reasoning_before_text(
openai_unit_test_env: dict[str, str],
) -> None:
"""Test that text_reasoning content appearing before text content is handled correctly.
Regression test for https://github.com/microsoft/agent-framework/issues/4384
@@ -711,7 +776,9 @@ def test_prepare_message_with_text_reasoning_before_text(openai_unit_test_env: d
assert prepared[0]["content"] == "The answer is 42."
def test_prepare_message_with_text_reasoning_before_function_call(openai_unit_test_env: dict[str, str]) -> None:
def test_prepare_message_with_text_reasoning_before_function_call(
openai_unit_test_env: dict[str, str],
) -> None:
"""Test that text_reasoning content appearing before a function call is handled correctly.
Regression test for https://github.com/microsoft/agent-framework/issues/4384
@@ -747,7 +814,9 @@ def test_prepare_message_with_text_reasoning_before_function_call(openai_unit_te
assert prepared[0]["role"] == "assistant"
def test_function_approval_content_is_skipped_in_preparation(openai_unit_test_env: dict[str, str]) -> None:
def test_function_approval_content_is_skipped_in_preparation(
openai_unit_test_env: dict[str, str],
) -> None:
"""Test that function approval request and response content are skipped."""
client = OpenAIChatClient()
@@ -793,7 +862,9 @@ def test_function_approval_content_is_skipped_in_preparation(openai_unit_test_en
assert prepared_mixed[0]["content"] == "I need approval for this action."
def test_usage_content_in_streaming_response(openai_unit_test_env: dict[str, str]) -> None:
def test_usage_content_in_streaming_response(
openai_unit_test_env: dict[str, str],
) -> None:
"""Test that UsageContent is correctly parsed from streaming response with usage data."""
from openai.types.chat.chat_completion_chunk import ChatCompletionChunk
from openai.types.completion_usage import CompletionUsage
@@ -829,13 +900,19 @@ def test_usage_content_in_streaming_response(openai_unit_test_env: dict[str, str
assert usage_content.usage_details["total_token_count"] == 150
def test_streaming_chunk_with_usage_and_text(openai_unit_test_env: dict[str, str]) -> None:
def test_streaming_chunk_with_usage_and_text(
openai_unit_test_env: dict[str, str],
) -> None:
"""Test that text content is not lost when usage data is in the same chunk.
Some providers (e.g. Gemini) include both usage and text content in the
same streaming chunk. See https://github.com/microsoft/agent-framework/issues/3434
"""
from openai.types.chat.chat_completion_chunk import ChatCompletionChunk, Choice, ChoiceDelta
from openai.types.chat.chat_completion_chunk import (
ChatCompletionChunk,
Choice,
ChoiceDelta,
)
from openai.types.completion_usage import CompletionUsage
client = OpenAIChatClient()
@@ -923,7 +1000,9 @@ def test_prepare_options_without_messages(openai_unit_test_env: dict[str, str])
client._prepare_options([], {})
def test_prepare_tools_with_web_search_no_location(openai_unit_test_env: dict[str, str]) -> None:
def test_prepare_tools_with_web_search_no_location(
openai_unit_test_env: dict[str, str],
) -> None:
"""Test preparing web search tool without user location."""
client = OpenAIChatClient()
@@ -937,7 +1016,9 @@ def test_prepare_tools_with_web_search_no_location(openai_unit_test_env: dict[st
assert result["web_search_options"] == {}
def test_prepare_options_with_instructions(openai_unit_test_env: dict[str, str]) -> None:
def test_prepare_options_with_instructions(
openai_unit_test_env: dict[str, str],
) -> None:
"""Test that instructions are prepended as system message."""
client = OpenAIChatClient()
@@ -969,7 +1050,9 @@ def test_prepare_message_with_author_name(openai_unit_test_env: dict[str, str])
assert prepared[0]["name"] == "TestUser"
def test_prepare_message_with_tool_result_author_name(openai_unit_test_env: dict[str, str]) -> None:
def test_prepare_message_with_tool_result_author_name(
openai_unit_test_env: dict[str, str],
) -> None:
"""Test that author_name is not included for TOOL role messages."""
client = OpenAIChatClient()
@@ -987,7 +1070,9 @@ def test_prepare_message_with_tool_result_author_name(openai_unit_test_env: dict
assert "name" not in prepared[0]
def test_prepare_system_message_content_is_string(openai_unit_test_env: dict[str, str]) -> None:
def test_prepare_system_message_content_is_string(
openai_unit_test_env: dict[str, str],
) -> None:
"""Test that system message content is a plain string, not a list.
Some OpenAI-compatible endpoints (e.g. NVIDIA NIM) reject system messages
@@ -1005,7 +1090,9 @@ def test_prepare_system_message_content_is_string(openai_unit_test_env: dict[str
assert prepared[0]["content"] == "You are a helpful assistant."
def test_prepare_developer_message_content_is_string(openai_unit_test_env: dict[str, str]) -> None:
def test_prepare_developer_message_content_is_string(
openai_unit_test_env: dict[str, str],
) -> None:
"""Test that developer message content is a plain string, not a list."""
client = OpenAIChatClient()
@@ -1019,7 +1106,9 @@ def test_prepare_developer_message_content_is_string(openai_unit_test_env: dict[
assert prepared[0]["content"] == "Follow these rules."
def test_prepare_system_message_multiple_text_contents_joined(openai_unit_test_env: dict[str, str]) -> None:
def test_prepare_system_message_multiple_text_contents_joined(
openai_unit_test_env: dict[str, str],
) -> None:
"""Test that system messages with multiple text contents are joined into a single string."""
client = OpenAIChatClient()
@@ -1039,7 +1128,9 @@ def test_prepare_system_message_multiple_text_contents_joined(openai_unit_test_e
assert prepared[0]["content"] == "You are a helpful assistant.\nBe concise."
def test_prepare_user_message_text_content_is_string(openai_unit_test_env: dict[str, str]) -> None:
def test_prepare_user_message_text_content_is_string(
openai_unit_test_env: dict[str, str],
) -> None:
"""Test that text-only user message content is flattened to a plain string.
Some OpenAI-compatible endpoints (e.g. Foundry Local) cannot deserialize
@@ -1057,7 +1148,9 @@ def test_prepare_user_message_text_content_is_string(openai_unit_test_env: dict[
assert prepared[0]["content"] == "Hello"
def test_prepare_user_message_multimodal_content_remains_list(openai_unit_test_env: dict[str, str]) -> None:
def test_prepare_user_message_multimodal_content_remains_list(
openai_unit_test_env: dict[str, str],
) -> None:
"""Test that multimodal user message content remains a list."""
client = OpenAIChatClient()
@@ -1076,7 +1169,9 @@ def test_prepare_user_message_multimodal_content_remains_list(openai_unit_test_e
assert has_list_content
def test_prepare_assistant_message_text_content_is_string(openai_unit_test_env: dict[str, str]) -> None:
def test_prepare_assistant_message_text_content_is_string(
openai_unit_test_env: dict[str, str],
) -> None:
"""Test that text-only assistant message content is flattened to a plain string."""
client = OpenAIChatClient()
@@ -1090,7 +1185,9 @@ def test_prepare_assistant_message_text_content_is_string(openai_unit_test_env:
assert prepared[0]["content"] == "Sure, I can help."
def test_tool_choice_required_with_function_name(openai_unit_test_env: dict[str, str]) -> None:
def test_tool_choice_required_with_function_name(
openai_unit_test_env: dict[str, str],
) -> None:
"""Test that tool_choice with required mode and function name is correctly prepared."""
client = OpenAIChatClient()
@@ -1125,7 +1222,9 @@ def test_response_format_dict_passthrough(openai_unit_test_env: dict[str, str])
assert prepared_options["response_format"] == custom_format
def test_multiple_function_calls_in_single_message(openai_unit_test_env: dict[str, str]) -> None:
def test_multiple_function_calls_in_single_message(
openai_unit_test_env: dict[str, str],
) -> None:
"""Test that multiple function calls in a message are correctly prepared."""
client = OpenAIChatClient()
@@ -1148,7 +1247,9 @@ def test_multiple_function_calls_in_single_message(openai_unit_test_env: dict[st
assert prepared[0]["tool_calls"][1]["id"] == "call_2"
def test_prepare_options_removes_parallel_tool_calls_when_no_tools(openai_unit_test_env: dict[str, str]) -> None:
def test_prepare_options_removes_parallel_tool_calls_when_no_tools(
openai_unit_test_env: dict[str, str],
) -> None:
"""Test that parallel_tool_calls is removed when no tools are present."""
client = OpenAIChatClient()
@@ -1176,7 +1277,9 @@ def test_prepare_options_excludes_conversation_id(openai_unit_test_env: dict[str
assert prepared_options["temperature"] == 0.7
async def test_streaming_exception_handling(openai_unit_test_env: dict[str, str]) -> None:
async def test_streaming_exception_handling(
openai_unit_test_env: dict[str, str],
) -> None:
"""Test that streaming errors are properly handled."""
client = OpenAIChatClient()
messages = [Message(role="user", text="test")]
@@ -1220,7 +1323,12 @@ class OutputStruct(BaseModel):
param("allow_multiple_tool_calls", True, False, id="allow_multiple_tool_calls"),
# OpenAIChatOptions - just verify they don't fail
param("logit_bias", {"50256": -1}, False, id="logit_bias"),
param("prediction", {"type": "content", "content": "hello world"}, False, id="prediction"),
param(
"prediction",
{"type": "content", "content": "hello world"},
False,
id="prediction",
),
# Complex options requiring output validation
param("tools", [get_weather], True, id="tools_function"),
param("tool_choice", "auto", True, id="tool_choice_auto"),
@@ -1249,7 +1357,12 @@ class OutputStruct(BaseModel):
"temperature_c": {"type": "number"},
"advisory": {"type": "string"},
},
"required": ["location", "conditions", "temperature_c", "advisory"],
"required": [
"location",
"conditions",
"temperature_c",
"advisory",
],
"additionalProperties": False,
},
},
@@ -1383,7 +1496,12 @@ async def test_integration_web_search() -> None:
}
)
content = {
"messages": [Message(role="user", text="What is the current weather? Do not ask for my current location.")],
"messages": [
Message(
role="user",
text="What is the current weather? Do not ask for my current location.",
)
],
"options": {
"tool_choice": "auto",
"tools": [web_search_tool_with_location],
python/packages/core/tests/openai/test_openai_responses_client.py
+134 -14
View File
@@ -4,6 +4,7 @@ import base64
import json
import os
from datetime import datetime, timezone
from pathlib import Path
from typing import Annotated, Any
from unittest.mock import MagicMock, patch
@@ -36,7 +37,10 @@ from agent_framework import (
SupportsChatGetResponse,
tool,
)
from agent_framework.exceptions import ChatClientException, ChatClientInvalidRequestException
from agent_framework.exceptions import (
ChatClientException,
ChatClientInvalidRequestException,
)
from agent_framework.openai import OpenAIResponsesClient
from agent_framework.openai._exceptions import OpenAIContentFilterException
from agent_framework.openai._responses_client import OPENAI_LOCAL_SHELL_CALL_ITEM_ID_KEY
@@ -1313,7 +1317,10 @@ def test_prepare_messages_for_openai_full_conversation_with_reasoning() -> None:
),
],
),
Message(role="assistant", contents=[Content.from_text(text="I found hotels for you")]),
Message(
role="assistant",
contents=[Content.from_text(text="I found hotels for you")],
),
]
result = client._prepare_messages_for_openai(messages)
@@ -1422,10 +1429,16 @@ def test_response_format_with_conflicting_definitions() -> None:
client = OpenAIResponsesClient(model_id="test-model", api_key="test-key")
# Mock response_format and text_config that conflict
response_format = {"type": "json_schema", "format": {"type": "json_schema", "name": "Test", "schema": {}}}
response_format = {
"type": "json_schema",
"format": {"type": "json_schema", "name": "Test", "schema": {}},
}
text_config = {"format": {"type": "json_object"}}
with pytest.raises(ChatClientInvalidRequestException, match="Conflicting response_format definitions"):
with pytest.raises(
ChatClientInvalidRequestException,
match="Conflicting response_format definitions",
):
client._prepare_response_and_text_format(response_format=response_format, text_config=text_config)
@@ -1457,7 +1470,13 @@ def test_response_format_with_format_key() -> None:
"""Test response_format that already has a format key."""
client = OpenAIResponsesClient(model_id="test-model", api_key="test-key")
response_format = {"format": {"type": "json_schema", "name": "MySchema", "schema": {"type": "object"}}}
response_format = {
"format": {
"type": "json_schema",
"name": "MySchema",
"schema": {"type": "object"},
}
}
_, text_config = client._prepare_response_and_text_format(response_format=response_format, text_config=None)
@@ -1487,7 +1506,11 @@ def test_response_format_json_schema_with_strict() -> None:
response_format = {
"type": "json_schema",
"json_schema": {"name": "StrictSchema", "schema": {"type": "object"}, "strict": True},
"json_schema": {
"name": "StrictSchema",
"schema": {"type": "object"},
"strict": True,
},
}
_, text_config = client._prepare_response_and_text_format(response_format=response_format, text_config=None)
@@ -1521,7 +1544,10 @@ def test_response_format_json_schema_missing_schema() -> None:
response_format = {"type": "json_schema", "json_schema": {"name": "NoSchema"}}
with pytest.raises(ChatClientInvalidRequestException, match="json_schema response_format requires a schema"):
with pytest.raises(
ChatClientInvalidRequestException,
match="json_schema response_format requires a schema",
):
client._prepare_response_and_text_format(response_format=response_format, text_config=None)
@@ -1541,7 +1567,10 @@ def test_response_format_invalid_type() -> None:
response_format = "invalid" # Not a Pydantic model or mapping
with pytest.raises(ChatClientInvalidRequestException, match="response_format must be a Pydantic model or mapping"):
with pytest.raises(
ChatClientInvalidRequestException,
match="response_format must be a Pydantic model or mapping",
):
client._prepare_response_and_text_format(response_format=response_format, text_config=None) # type: ignore
@@ -2198,7 +2227,9 @@ async def test_get_response_streaming_with_response_format() -> None:
async def run_streaming():
async for _ in client.get_response(
stream=True, messages=messages, options={"response_format": OutputStruct}
stream=True,
messages=messages,
options={"response_format": OutputStruct},
):
pass
@@ -2262,6 +2293,45 @@ def test_prepare_content_for_openai_unsupported_content() -> None:
assert result == {}
def test_prepare_content_for_openai_function_result_with_rich_items() -> None:
"""Test _prepare_content_for_openai with function_result containing rich items."""
client = OpenAIResponsesClient(model_id="test-model", api_key="test-key")
image_content = Content.from_data(data=b"image_bytes", media_type="image/png")
content = Content.from_function_result(
call_id="call_rich",
result=[Content.from_text("Result text"), image_content],
)
result = client._prepare_content_for_openai("user", content, {}) # type: ignore
assert result["type"] == "function_call_output"
assert result["call_id"] == "call_rich"
# Output should be a list with text and image parts
output = result["output"]
assert isinstance(output, list)
assert len(output) == 2
assert output[0]["type"] == "input_text"
assert output[0]["text"] == "Result text"
assert output[1]["type"] == "input_image"
def test_prepare_content_for_openai_function_result_without_items() -> None:
"""Test _prepare_content_for_openai with plain string function_result."""
client = OpenAIResponsesClient(model_id="test-model", api_key="test-key")
content = Content.from_function_result(
call_id="call_plain",
result="Simple result",
)
result = client._prepare_content_for_openai("user", content, {}) # type: ignore
assert result["type"] == "function_call_output"
assert result["call_id"] == "call_plain"
assert result["output"] == "Simple result"
def test_parse_chunk_from_openai_code_interpreter() -> None:
"""Test _parse_chunk_from_openai with code_interpreter_call."""
client = OpenAIResponsesClient(model_id="test-model", api_key="test-key")
@@ -2778,7 +2848,10 @@ async def test_instructions_sent_first_turn_then_skipped_for_continuation() -> N
await client.get_response(
messages=[Message(role="user", text="Tell me a joke")],
options={"instructions": "Reply in uppercase.", "conversation_id": "resp_123"},
options={
"instructions": "Reply in uppercase.",
"conversation_id": "resp_123",
},
)
second_input_messages = mock_create.call_args.kwargs["input"]
@@ -2788,7 +2861,9 @@ async def test_instructions_sent_first_turn_then_skipped_for_continuation() -> N
@pytest.mark.parametrize("conversation_id", ["resp_456", "conv_abc123"])
async def test_instructions_not_repeated_for_continuation_ids(conversation_id: str) -> None:
async def test_instructions_not_repeated_for_continuation_ids(
conversation_id: str,
) -> None:
client = OpenAIResponsesClient(model_id="test-model", api_key="test-key")
mock_response = _create_mock_responses_text_response(response_id="resp_456")
@@ -2889,7 +2964,12 @@ def test_with_callable_api_key() -> None:
"temperature_c": {"type": "number"},
"advisory": {"type": "string"},
},
"required": ["location", "conditions", "temperature_c", "advisory"],
"required": [
"location",
"conditions",
"temperature_c",
"advisory",
],
"additionalProperties": False,
},
},
@@ -3014,7 +3094,12 @@ async def test_integration_web_search() -> None:
user_location={"country": "US", "city": "Seattle"},
)
content = {
"messages": [Message(role="user", text="What is the current weather? Do not ask for my current location.")],
"messages": [
Message(
role="user",
text="What is the current weather? Do not ask for my current location.",
)
],
"options": {
"tool_choice": "auto",
"tools": [web_search_tool_with_location],
@@ -3105,7 +3190,42 @@ async def test_integration_streaming_file_search() -> None:
assert "75" in full_message
# region Background Response / ContinuationToken Tests
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_openai_integration_tests_disabled
async def test_integration_tool_rich_content_image() -> None:
"""Integration test: a tool returns an image and the model describes it."""
image_path = Path(__file__).parent.parent / "assets" / "sample_image.jpg"
image_bytes = image_path.read_bytes()
@tool(approval_mode="never_require")
def get_test_image() -> Content:
"""Return a test image for analysis."""
return Content.from_data(data=image_bytes, media_type="image/jpeg")
client = OpenAIResponsesClient()
client.function_invocation_configuration["max_iterations"] = 2
for streaming in [False, True]:
messages = [
Message(
role="user",
text="Call the get_test_image tool and describe what you see.",
)
]
options: dict[str, Any] = {"tools": [get_test_image], "tool_choice": "auto"}
if streaming:
response = await client.get_response(messages=messages, stream=True, options=options).get_final_response()
else:
response = await client.get_response(messages=messages, options=options)
assert response is not None
assert isinstance(response, ChatResponse)
assert response.text is not None
assert len(response.text) > 0
# sample_image.jpg contains a photo of a house; the model should mention it.
assert "house" in response.text.lower(), f"Model did not describe the house image. Response: {response.text}"
def test_continuation_token_json_serializable() -> None: