agent/agent-framework
1
0
Fork 0
mirror of https://github.com/microsoft/agent-framework.git synced 2026-06-16 21:04:09 +08:00
Code Issues Packages Projects Releases Wiki Activity

Python: [BREAKING] Python: Provider-leading client design & OpenAI package extraction (#4818)

Browse Source 
* Python: Provider-leading client design & OpenAI package extraction

Major refactoring of the Python Agent Framework client architecture:

- Extract OpenAI clients into new `agent-framework-openai` package
- Core package no longer depends on openai, azure-identity, azure-ai-projects
- Rename clients for discoverability: OpenAIResponsesClient → OpenAIChatClient,
  OpenAIChatClient → OpenAIChatCompletionClient
- Unify `model_id`/`deployment_name`/`model_deployment_name` → `model` param
- New FoundryChatClient for Azure AI Foundry Responses API
- New FoundryAgent/FoundryAgentClient for connecting to pre-configured Foundry agents
- Remove OpenAIBase/OpenAIConfigMixin from non-deprecated client MRO
- Deprecate AzureOpenAI* clients, AzureAIClient, OpenAIAssistantsClient
- Reorganize samples: azure_openai+azure_ai+azure_ai_agent → azure/
- ADR-0020: Provider-Leading Client Design

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: missing Agent imports in samples, .model_id → .model in foundry_local sample

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: CI failures — mypy errors, coverage targets, sample imports

- azure-ai mypy: add type ignores for TypedDict total=, model arg, forward ref
- Coverage: replace core.azure/openai targets with openai package target
- project_provider: add type annotation for opts dict

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: populate openai .pyi stub, fix broken README links, coverage targets

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fixes

* updated observabilitty

* reset azure init.pyi

* fix errors

* updated adr number

* fix foundry local

* fixed not renamed docstrings and comments, and added deprecated markers to old classes

* fix tests and pyprojects

* fix test vars

* updated function tests

* update durable

* updated test setup for functions

* Fix Foundry auth in workflow samples

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Stabilize Python integration workflows

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update hosting samples for Foundry

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Trigger full CI rerun

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Trigger CI rerun again

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* trigger rerun

* trigger rerun

* fix for litellm

* undo durabletask changes

* Move Foundry APIs into foundry namespace

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix Foundry pyproject formatting

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Split provider samples by Foundry surface

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Restore hosting sample requirements

Also fix the Foundry Local sample link after the provider sample move.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* updated tests

* udpated foundry integration tests

* removed dist from azurefunctions tests

* Use separate Foundry clients for concurrent agents

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix client setup in azfunc and durable

* disabled two tests

* updated setup for some function and durable tests

* improved azure openai setup with new clients

* ignore deprecated

* fixes

* skip 11

* remove openai assistants int tests

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
Eduard van Valkenburg
2026-03-25 10:56:29 +01:00
committed by GitHub
Unverified
parent 4b533608b6
commit 5e056b672e
485 changed files with 9784 additions and 12084 deletions
Download Patch File Download Diff File Expand all files Collapse all files
python/packages/core/tests/azure/conftest.py
-62
View File
@@ -1,62 +0,0 @@
# Copyright (c) Microsoft. All rights reserved.
from typing import Any
from pytest import fixture
from agent_framework import Message
# region: Connector Settings fixtures
@fixture
def exclude_list(request: Any) -> list[str]:
"""Fixture that returns a list of environment variables to exclude."""
return request.param if hasattr(request, "param") else []
@fixture
def override_env_param_dict(request: Any) -> dict[str, str]:
"""Fixture that returns a dict of environment variables to override."""
return request.param if hasattr(request, "param") else {}
# These two fixtures are used for multiple things, also non-connector tests
@fixture()
def azure_openai_unit_test_env(monkeypatch, exclude_list, override_env_param_dict): # type: ignore
"""Fixture to set environment variables for AzureOpenAISettings."""
if exclude_list is None:
exclude_list = []
if override_env_param_dict is None:
override_env_param_dict = {}
env_vars = {
"AZURE_OPENAI_ENDPOINT": "https://test-endpoint.com",
"AZURE_OPENAI_CHAT_DEPLOYMENT_NAME": "test_chat_deployment",
"AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME": "test_chat_deployment",
"AZURE_OPENAI_TEXT_DEPLOYMENT_NAME": "test_text_deployment",
"AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME": "test_embedding_deployment",
"AZURE_OPENAI_TEXT_TO_IMAGE_DEPLOYMENT_NAME": "test_text_to_image_deployment",
"AZURE_OPENAI_AUDIO_TO_TEXT_DEPLOYMENT_NAME": "test_audio_to_text_deployment",
"AZURE_OPENAI_TEXT_TO_AUDIO_DEPLOYMENT_NAME": "test_text_to_audio_deployment",
"AZURE_OPENAI_REALTIME_DEPLOYMENT_NAME": "test_realtime_deployment",
"AZURE_OPENAI_API_KEY": "test_api_key",
"AZURE_OPENAI_API_VERSION": "2023-03-15-preview",
"AZURE_OPENAI_BASE_URL": "https://test_text_deployment.test-base-url.com",
"AZURE_OPENAI_TOKEN_ENDPOINT": "https://test-token-endpoint.com",
}
env_vars.update(override_env_param_dict) # type: ignore
for key, value in env_vars.items():
if key in exclude_list:
monkeypatch.delenv(key, raising=False) # type: ignore
continue
monkeypatch.setenv(key, value) # type: ignore
return env_vars
@fixture(scope="function")
def chat_history() -> list[Message]:
return []
python/packages/core/tests/azure/test_azure_assistants_client.py
-725
View File
@@ -1,725 +0,0 @@
# Copyright (c) Microsoft. All rights reserved.
import os
from typing import Annotated
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from azure.identity import AzureCliCredential
from pydantic import Field
from agent_framework import (
Agent,
AgentResponse,
AgentResponseUpdate,
AgentSession,
ChatResponse,
ChatResponseUpdate,
Message,
SupportsChatGetResponse,
tool,
)
from agent_framework._settings import SecretString
from agent_framework.azure import AzureOpenAIAssistantsClient
skip_if_azure_integration_tests_disabled = pytest.mark.skipif(
os.getenv("AZURE_OPENAI_ENDPOINT", "") in ("", "https://test-endpoint.com"),
reason="No real AZURE_OPENAI_ENDPOINT provided; skipping integration tests.",
)
def create_test_azure_assistants_client(
mock_async_azure_openai: MagicMock,
deployment_name: str | None = None,
assistant_id: str | None = None,
assistant_name: str | None = None,
thread_id: str | None = None,
should_delete_assistant: bool = False,
) -> AzureOpenAIAssistantsClient:
"""Helper function to create AzureOpenAIAssistantsClient instances for testing."""
client = AzureOpenAIAssistantsClient(
deployment_name=deployment_name or "test_chat_deployment",
assistant_id=assistant_id,
assistant_name=assistant_name,
thread_id=thread_id,
api_key="test-api-key",
endpoint="https://test-endpoint.com",
async_client=mock_async_azure_openai,
)
# Set the _should_delete_assistant flag directly if needed
if should_delete_assistant:
object.__setattr__(client, "_should_delete_assistant", True)
return client
@pytest.fixture
def mock_async_azure_openai() -> MagicMock:
"""Mock AsyncAzureOpenAI client."""
mock_client = MagicMock()
# Mock beta.assistants
mock_client.beta.assistants.create = AsyncMock(return_value=MagicMock(id="test-assistant-id"))
mock_client.beta.assistants.delete = AsyncMock()
# Mock beta.threads
mock_client.beta.threads.create = AsyncMock(return_value=MagicMock(id="test-thread-id"))
mock_client.beta.threads.delete = AsyncMock()
# Mock beta.threads.runs
mock_client.beta.threads.runs.create = AsyncMock(return_value=MagicMock(id="test-run-id"))
mock_client.beta.threads.runs.retrieve = AsyncMock()
mock_client.beta.threads.runs.submit_tool_outputs = AsyncMock()
# Mock beta.threads.messages
mock_client.beta.threads.messages.create = AsyncMock()
mock_client.beta.threads.messages.list = AsyncMock(return_value=MagicMock(data=[]))
return mock_client
def test_azure_assistants_client_init_with_client(mock_async_azure_openai: MagicMock) -> None:
"""Test AzureOpenAIAssistantsClient initialization with existing client."""
client = create_test_azure_assistants_client(
mock_async_azure_openai,
deployment_name="test_chat_deployment",
assistant_id="existing-assistant-id",
thread_id="test-thread-id",
)
assert client.client is mock_async_azure_openai
assert client.model_id == "test_chat_deployment"
assert client.assistant_id == "existing-assistant-id"
assert client.thread_id == "test-thread-id"
assert not client._should_delete_assistant # type: ignore
assert isinstance(client, SupportsChatGetResponse)
def test_azure_assistants_client_init_auto_create_client(
azure_openai_unit_test_env: dict[str, str],
mock_async_azure_openai: MagicMock,
) -> None:
"""Test AzureOpenAIAssistantsClient initialization with auto-created client."""
client = AzureOpenAIAssistantsClient(
deployment_name=azure_openai_unit_test_env["AZURE_OPENAI_CHAT_DEPLOYMENT_NAME"],
assistant_name="TestAssistant",
api_key=azure_openai_unit_test_env["AZURE_OPENAI_API_KEY"],
endpoint=azure_openai_unit_test_env["AZURE_OPENAI_ENDPOINT"],
async_client=mock_async_azure_openai,
)
assert client.client is mock_async_azure_openai
assert client.model_id == azure_openai_unit_test_env["AZURE_OPENAI_CHAT_DEPLOYMENT_NAME"]
assert client.assistant_id is None
assert client.assistant_name == "TestAssistant"
assert not client._should_delete_assistant # type: ignore
def test_azure_assistants_client_init_validation_fail() -> None:
"""Test AzureOpenAIAssistantsClient initialization with validation failure."""
with pytest.raises(ValueError):
# Force failure by providing invalid deployment name type - this should cause validation to fail
AzureOpenAIAssistantsClient(deployment_name=123, api_key="valid-key") # type: ignore
@pytest.mark.parametrize("exclude_list", [["AZURE_OPENAI_CHAT_DEPLOYMENT_NAME"]], indirect=True)
def test_azure_assistants_client_init_missing_deployment_name(azure_openai_unit_test_env: dict[str, str]) -> None:
"""Test AzureOpenAIAssistantsClient initialization with missing deployment name."""
with pytest.raises(ValueError):
AzureOpenAIAssistantsClient(api_key=azure_openai_unit_test_env.get("AZURE_OPENAI_API_KEY", "test-key"))
def test_azure_assistants_client_init_with_default_headers(azure_openai_unit_test_env: dict[str, str]) -> None:
"""Test AzureOpenAIAssistantsClient initialization with default headers."""
default_headers = {"X-Unit-Test": "test-guid"}
client = AzureOpenAIAssistantsClient(
deployment_name="test_chat_deployment",
api_key=azure_openai_unit_test_env["AZURE_OPENAI_API_KEY"],
endpoint=azure_openai_unit_test_env["AZURE_OPENAI_ENDPOINT"],
default_headers=default_headers,
)
assert client.model_id == "test_chat_deployment"
assert isinstance(client, SupportsChatGetResponse)
# Assert that the default header we added is present in the client's default headers
for key, value in default_headers.items():
assert key in client.client.default_headers
assert client.client.default_headers[key] == value
async def test_azure_assistants_client_get_assistant_id_or_create_existing_assistant(
mock_async_azure_openai: MagicMock,
) -> None:
"""Test _get_assistant_id_or_create when assistant_id is already provided."""
client = create_test_azure_assistants_client(mock_async_azure_openai, assistant_id="existing-assistant-id")
assistant_id = await client._get_assistant_id_or_create() # type: ignore
assert assistant_id == "existing-assistant-id"
assert not client._should_delete_assistant # type: ignore
mock_async_azure_openai.beta.assistants.create.assert_not_called()
async def test_azure_assistants_client_get_assistant_id_or_create_create_new(
mock_async_azure_openai: MagicMock,
) -> None:
"""Test _get_assistant_id_or_create when creating a new assistant."""
client = create_test_azure_assistants_client(
mock_async_azure_openai, deployment_name="test_chat_deployment", assistant_name="TestAssistant"
)
assistant_id = await client._get_assistant_id_or_create() # type: ignore
assert assistant_id == "test-assistant-id"
assert client._should_delete_assistant # type: ignore
mock_async_azure_openai.beta.assistants.create.assert_called_once()
async def test_azure_assistants_client_aclose_should_not_delete(
mock_async_azure_openai: MagicMock,
) -> None:
"""Test close when assistant should not be deleted."""
client = create_test_azure_assistants_client(
mock_async_azure_openai, assistant_id="assistant-to-keep", should_delete_assistant=False
)
await client.close() # type: ignore
# Verify assistant deletion was not called
mock_async_azure_openai.beta.assistants.delete.assert_not_called()
assert not client._should_delete_assistant # type: ignore
async def test_azure_assistants_client_aclose_should_delete(mock_async_azure_openai: MagicMock) -> None:
"""Test close method calls cleanup."""
client = create_test_azure_assistants_client(
mock_async_azure_openai, assistant_id="assistant-to-delete", should_delete_assistant=True
)
await client.close()
# Verify assistant deletion was called
mock_async_azure_openai.beta.assistants.delete.assert_called_once_with("assistant-to-delete")
assert not client._should_delete_assistant # type: ignore
async def test_azure_assistants_client_async_context_manager(mock_async_azure_openai: MagicMock) -> None:
"""Test async context manager functionality."""
client = create_test_azure_assistants_client(
mock_async_azure_openai, assistant_id="assistant-to-delete", should_delete_assistant=True
)
# Test context manager
async with client:
pass # Just test that we can enter and exit
# Verify cleanup was called on exit
mock_async_azure_openai.beta.assistants.delete.assert_called_once_with("assistant-to-delete")
def test_azure_assistants_client_serialize(azure_openai_unit_test_env: dict[str, str]) -> None:
"""Test serialization of AzureOpenAIAssistantsClient."""
default_headers = {"X-Unit-Test": "test-guid"}
# Test basic initialization and to_dict
client = AzureOpenAIAssistantsClient(
deployment_name="test_chat_deployment",
assistant_id="test-assistant-id",
assistant_name="TestAssistant",
thread_id="test-thread-id",
api_key=azure_openai_unit_test_env["AZURE_OPENAI_API_KEY"],
endpoint=azure_openai_unit_test_env["AZURE_OPENAI_ENDPOINT"],
default_headers=default_headers,
)
dumped_settings = client.to_dict()
assert dumped_settings["model_id"] == "test_chat_deployment"
assert dumped_settings["assistant_id"] == "test-assistant-id"
assert dumped_settings["assistant_name"] == "TestAssistant"
assert dumped_settings["thread_id"] == "test-thread-id"
# Assert that the default header we added is present in the dumped_settings default headers
for key, value in default_headers.items():
assert key in dumped_settings["default_headers"]
assert dumped_settings["default_headers"][key] == value
# Assert that the 'User-Agent' header is not present in the dumped_settings default headers
assert "User-Agent" not in dumped_settings["default_headers"]
@tool(approval_mode="never_require")
def get_weather(
location: Annotated[str, Field(description="The location to get the weather for.")],
) -> str:
"""Get the weather for a given location."""
return f"The weather in {location} is sunny with a high of 25°C."
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_azure_assistants_client_get_response() -> None:
"""Test Azure Assistants Client response."""
async with AzureOpenAIAssistantsClient(credential=AzureCliCredential()) as azure_assistants_client:
assert isinstance(azure_assistants_client, SupportsChatGetResponse)
messages: list[Message] = []
messages.append(
Message(
role="user",
text="The weather in Seattle is currently sunny with a high of 25°C. "
"It's a beautiful day for outdoor activities.",
)
)
messages.append(Message(role="user", text="What's the weather like today?"))
# Test that the client can be used to get a response
response = await azure_assistants_client.get_response(messages=messages)
assert response is not None
assert isinstance(response, ChatResponse)
assert any(word in response.text.lower() for word in ["sunny", "25", "weather", "seattle"])
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_azure_assistants_client_get_response_tools() -> None:
"""Test Azure Assistants Client response with tools."""
async with AzureOpenAIAssistantsClient(credential=AzureCliCredential()) as azure_assistants_client:
assert isinstance(azure_assistants_client, SupportsChatGetResponse)
messages: list[Message] = []
messages.append(Message(role="user", text="What's the weather like in Seattle?"))
# Test that the client can be used to get a response
response = await azure_assistants_client.get_response(
messages=messages,
options={"tools": [get_weather], "tool_choice": "auto"},
)
assert response is not None
assert isinstance(response, ChatResponse)
assert any(word in response.text.lower() for word in ["sunny", "25", "weather"])
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_azure_assistants_client_streaming() -> None:
"""Test Azure Assistants Client streaming response."""
async with AzureOpenAIAssistantsClient(credential=AzureCliCredential()) as azure_assistants_client:
assert isinstance(azure_assistants_client, SupportsChatGetResponse)
messages: list[Message] = []
messages.append(
Message(
role="user",
text="The weather in Seattle is currently sunny with a high of 25°C. "
"It's a beautiful day for outdoor activities.",
)
)
messages.append(Message(role="user", text="What's the weather like today?"))
# Test that the client can be used to get a response
response = azure_assistants_client.get_response(messages=messages, stream=True)
full_message: str = ""
async for chunk in response:
assert chunk is not None
assert isinstance(chunk, ChatResponseUpdate)
for content in chunk.contents:
if content.type == "text" and content.text:
full_message += content.text
assert any(word in full_message.lower() for word in ["sunny", "25", "weather", "seattle"])
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_azure_assistants_client_streaming_tools() -> None:
"""Test Azure Assistants Client streaming response with tools."""
async with AzureOpenAIAssistantsClient(credential=AzureCliCredential()) as azure_assistants_client:
assert isinstance(azure_assistants_client, SupportsChatGetResponse)
messages: list[Message] = []
messages.append(Message(role="user", text="What's the weather like in Seattle?"))
# Test that the client can be used to get a response
response = azure_assistants_client.get_response(
messages=messages,
options={"tools": [get_weather], "tool_choice": "auto"},
stream=True,
)
full_message: str = ""
async for chunk in response:
assert chunk is not None
assert isinstance(chunk, ChatResponseUpdate)
for content in chunk.contents:
if content.type == "text" and content.text:
full_message += content.text
assert any(word in full_message.lower() for word in ["sunny", "25", "weather"])
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_azure_assistants_client_with_existing_assistant() -> None:
"""Test Azure Assistants Client with existing assistant ID."""
# First create an assistant to use in the test
async with AzureOpenAIAssistantsClient(credential=AzureCliCredential()) as temp_client:
# Get the assistant ID by triggering assistant creation
messages = [Message(role="user", text="Hello")]
await temp_client.get_response(messages=messages)
assistant_id = temp_client.assistant_id
# Now test using the existing assistant
async with AzureOpenAIAssistantsClient(
assistant_id=assistant_id, credential=AzureCliCredential()
) as azure_assistants_client:
assert isinstance(azure_assistants_client, SupportsChatGetResponse)
assert azure_assistants_client.assistant_id == assistant_id
messages = [Message(role="user", text="What can you do?")]
# Test that the client can be used to get a response
response = await azure_assistants_client.get_response(messages=messages)
assert response is not None
assert isinstance(response, ChatResponse)
assert len(response.text) > 0
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_azure_assistants_agent_basic_run():
"""Test Agent basic run functionality with AzureOpenAIAssistantsClient."""
async with Agent(
client=AzureOpenAIAssistantsClient(credential=AzureCliCredential()),
) as agent:
# Run a simple query
response = await agent.run("Hello! Please respond with 'Hello World' exactly.")
# Validate response
assert isinstance(response, AgentResponse)
assert response.text is not None
assert len(response.text) > 0
assert "Hello World" in response.text
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_azure_assistants_agent_basic_run_streaming():
"""Test Agent basic streaming functionality with AzureOpenAIAssistantsClient."""
async with Agent(
client=AzureOpenAIAssistantsClient(credential=AzureCliCredential()),
) as agent:
# Run streaming query
full_message: str = ""
async for chunk in agent.run("Please respond with exactly: 'This is a streaming response test.'", stream=True):
assert chunk is not None
assert isinstance(chunk, AgentResponseUpdate)
if chunk.text:
full_message += chunk.text
# Validate streaming response
assert len(full_message) > 0
assert "streaming response test" in full_message.lower()
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_azure_assistants_agent_session_persistence():
"""Test Agent session persistence across runs with AzureOpenAIAssistantsClient."""
async with Agent(
client=AzureOpenAIAssistantsClient(credential=AzureCliCredential()),
instructions="You are a helpful assistant with good memory.",
) as agent:
# Create a new session that will be reused
session = agent.create_session()
# First message - establish context
first_response = await agent.run(
"Remember this number: 42. What number did I just tell you to remember?", session=session
)
assert isinstance(first_response, AgentResponse)
assert "42" in first_response.text
# Second message - test conversation memory
second_response = await agent.run(
"What number did I tell you to remember in my previous message?", session=session
)
assert isinstance(second_response, AgentResponse)
assert "42" in second_response.text
# Verify session has been populated with conversation ID
assert session.service_session_id is not None
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_azure_assistants_agent_existing_session_id():
"""Test Agent with existing session ID to continue conversations across agent instances."""
# First, create a conversation and capture the session ID
existing_session_id = None
async with Agent(
client=AzureOpenAIAssistantsClient(credential=AzureCliCredential()),
instructions="You are a helpful weather agent.",
tools=[get_weather],
) as agent:
# Start a conversation and get the session ID
session = agent.create_session()
response1 = await agent.run("What's the weather in Paris?", session=session)
# Validate first response
assert isinstance(response1, AgentResponse)
assert response1.text is not None
assert any(word in response1.text.lower() for word in ["weather", "paris"])
# The session ID is set after the first response
existing_session_id = session.service_session_id
assert existing_session_id is not None
# Now continue with the same session ID in a new agent instance
async with Agent(
client=AzureOpenAIAssistantsClient(thread_id=existing_session_id, credential=AzureCliCredential()),
instructions="You are a helpful weather agent.",
tools=[get_weather],
) as agent:
# Create a session with the existing ID
session = AgentSession(service_session_id=existing_session_id)
# Ask about the previous conversation
response2 = await agent.run("What was the last city I asked about?", session=session)
# Validate that the agent remembers the previous conversation
assert isinstance(response2, AgentResponse)
assert response2.text is not None
# Should reference Paris from the previous conversation
assert "paris" in response2.text.lower()
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_azure_assistants_agent_code_interpreter():
"""Test Agent with code interpreter through AzureOpenAIAssistantsClient."""
async with Agent(
client=AzureOpenAIAssistantsClient(credential=AzureCliCredential()),
instructions="You are a helpful assistant that can write and execute Python code.",
tools=[AzureOpenAIAssistantsClient.get_code_interpreter_tool()],
) as agent:
# Request code execution
response = await agent.run("Write Python code to calculate the factorial of 5 and show the result.")
# Validate response
assert isinstance(response, AgentResponse)
assert response.text is not None
# Factorial of 5 is 120
assert "120" in response.text or "factorial" in response.text.lower()
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_azure_assistants_client_agent_level_tool_persistence():
"""Test that agent-level tools persist across multiple runs with Azure Assistants Client."""
async with Agent(
client=AzureOpenAIAssistantsClient(credential=AzureCliCredential()),
instructions="You are a helpful assistant that uses available tools.",
tools=[get_weather], # Agent-level tool
) as agent:
# First run - agent-level tool should be available
first_response = await agent.run("What's the weather like in Chicago?")
assert isinstance(first_response, AgentResponse)
assert first_response.text is not None
# Should use the agent-level weather tool
assert any(term in first_response.text.lower() for term in ["chicago", "sunny", "72"])
# Second run - agent-level tool should still be available (persistence test)
second_response = await agent.run("What's the weather in Miami?")
assert isinstance(second_response, AgentResponse)
assert second_response.text is not None
# Should use the agent-level weather tool again
assert any(term in second_response.text.lower() for term in ["miami", "sunny", "72"])
def test_azure_assistants_client_entra_id_authentication() -> None:
"""Test credential authentication path with sync credential."""
mock_credential = MagicMock()
mock_provider = MagicMock(return_value="token-string")
with (
patch("agent_framework.azure._assistants_client.load_settings") as mock_load_settings,
patch(
"agent_framework.azure._assistants_client.resolve_credential_to_token_provider",
return_value=mock_provider,
) as mock_resolve,
patch("agent_framework.azure._assistants_client.AsyncAzureOpenAI") as mock_azure_client,
patch("agent_framework.openai.OpenAIAssistantsClient.__init__", return_value=None),
):
mock_load_settings.return_value = {
"chat_deployment_name": "test-deployment",
"responses_deployment_name": None,
"api_key": None,
"token_endpoint": "https://cognitiveservices.azure.com/.default",
"api_version": "2024-05-01-preview",
"endpoint": "https://test-endpoint.openai.azure.com",
"base_url": None,
}
client = AzureOpenAIAssistantsClient(
deployment_name="test-deployment",
endpoint="https://test-endpoint.openai.azure.com",
credential=mock_credential,
token_endpoint="https://cognitiveservices.azure.com/.default",
)
# Verify credential was resolved to a token provider
mock_resolve.assert_called_once_with(mock_credential, "https://cognitiveservices.azure.com/.default")
# Verify client was created with the token provider
mock_azure_client.assert_called_once()
call_args = mock_azure_client.call_args[1]
assert call_args["azure_ad_token_provider"] is mock_provider
assert client is not None
assert isinstance(client, AzureOpenAIAssistantsClient)
def test_azure_assistants_client_no_authentication_error() -> None:
"""Test authentication validation error when no auth provided."""
with patch("agent_framework.azure._assistants_client.load_settings") as mock_load_settings:
mock_load_settings.return_value = {
"chat_deployment_name": "test-deployment",
"responses_deployment_name": None,
"api_key": None,
"token_endpoint": None,
"api_version": "2024-05-01-preview",
"endpoint": "https://test-endpoint.openai.azure.com",
"base_url": None,
}
# Test missing authentication raises error
with pytest.raises(ValueError, match="api_key, credential, or a client"):
AzureOpenAIAssistantsClient(
deployment_name="test-deployment",
endpoint="https://test-endpoint.openai.azure.com",
# No authentication provided at all
)
def test_azure_assistants_client_callable_credential() -> None:
"""Test callable token provider as credential."""
mock_provider = MagicMock(return_value="my-token")
with (
patch("agent_framework.azure._assistants_client.load_settings") as mock_load_settings,
patch(
"agent_framework.azure._assistants_client.resolve_credential_to_token_provider",
return_value=mock_provider,
),
patch("agent_framework.azure._assistants_client.AsyncAzureOpenAI") as mock_azure_client,
patch("agent_framework.openai.OpenAIAssistantsClient.__init__", return_value=None),
):
mock_load_settings.return_value = {
"chat_deployment_name": "test-deployment",
"responses_deployment_name": None,
"api_key": None,
"token_endpoint": "https://cognitiveservices.azure.com/.default",
"api_version": "2024-05-01-preview",
"endpoint": "https://test-endpoint.openai.azure.com",
"base_url": None,
}
client = AzureOpenAIAssistantsClient(
deployment_name="test-deployment",
endpoint="https://test-endpoint.openai.azure.com",
credential=mock_provider,
token_endpoint="https://cognitiveservices.azure.com/.default",
)
# Verify client was created with the token provider
mock_azure_client.assert_called_once()
call_args = mock_azure_client.call_args[1]
assert call_args["azure_ad_token_provider"] is mock_provider
assert client is not None
assert isinstance(client, AzureOpenAIAssistantsClient)
def test_azure_assistants_client_base_url_configuration() -> None:
"""Test base_url client parameter path."""
with (
patch("agent_framework.azure._assistants_client.load_settings") as mock_load_settings,
patch("agent_framework.azure._assistants_client.AsyncAzureOpenAI") as mock_azure_client,
patch("agent_framework.openai.OpenAIAssistantsClient.__init__", return_value=None),
):
mock_load_settings.return_value = {
"chat_deployment_name": "test-deployment",
"responses_deployment_name": None,
"api_key": SecretString("test-api-key"),
"token_endpoint": None,
"api_version": "2024-05-01-preview",
"endpoint": None,
"base_url": "https://custom-base-url.com",
}
client = AzureOpenAIAssistantsClient(
deployment_name="test-deployment", api_key="test-api-key", base_url="https://custom-base-url.com"
)
# base_url path
mock_azure_client.assert_called_once()
call_args = mock_azure_client.call_args[1]
assert call_args["base_url"] == "https://custom-base-url.com"
assert "azure_endpoint" not in call_args
assert client is not None
assert isinstance(client, AzureOpenAIAssistantsClient)
def test_azure_assistants_client_azure_endpoint_configuration() -> None:
"""Test azure_endpoint client parameter path."""
with (
patch("agent_framework.azure._assistants_client.load_settings") as mock_load_settings,
patch("agent_framework.azure._assistants_client.AsyncAzureOpenAI") as mock_azure_client,
patch("agent_framework.openai.OpenAIAssistantsClient.__init__", return_value=None),
):
mock_load_settings.return_value = {
"chat_deployment_name": "test-deployment",
"responses_deployment_name": None,
"api_key": SecretString("test-api-key"),
"token_endpoint": None,
"api_version": "2024-05-01-preview",
"endpoint": "https://test-endpoint.openai.azure.com",
"base_url": None,
}
client = AzureOpenAIAssistantsClient(
deployment_name="test-deployment",
api_key="test-api-key",
endpoint="https://test-endpoint.openai.azure.com",
)
# azure_endpoint path
mock_azure_client.assert_called_once()
call_args = mock_azure_client.call_args[1]
assert call_args["azure_endpoint"] == "https://test-endpoint.openai.azure.com"
assert "base_url" not in call_args
assert client is not None
assert isinstance(client, AzureOpenAIAssistantsClient)
python/packages/core/tests/azure/test_azure_chat_client.py
-1064
View File
File diff suppressed because it is too large Load Diff
python/packages/core/tests/azure/test_azure_embedding_client.py
-159
View File
@@ -1,159 +0,0 @@
# Copyright (c) Microsoft. All rights reserved.
from __future__ import annotations
import os
from unittest.mock import AsyncMock, MagicMock
import pytest
from openai.types import CreateEmbeddingResponse
from openai.types import Embedding as OpenAIEmbedding
from openai.types.create_embedding_response import Usage
from agent_framework.azure import AzureOpenAIEmbeddingClient
from agent_framework.openai import OpenAIEmbeddingOptions
def _make_openai_response(
embeddings: list[list[float]],
model: str = "text-embedding-3-small",
prompt_tokens: int = 5,
total_tokens: int = 5,
) -> CreateEmbeddingResponse:
"""Helper to create a mock OpenAI embeddings response."""
data = [OpenAIEmbedding(embedding=emb, index=i, object="embedding") for i, emb in enumerate(embeddings)]
return CreateEmbeddingResponse(
data=data,
model=model,
object="list",
usage=Usage(prompt_tokens=prompt_tokens, total_tokens=total_tokens),
)
@pytest.fixture
def azure_embedding_unit_test_env(monkeypatch: pytest.MonkeyPatch) -> None:
"""Clear ambient Azure OpenAI embedding env vars for deterministic unit tests."""
for key in (
"AZURE_OPENAI_ENDPOINT",
"AZURE_OPENAI_API_KEY",
"AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME",
"AZURE_OPENAI_BASE_URL",
"AZURE_OPENAI_TOKEN_ENDPOINT",
):
monkeypatch.delenv(key, raising=False)
def test_azure_construction_with_deployment_name(azure_embedding_unit_test_env: None) -> None:
client = AzureOpenAIEmbeddingClient(
deployment_name="text-embedding-3-small",
api_key="test-key",
endpoint="https://test.openai.azure.com/",
)
assert client.model_id == "text-embedding-3-small"
def test_azure_construction_with_existing_client(azure_embedding_unit_test_env: None) -> None:
mock_client = MagicMock()
client = AzureOpenAIEmbeddingClient(
deployment_name="my-deployment",
async_client=mock_client,
)
assert client.model_id == "my-deployment"
assert client.client is mock_client
def test_azure_construction_missing_deployment_name_raises(azure_embedding_unit_test_env: None) -> None:
with pytest.raises(ValueError, match="deployment name is required"):
AzureOpenAIEmbeddingClient(
api_key="test-key",
endpoint="https://test.openai.azure.com/",
)
def test_azure_construction_missing_credentials_raises(azure_embedding_unit_test_env: None) -> None:
with pytest.raises(ValueError, match="api_key, credential, or a client"):
AzureOpenAIEmbeddingClient(
deployment_name="test",
endpoint="https://test.openai.azure.com/",
)
async def test_azure_get_embeddings(azure_embedding_unit_test_env: None) -> None:
mock_response = _make_openai_response(
embeddings=[[0.1, 0.2]],
)
mock_async_client = MagicMock()
mock_async_client.embeddings = MagicMock()
mock_async_client.embeddings.create = AsyncMock(return_value=mock_response)
client = AzureOpenAIEmbeddingClient(
deployment_name="text-embedding-3-small",
async_client=mock_async_client,
)
result = await client.get_embeddings(["hello"])
assert len(result) == 1
assert result[0].vector == [0.1, 0.2]
def test_azure_otel_provider_name(azure_embedding_unit_test_env: None) -> None:
mock_client = MagicMock()
client = AzureOpenAIEmbeddingClient(
deployment_name="test",
async_client=mock_client,
)
assert client.OTEL_PROVIDER_NAME == "azure.ai.openai"
skip_if_azure_openai_integration_tests_disabled = pytest.mark.skipif(
not os.getenv("AZURE_OPENAI_ENDPOINT")
or (not os.getenv("AZURE_OPENAI_API_KEY") and not os.getenv("AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME")),
reason="No Azure OpenAI credentials provided; skipping integration tests.",
)
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_openai_integration_tests_disabled
async def test_integration_azure_openai_get_embeddings() -> None:
"""End-to-end test of Azure OpenAI embedding generation."""
client = AzureOpenAIEmbeddingClient()
result = await client.get_embeddings(["hello world"])
assert len(result) == 1
assert isinstance(result[0].vector, list)
assert len(result[0].vector) > 0
assert all(isinstance(v, float) for v in result[0].vector)
assert result[0].model_id is not None
assert result.usage is not None
assert result.usage["input_token_count"] > 0
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_openai_integration_tests_disabled
async def test_integration_azure_openai_get_embeddings_multiple() -> None:
"""Test Azure OpenAI embedding generation for multiple inputs."""
client = AzureOpenAIEmbeddingClient()
result = await client.get_embeddings(["hello", "world", "test"])
assert len(result) == 3
dims = [len(e.vector) for e in result]
assert all(d == dims[0] for d in dims)
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_openai_integration_tests_disabled
async def test_integration_azure_openai_get_embeddings_with_dimensions() -> None:
"""Test Azure OpenAI embedding generation with custom dimensions."""
client = AzureOpenAIEmbeddingClient()
options: OpenAIEmbeddingOptions = {"dimensions": 256}
result = await client.get_embeddings(["hello world"], options=options)
assert len(result) == 1
assert len(result[0].vector) == 256
python/packages/core/tests/azure/test_azure_responses_client.py
-730
View File
@@ -1,730 +0,0 @@
# Copyright (c) Microsoft. All rights reserved.
import json
import logging
import os
from pathlib import Path
from typing import Annotated, Any
from unittest.mock import MagicMock
import pytest
from azure.identity import AzureCliCredential
from pydantic import BaseModel
from pytest import param
from agent_framework import (
Agent,
AgentResponse,
ChatResponse,
Content,
Message,
SupportsChatGetResponse,
tool,
)
from agent_framework.azure import AzureOpenAIResponsesClient
skip_if_azure_integration_tests_disabled = pytest.mark.skipif(
os.getenv("AZURE_OPENAI_ENDPOINT", "") in ("", "https://test-endpoint.com"),
reason="No real AZURE_OPENAI_ENDPOINT provided; skipping integration tests.",
)
logger = logging.getLogger(__name__)
class OutputStruct(BaseModel):
"""A structured output for testing purposes."""
location: str
weather: str
@tool(approval_mode="never_require")
async def get_weather(location: Annotated[str, "The location as a city name"]) -> str:
"""Get the current weather in a given location."""
# Implementation of the tool to get weather
return f"The weather in {location} is sunny and 72°F."
async def create_vector_store(
client: AzureOpenAIResponsesClient,
) -> tuple[str, Content]:
"""Create a vector store with sample documents for testing."""
file = await client.client.files.create(
file=("todays_weather.txt", b"The weather today is sunny with a high of 75F."),
purpose="assistants",
)
vector_store = await client.client.vector_stores.create(
name="knowledge_base",
expires_after={"anchor": "last_active_at", "days": 1},
)
result = await client.client.vector_stores.files.create_and_poll(vector_store_id=vector_store.id, file_id=file.id)
if result.last_error is not None:
raise Exception(f"Vector store file processing failed with status: {result.last_error.message}")
return file.id, Content.from_hosted_vector_store(vector_store_id=vector_store.id)
async def delete_vector_store(client: AzureOpenAIResponsesClient, file_id: str, vector_store_id: str) -> None:
"""Delete the vector store after tests."""
await client.client.vector_stores.delete(vector_store_id=vector_store_id)
await client.client.files.delete(file_id=file_id)
def test_init(azure_openai_unit_test_env: dict[str, str]) -> None:
# Test successful initialization
azure_responses_client = AzureOpenAIResponsesClient(credential=AzureCliCredential())
assert azure_responses_client.model_id == azure_openai_unit_test_env["AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME"]
assert isinstance(azure_responses_client, SupportsChatGetResponse)
def test_init_validation_fail() -> None:
# Test successful initialization
with pytest.raises(ValueError):
AzureOpenAIResponsesClient(api_key="34523", deployment_name={"test": "dict"}) # type: ignore
def test_init_model_id_constructor(azure_openai_unit_test_env: dict[str, str]) -> None:
# Test successful initialization
model_id = "test_model_id"
azure_responses_client = AzureOpenAIResponsesClient(deployment_name=model_id)
assert azure_responses_client.model_id == model_id
assert isinstance(azure_responses_client, SupportsChatGetResponse)
def test_init_model_id_kwarg(azure_openai_unit_test_env: dict[str, str]) -> None:
"""Test that model_id kwarg correctly sets the deployment name (issue #4299)."""
azure_responses_client = AzureOpenAIResponsesClient(model_id="gpt-4o")
assert azure_responses_client.model_id == "gpt-4o"
assert isinstance(azure_responses_client, SupportsChatGetResponse)
def test_init_model_id_kwarg_does_not_override_deployment_name(
azure_openai_unit_test_env: dict[str, str],
) -> None:
"""Test that deployment_name takes precedence over model_id kwarg (issue #4299)."""
azure_responses_client = AzureOpenAIResponsesClient(deployment_name="my-deployment", model_id="gpt-4o")
assert azure_responses_client.model_id == "my-deployment"
assert isinstance(azure_responses_client, SupportsChatGetResponse)
def test_init_model_id_kwarg_none(azure_openai_unit_test_env: dict[str, str]) -> None:
"""Test that model_id=None does not override the env-var deployment name."""
azure_responses_client = AzureOpenAIResponsesClient(model_id=None)
assert azure_responses_client.model_id == azure_openai_unit_test_env["AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME"]
def test_init_with_default_header(azure_openai_unit_test_env: dict[str, str]) -> None:
default_headers = {"X-Unit-Test": "test-guid"}
# Test successful initialization
azure_responses_client = AzureOpenAIResponsesClient(
default_headers=default_headers,
)
assert azure_responses_client.model_id == azure_openai_unit_test_env["AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME"]
assert isinstance(azure_responses_client, SupportsChatGetResponse)
# Assert that the default header we added is present in the client's default headers
for key, value in default_headers.items():
assert key in azure_responses_client.client.default_headers
assert azure_responses_client.client.default_headers[key] == value
@pytest.mark.parametrize("exclude_list", [["AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME"]], indirect=True)
def test_init_with_empty_model_id(azure_openai_unit_test_env: dict[str, str]) -> None:
with pytest.raises(ValueError):
AzureOpenAIResponsesClient()
def test_init_with_project_client(azure_openai_unit_test_env: dict[str, str]) -> None:
"""Test initialization with an existing AIProjectClient."""
from unittest.mock import patch
from openai import AsyncOpenAI
# Create a mock AIProjectClient that returns a mock AsyncOpenAI client
mock_openai_client = MagicMock(spec=AsyncOpenAI)
mock_openai_client.default_headers = {}
mock_project_client = MagicMock()
mock_project_client.get_openai_client.return_value = mock_openai_client
with patch(
"agent_framework.azure._responses_client.AzureOpenAIResponsesClient._create_client_from_project",
return_value=mock_openai_client,
):
azure_responses_client = AzureOpenAIResponsesClient(
project_client=mock_project_client,
deployment_name="gpt-4o",
)
assert azure_responses_client.model_id == "gpt-4o"
assert azure_responses_client.client is mock_openai_client
assert isinstance(azure_responses_client, SupportsChatGetResponse)
def test_init_with_project_endpoint(azure_openai_unit_test_env: dict[str, str]) -> None:
"""Test initialization with a project endpoint and credential."""
from unittest.mock import patch
from openai import AsyncOpenAI
mock_openai_client = MagicMock(spec=AsyncOpenAI)
mock_openai_client.default_headers = {}
with patch(
"agent_framework.azure._responses_client.AzureOpenAIResponsesClient._create_client_from_project",
return_value=mock_openai_client,
):
azure_responses_client = AzureOpenAIResponsesClient(
project_endpoint="https://test-project.services.ai.azure.com",
deployment_name="gpt-4o",
credential=AzureCliCredential(),
)
assert azure_responses_client.model_id == "gpt-4o"
assert azure_responses_client.client is mock_openai_client
assert isinstance(azure_responses_client, SupportsChatGetResponse)
def test_create_client_from_project_with_project_client() -> None:
"""Test _create_client_from_project with an existing project client."""
from openai import AsyncOpenAI
mock_openai_client = MagicMock(spec=AsyncOpenAI)
mock_project_client = MagicMock()
mock_project_client.get_openai_client.return_value = mock_openai_client
result = AzureOpenAIResponsesClient._create_client_from_project(
project_client=mock_project_client,
project_endpoint=None,
credential=None,
)
assert result is mock_openai_client
mock_project_client.get_openai_client.assert_called_once()
def test_create_client_from_project_with_endpoint() -> None:
"""Test _create_client_from_project with a project endpoint."""
from unittest.mock import patch
from openai import AsyncOpenAI
mock_openai_client = MagicMock(spec=AsyncOpenAI)
mock_credential = MagicMock()
with patch("agent_framework.azure._responses_client.AIProjectClient") as MockAIProjectClient:
mock_instance = MockAIProjectClient.return_value
mock_instance.get_openai_client.return_value = mock_openai_client
result = AzureOpenAIResponsesClient._create_client_from_project(
project_client=None,
project_endpoint="https://test-project.services.ai.azure.com",
credential=mock_credential,
)
assert result is mock_openai_client
MockAIProjectClient.assert_called_once()
mock_instance.get_openai_client.assert_called_once()
def test_create_client_from_project_missing_endpoint() -> None:
"""Test _create_client_from_project raises error when endpoint is missing."""
with pytest.raises(ValueError, match="project endpoint is required"):
AzureOpenAIResponsesClient._create_client_from_project(
project_client=None,
project_endpoint=None,
credential=MagicMock(),
)
def test_create_client_from_project_missing_credential() -> None:
"""Test _create_client_from_project raises error when credential is missing."""
with pytest.raises(ValueError, match="credential is required"):
AzureOpenAIResponsesClient._create_client_from_project(
project_client=None,
project_endpoint="https://test-project.services.ai.azure.com",
credential=None,
)
def test_serialize(azure_openai_unit_test_env: dict[str, str]) -> None:
default_headers = {"X-Unit-Test": "test-guid"}
settings = {
"deployment_name": azure_openai_unit_test_env["AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME"],
"api_key": azure_openai_unit_test_env["AZURE_OPENAI_API_KEY"],
"default_headers": default_headers,
}
azure_responses_client = AzureOpenAIResponsesClient.from_dict(settings)
dumped_settings = azure_responses_client.to_dict()
assert dumped_settings["deployment_name"] == azure_openai_unit_test_env["AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME"]
assert "api_key" not in dumped_settings
# Assert that the default header we added is present in the dumped_settings default headers
for key, value in default_headers.items():
assert key in dumped_settings["default_headers"]
assert dumped_settings["default_headers"][key] == value
# Assert that the 'User-Agent' header is not present in the dumped_settings default headers
assert "User-Agent" not in dumped_settings["default_headers"]
# region Integration Tests
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
@pytest.mark.parametrize(
"option_name,option_value,needs_validation",
[
# Simple ChatOptions - just verify they don't fail
param("temperature", 0.7, False, id="temperature"),
param("top_p", 0.9, False, id="top_p"),
param("max_tokens", 500, False, id="max_tokens"),
param("seed", 123, False, id="seed"),
param("user", "test-user-id", False, id="user"),
param("metadata", {"test_key": "test_value"}, False, id="metadata"),
param("frequency_penalty", 0.5, False, id="frequency_penalty"),
param("presence_penalty", 0.3, False, id="presence_penalty"),
param("stop", ["END"], False, id="stop"),
param("allow_multiple_tool_calls", True, False, id="allow_multiple_tool_calls"),
param("tool_choice", "none", True, id="tool_choice_none"),
# OpenAIResponsesOptions - just verify they don't fail
param("safety_identifier", "user-hash-abc123", False, id="safety_identifier"),
param("truncation", "auto", False, id="truncation"),
param("top_logprobs", 5, False, id="top_logprobs"),
param("prompt_cache_key", "test-cache-key", False, id="prompt_cache_key"),
param("max_tool_calls", 3, False, id="max_tool_calls"),
# Complex options requiring output validation
param("tools", [get_weather], True, id="tools_function"),
param("tool_choice", "auto", True, id="tool_choice_auto"),
param(
"tool_choice",
{"mode": "required", "required_function_name": "get_weather"},
True,
id="tool_choice_required",
),
param("response_format", OutputStruct, True, id="response_format_pydantic"),
param(
"response_format",
{
"type": "json_schema",
"json_schema": {
"name": "WeatherDigest",
"strict": True,
"schema": {
"title": "WeatherDigest",
"type": "object",
"properties": {
"location": {"type": "string"},
"conditions": {"type": "string"},
"temperature_c": {"type": "number"},
"advisory": {"type": "string"},
},
"required": [
"location",
"conditions",
"temperature_c",
"advisory",
],
"additionalProperties": False,
},
},
},
True,
id="response_format_runtime_json_schema",
),
],
)
async def test_integration_options(
option_name: str,
option_value: Any,
needs_validation: bool,
) -> None:
"""Parametrized test covering all ChatOptions and OpenAIResponsesOptions.
Tests both streaming and non-streaming modes for each option to ensure
they don't cause failures. Options marked with needs_validation also
check that the feature actually works correctly.
"""
client = AzureOpenAIResponsesClient(credential=AzureCliCredential())
# Need at least 2 iterations for tool_choice tests: one to get function call, one to get final response
client.function_invocation_configuration["max_iterations"] = 2
for streaming in [False, True]:
# Prepare test message
if option_name == "tools" or option_name == "tool_choice":
# Use weather-related prompt for tool tests
messages = [Message(role="user", text="What is the weather in Seattle?")]
elif option_name == "response_format":
# Use prompt that works well with structured output
messages = [
Message(role="user", text="The weather in Seattle is sunny"),
Message(role="user", text="What is the weather in Seattle?"),
]
else:
# Generic prompt for simple options
messages = [Message(role="user", text="Say 'Hello World' briefly.")]
# Build options dict
options: dict[str, Any] = {option_name: option_value}
# Add tools if testing tool_choice to avoid errors
if option_name == "tool_choice":
options["tools"] = [get_weather]
if streaming:
# Test streaming mode
response_stream = client.get_response(
messages=messages,
stream=True,
options=options,
)
response = await response_stream.get_final_response()
else:
# Test non-streaming mode
response = await client.get_response(
messages=messages,
options=options,
)
assert response is not None
assert isinstance(response, ChatResponse)
assert response.text is not None, f"No text in response for option '{option_name}'"
assert len(response.text) > 0, f"Empty response for option '{option_name}'"
# Validate based on option type
if needs_validation:
if option_name == "tools" or option_name == "tool_choice":
# Should have called the weather function
text = response.text.lower()
assert "sunny" in text or "seattle" in text, f"Tool not invoked for {option_name}"
elif option_name == "response_format":
if option_value == OutputStruct:
# Should have structured output
assert response.value is not None, "No structured output"
assert isinstance(response.value, OutputStruct)
assert "seattle" in response.value.location.lower()
else:
# Runtime JSON schema
assert response.value is None, "No structured output, can't parse any json."
response_value = json.loads(response.text)
assert isinstance(response_value, dict)
assert "location" in response_value
assert "seattle" in response_value["location"].lower()
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_integration_web_search() -> None:
client = AzureOpenAIResponsesClient(credential=AzureCliCredential())
for streaming in [False, True]:
content = {
"messages": [
Message(
role="user",
text="Who are the main characters of Kpop Demon Hunters? Do a web search to find the answer.",
)
],
"options": {
"tool_choice": "auto",
"tools": [AzureOpenAIResponsesClient.get_web_search_tool()],
},
"stream": streaming,
}
if streaming:
response = await client.get_response(**content).get_final_response()
else:
response = await client.get_response(**content)
assert response is not None
assert isinstance(response, ChatResponse)
assert "Rumi" in response.text
assert "Mira" in response.text
assert "Zoey" in response.text
# Test that the client will use the web search tool with location
content = {
"messages": [
Message(
role="user",
text="What is the current weather? Do not ask for my current location.",
)
],
"options": {
"tool_choice": "auto",
"tools": [
AzureOpenAIResponsesClient.get_web_search_tool(user_location={"country": "US", "city": "Seattle"})
],
},
"stream": streaming,
}
if streaming:
response = await client.get_response(**content).get_final_response()
else:
response = await client.get_response(**content)
assert response.text is not None
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_integration_client_file_search() -> None:
"""Test Azure responses client with file search tool."""
azure_responses_client = AzureOpenAIResponsesClient(credential=AzureCliCredential())
file_id, vector_store = await create_vector_store(azure_responses_client)
try:
# Test that the client will use the file search tool
response = await azure_responses_client.get_response(
messages=[
Message(
role="user",
text="What is the weather today? Do a file search to find the answer.",
)
],
options={
"tools": [
AzureOpenAIResponsesClient.get_file_search_tool(vector_store_ids=[vector_store.vector_store_id])
],
"tool_choice": "auto",
},
)
assert "sunny" in response.text.lower()
assert "75" in response.text
finally:
await delete_vector_store(azure_responses_client, file_id, vector_store.vector_store_id)
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_integration_client_file_search_streaming() -> None:
"""Test Azure responses client with file search tool and streaming."""
azure_responses_client = AzureOpenAIResponsesClient(credential=AzureCliCredential())
file_id, vector_store = await create_vector_store(azure_responses_client)
# Test that the client will use the file search tool
try:
response_stream = azure_responses_client.get_response(
messages=[
Message(
role="user",
text="What is the weather today? Do a file search to find the answer.",
)
],
stream=True,
options={
"tools": [
AzureOpenAIResponsesClient.get_file_search_tool(vector_store_ids=[vector_store.vector_store_id])
],
"tool_choice": "auto",
},
)
full_response = await response_stream.get_final_response()
assert "sunny" in full_response.text.lower()
assert "75" in full_response.text
finally:
await delete_vector_store(azure_responses_client, file_id, vector_store.vector_store_id)
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_integration_client_agent_hosted_mcp_tool() -> None:
"""Integration test for MCP tool with Azure Response Agent using Microsoft Learn MCP."""
client = AzureOpenAIResponsesClient(credential=AzureCliCredential())
response = await client.get_response(
messages=[Message(role="user", text="How to create an Azure storage account using az cli?")],
options={
# this needs to be high enough to handle the full MCP tool response.
"max_tokens": 5000,
"tools": AzureOpenAIResponsesClient.get_mcp_tool(
name="Microsoft Learn MCP",
url="https://learn.microsoft.com/api/mcp",
),
},
)
assert isinstance(response, ChatResponse)
# MCP server may return empty response intermittently - skip test rather than fail
if not response.text:
pytest.skip("MCP server returned empty response - service-side issue")
# Should contain Azure-related content since it's asking about Azure CLI
assert any(term in response.text.lower() for term in ["azure", "storage", "account", "cli"])
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_integration_client_agent_hosted_code_interpreter_tool():
"""Test Azure Responses Client agent with code interpreter tool."""
client = AzureOpenAIResponsesClient(credential=AzureCliCredential())
response = await client.get_response(
messages=[
Message(
role="user",
text="Calculate the sum of numbers from 1 to 10 using Python code.",
)
],
options={
"tools": [AzureOpenAIResponsesClient.get_code_interpreter_tool()],
},
)
# Should contain calculation result (sum of 1-10 = 55) or code execution content
contains_relevant_content = any(
term in response.text.lower() for term in ["55", "sum", "code", "python", "calculate", "10"]
)
assert contains_relevant_content or len(response.text.strip()) > 10
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_integration_client_agent_existing_session():
"""Test Azure Responses Client agent with existing session to continue conversations across agent instances."""
# First conversation - capture the session
preserved_session = None
async with Agent(
client=AzureOpenAIResponsesClient(credential=AzureCliCredential()),
instructions="You are a helpful assistant with good memory.",
) as first_agent:
# Start a conversation and capture the session
session = first_agent.create_session()
first_response = await first_agent.run("My hobby is photography. Remember this.", session=session, store=True)
assert isinstance(first_response, AgentResponse)
assert first_response.text is not None
# Preserve the session for reuse
preserved_session = session
# Second conversation - reuse the session in a new agent instance
if preserved_session:
async with Agent(
client=AzureOpenAIResponsesClient(credential=AzureCliCredential()),
instructions="You are a helpful assistant with good memory.",
) as second_agent:
# Reuse the preserved session
second_response = await second_agent.run("What is my hobby?", session=preserved_session)
assert isinstance(second_response, AgentResponse)
assert second_response.text is not None
assert "photography" in second_response.text.lower()
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_azure_openai_responses_client_tool_rich_content_image() -> None:
"""Test that Azure OpenAI Responses client can handle tool results containing images."""
image_path = Path(__file__).parent.parent / "assets" / "sample_image.jpg"
image_bytes = image_path.read_bytes()
@tool(approval_mode="never_require")
def get_test_image() -> Content:
"""Return a test image for analysis."""
return Content.from_data(data=image_bytes, media_type="image/jpeg")
client = AzureOpenAIResponsesClient(credential=AzureCliCredential())
client.function_invocation_configuration["max_iterations"] = 2
for streaming in [False, True]:
messages = [
Message(
role="user",
text="Call the get_test_image tool and describe what you see.",
)
]
options: dict[str, Any] = {"tools": [get_test_image], "tool_choice": "auto"}
if streaming:
response = await client.get_response(messages=messages, stream=True, options=options).get_final_response()
else:
response = await client.get_response(messages=messages, options=options)
assert response is not None
assert isinstance(response, ChatResponse)
assert response.text is not None
assert len(response.text) > 0
# sample_image.jpg contains a photo of a house; the model should mention it.
assert "house" in response.text.lower(), f"Model did not describe the house image. Response: {response.text}"
# region Integration with Foundry V2
skip_if_azure_ai_integration_tests_disabled = pytest.mark.skipif(
os.getenv("AZURE_AI_PROJECT_ENDPOINT", "") in ("", "https://test-project.cognitiveservices.azure.com/")
or os.getenv("AZURE_AI_MODEL_DEPLOYMENT_NAME", "") == "",
reason="No real AZURE_AI_PROJECT_ENDPOINT or AZURE_AI_MODEL_DEPLOYMENT_NAME provided; skipping integration tests.",
)
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_ai_integration_tests_disabled
async def test_integration_function_call_roundtrip_preserves_fidelity():
"""Test that function calls roundtrip correctly with full fidelity preserved.
This verifies the changes where:
1. raw_representation is preserved when parsing function calls
2. fc_id and status are included in additional_properties
3. When re-sending messages, the full object fidelity is preserved
"""
call_count = 0
@tool(name="get_weather", approval_mode="never_require")
async def get_weather_tool(location: str) -> str:
"""Get weather for a location."""
nonlocal call_count
call_count += 1
return f"Weather in {location} is sunny, 72F"
client = AzureOpenAIResponsesClient(
project_endpoint=os.environ["AZURE_AI_PROJECT_ENDPOINT"],
deployment_name=os.environ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
credential=AzureCliCredential(),
)
async with Agent(
client=client,
name="WeatherAgent",
instructions="You help check weather. Use get_weather when asked about weather.",
tools=[get_weather_tool],
default_options={"store": False}, # Store messages locally to test fidelity across messages
) as agent:
session = agent.create_session()
# First request - should invoke the tool
response1 = await agent.run("What is the weather in Seattle?", session=session)
assert response1 is not None
assert response1.text is not None
assert call_count >= 1
# Verify the response contains expected content
response_text = response1.text.lower()
assert "seattle" in response_text or "sunny" in response_text or "72" in response_text
# Second request - should work correctly with the preserved conversation
response2 = await agent.run("And how about in Portland?", session=session)
assert response2 is not None
assert response2.text is not None
assert call_count >= 2
# endregion
python/packages/core/tests/azure/test_entra_id_authentication.py
-61
View File
@@ -1,61 +0,0 @@
# Copyright (c) Microsoft. All rights reserved.
from unittest.mock import MagicMock, patch
import pytest
from azure.core.credentials import TokenCredential
from azure.core.credentials_async import AsyncTokenCredential
from agent_framework.azure._entra_id_authentication import (
resolve_credential_to_token_provider,
)
from agent_framework.exceptions import ChatClientInvalidAuthException
TOKEN_ENDPOINT = "https://cognitiveservices.azure.com/.default"
def test_resolve_sync_credential_returns_provider() -> None:
"""Test that a sync TokenCredential is resolved via azure.identity.get_bearer_token_provider."""
mock_credential = MagicMock(spec=TokenCredential)
mock_provider = MagicMock(return_value="token-string")
with patch("azure.identity.get_bearer_token_provider", return_value=mock_provider) as mock_gbtp:
result = resolve_credential_to_token_provider(mock_credential, TOKEN_ENDPOINT)
mock_gbtp.assert_called_once_with(mock_credential, TOKEN_ENDPOINT)
assert result is mock_provider
def test_resolve_async_credential_returns_provider() -> None:
"""Test that an AsyncTokenCredential is resolved via azure.identity.aio.get_bearer_token_provider."""
mock_credential = MagicMock(spec=AsyncTokenCredential)
mock_provider = MagicMock(return_value="token-string")
with patch("azure.identity.aio.get_bearer_token_provider", return_value=mock_provider) as mock_gbtp:
result = resolve_credential_to_token_provider(mock_credential, TOKEN_ENDPOINT)
mock_gbtp.assert_called_once_with(mock_credential, TOKEN_ENDPOINT)
assert result is mock_provider
def test_resolve_callable_provider_passthrough() -> None:
"""Test that a callable token provider is returned as-is, without needing token_endpoint."""
my_provider = lambda: "my-token" # noqa: E731
# Works with token_endpoint
assert resolve_credential_to_token_provider(my_provider, TOKEN_ENDPOINT) is my_provider
# Also works without token_endpoint
assert resolve_credential_to_token_provider(my_provider, None) is my_provider
assert resolve_credential_to_token_provider(my_provider, "") is my_provider
def test_resolve_missing_endpoint_raises() -> None:
"""Test that missing token endpoint raises ChatClientInvalidAuthException."""
mock_credential = MagicMock(spec=TokenCredential)
with pytest.raises(ChatClientInvalidAuthException, match="A token endpoint must be provided"):
resolve_credential_to_token_provider(mock_credential, "")
with pytest.raises(ChatClientInvalidAuthException, match="A token endpoint must be provided"):
resolve_credential_to_token_provider(mock_credential, None) # type: ignore[arg-type]
python/packages/core/tests/core/test_agents.py
+3 -3
View File
@@ -1950,13 +1950,13 @@ async def test_shared_local_storage_cross_provider_responses_history_does_not_le
from openai.types.chat.chat_completion_message import ChatCompletionMessage
from agent_framework._sessions import InMemoryHistoryProvider
from agent_framework.openai import OpenAIChatClient, OpenAIResponsesClient
from agent_framework.openai import OpenAIChatClient, OpenAIChatCompletionClient
@tool(approval_mode="never_require")
def search_hotels(city: str) -> str:
return f"Found 3 hotels in {city}"
responses_client = OpenAIResponsesClient(model_id="test-model", api_key="test-key")
responses_client = OpenAIChatClient(model="test-model", api_key="test-key")
responses_agent = Agent(
client=responses_client,
tools=[search_hotels],
@@ -2024,7 +2024,7 @@ async def test_shared_local_storage_cross_provider_responses_history_does_not_le
responses_replay_call = next(item for item in responses_replay_input if item.get("type") == "function_call")
assert responses_replay_call["id"] == "fc_provider123"
chat_client = OpenAIChatClient(model_id="test-model", api_key="test-key")
chat_client = OpenAIChatCompletionClient(model="test-model", api_key="test-key")
chat_agent = Agent(client=chat_client)
chat_response = ChatCompletion(
python/packages/core/tests/core/test_clients.py
+14 -14
View File
@@ -66,10 +66,10 @@ def test_base_client_as_agent_uses_explicit_additional_properties(chat_client_ba
assert agent.additional_properties == {"team": "core"}
def test_openai_chat_client_get_response_docstring_surfaces_layered_runtime_docs() -> None:
from agent_framework.openai import OpenAIChatClient
def test_openai_chat_completion_client_get_response_docstring_surfaces_layered_runtime_docs() -> None:
from agent_framework.openai import OpenAIChatCompletionClient
docstring = inspect.getdoc(OpenAIChatClient.get_response)
docstring = inspect.getdoc(OpenAIChatCompletionClient.get_response)
assert docstring is not None
assert "Get a response from a chat client." in docstring
@@ -78,12 +78,12 @@ def test_openai_chat_client_get_response_docstring_surfaces_layered_runtime_docs
assert "function_middleware: Optional per-call function middleware." not in docstring
def test_openai_chat_client_get_response_is_defined_on_openai_class() -> None:
from agent_framework.openai import OpenAIChatClient
def test_openai_chat_completion_client_get_response_is_defined_on_openai_class() -> None:
from agent_framework.openai import OpenAIChatCompletionClient
signature = inspect.signature(OpenAIChatClient.get_response)
signature = inspect.signature(OpenAIChatCompletionClient.get_response)
assert OpenAIChatClient.get_response.__qualname__ == "OpenAIChatClient.get_response"
assert OpenAIChatCompletionClient.get_response.__qualname__ == "OpenAIChatCompletionClient.get_response"
assert "middleware" in signature.parameters
@@ -349,15 +349,15 @@ def test_openai_responses_client_supports_all_tool_protocols():
assert isinstance(OpenAIResponsesClient, SupportsFileSearchTool)
def test_openai_chat_client_supports_web_search_only():
def test_openai_chat_completion_client_supports_web_search_only():
"""Test that OpenAIChatClient only supports web search tool."""
from agent_framework.openai import OpenAIChatClient
from agent_framework.openai import OpenAIChatCompletionClient
assert not isinstance(OpenAIChatClient, SupportsCodeInterpreterTool)
assert isinstance(OpenAIChatClient, SupportsWebSearchTool)
assert not isinstance(OpenAIChatClient, SupportsImageGenerationTool)
assert not isinstance(OpenAIChatClient, SupportsMCPTool)
assert not isinstance(OpenAIChatClient, SupportsFileSearchTool)
assert not isinstance(OpenAIChatCompletionClient, SupportsCodeInterpreterTool)
assert isinstance(OpenAIChatCompletionClient, SupportsWebSearchTool)
assert not isinstance(OpenAIChatCompletionClient, SupportsImageGenerationTool)
assert not isinstance(OpenAIChatCompletionClient, SupportsMCPTool)
assert not isinstance(OpenAIChatCompletionClient, SupportsFileSearchTool)
def test_openai_assistants_client_supports_code_interpreter_and_file_search():
python/packages/core/tests/core/test_foundry_namespace.py
+24
View File
@@ -0,0 +1,24 @@
# Copyright (c) Microsoft. All rights reserved.
import pytest
from agent_framework_foundry import FoundryChatClient, FoundryMemoryProvider
from agent_framework_foundry_local import FoundryLocalClient
import agent_framework.azure as azure
import agent_framework.foundry as foundry
def test_foundry_namespace_exposes_cloud_and_local_symbols() -> None:
assert foundry.FoundryChatClient is FoundryChatClient
assert foundry.FoundryMemoryProvider is FoundryMemoryProvider
assert foundry.FoundryLocalClient is FoundryLocalClient
assert "FoundryChatClient" in dir(foundry)
assert "FoundryLocalClient" in dir(foundry)
def test_azure_namespace_no_longer_exposes_foundry_symbols() -> None:
assert "FoundryChatClient" not in dir(azure)
assert "FoundryLocalClient" not in dir(azure)
with pytest.raises(AttributeError, match="Module `azure` has no attribute FoundryChatClient\\."):
_ = azure.FoundryChatClient
python/packages/core/tests/core/test_mcp.py
+12 -7
View File
@@ -42,6 +42,14 @@ skip_if_mcp_integration_tests_disabled = pytest.mark.skipif(
)
def _mcp_result_to_text(result: str | list[Content]) -> str:
"""Normalize an MCP tool result to text for assertions."""
if isinstance(result, str):
return result
text = "\n".join(content.text for content in result if content.type == "text" and content.text)
return text or str(result)
# Helper function tests
def test_normalize_mcp_name():
"""Test MCP name normalization."""
@@ -1401,8 +1409,7 @@ async def test_streamable_http_integration():
assert hasattr(func, "name")
assert hasattr(func, "description")
result = await func.invoke(query="What is Agent Framework?")
assert isinstance(result, str)
result = _mcp_result_to_text(await func.invoke(query="What is Agent Framework?"))
assert len(result) > 0
@@ -1430,7 +1437,7 @@ async def test_mcp_connection_reset_integration():
# Get the first function and invoke it
func = tool.functions[0]
first_result = await func.invoke(query="What is Agent Framework?")
first_result = _mcp_result_to_text(await func.invoke(query="What is Agent Framework?"))
assert first_result is not None
assert len(first_result) > 0
@@ -1456,7 +1463,7 @@ async def test_mcp_connection_reset_integration():
tool.session.call_tool = call_tool_with_error
# Invoke the function again - this should trigger automatic reconnection on ClosedResourceError
second_result = await func.invoke(query="What is Agent Framework?")
second_result = _mcp_result_to_text(await func.invoke(query="What is Agent Framework?"))
assert second_result is not None
assert len(second_result) > 0
@@ -1469,10 +1476,8 @@ async def test_mcp_connection_reset_integration():
# Verify tools are still available after reconnection
assert len(tool.functions) > 0
# Both results should be valid strings (we don't compare content as it may vary)
assert isinstance(first_result, str)
# Both results should include text (we don't compare content as it may vary)
assert len(first_result) > 0
assert isinstance(second_result, str)
assert len(second_result) > 0
python/packages/core/tests/core/test_types.py
+13 -13
View File
@@ -1031,11 +1031,11 @@ def test_chat_tool_mode_from_dict():
def test_chat_options_init() -> None:
"""Test that ChatOptions can be created as a TypedDict."""
options: ChatOptions = {}
assert options.get("model_id") is None
assert options.get("model") is None
# With values
options_with_model: ChatOptions = {"model_id": "gpt-4o", "temperature": 0.7}
assert options_with_model.get("model_id") == "gpt-4o"
options_with_model: ChatOptions = {"model": "gpt-4o", "temperature": 0.7}
assert options_with_model.get("model") == "gpt-4o"
assert options_with_model.get("temperature") == 0.7
@@ -1069,18 +1069,18 @@ def test_chat_options_tool_choice_validation():
def test_chat_options_merge(tool_tool, ai_tool) -> None:
"""Test merge_chat_options utility function."""
options1: ChatOptions = {
"model_id": "gpt-4o",
"model": "gpt-4o",
"tools": [tool_tool],
"logit_bias": {"x": 1},
"metadata": {"a": "b"},
}
options2: ChatOptions = {"model_id": "gpt-4.1", "tools": [ai_tool]}
options2: ChatOptions = {"model": "gpt-4.1", "tools": [ai_tool]}
assert options1 != options2
# Merge options - override takes precedence for non-collection fields
options3 = merge_chat_options(options1, options2)
assert options3.get("model_id") == "gpt-4.1"
assert options3.get("model") == "gpt-4.1"
assert options3.get("tools") == [tool_tool, ai_tool] # tools are combined
assert options3.get("logit_bias") == {"x": 1} # base value preserved
assert options3.get("metadata") == {"a": "b"} # base value preserved
@@ -1089,7 +1089,7 @@ def test_chat_options_merge(tool_tool, ai_tool) -> None:
def test_chat_options_and_tool_choice_override() -> None:
"""Test that tool_choice from other takes precedence in ChatOptions merge."""
# Agent-level defaults to "auto"
agent_options: ChatOptions = {"model_id": "gpt-4o", "tool_choice": "auto"}
agent_options: ChatOptions = {"model": "gpt-4o", "tool_choice": "auto"}
# Run-level specifies "required"
run_options: ChatOptions = {"tool_choice": "required"}
@@ -1097,19 +1097,19 @@ def test_chat_options_and_tool_choice_override() -> None:
# Run-level should override agent-level
assert merged.get("tool_choice") == "required"
assert merged.get("model_id") == "gpt-4o" # Other fields preserved
assert merged.get("model") == "gpt-4o" # Other fields preserved
def test_chat_options_and_tool_choice_none_in_other_uses_self() -> None:
"""Test that when other.tool_choice is None, self.tool_choice is used."""
agent_options: ChatOptions = {"tool_choice": "auto"}
run_options: ChatOptions = {"model_id": "gpt-4.1"} # tool_choice is None
run_options: ChatOptions = {"model": "gpt-4.1"} # tool_choice is None
merged = merge_chat_options(agent_options, run_options)
# Should keep agent-level tool_choice since run-level is None
assert merged.get("tool_choice") == "auto"
assert merged.get("model_id") == "gpt-4.1"
assert merged.get("model") == "gpt-4.1"
def test_chat_options_and_tool_choice_with_tool_mode() -> None:
@@ -1845,7 +1845,7 @@ def test_chat_response_complex_serialization():
"output_token_count": 8,
"total_token_count": 13,
},
"model_id": "gpt-4", # Test alias handling
"model": "gpt-4", # Test alias handling
}
response = ChatResponse.from_dict(response_data)
@@ -1861,7 +1861,7 @@ def test_chat_response_complex_serialization():
assert isinstance(response_dict["messages"][0], dict)
assert isinstance(response_dict["finish_reason"], str) # FinishReason serializes to string
assert isinstance(response_dict["usage_details"], dict)
assert response_dict["model_id"] == "gpt-4" # Should serialize as model_id
assert response_dict["model"] == "gpt-4" # Should serialize as model_id
def test_chat_response_update_all_content_types():
@@ -2309,7 +2309,7 @@ def test_chat_response_deepcopy_deep_copies_additional_properties():
"total_token_count": 30,
},
"response_id": "resp-123",
"model_id": "gpt-4",
"model": "gpt-4",
},
id="chat_response",
),
python/packages/core/tests/openai/conftest.py
-51
View File
@@ -1,51 +0,0 @@
# Copyright (c) Microsoft. All rights reserved.
from typing import Any
from pytest import fixture
# region Connector Settings fixtures
@fixture
def exclude_list(request: Any) -> list[str]:
"""Fixture that returns a list of environment variables to exclude."""
return request.param if hasattr(request, "param") else []
@fixture
def override_env_param_dict(request: Any) -> dict[str, str]:
"""Fixture that returns a dict of environment variables to override."""
return request.param if hasattr(request, "param") else {}
@fixture()
def openai_unit_test_env(monkeypatch, exclude_list, override_env_param_dict): # type: ignore
"""Fixture to set environment variables for OpenAISettings."""
if exclude_list is None:
exclude_list = []
if override_env_param_dict is None:
override_env_param_dict = {}
env_vars = {
"OPENAI_API_KEY": "test-dummy-key",
"OPENAI_ORG_ID": "test_org_id",
"OPENAI_RESPONSES_MODEL_ID": "test_responses_model_id",
"OPENAI_CHAT_MODEL_ID": "test_chat_model_id",
"OPENAI_TEXT_MODEL_ID": "test_text_model_id",
"OPENAI_EMBEDDING_MODEL_ID": "test_embedding_model_id",
"OPENAI_TEXT_TO_IMAGE_MODEL_ID": "test_text_to_image_model_id",
"OPENAI_AUDIO_TO_TEXT_MODEL_ID": "test_audio_to_text_model_id",
"OPENAI_TEXT_TO_AUDIO_MODEL_ID": "test_text_to_audio_model_id",
"OPENAI_REALTIME_MODEL_ID": "test_realtime_model_id",
}
env_vars.update(override_env_param_dict) # type: ignore
for key, value in env_vars.items():
if key in exclude_list:
monkeypatch.delenv(key, raising=False) # type: ignore
continue
monkeypatch.setenv(key, value) # type: ignore
return env_vars
python/packages/core/tests/openai/test_assistant_provider.py
-814
View File
@@ -1,814 +0,0 @@
# Copyright (c) Microsoft. All rights reserved.
import os
from typing import Annotated, Any
from unittest.mock import AsyncMock, MagicMock
import pytest
from openai.types.beta.assistant import Assistant
from pydantic import BaseModel, Field
from agent_framework import Agent, normalize_tools, tool
from agent_framework.openai import OpenAIAssistantProvider, OpenAIAssistantsClient
from agent_framework.openai._shared import from_assistant_tools, to_assistant_tools
# region Test Helpers
def create_mock_assistant(
assistant_id: str = "asst_test123",
name: str = "TestAssistant",
model: str = "gpt-4",
instructions: str | None = "You are a helpful assistant.",
description: str | None = None,
tools: list[Any] | None = None,
) -> Assistant:
"""Create a mock Assistant object."""
mock = MagicMock(spec=Assistant)
mock.id = assistant_id
mock.name = name
mock.model = model
mock.instructions = instructions
mock.description = description
mock.tools = tools or []
return mock
def create_function_tool(name: str, description: str = "A test function") -> MagicMock:
"""Create a mock FunctionTool."""
mock = MagicMock()
mock.type = "function"
mock.function = MagicMock()
mock.function.name = name
mock.function.description = description
return mock
def create_code_interpreter_tool() -> MagicMock:
"""Create a mock CodeInterpreterTool."""
mock = MagicMock()
mock.type = "code_interpreter"
return mock
def create_file_search_tool() -> MagicMock:
"""Create a mock FileSearchTool."""
mock = MagicMock()
mock.type = "file_search"
return mock
@pytest.fixture
def mock_async_openai() -> MagicMock:
"""Mock AsyncOpenAI client."""
mock_client = MagicMock()
# Mock beta.assistants
mock_client.beta.assistants.create = AsyncMock(
return_value=create_mock_assistant(assistant_id="asst_created123", name="CreatedAssistant")
)
mock_client.beta.assistants.retrieve = AsyncMock(
return_value=create_mock_assistant(assistant_id="asst_retrieved123", name="RetrievedAssistant")
)
mock_client.beta.assistants.delete = AsyncMock()
# Mock close method
mock_client.close = AsyncMock()
return mock_client
# Test function for tool validation
def get_weather(location: Annotated[str, Field(description="The location")]) -> str:
"""Get the weather for a location."""
return f"Weather in {location}: sunny"
def search_database(query: Annotated[str, Field(description="Search query")]) -> str:
"""Search the database."""
return f"Results for: {query}"
# Pydantic model for structured output tests
class WeatherResponse(BaseModel):
location: str
temperature: float
conditions: str
# endregion
# region Initialization Tests
class TestOpenAIAssistantProviderInit:
"""Tests for provider initialization."""
def test_init_with_client(self, mock_async_openai: MagicMock) -> None:
"""Test initialization with existing AsyncOpenAI client."""
provider = OpenAIAssistantProvider(mock_async_openai)
assert provider._client is mock_async_openai # type: ignore[reportPrivateUsage]
assert provider._should_close_client is False # type: ignore[reportPrivateUsage]
def test_init_without_client_creates_one(self, openai_unit_test_env: dict[str, str]) -> None:
"""Test initialization creates client from settings."""
provider = OpenAIAssistantProvider()
assert provider._client is not None # type: ignore[reportPrivateUsage]
assert provider._should_close_client is True # type: ignore[reportPrivateUsage]
def test_init_with_api_key(self) -> None:
"""Test initialization with explicit API key."""
provider = OpenAIAssistantProvider(api_key="sk-test-key")
assert provider._client is not None # type: ignore[reportPrivateUsage]
assert provider._should_close_client is True # type: ignore[reportPrivateUsage]
def test_init_fails_without_api_key(self) -> None:
"""Test initialization fails without API key when settings return None."""
from unittest.mock import patch
# Mock load_settings to return a dict with None for api_key
with patch("agent_framework.openai._assistant_provider.load_settings") as mock_load:
mock_load.return_value = {
"api_key": None,
"org_id": None,
"base_url": None,
"chat_model_id": None,
"responses_model_id": None,
}
with pytest.raises(ValueError) as exc_info:
OpenAIAssistantProvider()
assert "API key is required" in str(exc_info.value)
def test_init_with_org_id_and_base_url(self) -> None:
"""Test initialization with organization ID and base URL."""
provider = OpenAIAssistantProvider(
api_key="sk-test-key",
org_id="org-123",
base_url="https://custom.openai.com",
)
assert provider._client is not None # type: ignore[reportPrivateUsage]
class TestOpenAIAssistantProviderContextManager:
"""Tests for async context manager."""
async def test_context_manager_enter_exit(self, mock_async_openai: MagicMock) -> None:
"""Test async context manager entry and exit."""
provider = OpenAIAssistantProvider(mock_async_openai)
async with provider as p:
assert p is provider
async def test_context_manager_closes_owned_client(self, openai_unit_test_env: dict[str, str]) -> None:
"""Test that owned client is closed on exit."""
provider = OpenAIAssistantProvider()
client = provider._client # type: ignore[reportPrivateUsage]
assert client is not None
client.close = AsyncMock()
async with provider:
pass
client.close.assert_called_once()
async def test_context_manager_does_not_close_external_client(self, mock_async_openai: MagicMock) -> None:
"""Test that external client is not closed on exit."""
provider = OpenAIAssistantProvider(mock_async_openai)
async with provider:
pass
mock_async_openai.close.assert_not_called()
# endregion
# region create_agent Tests
class TestOpenAIAssistantProviderCreateAgent:
"""Tests for create_agent method."""
async def test_create_agent_basic(self, mock_async_openai: MagicMock) -> None:
"""Test basic assistant creation."""
provider = OpenAIAssistantProvider(mock_async_openai)
agent = await provider.create_agent(
name="TestAgent",
model="gpt-4",
instructions="You are helpful.",
)
assert isinstance(agent, Agent)
assert agent.name == "CreatedAssistant"
mock_async_openai.beta.assistants.create.assert_called_once()
# Verify create was called with correct parameters
call_kwargs = mock_async_openai.beta.assistants.create.call_args.kwargs
assert call_kwargs["name"] == "TestAgent"
assert call_kwargs["model"] == "gpt-4"
assert call_kwargs["instructions"] == "You are helpful."
async def test_create_agent_with_description(self, mock_async_openai: MagicMock) -> None:
"""Test assistant creation with description."""
provider = OpenAIAssistantProvider(mock_async_openai)
await provider.create_agent(
name="TestAgent",
model="gpt-4",
description="A test agent description",
)
call_kwargs = mock_async_openai.beta.assistants.create.call_args.kwargs
assert call_kwargs["description"] == "A test agent description"
async def test_create_agent_with_function_tools(self, mock_async_openai: MagicMock) -> None:
"""Test assistant creation with function tools."""
provider = OpenAIAssistantProvider(mock_async_openai)
agent = await provider.create_agent(
name="WeatherAgent",
model="gpt-4",
tools=[get_weather],
)
assert isinstance(agent, Agent)
# Verify tools were passed to create
call_kwargs = mock_async_openai.beta.assistants.create.call_args.kwargs
assert "tools" in call_kwargs
assert len(call_kwargs["tools"]) == 1
assert call_kwargs["tools"][0]["type"] == "function"
assert call_kwargs["tools"][0]["function"]["name"] == "get_weather"
async def test_create_agent_with_tool(self, mock_async_openai: MagicMock) -> None:
"""Test assistant creation with FunctionTool."""
provider = OpenAIAssistantProvider(mock_async_openai)
@tool
def my_function(x: int) -> int:
"""Double a number."""
return x * 2
await provider.create_agent(
name="TestAgent",
model="gpt-4",
tools=[my_function],
)
call_kwargs = mock_async_openai.beta.assistants.create.call_args.kwargs
assert call_kwargs["tools"][0]["function"]["name"] == "my_function"
async def test_create_agent_with_code_interpreter(self, mock_async_openai: MagicMock) -> None:
"""Test assistant creation with code interpreter."""
provider = OpenAIAssistantProvider(mock_async_openai)
await provider.create_agent(
name="CodeAgent",
model="gpt-4",
tools=[OpenAIAssistantsClient.get_code_interpreter_tool()],
)
call_kwargs = mock_async_openai.beta.assistants.create.call_args.kwargs
assert {"type": "code_interpreter"} in call_kwargs["tools"]
async def test_create_agent_with_file_search(self, mock_async_openai: MagicMock) -> None:
"""Test assistant creation with file search."""
provider = OpenAIAssistantProvider(mock_async_openai)
await provider.create_agent(
name="SearchAgent",
model="gpt-4",
tools=[OpenAIAssistantsClient.get_file_search_tool()],
)
call_kwargs = mock_async_openai.beta.assistants.create.call_args.kwargs
assert any(t["type"] == "file_search" for t in call_kwargs["tools"])
async def test_create_agent_with_file_search_max_results(self, mock_async_openai: MagicMock) -> None:
"""Test assistant creation with file search and max_results."""
provider = OpenAIAssistantProvider(mock_async_openai)
await provider.create_agent(
name="SearchAgent",
model="gpt-4",
tools=[OpenAIAssistantsClient.get_file_search_tool(max_num_results=10)],
)
call_kwargs = mock_async_openai.beta.assistants.create.call_args.kwargs
file_search_tool = next(t for t in call_kwargs["tools"] if t["type"] == "file_search")
assert file_search_tool.get("file_search", {}).get("max_num_results") == 10
async def test_create_agent_with_mixed_tools(self, mock_async_openai: MagicMock) -> None:
"""Test assistant creation with multiple tool types."""
provider = OpenAIAssistantProvider(mock_async_openai)
await provider.create_agent(
name="MultiToolAgent",
model="gpt-4",
tools=[
get_weather,
OpenAIAssistantsClient.get_code_interpreter_tool(),
OpenAIAssistantsClient.get_file_search_tool(),
],
)
call_kwargs = mock_async_openai.beta.assistants.create.call_args.kwargs
assert len(call_kwargs["tools"]) == 3
async def test_create_agent_with_metadata(self, mock_async_openai: MagicMock) -> None:
"""Test assistant creation with metadata."""
provider = OpenAIAssistantProvider(mock_async_openai)
await provider.create_agent(
name="TestAgent",
model="gpt-4",
metadata={"env": "test", "version": "1.0"},
)
call_kwargs = mock_async_openai.beta.assistants.create.call_args.kwargs
assert call_kwargs["metadata"] == {"env": "test", "version": "1.0"}
async def test_create_agent_with_response_format_pydantic(self, mock_async_openai: MagicMock) -> None:
"""Test assistant creation with Pydantic response format via default_options."""
provider = OpenAIAssistantProvider(mock_async_openai)
await provider.create_agent(
name="StructuredAgent",
model="gpt-4",
default_options={"response_format": WeatherResponse},
)
call_kwargs = mock_async_openai.beta.assistants.create.call_args.kwargs
assert call_kwargs["response_format"]["type"] == "json_schema"
assert call_kwargs["response_format"]["json_schema"]["name"] == "WeatherResponse"
async def test_create_agent_returns_chat_agent(self, mock_async_openai: MagicMock) -> None:
"""Test that create_agent returns a Agent instance."""
provider = OpenAIAssistantProvider(mock_async_openai)
agent = await provider.create_agent(
name="TestAgent",
model="gpt-4",
)
assert isinstance(agent, Agent)
# endregion
# region get_agent Tests
class TestOpenAIAssistantProviderGetAgent:
"""Tests for get_agent method."""
async def test_get_agent_basic(self, mock_async_openai: MagicMock) -> None:
"""Test retrieving an existing assistant."""
provider = OpenAIAssistantProvider(mock_async_openai)
agent = await provider.get_agent(assistant_id="asst_123")
assert isinstance(agent, Agent)
mock_async_openai.beta.assistants.retrieve.assert_called_once_with("asst_123")
async def test_get_agent_with_instructions_override(self, mock_async_openai: MagicMock) -> None:
"""Test retrieving assistant with instruction override."""
provider = OpenAIAssistantProvider(mock_async_openai)
agent = await provider.get_agent(
assistant_id="asst_123",
instructions="Custom instructions",
)
# Agent should be created successfully with the custom instructions
assert isinstance(agent, Agent)
assert agent.id == "asst_retrieved123"
async def test_get_agent_with_function_tools(self, mock_async_openai: MagicMock) -> None:
"""Test retrieving assistant with function tools provided."""
# Setup assistant with function tool
assistant = create_mock_assistant(tools=[create_function_tool("get_weather")])
mock_async_openai.beta.assistants.retrieve = AsyncMock(return_value=assistant)
provider = OpenAIAssistantProvider(mock_async_openai)
agent = await provider.get_agent(
assistant_id="asst_123",
tools=[get_weather],
)
assert isinstance(agent, Agent)
async def test_get_agent_validates_missing_function_tools(self, mock_async_openai: MagicMock) -> None:
"""Test that missing function tools raise ValueError."""
# Setup assistant with function tool
assistant = create_mock_assistant(tools=[create_function_tool("get_weather")])
mock_async_openai.beta.assistants.retrieve = AsyncMock(return_value=assistant)
provider = OpenAIAssistantProvider(mock_async_openai)
with pytest.raises(ValueError) as exc_info:
await provider.get_agent(assistant_id="asst_123")
assert "get_weather" in str(exc_info.value)
assert "no implementation was provided" in str(exc_info.value)
async def test_get_agent_validates_multiple_missing_function_tools(self, mock_async_openai: MagicMock) -> None:
"""Test validation with multiple missing function tools."""
assistant = create_mock_assistant(
tools=[create_function_tool("get_weather"), create_function_tool("search_database")]
)
mock_async_openai.beta.assistants.retrieve = AsyncMock(return_value=assistant)
provider = OpenAIAssistantProvider(mock_async_openai)
with pytest.raises(ValueError) as exc_info:
await provider.get_agent(assistant_id="asst_123")
error_msg = str(exc_info.value)
assert "get_weather" in error_msg or "search_database" in error_msg
async def test_get_agent_merges_hosted_tools(self, mock_async_openai: MagicMock) -> None:
"""Test that hosted tools are automatically included."""
assistant = create_mock_assistant(tools=[create_code_interpreter_tool(), create_file_search_tool()])
mock_async_openai.beta.assistants.retrieve = AsyncMock(return_value=assistant)
provider = OpenAIAssistantProvider(mock_async_openai)
agent = await provider.get_agent(assistant_id="asst_123")
# Hosted tools should be merged automatically
assert isinstance(agent, Agent)
# endregion
# region as_agent Tests
class TestOpenAIAssistantProviderAsAgent:
"""Tests for as_agent method."""
def test_as_agent_no_http_call(self, mock_async_openai: MagicMock) -> None:
"""Test that as_agent doesn't make HTTP calls."""
provider = OpenAIAssistantProvider(mock_async_openai)
assistant = create_mock_assistant()
agent = provider.as_agent(assistant)
assert isinstance(agent, Agent)
# Verify no HTTP calls were made
mock_async_openai.beta.assistants.create.assert_not_called()
mock_async_openai.beta.assistants.retrieve.assert_not_called()
def test_as_agent_wraps_assistant(self, mock_async_openai: MagicMock) -> None:
"""Test wrapping an SDK Assistant object."""
provider = OpenAIAssistantProvider(mock_async_openai)
assistant = create_mock_assistant(
assistant_id="asst_wrap123",
name="WrappedAssistant",
instructions="Original instructions",
)
agent = provider.as_agent(assistant)
assert agent.id == "asst_wrap123"
assert agent.name == "WrappedAssistant"
# Instructions are passed to ChatOptions, not exposed as attribute
assert isinstance(agent, Agent)
def test_as_agent_with_instructions_override(self, mock_async_openai: MagicMock) -> None:
"""Test as_agent with instruction override."""
provider = OpenAIAssistantProvider(mock_async_openai)
assistant = create_mock_assistant(instructions="Original")
agent = provider.as_agent(assistant, instructions="Override")
# Agent should be created successfully with override instructions
assert isinstance(agent, Agent)
def test_as_agent_validates_function_tools(self, mock_async_openai: MagicMock) -> None:
"""Test that missing function tools raise ValueError."""
provider = OpenAIAssistantProvider(mock_async_openai)
assistant = create_mock_assistant(tools=[create_function_tool("get_weather")])
with pytest.raises(ValueError) as exc_info:
provider.as_agent(assistant)
assert "get_weather" in str(exc_info.value)
def test_as_agent_with_function_tools_provided(self, mock_async_openai: MagicMock) -> None:
"""Test as_agent with function tools provided."""
provider = OpenAIAssistantProvider(mock_async_openai)
assistant = create_mock_assistant(tools=[create_function_tool("get_weather")])
agent = provider.as_agent(assistant, tools=[get_weather])
assert isinstance(agent, Agent)
def test_as_agent_merges_hosted_tools(self, mock_async_openai: MagicMock) -> None:
"""Test that hosted tools are merged automatically."""
provider = OpenAIAssistantProvider(mock_async_openai)
assistant = create_mock_assistant(tools=[create_code_interpreter_tool()])
agent = provider.as_agent(assistant)
assert isinstance(agent, Agent)
def test_as_agent_hosted_tools_not_required(self, mock_async_openai: MagicMock) -> None:
"""Test that hosted tools don't require user implementations."""
provider = OpenAIAssistantProvider(mock_async_openai)
assistant = create_mock_assistant(tools=[create_code_interpreter_tool(), create_file_search_tool()])
# Should not raise - hosted tools don't need implementations
agent = provider.as_agent(assistant)
assert isinstance(agent, Agent)
# endregion
# region Tool Conversion Tests
class TestToolConversion:
"""Tests for tool conversion utilities (shared functions)."""
def test_to_assistant_tools_tool(self) -> None:
"""Test FunctionTool conversion to API format."""
@tool
def test_func(x: int) -> int:
"""Test function."""
return x
# Normalize tools first, then convert
normalized = normalize_tools([test_func])
api_tools = to_assistant_tools(normalized)
assert len(api_tools) == 1
assert api_tools[0]["type"] == "function"
assert api_tools[0]["function"]["name"] == "test_func"
def test_to_assistant_tools_callable(self) -> None:
"""Test raw callable conversion via normalize_tools."""
# normalize_tools converts callables to FunctionTool
normalized = normalize_tools([get_weather])
api_tools = to_assistant_tools(normalized)
assert len(api_tools) == 1
assert api_tools[0]["type"] == "function"
assert api_tools[0]["function"]["name"] == "get_weather"
def test_to_assistant_tools_code_interpreter(self) -> None:
"""Test code_interpreter tool dict conversion."""
api_tools = to_assistant_tools([OpenAIAssistantsClient.get_code_interpreter_tool()])
assert len(api_tools) == 1
assert api_tools[0] == {"type": "code_interpreter"}
def test_to_assistant_tools_file_search(self) -> None:
"""Test file_search tool dict conversion."""
api_tools = to_assistant_tools([OpenAIAssistantsClient.get_file_search_tool()])
assert len(api_tools) == 1
assert api_tools[0]["type"] == "file_search"
def test_to_assistant_tools_file_search_with_max_results(self) -> None:
"""Test file_search tool with max_results conversion."""
api_tools = to_assistant_tools([OpenAIAssistantsClient.get_file_search_tool(max_num_results=5)])
assert api_tools[0]["file_search"]["max_num_results"] == 5
def test_to_assistant_tools_dict(self) -> None:
"""Test raw dict tool passthrough."""
raw_tool = {"type": "function", "function": {"name": "custom", "description": "Custom tool"}}
api_tools = to_assistant_tools([raw_tool])
assert len(api_tools) == 1
assert api_tools[0] == raw_tool
def test_to_assistant_tools_empty(self) -> None:
"""Test conversion with no tools."""
api_tools = to_assistant_tools(None)
assert api_tools == []
def test_from_assistant_tools_code_interpreter(self) -> None:
"""Test converting code_interpreter tool from OpenAI format."""
assistant_tools = [create_code_interpreter_tool()]
tools = from_assistant_tools(assistant_tools)
assert len(tools) == 1
assert tools[0] == {"type": "code_interpreter"}
def test_from_assistant_tools_file_search(self) -> None:
"""Test converting file_search tool from OpenAI format."""
assistant_tools = [create_file_search_tool()]
tools = from_assistant_tools(assistant_tools)
assert len(tools) == 1
assert tools[0] == {"type": "file_search"}
def test_from_assistant_tools_function_skipped(self) -> None:
"""Test that function tools are skipped (no implementations)."""
assistant_tools = [create_function_tool("test_func")]
tools = from_assistant_tools(assistant_tools)
assert len(tools) == 0 # Function tools are skipped
def test_from_assistant_tools_empty(self) -> None:
"""Test conversion with no tools."""
tools = from_assistant_tools(None)
assert tools == []
# endregion
# region Tool Validation Tests
class TestToolValidation:
"""Tests for tool validation."""
def test_validate_missing_function_tool_raises(self, mock_async_openai: MagicMock) -> None:
"""Test that missing function tools raise ValueError."""
provider = OpenAIAssistantProvider(mock_async_openai)
assistant_tools = [create_function_tool("my_function")]
with pytest.raises(ValueError) as exc_info:
provider._validate_function_tools(assistant_tools, None) # type: ignore[reportPrivateUsage]
assert "my_function" in str(exc_info.value)
def test_validate_all_tools_provided_passes(self, mock_async_openai: MagicMock) -> None:
"""Test that validation passes when all tools provided."""
provider = OpenAIAssistantProvider(mock_async_openai)
assistant_tools = [create_function_tool("get_weather")]
# Should not raise
provider._validate_function_tools(assistant_tools, [get_weather]) # type: ignore[reportPrivateUsage]
def test_validate_hosted_tools_not_required(self, mock_async_openai: MagicMock) -> None:
"""Test that hosted tools don't require implementations."""
provider = OpenAIAssistantProvider(mock_async_openai)
assistant_tools = [create_code_interpreter_tool(), create_file_search_tool()]
# Should not raise
provider._validate_function_tools(assistant_tools, None) # type: ignore[reportPrivateUsage]
def test_validate_with_tool(self, mock_async_openai: MagicMock) -> None:
"""Test validation with FunctionTool."""
provider = OpenAIAssistantProvider(mock_async_openai)
assistant_tools = [create_function_tool("get_weather")]
wrapped = tool(get_weather)
# Should not raise
provider._validate_function_tools(assistant_tools, [wrapped]) # type: ignore[reportPrivateUsage]
def test_validate_partial_tools_raises(self, mock_async_openai: MagicMock) -> None:
"""Test that partial tool provision raises error."""
provider = OpenAIAssistantProvider(mock_async_openai)
assistant_tools = [
create_function_tool("get_weather"),
create_function_tool("search_database"),
]
with pytest.raises(ValueError) as exc_info:
provider._validate_function_tools(assistant_tools, [get_weather]) # type: ignore[reportPrivateUsage]
assert "search_database" in str(exc_info.value)
# endregion
# region Tool Merging Tests
class TestToolMerging:
"""Tests for tool merging."""
def test_merge_code_interpreter(self, mock_async_openai: MagicMock) -> None:
"""Test merging code interpreter tool."""
provider = OpenAIAssistantProvider(mock_async_openai)
assistant_tools = [create_code_interpreter_tool()]
merged = provider._merge_tools(assistant_tools, None) # type: ignore[reportPrivateUsage]
assert len(merged) == 1
assert merged[0] == {"type": "code_interpreter"}
def test_merge_file_search(self, mock_async_openai: MagicMock) -> None:
"""Test merging file search tool."""
provider = OpenAIAssistantProvider(mock_async_openai)
assistant_tools = [create_file_search_tool()]
merged = provider._merge_tools(assistant_tools, None) # type: ignore[reportPrivateUsage]
assert len(merged) == 1
assert merged[0] == {"type": "file_search"}
def test_merge_with_user_tools(self, mock_async_openai: MagicMock) -> None:
"""Test merging hosted and user tools."""
provider = OpenAIAssistantProvider(mock_async_openai)
assistant_tools = [create_code_interpreter_tool()]
merged = provider._merge_tools(assistant_tools, [get_weather]) # type: ignore[reportPrivateUsage]
assert len(merged) == 2
assert merged[0] == {"type": "code_interpreter"}
def test_merge_multiple_hosted_tools(self, mock_async_openai: MagicMock) -> None:
"""Test merging multiple hosted tools."""
provider = OpenAIAssistantProvider(mock_async_openai)
assistant_tools = [create_code_interpreter_tool(), create_file_search_tool()]
merged = provider._merge_tools(assistant_tools, None) # type: ignore[reportPrivateUsage]
assert len(merged) == 2
def test_merge_single_user_tool(self, mock_async_openai: MagicMock) -> None:
"""Test merging with single user tool (not list)."""
provider = OpenAIAssistantProvider(mock_async_openai)
assistant_tools: list[Any] = []
merged = provider._merge_tools(assistant_tools, get_weather) # type: ignore[reportPrivateUsage]
assert len(merged) == 1
# endregion
# region Integration Tests
skip_if_openai_integration_tests_disabled = pytest.mark.skipif(
os.getenv("OPENAI_API_KEY", "") in ("", "test-dummy-key"),
reason="No real OPENAI_API_KEY provided; skipping integration tests.",
)
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_openai_integration_tests_disabled
class TestOpenAIAssistantProviderIntegration:
"""Integration tests requiring real OpenAI API."""
async def test_create_and_run_agent(self) -> None:
"""End-to-end test of creating and running an agent."""
provider = OpenAIAssistantProvider()
agent = await provider.create_agent(
name="IntegrationTestAgent",
model=os.environ.get("OPENAI_CHAT_MODEL_ID", "gpt-4"),
instructions="You are a helpful assistant. Respond briefly.",
)
try:
result = await agent.run("Say 'hello' and nothing else.")
result_text = str(result)
assert "hello" in result_text.lower()
finally:
# Clean up the assistant
await provider._client.beta.assistants.delete(agent.id) # type: ignore[reportPrivateUsage, union-attr]
async def test_create_agent_with_function_tools_integration(self) -> None:
"""Integration test with function tools."""
provider = OpenAIAssistantProvider()
@tool(approval_mode="never_require")
def get_current_time() -> str:
"""Get the current time."""
from datetime import datetime
return datetime.now().strftime("%H:%M")
agent = await provider.create_agent(
name="TimeAgent",
model=os.environ.get("OPENAI_CHAT_MODEL_ID", "gpt-4"),
instructions="You are a helpful assistant.",
tools=[get_current_time],
)
try:
result = await agent.run("What time is it? Use the get_current_time function.")
result_text = str(result)
# The response should contain time information
assert ":" in result_text or "time" in result_text.lower()
finally:
await provider._client.beta.assistants.delete(agent.id) # type: ignore[reportPrivateUsage, union-attr]
# endregion
python/packages/core/tests/openai/test_openai_assistants_client.py
-1833
View File
File diff suppressed because it is too large Load Diff
python/packages/core/tests/openai/test_openai_chat_client.py
-1607
View File
File diff suppressed because it is too large Load Diff
python/packages/core/tests/openai/test_openai_chat_client_base.py
-425
View File
@@ -1,425 +0,0 @@
# Copyright (c) Microsoft. All rights reserved.
from copy import deepcopy
from datetime import datetime, timezone
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from openai import AsyncStream
from openai.resources.chat.completions import AsyncCompletions as AsyncChatCompletions
from openai.types.chat import ChatCompletion, ChatCompletionChunk
from openai.types.chat.chat_completion import Choice
from openai.types.chat.chat_completion_chunk import Choice as ChunkChoice
from openai.types.chat.chat_completion_chunk import ChoiceDelta as ChunkChoiceDelta
from openai.types.chat.chat_completion_message import ChatCompletionMessage
from pydantic import BaseModel
from agent_framework import ChatResponseUpdate, Message
from agent_framework.exceptions import ChatClientException
from agent_framework.openai import OpenAIChatClient
async def mock_async_process_chat_stream_response(_):
mock_content = MagicMock(spec=ChatResponseUpdate)
yield mock_content, None
@pytest.fixture(scope="function")
def chat_history() -> list[Message]:
return []
@pytest.fixture
def mock_chat_completion_response() -> ChatCompletion:
return ChatCompletion(
id="test_id",
choices=[
Choice(index=0, message=ChatCompletionMessage(content="test", role="assistant"), finish_reason="stop")
],
created=0,
model="test",
object="chat.completion",
)
@pytest.fixture
def mock_streaming_chat_completion_response() -> AsyncStream[ChatCompletionChunk]:
content = ChatCompletionChunk(
id="test_id",
choices=[ChunkChoice(index=0, delta=ChunkChoiceDelta(content="test", role="assistant"), finish_reason="stop")],
created=0,
model="test",
object="chat.completion.chunk",
)
stream = MagicMock(spec=AsyncStream)
stream.__aiter__.return_value = [content]
return stream
# region Chat Message Content
@patch.object(AsyncChatCompletions, "create", new_callable=AsyncMock)
async def test_cmc(
mock_create: AsyncMock,
chat_history: list[Message],
mock_chat_completion_response: ChatCompletion,
openai_unit_test_env: dict[str, str],
):
mock_create.return_value = mock_chat_completion_response
chat_history.append(Message(role="user", text="hello world"))
openai_chat_completion = OpenAIChatClient()
await openai_chat_completion.get_response(messages=chat_history)
mock_create.assert_awaited_once_with(
model=openai_unit_test_env["OPENAI_CHAT_MODEL_ID"],
stream=False,
messages=openai_chat_completion._prepare_messages_for_openai(chat_history), # type: ignore
)
@patch.object(AsyncChatCompletions, "create", new_callable=AsyncMock)
async def test_cmc_chat_options(
mock_create: AsyncMock,
chat_history: list[Message],
mock_chat_completion_response: ChatCompletion,
openai_unit_test_env: dict[str, str],
):
mock_create.return_value = mock_chat_completion_response
chat_history.append(Message(role="user", text="hello world"))
openai_chat_completion = OpenAIChatClient()
await openai_chat_completion.get_response(
messages=chat_history,
)
mock_create.assert_awaited_once_with(
model=openai_unit_test_env["OPENAI_CHAT_MODEL_ID"],
stream=False,
messages=openai_chat_completion._prepare_messages_for_openai(chat_history), # type: ignore
)
@patch.object(AsyncChatCompletions, "create", new_callable=AsyncMock)
async def test_cmc_no_fcc_in_response(
mock_create: AsyncMock,
chat_history: list[Message],
mock_chat_completion_response: ChatCompletion,
openai_unit_test_env: dict[str, str],
):
mock_create.return_value = mock_chat_completion_response
chat_history.append(Message(role="user", text="hello world"))
orig_chat_history = deepcopy(chat_history)
openai_chat_completion = OpenAIChatClient()
await openai_chat_completion.get_response(
messages=chat_history,
)
mock_create.assert_awaited_once_with(
model=openai_unit_test_env["OPENAI_CHAT_MODEL_ID"],
stream=False,
messages=openai_chat_completion._prepare_messages_for_openai(orig_chat_history), # type: ignore
)
@patch.object(AsyncChatCompletions, "create", new_callable=AsyncMock)
async def test_cmc_structured_output_no_fcc(
mock_create: AsyncMock,
chat_history: list[Message],
mock_chat_completion_response: ChatCompletion,
openai_unit_test_env: dict[str, str],
):
mock_create.return_value = mock_chat_completion_response
chat_history.append(Message(role="user", text="hello world"))
# Define a mock response format
class Test(BaseModel):
name: str
openai_chat_completion = OpenAIChatClient()
await openai_chat_completion.get_response(
messages=chat_history,
response_format=Test,
)
mock_create.assert_awaited_once()
@patch.object(AsyncChatCompletions, "create", new_callable=AsyncMock)
async def test_scmc_chat_options(
mock_create: AsyncMock,
chat_history: list[Message],
mock_streaming_chat_completion_response: AsyncStream[ChatCompletionChunk],
openai_unit_test_env: dict[str, str],
):
mock_create.return_value = mock_streaming_chat_completion_response
chat_history.append(Message(role="user", text="hello world"))
openai_chat_completion = OpenAIChatClient()
async for msg in openai_chat_completion.get_response(
stream=True,
messages=chat_history,
):
assert isinstance(msg, ChatResponseUpdate)
assert msg.message_id is not None
assert msg.response_id is not None
mock_create.assert_awaited_once_with(
model=openai_unit_test_env["OPENAI_CHAT_MODEL_ID"],
stream=True,
stream_options={"include_usage": True},
messages=openai_chat_completion._prepare_messages_for_openai(chat_history), # type: ignore
)
@patch.object(AsyncChatCompletions, "create", new_callable=AsyncMock, side_effect=Exception)
async def test_cmc_general_exception(
mock_create: AsyncMock,
chat_history: list[Message],
mock_chat_completion_response: ChatCompletion,
openai_unit_test_env: dict[str, str],
):
mock_create.return_value = mock_chat_completion_response
chat_history.append(Message(role="user", text="hello world"))
openai_chat_completion = OpenAIChatClient()
with pytest.raises(ChatClientException):
await openai_chat_completion.get_response(
messages=chat_history,
)
@patch.object(AsyncChatCompletions, "create", new_callable=AsyncMock)
async def test_cmc_additional_properties(
mock_create: AsyncMock,
chat_history: list[Message],
mock_chat_completion_response: ChatCompletion,
openai_unit_test_env: dict[str, str],
):
mock_create.return_value = mock_chat_completion_response
chat_history.append(Message(role="user", text="hello world"))
openai_chat_completion = OpenAIChatClient()
await openai_chat_completion.get_response(messages=chat_history, options={"reasoning_effort": "low"})
mock_create.assert_awaited_once_with(
model=openai_unit_test_env["OPENAI_CHAT_MODEL_ID"],
stream=False,
messages=openai_chat_completion._prepare_messages_for_openai(chat_history), # type: ignore
reasoning_effort="low",
)
# region Streaming
@patch.object(AsyncChatCompletions, "create", new_callable=AsyncMock)
async def test_get_streaming(
mock_create: AsyncMock,
chat_history: list[Message],
openai_unit_test_env: dict[str, str],
):
content1 = ChatCompletionChunk(
id="test_id",
choices=[],
created=0,
model="test",
object="chat.completion.chunk",
)
content2 = ChatCompletionChunk(
id="test_id",
choices=[ChunkChoice(index=0, delta=ChunkChoiceDelta(content="test", role="assistant"), finish_reason="stop")],
created=0,
model="test",
object="chat.completion.chunk",
)
stream = MagicMock(spec=AsyncStream)
stream.__aiter__.return_value = [content1, content2]
mock_create.return_value = stream
chat_history.append(Message(role="user", text="hello world"))
orig_chat_history = deepcopy(chat_history)
openai_chat_completion = OpenAIChatClient()
async for msg in openai_chat_completion.get_response(
stream=True,
messages=chat_history,
):
assert isinstance(msg, ChatResponseUpdate)
mock_create.assert_awaited_once_with(
model=openai_unit_test_env["OPENAI_CHAT_MODEL_ID"],
stream=True,
stream_options={"include_usage": True},
messages=openai_chat_completion._prepare_messages_for_openai(orig_chat_history), # type: ignore
)
@patch.object(AsyncChatCompletions, "create", new_callable=AsyncMock)
async def test_get_streaming_singular(
mock_create: AsyncMock,
chat_history: list[Message],
openai_unit_test_env: dict[str, str],
):
content1 = ChatCompletionChunk(
id="test_id",
choices=[],
created=0,
model="test",
object="chat.completion.chunk",
)
content2 = ChatCompletionChunk(
id="test_id",
choices=[ChunkChoice(index=0, delta=ChunkChoiceDelta(content="test", role="assistant"), finish_reason="stop")],
created=0,
model="test",
object="chat.completion.chunk",
)
stream = MagicMock(spec=AsyncStream)
stream.__aiter__.return_value = [content1, content2]
mock_create.return_value = stream
chat_history.append(Message(role="user", text="hello world"))
orig_chat_history = deepcopy(chat_history)
openai_chat_completion = OpenAIChatClient()
async for msg in openai_chat_completion.get_response(
stream=True,
messages=chat_history,
):
assert isinstance(msg, ChatResponseUpdate)
mock_create.assert_awaited_once_with(
model=openai_unit_test_env["OPENAI_CHAT_MODEL_ID"],
stream=True,
stream_options={"include_usage": True},
messages=openai_chat_completion._prepare_messages_for_openai(orig_chat_history), # type: ignore
)
@patch.object(AsyncChatCompletions, "create", new_callable=AsyncMock)
async def test_get_streaming_structured_output_no_fcc(
mock_create: AsyncMock,
chat_history: list[Message],
openai_unit_test_env: dict[str, str],
):
content1 = ChatCompletionChunk(
id="test_id",
choices=[],
created=0,
model="test",
object="chat.completion.chunk",
)
content2 = ChatCompletionChunk(
id="test_id",
choices=[ChunkChoice(index=0, delta=ChunkChoiceDelta(content="test", role="assistant"), finish_reason="stop")],
created=0,
model="test",
object="chat.completion.chunk",
)
stream = MagicMock(spec=AsyncStream)
stream.__aiter__.return_value = [content1, content2]
mock_create.return_value = stream
chat_history.append(Message(role="user", text="hello world"))
# Define a mock response format
class Test(BaseModel):
name: str
openai_chat_completion = OpenAIChatClient()
async for msg in openai_chat_completion.get_response(
stream=True,
messages=chat_history,
response_format=Test,
):
assert isinstance(msg, ChatResponseUpdate)
mock_create.assert_awaited_once()
@patch.object(AsyncChatCompletions, "create", new_callable=AsyncMock)
async def test_get_streaming_no_fcc_in_response(
mock_create: AsyncMock,
chat_history: list[Message],
mock_streaming_chat_completion_response: ChatCompletion,
openai_unit_test_env: dict[str, str],
):
mock_create.return_value = mock_streaming_chat_completion_response
chat_history.append(Message(role="user", text="hello world"))
orig_chat_history = deepcopy(chat_history)
openai_chat_completion = OpenAIChatClient()
[
msg
async for msg in openai_chat_completion.get_response(
stream=True,
messages=chat_history,
)
]
mock_create.assert_awaited_once_with(
model=openai_unit_test_env["OPENAI_CHAT_MODEL_ID"],
stream=True,
stream_options={"include_usage": True},
messages=openai_chat_completion._prepare_messages_for_openai(orig_chat_history), # type: ignore
)
# region UTC Timestamp Tests
def test_chat_response_created_at_uses_utc(openai_unit_test_env: dict[str, str]):
"""Test that ChatResponse.created_at uses UTC timestamp, not local time.
This is a regression test for the issue where created_at was using local time
but labeling it as UTC (with 'Z' suffix).
"""
# Use a specific Unix timestamp: 1733011890 = 2024-12-01T00:31:30Z (UTC)
# This ensures we test that the timestamp is actually converted to UTC
utc_timestamp = 1733011890
mock_response = ChatCompletion(
id="test_id",
choices=[
Choice(index=0, message=ChatCompletionMessage(content="test", role="assistant"), finish_reason="stop")
],
created=utc_timestamp,
model="test",
object="chat.completion",
)
client = OpenAIChatClient()
response = client._parse_response_from_openai(mock_response, {})
# Verify that created_at is correctly formatted as UTC
assert response.created_at is not None
assert response.created_at.endswith("Z"), "Timestamp should end with 'Z' for UTC"
# Parse the timestamp and verify it matches UTC time
expected_utc_time = datetime.fromtimestamp(utc_timestamp, tz=timezone.utc)
expected_formatted = expected_utc_time.strftime("%Y-%m-%dT%H:%M:%S.%fZ")
assert response.created_at == expected_formatted, (
f"Expected UTC timestamp {expected_formatted}, got {response.created_at}"
)
def test_chat_response_update_created_at_uses_utc(openai_unit_test_env: dict[str, str]):
"""Test that ChatResponseUpdate.created_at uses UTC timestamp, not local time.
This is a regression test for the issue where created_at was using local time
but labeling it as UTC (with 'Z' suffix).
"""
# Use a specific Unix timestamp: 1733011890 = 2024-12-01T00:31:30Z (UTC)
utc_timestamp = 1733011890
mock_chunk = ChatCompletionChunk(
id="test_id",
choices=[ChunkChoice(index=0, delta=ChunkChoiceDelta(content="test", role="assistant"), finish_reason="stop")],
created=utc_timestamp,
model="test",
object="chat.completion.chunk",
)
client = OpenAIChatClient()
response_update = client._parse_response_update_from_openai(mock_chunk)
# Verify that created_at is correctly formatted as UTC
assert response_update.created_at is not None
assert response_update.created_at.endswith("Z"), "Timestamp should end with 'Z' for UTC"
# Parse the timestamp and verify it matches UTC time
expected_utc_time = datetime.fromtimestamp(utc_timestamp, tz=timezone.utc)
expected_formatted = expected_utc_time.strftime("%Y-%m-%dT%H:%M:%S.%fZ")
assert response_update.created_at == expected_formatted, (
f"Expected UTC timestamp {expected_formatted}, got {response_update.created_at}"
)
python/packages/core/tests/openai/test_openai_embedding_client.py
-243
View File
@@ -1,243 +0,0 @@
# Copyright (c) Microsoft. All rights reserved.
from __future__ import annotations
import os
from unittest.mock import AsyncMock, MagicMock
import pytest
from openai.types import CreateEmbeddingResponse
from openai.types import Embedding as OpenAIEmbedding
from openai.types.create_embedding_response import Usage
from agent_framework.openai import (
OpenAIEmbeddingClient,
OpenAIEmbeddingOptions,
)
def _make_openai_response(
embeddings: list[list[float]],
model: str = "text-embedding-3-small",
prompt_tokens: int = 5,
total_tokens: int = 5,
) -> CreateEmbeddingResponse:
"""Helper to create a mock OpenAI embeddings response."""
data = [OpenAIEmbedding(embedding=emb, index=i, object="embedding") for i, emb in enumerate(embeddings)]
return CreateEmbeddingResponse(
data=data,
model=model,
object="list",
usage=Usage(prompt_tokens=prompt_tokens, total_tokens=total_tokens),
)
@pytest.fixture
def openai_unit_test_env(monkeypatch: pytest.MonkeyPatch) -> None:
"""Set up environment variables for OpenAI embedding client."""
monkeypatch.setenv("OPENAI_API_KEY", "test-api-key")
monkeypatch.setenv("OPENAI_EMBEDDING_MODEL_ID", "text-embedding-3-small")
# --- OpenAI unit tests ---
def test_openai_construction_with_explicit_params() -> None:
client = OpenAIEmbeddingClient(
model_id="text-embedding-3-small",
api_key="test-key",
)
assert client.model_id == "text-embedding-3-small"
def test_openai_construction_from_env(openai_unit_test_env: None) -> None:
client = OpenAIEmbeddingClient()
assert client.model_id == "text-embedding-3-small"
def test_openai_construction_missing_api_key_raises(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.delenv("OPENAI_API_KEY", raising=False)
with pytest.raises(ValueError, match="API key is required"):
OpenAIEmbeddingClient(model_id="text-embedding-3-small")
def test_openai_construction_missing_model_raises(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.delenv("OPENAI_EMBEDDING_MODEL_ID", raising=False)
with pytest.raises(ValueError, match="model ID is required"):
OpenAIEmbeddingClient(api_key="test-key")
async def test_openai_get_embeddings(openai_unit_test_env: None) -> None:
mock_response = _make_openai_response(
embeddings=[[0.1, 0.2, 0.3], [0.4, 0.5, 0.6]],
)
client = OpenAIEmbeddingClient()
client.client = MagicMock()
client.client.embeddings = MagicMock()
client.client.embeddings.create = AsyncMock(return_value=mock_response)
result = await client.get_embeddings(["hello", "world"])
assert len(result) == 2
assert result[0].vector == [0.1, 0.2, 0.3]
assert result[1].vector == [0.4, 0.5, 0.6]
assert result[0].model_id == "text-embedding-3-small"
assert result[0].dimensions == 3
async def test_openai_get_embeddings_usage(openai_unit_test_env: None) -> None:
mock_response = _make_openai_response(
embeddings=[[0.1]],
prompt_tokens=10,
total_tokens=10,
)
client = OpenAIEmbeddingClient()
client.client = MagicMock()
client.client.embeddings = MagicMock()
client.client.embeddings.create = AsyncMock(return_value=mock_response)
result = await client.get_embeddings(["test"])
assert result.usage is not None
assert result.usage["input_token_count"] == 10
assert result.usage["total_token_count"] == 10
async def test_openai_options_passthrough_dimensions(openai_unit_test_env: None) -> None:
mock_response = _make_openai_response(embeddings=[[0.1]])
client = OpenAIEmbeddingClient()
client.client = MagicMock()
client.client.embeddings = MagicMock()
client.client.embeddings.create = AsyncMock(return_value=mock_response)
options: OpenAIEmbeddingOptions = {"dimensions": 256}
result = await client.get_embeddings(["test"], options=options)
call_kwargs = client.client.embeddings.create.call_args[1]
assert call_kwargs["dimensions"] == 256
assert result.options is options
async def test_openai_options_passthrough_encoding_format(openai_unit_test_env: None) -> None:
mock_response = _make_openai_response(embeddings=[[0.1]])
client = OpenAIEmbeddingClient()
client.client = MagicMock()
client.client.embeddings = MagicMock()
client.client.embeddings.create = AsyncMock(return_value=mock_response)
options: OpenAIEmbeddingOptions = {"encoding_format": "base64"}
await client.get_embeddings(["test"], options=options)
call_kwargs = client.client.embeddings.create.call_args[1]
assert call_kwargs["encoding_format"] == "base64"
async def test_openai_base64_decoding(openai_unit_test_env: None) -> None:
import base64
import struct
# Encode [0.1, 0.2, 0.3] as base64 little-endian floats
raw_floats = [0.1, 0.2, 0.3]
b64_str = base64.b64encode(struct.pack(f"<{len(raw_floats)}f", *raw_floats)).decode()
# Mock the embedding item to return a base64 string (as the API does with encoding_format=base64)
mock_item = MagicMock()
mock_item.embedding = b64_str
mock_item.index = 0
mock_response = MagicMock()
mock_response.data = [mock_item]
mock_response.model = "text-embedding-3-small"
mock_response.usage = MagicMock(prompt_tokens=3, total_tokens=3)
client = OpenAIEmbeddingClient()
client.client = MagicMock()
client.client.embeddings = MagicMock()
client.client.embeddings.create = AsyncMock(return_value=mock_response)
options: OpenAIEmbeddingOptions = {"encoding_format": "base64"}
result = await client.get_embeddings(["test"], options=options)
assert len(result) == 1
assert len(result[0].vector) == 3
assert result[0].dimensions == 3
for expected, actual in zip(raw_floats, result[0].vector):
assert abs(expected - actual) < 1e-6
async def test_openai_error_when_no_model_id() -> None:
client = OpenAIEmbeddingClient.__new__(OpenAIEmbeddingClient)
client.model_id = None
client.client = MagicMock()
client.additional_properties = {}
client.otel_provider_name = "openai"
with pytest.raises(ValueError, match="model_id is required"):
await client.get_embeddings(["test"])
async def test_openai_empty_values_returns_empty(openai_unit_test_env: None) -> None:
client = OpenAIEmbeddingClient()
client.client = MagicMock()
client.client.embeddings = MagicMock()
client.client.embeddings.create = AsyncMock()
result = await client.get_embeddings([])
assert len(result) == 0
assert result.usage is None
client.client.embeddings.create.assert_not_called()
# --- Integration tests ---
skip_if_openai_integration_tests_disabled = pytest.mark.skipif(
os.getenv("OPENAI_API_KEY", "") in ("", "test-dummy-key"),
reason="No real OPENAI_API_KEY provided; skipping integration tests.",
)
@skip_if_openai_integration_tests_disabled
@pytest.mark.flaky
@pytest.mark.integration
async def test_integration_openai_get_embeddings() -> None:
"""End-to-end test of OpenAI embedding generation."""
client = OpenAIEmbeddingClient(model_id="text-embedding-3-small")
result = await client.get_embeddings(["hello world"])
assert len(result) == 1
assert isinstance(result[0].vector, list)
assert len(result[0].vector) > 0
assert all(isinstance(v, float) for v in result[0].vector)
assert result[0].model_id is not None
assert result.usage is not None
assert result.usage["input_token_count"] > 0
@skip_if_openai_integration_tests_disabled
@pytest.mark.flaky
@pytest.mark.integration
async def test_integration_openai_get_embeddings_multiple() -> None:
"""Test embedding generation for multiple inputs."""
client = OpenAIEmbeddingClient(model_id="text-embedding-3-small")
result = await client.get_embeddings(["hello", "world", "test"])
assert len(result) == 3
dims = [len(e.vector) for e in result]
assert all(d == dims[0] for d in dims)
@skip_if_openai_integration_tests_disabled
@pytest.mark.flaky
@pytest.mark.integration
async def test_integration_openai_get_embeddings_with_dimensions() -> None:
"""Test embedding generation with custom dimensions."""
client = OpenAIEmbeddingClient(model_id="text-embedding-3-small")
options: OpenAIEmbeddingOptions = {"dimensions": 256}
result = await client.get_embeddings(["hello world"], options=options)
assert len(result) == 1
assert len(result[0].vector) == 256
python/packages/core/tests/openai/test_openai_responses_client.py
-3776
View File
File diff suppressed because it is too large Load Diff