agent/agent-framework
1
0
Fork 0
mirror of https://github.com/microsoft/agent-framework.git synced 2026-06-16 21:04:09 +08:00
Code Issues Packages Projects Releases Wiki Activity

Python: updated integration tests and guidance (#4181)

Browse Source 
* updated integration tests and guidance

* fixed merge test

* updated integration tests

* fix: remove duplicate --dist loadfile flag from pytest-xdist config

Only one --dist mode can be active at a time; the second value silently
overrides the first. Keep --dist worksteal (dynamic load balancing) and
remove the redundant --dist loadfile from all workflow files and
pyproject.toml configs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: add keep-in-sync notes for merge and integration test workflows

Both python-merge-tests.yml and python-integration-tests.yml share the
same parallel job structure. Added sync reminders in workflow file
comments, the python-testing SKILL.md, and CODING_STANDARD.md.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* refactor: remove RUN_INTEGRATION_TESTS flag

Integration test gating now uses two mechanisms:
- `@pytest.mark.integration` for test selection via `-m` filtering
- `skip_if_*_disabled` for credential/service availability checks

The RUN_INTEGRATION_TESTS env var was redundant since the marker handles
selection and the skip decorators already check for actual credentials.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: sync missing env vars from merge-tests to integration-tests

Add OPENAI_EMBEDDINGS_MODEL_ID and AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME
to python-integration-tests.yml to match python-merge-tests.yml.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: remove remaining RUN_INTEGRATION_TESTS from embedding tests and docs

Missed test_openai_embedding_client.py and vector-stores README in the
earlier cleanup.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* set functions tests to 3.10

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
Eduard van Valkenburg
2026-02-24 10:35:46 +01:00
committed by GitHub
Unverified
parent 6305e3e092
commit acc49196c1
52 changed files with 731 additions and 212 deletions
Download Patch File Download Diff File Expand all files Collapse all files
python/packages/ag-ui/pyproject.toml
+1 -1
View File
@@ -70,4 +70,4 @@ include = "../../shared_tasks.toml"
[tool.poe.tasks]
mypy = "mypy --config-file $POE_ROOT/pyproject.toml agent_framework_ag_ui"
test = "pytest --cov=agent_framework_ag_ui --cov-report=term-missing:skip-covered tests/ag_ui"
test = "pytest --cov=agent_framework_ag_ui --cov-report=term-missing:skip-covered -n auto --dist worksteal tests/ag_ui"
python/packages/anthropic/pyproject.toml
+1 -1
View File
@@ -82,7 +82,7 @@ executor.type = "uv"
include = "../../shared_tasks.toml"
[tool.poe.tasks]
mypy = "mypy --config-file $POE_ROOT/pyproject.toml agent_framework_anthropic"
test = "pytest --cov=agent_framework_anthropic --cov-report=term-missing:skip-covered tests"
test = "pytest --cov=agent_framework_anthropic --cov-report=term-missing:skip-covered -n auto --dist worksteal tests"
[build-system]
requires = ["flit-core >= 3.11,<4.0"]
python/packages/anthropic/tests/test_anthropic_client.py
+10 -5
View File
@@ -29,11 +29,8 @@ from agent_framework_anthropic._chat_client import AnthropicSettings
VALID_PNG_BASE64 = b"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg=="
skip_if_anthropic_integration_tests_disabled = pytest.mark.skipif(
os.getenv("RUN_INTEGRATION_TESTS", "false").lower() != "true"
or os.getenv("ANTHROPIC_API_KEY", "") in ("", "test-api-key-12345"),
reason="No real ANTHROPIC_API_KEY provided; skipping integration tests."
if os.getenv("RUN_INTEGRATION_TESTS", "false").lower() == "true"
else "Integration tests are disabled.",
os.getenv("ANTHROPIC_API_KEY", "") in ("", "test-api-key-12345"),
reason="No real ANTHROPIC_API_KEY provided; skipping integration tests.",
)
@@ -915,6 +912,7 @@ def get_weather(
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_anthropic_integration_tests_disabled
async def test_anthropic_client_integration_basic_chat() -> None:
"""Integration test for basic chat completion."""
@@ -932,6 +930,7 @@ async def test_anthropic_client_integration_basic_chat() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_anthropic_integration_tests_disabled
async def test_anthropic_client_integration_streaming_chat() -> None:
"""Integration test for streaming chat completion."""
@@ -948,6 +947,7 @@ async def test_anthropic_client_integration_streaming_chat() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_anthropic_integration_tests_disabled
async def test_anthropic_client_integration_function_calling() -> None:
"""Integration test for function calling."""
@@ -968,6 +968,7 @@ async def test_anthropic_client_integration_function_calling() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_anthropic_integration_tests_disabled
async def test_anthropic_client_integration_hosted_tools() -> None:
"""Integration test for hosted tools."""
@@ -993,6 +994,7 @@ async def test_anthropic_client_integration_hosted_tools() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_anthropic_integration_tests_disabled
async def test_anthropic_client_integration_with_system_message() -> None:
"""Integration test with system message."""
@@ -1010,6 +1012,7 @@ async def test_anthropic_client_integration_with_system_message() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_anthropic_integration_tests_disabled
async def test_anthropic_client_integration_temperature_control() -> None:
"""Integration test with temperature control."""
@@ -1027,6 +1030,7 @@ async def test_anthropic_client_integration_temperature_control() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_anthropic_integration_tests_disabled
async def test_anthropic_client_integration_ordering() -> None:
"""Integration test with ordering."""
@@ -1047,6 +1051,7 @@ async def test_anthropic_client_integration_ordering() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_anthropic_integration_tests_disabled
async def test_anthropic_client_integration_images() -> None:
"""Integration test with images."""
python/packages/azure-ai/pyproject.toml
+1 -1
View File
@@ -85,7 +85,7 @@ test = "pytest --cov=agent_framework_azure_ai --cov-report=term-missing:skip-cov
[tool.poe.tasks.integration-tests]
cmd = """
pytest --import-mode=importlib
-n logical --dist loadfile --dist worksteal
-n logical --dist worksteal
tests
"""
python/packages/azure-ai/tests/test_agent_provider.py
+8 -5
View File
@@ -29,11 +29,8 @@ from agent_framework_azure_ai._shared import (
)
skip_if_azure_ai_integration_tests_disabled = pytest.mark.skipif(
os.getenv("RUN_INTEGRATION_TESTS", "false").lower() != "true"
or os.getenv("AZURE_AI_PROJECT_ENDPOINT", "") in ("", "https://test-project.cognitiveservices.azure.com/"),
reason="No real AZURE_AI_PROJECT_ENDPOINT provided; skipping integration tests."
if os.getenv("RUN_INTEGRATION_TESTS", "false").lower() == "true"
else "Integration tests are disabled.",
os.getenv("AZURE_AI_PROJECT_ENDPOINT", "") in ("", "https://test-project.cognitiveservices.azure.com/"),
reason="No real AZURE_AI_PROJECT_ENDPOINT provided; skipping integration tests.",
)
# region Provider Initialization Tests
@@ -779,6 +776,8 @@ def test_from_azure_ai_agent_tools_unknown_dict() -> None:
# region Integration Tests
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_ai_integration_tests_disabled
async def test_integration_create_agent() -> None:
"""Integration test: Create an agent using the provider."""
@@ -801,6 +800,8 @@ async def test_integration_create_agent() -> None:
await provider._agents_client.delete_agent(agent.id) # type: ignore
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_ai_integration_tests_disabled
async def test_integration_get_agent() -> None:
"""Integration test: Get an existing agent using the provider."""
@@ -825,6 +826,8 @@ async def test_integration_get_agent() -> None:
await provider._agents_client.delete_agent(created.id) # type: ignore
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_ai_integration_tests_disabled
async def test_integration_create_and_run() -> None:
"""Integration test: Create an agent and run a conversation."""
python/packages/azure-ai/tests/test_azure_ai_agent_client.py
+16 -5
View File
@@ -50,11 +50,8 @@ from pydantic import BaseModel, Field
from agent_framework_azure_ai import AzureAIAgentClient, AzureAISettings
skip_if_azure_ai_integration_tests_disabled = pytest.mark.skipif(
os.getenv("RUN_INTEGRATION_TESTS", "false").lower() != "true"
or os.getenv("AZURE_AI_PROJECT_ENDPOINT", "") in ("", "https://test-project.cognitiveservices.azure.com/"),
reason="No real AZURE_AI_PROJECT_ENDPOINT provided; skipping integration tests."
if os.getenv("RUN_INTEGRATION_TESTS", "false").lower() == "true"
else "Integration tests are disabled.",
os.getenv("AZURE_AI_PROJECT_ENDPOINT", "") in ("", "https://test-project.cognitiveservices.azure.com/"),
reason="No real AZURE_AI_PROJECT_ENDPOINT provided; skipping integration tests.",
)
@@ -1379,6 +1376,7 @@ def get_weather(
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_ai_integration_tests_disabled
async def test_azure_ai_chat_client_get_response() -> None:
"""Test Azure AI Chat Client response."""
@@ -1404,6 +1402,7 @@ async def test_azure_ai_chat_client_get_response() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_ai_integration_tests_disabled
async def test_azure_ai_chat_client_get_response_tools() -> None:
"""Test Azure AI Chat Client response with tools."""
@@ -1425,6 +1424,7 @@ async def test_azure_ai_chat_client_get_response_tools() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_ai_integration_tests_disabled
async def test_azure_ai_chat_client_streaming() -> None:
"""Test Azure AI Chat Client streaming response."""
@@ -1456,6 +1456,7 @@ async def test_azure_ai_chat_client_streaming() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_ai_integration_tests_disabled
async def test_azure_ai_chat_client_streaming_tools() -> None:
"""Test Azure AI Chat Client streaming response with tools."""
@@ -1483,6 +1484,7 @@ async def test_azure_ai_chat_client_streaming_tools() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_ai_integration_tests_disabled
async def test_azure_ai_chat_client_agent_basic_run() -> None:
"""Test Agent basic run functionality with AzureAIAgentClient."""
@@ -1500,6 +1502,7 @@ async def test_azure_ai_chat_client_agent_basic_run() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_ai_integration_tests_disabled
async def test_azure_ai_chat_client_agent_basic_run_streaming() -> None:
"""Test Agent basic streaming functionality with AzureAIAgentClient."""
@@ -1520,6 +1523,7 @@ async def test_azure_ai_chat_client_agent_basic_run_streaming() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_ai_integration_tests_disabled
async def test_azure_ai_chat_client_agent_thread_persistence() -> None:
"""Test Agent session persistence across runs with AzureAIAgentClient."""
@@ -1546,6 +1550,7 @@ async def test_azure_ai_chat_client_agent_thread_persistence() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_ai_integration_tests_disabled
async def test_azure_ai_chat_client_agent_existing_thread_id() -> None:
"""Test Agent existing thread ID functionality with AzureAIAgentClient."""
@@ -1584,6 +1589,7 @@ async def test_azure_ai_chat_client_agent_existing_thread_id() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_ai_integration_tests_disabled
async def test_azure_ai_chat_client_agent_code_interpreter():
"""Test Agent with code interpreter through AzureAIAgentClient."""
@@ -1604,6 +1610,7 @@ async def test_azure_ai_chat_client_agent_code_interpreter():
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_ai_integration_tests_disabled
async def test_azure_ai_chat_client_agent_file_search():
"""Test Agent with file search through AzureAIAgentClient."""
@@ -1651,6 +1658,7 @@ async def test_azure_ai_chat_client_agent_file_search():
await client.close()
@pytest.mark.integration
@skip_if_azure_ai_integration_tests_disabled
async def test_azure_ai_chat_client_agent_hosted_mcp_tool() -> None:
"""Integration test for MCP tool with Azure AI Agent using Microsoft Learn MCP."""
@@ -1686,6 +1694,7 @@ async def test_azure_ai_chat_client_agent_hosted_mcp_tool() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_ai_integration_tests_disabled
async def test_azure_ai_chat_client_agent_level_tool_persistence():
"""Test that agent-level tools persist across multiple runs with AzureAIAgentClient."""
@@ -1711,6 +1720,7 @@ async def test_azure_ai_chat_client_agent_level_tool_persistence():
assert any(term in second_response.text.lower() for term in ["miami", "sunny", "25"])
@pytest.mark.integration
@skip_if_azure_ai_integration_tests_disabled
async def test_azure_ai_chat_client_agent_chat_options_run_level() -> None:
"""Test ChatOptions parameter coverage at run level."""
@@ -1735,6 +1745,7 @@ async def test_azure_ai_chat_client_agent_chat_options_run_level() -> None:
assert len(response.text) > 0
@pytest.mark.integration
@skip_if_azure_ai_integration_tests_disabled
async def test_azure_ai_chat_client_agent_chat_options_agent_level() -> None:
"""Test ChatOptions parameter coverage agent level."""
python/packages/azure-ai/tests/test_azure_ai_client.py
+8 -7
View File
@@ -47,14 +47,9 @@ from agent_framework_azure_ai import AzureAIClient, AzureAISettings
from agent_framework_azure_ai._shared import from_azure_ai_tools
skip_if_azure_ai_integration_tests_disabled = pytest.mark.skipif(
os.getenv("RUN_INTEGRATION_TESTS", "false").lower() != "true"
or os.getenv("AZURE_AI_PROJECT_ENDPOINT", "") in ("", "https://test-project.cognitiveservices.azure.com/")
os.getenv("AZURE_AI_PROJECT_ENDPOINT", "") in ("", "https://test-project.cognitiveservices.azure.com/")
or os.getenv("AZURE_AI_MODEL_DEPLOYMENT_NAME", "") == "",
reason=(
"No real AZURE_AI_PROJECT_ENDPOINT or AZURE_AI_MODEL_DEPLOYMENT_NAME provided; skipping integration tests."
if os.getenv("RUN_INTEGRATION_TESTS", "false").lower() == "true"
else "Integration tests are disabled."
),
reason="No real AZURE_AI_PROJECT_ENDPOINT or AZURE_AI_MODEL_DEPLOYMENT_NAME provided; skipping integration tests.",
)
@@ -1329,6 +1324,7 @@ async def client() -> AsyncGenerator[AzureAIClient, None]:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_ai_integration_tests_disabled
@pytest.mark.parametrize(
"option_name,option_value,needs_validation",
@@ -1443,6 +1439,7 @@ async def test_integration_options(
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_ai_integration_tests_disabled
@pytest.mark.parametrize(
"option_name,option_value,needs_validation",
@@ -1541,6 +1538,7 @@ async def test_integration_agent_options(
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_ai_integration_tests_disabled
async def test_integration_web_search() -> None:
async with temporary_chat_client(agent_name="af-int-test-web-search") as client:
@@ -1586,6 +1584,7 @@ async def test_integration_web_search() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_ai_integration_tests_disabled
async def test_integration_agent_hosted_mcp_tool() -> None:
"""Integration test for MCP tool with Azure Response Agent using Microsoft Learn MCP."""
@@ -1610,6 +1609,7 @@ async def test_integration_agent_hosted_mcp_tool() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_ai_integration_tests_disabled
async def test_integration_agent_hosted_code_interpreter_tool():
"""Test Azure Responses Client agent with code interpreter tool through AzureAIClient."""
@@ -1628,6 +1628,7 @@ async def test_integration_agent_hosted_code_interpreter_tool():
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_ai_integration_tests_disabled
async def test_integration_agent_existing_session():
"""Test Azure Responses Client agent with existing session to continue conversations across agent instances."""
python/packages/azure-ai/tests/test_provider.py
+3 -7
View File
@@ -20,14 +20,9 @@ from azure.identity.aio import AzureCliCredential
from agent_framework_azure_ai import AzureAIProjectAgentProvider
skip_if_azure_ai_integration_tests_disabled = pytest.mark.skipif(
os.getenv("RUN_INTEGRATION_TESTS", "false").lower() != "true"
or os.getenv("AZURE_AI_PROJECT_ENDPOINT", "") in ("", "https://test-project.cognitiveservices.azure.com/")
os.getenv("AZURE_AI_PROJECT_ENDPOINT", "") in ("", "https://test-project.cognitiveservices.azure.com/")
or os.getenv("AZURE_AI_MODEL_DEPLOYMENT_NAME", "") == "",
reason=(
"No real AZURE_AI_PROJECT_ENDPOINT or AZURE_AI_MODEL_DEPLOYMENT_NAME provided; skipping integration tests."
if os.getenv("RUN_INTEGRATION_TESTS", "false").lower() == "true"
else "Integration tests are disabled."
),
reason="No real AZURE_AI_PROJECT_ENDPOINT or AZURE_AI_MODEL_DEPLOYMENT_NAME provided; skipping integration tests.",
)
@@ -698,6 +693,7 @@ async def test_provider_create_agent_with_mcp_and_regular_tools(
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_ai_integration_tests_disabled
async def test_provider_create_and_get_agent_integration() -> None:
"""Integration test for provider create_agent and get_agent."""
python/packages/azurefunctions/tests/integration_tests/.env.example
-1
View File
@@ -2,7 +2,6 @@
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
AZURE_OPENAI_CHAT_DEPLOYMENT_NAME=your-deployment-name
FUNCTIONS_WORKER_RUNTIME=python
RUN_INTEGRATION_TESTS=true
# Azure Functions Configuration
AzureWebJobsStorage=UseDevelopmentStorage=true
python/packages/azurefunctions/tests/integration_tests/conftest.py
-7
View File
@@ -90,13 +90,6 @@ def _should_skip_azure_functions_integration_tests() -> tuple[bool, str]:
"""Determine whether Azure Functions integration tests should be skipped."""
_load_env_file_if_present()
run_integration_tests = os.getenv("RUN_INTEGRATION_TESTS", "false").lower() == "true"
if not run_integration_tests:
return (
True,
"Integration tests are disabled. Set RUN_INTEGRATION_TESTS=true to enable Azure Functions sample tests.",
)
# Check for Azure Functions Core Tools
if not _check_func_cli_available():
return (
python/packages/azurefunctions/tests/integration_tests/test_01_single_agent.py
+2
View File
@@ -19,6 +19,8 @@ from agent_framework_durabletask import THREAD_ID_HEADER
# Module-level markers - applied to all tests in this file
pytestmark = [
pytest.mark.flaky,
pytest.mark.integration,
pytest.mark.sample("01_single_agent"),
pytest.mark.usefixtures("function_app_for_test"),
]
python/packages/azurefunctions/tests/integration_tests/test_02_multi_agent.py
+2
View File
@@ -18,6 +18,8 @@ import pytest
# Module-level markers - applied to all tests in this file
pytestmark = [
pytest.mark.flaky,
pytest.mark.integration,
pytest.mark.sample("02_multi_agent"),
pytest.mark.usefixtures("function_app_for_test"),
]
python/packages/azurefunctions/tests/integration_tests/test_03_reliable_streaming.py
+2
View File
@@ -22,6 +22,8 @@ import requests
# Module-level markers - applied to all tests in this file
pytestmark = [
pytest.mark.flaky,
pytest.mark.integration,
pytest.mark.sample("03_reliable_streaming"),
pytest.mark.usefixtures("function_app_for_test"),
pytest.mark.skip(reason="Temp disabled to fix test instability - needs investigation into root cause"),
python/packages/azurefunctions/tests/integration_tests/test_04_single_agent_orchestration_chaining.py
+2
View File
@@ -22,6 +22,8 @@ import pytest
# Module-level markers - applied to all tests in this file
pytestmark = [
pytest.mark.flaky,
pytest.mark.integration,
pytest.mark.sample("04_single_agent_orchestration_chaining"),
pytest.mark.usefixtures("function_app_for_test"),
]
python/packages/azurefunctions/tests/integration_tests/test_05_multi_agent_orchestration_concurrency.py
+2
View File
@@ -22,6 +22,8 @@ import pytest
# Module-level markers - applied to all tests in this file
pytestmark = [
pytest.mark.flaky,
pytest.mark.integration,
pytest.mark.orchestration,
pytest.mark.sample("05_multi_agent_orchestration_concurrency"),
pytest.mark.usefixtures("function_app_for_test"),
python/packages/azurefunctions/tests/integration_tests/test_06_multi_agent_orchestration_conditionals.py
+2
View File
@@ -22,6 +22,8 @@ import pytest
# Module-level markers - applied to all tests in this file
pytestmark = [
pytest.mark.flaky,
pytest.mark.integration,
pytest.mark.orchestration,
pytest.mark.sample("06_multi_agent_orchestration_conditionals"),
pytest.mark.usefixtures("function_app_for_test"),
python/packages/azurefunctions/tests/integration_tests/test_07_single_agent_orchestration_hitl.py
+2
View File
@@ -24,6 +24,8 @@ import pytest
# Module-level markers - applied to all tests in this file
pytestmark = [
pytest.mark.flaky,
pytest.mark.integration,
pytest.mark.sample("07_single_agent_orchestration_hitl"),
pytest.mark.usefixtures("function_app_for_test"),
]
python/packages/azurefunctions/tests/integration_tests/test_09_workflow_shared_state.py
+2
View File
@@ -23,6 +23,8 @@ import pytest
# Module-level markers - applied to all tests in this file
pytestmark = [
pytest.mark.flaky,
pytest.mark.integration,
pytest.mark.sample("09_workflow_shared_state"),
pytest.mark.usefixtures("function_app_for_test"),
]
python/packages/azurefunctions/tests/integration_tests/test_10_workflow_no_shared_state.py
+2
View File
@@ -23,6 +23,8 @@ import pytest
# Module-level markers - applied to all tests in this file
pytestmark = [
pytest.mark.flaky,
pytest.mark.integration,
pytest.mark.sample("10_workflow_no_shared_state"),
pytest.mark.usefixtures("function_app_for_test"),
]
python/packages/azurefunctions/tests/integration_tests/test_11_workflow_parallel.py
+2
View File
@@ -25,6 +25,8 @@ import pytest
# Module-level markers - applied to all tests in this file
pytestmark = [
pytest.mark.flaky,
pytest.mark.integration,
pytest.mark.sample("11_workflow_parallel"),
pytest.mark.usefixtures("function_app_for_test"),
]
python/packages/azurefunctions/tests/integration_tests/test_12_workflow_hitl.py
+2
View File
@@ -25,6 +25,8 @@ import pytest
# Module-level markers - applied to all tests in this file
pytestmark = [
pytest.mark.flaky,
pytest.mark.integration,
pytest.mark.sample("12_workflow_hitl"),
pytest.mark.usefixtures("function_app_for_test"),
]
python/packages/core/agent_framework/observability.py
+2 -4
View File
@@ -28,7 +28,7 @@ from dotenv import load_dotenv
from opentelemetry import metrics, trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.semconv.attributes import service_attributes
from opentelemetry.semconv_ai import Meters, SpanAttributes
from opentelemetry.semconv_ai import Meters
from . import __version__ as version_info
from ._settings import load_settings
@@ -1826,9 +1826,7 @@ def _capture_response(
span.set_attributes(attributes)
attrs: dict[str, Any] = {k: v for k, v in attributes.items() if k in GEN_AI_METRIC_ATTRIBUTES}
if token_usage_histogram and (input_tokens := attributes.get(OtelAttr.INPUT_TOKENS)):
token_usage_histogram.record(
input_tokens, attributes={**attrs, OtelAttr.T_TYPE: OtelAttr.T_TYPE_INPUT}
)
token_usage_histogram.record(input_tokens, attributes={**attrs, OtelAttr.T_TYPE: OtelAttr.T_TYPE_INPUT})
if token_usage_histogram and (output_tokens := attributes.get(OtelAttr.OUTPUT_TOKENS)):
token_usage_histogram.record(output_tokens, {**attrs, OtelAttr.T_TYPE: OtelAttr.T_TYPE_OUTPUT})
if operation_duration_histogram and duration is not None:
python/packages/core/agent_framework/openai/_chat_client.py
+4 -4
View File
@@ -411,9 +411,7 @@ class RawOpenAIChatClient( # type: ignore[misc]
# See https://github.com/microsoft/agent-framework/issues/3434
if chunk.usage:
contents.append(
Content.from_usage(
usage_details=self._parse_usage_from_openai(chunk.usage), raw_representation=chunk
)
Content.from_usage(usage_details=self._parse_usage_from_openai(chunk.usage), raw_representation=chunk)
)
for choice in chunk.choices:
@@ -591,7 +589,9 @@ class RawOpenAIChatClient( # type: ignore[misc]
# See https://github.com/microsoft/agent-framework/issues/4084
for msg in all_messages:
msg_content: Any = msg.get("content")
if isinstance(msg_content, list) and all(isinstance(c, dict) and c.get("type") == "text" for c in msg_content):
if isinstance(msg_content, list) and all(
isinstance(c, dict) and c.get("type") == "text" for c in msg_content
):
msg["content"] = "\n".join(c.get("text", "") for c in msg_content)
return all_messages
python/packages/core/pyproject.toml
+1 -1
View File
@@ -125,7 +125,7 @@ executor.type = "uv"
include = "../../shared_tasks.toml"
[tool.poe.tasks]
mypy = "mypy --config-file $POE_ROOT/pyproject.toml agent_framework"
test = "pytest --cov=agent_framework --cov-report=term-missing:skip-covered tests"
test = "pytest --cov=agent_framework --cov-report=term-missing:skip-covered -n auto --dist worksteal tests"
[tool.flit.module]
name = "agent_framework"
python/packages/core/tests/azure/test_azure_assistants_client.py
+13 -5
View File
@@ -23,11 +23,8 @@ from agent_framework._settings import SecretString
from agent_framework.azure import AzureOpenAIAssistantsClient
skip_if_azure_integration_tests_disabled = pytest.mark.skipif(
os.getenv("RUN_INTEGRATION_TESTS", "false").lower() != "true"
or os.getenv("AZURE_OPENAI_ENDPOINT", "") in ("", "https://test-endpoint.com"),
reason="No real AZURE_OPENAI_ENDPOINT provided; skipping integration tests."
if os.getenv("RUN_INTEGRATION_TESTS", "false").lower() == "true"
else "Integration tests are disabled.",
os.getenv("AZURE_OPENAI_ENDPOINT", "") in ("", "https://test-endpoint.com"),
reason="No real AZURE_OPENAI_ENDPOINT provided; skipping integration tests.",
)
@@ -260,6 +257,7 @@ def get_weather(
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_azure_assistants_client_get_response() -> None:
"""Test Azure Assistants Client response."""
@@ -285,6 +283,7 @@ async def test_azure_assistants_client_get_response() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_azure_assistants_client_get_response_tools() -> None:
"""Test Azure Assistants Client response with tools."""
@@ -306,6 +305,7 @@ async def test_azure_assistants_client_get_response_tools() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_azure_assistants_client_streaming() -> None:
"""Test Azure Assistants Client streaming response."""
@@ -337,6 +337,7 @@ async def test_azure_assistants_client_streaming() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_azure_assistants_client_streaming_tools() -> None:
"""Test Azure Assistants Client streaming response with tools."""
@@ -364,6 +365,7 @@ async def test_azure_assistants_client_streaming_tools() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_azure_assistants_client_with_existing_assistant() -> None:
"""Test Azure Assistants Client with existing assistant ID."""
@@ -392,6 +394,7 @@ async def test_azure_assistants_client_with_existing_assistant() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_azure_assistants_agent_basic_run():
"""Test Agent basic run functionality with AzureOpenAIAssistantsClient."""
@@ -409,6 +412,7 @@ async def test_azure_assistants_agent_basic_run():
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_azure_assistants_agent_basic_run_streaming():
"""Test Agent basic streaming functionality with AzureOpenAIAssistantsClient."""
@@ -429,6 +433,7 @@ async def test_azure_assistants_agent_basic_run_streaming():
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_azure_assistants_agent_session_persistence():
"""Test Agent session persistence across runs with AzureOpenAIAssistantsClient."""
@@ -458,6 +463,7 @@ async def test_azure_assistants_agent_session_persistence():
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_azure_assistants_agent_existing_session_id():
"""Test Agent with existing session ID to continue conversations across agent instances."""
@@ -503,6 +509,7 @@ async def test_azure_assistants_agent_existing_session_id():
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_azure_assistants_agent_code_interpreter():
"""Test Agent with code interpreter through AzureOpenAIAssistantsClient."""
@@ -523,6 +530,7 @@ async def test_azure_assistants_agent_code_interpreter():
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_azure_assistants_client_agent_level_tool_persistence():
"""Test that agent-level tools persist across multiple runs with Azure Assistants Client."""
python/packages/core/tests/azure/test_azure_chat_client.py
+11 -5
View File
@@ -37,11 +37,8 @@ from agent_framework.openai import (
# region Service Setup
skip_if_azure_integration_tests_disabled = pytest.mark.skipif(
os.getenv("RUN_INTEGRATION_TESTS", "false").lower() != "true"
or os.getenv("AZURE_OPENAI_ENDPOINT", "") in ("", "https://test-endpoint.com"),
reason="No real AZURE_OPENAI_ENDPOINT provided; skipping integration tests."
if os.getenv("RUN_INTEGRATION_TESTS", "false").lower() == "true"
else "Integration tests are disabled.",
os.getenv("AZURE_OPENAI_ENDPOINT", "") in ("", "https://test-endpoint.com"),
reason="No real AZURE_OPENAI_ENDPOINT provided; skipping integration tests.",
)
@@ -647,6 +644,7 @@ def get_weather(location: str) -> str:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_azure_openai_chat_client_response() -> None:
"""Test Azure OpenAI chat completion responses."""
@@ -677,6 +675,7 @@ async def test_azure_openai_chat_client_response() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_azure_openai_chat_client_response_tools() -> None:
"""Test AzureOpenAI chat completion responses."""
@@ -698,6 +697,7 @@ async def test_azure_openai_chat_client_response_tools() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_azure_openai_chat_client_streaming() -> None:
"""Test Azure OpenAI chat completion responses."""
@@ -733,6 +733,7 @@ async def test_azure_openai_chat_client_streaming() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_azure_openai_chat_client_streaming_tools() -> None:
"""Test AzureOpenAI chat completion responses."""
@@ -760,6 +761,7 @@ async def test_azure_openai_chat_client_streaming_tools() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_azure_openai_chat_client_agent_basic_run():
"""Test Azure OpenAI chat client agent basic run functionality with AzureOpenAIChatClient."""
@@ -776,6 +778,7 @@ async def test_azure_openai_chat_client_agent_basic_run():
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_azure_openai_chat_client_agent_basic_run_streaming():
"""Test Azure OpenAI chat client agent basic streaming functionality with AzureOpenAIChatClient."""
@@ -794,6 +797,7 @@ async def test_azure_openai_chat_client_agent_basic_run_streaming():
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_azure_openai_chat_client_agent_session_persistence():
"""Test Azure OpenAI chat client agent session persistence across runs with AzureOpenAIChatClient."""
@@ -819,6 +823,7 @@ async def test_azure_openai_chat_client_agent_session_persistence():
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_azure_openai_chat_client_agent_existing_session():
"""Test Azure OpenAI chat client agent with existing session to continue conversations across agent instances."""
@@ -854,6 +859,7 @@ async def test_azure_openai_chat_client_agent_existing_session():
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_azure_chat_client_agent_level_tool_persistence():
"""Test that agent-level tools persist across multiple runs with Azure Chat Client."""
python/packages/core/tests/azure/test_azure_responses_client.py
+9 -5
View File
@@ -23,11 +23,8 @@ from agent_framework import (
from agent_framework.azure import AzureOpenAIResponsesClient
skip_if_azure_integration_tests_disabled = pytest.mark.skipif(
os.getenv("RUN_INTEGRATION_TESTS", "false").lower() != "true"
or os.getenv("AZURE_OPENAI_ENDPOINT", "") in ("", "https://test-endpoint.com"),
reason="No real AZURE_OPENAI_ENDPOINT provided; skipping integration tests."
if os.getenv("RUN_INTEGRATION_TESTS", "false").lower() == "true"
else "Integration tests are disabled.",
os.getenv("AZURE_OPENAI_ENDPOINT", "") in ("", "https://test-endpoint.com"),
reason="No real AZURE_OPENAI_ENDPOINT provided; skipping integration tests.",
)
logger = logging.getLogger(__name__)
@@ -254,6 +251,7 @@ def test_serialize(azure_openai_unit_test_env: dict[str, str]) -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
@pytest.mark.parametrize(
"option_name,option_value,needs_validation",
@@ -392,6 +390,7 @@ async def test_integration_options(
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_integration_web_search() -> None:
client = AzureOpenAIResponsesClient(credential=AzureCliCredential())
@@ -440,6 +439,7 @@ async def test_integration_web_search() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_integration_client_file_search() -> None:
"""Test Azure responses client with file search tool."""
@@ -469,6 +469,7 @@ async def test_integration_client_file_search() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_integration_client_file_search_streaming() -> None:
"""Test Azure responses client with file search tool and streaming."""
@@ -500,6 +501,7 @@ async def test_integration_client_file_search_streaming() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_integration_client_agent_hosted_mcp_tool() -> None:
"""Integration test for MCP tool with Azure Response Agent using Microsoft Learn MCP."""
@@ -524,6 +526,7 @@ async def test_integration_client_agent_hosted_mcp_tool() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_integration_client_agent_hosted_code_interpreter_tool():
"""Test Azure Responses Client agent with code interpreter tool."""
@@ -543,6 +546,7 @@ async def test_integration_client_agent_hosted_code_interpreter_tool():
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_integration_client_agent_existing_session():
"""Test Azure Responses Client agent with existing session to continue conversations across agent instances."""
python/packages/core/tests/core/test_mcp.py
+4 -6
View File
@@ -34,12 +34,8 @@ from agent_framework.exceptions import ToolException, ToolExecutionException
# Integration test skip condition
skip_if_mcp_integration_tests_disabled = pytest.mark.skipif(
os.getenv("RUN_INTEGRATION_TESTS", "false").lower() != "true" or os.getenv("LOCAL_MCP_URL", "") == "",
reason=(
"No LOCAL_MCP_URL provided; skipping integration tests."
if os.getenv("RUN_INTEGRATION_TESTS", "false").lower() == "true"
else "Integration tests are disabled."
),
os.getenv("LOCAL_MCP_URL", "") == "",
reason="No LOCAL_MCP_URL provided; skipping integration tests.",
)
@@ -1105,6 +1101,7 @@ def test_local_mcp_streamable_http_tool_init():
# Integration test
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_mcp_integration_tests_disabled
async def test_streamable_http_integration():
"""Test MCP StreamableHTTP integration."""
@@ -1133,6 +1130,7 @@ async def test_streamable_http_integration():
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_mcp_integration_tests_disabled
async def test_mcp_connection_reset_integration():
"""Test that connection reset works correctly with a real MCP server.
python/packages/core/tests/openai/test_assistant_provider.py
+4 -5
View File
@@ -755,14 +755,13 @@ class TestToolMerging:
# region Integration Tests
skip_if_openai_integration_tests_disabled = pytest.mark.skipif(
os.getenv("RUN_INTEGRATION_TESTS", "false").lower() != "true"
or os.getenv("OPENAI_API_KEY", "") in ("", "test-dummy-key"),
reason="No real OPENAI_API_KEY provided; skipping integration tests."
if os.getenv("RUN_INTEGRATION_TESTS", "false").lower() == "true"
else "Integration tests are disabled.",
os.getenv("OPENAI_API_KEY", "") in ("", "test-dummy-key"),
reason="No real OPENAI_API_KEY provided; skipping integration tests.",
)
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_openai_integration_tests_disabled
class TestOpenAIAssistantProviderIntegration:
"""Integration tests requiring real OpenAI API."""
python/packages/core/tests/openai/test_openai_assistants_client.py
+15 -5
View File
@@ -25,11 +25,8 @@ from agent_framework import (
from agent_framework.openai import OpenAIAssistantsClient
skip_if_openai_integration_tests_disabled = pytest.mark.skipif(
os.getenv("RUN_INTEGRATION_TESTS", "false").lower() != "true"
or os.getenv("OPENAI_API_KEY", "") in ("", "test-dummy-key"),
reason="No real OPENAI_API_KEY provided; skipping integration tests."
if os.getenv("RUN_INTEGRATION_TESTS", "false").lower() == "true"
else "Integration tests are disabled.",
os.getenv("OPENAI_API_KEY", "") in ("", "test-dummy-key"),
reason="No real OPENAI_API_KEY provided; skipping integration tests.",
)
INTEGRATION_TEST_MODEL = "gpt-4.1-nano"
@@ -1075,6 +1072,7 @@ def get_weather(
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_openai_integration_tests_disabled
async def test_get_response() -> None:
"""Test OpenAI Assistants Client response."""
@@ -1100,6 +1098,7 @@ async def test_get_response() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_openai_integration_tests_disabled
async def test_get_response_tools() -> None:
"""Test OpenAI Assistants Client response with tools."""
@@ -1121,6 +1120,7 @@ async def test_get_response_tools() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_openai_integration_tests_disabled
async def test_streaming() -> None:
"""Test OpenAI Assistants Client streaming response."""
@@ -1152,6 +1152,7 @@ async def test_streaming() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_openai_integration_tests_disabled
async def test_streaming_tools() -> None:
"""Test OpenAI Assistants Client streaming response with tools."""
@@ -1182,6 +1183,7 @@ async def test_streaming_tools() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_openai_integration_tests_disabled
async def test_with_existing_assistant() -> None:
"""Test OpenAI Assistants Client with existing assistant ID."""
@@ -1210,6 +1212,7 @@ async def test_with_existing_assistant() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_openai_integration_tests_disabled
@pytest.mark.skip(reason="OpenAI file search functionality is currently broken - tracked in GitHub issue")
async def test_file_search() -> None:
@@ -1236,6 +1239,7 @@ async def test_file_search() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_openai_integration_tests_disabled
@pytest.mark.skip(reason="OpenAI file search functionality is currently broken - tracked in GitHub issue")
async def test_file_search_streaming() -> None:
@@ -1270,6 +1274,7 @@ async def test_file_search_streaming() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_openai_integration_tests_disabled
async def test_openai_assistants_agent_basic_run():
"""Test Agent basic run functionality with OpenAIAssistantsClient."""
@@ -1287,6 +1292,7 @@ async def test_openai_assistants_agent_basic_run():
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_openai_integration_tests_disabled
async def test_openai_assistants_agent_basic_run_streaming():
"""Test Agent basic streaming functionality with OpenAIAssistantsClient."""
@@ -1307,6 +1313,7 @@ async def test_openai_assistants_agent_basic_run_streaming():
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_openai_integration_tests_disabled
async def test_openai_assistants_agent_session_persistence():
"""Test Agent session persistence across runs with OpenAIAssistantsClient."""
@@ -1336,6 +1343,7 @@ async def test_openai_assistants_agent_session_persistence():
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_openai_integration_tests_disabled
async def test_openai_assistants_agent_existing_session_id():
"""Test Agent with existing session ID to continue conversations across agent instances."""
@@ -1381,6 +1389,7 @@ async def test_openai_assistants_agent_existing_session_id():
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_openai_integration_tests_disabled
async def test_openai_assistants_agent_code_interpreter():
"""Test Agent with code interpreter through OpenAIAssistantsClient."""
@@ -1401,6 +1410,7 @@ async def test_openai_assistants_agent_code_interpreter():
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_openai_integration_tests_disabled
async def test_agent_level_tool_persistence():
"""Test that agent-level tools persist across multiple runs with OpenAI Assistants Client."""
python/packages/core/tests/openai/test_openai_chat_client.py
+4 -5
View File
@@ -24,11 +24,8 @@ from agent_framework.openai import OpenAIChatClient
from agent_framework.openai._exceptions import OpenAIContentFilterException
skip_if_openai_integration_tests_disabled = pytest.mark.skipif(
os.getenv("RUN_INTEGRATION_TESTS", "false").lower() != "true"
or os.getenv("OPENAI_API_KEY", "") in ("", "test-dummy-key"),
reason="No real OPENAI_API_KEY provided; skipping integration tests."
if os.getenv("RUN_INTEGRATION_TESTS", "false").lower() == "true"
else "Integration tests are disabled.",
os.getenv("OPENAI_API_KEY", "") in ("", "test-dummy-key"),
reason="No real OPENAI_API_KEY provided; skipping integration tests.",
)
@@ -1087,6 +1084,7 @@ class OutputStruct(BaseModel):
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_openai_integration_tests_disabled
@pytest.mark.parametrize(
"option_name,option_value,needs_validation",
@@ -1225,6 +1223,7 @@ async def test_integration_options(
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_openai_integration_tests_disabled
async def test_integration_web_search() -> None:
client = OpenAIChatClient(model_id="gpt-4o-search-preview")
python/packages/core/tests/openai/test_openai_embedding_client.py
+4 -10
View File
@@ -259,20 +259,14 @@ def test_azure_otel_provider_name() -> None:
# --- Integration tests ---
skip_if_openai_integration_tests_disabled = pytest.mark.skipif(
os.getenv("RUN_INTEGRATION_TESTS", "false").lower() != "true"
or os.getenv("OPENAI_API_KEY", "") in ("", "test-dummy-key"),
reason="No real OPENAI_API_KEY provided; skipping integration tests."
if os.getenv("RUN_INTEGRATION_TESTS", "false").lower() == "true"
else "Integration tests are disabled.",
os.getenv("OPENAI_API_KEY", "") in ("", "test-dummy-key"),
reason="No real OPENAI_API_KEY provided; skipping integration tests.",
)
skip_if_azure_openai_integration_tests_disabled = pytest.mark.skipif(
os.getenv("RUN_INTEGRATION_TESTS", "false").lower() != "true"
or not os.getenv("AZURE_OPENAI_ENDPOINT")
not os.getenv("AZURE_OPENAI_ENDPOINT")
or (not os.getenv("AZURE_OPENAI_API_KEY") and not os.getenv("AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME")),
reason="No Azure OpenAI credentials provided; skipping integration tests."
if os.getenv("RUN_INTEGRATION_TESTS", "false").lower() == "true"
else "Integration tests are disabled.",
reason="No Azure OpenAI credentials provided; skipping integration tests.",
)
python/packages/core/tests/openai/test_openai_responses_client.py
+6 -5
View File
@@ -40,11 +40,8 @@ from agent_framework.openai import OpenAIResponsesClient
from agent_framework.openai._exceptions import OpenAIContentFilterException
skip_if_openai_integration_tests_disabled = pytest.mark.skipif(
os.getenv("RUN_INTEGRATION_TESTS", "false").lower() != "true"
or os.getenv("OPENAI_API_KEY", "") in ("", "test-dummy-key"),
reason="No real OPENAI_API_KEY provided; skipping integration tests."
if os.getenv("RUN_INTEGRATION_TESTS", "false").lower() == "true"
else "Integration tests are disabled.",
os.getenv("OPENAI_API_KEY", "") in ("", "test-dummy-key"),
reason="No real OPENAI_API_KEY provided; skipping integration tests.",
)
@@ -2362,6 +2359,7 @@ def test_with_callable_api_key() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_openai_integration_tests_disabled
@pytest.mark.parametrize(
"option_name,option_value,needs_validation",
@@ -2500,6 +2498,7 @@ async def test_integration_options(
@pytest.mark.timeout(300)
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_openai_integration_tests_disabled
async def test_integration_web_search() -> None:
client = OpenAIResponsesClient(model_id="gpt-5")
@@ -2553,6 +2552,7 @@ async def test_integration_web_search() -> None:
"race condition. See https://github.com/microsoft/agent-framework/issues/1669"
)
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_openai_integration_tests_disabled
async def test_integration_file_search() -> None:
openai_responses_client = OpenAIResponsesClient()
@@ -2586,6 +2586,7 @@ async def test_integration_file_search() -> None:
"potential race condition. See https://github.com/microsoft/agent-framework/issues/1669"
)
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_openai_integration_tests_disabled
async def test_integration_streaming_file_search() -> None:
openai_responses_client = OpenAIResponsesClient()
python/packages/durabletask/tests/integration_tests/.env.example
-4
View File
@@ -11,7 +11,3 @@ TASKHUB=default
# Redis Configuration (for streaming tests)
REDIS_CONNECTION_STRING=redis://localhost:6379
REDIS_STREAM_TTL_MINUTES=10
# Integration Test Control
# Set to 'true' to enable integration tests
RUN_INTEGRATION_TESTS=true
python/packages/durabletask/tests/integration_tests/README.md
+1 -2
View File
@@ -16,7 +16,6 @@ Required variables:
- `AZURE_OPENAI_ENDPOINT`
- `AZURE_OPENAI_CHAT_DEPLOYMENT_NAME`
- `AZURE_OPENAI_API_KEY` (optional if using Azure CLI authentication)
- `RUN_INTEGRATION_TESTS` (set to `true`)
- `ENDPOINT` (default: http://localhost:8080)
- `TASKHUB` (default: default)
@@ -75,7 +74,7 @@ pytestmark = [
## Troubleshooting
**Tests are skipped:**
Ensure `RUN_INTEGRATION_TESTS=true` is set in your `.env` file.
Ensure the required environment variables (e.g., `AZURE_OPENAI_ENDPOINT`) are set in your `.env` file.
**DTS connection failed:**
Check that the DTS emulator container is running: `docker ps | grep dts-emulator`
python/packages/durabletask/tests/integration_tests/conftest.py
-5
View File
@@ -289,9 +289,6 @@ def pytest_configure(config: pytest.Config) -> None:
def pytest_collection_modifyitems(config: pytest.Config, items: list[pytest.Item]) -> None:
"""Skip tests based on markers and environment availability."""
run_integration = os.getenv("RUN_INTEGRATION_TESTS", "false").lower() == "true"
skip_integration = pytest.mark.skip(reason="RUN_INTEGRATION_TESTS not set to 'true'")
# Check Azure OpenAI environment variables
azure_openai_vars = ["AZURE_OPENAI_ENDPOINT", "AZURE_OPENAI_CHAT_DEPLOYMENT_NAME"]
azure_openai_available = all(os.getenv(var) for var in azure_openai_vars)
@@ -308,8 +305,6 @@ def pytest_collection_modifyitems(config: pytest.Config, items: list[pytest.Item
skip_redis = pytest.mark.skip(reason="Redis is not available at redis://localhost:6379")
for item in items:
if "integration_test" in item.keywords and not run_integration:
item.add_marker(skip_integration)
if "requires_azure_openai" in item.keywords and not azure_openai_available:
item.add_marker(skip_azure_openai)
if "requires_dts" in item.keywords and not dts_available:
python/packages/durabletask/tests/integration_tests/test_01_dt_single_agent.py
+2
View File
@@ -14,6 +14,8 @@ import pytest
# Module-level markers - applied to all tests in this module
pytestmark = [
pytest.mark.flaky,
pytest.mark.integration,
pytest.mark.sample("01_single_agent"),
pytest.mark.integration_test,
pytest.mark.requires_azure_openai,
python/packages/durabletask/tests/integration_tests/test_02_dt_multi_agent.py
+2
View File
@@ -18,6 +18,8 @@ MATH_AGENT_NAME: str = "MathAgent"
# Module-level markers - applied to all tests in this module
pytestmark = [
pytest.mark.flaky,
pytest.mark.integration,
pytest.mark.sample("02_multi_agent"),
pytest.mark.integration_test,
pytest.mark.requires_azure_openai,
python/packages/durabletask/tests/integration_tests/test_03_dt_single_agent_streaming.py
+2
View File
@@ -34,6 +34,8 @@ from redis_stream_response_handler import RedisStreamResponseHandler # type: ig
# Module-level markers - applied to all tests in this file
pytestmark = [
pytest.mark.flaky,
pytest.mark.integration,
pytest.mark.sample("03_single_agent_streaming"),
pytest.mark.integration_test,
pytest.mark.requires_azure_openai,
python/packages/durabletask/tests/integration_tests/test_04_dt_single_agent_orchestration_chaining.py
+2
View File
@@ -23,6 +23,8 @@ logging.basicConfig(level=logging.WARNING)
# Module-level markers - applied to all tests in this module
pytestmark = [
pytest.mark.flaky,
pytest.mark.integration,
pytest.mark.sample("04_single_agent_orchestration_chaining"),
pytest.mark.integration_test,
pytest.mark.requires_azure_openai,
python/packages/durabletask/tests/integration_tests/test_05_dt_multi_agent_orchestration_concurrency.py
+2
View File
@@ -24,6 +24,8 @@ logging.basicConfig(level=logging.WARNING)
# Module-level markers
pytestmark = [
pytest.mark.flaky,
pytest.mark.integration,
pytest.mark.sample("05_multi_agent_orchestration_concurrency"),
pytest.mark.integration_test,
pytest.mark.requires_dts,
python/packages/durabletask/tests/integration_tests/test_06_dt_multi_agent_orchestration_conditionals.py
+2
View File
@@ -24,6 +24,8 @@ logging.basicConfig(level=logging.WARNING)
# Module-level markers
pytestmark = [
pytest.mark.flaky,
pytest.mark.integration,
pytest.mark.sample("06_multi_agent_orchestration_conditionals"),
pytest.mark.integration_test,
pytest.mark.requires_dts,
python/packages/durabletask/tests/integration_tests/test_07_dt_single_agent_orchestration_hitl.py
+2
View File
@@ -24,6 +24,8 @@ logging.basicConfig(level=logging.WARNING)
# Module-level markers
pytestmark = [
pytest.mark.flaky,
pytest.mark.integration,
pytest.mark.sample("07_single_agent_orchestration_hitl"),
pytest.mark.integration_test,
pytest.mark.requires_dts,
python/packages/ollama/tests/test_ollama_chat_client.py
+10 -5
View File
@@ -26,11 +26,8 @@ from agent_framework_ollama import OllamaChatClient
# region Service Setup
skip_if_azure_integration_tests_disabled = pytest.mark.skipif(
os.getenv("RUN_INTEGRATION_TESTS", "false").lower() != "true"
or os.getenv("OLLAMA_MODEL_ID", "") in ("", "test-model"),
reason="No real Ollama chat model provided; skipping integration tests."
if os.getenv("RUN_INTEGRATION_TESTS", "false").lower() == "true"
else "Integration tests are disabled.",
os.getenv("OLLAMA_MODEL_ID", "") in ("", "test-model"),
reason="No real Ollama chat model provided; skipping integration tests.",
)
@@ -470,6 +467,8 @@ async def test_cmc_with_invalid_content_type(
await ollama_client.get_response(messages=chat_history)
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_cmc_integration_with_tool_call(
chat_history: list[Message],
@@ -485,6 +484,8 @@ async def test_cmc_integration_with_tool_call(
assert tool_result.result == "Hello World"
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_cmc_integration_with_chat_completion(
chat_history: list[Message],
@@ -497,6 +498,8 @@ async def test_cmc_integration_with_chat_completion(
assert "hello" in result.text.lower()
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_cmc_streaming_integration_with_tool_call(
chat_history: list[Message],
@@ -522,6 +525,8 @@ async def test_cmc_streaming_integration_with_tool_call(
assert tool_call.name == "hello_world"
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_azure_integration_tests_disabled
async def test_cmc_streaming_integration_with_chat_completion(
chat_history: list[Message],
python/packages/orchestrations/pyproject.toml
+1 -1
View File
@@ -80,7 +80,7 @@ executor.type = "uv"
include = "../../shared_tasks.toml"
[tool.poe.tasks]
mypy = "mypy --config-file $POE_ROOT/pyproject.toml agent_framework_orchestrations"
test = "pytest --cov=agent_framework_orchestrations --cov-report=term-missing:skip-covered tests"
test = "pytest --cov=agent_framework_orchestrations --cov-report=term-missing:skip-covered -n auto --dist worksteal tests"
[build-system]
requires = ["flit-core >= 3.11,<4.0"]