Files
agent-framework/python
T
Giles Odigwe c06af9a1b3 .NET: Python: Add dotnet integration test report to CI (#5515)
* Add dotnet integration test report to CI

- Add --report-junit flag to dotnet integration test step to generate
  JUnit XML alongside TRX, with explicit --results-directory to
  centralize output in IntegrationTestResults/
- Upload JUnit XML artifacts from each matrix leg (net10.0/ubuntu,
  net472/windows) as dotnet-test-results-{framework}-{os}
- Add dotnet-integration-test-report job that downloads artifacts,
  runs the existing aggregate.py script, posts markdown to Job Summary,
  and saves trend history via actions/cache
- Refactor aggregate.py to discover JUnit XML files recursively,
  supporting both pytest (pytest.xml) and xunit (*.junit.xml) layouts
- Handle provider name derivation for dotnet artifact naming convention
- Fix nodeid collision when same test runs under multiple frameworks
  by qualifying keys with provider when collisions are detected
- Improve module extraction for dotnet C# classnames (recognizes
  IntegrationTests/UnitTests namespace segments)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chore: trigger dotnet CI for report validation

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: use .junit extension (not .junit.xml) for xunit v3 output

xUnit v3 generates files with .junit extension, not .junit.xml.
Update upload glob and aggregate.py discovery to match.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: use deterministic provider-qualified keys for dotnet tests

Always prefix dotnet test keys with provider (e.g. net10.0 (ubuntu)::TestName)
to ensure stable, comparable counts across runs regardless of file parse order.
Also show Executed (passed+failed) instead of Total in summary table.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: match Python report summary format (Total, passed/total, etc.)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat: split dotnet report into per-framework tables

Dotnet tests run on multiple frameworks (net10.0, net472). Instead of
one combined table with unstable totals, show separate sections per
framework — each with its own summary row and per-test table. Python
reports retain the original single-table format.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Re-enable 7 flaky dotnet integration tests with increased timeouts

Increase timeouts to reduce timing-related flakiness in LLM-backed
integration tests (issue #4971):

- ExternalClientTests: 60s -> 120s default timeout
- SamplesValidationBase: 60s -> 120s default timeout
- ConsoleAppSamplesValidation: 90s -> 150s for long-running tests
- AzureFunctions SamplesValidation: 2min -> 3min orchestration timeout,
  60s -> 90s per-step WaitForConditionAsync timeouts

Remove all Skip=Flaky annotations and unused SkipFlakyTimingTest constants.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Re-skip LLM non-determinism flaky tests, keep timeout fixes

Re-skip SingleAgentOrchestrationHITLSampleValidationAsync and
LongRunningToolsSampleValidationAsync - these fail due to LLM producing
extra review notifications, not timeouts. Updated skip reasons to
accurately describe the root cause. Reverted unnecessary timeout change
on the skipped LongRunningTools test.

The remaining 5 re-enabled tests with timeout increases are stable.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Enable Anthropic integration tests in CI

Replace hardcoded skip with conditional skip pattern (matching
CopilotStudio approach): tests gracefully skip when ANTHROPIC_API_KEY
is missing, and run when present.

Changes:
- AnthropicChatCompletionFixture: try/catch in InitializeAsync with
  Assert.Skip on missing config (replaces hardcoded SkipReason)
- AnthropicSkillsIntegrationTests: same pattern per test method
- dotnet-build-and-test.yml: wire up ANTHROPIC_API_KEY,
  ANTHROPIC_CHAT_MODEL_NAME, and ANTHROPIC_REASONING_MODEL_NAME
  env vars to the integration test step

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix missing System using in AnthropicSkillsIntegrationTests

Add 'using System;' for InvalidOperationException in try/catch blocks.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Skip flaky SingleAgentOrchestrationChainingSampleValidationAsync

LLM non-determinism causes Assert.NotNull failures on orchestration
results. Skip until test logic is hardened against non-deterministic
LLM responses.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Re-enable HITL and LongRunningTools tests with timeout and flexibility fixes

- Remove Skip attribute from SingleAgentOrchestrationHITLSampleValidationAsync
- Remove Skip attribute from LongRunningToolsSampleValidationAsync
- Increase timeout from 120s/90s to 180s to accommodate 2+ LLM round-trips
- Replace rigid 2-cycle assertion with flexible approval logic that handles
  extra review cycles from LLM non-determinism

Fixes the two failure modes identified in #4971:
1. Timeout: 120s/90s was insufficient for multiple LLM calls under CI load
2. Extra notifications: Assert.Fail on 3rd+ review cycle was too rigid

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Increase AzureFunctions LongRunningTools test timeouts from 90s to 180s

The LongRunningToolsSampleValidationAsync test in the AzureFunctions integration
tests was failing in CI with TimeoutException at the 'Content published
notification is logged' step. The 90-second timeouts are too tight for CI
environments where LLM calls and orchestration overhead can be slow.

Increased all three WaitForConditionAsync timeouts from 90s to 180s:
- Waiting for human feedback notification
- Waiting for publish notification (the step that was failing)
- Waiting for orchestration completion

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Merge main and fix dotnet report path after flaky_report rename

Merge upstream/main which renamed scripts/flaky_report/ to
scripts/integration_test_report/ (from Python PR #5454). Update the
dotnet-build-and-test workflow to reference the new path.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add RetryFact to DurableTask and AzureFunctions integration tests

These tests interact with LLMs via stdin/stdout (DurableTask) or HTTP
(AzureFunctions) and are inherently non-deterministic. Unlike the Python
side which uses pytest-retry, the dotnet tests had no retry mechanism
and a single transient failure would fail the entire CI run.

Changes:
- Switch [Fact] to [RetryFact(2, 5000)] on all LLM-dependent tests
  across ConsoleAppSamplesValidation, ExternalClientTests,
  WorkflowConsoleAppSamplesValidation, and AzureFunctions SamplesValidation
- Add re-prompt mechanism to LongRunningToolsSampleValidationAsync:
  if the LLM doesn't invoke the tool within 60s, re-send the prompt
  (up to 2 retries) instead of burning the full timeout
- Reduce LongRunningTools timeout from 240s to 180s (re-prompt makes
  the extra buffer unnecessary)
- Leave simple/deterministic tests as [Fact] (SingleAgent, unit tests)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add persist-credentials: false to Integration Test Report checkout step

Matches the convention used by other checkout steps in this workflow
to avoid leaving GITHUB_TOKEN credentials in the local git config.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* small fixes

* disable anthropic failing tests

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
c06af9a1b3 · 2026-05-07 20:39:32 +00:00
History
..
2025-10-01 11:54:26 +00:00

Get Started with Microsoft Agent Framework for Python Developers

Quick Install

We recommend two common installation paths depending on your use case.

1. Development mode

If you are exploring or developing locally, install the entire framework with all sub-packages:

pip install agent-framework

This installs the core and every integration package, making sure that all features are available without additional steps. This is the simplest way to get started.

2. Selective install

If you only need specific integrations, you can install at a more granular level. This keeps dependencies lighter and focuses on what you actually plan to use. Some examples:

# Core only
# includes Azure OpenAI and OpenAI support by default
# also includes workflows and orchestrations
pip install agent-framework-core

# Core + Azure AI Foundry integration
pip install agent-framework-foundry

# Core + Microsoft Copilot Studio integration (preview package)
pip install agent-framework-copilotstudio --pre

# Core + both Microsoft Copilot Studio and Azure AI Foundry integration
pip install --pre agent-framework-copilotstudio agent-framework-foundry

This selective approach is useful when you know which integrations you need, and it is the recommended way to set up lightweight environments. Released packages such as agent-framework, agent-framework-core, and agent-framework-foundry no longer require --pre, while preview connectors such as agent-framework-copilotstudio still do.

Supported Platforms:

  • Python: 3.10+
  • OS: Windows, macOS, Linux

1. Setup API Keys

Set as environment variables, or create a .env file at your project root:

OPENAI_API_KEY=sk-...
OPENAI_MODEL=...
...
AZURE_OPENAI_API_KEY=...
AZURE_OPENAI_ENDPOINT=...
AZURE_OPENAI_MODEL=...
...
FOUNDRY_PROJECT_ENDPOINT=...
FOUNDRY_MODEL=...

For the generic OpenAI clients (OpenAIChatClient and OpenAIChatCompletionClient), configuration resolves in this order:

  1. Explicit Azure inputs such as credential or azure_endpoint
  2. OPENAI_API_KEY / explicit OpenAI API-key parameters
  3. Azure environment fallback such as AZURE_OPENAI_ENDPOINT and AZURE_OPENAI_API_KEY

This means mixed shells default to OpenAI when OPENAI_API_KEY is present. To force Azure routing, pass an explicit Azure input such as credential=AzureCliCredential().

You can also override environment variables by explicitly passing configuration parameters to the chat client constructor:

from agent_framework.openai import OpenAIChatClient

client = OpenAIChatClient(
    api_key='',
    azure_endpoint='',
    model='',
    api_version='',
)

See the following setup guide for more information.

2. Create a Simple Agent

Create agents and invoke them directly:

import asyncio
from agent_framework import Agent
from agent_framework.openai import OpenAIChatClient

async def main():
    agent = Agent(
        client=OpenAIChatClient(),
        instructions="""
        1) A robot may not injure a human being...
        2) A robot must obey orders given it by human beings...
        3) A robot must protect its own existence...

        Give me the TLDR in exactly 5 words.
        """
    )

    result = await agent.run("Summarize the Three Laws of Robotics")
    print(result)

asyncio.run(main())
# Output: Protect humans, obey, self-preserve, prioritized.

3. Directly Use Chat Clients (No Agent Required)

You can use the chat client classes directly for advanced workflows:

import asyncio
from agent_framework import Message
from agent_framework.openai import OpenAIChatClient

async def main():
    client = OpenAIChatClient()

    messages = [
        Message("system", ["You are a helpful assistant."]),
        Message("user", ["Write a haiku about Agent Framework."])
    ]

    response = await client.get_response(messages)
    print(response.messages[0].text)

    """
    Output:

    Agents work in sync,
    Framework threads through each task—
    Code sparks collaboration.
    """

asyncio.run(main())

4. Build an Agent with Tools and Functions

Enhance your agent with custom tools and function calling:

import asyncio
from typing import Annotated
from random import randint
from pydantic import Field
from agent_framework import Agent
from agent_framework.openai import OpenAIChatClient


def get_weather(
    location: Annotated[str, Field(description="The location to get the weather for.")],
) -> str:
    """Get the weather for a given location."""
    conditions = ["sunny", "cloudy", "rainy", "stormy"]
    return f"The weather in {location} is {conditions[randint(0, 3)]} with a high of {randint(10, 30)}°C."


def get_menu_specials() -> str:
    """Get today's menu specials."""
    return """
    Special Soup: Clam Chowder
    Special Salad: Cobb Salad
    Special Drink: Chai Tea
    """


async def main():
    agent = Agent(
        client=OpenAIChatClient(),
        instructions="You are a helpful assistant that can provide weather and restaurant information.",
        tools=[get_weather, get_menu_specials]
    )

    response = await agent.run("What's the weather in Amsterdam and what are today's specials?")
    print(response)

    """
    Output:
    The weather in Amsterdam is sunny with a high of 22°C. Today's specials include
    Clam Chowder soup, Cobb Salad, and Chai Tea as the special drink.
    """

if __name__ == "__main__":
    asyncio.run(main())

You can explore additional agent samples here.

5. Multi-Agent Orchestration

Coordinate multiple agents to collaborate on complex tasks using orchestration patterns:

import asyncio
from agent_framework import Agent
from agent_framework.openai import OpenAIChatClient


async def main():
    # Create specialized agents
    writer = Agent(
        client=OpenAIChatClient(),
        name="Writer",
        instructions="You are a creative content writer. Generate and refine slogans based on feedback."
    )

    reviewer = Agent(
        client=OpenAIChatClient(),
        name="Reviewer",
        instructions="You are a critical reviewer. Provide detailed feedback on proposed slogans."
    )

    # Sequential workflow: Writer creates, Reviewer provides feedback
    task = "Create a slogan for a new electric SUV that is affordable and fun to drive."

    # Step 1: Writer creates initial slogan
    initial_result = await writer.run(task)
    print(f"Writer: {initial_result}")

    # Step 2: Reviewer provides feedback
    feedback_request = f"Please review this slogan: {initial_result}"
    feedback = await reviewer.run(feedback_request)
    print(f"Reviewer: {feedback}")

    # Step 3: Writer refines based on feedback
    refinement_request = f"Please refine this slogan based on the feedback: {initial_result}\nFeedback: {feedback}"
    final_result = await writer.run(refinement_request)
    print(f"Final Slogan: {final_result}")

    # Example Output:
    # Writer: "Charge Forward: Affordable Adventure Awaits!"
    # Reviewer: "Good energy, but 'Charge Forward' is overused in EV marketing..."
    # Final Slogan: "Power Up Your Adventure: Premium Feel, Smart Price!"

if __name__ == "__main__":
    asyncio.run(main())

For more advanced orchestration patterns including Sequential, Concurrent, Group Chat, Handoff, and Magentic orchestrations, see the orchestration samples.

More Examples & Samples

Agent Framework Documentation