Compare commits

..
Author SHA1 Message Date
Giles OdigweandCopilot 7cc5ff771b Remove duplicate WebhookKey properties from merge
Both our branch and main added WebhookKey to the Anthropic test
mock classes, resulting in CS0102 duplicate definition errors.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-12 10:02:56 -07:00
Giles Odigwe 0269529ccf Merge remote-tracking branch 'upstream/main' into dotnet-split-integration-tests
# Conflicts:
#	dotnet/tests/AnthropicChatCompletion.IntegrationTests/AnthropicSkillsIntegrationTests.cs
2026-05-12 09:54:58 -07:00
Giles OdigweandCopilot e514fc8837 Remove unnecessary RT0003 warning suppression
The RT0003 suppression was added during the Anthropic SDK 12.20.0
upgrade but the warning no longer fires. Removing it to keep the
NoWarn list minimal.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-12 08:16:42 -07:00
Giles OdigweandCopilot 049e823177 Rename filter parameters for consistency
TestProjectNameFilter  -> TestProjectNameIncludeFilter
TestProjectNameExclude -> TestProjectNameExcludeFilter

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-12 07:22:36 -07:00
Giles OdigweandCopilot c4f9d0d4cf Include workflow file in functions/core path filters
A PR editing only dotnet-build-and-test.yml would skip
dotnet-test-functions because the workflow path was missing
from both the functions and core path filter lists.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-11 08:10:58 -07:00
Giles Odigwe 1929f73959 Merge branch 'main' into dotnet-split-integration-tests 2026-05-11 06:34:47 -07:00
Giles OdigweandCopilot 1b0fbb808e Re-enable Foundry OpenAPI server-side tool integration test
Remove Skip="For manual testing only" from
AsAIAgent_WithOpenAPITool_NativeSDKCreation_InvokesServerSideToolAsync.
The test already uses RetryFact(3 retries, 5s delay) to handle
transient failures from the external restcountries.com API.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-10 16:33:37 -07:00
Giles OdigweandCopilot c799c61ff1 Fix CheckSystem test case to expect 1 response
The CheckSystem workflow sends a 'PASSED!' SendActivity when all system
variables are populated, producing 1 AgentResponseEvent. The test case
had min_response_count: 0 with no max, so the assertion defaulted max
to 0 and failed with 'Response count greater than expected: 0 (Actual: 1)'.
Updated to expect exactly 1 response, matching the SendActivity pattern.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-10 13:32:10 -07:00
Giles OdigweandCopilot 158ecb7b40 Re-enable CheckSystem declarative integration tests
The CheckSystem.yaml tests were temporarily skipped in PR #4270 during
the Azure.AI.Projects 2.0.0-beta.1 SDK update. Since then, the system
variable plumbing (SystemScope, SetLastMessageAsync, conversation
initialization) has been significantly updated and stabilized. The
other tests in these same files pass reliably using the same
infrastructure.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-10 12:20:25 -07:00
Giles OdigweandCopilot 3c8fdb6f49 Fix Anthropic unit test mocks for SDK 12.20.0 interface changes
Add missing interface members: IAnthropicClient.WebhookKey,
IBetaService.MemoryStores, IBetaService.Webhooks, IBetaService.UserProfiles

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-08 14:47:49 -07:00
Giles OdigweandCopilot 5ec3bcf390 Upgrade Anthropic SDK 12.13.0 -> 12.20.0 to fix M.E.AI incompatibility
Fixes MissingMethodException on WebSearchToolResultContent.get_Results()
caused by Anthropic 12.13.0 being compiled against an older
Microsoft.Extensions.AI.Abstractions version.

Suppress RT0003 in AI.Abstractions.csproj as the transitive reference
from the upgraded Anthropic SDK conflicts with the explicit one.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-08 14:29:37 -07:00
Giles OdigweandCopilot 9ce21a2a2f Re-enable Anthropic integration tests
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-08 09:48:57 -07:00
Giles OdigweandCopilot 2a55b35176 Address PR feedback: add Workflows.Generators to core filter, drop dotnetChanges gate from functions job
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-08 07:48:34 -07:00
Giles OdigweandCopilot 08dcf74cf4 Split DurableTask/AzureFunctions integration tests into dedicated CI job
- Add -TestProjectNameExclude parameter to New-FilteredSolution.ps1
- Add 'functions' and 'core' path filters to paths-filter job
- Exclude DurableTask/AzureFunctions from main dotnet-test job
- Remove emulator setup from dotnet-test (no longer needed)
- Add new dotnet-test-functions job (ubuntu/net10.0 only, path-conditional)
- Update merge gate and report job to include dotnet-test-functions

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-08 07:25:33 -07:00
84 changed files with 1062 additions and 6426 deletions
+13 -95
View File
@@ -2,7 +2,7 @@ name: Merge Gatekeeper
on:
pull_request:
branches: ["main", "feature*"]
branches: [ "main", "feature*" ]
merge_group:
branches: ["main"]
@@ -13,105 +13,23 @@ concurrency:
jobs:
merge-gatekeeper:
runs-on: ubuntu-latest
# Restrict permissions of the GITHUB_TOKEN.
# Docs: https://docs.github.com/en/actions/using-jobs/assigning-permissions-to-jobs
permissions:
checks: read
statuses: read
steps:
- name: Wait for required checks
- name: Run Merge Gatekeeper
# NOTE: v1 is updated to reflect the latest v1.x.y. Please use any tag/branch that suits your needs:
# https://github.com/upsidr/merge-gatekeeper/tags
# https://github.com/upsidr/merge-gatekeeper/branches
uses: upsidr/merge-gatekeeper@v1
if: github.event_name == 'pull_request'
uses: actions/github-script@v8
env:
TIMEOUT_SECONDS: "3600"
INTERVAL_SECONDS: "30"
SELF_JOB_NAME: ${{ github.job }}
with:
token: ${{ secrets.GITHUB_TOKEN }}
timeout: 3600
interval: 30
# "Cleanup artifacts", "Agent", "Prepare", and "Upload results" are check runs
# created by an org-level GitHub App (MSDO), not by any workflow in this repo.
# They are outside our control and their transient failures should not block merges.
IGNORED_NAMES: "CodeQL,CodeQL analysis (csharp),Cleanup artifacts,Agent,Prepare,Upload results"
with:
script: |
const timeoutSeconds = Number(process.env.TIMEOUT_SECONDS);
const intervalSeconds = Number(process.env.INTERVAL_SECONDS);
const selfName = process.env.SELF_JOB_NAME;
const ignored = new Set(
process.env.IGNORED_NAMES.split(',').map((s) => s.trim()).filter(Boolean),
);
const sha = context.payload.pull_request.head.sha;
const { owner, repo } = context.repo;
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
// Mirrors upsidr/merge-gatekeeper: merge combined-statuses and check-runs
// for the PR head SHA, with combined-statuses winning on name collision.
async function collectChecks() {
const merged = new Map();
const combined = await github.rest.repos.getCombinedStatusForRef({
owner, repo, ref: sha, per_page: 100,
});
for (const s of combined.data.statuses ?? []) {
if (!merged.has(s.context)) {
// Combined-status states: success | pending | error | failure
merged.set(s.context, { name: s.context, state: s.state });
}
}
const runs = await github.paginate(github.rest.checks.listForRef, {
owner, repo, ref: sha, per_page: 100,
});
for (const r of runs) {
if (merged.has(r.name)) continue;
let state;
if (r.status !== 'completed') {
state = 'pending';
} else if (r.conclusion === 'skipped') {
continue; // Skipped runs are dropped, matching the original action.
} else if (r.conclusion === 'success' || r.conclusion === 'neutral') {
state = 'success';
} else {
// cancelled | timed_out | action_required | stale | failure
state = 'error';
}
merged.set(r.name, { name: r.name, state });
}
return [...merged.values()];
}
function evaluate(entries) {
const failed = [];
const pending = [];
const succeeded = [];
for (const e of entries) {
if (e.name === selfName || ignored.has(e.name)) continue;
if (e.state === 'success') succeeded.push(e.name);
else if (e.state === 'error' || e.state === 'failure') failed.push(e.name);
else pending.push(e.name);
}
return { failed, pending, succeeded };
}
const deadline = Date.now() + timeoutSeconds * 1000;
for (;;) {
const entries = await collectChecks();
const { failed, pending, succeeded } = evaluate(entries);
core.info(
`succeeded=${succeeded.length} pending=${pending.length} failed=${failed.length}`,
);
if (failed.length) {
core.setFailed(`Failing checks: ${failed.join(', ')}`);
return;
}
if (pending.length === 0) {
core.info(`All required checks passed: ${succeeded.join(', ') || '(none)'}`);
return;
}
if (Date.now() > deadline) {
core.setFailed(`Timed out waiting for: ${pending.join(', ')}`);
return;
}
core.info(`Waiting on (${pending.length}): ${pending.slice(0, 10).join(', ')}${pending.length > 10 ? ', …' : ''}`);
await sleep(intervalSeconds * 1000);
}
ignored: CodeQL,CodeQL analysis (csharp),Cleanup artifacts,Agent,Prepare,Upload results
-2
View File
@@ -246,5 +246,3 @@ dotnet/filtered-*.slnx
# Local tool state
.omc/
.omx/
**/issues/
-3
View File
@@ -298,7 +298,6 @@
</Folder>
<Folder Name="/Samples/03-workflows/Evaluation/">
<Project Path="samples/03-workflows/Evaluation/Evaluation_WorkflowEval/Evaluation_WorkflowEval.csproj" />
<Project Path="samples/03-workflows/Evaluation/Evaluation_WorkflowExpectedOutputs/Evaluation_WorkflowExpectedOutputs.csproj" />
</Folder>
<Folder Name="/Samples/04-hosting/">
</Folder>
@@ -583,7 +582,6 @@
<Project Path="src/Microsoft.Agents.AI.DurableTask/Microsoft.Agents.AI.DurableTask.csproj" />
<Project Path="src/Microsoft.Agents.AI.Foundry.Hosting/Microsoft.Agents.AI.Foundry.Hosting.csproj" />
<Project Path="src/Microsoft.Agents.AI.Foundry/Microsoft.Agents.AI.Foundry.csproj" />
<Project Path="src/Microsoft.Agents.AI.Harness/Microsoft.Agents.AI.Harness.csproj" />
<Project Path="src/Microsoft.Agents.AI.GitHub.Copilot/Microsoft.Agents.AI.GitHub.Copilot.csproj" />
<Project Path="src/Microsoft.Agents.AI.Hosting.A2A.AspNetCore/Microsoft.Agents.AI.Hosting.A2A.AspNetCore.csproj" />
<Project Path="src/Microsoft.Agents.AI.Hosting.A2A/Microsoft.Agents.AI.Hosting.A2A.csproj" />
@@ -638,7 +636,6 @@
<Project Path="tests/Microsoft.Agents.AI.Foundry.UnitTests/Microsoft.Agents.AI.Foundry.UnitTests.csproj" />
<Project Path="tests/Microsoft.Agents.AI.Foundry.Hosting.UnitTests/Microsoft.Agents.AI.Foundry.Hosting.UnitTests.csproj" />
<Project Path="tests/Microsoft.Agents.AI.GitHub.Copilot.UnitTests/Microsoft.Agents.AI.GitHub.Copilot.UnitTests.csproj" />
<Project Path="tests/Microsoft.Agents.AI.Harness.UnitTests/Microsoft.Agents.AI.Harness.UnitTests.csproj" />
<Project Path="tests/Microsoft.Agents.AI.Hosting.A2A.UnitTests/Microsoft.Agents.AI.Hosting.A2A.UnitTests.csproj" />
<Project Path="tests/Microsoft.Agents.AI.Hosting.AGUI.AspNetCore.UnitTests/Microsoft.Agents.AI.Hosting.AGUI.AspNetCore.UnitTests.csproj" />
<Project Path="tests/Microsoft.Agents.AI.Hosting.AzureFunctions.UnitTests/Microsoft.Agents.AI.Hosting.AzureFunctions.UnitTests.csproj" />
-1
View File
@@ -7,7 +7,6 @@
"src\\Microsoft.Agents.AI.AGUI\\Microsoft.Agents.AI.AGUI.csproj",
"src\\Microsoft.Agents.AI.Anthropic\\Microsoft.Agents.AI.Anthropic.csproj",
"src\\Microsoft.Agents.AI.GitHub.Copilot\\Microsoft.Agents.AI.GitHub.Copilot.csproj",
"src\\Microsoft.Agents.AI.Harness\\Microsoft.Agents.AI.Harness.csproj",
"src\\Microsoft.Agents.AI.AzureAI.Persistent\\Microsoft.Agents.AI.AzureAI.Persistent.csproj",
"src\\Microsoft.Agents.AI.Foundry\\Microsoft.Agents.AI.Foundry.csproj",
"src\\Microsoft.Agents.AI.Foundry.Hosting\\Microsoft.Agents.AI.Foundry.Hosting.csproj",
+3 -3
View File
@@ -1,14 +1,14 @@
<Project>
<PropertyGroup>
<!-- Central version prefix - applies to all nuget packages. -->
<VersionPrefix>1.6.1</VersionPrefix>
<VersionPrefix>1.5.0</VersionPrefix>
<RCNumber>1</RCNumber>
<DateSuffix>260514</DateSuffix>
<DateSuffix>260507</DateSuffix>
<PackageVersion Condition="'$(IsReleaseCandidate)' == 'true'">$(VersionPrefix)-rc$(RCNumber)</PackageVersion>
<PackageVersion Condition="'$(IsReleaseCandidate)' != 'true' AND '$(VersionSuffix)' != ''">$(VersionPrefix)-$(VersionSuffix).$(DateSuffix).1</PackageVersion>
<PackageVersion Condition="'$(IsReleaseCandidate)' != 'true' AND '$(VersionSuffix)' == ''">$(VersionPrefix)-preview.$(DateSuffix).1</PackageVersion>
<PackageVersion Condition="'$(IsReleased)' == 'true'">$(VersionPrefix)</PackageVersion>
<GitTag>1.6.1</GitTag>
<GitTag>1.5.0</GitTag>
<Configurations>Debug;Release;Publish</Configurations>
<IsPackable>true</IsPackable>
@@ -20,18 +20,22 @@ using OpenAI.Responses;
#pragma warning disable OPENAI001 // Experimental API
#pragma warning disable AAIP001 // AgentToolboxes is experimental
// Name of the toolbox to create and connect to.
// Must match the `<name>` segment of FOUNDRY_TOOLBOX_ENDPOINT.
const string ToolboxName = "research_toolbox";
const string Query = "What tools do you have access to?";
string endpoint = Environment.GetEnvironmentVariable("AZURE_AI_PROJECT_ENDPOINT")
?? throw new InvalidOperationException("AZURE_AI_PROJECT_ENDPOINT is not set.");
string deploymentName = Environment.GetEnvironmentVariable("AZURE_AI_MODEL_DEPLOYMENT_NAME") ?? "gpt-5.4-mini";
string toolboxEndpoint = Environment.GetEnvironmentVariable("FOUNDRY_TOOLBOX_ENDPOINT")
?? throw new InvalidOperationException(
"FOUNDRY_TOOLBOX_ENDPOINT is not set. Example: " +
"https://<account>.services.ai.azure.com/api/projects/<project>/toolsets/<name>/mcp?api-version=2025-05-01-preview");
TokenCredential credential = new DefaultAzureCredential();
// Comment out if the toolbox already exists in your Foundry project.
var toolboxEndpoint = await CreateSampleToolboxAsync(ToolboxName, endpoint, credential);
await CreateSampleToolboxAsync(ToolboxName, endpoint, credential);
// Inject a fresh Azure AI bearer token on every MCP request.
using var httpClient = new HttpClient(new BearerTokenHandler(credential, "https://ai.azure.com/.default")
@@ -47,11 +51,6 @@ await using McpClient mcpClient = await McpClient.CreateAsync(
{
Endpoint = new Uri(toolboxEndpoint),
Name = "foundry_toolbox",
TransportMode = HttpTransportMode.StreamableHttp,
AdditionalHeaders = new Dictionary<string, string>
{
["Foundry-Features"] = "Toolboxes=V1Preview",
},
},
httpClient));
@@ -75,7 +74,7 @@ Console.WriteLine($"Assistant: {await agent.RunAsync(Query)}");
// ---------------------------------------------------------------------------
// Helper: create (or replace) a sample toolbox so the sample runs end-to-end
// ---------------------------------------------------------------------------
static async Task<string> CreateSampleToolboxAsync(string name, string endpoint, TokenCredential credential)
static async Task CreateSampleToolboxAsync(string name, string endpoint, TokenCredential credential)
{
// Toolboxes are normally configured in the Foundry portal or a deployment
// script, not the application itself. This helper exists so the sample can
@@ -104,13 +103,12 @@ static async Task<string> CreateSampleToolboxAsync(string name, string endpoint,
serverUri: new Uri("https://gitmcp.io/Azure/azure-rest-api-specs"),
toolCallApprovalPolicy: new McpToolCallApprovalPolicy(GlobalMcpToolCallApprovalPolicy.NeverRequireApproval)));
ToolboxVersion created = (await toolboxClient.CreateToolboxVersionAsync(
var created = (await toolboxClient.CreateToolboxVersionAsync(
name: name,
tools: [mcpTool],
description: "Sample toolbox with an MCP tool — created by Agent_Step25 sample.")).Value;
Console.WriteLine($"Created toolbox '{created.Name}' v{created.Version} ({created.Tools.Count} tool(s))");
return $"{endpoint}/toolboxes/{created.Name}/mcp?api-version=v{created.Version}";
}
// ---------------------------------------------------------------------------
@@ -19,11 +19,10 @@ Set the following environment variables:
```powershell
$env:AZURE_AI_PROJECT_ENDPOINT="https://your-foundry-service.services.ai.azure.com/api/projects/your-foundry-project"
$env:AZURE_AI_MODEL_DEPLOYMENT_NAME="gpt-5.4-mini"
$env:FOUNDRY_TOOLBOX_ENDPOINT="https://your-foundry-service.services.ai.azure.com/api/projects/your-foundry-project/toolsets/research_toolbox/mcp?api-version=2025-05-01-preview"
```
The sample creates a toolbox named `research_toolbox` in your Foundry project on
startup, then connects to its MCP endpoint at
`{AZURE_AI_PROJECT_ENDPOINT}/toolboxes/research_toolbox/mcp?api-version=v{version}`.
The `<name>` segment of `FOUNDRY_TOOLBOX_ENDPOINT` must match the `ToolboxName` constant in `Program.cs`.
## Run the sample
@@ -13,7 +13,6 @@
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\..\..\..\src\Microsoft.Agents.AI.Harness\Microsoft.Agents.AI.Harness.csproj" />
<ProjectReference Include="..\..\..\..\src\Microsoft.Agents.AI.OpenAI\Microsoft.Agents.AI.OpenAI.csproj" />
<ProjectReference Include="..\Harness_Shared_Console\Harness_Shared_Console.csproj" />
</ItemGroup>
@@ -1,6 +1,6 @@
// Copyright (c) Microsoft. All rights reserved.
// This sample demonstrates how to use a HarnessAgent with the Harness AIContextProviders
// This sample demonstrates how to use a ChatClientAgent with the Harness AIContextProviders
// (TodoProvider and AgentModeProvider) for interactive research tasks with web search
// capabilities powered by Azure AI Foundry.
// The agent plans research tasks, creates a todo list, gets user approval,
@@ -17,6 +17,7 @@ using System.ClientModel.Primitives;
using Azure.Identity;
using Harness.Shared.Console;
using Microsoft.Agents.AI;
using Microsoft.Agents.AI.Compaction;
using Microsoft.Extensions.AI;
using OpenAI;
using OpenAI.Responses;
@@ -28,7 +29,7 @@ var deploymentName = Environment.GetEnvironmentVariable("AZURE_AI_MODEL_DEPLOYME
const int MaxContextWindowTokens = 1_050_000;
const int MaxOutputTokens = 128_000;
// Create a HarnessAgent with the Harness providers (TodoProvider and AgentModeProvider)
// Create a ChatClientAgent with the Harness providers (TodoProvider and AgentModeProvider)
// and research-focused instructions including the mandatory planning workflow.
var instructions =
"""
@@ -109,9 +110,13 @@ var instructions =
- Check for relevant previously downloaded data / findings before starting new research.
""";
// Create the agent using AsHarnessAgent, which pre-configures function invocation,
// per-service-call chat history persistence, and in-loop compaction.
// Then wrap with UseToolApproval to allow auto-approving tools once confirmed.
// Create a compaction strategy based on the model's context window.
// gpt-5.4: 1,050,000 token context window, 128,000 max output tokens.
// Defaults: tool result eviction at 50% of input budget, truncation at 80%.
var compactionStrategy = new ContextWindowCompactionStrategy(
maxContextWindowTokens: MaxContextWindowTokens,
maxOutputTokens: MaxOutputTokens);
AIAgent agent =
// Create an OpenAIClient that communicates with the Foundry responses service.
new OpenAIClient(
@@ -125,32 +130,49 @@ AIAgent agent =
RetryPolicy = new ClientRetryPolicy(3) // Enable retries to improve resiliency.
})
.GetResponsesClient()
.AsIChatClientWithStoredOutputDisabled(deploymentName) // We want to manage chat history locally (not stored in the responses service), so that we can manage compaction ourselves.
.AsHarnessAgent(MaxContextWindowTokens, MaxOutputTokens, new HarnessAgentOptions
{
Name = "ResearchAgent",
Description = "A research assistant that plans and executes research tasks.",
AIContextProviders =
[
new TodoProvider(), // Add an AIContextProvider to allow the agent to create a TODO list, which is stored in the session.
new AgentModeProvider(), // Add an AIContextProvider that tracks the agent mode and allows switching mode. Current mode is stored in the session.
new FileMemoryProvider( // Add an AIContextProvider that can store memories in files under a session specific working folder.
new FileSystemAgentFileStore(Path.Combine(AppContext.BaseDirectory, "agent-files")),
(_) => new FileMemoryState() { WorkingFolder = DateTime.UtcNow.ToString("yyyyMMdd_HHmmss") + "_" + Guid.NewGuid().ToString() })
],
ChatOptions = new ChatOptions
.AsIChatClientWithStoredOutputDisabled(deploymentName) // We want to manage chat history locally (not stored in the responses service), so that we can manage compaction ourselves.
// Build a ChatClient Pipeline
.AsBuilder()
.UseFunctionInvocation() // We are building our own stack from scratch so we need to include Function Invocation ourselves.
.UseMessageInjection() // Allow message injection during the function call loop.
.UsePerServiceCallChatHistoryPersistence() // Save chat history updates to the session after each service call, rather than only at the end of the run.
.UseAIContextProviders(new CompactionProvider(compactionStrategy)) // Add Compaction before each service call to responses so that long function invocation loops don't overflow the context.
// Build our agent on top of the ChatClient Pipeline
.BuildAIAgent(
new ChatClientAgentOptions
{
Instructions = instructions,
Tools =
Name = "ResearchAgent",
Description = "A research assistant that plans and executes research tasks.",
UseProvidedChatClientAsIs = true, // Since we built our own stack from scratch we need to tell the agent not to also add defaults like Function Invocation.
RequirePerServiceCallChatHistoryPersistence = true, // Since we are added the per service call persistence ChatClient, we need to tell the agent to not also store chat history at the end of the run.
ChatHistoryProvider = new InMemoryChatHistoryProvider( // Store chat history in memory in the session object. Will persist if the session is persisted.
new InMemoryChatHistoryProviderOptions
{
ChatReducer = compactionStrategy.AsChatReducer(), // Run compaction on the InMemory chat history when it gets too large.
}),
AIContextProviders =
[
ResponseTool.CreateWebSearchTool().AsAITool(), // Add the foundry hosted web search tool that runs in the service.
new WebBrowsingTool( // Add a local web browsing tool that converts html to markdown.
new WebBrowsingToolOptions { AllowPublicNetworks = true }),
new TodoProvider(), // Add an AIContextProvider to allow the agent to create a TODO list, which is stored in the session.
new AgentModeProvider(), // Add an AIContextProvider that tracks the agent mode and allows switching mode. Current mode is stored in the session.
new FileMemoryProvider( // Add an AIContextProvider that can store memories in files under a session specific working folder.
new FileSystemAgentFileStore(Path.Combine(AppContext.BaseDirectory, "agent-files")),
(_) => new FileMemoryState() { WorkingFolder = DateTime.UtcNow.ToString("yyyyMMdd_HHmmss") + "_" + Guid.NewGuid().ToString() })
],
MaxOutputTokens = MaxOutputTokens, // Set a high token limit for long research tasks with many tool calls and long outputs.
Reasoning = new() { Effort = ReasoningEffort.Medium },
},
})
ChatOptions = new ChatOptions
{
Instructions = instructions,
Tools =
[
ResponseTool.CreateWebSearchTool().AsAITool(), // Add the foundry hosted web search tool that runs in the service.
new WebBrowsingTool( // Add a local web browsing tool that converts html to markdown.
new WebBrowsingToolOptions { AllowPublicNetworks = true }),
],
MaxOutputTokens = MaxOutputTokens, // Set a high token limit for long research tasks with many tool calls and long outputs.
Reasoning = new() { Effort = ReasoningEffort.Medium },
},
})
.AsBuilder()
.UseToolApproval() // Add the ability to auto approve tools once a user has said they don't want to be asked again. Approval rules are tied to the session.
.Build();
@@ -1,11 +1,10 @@
# What this sample demonstrates
This sample demonstrates how to use a `HarnessAgent` with the Harness `AIContextProviders` (`TodoProvider` and `AgentModeProvider`) for interactive research tasks with web search capabilities powered by Azure AI Foundry. The `HarnessAgent` pre-configures function invocation, per-service-call chat history persistence, and context-window compaction.
This sample demonstrates how to use a `ChatClientAgent` with the Harness `AIContextProviders` (`TodoProvider` and `AgentModeProvider`) for interactive research tasks with web search capabilities powered by Azure AI Foundry.
Key features showcased:
- **HarnessAgent** — a pre-configured agent that wraps a `ChatClientAgent` with function invocation, per-service-call persistence, and context-window compaction
- **ToolApproval** — the agent is wrapped with `UseToolApproval()` to allow auto-approving tools once confirmed
- **ChatClientAgent** — configured directly with Harness providers for planning and task management
- **Web Search** — the agent can search the web for current information via `ResponseTool.CreateWebSearchTool()`
- **TodoProvider** — the agent creates and manages a todo list to track research questions
- **AgentModeProvider** — the agent switches between "plan" mode (breaking down the topic) and "execute" mode (answering each research question)
@@ -13,7 +13,6 @@
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\..\..\..\src\Microsoft.Agents.AI.Harness\Microsoft.Agents.AI.Harness.csproj" />
<ProjectReference Include="..\..\..\..\src\Microsoft.Agents.AI.OpenAI\Microsoft.Agents.AI.OpenAI.csproj" />
<ProjectReference Include="..\Harness_Shared_Console\Harness_Shared_Console.csproj" />
</ItemGroup>
@@ -22,9 +22,6 @@ using OpenAI.Responses;
var endpoint = Environment.GetEnvironmentVariable("AZURE_FOUNDRY_OPENAI_ENDPOINT") ?? throw new InvalidOperationException("AZURE_FOUNDRY_OPENAI_ENDPOINT is not set.");
var deploymentName = Environment.GetEnvironmentVariable("AZURE_AI_MODEL_DEPLOYMENT_NAME") ?? "gpt-5.4";
const int MaxContextWindowTokens = 1_050_000;
const int MaxOutputTokens = 128_000;
// --- Sub-agent: Web Search Agent ---
// This agent can search the web and is used by the parent agent to look up stock prices.
AIAgent webSearchAgent =
@@ -37,19 +34,20 @@ AIAgent webSearchAgent =
})
.GetResponsesClient()
.AsIChatClientWithStoredOutputDisabled(deploymentName)
.AsHarnessAgent(MaxContextWindowTokens, MaxOutputTokens, new HarnessAgentOptions
{
Name = "WebSearchAgent",
Description = "An agent that can search the web to find information.",
ChatOptions = new ChatOptions
.AsAIAgent(
new ChatClientAgentOptions
{
Instructions = "You are a web search assistant. When asked to find information, use the web search tool to look it up and return a concise, factual answer.",
Tools =
[
ResponseTool.CreateWebSearchTool().AsAITool(),
],
},
});
Name = "WebSearchAgent",
Description = "An agent that can search the web to find information.",
ChatOptions = new ChatOptions
{
Instructions = "You are a web search assistant. When asked to find information, use the web search tool to look it up and return a concise, factual answer.",
Tools =
[
ResponseTool.CreateWebSearchTool().AsAITool(),
],
},
});
// --- Parent agent: Stock Price Researcher ---
// This agent orchestrates the sub-agent to look up stock prices in parallel.
@@ -85,20 +83,21 @@ AIAgent parentAgent =
})
.GetResponsesClient()
.AsIChatClientWithStoredOutputDisabled(deploymentName)
.AsHarnessAgent(MaxContextWindowTokens, MaxOutputTokens, new HarnessAgentOptions
{
Name = "StockPriceResearcher",
Description = "An agent that researches stock prices using sub-agents.",
AIContextProviders =
[
new SubAgentsProvider([webSearchAgent]),
],
ChatOptions = new ChatOptions
.AsAIAgent(
new ChatClientAgentOptions
{
Instructions = parentInstructions,
MaxOutputTokens = 16_000,
},
});
Name = "StockPriceResearcher",
Description = "An agent that researches stock prices using sub-agents.",
AIContextProviders =
[
new SubAgentsProvider([webSearchAgent]),
],
ChatOptions = new ChatOptions
{
Instructions = parentInstructions,
MaxOutputTokens = 16_000,
},
});
// Run the interactive console session.
await HarnessConsole.RunAgentAsync(
@@ -1,6 +1,6 @@
# Harness Step 02 — SubAgents (Stock Price Research)
This sample demonstrates how to use the **SubAgentsProvider** to delegate work from a parent agent to sub-agents. Both agents use `HarnessAgent` for pre-configured function invocation, per-service-call persistence, and context-window compaction.
This sample demonstrates how to use the **SubAgentsProvider** to delegate work from a parent agent to sub-agents.
## What It Does
@@ -13,7 +13,6 @@
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\..\..\..\src\Microsoft.Agents.AI.Harness\Microsoft.Agents.AI.Harness.csproj" />
<ProjectReference Include="..\..\..\..\src\Microsoft.Agents.AI.OpenAI\Microsoft.Agents.AI.OpenAI.csproj" />
<ProjectReference Include="..\Harness_Shared_Console\Harness_Shared_Console.csproj" />
</ItemGroup>
@@ -1,6 +1,6 @@
// Copyright (c) Microsoft. All rights reserved.
// This sample demonstrates how to use a HarnessAgent with the FileAccessProvider
// This sample demonstrates how to use a ChatClientAgent with the FileAccessProvider
// to give an agent access to a folder of CSV data files. The agent can read, analyze,
// and extract information from the data, then write results back as new files.
//
@@ -17,6 +17,7 @@ using System.ClientModel.Primitives;
using Azure.Identity;
using Harness.Shared.Console;
using Microsoft.Agents.AI;
using Microsoft.Agents.AI.Compaction;
using Microsoft.Extensions.AI;
using OpenAI;
using OpenAI.Responses;
@@ -56,7 +57,11 @@ var instructions =
- Always explain what you learned and what you are going to do next between tool calls, so the user can follow along with your thought process.
""";
// Create the chat client from the OpenAI provider.
// Create a compaction strategy based on the model's context window.
var compactionStrategy = new ContextWindowCompactionStrategy(
maxContextWindowTokens: MaxContextWindowTokens,
maxOutputTokens: MaxOutputTokens);
AIAgent agent =
new OpenAIClient(
new BearerTokenPolicy(new DefaultAzureCredential(), "https://ai.azure.com/.default"),
@@ -67,20 +72,36 @@ AIAgent agent =
})
.GetResponsesClient()
.AsIChatClientWithStoredOutputDisabled(deploymentName)
.AsHarnessAgent(MaxContextWindowTokens, MaxOutputTokens, new HarnessAgentOptions
{
Name = "DataAnalyst",
Description = "A data analyst assistant that reads, analyzes, and processes data files.",
AIContextProviders =
[
new FileAccessProvider(fileStore),
],
ChatOptions = new ChatOptions
.AsBuilder()
.UseFunctionInvocation()
.UsePerServiceCallChatHistoryPersistence()
.UseAIContextProviders(new CompactionProvider(compactionStrategy))
.BuildAIAgent(
new ChatClientAgentOptions
{
Instructions = instructions,
MaxOutputTokens = MaxOutputTokens,
},
});
Name = "DataAnalyst",
Description = "A data analyst assistant that reads, analyzes, and processes data files.",
UseProvidedChatClientAsIs = true,
RequirePerServiceCallChatHistoryPersistence = true,
ChatHistoryProvider = new InMemoryChatHistoryProvider(
new InMemoryChatHistoryProviderOptions
{
ChatReducer = compactionStrategy.AsChatReducer(),
}),
AIContextProviders =
[
new FileAccessProvider(fileStore),
],
ChatOptions = new ChatOptions
{
Instructions = instructions,
MaxOutputTokens = MaxOutputTokens,
},
})
.AsBuilder()
.Build();
// Run the interactive console session.
await HarnessConsole.RunAgentAsync(
@@ -1,10 +1,9 @@
# What this sample demonstrates
This sample demonstrates how to use a `HarnessAgent` with the `FileAccessProvider` to give an agent access to a folder of data files for reading, analyzing, and writing results. The `HarnessAgent` pre-configures function invocation, per-service-call chat history persistence, and in-loop compaction — so the sample only needs to supply the chat client, token limits, and application-specific options.
This sample demonstrates how to use a `ChatClientAgent` with the `FileAccessProvider` to give an agent access to a folder of data files for reading, analyzing, and writing results.
Key features showcased:
- **HarnessAgent** — a pre-configured agent that wraps a `ChatClientAgent` with function invocation, per-service-call persistence, and context-window compaction
- **FileAccessProvider** — gives the agent tools to read, write, list, search, and delete files in a shared data folder
- **CSV data processing** — the agent reads sales transaction data and performs analysis on demand
- **Output file creation** — the agent can write summaries, filtered data, or reports back to the data folder
@@ -1,16 +0,0 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFrameworks>net10.0</TargetFrameworks>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\..\..\..\src\Microsoft.Agents.AI.Foundry\Microsoft.Agents.AI.Foundry.csproj" />
<ProjectReference Include="..\..\..\..\src\Microsoft.Agents.AI.Workflows\Microsoft.Agents.AI.Workflows.csproj" />
</ItemGroup>
</Project>
@@ -1,76 +0,0 @@
// Copyright (c) Microsoft. All rights reserved.
// This sample demonstrates evaluating a multi-agent workflow against a
// golden answer using Foundry's reference-based Similarity evaluator.
using Azure.AI.Projects;
using Azure.Identity;
using Microsoft.Agents.AI;
using Microsoft.Agents.AI.Workflows;
using Microsoft.Extensions.AI;
using FoundryEvals = Microsoft.Agents.AI.Foundry.FoundryEvals;
string endpoint = Environment.GetEnvironmentVariable("AZURE_AI_PROJECT_ENDPOINT")
?? throw new InvalidOperationException("AZURE_AI_PROJECT_ENDPOINT is not set.");
string deploymentName = Environment.GetEnvironmentVariable("AZURE_AI_MODEL_DEPLOYMENT_NAME") ?? "gpt-4o-mini";
// WARNING: DefaultAzureCredential is convenient for development but requires careful consideration in production.
// In production, consider using a specific credential (e.g., ManagedIdentityCredential) to avoid
// latency issues, unintended credential probing, and potential security risks from fallback mechanisms.
AIProjectClient projectClient = new(new Uri(endpoint), new DefaultAzureCredential());
// Build a two-agent workflow: a researcher writes a draft answer, then an
// editor polishes it into the final response that we compare to ground truth.
// EmitAgentResponseEvents is enabled so the workflow surfaces an AgentResponseEvent
// for each agent — this is what EvaluateAsync uses to find the overall final answer.
var hostOptions = new AIAgentHostOptions { EmitAgentResponseEvents = true };
AIAgent researcher = projectClient.AsAIAgent(
model: deploymentName,
instructions: "You research questions and produce a short factual draft answer.",
name: "researcher");
AIAgent editor = projectClient.AsAIAgent(
model: deploymentName,
instructions: "You take a draft answer and produce the final concise response.",
name: "editor");
ExecutorBinding researcherExecutor = researcher.BindAsExecutor(hostOptions);
ExecutorBinding editorExecutor = editor.BindAsExecutor(hostOptions);
Workflow workflow = new WorkflowBuilder(researcherExecutor)
.AddEdge(researcherExecutor, editorExecutor)
.Build();
// Run the workflow against the user question.
const string Query = "What is the capital of France?";
const string GroundTruth = "Paris";
await using Run run = await InProcessExecution.RunAsync(
workflow,
new ChatMessage(ChatRole.User, Query));
// Evaluate the overall workflow output against a golden answer using the
// reference-based Similarity evaluator. The 'expectedOutput' value is stamped
// onto the overall EvalItem.ExpectedOutput and is surfaced to Foundry as
// `ground_truth` in the underlying JSONL payload.
//
// Per-agent breakdown is disabled here: ground truth applies to the workflow's
// final answer, not to each sub-agent's intermediate output. Without
// includePerAgent: false, the evaluator would be invoked for per-agent items
// (which have no ExpectedOutput) and Similarity would fail validation.
FoundryEvals similarity = new(projectClient, deploymentName, FoundryEvals.Similarity);
AgentEvaluationResults results = await run.EvaluateAsync(
similarity,
includePerAgent: false,
expectedOutput: GroundTruth);
Console.WriteLine($"Query: {Query}");
Console.WriteLine($"Expected: {GroundTruth}");
Console.WriteLine($"Provider: {results.ProviderName}");
Console.WriteLine($"Passed: {results.Passed}/{results.Total}");
if (results.ReportUrl is not null)
{
Console.WriteLine($"Report: {results.ReportUrl}");
}
@@ -1,37 +0,0 @@
# Evaluation - Workflow Expected Outputs
This sample demonstrates evaluating a multi-agent workflow's final answer
against a golden expected output using Foundry's reference-based **Similarity**
evaluator.
## What this sample demonstrates
- Building a small researcher → editor workflow
- Running the workflow and obtaining a `Run`
- Calling `run.EvaluateAsync(evaluator, expectedOutput: ...)` to attach a
ground-truth answer to the overall workflow item
- Using `FoundryEvals.Similarity`, which requires a `ground_truth` value
per item
The `expectedOutput` value is stamped onto the overall `EvalItem.ExpectedOutput`
and is surfaced to Foundry as `ground_truth` in the JSONL payload sent to the
Evals API.
## Prerequisites
- .NET 10 SDK or later
- Azure CLI installed and authenticated (`az login`)
Set the following environment variables:
```powershell
$env:AZURE_AI_PROJECT_ENDPOINT="https://your-foundry-service.services.ai.azure.com/api/projects/your-foundry-project"
$env:AZURE_AI_MODEL_DEPLOYMENT_NAME="gpt-4o-mini"
```
## Run the sample
```powershell
cd dotnet/samples/03-workflows/Evaluation
dotnet run --project .\Evaluation_WorkflowExpectedOutputs
```
@@ -163,22 +163,10 @@ app.MapA2AHttpJson(knightsKnavesAgentBuilder, path: "/a2a/knights-and-knaves");
app.MapDevUI();
app.MapOpenAIResponses();
app.MapOpenAIResponses(pirateAgentBuilder);
app.MapOpenAIResponses(knightsKnavesAgentBuilder);
app.MapOpenAIResponses(chemistryAgent);
app.MapOpenAIResponses(mathsAgent);
app.MapOpenAIResponses(literatureAgent);
app.MapOpenAIResponses(scienceSequentialWorkflow);
app.MapOpenAIResponses(scienceConcurrentWorkflow);
app.MapOpenAIConversations();
app.MapOpenAIChatCompletions(pirateAgentBuilder);
app.MapOpenAIChatCompletions(knightsKnavesAgentBuilder);
app.MapOpenAIChatCompletions(chemistryAgent);
app.MapOpenAIChatCompletions(mathsAgent);
app.MapOpenAIChatCompletions(literatureAgent);
app.MapOpenAIChatCompletions(scienceSequentialWorkflow);
app.MapOpenAIChatCompletions(scienceConcurrentWorkflow);
// Map the agents HTTP endpoints
app.MapAgentDiscovery("/agents");
@@ -24,7 +24,7 @@ internal sealed class OpenAIChatCompletionsAgentClient(HttpClient httpClient) :
{
OpenAIClientOptions options = new()
{
Endpoint = new Uri(httpClient.BaseAddress!, $"/{Uri.EscapeDataString(agentName)}/v1/"),
Endpoint = new Uri(httpClient.BaseAddress!, $"/{agentName}/v1/"),
Transport = new HttpClientPipelineTransport(httpClient)
};
@@ -23,7 +23,7 @@ internal sealed class OpenAIResponsesAgentClient(HttpClient httpClient) : AgentC
{
OpenAIClientOptions options = new()
{
Endpoint = new Uri(httpClient.BaseAddress!, $"/{Uri.EscapeDataString(agentName)}/v1/"),
Endpoint = new Uri(httpClient.BaseAddress!, "/v1/"),
Transport = new HttpClientPipelineTransport(httpClient)
};
@@ -51,7 +51,9 @@ internal sealed class DevUIAuthFilter : IEndpointFilter
if (!isLoopback && !this._options.AllowRemoteAccess)
{
DevUILog.RejectedNonLoopbackRequest(this._logger, remoteIp);
this._logger.LogWarning(
"Rejected non-loopback DevUI request from {RemoteIp}. Set DevUIOptions.AllowRemoteAccess to permit remote callers.",
remoteIp);
return Results.Problem(
statusCode: StatusCodes.Status403Forbidden,
title: "DevUI access denied",
@@ -100,7 +100,10 @@ public static class DevUIExtensions
if (options.AllowRemoteAccess && !tokenConfigured && options.ConfigureEndpoints is null)
{
DevUILog.InsecurelyExposed(logger, DevUIOptions.AuthTokenEnvironmentVariable);
logger.LogWarning(
"DevUI is configured with AllowRemoteAccess=true and no authentication. " +
"Set DevUIOptions.AuthToken, the {EnvVar} environment variable, or attach an authorization policy via ConfigureEndpoints.",
DevUIOptions.AuthTokenEnvironmentVariable);
}
}
}
@@ -1,20 +0,0 @@
// Copyright (c) Microsoft. All rights reserved.
using System.Net;
namespace Microsoft.Agents.AI.DevUI;
internal static partial class DevUILog
{
[LoggerMessage(
EventId = 1,
Level = LogLevel.Warning,
Message = "Rejected non-loopback DevUI request from {RemoteIp}. Set DevUIOptions.AllowRemoteAccess to permit remote callers.")]
public static partial void RejectedNonLoopbackRequest(ILogger logger, IPAddress? remoteIp);
[LoggerMessage(
EventId = 2,
Level = LogLevel.Warning,
Message = "DevUI is configured with AllowRemoteAccess=true and no authentication. Set DevUIOptions.AuthToken, the {EnvVar} environment variable, or attach an authorization policy via ConfigureEndpoints.")]
public static partial void InsecurelyExposed(ILogger logger, string envVar);
}
@@ -130,7 +130,6 @@ internal static class FoundryEvalConverter
QueryMessages = ConvertMessages(queryMessages),
ResponseMessages = ConvertMessages(responseMessages),
Context = item.Context,
GroundTruth = item.ExpectedOutput,
ToolDefinitions = item.Tools is { Count: > 0 }
? item.Tools
.OfType<AIFunction>()
@@ -186,11 +185,6 @@ internal static class FoundryEvalConverter
dataMapping["context"] = "{{item.context}}";
}
if (GroundTruthEvaluators.Contains(qualified))
{
dataMapping["ground_truth"] = "{{item.ground_truth}}";
}
if (ToolEvaluators.Contains(qualified))
{
dataMapping["tool_definitions"] = "{{item.tool_definitions}}";
@@ -212,7 +206,7 @@ internal static class FoundryEvalConverter
/// <summary>
/// Builds the <c>item_schema</c> for custom JSONL eval definitions.
/// </summary>
internal static WireItemSchema BuildItemSchema(bool hasContext = false, bool hasTools = false, bool hasGroundTruth = false)
internal static WireItemSchema BuildItemSchema(bool hasContext = false, bool hasTools = false)
{
var properties = new Dictionary<string, WireSchemaProperty>
{
@@ -227,11 +221,6 @@ internal static class FoundryEvalConverter
properties["context"] = new WireSchemaProperty { Type = "string" };
}
if (hasGroundTruth)
{
properties["ground_truth"] = new WireSchemaProperty { Type = "string" };
}
if (hasTools)
{
properties["tool_definitions"] = new WireSchemaProperty { Type = "array" };
@@ -244,31 +233,6 @@ internal static class FoundryEvalConverter
};
}
/// <summary>
/// Returns the subset of <paramref name="evaluators"/> that require a ground-truth
/// (reference) value but cannot be evaluated because no item provided one.
/// </summary>
internal static List<string> FindMissingGroundTruthEvaluators(
IEnumerable<string> evaluators,
bool hasGroundTruth)
{
if (hasGroundTruth)
{
return [];
}
var missing = new List<string>();
foreach (var name in evaluators)
{
if (GroundTruthEvaluators.Contains(ResolveEvaluator(name)))
{
missing.Add(name);
}
}
return missing;
}
/// <summary>
/// Resolves a short evaluator name to its fully-qualified <c>builtin.*</c> form.
/// </summary>
@@ -313,12 +277,6 @@ internal static class FoundryEvalConverter
"builtin.tool_call_success",
};
// Evaluators that require a ground_truth (reference) value per item.
internal static readonly HashSet<string> GroundTruthEvaluators = new(StringComparer.OrdinalIgnoreCase)
{
"builtin.similarity",
};
// Short name → fully-qualified name mapping.
internal static readonly Dictionary<string, string> BuiltinEvaluators = new(StringComparer.OrdinalIgnoreCase)
{
@@ -103,9 +103,6 @@ internal sealed class WireEvalItemPayload
[JsonPropertyName("context")]
public string? Context { get; init; }
[JsonPropertyName("ground_truth")]
public string? GroundTruth { get; init; }
[JsonPropertyName("tool_definitions")]
public List<WireToolDefinition>? ToolDefinitions { get; init; }
}
@@ -145,8 +145,6 @@ public sealed class FoundryEvals : IAgentEvaluator
bool hasContext = payloads.Any(p => p.Context is not null);
bool hasTools = payloads.Any(p => p.ToolDefinitions is { Count: > 0 });
bool hasGroundTruth = payloads.Any(p => p.GroundTruth is not null);
bool allHaveGroundTruth = payloads.Count > 0 && payloads.All(p => p.GroundTruth is not null);
// Filter out tool evaluators if no items have tools; auto-add ToolCallAccuracy if tools present
var evaluators = FilterToolEvaluators(this._evaluatorNames, hasTools);
@@ -155,27 +153,13 @@ public sealed class FoundryEvals : IAgentEvaluator
evaluators = [.. evaluators, ToolCallAccuracy];
}
// Fail fast if a ground-truth evaluator (e.g. similarity) is requested but not
// every item carries an ExpectedOutput. Reference-based evaluators score each
// item against its own ground truth, so even one missing value will surface as
// a provider-side validation error. Catch it here with a clearer message.
var missingGroundTruth = FoundryEvalConverter.FindMissingGroundTruthEvaluators(evaluators, allHaveGroundTruth);
if (missingGroundTruth.Count > 0)
{
throw new InvalidOperationException(
"The following evaluator(s) require a ground-truth/expected output on every item but " +
$"at least one item is missing an {nameof(EvalItem.ExpectedOutput)}: {string.Join(", ", missingGroundTruth)}. " +
"Provide an expected output per item (for example via the 'expectedOutput' parameter on EvaluateAsync), " +
"or set 'includePerAgent: false' so the evaluator only runs on the overall item.");
}
// 2. Create the evaluation definition
var createEvalPayload = new WireCreateEvalRequest
{
Name = evalName,
DataSourceConfig = new WireCustomDataSourceConfig
{
ItemSchema = FoundryEvalConverter.BuildItemSchema(hasContext, hasTools, hasGroundTruth),
ItemSchema = FoundryEvalConverter.BuildItemSchema(hasContext, hasTools),
},
TestingCriteria = FoundryEvalConverter.BuildTestingCriteria(
evaluators, this._model, includeDataMapping: true),
@@ -838,15 +822,15 @@ public sealed class FoundryEvals : IAgentEvaluator
var result = new EvalItemResult(itemId, status, scores);
// Extract error info from sample
if (outputItem.TryGetProperty("sample", out var sample) && sample.ValueKind == JsonValueKind.Object)
if (outputItem.TryGetProperty("sample", out var sample))
{
if (sample.TryGetProperty("error", out var errObj) && errObj.ValueKind == JsonValueKind.Object)
if (sample.TryGetProperty("error", out var errObj))
{
result.ErrorCode = errObj.TryGetProperty("code", out var code) ? code.GetString() : null;
result.ErrorMessage = errObj.TryGetProperty("message", out var msg) ? msg.GetString() : null;
}
if (sample.TryGetProperty("usage", out var usage) && usage.ValueKind == JsonValueKind.Object && usage.TryGetProperty("total_tokens", out var tt) && tt.ValueKind == JsonValueKind.Number)
if (sample.TryGetProperty("usage", out var usage) && usage.TryGetProperty("total_tokens", out var tt) && tt.ValueKind == JsonValueKind.Number)
{
var tokenUsage = new Dictionary<string, int>();
if (usage.TryGetProperty("prompt_tokens", out var pt) && pt.ValueKind == JsonValueKind.Number)
@@ -902,7 +886,7 @@ public sealed class FoundryEvals : IAgentEvaluator
}
// Extract response_id from datasource_item
if (outputItem.TryGetProperty("datasource_item", out var dsItem) && dsItem.ValueKind == JsonValueKind.Object)
if (outputItem.TryGetProperty("datasource_item", out var dsItem))
{
if (dsItem.TryGetProperty("resp_id", out var respId))
{
@@ -1,42 +0,0 @@
// Copyright (c) Microsoft. All rights reserved.
using System.Diagnostics.CodeAnalysis;
using Microsoft.Agents.AI;
using Microsoft.Shared.DiagnosticIds;
namespace Microsoft.Extensions.AI;
/// <summary>
/// Provides extension methods for creating a <see cref="HarnessAgent"/> from an <see cref="IChatClient"/>.
/// </summary>
[Experimental(DiagnosticIds.Experiments.AgentsAIExperiments)]
public static class ChatClientHarnessExtensions
{
/// <summary>
/// Creates a new <see cref="HarnessAgent"/> that wraps this <see cref="IChatClient"/> with a pre-configured
/// pipeline including function invocation, per-service-call chat history persistence, and in-loop compaction.
/// </summary>
/// <param name="chatClient">
/// The <see cref="IChatClient"/> that provides access to the underlying AI model.
/// </param>
/// <param name="maxContextWindowTokens">
/// The maximum number of tokens the model's context window supports (e.g., 1,050,000 for gpt-5.4).
/// Used to configure the compaction strategy.
/// </param>
/// <param name="maxOutputTokens">
/// The maximum number of output tokens the model can generate per response (e.g., 128,000 for gpt-5.4).
/// Used to configure the compaction strategy.
/// </param>
/// <param name="options">
/// Optional configuration options for the agent, including instructions override, tools,
/// additional context providers, and chat history provider.
/// When <see langword="null"/>, the agent uses built-in default settings.
/// </param>
/// <returns>A new <see cref="HarnessAgent"/> instance.</returns>
public static HarnessAgent AsHarnessAgent(
this IChatClient chatClient,
int maxContextWindowTokens,
int maxOutputTokens,
HarnessAgentOptions? options = null) =>
new(chatClient, maxContextWindowTokens, maxOutputTokens, options);
}
@@ -1,135 +0,0 @@
// Copyright (c) Microsoft. All rights reserved.
using System.Diagnostics.CodeAnalysis;
using Microsoft.Agents.AI.Compaction;
using Microsoft.Extensions.AI;
using Microsoft.Shared.DiagnosticIds;
using Microsoft.Shared.Diagnostics;
namespace Microsoft.Agents.AI;
/// <summary>
/// A pre-configured <see cref="DelegatingAIAgent"/> that wraps a <see cref="ChatClientAgent"/> with
/// function invocation, per-service-call chat history persistence, and in-loop compaction.
/// </summary>
/// <remarks>
/// <para>
/// <see cref="HarnessAgent"/> assembles the following pipeline from a caller-supplied <see cref="IChatClient"/>:
/// <list type="number">
/// <item><description><see cref="FunctionInvokingChatClient"/> — automatic function/tool invocation.</description></item>
/// <item><description><see cref="PerServiceCallChatHistoryPersistingChatClient"/> — persists chat history after every individual service call within a function-invocation loop.</description></item>
/// <item><description><see cref="AIContextProviderChatClient"/> with a <see cref="CompactionProvider"/> — applies context-window compaction before each call so long function-invocation loops do not overflow the context window.</description></item>
/// </list>
/// </para>
/// <para>
/// The underlying <see cref="ChatClientAgent"/> is configured with
/// <see cref="ChatClientAgentOptions.UseProvidedChatClientAsIs"/> and
/// <see cref="ChatClientAgentOptions.RequirePerServiceCallChatHistoryPersistence"/> set to <see langword="true"/>
/// to match the manually-assembled pipeline.
/// </para>
/// <para>
/// When no <see cref="HarnessAgentOptions.ChatHistoryProvider"/> is supplied, the agent defaults to an
/// <see cref="InMemoryChatHistoryProvider"/> whose chat reducer applies the same compaction strategy,
/// keeping in-memory history from growing unboundedly across sessions.
/// </para>
/// </remarks>
[Experimental(DiagnosticIds.Experiments.AgentsAIExperiments)]
public sealed class HarnessAgent : DelegatingAIAgent
{
/// <summary>
/// The built-in default system instructions used when <see cref="ChatOptions.Instructions"/> is not set.
/// </summary>
public const string DefaultInstructions =
"""
You are a helpful AI assistant that uses tools to complete tasks.
## General guidelines
- Think through the task before acting. Break complex work into clear steps.
- Use the tools available to you to gather information, perform actions, and verify results.
- Explain your reasoning between tool calls so the user can follow your progress.
- If a tool call fails or returns unexpected results, adapt your approach rather than repeating the same call.
- When you have completed the task, present a clear and concise summary of what you did and what you found.
""";
/// <summary>
/// Initializes a new instance of the <see cref="HarnessAgent"/> class.
/// </summary>
/// <param name="chatClient">
/// The <see cref="IChatClient"/> that provides access to the underlying AI model.
/// The agent wraps this client in a function-invocation, per-service-call persistence,
/// and compaction pipeline automatically.
/// </param>
/// <param name="maxContextWindowTokens">
/// The maximum number of tokens the model's context window supports (e.g., 1,050,000 for gpt-5.4).
/// Used to configure the compaction strategy.
/// </param>
/// <param name="maxOutputTokens">
/// The maximum number of output tokens the model can generate per response (e.g., 128,000 for gpt-5.4).
/// Used to configure the compaction strategy and to limit the model's output.
/// </param>
/// <param name="options">
/// Optional configuration options for the agent, including instructions override, tools,
/// additional context providers, and chat history provider.
/// When <see langword="null"/>, the agent uses built-in default settings.
/// </param>
/// <exception cref="System.ArgumentNullException">
/// <paramref name="chatClient"/> is <see langword="null"/>.
/// </exception>
/// <exception cref="System.ArgumentOutOfRangeException">
/// <paramref name="maxContextWindowTokens"/> is not positive, or
/// <paramref name="maxOutputTokens"/> is negative or greater than or equal to <paramref name="maxContextWindowTokens"/>.
/// </exception>
public HarnessAgent(IChatClient chatClient, int maxContextWindowTokens, int maxOutputTokens, HarnessAgentOptions? options = null)
: base(BuildInnerAgent(
Throw.IfNull(chatClient),
maxContextWindowTokens,
maxOutputTokens,
options))
{
}
private static ChatClientAgent BuildInnerAgent(IChatClient chatClient, int maxContextWindowTokens, int maxOutputTokens, HarnessAgentOptions? options)
{
var compactionStrategy = new ContextWindowCompactionStrategy(
maxContextWindowTokens: maxContextWindowTokens,
maxOutputTokens: maxOutputTokens);
ChatHistoryProvider chatHistoryProvider = options?.ChatHistoryProvider
?? new InMemoryChatHistoryProvider(new InMemoryChatHistoryProviderOptions
{
ChatReducer = compactionStrategy.AsChatReducer(),
});
string instructions = options?.ChatOptions?.Instructions ?? DefaultInstructions;
ChatOptions chatOptions = BuildChatOptions(options?.ChatOptions, instructions, maxOutputTokens);
var compactionProvider = new CompactionProvider(compactionStrategy);
return chatClient
.AsBuilder()
.UseFunctionInvocation()
.UsePerServiceCallChatHistoryPersistence()
.UseAIContextProviders(compactionProvider)
.BuildAIAgent(new ChatClientAgentOptions
{
Id = options?.Id,
Name = options?.Name,
Description = options?.Description,
ChatOptions = chatOptions,
ChatHistoryProvider = chatHistoryProvider,
AIContextProviders = options?.AIContextProviders,
UseProvidedChatClientAsIs = true,
RequirePerServiceCallChatHistoryPersistence = true,
});
}
private static ChatOptions BuildChatOptions(ChatOptions? source, string instructions, int maxOutputTokens)
{
ChatOptions result = source?.Clone() ?? new ChatOptions();
result.Instructions = instructions;
result.MaxOutputTokens ??= maxOutputTokens;
return result;
}
}
@@ -1,64 +0,0 @@
// Copyright (c) Microsoft. All rights reserved.
using System.Collections.Generic;
using System.Diagnostics.CodeAnalysis;
using Microsoft.Extensions.AI;
using Microsoft.Shared.DiagnosticIds;
namespace Microsoft.Agents.AI;
/// <summary>
/// Represents configuration options for a <see cref="HarnessAgent"/>.
/// </summary>
[Experimental(DiagnosticIds.Experiments.AgentsAIExperiments)]
public sealed class HarnessAgentOptions
{
/// <summary>
/// Gets or sets the agent id.
/// </summary>
public string? Id { get; set; }
/// <summary>
/// Gets or sets the agent name.
/// </summary>
public string? Name { get; set; }
/// <summary>
/// Gets or sets the agent description.
/// </summary>
public string? Description { get; set; }
/// <summary>
/// Gets or sets additional chat options such as tools for the agent to use.
/// </summary>
/// <remarks>
/// <para>
/// Use <see cref="ChatOptions.Tools"/> to supply additional tools the agent can invoke.
/// </para>
/// <para>
/// Use <see cref="ChatOptions.Instructions"/> to override the <see cref="HarnessAgent"/>'s built-in
/// default instructions. When <see cref="ChatOptions.Instructions"/> is <see langword="null"/> or not set,
/// the default instructions are used.
/// </para>
/// </remarks>
public ChatOptions? ChatOptions { get; set; }
/// <summary>
/// Gets or sets the <see cref="ChatHistoryProvider"/> to use for storing chat history.
/// </summary>
/// <remarks>
/// When <see langword="null"/>, the agent defaults to an <see cref="InMemoryChatHistoryProvider"/>
/// configured with a compaction-based chat reducer derived from the <c>maxContextWindowTokens</c>
/// and <c>maxOutputTokens</c> constructor parameters of <see cref="HarnessAgent"/>.
/// </remarks>
public ChatHistoryProvider? ChatHistoryProvider { get; set; }
/// <summary>
/// Gets or sets additional <see cref="AIContextProvider"/> instances to include in the agent pipeline.
/// </summary>
/// <remarks>
/// These providers are passed to the underlying <see cref="ChatClientAgent"/> via
/// <see cref="ChatClientAgentOptions.AIContextProviders"/>.
/// </remarks>
public IEnumerable<AIContextProvider>? AIContextProviders { get; set; }
}
@@ -1,28 +0,0 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<IsReleaseCandidate>false</IsReleaseCandidate>
<InjectSharedThrow>true</InjectSharedThrow>
<InjectSharedDiagnosticIds>true</InjectSharedDiagnosticIds>
<InjectExperimentalAttributeOnLegacy>true</InjectExperimentalAttributeOnLegacy>
<InjectIsExternalInitOnLegacy>true</InjectIsExternalInitOnLegacy>
<InjectTrimAttributesOnLegacy>true</InjectTrimAttributesOnLegacy>
</PropertyGroup>
<Import Project="$(RepoRoot)/dotnet/nuget/nuget-package.props" />
<ItemGroup>
<ProjectReference Include="..\Microsoft.Agents.AI\Microsoft.Agents.AI.csproj" />
</ItemGroup>
<PropertyGroup>
<!-- NuGet Package Settings -->
<Title>Microsoft Agent Framework Harness</Title>
<Description>Provides the HarnessAgent, a pre-configured AI agent that can be used for long running tasks.</Description>
</PropertyGroup>
<ItemGroup>
<InternalsVisibleTo Include="Microsoft.Agents.AI.Harness.UnitTests" />
</ItemGroup>
</Project>
@@ -32,8 +32,38 @@ internal static class AIAgentsAbstractionsExtensions
return message;
}
public static List<ChatMessage> CopyWithAssistantToUserForOtherParticipants(
this IEnumerable<ChatMessage> messages,
string targetAgentName)
=> messages.Select(m => m.ChatAssistantToUserIfNotFromNamed(targetAgentName, out _, false)).ToList();
/// <summary>
/// Iterates through <paramref name="messages"/> looking for <see cref="ChatRole.Assistant"/> messages and swapping
/// any that have a different <see cref="ChatMessage.AuthorName"/> from <paramref name="targetAgentName"/> to
/// <see cref="ChatRole.User"/>.
/// </summary>
public static List<ChatMessage>? ChangeAssistantToUserForOtherParticipants(this IEnumerable<ChatMessage> messages, string targetAgentName)
{
List<ChatMessage>? roleChanged = null;
foreach (var m in messages)
{
m.ChatAssistantToUserIfNotFromNamed(targetAgentName, out bool changed);
if (changed)
{
(roleChanged ??= []).Add(m);
}
}
return roleChanged;
}
/// <summary>
/// Undoes changes made by <see cref="ChangeAssistantToUserForOtherParticipants"/> when passed the list of changes
/// made by that method.
/// </summary>
public static void ResetUserToAssistantForChangedRoles(this List<ChatMessage>? roleChanged)
{
if (roleChanged is not null)
{
foreach (var m in roleChanged)
{
m.Role = ChatRole.Assistant;
}
}
}
}
@@ -28,17 +28,6 @@ public static class WorkflowEvaluationExtensions
/// Use <see cref="ConversationSplitters.LastTurn"/>, <see cref="ConversationSplitters.Full"/>,
/// or a custom <see cref="IConversationSplitter"/> implementation.
/// </param>
/// <param name="expectedOutput">
/// Optional ground-truth/expected output for the workflow's overall final answer.
/// When provided, it is stamped onto the overall <see cref="EvalItem.ExpectedOutput"/>
/// so reference-based evaluators (for example, similarity) can compare the
/// workflow's response against a golden answer. Ground truth is only applied
/// to the overall item; per-agent items are intentionally left without an
/// expected output, since ground truth is defined against the final response.
/// When using a reference-based evaluator that requires ground truth, set
/// <paramref name="includePerAgent"/> to <see langword="false"/> to avoid
/// invoking the evaluator on per-agent items that have no expected output.
/// </param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>Evaluation results with optional per-agent sub-results.</returns>
public static async Task<AgentEvaluationResults> EvaluateAsync(
@@ -48,7 +37,6 @@ public static class WorkflowEvaluationExtensions
bool includePerAgent = true,
string evalName = "Workflow Eval",
IConversationSplitter? splitter = null,
string? expectedOutput = null,
CancellationToken cancellationToken = default)
{
var events = run.OutgoingEvents.ToList();
@@ -60,26 +48,28 @@ public static class WorkflowEvaluationExtensions
var overallItems = new List<EvalItem>();
if (includeOverall)
{
var overallItem = BuildOverallItem(events, splitter, expectedOutput);
if (overallItem is not null)
var finalResponse = events.OfType<AgentResponseEvent>().LastOrDefault();
if (finalResponse is not null)
{
overallItems.Add(overallItem);
}
else
{
// The caller asked for an overall evaluation but we couldn't find a final
// response to score — almost always because the workflow's agents weren't
// built with EmitAgentResponseEvents enabled (so no AgentResponseEvent was
// emitted) and no terminal ExecutorCompletedEvent carried an AgentResponse
// / ChatMessage / string payload. Fail loudly instead of silently returning
// 0/0 (or skipping evaluation against a supplied expectedOutput).
throw new InvalidOperationException(
"Cannot evaluate the overall workflow output: no AgentResponseEvent or " +
"ExecutorCompletedEvent with an AgentResponse/ChatMessage/string payload " +
"was found in the run. Bind agents with " +
"AIAgentHostOptions { EmitAgentResponseEvents = true } " +
"(for example via agent.BindAsExecutor(new AIAgentHostOptions { EmitAgentResponseEvents = true })) " +
"so the workflow surfaces the final agent response, or set 'includeOverall: false'.");
var firstInvoked = events.OfType<ExecutorInvokedEvent>().FirstOrDefault();
var query = firstInvoked?.Data switch
{
ChatMessage cm => cm.Text ?? string.Empty,
IReadOnlyList<ChatMessage> msgs => msgs.LastOrDefault(m => m.Role == ChatRole.User)?.Text ?? string.Empty,
string s => s,
_ => firstInvoked?.Data?.ToString() ?? string.Empty,
};
var conversation = new List<ChatMessage>
{
new(ChatRole.User, query),
};
conversation.AddRange(finalResponse.Response.Messages);
overallItems.Add(new EvalItem(query, finalResponse.Response.Text, conversation)
{
Splitter = splitter,
});
}
}
@@ -107,86 +97,6 @@ public static class WorkflowEvaluationExtensions
return overallResult;
}
internal static EvalItem? BuildOverallItem(
IReadOnlyList<WorkflowEvent> events,
IConversationSplitter? splitter,
string? expectedOutput)
{
var firstInvoked = events.OfType<ExecutorInvokedEvent>().FirstOrDefault();
var query = firstInvoked?.Data switch
{
ChatMessage cm => cm.Text ?? string.Empty,
IReadOnlyList<ChatMessage> msgs => msgs.LastOrDefault(m => m.Role == ChatRole.User)?.Text ?? string.Empty,
string s => s,
_ => firstInvoked?.Data?.ToString() ?? string.Empty,
};
var conversation = new List<ChatMessage>
{
new(ChatRole.User, query),
};
// Prefer AgentResponseEvent (only emitted when AIAgentHostOptions.EmitAgentResponseEvents
// is enabled). Otherwise fall back to the last ExecutorCompletedEvent that carries an
// AgentResponse / ChatMessage / string payload — these are always emitted by the runtime.
var finalResponse = events.OfType<AgentResponseEvent>().LastOrDefault();
string responseText;
if (finalResponse is not null)
{
responseText = finalResponse.Response.Text;
conversation.AddRange(finalResponse.Response.Messages);
}
else
{
ExecutorCompletedEvent? finalCompleted = null;
for (int i = events.Count - 1; i >= 0; i--)
{
if (events[i] is ExecutorCompletedEvent completed
&& !IsInternalExecutor(completed.ExecutorId)
&& completed.Data is AgentResponse or ChatMessage or string)
{
finalCompleted = completed;
break;
}
}
if (finalCompleted is null)
{
return null;
}
switch (finalCompleted.Data)
{
case AgentResponse ar:
responseText = ar.Text;
conversation.AddRange(ar.Messages);
break;
case ChatMessage cm:
responseText = cm.Text ?? string.Empty;
conversation.Add(cm);
break;
case string s:
responseText = s;
conversation.Add(new ChatMessage(ChatRole.Assistant, s));
break;
default:
// Unreachable — the for-loop above already constrains Data to one of the
// three handled types. Throw if the contract drifts so the bug is visible
// instead of silently dropping the overall item.
throw new InvalidOperationException(
"BuildOverallItem: unexpected ExecutorCompletedEvent.Data type " +
$"'{finalCompleted.Data?.GetType().FullName ?? "null"}'. Expected " +
$"{nameof(AgentResponse)}, {nameof(ChatMessage)}, or string.");
}
}
return new EvalItem(query, responseText, conversation)
{
Splitter = splitter,
ExpectedOutput = expectedOutput,
};
}
internal static Dictionary<string, List<EvalItem>> ExtractAgentData(
List<WorkflowEvent> events,
IConversationSplitter? splitter)
@@ -235,10 +235,11 @@ internal sealed class HandoffAgentExecutor :
// This will not filter out tool responses and approval responses that are part of this agent's turn, which is
// the expected behavior since those are part of the agent's reasoning process.
HandoffMessagesFilter handoffMessagesFilter = new(this._options.ToolCallFilteringBehavior);
List<ChatMessage> messagesForAgent = (state.IncomingState.RequestedHandoffTargetAgentId is not null
IEnumerable<ChatMessage> messagesForAgent = state.IncomingState.RequestedHandoffTargetAgentId is not null
? handoffMessagesFilter.FilterMessages(incomingMessages)
: incomingMessages)
.CopyWithAssistantToUserForOtherParticipants(this._agent.Name ?? this._agent.Id);
: incomingMessages;
List<ChatMessage>? roleChanges = messagesForAgent.ChangeAssistantToUserForOtherParticipants(this._agent.Name ?? this._agent.Id);
bool emitUpdateEvents = state.IncomingState!.ShouldEmitStreamingEvents(this._options.EmitAgentResponseUpdateEvents);
AgentInvocationResult result = await this.InvokeAgentAsync(messagesForAgent, context, emitUpdateEvents, cancellationToken)
@@ -249,6 +250,8 @@ internal sealed class HandoffAgentExecutor :
throw new InvalidOperationException("Cannot request a handoff while holding pending requests.");
}
roleChanges.ResetUserToAssistantForChangedRoles();
int newConversationBookmark = state.ConversationBookmark;
await this._sharedStateRef.InvokeWithStateAsync(
(sharedState, ctx, ct) =>
@@ -36,13 +36,9 @@ public sealed class SwitchBuilder
Throw.IfNull(executors);
HashSet<int> indicies = [];
int executorIndex = 0;
foreach (ExecutorBinding executor in executors)
{
// Explicit name: null element inside the collection argument.
Throw.IfNull(executor, $"{nameof(executors)}[{executorIndex++}]");
if (!this._executorIndicies.TryGetValue(executor.Id, out int index))
{
index = this._executors.Count;
@@ -68,13 +64,8 @@ public sealed class SwitchBuilder
{
Throw.IfNull(executors);
int executorIndex = 0;
foreach (ExecutorBinding executor in executors)
{
// Explicit name: null element inside the collection argument.
Throw.IfNull(executor, $"{nameof(executors)}[{executorIndex++}]");
if (!this._executorIndicies.TryGetValue(executor.Id, out int index))
{
index = this._executors.Count;
@@ -25,11 +25,7 @@ public static class WorkflowBuilderExtensions
/// <param name="target">The target executor to which messages will be forwarded.</param>
/// <returns>The updated <see cref="WorkflowBuilder"/> instance.</returns>
public static WorkflowBuilder ForwardMessage<TMessage>(this WorkflowBuilder builder, ExecutorBinding source, ExecutorBinding target)
{
Throw.IfNull(target, nameof(target));
return builder.ForwardMessage<TMessage>(source, [target], condition: null);
}
=> builder.ForwardMessage<TMessage>(source, [target], condition: null);
/// <summary>
/// Adds edges to the workflow that forward messages of the specified type from the source executor to
@@ -56,8 +52,6 @@ public static class WorkflowBuilderExtensions
/// <returns>The updated <see cref="WorkflowBuilder"/> instance.</returns>
public static WorkflowBuilder ForwardMessage<TMessage>(this WorkflowBuilder builder, ExecutorBinding source, IEnumerable<ExecutorBinding> targets, Func<TMessage, bool>? condition = null)
{
Throw.IfNull(builder);
Throw.IfNull(source);
Throw.IfNull(targets);
Func<object?, bool> predicate = WorkflowBuilder.CreateConditionFunc<TMessage>(IsAllowedTypeAndMatchingCondition)!;
@@ -68,16 +62,14 @@ public static class WorkflowBuilderExtensions
if (targets is ICollection<ExecutorBinding> { Count: 1 })
#endif
{
return builder.AddEdge(source, Throw.IfNull(targets.First(), nameof(targets)), predicate);
return builder.AddEdge(source, targets.First(), predicate);
}
return builder.AddSwitch(source, (switch_) => switch_.AddCase(predicate, targets.Select(ValidateTarget)));
return builder.AddSwitch(source, (switch_) => switch_.AddCase(predicate, targets));
// The reason we can check for "not null" here is that CreateConditionFunc<T> will do the correct unwrapping
// logic for PortableValues.
bool IsAllowedTypeAndMatchingCondition(TMessage? message) => message != null && (condition == null || condition(message));
ExecutorBinding ValidateTarget(ExecutorBinding target) => Throw.IfNull(target, nameof(targets));
}
/// <summary>
@@ -89,11 +81,7 @@ public static class WorkflowBuilderExtensions
/// <param name="target">The target executor to which messages, except those of type <typeparamref name="TMessage"/>, will be forwarded.</param>
/// <returns>The updated <see cref="WorkflowBuilder"/> instance with the added edges.</returns>
public static WorkflowBuilder ForwardExcept<TMessage>(this WorkflowBuilder builder, ExecutorBinding source, ExecutorBinding target)
{
Throw.IfNull(target, nameof(target));
return builder.ForwardExcept<TMessage>(source, [target]);
}
=> builder.ForwardExcept<TMessage>(source, [target]);
/// <summary>
/// Adds edges from the specified source to the provided executors, excluding messages of a specified type.
@@ -105,8 +93,6 @@ public static class WorkflowBuilderExtensions
/// <returns>The updated <see cref="WorkflowBuilder"/> instance with the added edges.</returns>
public static WorkflowBuilder ForwardExcept<TMessage>(this WorkflowBuilder builder, ExecutorBinding source, IEnumerable<ExecutorBinding> targets)
{
Throw.IfNull(builder);
Throw.IfNull(source);
Throw.IfNull(targets);
Func<object?, bool> predicate = WorkflowBuilder.CreateConditionFunc<TMessage>((Func<object?, bool>)IsAllowedType)!;
@@ -117,16 +103,14 @@ public static class WorkflowBuilderExtensions
if (targets is ICollection<ExecutorBinding> { Count: 1 })
#endif
{
return builder.AddEdge(source, Throw.IfNull(targets.First(), nameof(targets)), predicate);
return builder.AddEdge(source, targets.First(), predicate);
}
return builder.AddSwitch(source, (switch_) => switch_.AddCase(predicate, targets.Select(ValidateTarget)));
return builder.AddSwitch(source, (switch_) => switch_.AddCase(predicate, targets));
// The reason we can check for "null" here is that CreateConditionFunc<T> will do the correct unwrapping
// logic for PortableValues.
static bool IsAllowedType(object? message) => message is null;
ExecutorBinding ValidateTarget(ExecutorBinding target) => Throw.IfNull(target, nameof(targets));
}
/// <summary>
@@ -145,7 +129,6 @@ public static class WorkflowBuilderExtensions
{
Throw.IfNull(builder);
Throw.IfNull(source);
Throw.IfNull(executors);
HashSet<string> seenExecutors = [source.Id];
@@ -122,7 +122,6 @@ public sealed class FileSystemAgentFileStore : AgentFileStore
}
var files = Directory.GetFiles(fullDir)
.Where(f => (File.GetAttributes(f) & FileAttributes.ReparsePoint) == 0)
.Select(Path.GetFileName)
.Where(name => name is not null)
.ToList();
@@ -158,12 +157,6 @@ public sealed class FileSystemAgentFileStore : AgentFileStore
foreach (string filePath in Directory.GetFiles(fullDir))
{
// Skip files that are symlinks/reparse points to prevent reading outside the root.
if ((File.GetAttributes(filePath) & FileAttributes.ReparsePoint) != 0)
{
continue;
}
string? fileName = Path.GetFileName(filePath);
if (fileName is null)
{
@@ -238,7 +231,7 @@ public sealed class FileSystemAgentFileStore : AgentFileStore
/// <summary>
/// Resolves a relative file path to a safe absolute path under the root directory.
/// Rejects paths that would escape the root via traversal, rooted paths, or symbolic links.
/// Rejects paths that would escape the root via traversal or rooted paths.
/// </summary>
private string ResolveSafePath(string relativePath)
{
@@ -257,55 +250,9 @@ public sealed class FileSystemAgentFileStore : AgentFileStore
nameof(relativePath));
}
// Reject symlinks/reparse points in any path segment to prevent escaping the root.
ThrowIfContainsSymlink(fullPath, this._rootPath);
return fullPath;
}
/// <summary>
/// Checks each path segment between the trusted root and the resolved path for symbolic links
/// or reparse points. Throws <see cref="ArgumentException"/> if any segment is a symlink.
/// Stops checking at the first segment that does not exist on disk (for write scenarios).
/// Uses <see cref="File.GetAttributes(string)"/> directly so that dangling symlinks (whose targets
/// do not exist) are still detected via their <see cref="FileAttributes.ReparsePoint"/> flag.
/// </summary>
private static void ThrowIfContainsSymlink(string fullPath, string rootPath)
{
string rootTrimmed = rootPath.TrimEnd(Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar);
string relative = fullPath.Substring(rootTrimmed.Length);
string[] segments = relative.Split(
[Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar],
StringSplitOptions.RemoveEmptyEntries);
string current = rootTrimmed;
foreach (string segment in segments)
{
current = Path.Combine(current, segment);
FileAttributes attributes;
try
{
attributes = File.GetAttributes(current);
}
catch (FileNotFoundException)
{
// Segment does not exist on disk (write scenario); stop checking.
break;
}
catch (DirectoryNotFoundException)
{
break;
}
if ((attributes & FileAttributes.ReparsePoint) != 0)
{
throw new ArgumentException(
"Invalid path: the resolved path contains a symbolic link or reparse point.");
}
}
}
/// <summary>
/// Resolves a relative directory path to a safe absolute path under the root directory.
/// An empty string resolves to the root directory itself.
@@ -3,12 +3,10 @@
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Diagnostics.CodeAnalysis;
using System.Runtime.CompilerServices;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.AI;
using Microsoft.Shared.DiagnosticIds;
namespace Microsoft.Agents.AI;
@@ -34,13 +32,6 @@ public sealed class OpenTelemetryAgent : DelegatingAIAgent, IDisposable
private readonly OpenTelemetryChatClient _otelClient;
/// <summary>The provider name extracted from <see cref="AIAgentMetadata"/>.</summary>
private readonly string? _providerName;
/// <summary>The resolved source name for telemetry. Always non-empty; defaults to <see cref="OpenTelemetryConsts.DefaultSourceName"/>.</summary>
private readonly string _sourceName;
/// <summary>
/// Indicates whether the underlying <see cref="IChatClient"/> of a <see cref="ChatClientAgent"/> inner agent
/// should be automatically wrapped with <see cref="OpenTelemetryChatClient"/> on each invocation.
/// </summary>
private readonly bool _autoWireChatClient;
/// <summary>Initializes a new instance of the <see cref="OpenTelemetryAgent"/> class.</summary>
/// <param name="innerAgent">The underlying <see cref="AIAgent"/> to be augmented with telemetry capabilities.</param>
@@ -53,44 +44,13 @@ public sealed class OpenTelemetryAgent : DelegatingAIAgent, IDisposable
/// The constructor automatically extracts provider metadata from the inner agent and configures
/// telemetry collection according to OpenTelemetry semantic conventions for AI systems.
/// </remarks>
public OpenTelemetryAgent(AIAgent innerAgent, string? sourceName = null)
#pragma warning disable MAAI001 // Auto-wiring is the new default; the experimental opt-out lives on the 3-arg overload.
: this(innerAgent, sourceName, autoWireChatClient: true)
#pragma warning restore MAAI001
{
}
/// <summary>Initializes a new instance of the <see cref="OpenTelemetryAgent"/> class.</summary>
/// <param name="innerAgent">The underlying <see cref="AIAgent"/> to be augmented with telemetry capabilities.</param>
/// <param name="sourceName">
/// An optional source name that will be used to identify telemetry data from this agent.
/// If not provided, a default source name will be used for telemetry identification.
/// </param>
/// <param name="autoWireChatClient">
/// When <see langword="true"/> and the inner agent is a <see cref="ChatClientAgent"/>, the underlying
/// <see cref="IChatClient"/> is automatically wrapped with <see cref="OpenTelemetryChatClient"/> for each invocation
/// so that chat-level telemetry flows alongside agent-level telemetry. If the underlying chat client is already
/// instrumented, no additional wrapping is applied. Set to <see langword="false"/> to opt-out of this behavior.
/// </param>
/// <exception cref="ArgumentNullException"><paramref name="innerAgent"/> is <see langword="null"/>.</exception>
/// <remarks>
/// The constructor automatically extracts provider metadata from the inner agent and configures
/// telemetry collection according to OpenTelemetry semantic conventions for AI systems.
/// </remarks>
[Experimental(DiagnosticIds.Experiments.AgentsAIExperiments)]
public OpenTelemetryAgent(AIAgent innerAgent, string? sourceName, bool autoWireChatClient) : base(innerAgent)
public OpenTelemetryAgent(AIAgent innerAgent, string? sourceName = null) : base(innerAgent)
{
this._providerName = innerAgent.GetService<AIAgentMetadata>()?.ProviderName;
// Resolve once so the outer OpenTelemetryChatClient and the auto-wired inner
// OpenTelemetryChatClient always emit spans under the same ActivitySource, even when
// the caller passes "" or whitespace (which neither client should treat as a real source).
this._sourceName = string.IsNullOrWhiteSpace(sourceName) ? OpenTelemetryConsts.DefaultSourceName : sourceName!;
this._autoWireChatClient = autoWireChatClient;
this._otelClient = new OpenTelemetryChatClient(
new ForwardingChatClient(this),
sourceName: this._sourceName);
sourceName: string.IsNullOrEmpty(sourceName) ? OpenTelemetryConsts.DefaultSourceName : sourceName!);
}
/// <inheritdoc/>
@@ -203,85 +163,6 @@ public sealed class OpenTelemetryAgent : DelegatingAIAgent, IDisposable
public Activity? CurrentActivity { get; }
}
/// <summary>
/// If auto-wiring is enabled and the inner agent is a <see cref="ChatClientAgent"/> whose underlying
/// <see cref="IChatClient"/> is not already instrumented with <see cref="OpenTelemetryChatClient"/>, returns a
/// new <see cref="ChatClientAgentRunOptions"/> with a <see cref="ChatClientAgentRunOptions.ChatClientFactory"/>
/// that wraps the chat client with <see cref="OpenTelemetryChatClient"/>. When <paramref name="options"/> is a
/// plain <see cref="AgentRunOptions"/> (the base type, not <see cref="ChatClientAgentRunOptions"/>), the base
/// properties are copied onto the new <see cref="ChatClientAgentRunOptions"/> so high-level callers that pass
/// the abstract <see cref="AgentRunOptions"/> still benefit from auto-wiring and propagate their settings to
/// the inner agent. Otherwise, returns <paramref name="options"/> unchanged.
/// </summary>
private AgentRunOptions? GetRunOptionsWithChatClientWiring(AgentRunOptions? options)
{
if (!this._autoWireChatClient)
{
return options;
}
// The auto-wiring only applies when a ChatClientAgent is reachable from the inner agent. Otherwise, no-op.
// Use GetService rather than a type check so wrapping agents that expose a nested ChatClientAgent are supported.
var chatClientAgent = this.InnerAgent.GetService<ChatClientAgent>();
if (chatClientAgent is null)
{
return options;
}
// Respect ChatClientAgentOptions.UseProvidedChatClientAsIs: don't decorate the chat client when the user opted out.
if (chatClientAgent.GetService<ChatClientAgentOptions>()?.UseProvidedChatClientAsIs is true)
{
return options;
}
// Capture the underlying IChatClient and check whether it is already instrumented.
var chatClient = chatClientAgent.GetService<IChatClient>();
if (chatClient is null || chatClient.GetService(typeof(OpenTelemetryChatClient)) is not null)
{
return options;
}
string sourceName = this._sourceName;
static IChatClient WrapIfNeeded(IChatClient cc, string sourceName) =>
cc.GetService(typeof(OpenTelemetryChatClient)) is not null
? cc
: cc.AsBuilder().UseOpenTelemetry(sourceName: sourceName).Build();
if (options is ChatClientAgentRunOptions ccOptions)
{
// Don't mutate the caller's options; clone and chain any caller-provided factory.
// If the user factory already returns an OpenTelemetry-instrumented client, don't double-wrap.
var clone = (ChatClientAgentRunOptions)ccOptions.Clone();
var userFactory = clone.ChatClientFactory;
clone.ChatClientFactory = cc => WrapIfNeeded(userFactory is null ? cc : userFactory(cc), sourceName);
return clone;
}
// For a plain AgentRunOptions (or null), create a ChatClientAgentRunOptions and preserve
// any base AgentRunOptions properties from the caller so they reach the inner agent.
var newOptions = new ChatClientAgentRunOptions
{
ChatClientFactory = cc => WrapIfNeeded(cc, sourceName),
};
if (options is not null)
{
CopyBaseAgentRunOptions(options, newOptions);
}
return newOptions;
}
#pragma warning disable MEAI001 // ContinuationToken is experimental; copy it through to preserve caller-provided value.
private static void CopyBaseAgentRunOptions(AgentRunOptions source, AgentRunOptions target)
{
target.ContinuationToken = source.ContinuationToken;
target.AllowBackgroundResponses = source.AllowBackgroundResponses;
target.AdditionalProperties = source.AdditionalProperties?.Clone();
target.ResponseFormat = source.ResponseFormat;
}
#pragma warning restore MEAI001
/// <summary>The stub <see cref="IChatClient"/> used to delegate from the <see cref="OpenTelemetryChatClient"/> into the inner <see cref="AIAgent"/>.</summary>
/// <param name="parentAgent"></param>
private sealed class ForwardingChatClient(OpenTelemetryAgent parentAgent) : IChatClient
@@ -294,11 +175,8 @@ public sealed class OpenTelemetryAgent : DelegatingAIAgent, IDisposable
// Update the current activity to reflect the agent invocation.
parentAgent.UpdateCurrentActivity(fo?.CurrentActivity);
// If enabled, wire the underlying chat client with OpenTelemetryChatClient via ChatClientFactory.
var runOptions = parentAgent.GetRunOptionsWithChatClientWiring(fo?.Options);
// Invoke the inner agent.
var response = await parentAgent.InnerAgent.RunAsync(messages, fo?.Session, runOptions, cancellationToken).ConfigureAwait(false);
var response = await parentAgent.InnerAgent.RunAsync(messages, fo?.Session, fo?.Options, cancellationToken).ConfigureAwait(false);
// Wrap the response in a ChatResponse so we can pass it back through OpenTelemetryChatClient.
return response.AsChatResponse();
@@ -312,11 +190,8 @@ public sealed class OpenTelemetryAgent : DelegatingAIAgent, IDisposable
// Update the current activity to reflect the agent invocation.
parentAgent.UpdateCurrentActivity(fo?.CurrentActivity);
// If enabled, wire the underlying chat client with OpenTelemetryChatClient via ChatClientFactory.
var runOptions = parentAgent.GetRunOptionsWithChatClientWiring(fo?.Options);
// Invoke the inner agent.
await foreach (var update in parentAgent.InnerAgent.RunStreamingAsync(messages, fo?.Session, runOptions, cancellationToken).ConfigureAwait(false))
await foreach (var update in parentAgent.InnerAgent.RunStreamingAsync(messages, fo?.Session, fo?.Options, cancellationToken).ConfigureAwait(false))
{
// Wrap the response updates in ChatResponseUpdates so we can pass them back through OpenTelemetryChatClient.
yield return update.AsChatResponseUpdate();
@@ -218,7 +218,7 @@ public class DevUIIntegrationTests
Assert.Contains(discoveryResponse.Entities, e => e.Name == "default-workflow" && e.Type == "workflow");
}
[Fact(Skip = "Flaky in merge_group; see https://github.com/microsoft/agent-framework/issues/5845")]
[Fact]
public async Task TestServerWithDevUI_ResolvesMixedAgentsAndWorkflows_AllRegistrationsAsync()
{
// Arrange
@@ -179,35 +179,6 @@ public sealed class FoundryEvalConverterTests
Assert.Null(payload.Context);
}
[Fact]
public void ConvertEvalItem_WithExpectedOutput_PopulatesGroundTruth()
{
// Arrange
var item = new EvalItem(query: "q", response: "r")
{
ExpectedOutput = "the golden answer",
};
// Act
var payload = FoundryEvalConverter.ConvertEvalItem(item);
// Assert
Assert.Equal("the golden answer", payload.GroundTruth);
}
[Fact]
public void ConvertEvalItem_WithoutExpectedOutput_OmitsGroundTruth()
{
// Arrange
var item = new EvalItem(query: "q", response: "r");
// Act
var payload = FoundryEvalConverter.ConvertEvalItem(item);
// Assert
Assert.Null(payload.GroundTruth);
}
// ---------------------------------------------------------------
// FoundryEvalConverter.BuildTestingCriteria tests
// ---------------------------------------------------------------
@@ -268,33 +239,6 @@ public sealed class FoundryEvalConverterTests
Assert.Equal("{{item.context}}", mapping["context"]);
}
[Fact]
public void BuildTestingCriteria_SimilarityEvaluator_IncludesGroundTruth()
{
// Act
var criteria = FoundryEvalConverter.BuildTestingCriteria(
["similarity"], "gpt-4o-mini", includeDataMapping: true);
// Assert
Assert.Single(criteria);
Assert.Equal("builtin.similarity", criteria[0].EvaluatorName);
var mapping = criteria[0].DataMapping;
Assert.NotNull(mapping);
Assert.True(mapping.ContainsKey("ground_truth"));
Assert.Equal("{{item.ground_truth}}", mapping["ground_truth"]);
}
[Fact]
public void BuildTestingCriteria_NonGroundTruthEvaluator_OmitsGroundTruth()
{
var criteria = FoundryEvalConverter.BuildTestingCriteria(
["relevance"], "gpt-4o-mini", includeDataMapping: true);
var mapping = criteria[0].DataMapping;
Assert.NotNull(mapping);
Assert.False(mapping.ContainsKey("ground_truth"));
}
[Fact]
public void BuildTestingCriteria_WithoutDataMapping_OmitsMappingField()
{
@@ -338,59 +282,6 @@ public sealed class FoundryEvalConverterTests
Assert.True(schema.Properties.ContainsKey("tool_definitions"));
}
[Fact]
public void BuildItemSchema_WithGroundTruth_IncludesGroundTruthProperty()
{
// Act
var schema = FoundryEvalConverter.BuildItemSchema(hasGroundTruth: true);
// Assert
Assert.True(schema.Properties.ContainsKey("ground_truth"));
Assert.Equal("string", schema.Properties["ground_truth"].Type);
}
[Fact]
public void BuildItemSchema_WithoutGroundTruth_OmitsGroundTruthProperty()
{
var schema = FoundryEvalConverter.BuildItemSchema();
Assert.False(schema.Properties.ContainsKey("ground_truth"));
}
// ---------------------------------------------------------------
// FoundryEvalConverter.FindMissingGroundTruthEvaluators tests
// ---------------------------------------------------------------
[Fact]
public void FindMissingGroundTruthEvaluators_NoGroundTruth_ReturnsSimilarity()
{
// Act
var missing = FoundryEvalConverter.FindMissingGroundTruthEvaluators(
["similarity", "relevance"], hasGroundTruth: false);
// Assert
Assert.Single(missing);
Assert.Equal("similarity", missing[0]);
}
[Fact]
public void FindMissingGroundTruthEvaluators_HasGroundTruth_ReturnsEmpty()
{
var missing = FoundryEvalConverter.FindMissingGroundTruthEvaluators(
["similarity"], hasGroundTruth: true);
Assert.Empty(missing);
}
[Fact]
public void FindMissingGroundTruthEvaluators_NoGroundTruthEvaluators_ReturnsEmpty()
{
var missing = FoundryEvalConverter.FindMissingGroundTruthEvaluators(
["relevance", "coherence"], hasGroundTruth: false);
Assert.Empty(missing);
}
// ---------------------------------------------------------------
// FoundryEvalConverter.ConvertMessage DataContent test
// ---------------------------------------------------------------
@@ -1,56 +0,0 @@
// Copyright (c) Microsoft. All rights reserved.
namespace Microsoft.Agents.AI.UnitTests;
public class HarnessAgentOptionsTests
{
/// <summary>
/// Verify that default property values are as expected.
/// </summary>
[Fact]
public void DefaultPropertyValues()
{
// Arrange & Act
var options = new HarnessAgentOptions();
// Assert
Assert.Null(options.Id);
Assert.Null(options.Name);
Assert.Null(options.Description);
Assert.Null(options.ChatOptions);
Assert.Null(options.ChatHistoryProvider);
Assert.Null(options.AIContextProviders);
}
/// <summary>
/// Verify that all properties can be set and retrieved.
/// </summary>
[Fact]
public void PropertiesCanBeSetAndRetrieved()
{
// Arrange
var chatHistoryProvider = new InMemoryChatHistoryProvider();
var contextProviders = new AIContextProvider[] { new TodoProvider() };
// Act
var options = new HarnessAgentOptions
{
Id = "test-id",
Name = "test-name",
Description = "test-description",
ChatOptions = new() { Temperature = 0.5f, Instructions = "custom instructions" },
ChatHistoryProvider = chatHistoryProvider,
AIContextProviders = contextProviders,
};
// Assert
Assert.Equal("test-id", options.Id);
Assert.Equal("test-name", options.Name);
Assert.Equal("test-description", options.Description);
Assert.NotNull(options.ChatOptions);
Assert.Equal(0.5f, options.ChatOptions!.Temperature);
Assert.Equal("custom instructions", options.ChatOptions.Instructions);
Assert.Same(chatHistoryProvider, options.ChatHistoryProvider);
Assert.Same(contextProviders, options.AIContextProviders);
}
}
@@ -1,516 +0,0 @@
// Copyright (c) Microsoft. All rights reserved.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.AI;
using Moq;
namespace Microsoft.Agents.AI.UnitTests;
public class HarnessAgentTests
{
private const int TestMaxContextWindowTokens = 100_000;
private const int TestMaxOutputTokens = 10_000;
#region Constructor Validation
/// <summary>
/// Verify that the constructor throws when chatClient is null.
/// </summary>
[Fact]
public void Constructor_ThrowsWhenChatClientIsNull()
{
// Act & Assert
Assert.Throws<ArgumentNullException>(() => new HarnessAgent(null!, TestMaxContextWindowTokens, TestMaxOutputTokens));
}
/// <summary>
/// Verify that the constructor throws when MaxContextWindowTokens is invalid (zero).
/// </summary>
[Fact]
public void Constructor_ThrowsWhenMaxContextWindowTokensIsZero()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
// Act & Assert
Assert.Throws<ArgumentOutOfRangeException>(() => new HarnessAgent(chatClient, 0, TestMaxOutputTokens));
}
/// <summary>
/// Verify that the constructor throws when MaxOutputTokens equals MaxContextWindowTokens.
/// </summary>
[Fact]
public void Constructor_ThrowsWhenMaxOutputTokensEqualsContextWindow()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
// Act & Assert
Assert.Throws<ArgumentOutOfRangeException>(() => new HarnessAgent(chatClient, 100_000, 100_000));
}
/// <summary>
/// Verify that the constructor succeeds when options is null.
/// </summary>
[Fact]
public void Constructor_SucceedsWhenOptionsIsNull()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
// Act
var agent = new HarnessAgent(chatClient, TestMaxContextWindowTokens, TestMaxOutputTokens);
// Assert
Assert.NotNull(agent);
}
#endregion
#region Agent Identity
/// <summary>
/// Verify that Name and Description are passed through to the inner agent.
/// </summary>
[Fact]
public void NameAndDescription_ArePassedThrough()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
// Act
var agent = new HarnessAgent(chatClient, TestMaxContextWindowTokens, TestMaxOutputTokens, new HarnessAgentOptions
{
Name = "TestAgent",
Description = "A test agent",
});
// Assert
Assert.Equal("TestAgent", agent.Name);
Assert.Equal("A test agent", agent.Description);
}
/// <summary>
/// Verify that Id is passed through to the inner agent.
/// </summary>
[Fact]
public void Id_IsPassedThrough()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
// Act
var agent = new HarnessAgent(chatClient, TestMaxContextWindowTokens, TestMaxOutputTokens, new HarnessAgentOptions
{
Id = "my-agent-id",
});
// Assert
Assert.Equal("my-agent-id", agent.Id);
}
#endregion
#region Instructions
/// <summary>
/// Verify that default instructions are used when none are provided.
/// </summary>
[Fact]
public void Instructions_DefaultsToBuiltInInstructions()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
// Act
var agent = new HarnessAgent(chatClient, TestMaxContextWindowTokens, TestMaxOutputTokens);
var innerAgent = agent.GetService<ChatClientAgent>();
// Assert
Assert.NotNull(innerAgent);
Assert.Equal(HarnessAgent.DefaultInstructions, innerAgent!.Instructions);
}
/// <summary>
/// Verify that default instructions are used when options is provided but ChatOptions.Instructions is null.
/// </summary>
[Fact]
public void Instructions_DefaultsWhenChatOptionsInstructionsIsNull()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
// Act
var agent = new HarnessAgent(chatClient, TestMaxContextWindowTokens, TestMaxOutputTokens, new HarnessAgentOptions
{
ChatOptions = new ChatOptions { Temperature = 0.5f },
});
var innerAgent = agent.GetService<ChatClientAgent>();
// Assert
Assert.NotNull(innerAgent);
Assert.Equal(HarnessAgent.DefaultInstructions, innerAgent!.Instructions);
}
/// <summary>
/// Verify that ChatOptions.Instructions overrides the defaults.
/// </summary>
[Fact]
public void Instructions_CanBeOverriddenViaChatOptions()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
// Act
var agent = new HarnessAgent(chatClient, TestMaxContextWindowTokens, TestMaxOutputTokens, new HarnessAgentOptions
{
ChatOptions = new ChatOptions { Instructions = "You are a custom assistant." },
});
var innerAgent = agent.GetService<ChatClientAgent>();
// Assert
Assert.NotNull(innerAgent);
Assert.Equal("You are a custom assistant.", innerAgent!.Instructions);
}
#endregion
#region ChatHistoryProvider
/// <summary>
/// Verify that the default ChatHistoryProvider is InMemoryChatHistoryProvider when none is specified.
/// </summary>
[Fact]
public void ChatHistoryProvider_DefaultsToInMemory()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
// Act
var agent = new HarnessAgent(chatClient, TestMaxContextWindowTokens, TestMaxOutputTokens);
var innerAgent = agent.GetService<ChatClientAgent>();
// Assert
Assert.NotNull(innerAgent);
Assert.IsType<InMemoryChatHistoryProvider>(innerAgent!.ChatHistoryProvider);
}
/// <summary>
/// Verify that a custom ChatHistoryProvider is used when provided.
/// </summary>
[Fact]
public void ChatHistoryProvider_UsesCustomProviderWhenSpecified()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
var customProvider = new InMemoryChatHistoryProvider();
// Act
var agent = new HarnessAgent(chatClient, TestMaxContextWindowTokens, TestMaxOutputTokens, new HarnessAgentOptions
{
ChatHistoryProvider = customProvider,
});
var innerAgent = agent.GetService<ChatClientAgent>();
// Assert
Assert.NotNull(innerAgent);
Assert.Same(customProvider, innerAgent!.ChatHistoryProvider);
}
#endregion
#region ChatClient Pipeline
/// <summary>
/// Verify that the inner agent's ChatClient includes FunctionInvokingChatClient in the pipeline.
/// </summary>
[Fact]
public void Pipeline_IncludesFunctionInvokingChatClient()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
// Act
var agent = new HarnessAgent(chatClient, TestMaxContextWindowTokens, TestMaxOutputTokens);
var innerAgent = agent.GetService<ChatClientAgent>();
// Assert
Assert.NotNull(innerAgent);
var ficc = innerAgent!.ChatClient.GetService<FunctionInvokingChatClient>();
Assert.NotNull(ficc);
}
/// <summary>
/// Verify that the inner agent's ChatClient pipeline includes more than just the raw chat client,
/// confirming that per-service-call persistence and other decorators have been applied.
/// </summary>
[Fact]
public void Pipeline_HasDecoratedChatClient()
{
// Arrange
var mockClient = new Mock<IChatClient>();
var rawClient = mockClient.Object;
// Act
var agent = new HarnessAgent(rawClient, TestMaxContextWindowTokens, TestMaxOutputTokens);
var innerAgent = agent.GetService<ChatClientAgent>();
// Assert — the pipeline wraps the raw client, so the outer client is not the same object.
Assert.NotNull(innerAgent);
Assert.NotSame(rawClient, innerAgent!.ChatClient);
}
#endregion
#region AIContextProviders
/// <summary>
/// Verify that additional AIContextProviders from options are passed to the inner ChatClientAgent,
/// not merged into the chat client builder pipeline.
/// </summary>
[Fact]
public void AIContextProviders_ArePassedToInnerAgent()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
var todoProvider = new TodoProvider();
// Act
var agent = new HarnessAgent(chatClient, TestMaxContextWindowTokens, TestMaxOutputTokens, new HarnessAgentOptions
{
AIContextProviders = [todoProvider],
});
var innerAgent = agent.GetService<ChatClientAgent>();
// Assert — the TodoProvider should appear in the inner agent's AIContextProviders.
Assert.NotNull(innerAgent);
Assert.NotNull(innerAgent!.AIContextProviders);
Assert.Contains(todoProvider, innerAgent.AIContextProviders!);
}
/// <summary>
/// Verify that when no AIContextProviders are specified, the inner agent has no additional providers.
/// </summary>
[Fact]
public void AIContextProviders_IsNullWhenNoneSpecified()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
// Act
var agent = new HarnessAgent(chatClient, TestMaxContextWindowTokens, TestMaxOutputTokens);
var innerAgent = agent.GetService<ChatClientAgent>();
// Assert
Assert.NotNull(innerAgent);
Assert.Null(innerAgent!.AIContextProviders);
}
#endregion
#region ChatOptions and Tools
/// <summary>
/// Verify that tools from ChatOptions are passed to the model during invocation.
/// </summary>
[Fact]
public async Task ChatOptions_ToolsArePreservedAsync()
{
// Arrange
var tool = AIFunctionFactory.Create(() => "test", "TestTool");
var mockClient = new Mock<IChatClient>();
ChatOptions? capturedOptions = null;
mockClient
.Setup(c => c.GetResponseAsync(
It.IsAny<IEnumerable<ChatMessage>>(),
It.IsAny<ChatOptions>(),
It.IsAny<CancellationToken>()))
.Callback<IEnumerable<ChatMessage>, ChatOptions?, CancellationToken>((_, opts, _) => capturedOptions = opts)
.ReturnsAsync(new ChatResponse(new ChatMessage(ChatRole.Assistant, "Done")));
var agent = new HarnessAgent(mockClient.Object, TestMaxContextWindowTokens, TestMaxOutputTokens, new HarnessAgentOptions
{
ChatOptions = new ChatOptions
{
Tools = [tool],
},
});
var session = await agent.CreateSessionAsync();
// Act
await agent.RunAsync([new ChatMessage(ChatRole.User, "Hi")], session);
// Assert — verify the tool was included in the ChatOptions passed to the model.
Assert.NotNull(capturedOptions);
Assert.NotNull(capturedOptions!.Tools);
Assert.Contains(capturedOptions.Tools, t => t == tool);
}
/// <summary>
/// Verify that the source ChatOptions are cloned and not modified.
/// </summary>
[Fact]
public void ChatOptions_SourceIsNotModified()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
var sourceChatOptions = new ChatOptions
{
Instructions = "original instructions",
Temperature = 0.7f,
};
// Act
_ = new HarnessAgent(chatClient, TestMaxContextWindowTokens, TestMaxOutputTokens, new HarnessAgentOptions
{
ChatOptions = sourceChatOptions,
});
// Assert — source ChatOptions should not be mutated.
Assert.Equal("original instructions", sourceChatOptions.Instructions);
Assert.Equal(0.7f, sourceChatOptions.Temperature);
}
#endregion
#region GetService
/// <summary>
/// Verify that GetService returns the HarnessAgent for its own type.
/// </summary>
[Fact]
public void GetService_ReturnsSelfForHarnessAgentType()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
// Act
var agent = new HarnessAgent(chatClient, TestMaxContextWindowTokens, TestMaxOutputTokens);
// Assert
Assert.Same(agent, agent.GetService<HarnessAgent>());
}
/// <summary>
/// Verify that GetService returns the inner ChatClientAgent.
/// </summary>
[Fact]
public void GetService_ReturnsInnerChatClientAgent()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
// Act
var agent = new HarnessAgent(chatClient, TestMaxContextWindowTokens, TestMaxOutputTokens);
// Assert
Assert.NotNull(agent.GetService<ChatClientAgent>());
}
#endregion
#region RunAsync Delegation
/// <summary>
/// Verify that RunAsync delegates to the inner ChatClientAgent.
/// </summary>
[Fact]
public async Task RunAsync_DelegatesToInnerAgentAsync()
{
// Arrange
var mockClient = new Mock<IChatClient>();
mockClient
.Setup(c => c.GetResponseAsync(
It.IsAny<IEnumerable<ChatMessage>>(),
It.IsAny<ChatOptions>(),
It.IsAny<CancellationToken>()))
.ReturnsAsync(new ChatResponse(new ChatMessage(ChatRole.Assistant, "Hello!")));
var agent = new HarnessAgent(mockClient.Object, TestMaxContextWindowTokens, TestMaxOutputTokens);
var session = await agent.CreateSessionAsync();
// Act
var response = await agent.RunAsync(
[new ChatMessage(ChatRole.User, "Hi")],
session);
// Assert
Assert.NotNull(response);
Assert.True(response.Messages.Any());
}
#endregion
#region DefaultInstructions
/// <summary>
/// Verify that DefaultInstructions is a non-empty public constant.
/// </summary>
[Fact]
public void DefaultInstructions_IsNonEmpty()
{
// Assert
Assert.False(string.IsNullOrWhiteSpace(HarnessAgent.DefaultInstructions));
}
#endregion
#region AsHarnessAgent Extension Method
/// <summary>
/// Verify that AsHarnessAgent creates a HarnessAgent with default options.
/// </summary>
[Fact]
public void AsHarnessAgent_CreatesAgentWithDefaults()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
// Act
var agent = chatClient.AsHarnessAgent(TestMaxContextWindowTokens, TestMaxOutputTokens);
// Assert
Assert.NotNull(agent);
Assert.IsType<HarnessAgent>(agent);
Assert.Equal(HarnessAgent.DefaultInstructions, agent.GetService<ChatClientAgent>()!.Instructions);
}
/// <summary>
/// Verify that AsHarnessAgent passes options through to the HarnessAgent.
/// </summary>
[Fact]
public void AsHarnessAgent_PassesOptionsThrough()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
// Act
var agent = chatClient.AsHarnessAgent(TestMaxContextWindowTokens, TestMaxOutputTokens, new HarnessAgentOptions
{
Name = "ExtensionAgent",
ChatOptions = new ChatOptions { Instructions = "Custom instructions" },
});
var innerAgent = agent.GetService<ChatClientAgent>();
// Assert
Assert.Equal("ExtensionAgent", agent.Name);
Assert.NotNull(innerAgent);
Assert.Equal("Custom instructions", innerAgent!.Instructions);
}
/// <summary>
/// Verify that AsHarnessAgent throws when chatClient is null.
/// </summary>
[Fact]
public void AsHarnessAgent_ThrowsWhenChatClientIsNull()
{
// Act & Assert
Assert.Throws<ArgumentNullException>(() => ((IChatClient)null!).AsHarnessAgent(TestMaxContextWindowTokens, TestMaxOutputTokens));
}
#endregion
}
@@ -1,11 +0,0 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<NoWarn>$(NoWarn);MAAI001</NoWarn>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\..\src\Microsoft.Agents.AI.Harness\Microsoft.Agents.AI.Harness.csproj" />
</ItemGroup>
</Project>
@@ -267,43 +267,7 @@ public sealed class OpenAIResponsesAgentResolutionIntegrationTests : IAsyncDispo
Assert.Equal(System.Net.HttpStatusCode.BadRequest, httpResponse.StatusCode);
string responseJson = await httpResponse.Content.ReadAsStringAsync();
using JsonDocument errorDoc1 = JsonDocument.Parse(responseJson);
string? errorCode = errorDoc1.RootElement.GetProperty("error").GetProperty("code").GetString();
Assert.Equal("missing_required_parameter", errorCode);
}
/// <summary>
/// Verifies that the model field alone is not used for agent resolution.
/// The multi-agent endpoint requires agent.name or metadata.entity_id; setting only model returns 400.
/// </summary>
[Fact]
public async Task CreateResponse_WithModelOnly_ReturnsBadRequestAsync()
{
// Arrange
const string AgentName = "test-agent";
this._httpClient = await this.CreateTestServerWithAgentResolutionAsync(
(AgentName, "Instructions", "Response"));
// Act - Send request with model=agentName but no agent.name or metadata.entity_id
using StringContent requestContent = new(JsonSerializer.Serialize(new
{
model = AgentName,
input = new[]
{
new { type = "message", role = "user", content = "Test message" }
}
}), Encoding.UTF8, "application/json");
using HttpResponseMessage httpResponse = await this._httpClient!.PostAsync(new Uri("/v1/responses", UriKind.Relative), requestContent);
// Assert - model is not used for agent resolution
Assert.Equal(System.Net.HttpStatusCode.BadRequest, httpResponse.StatusCode);
string responseJson = await httpResponse.Content.ReadAsStringAsync();
using JsonDocument errorDoc2 = JsonDocument.Parse(responseJson);
string? errorCode = errorDoc2.RootElement.GetProperty("error").GetProperty("code").GetString();
Assert.Equal("missing_required_parameter", errorCode);
Assert.Contains("agent.name", responseJson, StringComparison.OrdinalIgnoreCase);
}
/// <summary>
@@ -334,475 +334,4 @@ public sealed class FileSystemAgentFileStoreTests : IDisposable
}
#endregion
#region Symlink Escape Rejection
#if NET
/// <summary>
/// Attempts to create a file symlink. Returns false if the platform does not support
/// symlink creation (e.g., Windows without developer mode) or if creation fails.
/// </summary>
private static bool TryCreateFileSymbolicLink(string linkPath, string targetPath)
{
try
{
File.CreateSymbolicLink(linkPath, targetPath);
}
catch (IOException)
{
return false;
}
// Verify the symlink was actually created as a reparse point.
return File.Exists(linkPath)
&& (File.GetAttributes(linkPath) & FileAttributes.ReparsePoint) != 0;
}
/// <summary>
/// Attempts to create a directory symlink. Returns false if the platform does not support
/// symlink creation (e.g., Windows without developer mode) or if creation fails.
/// </summary>
private static bool TryCreateDirectorySymbolicLink(string linkPath, string targetPath)
{
try
{
Directory.CreateSymbolicLink(linkPath, targetPath);
}
catch (IOException)
{
return false;
}
// Verify the symlink was actually created as a reparse point.
return Directory.Exists(linkPath)
&& (File.GetAttributes(linkPath) & FileAttributes.ReparsePoint) != 0;
}
[Fact]
public async Task ReadFileAsync_SymlinkedFile_ThrowsAsync()
{
// Arrange — create a file outside the root and symlink to it from inside.
string outsideFile = Path.Combine(Path.GetTempPath(), "symlink_target_read_" + Guid.NewGuid().ToString("N") + ".txt");
File.WriteAllText(outsideFile, "SECRET_OUTSIDE_ROOT");
string linkPath = Path.Combine(this._rootDir, "leak.txt");
try
{
if (!TryCreateFileSymbolicLink(linkPath, outsideFile))
{
return; // Cannot create symlinks in this environment; skip.
}
// Act & Assert — reading through the symlink should be rejected.
await Assert.ThrowsAsync<ArgumentException>(() => this._store.ReadFileAsync("leak.txt"));
}
finally
{
if (File.Exists(linkPath))
{
File.Delete(linkPath);
}
File.Delete(outsideFile);
}
}
[Fact]
public async Task WriteFileAsync_SymlinkedFile_ThrowsAsync()
{
// Arrange — create a file outside the root and symlink to it from inside.
string outsideFile = Path.Combine(Path.GetTempPath(), "symlink_target_write_" + Guid.NewGuid().ToString("N") + ".txt");
File.WriteAllText(outsideFile, "ORIGINAL_CONTENT");
string linkPath = Path.Combine(this._rootDir, "overwrite.txt");
try
{
if (!TryCreateFileSymbolicLink(linkPath, outsideFile))
{
return;
}
// Act & Assert — writing through the symlink should be rejected.
await Assert.ThrowsAsync<ArgumentException>(() => this._store.WriteFileAsync("overwrite.txt", "EVIL_CONTENT"));
// Verify the outside file was NOT modified.
Assert.Equal("ORIGINAL_CONTENT", await File.ReadAllTextAsync(outsideFile));
}
finally
{
if (File.Exists(linkPath))
{
File.Delete(linkPath);
}
File.Delete(outsideFile);
}
}
[Fact]
public async Task DeleteFileAsync_SymlinkedFile_ThrowsAsync()
{
// Arrange
string outsideFile = Path.Combine(Path.GetTempPath(), "symlink_target_delete_" + Guid.NewGuid().ToString("N") + ".txt");
File.WriteAllText(outsideFile, "DO_NOT_DELETE");
string linkPath = Path.Combine(this._rootDir, "trap.txt");
try
{
if (!TryCreateFileSymbolicLink(linkPath, outsideFile))
{
return;
}
// Act & Assert
await Assert.ThrowsAsync<ArgumentException>(() => this._store.DeleteFileAsync("trap.txt"));
// Verify the outside file still exists.
Assert.True(File.Exists(outsideFile));
}
finally
{
if (File.Exists(linkPath))
{
File.Delete(linkPath);
}
File.Delete(outsideFile);
}
}
[Fact]
public async Task FileExistsAsync_SymlinkedFile_ThrowsAsync()
{
// Arrange
string outsideFile = Path.Combine(Path.GetTempPath(), "symlink_target_exists_" + Guid.NewGuid().ToString("N") + ".txt");
File.WriteAllText(outsideFile, "EXISTS_OUTSIDE");
string linkPath = Path.Combine(this._rootDir, "phantom.txt");
try
{
if (!TryCreateFileSymbolicLink(linkPath, outsideFile))
{
return;
}
// Act & Assert
await Assert.ThrowsAsync<ArgumentException>(() => this._store.FileExistsAsync("phantom.txt"));
}
finally
{
if (File.Exists(linkPath))
{
File.Delete(linkPath);
}
File.Delete(outsideFile);
}
}
[Fact]
public async Task WriteFileAsync_DanglingSymlink_ThrowsAsync()
{
// Arrange — create a symlink pointing to a non-existent target.
string nonExistentTarget = Path.Combine(Path.GetTempPath(), "dangling_target_" + Guid.NewGuid().ToString("N") + ".txt");
string linkPath = Path.Combine(this._rootDir, "dangling.txt");
try
{
if (!TryCreateFileSymbolicLink(linkPath, nonExistentTarget))
{
return;
}
// Act & Assert — even a dangling symlink must be rejected.
await Assert.ThrowsAsync<ArgumentException>(() => this._store.WriteFileAsync("dangling.txt", "CONTENT"));
// Verify the target was NOT created by following the dangling link.
Assert.False(File.Exists(nonExistentTarget));
}
finally
{
// Dangling symlinks: File.Exists returns false, but the link entry still exists.
// Use FileInfo to delete the link itself.
var linkInfo = new FileInfo(linkPath);
if (linkInfo.Exists || (linkInfo.Attributes & FileAttributes.ReparsePoint) != 0)
{
linkInfo.Delete();
}
}
}
[Fact]
public async Task ListFilesAsync_SymlinkedDirectory_ThrowsAsync()
{
// Arrange — create a directory outside root and symlink a directory inside root to it.
string outsideDir = Path.Combine(Path.GetTempPath(), "symlink_dir_target_" + Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(outsideDir);
File.WriteAllText(Path.Combine(outsideDir, "secret.txt"), "SECRET");
string linkDir = Path.Combine(this._rootDir, "linked-dir");
try
{
if (!TryCreateDirectorySymbolicLink(linkDir, outsideDir))
{
return;
}
// Act & Assert
await Assert.ThrowsAsync<ArgumentException>(() => this._store.ListFilesAsync("linked-dir"));
}
finally
{
if (Directory.Exists(linkDir))
{
Directory.Delete(linkDir);
}
Directory.Delete(outsideDir, recursive: true);
}
}
[Fact]
public async Task SearchFilesAsync_SymlinkedDirectory_ThrowsAsync()
{
// Arrange
string outsideDir = Path.Combine(Path.GetTempPath(), "symlink_search_target_" + Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(outsideDir);
File.WriteAllText(Path.Combine(outsideDir, "data.txt"), "SENSITIVE_DATA");
string linkDir = Path.Combine(this._rootDir, "search-link");
try
{
if (!TryCreateDirectorySymbolicLink(linkDir, outsideDir))
{
return;
}
// Act & Assert
await Assert.ThrowsAsync<ArgumentException>(() => this._store.SearchFilesAsync("search-link", "SENSITIVE"));
}
finally
{
if (Directory.Exists(linkDir))
{
Directory.Delete(linkDir);
}
Directory.Delete(outsideDir, recursive: true);
}
}
[Fact]
public async Task ReadFileAsync_ThroughDirectorySymlink_ThrowsAsync()
{
// Arrange — directory symlink inside root pointing outside; read a file through it.
string outsideDir = Path.Combine(Path.GetTempPath(), "symlink_dir_read_" + Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(outsideDir);
File.WriteAllText(Path.Combine(outsideDir, "secret.txt"), "DIR_SYMLINK_SECRET");
string linkDir = Path.Combine(this._rootDir, "linked-output");
try
{
if (!TryCreateDirectorySymbolicLink(linkDir, outsideDir))
{
return;
}
// Act & Assert — reading through a directory symlink should be rejected.
await Assert.ThrowsAsync<ArgumentException>(() => this._store.ReadFileAsync("linked-output/secret.txt"));
}
finally
{
if (Directory.Exists(linkDir))
{
Directory.Delete(linkDir);
}
Directory.Delete(outsideDir, recursive: true);
}
}
[Fact]
public async Task WriteFileAsync_ThroughDirectorySymlink_ThrowsAsync()
{
// Arrange — directory symlink; attempt to create/overwrite a file through it.
string outsideDir = Path.Combine(Path.GetTempPath(), "symlink_dir_write_" + Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(outsideDir);
string linkDir = Path.Combine(this._rootDir, "linked-output");
try
{
if (!TryCreateDirectorySymbolicLink(linkDir, outsideDir))
{
return;
}
// Act & Assert
await Assert.ThrowsAsync<ArgumentException>(() => this._store.WriteFileAsync("linked-output/created-by-agent.txt", "CONTENT"));
// Verify no file was created outside.
Assert.False(File.Exists(Path.Combine(outsideDir, "created-by-agent.txt")));
}
finally
{
if (Directory.Exists(linkDir))
{
Directory.Delete(linkDir);
}
Directory.Delete(outsideDir, recursive: true);
}
}
[Fact]
public async Task DeleteFileAsync_ThroughDirectorySymlink_ThrowsAsync()
{
// Arrange — directory symlink; attempt to delete a file through it.
string outsideDir = Path.Combine(Path.GetTempPath(), "symlink_dir_delete_" + Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(outsideDir);
string outsideFile = Path.Combine(outsideDir, "delete-me.txt");
File.WriteAllText(outsideFile, "DO_NOT_DELETE");
string linkDir = Path.Combine(this._rootDir, "linked-output");
try
{
if (!TryCreateDirectorySymbolicLink(linkDir, outsideDir))
{
return;
}
// Act & Assert
await Assert.ThrowsAsync<ArgumentException>(() => this._store.DeleteFileAsync("linked-output/delete-me.txt"));
// Verify the outside file was NOT deleted.
Assert.True(File.Exists(outsideFile));
}
finally
{
if (Directory.Exists(linkDir))
{
Directory.Delete(linkDir);
}
Directory.Delete(outsideDir, recursive: true);
}
}
[Fact]
public async Task CreateDirectoryAsync_ThroughDirectorySymlink_ThrowsAsync()
{
// Arrange — directory symlink; attempt to create a subdirectory through it.
string outsideDir = Path.Combine(Path.GetTempPath(), "symlink_dir_mkdir_" + Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(outsideDir);
string linkDir = Path.Combine(this._rootDir, "linked-output");
try
{
if (!TryCreateDirectorySymbolicLink(linkDir, outsideDir))
{
return;
}
// Act & Assert
await Assert.ThrowsAsync<ArgumentException>(() => this._store.CreateDirectoryAsync("linked-output/created-directory"));
// Verify no directory was created outside.
Assert.False(Directory.Exists(Path.Combine(outsideDir, "created-directory")));
}
finally
{
if (Directory.Exists(linkDir))
{
Directory.Delete(linkDir);
}
Directory.Delete(outsideDir, recursive: true);
}
}
[Fact]
public async Task SearchFilesAsync_RootWithSymlinkedFile_DoesNotLeakContentAsync()
{
// Arrange — symlinked file at root level; search should not return its content.
string outsideFile = Path.Combine(Path.GetTempPath(), "symlink_search_root_" + Guid.NewGuid().ToString("N") + ".txt");
File.WriteAllText(outsideFile, "ROOT_LEVEL_SECRET_CONTENT");
string linkPath = Path.Combine(this._rootDir, "env-link.txt");
try
{
if (!TryCreateFileSymbolicLink(linkPath, outsideFile))
{
return;
}
// Also add a normal file to confirm search still works for non-symlinks.
await this._store.WriteFileAsync("normal.txt", "NORMAL_CONTENT");
// Act — search at root should skip the symlinked file.
var results = await this._store.SearchFilesAsync("", "SECRET_CONTENT");
// Assert — no results from the symlinked file.
Assert.Empty(results);
}
finally
{
if (File.Exists(linkPath))
{
File.Delete(linkPath);
}
File.Delete(outsideFile);
}
}
[Fact]
public async Task ListFilesAsync_RootWithSymlinkedFile_ExcludesSymlinkAsync()
{
// Arrange — symlinked file at root level; listing should not include it.
string outsideFile = Path.Combine(Path.GetTempPath(), "symlink_list_root_" + Guid.NewGuid().ToString("N") + ".txt");
File.WriteAllText(outsideFile, "OUTSIDE");
string linkPath = Path.Combine(this._rootDir, "hidden-link.txt");
try
{
if (!TryCreateFileSymbolicLink(linkPath, outsideFile))
{
return;
}
// Also add a normal file.
await this._store.WriteFileAsync("visible.txt", "VISIBLE");
// Act
var files = await this._store.ListFilesAsync("");
// Assert — symlinked file should not appear in listing.
Assert.DoesNotContain("hidden-link.txt", files);
Assert.Contains("visible.txt", files);
}
finally
{
if (File.Exists(linkPath))
{
File.Delete(linkPath);
}
File.Delete(outsideFile);
}
}
#endif
#endregion
}
@@ -627,455 +627,4 @@ public class OpenTelemetryAgentTests
}
private static string ReplaceWhitespace(string? input) => Regex.Replace(input ?? "", @"\s+", "").Trim();
#region AutoWireChatClient
[Fact]
public async Task AutoWireChatClient_DefaultsToEnabled_EmitsChatSpan_Async()
{
var sourceName = Guid.NewGuid().ToString();
var activities = new List<Activity>();
using var tracerProvider = OpenTelemetry.Sdk.CreateTracerProviderBuilder()
.AddSource(sourceName)
.AddInMemoryExporter(activities)
.Build();
var fakeChatClient = new AutoWireTestChatClient();
var inner = new ChatClientAgent(fakeChatClient);
using var agent = new OpenTelemetryAgent(inner, sourceName);
_ = await agent.RunAsync("hi");
// Expect 2 activities: the inner chat span (from auto-wired OpenTelemetryChatClient) and the invoke_agent span.
Assert.Equal(2, activities.Count);
Assert.Contains(activities, a => a.DisplayName.StartsWith("invoke_agent", StringComparison.Ordinal));
Assert.Contains(activities, a => string.Equals(a.GetTagItem("gen_ai.operation.name") as string, "chat", StringComparison.Ordinal));
}
[Fact]
public async Task AutoWireChatClient_Streaming_EmitsChatSpan_Async()
{
var sourceName = Guid.NewGuid().ToString();
var activities = new List<Activity>();
using var tracerProvider = OpenTelemetry.Sdk.CreateTracerProviderBuilder()
.AddSource(sourceName)
.AddInMemoryExporter(activities)
.Build();
var fakeChatClient = new AutoWireTestChatClient();
var inner = new ChatClientAgent(fakeChatClient);
using var agent = new OpenTelemetryAgent(inner, sourceName);
await foreach (var _ in agent.RunStreamingAsync("hi"))
{
}
Assert.Equal(2, activities.Count);
Assert.Contains(activities, a => a.DisplayName.StartsWith("invoke_agent", StringComparison.Ordinal));
Assert.Contains(activities, a => string.Equals(a.GetTagItem("gen_ai.operation.name") as string, "chat", StringComparison.Ordinal));
}
[Fact]
public async Task AutoWireChatClient_Disabled_DoesNotEmitChatSpan_Async()
{
var sourceName = Guid.NewGuid().ToString();
var activities = new List<Activity>();
using var tracerProvider = OpenTelemetry.Sdk.CreateTracerProviderBuilder()
.AddSource(sourceName)
.AddInMemoryExporter(activities)
.Build();
var fakeChatClient = new AutoWireTestChatClient();
var inner = new ChatClientAgent(fakeChatClient);
using var agent = new OpenTelemetryAgent(inner, sourceName, autoWireChatClient: false);
_ = await agent.RunAsync("hi");
// Only the invoke_agent activity should be emitted; no chat span.
var activity = Assert.Single(activities);
Assert.StartsWith("invoke_agent", activity.DisplayName);
}
[Fact]
public async Task AutoWireChatClient_NonChatClientAgent_NoOp_Async()
{
// Inner is not a ChatClientAgent — auto-wiring must be a no-op and options must remain null.
AgentRunOptions? observedOptions = null;
var inner = new TestAIAgent
{
RunAsyncFunc = (messages, session, options, ct) =>
{
observedOptions = options;
return Task.FromResult(new AgentResponse(new ChatMessage(ChatRole.Assistant, "ok")));
},
};
using var agent = new OpenTelemetryAgent(inner);
_ = await agent.RunAsync("hi");
Assert.Null(observedOptions);
}
[Fact]
public async Task AutoWireChatClient_UseProvidedChatClientAsIs_DoesNotEmitChatSpan_Async()
{
var sourceName = Guid.NewGuid().ToString();
var activities = new List<Activity>();
using var tracerProvider = OpenTelemetry.Sdk.CreateTracerProviderBuilder()
.AddSource(sourceName)
.AddInMemoryExporter(activities)
.Build();
var fakeChatClient = new AutoWireTestChatClient();
var inner = new ChatClientAgent(fakeChatClient, new ChatClientAgentOptions { UseProvidedChatClientAsIs = true });
using var agent = new OpenTelemetryAgent(inner, sourceName);
_ = await agent.RunAsync("hi");
// UseProvidedChatClientAsIs opts out of auto-wiring, so only the invoke_agent span should be emitted.
var activity = Assert.Single(activities);
Assert.StartsWith("invoke_agent", activity.DisplayName);
}
[Fact]
public async Task AutoWireChatClient_AlreadyInstrumented_DoesNotDoubleWrap_Async()
{
var sourceName = Guid.NewGuid().ToString();
var activities = new List<Activity>();
using var tracerProvider = OpenTelemetry.Sdk.CreateTracerProviderBuilder()
.AddSource(sourceName)
.AddInMemoryExporter(activities)
.Build();
var fakeChatClient = new AutoWireTestChatClient();
// Pre-wrap with OpenTelemetryChatClient on the same source so spans flow through the tracer.
IChatClient preWrapped = fakeChatClient.AsBuilder().UseOpenTelemetry(sourceName: sourceName).Build();
var inner = new ChatClientAgent(preWrapped);
using var agent = new OpenTelemetryAgent(inner, sourceName);
_ = await agent.RunAsync("hi");
// Expect exactly 2 activities (one invoke_agent + one chat from the pre-existing wrapper). If we had double-wrapped, we would see 3.
Assert.Equal(2, activities.Count);
}
[Fact]
public async Task AutoWireChatClient_PreservesUserChatClientFactory_Async()
{
var sourceName = Guid.NewGuid().ToString();
var activities = new List<Activity>();
using var tracerProvider = OpenTelemetry.Sdk.CreateTracerProviderBuilder()
.AddSource(sourceName)
.AddInMemoryExporter(activities)
.Build();
bool userFactoryCalled = false;
var fakeChatClient = new AutoWireTestChatClient();
var inner = new ChatClientAgent(fakeChatClient);
using var agent = new OpenTelemetryAgent(inner, sourceName);
var runOptions = new ChatClientAgentRunOptions
{
ChatClientFactory = cc =>
{
userFactoryCalled = true;
return cc;
},
};
_ = await agent.RunAsync("hi", options: runOptions);
Assert.True(userFactoryCalled);
// Auto-wiring should still produce a chat span on top of the user's factory.
Assert.Equal(2, activities.Count);
Assert.Contains(activities, a => string.Equals(a.GetTagItem("gen_ai.operation.name") as string, "chat", StringComparison.Ordinal));
}
[Fact]
public async Task AutoWireChatClient_PlainAgentRunOptions_PreservesBaseProperties_Async()
{
// Auto-wiring converts a plain AgentRunOptions into a ChatClientAgentRunOptions. The base
// properties (ContinuationToken, AllowBackgroundResponses, AdditionalProperties, ResponseFormat)
// must be preserved so they reach the inner agent.
AgentRunOptions? observedOptions = null;
var fakeChatClient = new AutoWireTestChatClient();
var innerChatClientAgent = new ChatClientAgent(fakeChatClient);
// Wrapping agent: surfaces the ChatClientAgent via GetService (so auto-wiring activates),
// but captures the AgentRunOptions passed to RunAsync by the OpenTelemetryAgent.
var wrapper = new TestAIAgent
{
GetServiceFunc = (type, key) =>
type == typeof(ChatClientAgent) ? innerChatClientAgent : null,
RunAsyncFunc = (messages, session, options, ct) =>
{
observedOptions = options;
return Task.FromResult(new AgentResponse(new ChatMessage(ChatRole.Assistant, "ok")));
},
};
using var agent = new OpenTelemetryAgent(wrapper);
var additionalProps = new AdditionalPropertiesDictionary { ["customKey"] = "customValue" };
var inputOptions = new AgentRunOptions
{
AllowBackgroundResponses = true,
AdditionalProperties = additionalProps,
ResponseFormat = ChatResponseFormat.Json,
};
_ = await agent.RunAsync("hi", options: inputOptions);
Assert.NotNull(observedOptions);
Assert.IsType<ChatClientAgentRunOptions>(observedOptions);
Assert.Equal(true, observedOptions!.AllowBackgroundResponses);
Assert.Same(ChatResponseFormat.Json, observedOptions.ResponseFormat);
Assert.NotNull(observedOptions.AdditionalProperties);
Assert.Equal("customValue", observedOptions.AdditionalProperties!["customKey"]);
}
[Fact]
public async Task AutoWireChatClient_UserFactoryReturnsInstrumentedClient_DoesNotDoubleWrap_Async()
{
var sourceName = Guid.NewGuid().ToString();
var activities = new List<Activity>();
using var tracerProvider = OpenTelemetry.Sdk.CreateTracerProviderBuilder()
.AddSource(sourceName)
.AddInMemoryExporter(activities)
.Build();
var fakeChatClient = new AutoWireTestChatClient();
var inner = new ChatClientAgent(fakeChatClient);
using var agent = new OpenTelemetryAgent(inner, sourceName);
// User factory wraps the chat client with OpenTelemetryChatClient itself.
var runOptions = new ChatClientAgentRunOptions
{
ChatClientFactory = cc => cc.AsBuilder().UseOpenTelemetry(sourceName: sourceName).Build(),
};
_ = await agent.RunAsync("hi", options: runOptions);
// Expect 2 activities (invoke_agent + a single chat span). If we double-wrapped, we would see 3.
Assert.Equal(2, activities.Count);
}
[Theory]
[InlineData(null)]
[InlineData("")]
[InlineData(" ")]
[InlineData("\t")]
public async Task Ctor_NullOrWhitespaceSourceName_AutoWiredChatClientUsesDefaultSource_Async(string? sourceName)
{
// Both the agent-level invoke_agent span and the auto-wired chat span must be emitted under
// OpenTelemetryConsts.DefaultSourceName when the caller passes null, "", or whitespace, so they reach
// the same ActivitySource and are not silently dropped by the exporter.
var activities = new List<Activity>();
using var tracerProvider = OpenTelemetry.Sdk.CreateTracerProviderBuilder()
.AddSource("Experimental.Microsoft.Agents.AI")
.AddInMemoryExporter(activities)
.Build();
var fakeChatClient = new AutoWireTestChatClient();
var inner = new ChatClientAgent(fakeChatClient);
using var agent = new OpenTelemetryAgent(inner, sourceName);
_ = await agent.RunAsync("hi");
Assert.Equal(2, activities.Count);
Assert.All(activities, a => Assert.Equal("Experimental.Microsoft.Agents.AI", a.Source.Name));
Assert.Contains(activities, a => a.DisplayName.StartsWith("invoke_agent", StringComparison.Ordinal));
Assert.Contains(activities, a => string.Equals(a.GetTagItem("gen_ai.operation.name") as string, "chat", StringComparison.Ordinal));
}
#pragma warning disable MEAI001 // ResponseContinuationToken is experimental.
[Fact]
public async Task AutoWireChatClient_PlainAgentRunOptions_PreservesContinuationToken_Async()
{
// ContinuationToken is the fourth base AgentRunOptions property copied by CopyBaseAgentRunOptions
// and is not exercised by AutoWireChatClient_PlainAgentRunOptions_PreservesBaseProperties_Async.
AgentRunOptions? observedOptions = null;
var fakeChatClient = new AutoWireTestChatClient();
var innerChatClientAgent = new ChatClientAgent(fakeChatClient);
var wrapper = new TestAIAgent
{
GetServiceFunc = (type, key) =>
type == typeof(ChatClientAgent) ? innerChatClientAgent : null,
RunAsyncFunc = (messages, session, options, ct) =>
{
observedOptions = options;
return Task.FromResult(new AgentResponse(new ChatMessage(ChatRole.Assistant, "ok")));
},
};
using var agent = new OpenTelemetryAgent(wrapper);
var token = ResponseContinuationToken.FromBytes(new byte[] { 1, 2, 3 });
var inputOptions = new AgentRunOptions
{
ContinuationToken = token,
};
_ = await agent.RunAsync("hi", options: inputOptions);
Assert.NotNull(observedOptions);
Assert.IsType<ChatClientAgentRunOptions>(observedOptions);
Assert.Same(token, observedOptions!.ContinuationToken);
}
#pragma warning restore MEAI001
[Fact]
public async Task AutoWireChatClient_ChatClientAgentRunOptions_NoUserFactory_PreservesChatOptions_Async()
{
// When the caller passes a ChatClientAgentRunOptions without a ChatClientFactory, the auto-wiring
// must clone (not mutate) the caller's options, set the factory, and preserve nested ChatOptions.
var sourceName = Guid.NewGuid().ToString();
var activities = new List<Activity>();
using var tracerProvider = OpenTelemetry.Sdk.CreateTracerProviderBuilder()
.AddSource(sourceName)
.AddInMemoryExporter(activities)
.Build();
ChatOptions? observedChatOptions = null;
var fakeChatClient = new AutoWireTestChatClient
{
OnGetResponseAsync = (msgs, opts) => observedChatOptions = opts,
};
var inner = new ChatClientAgent(fakeChatClient);
using var agent = new OpenTelemetryAgent(inner, sourceName);
var inputChatOptions = new ChatOptions { Temperature = 0.42f, ModelId = "test-model" };
var inputOptions = new ChatClientAgentRunOptions(inputChatOptions);
_ = await agent.RunAsync("hi", options: inputOptions);
// Caller's options must not have been mutated (no factory installed on the caller's instance).
Assert.Null(inputOptions.ChatClientFactory);
// Inner chat client must observe the caller-supplied ChatOptions.
Assert.NotNull(observedChatOptions);
Assert.Equal(0.42f, observedChatOptions!.Temperature);
Assert.Equal("test-model", observedChatOptions.ModelId);
// Auto-wiring still produces a chat span.
Assert.Equal(2, activities.Count);
Assert.Contains(activities, a => string.Equals(a.GetTagItem("gen_ai.operation.name") as string, "chat", StringComparison.Ordinal));
}
[Fact]
public async Task AutoWireChatClient_StreamingDisabled_DoesNotEmitChatSpan_Async()
{
// Symmetry with AutoWireChatClient_Disabled_DoesNotEmitChatSpan_Async for the streaming path.
var sourceName = Guid.NewGuid().ToString();
var activities = new List<Activity>();
using var tracerProvider = OpenTelemetry.Sdk.CreateTracerProviderBuilder()
.AddSource(sourceName)
.AddInMemoryExporter(activities)
.Build();
var fakeChatClient = new AutoWireTestChatClient();
var inner = new ChatClientAgent(fakeChatClient);
using var agent = new OpenTelemetryAgent(inner, sourceName, autoWireChatClient: false);
await foreach (var _ in agent.RunStreamingAsync("hi"))
{
}
var activity = Assert.Single(activities);
Assert.StartsWith("invoke_agent", activity.DisplayName);
}
[Fact]
public async Task AutoWireChatClient_PlainAgentRunOptions_RealChatClientAgent_EmitsChatSpan_Async()
{
// High-level callers may pass the abstract base AgentRunOptions (not ChatClientAgentRunOptions) when
// wiring a ChatClientAgent. Auto-wiring must still kick in: convert to ChatClientAgentRunOptions,
// install the OTel-wrapping factory, and produce both the invoke_agent and chat spans end-to-end.
var sourceName = Guid.NewGuid().ToString();
var activities = new List<Activity>();
using var tracerProvider = OpenTelemetry.Sdk.CreateTracerProviderBuilder()
.AddSource(sourceName)
.AddInMemoryExporter(activities)
.Build();
ChatOptions? observedChatOptions = null;
var fakeChatClient = new AutoWireTestChatClient
{
OnGetResponseAsync = (_, opts) => observedChatOptions = opts,
};
var inner = new ChatClientAgent(fakeChatClient);
using var agent = new OpenTelemetryAgent(inner, sourceName);
// Pass the base AgentRunOptions, not ChatClientAgentRunOptions.
var inputOptions = new AgentRunOptions { AllowBackgroundResponses = false };
_ = await agent.RunAsync("hi", options: inputOptions);
// Inner chat client was actually invoked (auto-wired factory ran without breaking the pipeline).
Assert.NotNull(observedChatOptions);
Assert.Equal(2, activities.Count);
Assert.Contains(activities, a => a.DisplayName.StartsWith("invoke_agent", StringComparison.Ordinal));
Assert.Contains(activities, a => string.Equals(a.GetTagItem("gen_ai.operation.name") as string, "chat", StringComparison.Ordinal));
}
[Fact]
public async Task AutoWireChatClient_PlainAgentRunOptions_RealChatClientAgent_StreamingEmitsChatSpan_Async()
{
// Same as the sync test above but for the streaming path so both invocation paths
// are covered when callers pass a base AgentRunOptions.
var sourceName = Guid.NewGuid().ToString();
var activities = new List<Activity>();
using var tracerProvider = OpenTelemetry.Sdk.CreateTracerProviderBuilder()
.AddSource(sourceName)
.AddInMemoryExporter(activities)
.Build();
ChatOptions? observedChatOptions = null;
var fakeChatClient = new AutoWireTestChatClient
{
OnGetResponseAsync = (_, opts) => observedChatOptions = opts,
};
var inner = new ChatClientAgent(fakeChatClient);
using var agent = new OpenTelemetryAgent(inner, sourceName);
var inputOptions = new AgentRunOptions { AllowBackgroundResponses = false };
await foreach (var _ in agent.RunStreamingAsync("hi", options: inputOptions))
{
}
Assert.NotNull(observedChatOptions);
Assert.Equal(2, activities.Count);
Assert.Contains(activities, a => a.DisplayName.StartsWith("invoke_agent", StringComparison.Ordinal));
Assert.Contains(activities, a => string.Equals(a.GetTagItem("gen_ai.operation.name") as string, "chat", StringComparison.Ordinal));
}
private sealed class AutoWireTestChatClient : IChatClient
{
public Action<IEnumerable<ChatMessage>, ChatOptions?>? OnGetResponseAsync { get; set; }
public Task<ChatResponse> GetResponseAsync(IEnumerable<ChatMessage> messages, ChatOptions? options = null, CancellationToken cancellationToken = default)
{
this.OnGetResponseAsync?.Invoke(messages, options);
return Task.FromResult(new ChatResponse(new ChatMessage(ChatRole.Assistant, "ok")));
}
public async IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(IEnumerable<ChatMessage> messages, ChatOptions? options = null, [EnumeratorCancellation] CancellationToken cancellationToken = default)
{
this.OnGetResponseAsync?.Invoke(messages, options);
await Task.Yield();
yield return new ChatResponseUpdate(ChatRole.Assistant, "ok");
}
public object? GetService(Type serviceType, object? serviceKey = null) =>
serviceType?.IsInstanceOfType(this) == true ? this : null;
public void Dispose() { }
}
#endregion
}
@@ -1,26 +0,0 @@
// Copyright (c) Microsoft. All rights reserved.
using System.Collections.Generic;
using Microsoft.Extensions.AI;
namespace Microsoft.Agents.AI.Workflows.UnitTests;
public class AIAgentsAbstractionsExtensionsTests
{
[Fact]
public void CopyWithAssistantToUserForOtherParticipants_DoesNotMutateOriginalMessages()
{
ChatMessage original = new(ChatRole.Assistant, "from first agent")
{
AuthorName = "firstAgent"
};
List<ChatMessage> copied = new[] { original }
.CopyWithAssistantToUserForOtherParticipants("secondAgent");
Assert.Single(copied);
Assert.Equal(ChatRole.Assistant, original.Role);
Assert.Equal(ChatRole.User, copied[0].Role);
Assert.NotSame(original, copied[0]);
}
}
@@ -209,36 +209,6 @@ public class HandoffOrchestrationTests
Assert.DoesNotContain(capturedNextAgentMessages, m => m.Role == ChatRole.Tool && m.Contents.Any(c => c is FunctionResultContent frc && frc.Result?.ToString() == "Transferred."));
}
[Fact]
public async Task Handoffs_ReassignedMessagesDoNotMutateSharedConversationAsync()
{
var firstAgent = new ChatClientAgent(new MockChatClient((_, options) =>
{
string? transferFuncName = options?.Tools?.FirstOrDefault(t => t.Name.StartsWith("handoff_to_", StringComparison.Ordinal))?.Name;
Assert.NotNull(transferFuncName);
return new ChatResponse([
new ChatMessage(ChatRole.Assistant, "Context from first agent"),
new ChatMessage(ChatRole.Assistant, [new FunctionCallContent("call1", transferFuncName)]),
]);
}), name: "firstAgent");
CapturingAgent secondAgent = new("secondAgent", "The second agent", "Context from first agent");
var workflow =
AgentWorkflowBuilder.CreateHandoffBuilderWith(firstAgent)
.WithHandoff(firstAgent, secondAgent)
.Build();
(_, List<ChatMessage>? result, _, _) = await RunWorkflowAsync(workflow, [new ChatMessage(ChatRole.User, "start")]);
Assert.NotNull(result);
Assert.Equal(ChatRole.User, secondAgent.RoleSeenDuringRun);
ChatMessage sharedMessage = Assert.Single(result, m => m.Text == "Context from first agent");
Assert.Equal(ChatRole.Assistant, sharedMessage.Role);
Assert.NotSame(sharedMessage, secondAgent.MessageSeenDuringRun);
}
[Fact]
public async Task Handoffs_TwoTransfers_HandoffTargetsDoNotReceiveHandoffFunctionMessagesAsync()
{
@@ -1228,44 +1198,6 @@ public class HandoffOrchestrationTests
Workflow workflow, List<ChatMessage> input, ExecutionEnvironment executionEnvironment = ExecutionEnvironment.InProcess_Lockstep)
=> RunWorkflowCheckpointedAsync(workflow, input, executionEnvironment.ToWorkflowExecutionEnvironment());
private sealed class CapturingAgent(string name, string description, string textToCapture) : AIAgent
{
public override string Name => name;
public override string Description => description;
public ChatMessage? MessageSeenDuringRun { get; private set; }
public ChatRole? RoleSeenDuringRun { get; private set; }
protected override ValueTask<AgentSession> CreateSessionCoreAsync(CancellationToken cancellationToken = default)
=> new(new TestAgentSession());
protected override ValueTask<AgentSession> DeserializeSessionCoreAsync(JsonElement serializedState, JsonSerializerOptions? jsonSerializerOptions = null, CancellationToken cancellationToken = default)
=> new(new TestAgentSession());
protected override ValueTask<JsonElement> SerializeSessionCoreAsync(AgentSession session, JsonSerializerOptions? jsonSerializerOptions = null, CancellationToken cancellationToken = default)
=> default;
protected override Task<AgentResponse> RunCoreAsync(
IEnumerable<ChatMessage> messages, AgentSession? session = null, AgentRunOptions? options = null, CancellationToken cancellationToken = default) =>
throw new NotImplementedException();
protected override async IAsyncEnumerable<AgentResponseUpdate> RunCoreStreamingAsync(
IEnumerable<ChatMessage> messages, AgentSession? session = null, AgentRunOptions? options = null, [EnumeratorCancellation] CancellationToken cancellationToken = default)
{
await Task.Yield();
this.MessageSeenDuringRun = messages.Single(m => m.Text == textToCapture);
this.RoleSeenDuringRun = this.MessageSeenDuringRun.Role;
yield return new AgentResponseUpdate(ChatRole.Assistant, "Done")
{
AuthorName = this.Name,
MessageId = Guid.NewGuid().ToString("N"),
};
}
}
private sealed class TestAgentSession() : AgentSession();
private sealed class DoubleEchoAgent(string name) : AIAgent
{
public override string Name => name;
@@ -36,18 +36,13 @@ public sealed class InputWaiterTests : IDisposable
{
Task waitTask = this._waiter.WaitForInputAsync(TimeSpan.FromSeconds(5));
Task completedBeforeSignal = await Task.WhenAny(waitTask, Task.Delay(100));
completedBeforeSignal.Should().NotBeSameAs(
waitTask,
"the waiter should not complete before input is signaled");
await Task.Delay(50);
waitTask.IsCompleted.Should().BeFalse("the waiter should block until input is signaled");
this._waiter.SignalInput();
Task completedAfterSignal = await Task.WhenAny(waitTask, Task.Delay(TimeSpan.FromSeconds(1)));
completedAfterSignal.Should().BeSameAs(
waitTask,
"the wait task should complete after being signaled");
Task completed = await Task.WhenAny(waitTask, Task.Delay(TimeSpan.FromSeconds(1)));
completed.Should().BeSameAs(waitTask, "the wait task should complete after being signaled");
await waitTask;
}
@@ -1,546 +0,0 @@
// Copyright (c) Microsoft. All rights reserved.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using FluentAssertions;
using Microsoft.Agents.AI.Workflows.Execution;
namespace Microsoft.Agents.AI.Workflows.UnitTests;
public sealed class RouteBuilderTests
{
public enum HandlerOverload
{
SyncWithCancellation = 0,
SyncWithoutCancellation = 1,
AsyncWithCancellation = 2,
AsyncWithoutCancellation = 3,
}
private sealed record TestPayload(string Value);
private sealed class HandlerInvocation
{
public object? Message { get; private set; }
public IWorkflowContext? Context { get; private set; }
public CancellationToken CancellationToken { get; private set; }
public int InvocationCount { get; private set; }
public void Capture(object? message, IWorkflowContext context, CancellationToken cancellationToken = default)
{
this.Message = message;
this.Context = context;
this.CancellationToken = cancellationToken;
this.InvocationCount++;
}
}
private sealed class TestExternalRequestContext : IExternalRequestContext, IExternalRequestSink
{
public List<RequestPort> RegisteredPorts { get; } = [];
public List<ExternalRequest> PostedRequests { get; } = [];
public IExternalRequestSink RegisterPort(RequestPort port)
{
this.RegisteredPorts.Add(port);
return this;
}
public ValueTask PostAsync(ExternalRequest request)
{
this.PostedRequests.Add(request);
return default;
}
}
[Theory]
[InlineData(HandlerOverload.SyncWithCancellation)]
[InlineData(HandlerOverload.SyncWithoutCancellation)]
[InlineData(HandlerOverload.AsyncWithCancellation)]
[InlineData(HandlerOverload.AsyncWithoutCancellation)]
public async Task AddHandler_VoidOverloads_RouteExpectedMessageAsync(HandlerOverload overload)
{
// Arrange
RouteBuilder routeBuilder = new(null);
HandlerInvocation invocation = new();
CancellationToken cancellationToken = new CancellationTokenSource().Token;
RegisterVoidHandler(routeBuilder, invocation, overload);
MessageRouter router = routeBuilder.Build();
TestWorkflowContext context = new("executor");
// Act
CallResult? result = await router.RouteMessageAsync("hello", context, cancellationToken: cancellationToken);
// Assert
result.Should().NotBeNull();
result!.IsSuccess.Should().BeTrue();
result.IsVoid.Should().BeTrue();
result.Result.Should().BeNull();
invocation.InvocationCount.Should().Be(1);
invocation.Message.Should().Be("hello");
invocation.Context.Should().BeSameAs(context);
if (UsesCancellationToken(overload))
{
invocation.CancellationToken.Should().Be(cancellationToken);
}
}
[Theory]
[InlineData(HandlerOverload.SyncWithCancellation)]
[InlineData(HandlerOverload.SyncWithoutCancellation)]
[InlineData(HandlerOverload.AsyncWithCancellation)]
[InlineData(HandlerOverload.AsyncWithoutCancellation)]
public async Task AddHandler_ResultOverloads_RouteExpectedMessageAsync(HandlerOverload overload)
{
// Arrange
RouteBuilder routeBuilder = new(null);
HandlerInvocation invocation = new();
CancellationToken cancellationToken = new CancellationTokenSource().Token;
RegisterResultHandler(routeBuilder, invocation, overload);
MessageRouter router = routeBuilder.Build();
TestWorkflowContext context = new("executor");
// Act
CallResult? result = await router.RouteMessageAsync("hello", context, cancellationToken: cancellationToken);
// Assert
result.Should().NotBeNull();
result!.IsSuccess.Should().BeTrue();
result.IsVoid.Should().BeFalse();
result.Result.Should().Be("HELLO");
router.DefaultOutputTypes.Should().Contain(typeof(string));
invocation.InvocationCount.Should().Be(1);
invocation.Message.Should().Be("hello");
invocation.Context.Should().BeSameAs(context);
if (UsesCancellationToken(overload))
{
invocation.CancellationToken.Should().Be(cancellationToken);
}
}
[Theory]
[InlineData(HandlerOverload.SyncWithCancellation)]
[InlineData(HandlerOverload.SyncWithoutCancellation)]
[InlineData(HandlerOverload.AsyncWithCancellation)]
[InlineData(HandlerOverload.AsyncWithoutCancellation)]
public async Task AddCatchAll_VoidOverloads_RouteUnexpectedMessageAsync(HandlerOverload overload)
{
// Arrange
RouteBuilder routeBuilder = new(null);
HandlerInvocation invocation = new();
CancellationToken cancellationToken = new CancellationTokenSource().Token;
TestPayload payload = new("hello");
RegisterVoidCatchAll(routeBuilder, invocation, overload);
MessageRouter router = routeBuilder.Build();
TestWorkflowContext context = new("executor");
// Act
CallResult? result = await router.RouteMessageAsync(payload, context, cancellationToken: cancellationToken);
// Assert
result.Should().NotBeNull();
result!.IsSuccess.Should().BeTrue();
result.IsVoid.Should().BeTrue();
result.Result.Should().BeNull();
invocation.InvocationCount.Should().Be(1);
invocation.Message.Should().BeEquivalentTo(new PortableValue(payload));
invocation.Context.Should().BeSameAs(context);
if (UsesCancellationToken(overload))
{
invocation.CancellationToken.Should().Be(cancellationToken);
}
}
[Theory]
[InlineData(HandlerOverload.SyncWithCancellation)]
[InlineData(HandlerOverload.SyncWithoutCancellation)]
[InlineData(HandlerOverload.AsyncWithCancellation)]
[InlineData(HandlerOverload.AsyncWithoutCancellation)]
public async Task AddCatchAll_ResultOverloads_RouteUnexpectedMessageAsync(HandlerOverload overload)
{
// Arrange
RouteBuilder routeBuilder = new(null);
HandlerInvocation invocation = new();
CancellationToken cancellationToken = new CancellationTokenSource().Token;
TestPayload payload = new("hello");
RegisterResultCatchAll(routeBuilder, invocation, overload);
MessageRouter router = routeBuilder.Build();
TestWorkflowContext context = new("executor");
// Act
CallResult? result = await router.RouteMessageAsync(payload, context, cancellationToken: cancellationToken);
// Assert
result.Should().NotBeNull();
result!.IsSuccess.Should().BeTrue();
result.IsVoid.Should().BeFalse();
result.Result.Should().Be("HELLO");
invocation.InvocationCount.Should().Be(1);
invocation.Message.Should().BeEquivalentTo(new PortableValue(payload));
invocation.Context.Should().BeSameAs(context);
if (UsesCancellationToken(overload))
{
invocation.CancellationToken.Should().Be(cancellationToken);
}
}
[Fact]
public async Task AddHandlerUntyped_VoidAndResultOverloads_RouteExpectedMessageAsync()
{
// Arrange
RouteBuilder routeBuilder = new(null);
HandlerInvocation voidInvocation = new();
HandlerInvocation resultInvocation = new();
CancellationToken cancellationToken = new CancellationTokenSource().Token;
routeBuilder.AddHandlerUntyped(typeof(string), (message, context, token) =>
{
voidInvocation.Capture(message, context, token);
return default;
});
routeBuilder.AddHandlerUntyped<int>(typeof(int), (message, context, token) =>
{
resultInvocation.Capture(message, context, token);
return new((int)message + 1);
});
MessageRouter router = routeBuilder.Build();
TestWorkflowContext context = new("executor");
// Act
CallResult? voidResult = await router.RouteMessageAsync("hello", context, cancellationToken: cancellationToken);
CallResult? typedResult = await router.RouteMessageAsync(41, context, cancellationToken: cancellationToken);
// Assert
voidResult.Should().NotBeNull();
voidResult!.IsVoid.Should().BeTrue();
voidInvocation.Message.Should().Be("hello");
voidInvocation.Context.Should().BeSameAs(context);
voidInvocation.CancellationToken.Should().Be(cancellationToken);
typedResult.Should().NotBeNull();
typedResult!.Result.Should().Be(42);
router.DefaultOutputTypes.Should().Contain(typeof(int));
resultInvocation.Message.Should().Be(41);
resultInvocation.Context.Should().BeSameAs(context);
resultInvocation.CancellationToken.Should().Be(cancellationToken);
}
[Fact]
public void AddHandler_ForPortableValue_ThrowsInvalidOperationException()
{
// Arrange
RouteBuilder routeBuilder = new(null);
// Act
Action act = () => routeBuilder.AddHandler<PortableValue>((message, context) => { });
// Assert
act.Should().Throw<InvalidOperationException>()
.WithMessage("*Use AddCatchAll()*");
}
[Fact]
public void AddHandler_DuplicateRegistrationWithoutOverwrite_ThrowsArgumentException()
{
// Arrange
RouteBuilder routeBuilder = new(null);
routeBuilder.AddHandler<string>((message, context) => { });
// Act
Action act = () => routeBuilder.AddHandler<string>((message, context) => { });
// Assert
act.Should().Throw<ArgumentException>()
.WithMessage("*already registered*");
}
[Fact]
public void AddHandler_OverwriteWithoutExistingRegistration_ThrowsArgumentException()
{
// Arrange
RouteBuilder routeBuilder = new(null);
// Act
Action act = () => routeBuilder.AddHandler<string>((message, context) => { }, overwrite: true);
// Assert
act.Should().Throw<ArgumentException>()
.WithMessage("*has not yet been registered*");
}
[Fact]
public async Task AddHandler_OverwriteExistingRegistration_RoutesUpdatedHandlerAsync()
{
// Arrange
RouteBuilder routeBuilder = new(null);
routeBuilder.AddHandler<string>((message, context) => context.SendMessageAsync("first"));
routeBuilder.AddHandler<string>((message, context) => context.SendMessageAsync("second"), overwrite: true);
MessageRouter router = routeBuilder.Build();
TestWorkflowContext context = new("executor");
// Act
_ = await router.RouteMessageAsync("hello", context);
// Assert
context.SentMessages.Should().ContainSingle().Which.Should().Be("second");
}
[Fact]
public void AddCatchAll_DuplicateRegistrationWithoutOverwrite_ThrowsInvalidOperationException()
{
// Arrange
RouteBuilder routeBuilder = new(null);
routeBuilder.AddCatchAll((message, context) => { });
// Act
Action act = () => routeBuilder.AddCatchAll((message, context) => { });
// Assert
act.Should().Throw<InvalidOperationException>()
.WithMessage("*already registered*");
}
[Fact]
public async Task AddCatchAll_OverwriteExistingRegistration_RoutesUpdatedHandlerAsync()
{
// Arrange
RouteBuilder routeBuilder = new(null);
routeBuilder.AddCatchAll((message, context) => context.SendMessageAsync("first"));
routeBuilder.AddCatchAll((message, context) => context.SendMessageAsync("second"), overwrite: true);
MessageRouter router = routeBuilder.Build();
TestWorkflowContext context = new("executor");
// Act
_ = await router.RouteMessageAsync(new TestPayload("hello"), context);
// Assert
context.SentMessages.Should().ContainSingle().Which.Should().Be("second");
}
[Fact]
public void AddPortHandler_WithoutExternalRequestContext_ThrowsInvalidOperationException()
{
// Arrange
RouteBuilder routeBuilder = new(null);
// Act
Action act = () => routeBuilder.AddPortHandler<string, int>("port", (response, context, cancellationToken) => default, out _);
// Assert
act.Should().Throw<InvalidOperationException>()
.WithMessage("*external request context is required*");
}
[Fact]
public async Task AddPortHandler_RoutesMatchingExternalResponseAsync()
{
// Arrange
TestExternalRequestContext externalRequestContext = new();
RouteBuilder routeBuilder = new(externalRequestContext);
HandlerInvocation invocation = new();
routeBuilder.AddPortHandler<string, int>("port", (response, context, cancellationToken) =>
{
invocation.Capture(response, context, cancellationToken);
return default;
}, out PortBinding portBinding);
await portBinding.PostRequestAsync("request", requestId: "req-1");
MessageRouter router = routeBuilder.Build();
TestWorkflowContext context = new("executor");
CancellationToken cancellationToken = new CancellationTokenSource().Token;
ExternalResponse response = externalRequestContext.PostedRequests.Single().CreateResponse(42);
// Act
CallResult? result = await router.RouteMessageAsync(response, context, cancellationToken: cancellationToken);
// Assert
externalRequestContext.RegisteredPorts.Should().ContainSingle(port => port.Id == "port");
externalRequestContext.PostedRequests.Should().ContainSingle(request => request.RequestId == "req-1");
result.Should().NotBeNull();
result!.IsSuccess.Should().BeTrue();
result.Result.Should().BeSameAs(response);
invocation.InvocationCount.Should().Be(1);
invocation.Message.Should().Be(42);
invocation.Context.Should().BeSameAs(context);
invocation.CancellationToken.Should().Be(cancellationToken);
}
[Fact]
public async Task AddPortHandler_UnknownPort_ReturnsExceptionResultAsync()
{
// Arrange
TestExternalRequestContext externalRequestContext = new();
RouteBuilder routeBuilder = new(externalRequestContext);
routeBuilder.AddPortHandler<string, int>("port", (response, context, cancellationToken) => default, out _);
MessageRouter router = routeBuilder.Build();
ExternalRequest request = ExternalRequest.Create(RequestPort.Create<string, int>("other"), "request", requestId: "req-1");
// Act
CallResult? result = await router.RouteMessageAsync(request.CreateResponse(42), new TestWorkflowContext("executor"));
// Assert
result.Should().NotBeNull();
result!.IsSuccess.Should().BeFalse();
result.Exception.Should().BeOfType<InvalidOperationException>();
result.Exception!.Message.Should().Contain("Unknown port");
}
private static void RegisterVoidHandler(RouteBuilder routeBuilder, HandlerInvocation invocation, HandlerOverload overload)
{
switch (overload)
{
case HandlerOverload.SyncWithCancellation:
routeBuilder.AddHandler<string>((message, context, cancellationToken) => invocation.Capture(message, context, cancellationToken));
break;
case HandlerOverload.SyncWithoutCancellation:
routeBuilder.AddHandler<string>((message, context) => invocation.Capture(message, context));
break;
case HandlerOverload.AsyncWithCancellation:
routeBuilder.AddHandler<string>((message, context, cancellationToken) =>
{
invocation.Capture(message, context, cancellationToken);
return default;
});
break;
case HandlerOverload.AsyncWithoutCancellation:
routeBuilder.AddHandler<string>((message, context) =>
{
invocation.Capture(message, context);
return default;
});
break;
default:
throw new ArgumentOutOfRangeException(nameof(overload));
}
}
private static void RegisterResultHandler(RouteBuilder routeBuilder, HandlerInvocation invocation, HandlerOverload overload)
{
switch (overload)
{
case HandlerOverload.SyncWithCancellation:
routeBuilder.AddHandler<string, string>((message, context, cancellationToken) =>
{
invocation.Capture(message, context, cancellationToken);
return NormalizeHandlerResult(message);
});
break;
case HandlerOverload.SyncWithoutCancellation:
routeBuilder.AddHandler<string, string>((message, context) =>
{
invocation.Capture(message, context);
return NormalizeHandlerResult(message);
});
break;
case HandlerOverload.AsyncWithCancellation:
Func<string, IWorkflowContext, CancellationToken, ValueTask<string>> asyncHandlerWithCancellation = (message, context, cancellationToken) =>
{
invocation.Capture(message, context, cancellationToken);
return new ValueTask<string>(NormalizeHandlerResult(message));
};
routeBuilder.AddHandler(asyncHandlerWithCancellation);
break;
case HandlerOverload.AsyncWithoutCancellation:
Func<string, IWorkflowContext, ValueTask<string>> asyncHandler = (message, context) =>
{
invocation.Capture(message, context);
return new ValueTask<string>(NormalizeHandlerResult(message));
};
routeBuilder.AddHandler(asyncHandler);
break;
default:
throw new ArgumentOutOfRangeException(nameof(overload));
}
}
private static void RegisterVoidCatchAll(RouteBuilder routeBuilder, HandlerInvocation invocation, HandlerOverload overload)
{
switch (overload)
{
case HandlerOverload.SyncWithCancellation:
routeBuilder.AddCatchAll((message, context, cancellationToken) => invocation.Capture(message, context, cancellationToken));
break;
case HandlerOverload.SyncWithoutCancellation:
routeBuilder.AddCatchAll((message, context) => invocation.Capture(message, context));
break;
case HandlerOverload.AsyncWithCancellation:
routeBuilder.AddCatchAll((message, context, cancellationToken) =>
{
invocation.Capture(message, context, cancellationToken);
return default;
});
break;
case HandlerOverload.AsyncWithoutCancellation:
routeBuilder.AddCatchAll((message, context) =>
{
invocation.Capture(message, context);
return default;
});
break;
default:
throw new ArgumentOutOfRangeException(nameof(overload));
}
}
private static void RegisterResultCatchAll(RouteBuilder routeBuilder, HandlerInvocation invocation, HandlerOverload overload)
{
switch (overload)
{
case HandlerOverload.SyncWithCancellation:
routeBuilder.AddCatchAll((message, context, cancellationToken) =>
{
invocation.Capture(message, context, cancellationToken);
return NormalizeCatchAllResult(message);
});
break;
case HandlerOverload.SyncWithoutCancellation:
routeBuilder.AddCatchAll((message, context) =>
{
invocation.Capture(message, context);
return NormalizeCatchAllResult(message);
});
break;
case HandlerOverload.AsyncWithCancellation:
Func<PortableValue, IWorkflowContext, CancellationToken, ValueTask<string>> asyncCatchAllWithCancellation = (message, context, cancellationToken) =>
{
invocation.Capture(message, context, cancellationToken);
return new ValueTask<string>(NormalizeCatchAllResult(message));
};
routeBuilder.AddCatchAll(asyncCatchAllWithCancellation);
break;
case HandlerOverload.AsyncWithoutCancellation:
Func<PortableValue, IWorkflowContext, ValueTask<string>> asyncCatchAll = (message, context) =>
{
invocation.Capture(message, context);
return new ValueTask<string>(NormalizeCatchAllResult(message));
};
routeBuilder.AddCatchAll(asyncCatchAll);
break;
default:
throw new ArgumentOutOfRangeException(nameof(overload));
}
}
private static bool UsesCancellationToken(HandlerOverload overload) =>
overload is HandlerOverload.SyncWithCancellation or HandlerOverload.AsyncWithCancellation;
private static string NormalizeHandlerResult(string message) => message.ToUpperInvariant();
private static string NormalizeCatchAllResult(PortableValue message) => GetPayloadValue(message).ToUpperInvariant();
private static string GetPayloadValue(PortableValue message)
{
return message.As<TestPayload>() is TestPayload payload
? payload.Value
: throw new InvalidOperationException("Expected catch-all message payload to deserialize as TestPayload.");
}
}
@@ -1,7 +1,6 @@
// Copyright (c) Microsoft. All rights reserved.
using System;
using System.Collections.Generic;
using FluentAssertions;
namespace Microsoft.Agents.AI.Workflows.UnitTests;
@@ -158,301 +157,4 @@ public partial class WorkflowBuilderSmokeTests
workflow3.Name.Should().Be("Named Only");
workflow3.Description.Should().BeNull();
}
[Fact]
public void ForwardMessage_WithSingleTarget_CreatesDirectEdge()
{
// Arrange
NoOpExecutor source = new("start");
NoOpExecutor target = new("target");
// Act
Workflow workflow = new WorkflowBuilder(source.Id)
.ForwardMessage<string>(source, target)
.Build();
// Assert
Edge edge = GetSingleEdge(workflow, source.Id);
edge.Kind.Should().Be(EdgeKind.Direct);
edge.DirectEdgeData.Should().NotBeNull();
edge.DirectEdgeData!.SourceId.Should().Be(source.Id);
edge.DirectEdgeData!.SinkId.Should().Be(target.Id);
edge.DirectEdgeData.Condition.Should().NotBeNull();
edge.DirectEdgeData.Condition!("message").Should().BeTrue();
edge.DirectEdgeData.Condition!(42).Should().BeFalse();
edge.DirectEdgeData.Condition!(null).Should().BeFalse();
}
[Fact]
public void ForwardMessage_WithMultipleTargets_CreatesFanOutEdge()
{
// Arrange
NoOpExecutor source = new("start");
NoOpExecutor target1 = new("target1");
NoOpExecutor target2 = new("target2");
// Act
Workflow workflow = new WorkflowBuilder(source.Id)
.ForwardMessage<string>(source, [target1, target2], message => message == "match")
.Build();
// Assert
Edge edge = GetSingleEdge(workflow, source.Id);
edge.Kind.Should().Be(EdgeKind.FanOut);
edge.FanOutEdgeData.Should().NotBeNull();
edge.FanOutEdgeData!.SourceId.Should().Be(source.Id);
edge.FanOutEdgeData!.SinkIds.Should().Equal([target1.Id, target2.Id]);
edge.FanOutEdgeData.EdgeAssigner.Should().NotBeNull();
edge.FanOutEdgeData.EdgeAssigner!("match", 2).Should().Equal([0, 1]);
edge.FanOutEdgeData.EdgeAssigner!("other", 2).Should().BeEmpty();
edge.FanOutEdgeData.EdgeAssigner!(42, 2).Should().BeEmpty();
}
[Fact]
public void ForwardExcept_WithSingleTarget_CreatesDirectEdge()
{
// Arrange
NoOpExecutor source = new("start");
NoOpExecutor target = new("target");
// Act
Workflow workflow = new WorkflowBuilder(source.Id)
.ForwardExcept<string>(source, target)
.Build();
// Assert
Edge edge = GetSingleEdge(workflow, source.Id);
edge.Kind.Should().Be(EdgeKind.Direct);
edge.DirectEdgeData.Should().NotBeNull();
edge.DirectEdgeData!.SourceId.Should().Be(source.Id);
edge.DirectEdgeData!.SinkId.Should().Be(target.Id);
edge.DirectEdgeData.Condition.Should().NotBeNull();
edge.DirectEdgeData.Condition!("message").Should().BeFalse();
edge.DirectEdgeData.Condition!(42).Should().BeTrue();
edge.DirectEdgeData.Condition!(null).Should().BeTrue();
}
[Fact]
public void ForwardExcept_WithMultipleTargets_CreatesFanOutEdge()
{
// Arrange
NoOpExecutor source = new("start");
NoOpExecutor target1 = new("target1");
NoOpExecutor target2 = new("target2");
// Act
Workflow workflow = new WorkflowBuilder(source.Id)
.ForwardExcept<string>(source, [target1, target2])
.Build();
// Assert
Edge edge = GetSingleEdge(workflow, source.Id);
edge.Kind.Should().Be(EdgeKind.FanOut);
edge.FanOutEdgeData.Should().NotBeNull();
edge.FanOutEdgeData!.SourceId.Should().Be(source.Id);
edge.FanOutEdgeData!.SinkIds.Should().Equal([target1.Id, target2.Id]);
edge.FanOutEdgeData.EdgeAssigner.Should().NotBeNull();
edge.FanOutEdgeData.EdgeAssigner!(42, 2).Should().Equal([0, 1]);
edge.FanOutEdgeData.EdgeAssigner!("message", 2).Should().BeEmpty();
}
[Fact]
public void AddChain_CreatesSequentialDirectEdges()
{
// Arrange
NoOpExecutor source = new("start");
NoOpExecutor middle = new("middle");
NoOpExecutor end = new("end");
// Act
Workflow workflow = new WorkflowBuilder(source.Id)
.AddChain(source, [middle, end])
.Build();
// Assert
Edge firstEdge = GetSingleEdge(workflow, source.Id);
firstEdge.Kind.Should().Be(EdgeKind.Direct);
firstEdge.DirectEdgeData!.SourceId.Should().Be(source.Id);
firstEdge.DirectEdgeData.SinkId.Should().Be(middle.Id);
Edge secondEdge = GetSingleEdge(workflow, middle.Id);
secondEdge.Kind.Should().Be(EdgeKind.Direct);
secondEdge.DirectEdgeData!.SourceId.Should().Be(middle.Id);
secondEdge.DirectEdgeData.SinkId.Should().Be(end.Id);
}
[Fact]
public void AddChain_WhenExecutorRepeats_Throws()
{
// Arrange
NoOpExecutor source = new("start");
NoOpExecutor middle = new("middle");
// Act
Action act = () => new WorkflowBuilder(source.Id)
.AddChain(source, [middle, source]);
// Assert
act.Should().Throw<ArgumentException>()
.WithParameterName("executors");
}
[Fact]
public void AddExternalCall_CreatesRequestPortAndRoundTripEdges()
{
// Arrange
const string PortId = "port1";
NoOpExecutor source = new("start");
// Act
Workflow workflow = new WorkflowBuilder(source.Id)
.AddExternalCall<string, int>(source, PortId)
.Build();
// Assert
workflow.Ports.Should().ContainKey(PortId);
workflow.Ports[PortId].Request.Should().Be(typeof(string));
workflow.Ports[PortId].Response.Should().Be(typeof(int));
workflow.ExecutorBindings.Should().ContainKey(PortId);
Edge requestEdge = GetSingleEdge(workflow, source.Id);
requestEdge.Kind.Should().Be(EdgeKind.Direct);
requestEdge.DirectEdgeData!.SourceId.Should().Be(source.Id);
requestEdge.DirectEdgeData.SinkId.Should().Be(PortId);
Edge responseEdge = GetSingleEdge(workflow, PortId);
responseEdge.Kind.Should().Be(EdgeKind.Direct);
responseEdge.DirectEdgeData!.SourceId.Should().Be(PortId);
responseEdge.DirectEdgeData.SinkId.Should().Be(source.Id);
}
[Fact]
public void AddSwitch_CreatesFanOutEdgeWithCasesAndDefault()
{
// Arrange
NoOpExecutor source = new("start");
NoOpExecutor stringTarget = new("string-target");
NoOpExecutor intTarget = new("int-target");
NoOpExecutor defaultTarget = new("default-target");
// Act
Workflow workflow = new WorkflowBuilder(source.Id)
.AddSwitch(source, switchBuilder => switchBuilder
.AddCase<string>(message => message == "match", [stringTarget])
.AddCase<int>(message => message > 0, [intTarget])
.WithDefault([defaultTarget]))
.Build();
// Assert
Edge edge = GetSingleEdge(workflow, source.Id);
edge.Kind.Should().Be(EdgeKind.FanOut);
edge.FanOutEdgeData.Should().NotBeNull();
edge.FanOutEdgeData!.SourceId.Should().Be(source.Id);
edge.FanOutEdgeData!.SinkIds.Should().Equal([stringTarget.Id, intTarget.Id, defaultTarget.Id]);
edge.FanOutEdgeData.EdgeAssigner.Should().NotBeNull();
edge.FanOutEdgeData.EdgeAssigner!("match", 3).Should().Equal([0]);
edge.FanOutEdgeData.EdgeAssigner!(2, 3).Should().Equal([1]);
edge.FanOutEdgeData.EdgeAssigner!("other", 3).Should().Equal([2]);
}
[Fact]
public void ForwardMessage_InvalidArguments_Throw()
{
// Arrange
WorkflowBuilder builder = new("start");
NoOpExecutor source = new("start");
NoOpExecutor target = new("target");
// Act/Assert
Assert.Throws<ArgumentNullException>(() => ((WorkflowBuilder)null!).ForwardMessage<string>(source, target));
Assert.Throws<ArgumentNullException>("source", () => builder.ForwardMessage<string>(null!, target));
Assert.Throws<ArgumentNullException>("target", () => builder.ForwardMessage<string>(source, (ExecutorBinding)null!));
Assert.Throws<ArgumentNullException>("targets", () => builder.ForwardMessage<string>(source, (IEnumerable<ExecutorBinding>)null!));
Assert.Throws<ArgumentNullException>("targets", () => builder.ForwardMessage<string>(source, [target, null!]));
Assert.Throws<ArgumentException>("targets", () => builder.ForwardMessage<string>(source, []));
}
[Fact]
public void ForwardExcept_InvalidArguments_Throw()
{
// Arrange
WorkflowBuilder builder = new("start");
NoOpExecutor source = new("start");
NoOpExecutor target = new("target");
// Act/Assert
Assert.Throws<ArgumentNullException>(() => ((WorkflowBuilder)null!).ForwardExcept<string>(source, target));
Assert.Throws<ArgumentNullException>("source", () => builder.ForwardExcept<string>(null!, target));
Assert.Throws<ArgumentNullException>("target", () => builder.ForwardExcept<string>(source, (ExecutorBinding)null!));
Assert.Throws<ArgumentNullException>("targets", () => builder.ForwardExcept<string>(source, (IEnumerable<ExecutorBinding>)null!));
Assert.Throws<ArgumentNullException>("targets", () => builder.ForwardExcept<string>(source, [target, null!]));
Assert.Throws<ArgumentException>("targets", () => builder.ForwardExcept<string>(source, []));
}
[Fact]
public void AddChain_InvalidArguments_Throw()
{
// Arrange
WorkflowBuilder builder = new("start");
NoOpExecutor source = new("start");
NoOpExecutor target = new("target");
NoOpExecutor otherTarget = new("other-target");
// Act/Assert
Assert.Throws<ArgumentNullException>(() => ((WorkflowBuilder)null!).AddChain(source, [target]));
Assert.Throws<ArgumentNullException>("source", () => builder.AddChain(null!, [target]));
Assert.Throws<ArgumentNullException>("executors", () => builder.AddChain(source, null!));
Assert.Throws<ArgumentNullException>("executors", () => builder.AddChain(source, [target, null!]));
Assert.Throws<ArgumentException>("executors", () => builder.AddChain(source, [target, source]));
Assert.Throws<ArgumentException>("executors", () => builder.AddChain(source, [target, otherTarget, target]));
}
[Fact]
public void AddExternalCall_InvalidArguments_Throw()
{
// Arrange
WorkflowBuilder builder = new("start");
NoOpExecutor source = new("start");
// Act/Assert
Assert.Throws<ArgumentNullException>(() => ((WorkflowBuilder)null!).AddExternalCall<string, int>(source, "port"));
Assert.Throws<ArgumentNullException>("source", () => builder.AddExternalCall<string, int>(null!, "port"));
Assert.Throws<ArgumentNullException>("portId", () => builder.AddExternalCall<string, int>(source, null!));
}
[Fact]
public void AddSwitch_InvalidArguments_Throw()
{
// Arrange
WorkflowBuilder builder = new("start");
NoOpExecutor source = new("start");
// Act/Assert
Assert.Throws<ArgumentNullException>(() => ((WorkflowBuilder)null!).AddSwitch(source, _ => { }));
Assert.Throws<ArgumentNullException>("source", () => builder.AddSwitch(null!, _ => { }));
Assert.Throws<ArgumentNullException>("configureSwitch", () => builder.AddSwitch(source, null!));
Assert.Throws<ArgumentException>("targets", () => builder.AddSwitch(source, _ => { }));
Assert.Throws<ArgumentException>("targets", () => builder.AddSwitch(source, switchBuilder => switchBuilder.AddCase<string>(_ => true, [])));
}
[Fact]
public void SwitchBuilder_InvalidArguments_Throw()
{
// Arrange
SwitchBuilder switchBuilder = new();
NoOpExecutor target = new("target");
// Act/Assert
Assert.Throws<ArgumentNullException>("predicate", () => switchBuilder.AddCase<string>(null!, [target]));
Assert.Throws<ArgumentNullException>("executors", () => switchBuilder.AddCase<string>(_ => true, null!));
Assert.Throws<ArgumentNullException>("executors[1]", () => switchBuilder.AddCase<string>(_ => true, [target, null!]));
Assert.Throws<ArgumentNullException>("executors", () => switchBuilder.WithDefault(null!));
Assert.Throws<ArgumentNullException>("executors[1]", () => switchBuilder.WithDefault([target, null!]));
}
/// <summary>
/// Gets the only edge emitted by the specified workflow source.
/// </summary>
private static Edge GetSingleEdge(Workflow workflow, string sourceId)
=> workflow.Edges[sourceId].Should().ContainSingle().Subject;
}
@@ -1,6 +1,5 @@
// Copyright (c) Microsoft. All rights reserved.
using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Microsoft.Extensions.AI;
@@ -291,121 +290,6 @@ public sealed class WorkflowEvaluationTests
Assert.DoesNotContain("end", result.Keys);
}
// ---------------------------------------------------------------
// BuildOverallItem tests (expected output / ground truth)
// ---------------------------------------------------------------
[Fact]
public void BuildOverallItem_NoCompletedExecutorWithResponse_ReturnsNull()
{
// Arrange — no ExecutorCompletedEvent with usable response data and no AgentResponseEvent
var events = new List<WorkflowEvent>
{
new ExecutorInvokedEvent("agent-1", "query"),
};
// Act
var item = WorkflowEvaluationExtensions.BuildOverallItem(events, splitter: null, expectedOutput: null);
// Assert
Assert.Null(item);
}
[Fact]
public void BuildOverallItem_NoAgentResponseEvent_FallsBackToLastExecutorCompleted()
{
// Arrange — only ExecutorCompletedEvent (the default when EmitAgentResponseEvents is false)
var finalResponse = new AgentResponse(new ChatMessage(ChatRole.Assistant, "Paris"));
var events = new List<WorkflowEvent>
{
new ExecutorInvokedEvent("researcher", "What is the capital of France?"),
new ExecutorCompletedEvent("researcher", new AgentResponse(new ChatMessage(ChatRole.Assistant, "draft"))),
new ExecutorInvokedEvent("editor", "draft"),
new ExecutorCompletedEvent("editor", finalResponse),
};
// Act
var item = WorkflowEvaluationExtensions.BuildOverallItem(
events, splitter: null, expectedOutput: "Paris");
// Assert
Assert.NotNull(item);
Assert.Equal("What is the capital of France?", item.Query);
Assert.Equal("Paris", item.Response);
Assert.Equal("Paris", item.ExpectedOutput);
}
[Fact]
public void BuildOverallItem_WithFinalResponseAndExpectedOutput_StampsExpectedOutput()
{
// Arrange
var finalResponse = new AgentResponse(new ChatMessage(ChatRole.Assistant, "Ofrece 41 planes"));
var events = new List<WorkflowEvent>
{
new ExecutorInvokedEvent("agent-1", "How many plans does Netlife offer?"),
new ExecutorCompletedEvent("agent-1", finalResponse),
new AgentResponseEvent("agent-1", finalResponse),
};
// Act
var item = WorkflowEvaluationExtensions.BuildOverallItem(
events, splitter: null, expectedOutput: "Ofrece 41 planes");
// Assert
Assert.NotNull(item);
Assert.Equal("How many plans does Netlife offer?", item.Query);
Assert.Equal("Ofrece 41 planes", item.Response);
Assert.Equal("Ofrece 41 planes", item.ExpectedOutput);
}
[Fact]
public void BuildOverallItem_WithFinalResponseAndNoExpectedOutput_LeavesExpectedOutputNull()
{
// Arrange
var finalResponse = new AgentResponse(new ChatMessage(ChatRole.Assistant, "answer"));
var events = new List<WorkflowEvent>
{
new ExecutorInvokedEvent("agent-1", "query"),
new ExecutorCompletedEvent("agent-1", finalResponse),
new AgentResponseEvent("agent-1", finalResponse),
};
// Act
var item = WorkflowEvaluationExtensions.BuildOverallItem(events, splitter: null, expectedOutput: null);
// Assert
Assert.NotNull(item);
Assert.Null(item.ExpectedOutput);
}
[Fact]
public async Task EvaluateAsync_WithIncludeOverallButNoFinalResponse_ThrowsAsync()
{
// Arrange — build a workflow whose AIAgentHostExecutor is NOT bound with
// EmitAgentResponseEvents=true, so no AgentResponseEvent is emitted, and the
// ExecutorCompletedEvent for the host carries null Data. That is the scenario
// where BuildOverallItem returns null. When the caller asks for an overall
// evaluation (includeOverall: true), we should fail fast rather than silently
// returning empty results — regardless of whether expectedOutput was supplied.
var agent = new TestEchoAgent(name: "echo");
var workflow = AgentWorkflowBuilder.BuildSequential(agent);
var input = new List<ChatMessage> { new(ChatRole.User, "Hello") };
var evaluator = new LocalEvaluator(
FunctionEvaluator.Create("noop", (EvalItem _) => true));
await using var run = await InProcessExecution.RunAsync(workflow, input);
// Act + Assert — throws even without expectedOutput
var ex = await Assert.ThrowsAsync<InvalidOperationException>(() =>
run.EvaluateAsync(
evaluator,
includeOverall: true,
includePerAgent: false));
Assert.Contains("EmitAgentResponseEvents", ex.Message);
}
// ---------------------------------------------------------------
// EvaluateAsync integration test
// ---------------------------------------------------------------
-24
View File
@@ -99,30 +99,6 @@ The `AGUIChatClient` supports:
- Integration with `Agent` for client-side history management
- Interrupt metadata passthrough (`availableInterrupts` and `resume`)
## Tool Return Helpers
Use `state_update` when a backend tool needs to send different payloads to the model, the UI, and shared state. The `text` value remains the LLM-bound tool result, `tool_result` becomes the AG-UI `ToolCallResultEvent.content` for frontend rendering, and `state` is merged into durable shared state.
```python
from agent_framework import Content, tool
from agent_framework.ag_ui import state_update
@tool
async def get_weather(city: str) -> Content:
data = await fetch_weather(city)
return state_update(
text=f"{city}: {data['temp']}°C and {data['conditions']}",
tool_result={
"component": "weather-card",
"city": city,
"temperature": data["temp"],
"conditions": data["conditions"],
"humidity": data["humidity"],
},
state={"weather": {"city": city, **data}},
)
```
## Documentation
- **[Getting Started Tutorial](getting_started/)** - Step-by-step guide to building AG-UI servers and clients
@@ -49,11 +49,8 @@ from ._run_common import (
_close_reasoning_block, # type: ignore
_emit_content, # type: ignore
_extract_resume_payload, # type: ignore
_extract_tool_result_display, # type: ignore
_has_only_tool_calls, # type: ignore
_normalize_resume_interrupts, # type: ignore
_resolve_ui_payload, # type: ignore
_stringify_tool_result, # type: ignore
)
from ._utils import (
convert_agui_tools_to_agent_framework,
@@ -384,23 +381,17 @@ def _handle_step_based_approval(messages: list[Any]) -> list[BaseEvent]:
def _make_approval_tool_result_events(resolved_approval_results: list[Content]) -> list[ToolCallResultEvent]:
"""Build TOOL_CALL_RESULT events for tools executed during approval resolution.
Honors ``TOOL_RESULT_DISPLAY_KEY`` so tools returning
``state_update(..., tool_result=...)`` route the display payload to the UI
event even when gated by HITL approval.
"""
"""Build TOOL_CALL_RESULT events for tools executed during approval resolution."""
events: list[ToolCallResultEvent] = []
for resolved in resolved_approval_results:
if resolved.call_id:
raw = resolved.result if resolved.result is not None else ""
llm_str = _stringify_tool_result(raw)
ui_str = _resolve_ui_payload(llm_str, _extract_tool_result_display(resolved))
result_str = raw if isinstance(raw, str) else json.dumps(make_json_safe(raw))
events.append(
ToolCallResultEvent(
message_id=generate_event_id(),
tool_call_id=resolved.call_id,
content=ui_str,
content=result_str,
role="tool",
)
)
@@ -32,14 +32,11 @@ from ag_ui.core import (
from agent_framework import Content
from ._orchestration._predictive_state import PredictiveStateHandler
from ._state import TOOL_RESULT_DISPLAY_KEY, TOOL_RESULT_STATE_KEY
from ._state import TOOL_RESULT_STATE_KEY
from ._utils import generate_event_id, make_json_safe
logger = logging.getLogger(__name__)
# Sentinel for an unset display_result; distinguishes "caller didn't pass" from None/{}/"".
_UNSET = object()
def _has_only_tool_calls(contents: list[Any]) -> bool:
"""Check if contents have only tool calls (no text)."""
@@ -238,22 +235,6 @@ def _emit_tool_call(
return events
def _extract_tool_result_marker_values(content: Content, key: str) -> list[Any]:
"""Extract marker values from outer and inner tool-result content."""
values: list[Any] = []
outer_ap = getattr(content, "additional_properties", None) or {}
if key in outer_ap:
values.append(outer_ap[key])
for item in content.items or ():
item_ap = getattr(item, "additional_properties", None) or {}
if key in item_ap:
values.append(item_ap[key])
return values
def _extract_tool_result_state(content: Content) -> dict[str, Any] | None:
"""Extract a deterministic AG-UI state update from a tool-result ``Content``.
@@ -271,7 +252,14 @@ def _extract_tool_result_state(content: Content) -> dict[str, Any] | None:
"""
merged: dict[str, Any] | None = None
for item_state in _extract_tool_result_marker_values(content, TOOL_RESULT_STATE_KEY):
outer_ap = getattr(content, "additional_properties", None) or {}
outer_state = outer_ap.get(TOOL_RESULT_STATE_KEY)
if isinstance(outer_state, dict):
merged = dict(outer_state)
for item in content.items or ():
item_ap = getattr(item, "additional_properties", None) or {}
item_state = item_ap.get(TOOL_RESULT_STATE_KEY)
if isinstance(item_state, dict):
if merged is None:
merged = dict(item_state)
@@ -281,21 +269,6 @@ def _extract_tool_result_state(content: Content) -> dict[str, Any] | None:
return merged
def _extract_tool_result_display(content: Content) -> Any: # noqa: ANN401
"""Extract a UI-only AG-UI tool result display payload, if present."""
display_values = _extract_tool_result_marker_values(content, TOOL_RESULT_DISPLAY_KEY)
return display_values[-1] if display_values else _UNSET
def _stringify_tool_result(raw_result: Any) -> str: # noqa: ANN401
return raw_result if isinstance(raw_result, str) else json.dumps(make_json_safe(raw_result))
def _resolve_ui_payload(llm_str: str, display_result: Any) -> str: # noqa: ANN401
"""Pick the UI-bound string: the serialized display payload when set, else the LLM string."""
return llm_str if display_result is _UNSET else _stringify_tool_result(display_result)
def _emit_tool_result_common(
call_id: str,
raw_result: Any,
@@ -303,7 +276,6 @@ def _emit_tool_result_common(
predictive_handler: PredictiveStateHandler | None = None,
*,
state_update: Mapping[str, Any] | None = None,
display_result: Any = _UNSET, # noqa: ANN401
) -> list[BaseEvent]:
"""Shared helper for emitting ToolCallEnd + ToolCallResult events and performing FlowState cleanup.
@@ -329,14 +301,13 @@ def _emit_tool_result_common(
events.append(ToolCallEndEvent(tool_call_id=call_id))
flow.tool_calls_ended.add(call_id)
result_content = _stringify_tool_result(raw_result)
ui_result_content = _resolve_ui_payload(result_content, display_result)
result_content = raw_result if isinstance(raw_result, str) else json.dumps(make_json_safe(raw_result))
message_id = generate_event_id()
events.append(
ToolCallResultEvent(
message_id=message_id,
tool_call_id=call_id,
content=ui_result_content,
content=result_content,
role="tool",
)
)
@@ -387,14 +358,12 @@ def _emit_tool_result(
return []
raw_result = content.result if content.result is not None else ""
state_update = _extract_tool_result_state(content)
display_result = _extract_tool_result_display(content)
return _emit_tool_result_common(
content.call_id,
raw_result,
flow,
predictive_handler,
state_update=state_update,
display_result=display_result,
)
@@ -561,14 +530,12 @@ def _emit_mcp_tool_result(
return []
raw_output = content.output if content.output is not None else ""
state_update = _extract_tool_result_state(content)
display_result = _extract_tool_result_display(content)
return _emit_tool_result_common(
content.call_id,
raw_output,
flow,
predictive_handler,
state_update=state_update,
display_result=display_result,
)
@@ -1,13 +1,12 @@
# Copyright (c) Microsoft. All rights reserved.
"""Deterministic tool-driven AG-UI state updates and display payloads.
"""Deterministic tool-driven AG-UI state updates.
Tools wired into the :mod:`agent_framework_ag_ui` endpoint can push a
deterministic state update or a per-call tool result display payload by
returning :func:`state_update`. Unlike ``predict_state_config`` — which emits
``StateDeltaEvent``s optimistically from LLM-predicted tool call arguments —
``state_update`` runs *after* the tool executes, so AG-UI state and display
content always reflect the tool's actual return value.
deterministic state update by returning :func:`state_update`. Unlike
``predict_state_config`` — which emits ``StateDeltaEvent``s optimistically from
LLM-predicted tool call arguments — ``state_update`` runs *after* the tool
executes, so the AG-UI state always reflects the tool's actual return value.
See issue https://github.com/microsoft/agent-framework/issues/3167 for the
motivating discussion.
@@ -15,48 +14,33 @@ motivating discussion.
from __future__ import annotations
import json
from collections.abc import Mapping
from typing import Any
from agent_framework import Content
from ._utils import make_json_safe
__all__ = ["TOOL_RESULT_DISPLAY_KEY", "TOOL_RESULT_STATE_KEY", "state_update"]
__all__ = ["TOOL_RESULT_STATE_KEY", "state_update"]
TOOL_RESULT_STATE_KEY = "__ag_ui_tool_result_state__"
"""Reserved ``Content.additional_properties`` key used to carry a tool-driven
state snapshot from a tool return value through to the AG-UI emitter."""
TOOL_RESULT_DISPLAY_KEY = "__ag_ui_tool_result_display__"
"""Reserved ``Content.additional_properties`` key used to carry UI-only tool result display content from a tool return value through to the AG-UI emitter."""
_UNSET = object()
def _serialize_tool_result(value: Any) -> str: # noqa: ANN401
return value if isinstance(value, str) else json.dumps(make_json_safe(value))
def state_update(
text: str = "",
*,
state: Mapping[str, Any] | None = None,
tool_result: Any = _UNSET, # noqa: ANN401
state: Mapping[str, Any],
) -> Content:
"""Build a tool return value that updates AG-UI shared state or display content.
"""Build a tool return value that deterministically updates AG-UI shared state.
Return the result of this helper from an agent tool to push a state update
or UI-only display payload to AG-UI clients using the actual tool output,
rather than LLM-predicted tool arguments.
to AG-UI clients using the actual tool output, rather than LLM-predicted
tool arguments.
When the AG-UI endpoint emits the tool result, it will:
* Forward ``text`` to the LLM as the normal ``function_result`` content.
* Use ``tool_result`` as the ``ToolCallResultEvent.content`` payload shown
to AG-UI clients, falling back to ``text`` when no display payload is set.
* Merge ``state`` into ``FlowState.current_state``.
* Emit a deterministic ``StateSnapshotEvent`` after the ``ToolCallResult``
event so frontends observe the updated state deterministically. If
@@ -65,7 +49,7 @@ def state_update(
Example:
.. code-block:: python
from agent_framework import Content, tool
from agent_framework import tool
from agent_framework_ag_ui import state_update
@@ -77,61 +61,24 @@ def state_update(
state={"weather": {"city": city, **data}},
)
Example:
.. code-block:: python
from agent_framework import Content, tool
from agent_framework_ag_ui import state_update
@tool
async def get_weather(city: str) -> Content:
data = await _fetch_weather(city)
return state_update(
text=f"{city}: {data['temp']}°C and {data['conditions']}",
tool_result={
"component": "weather-card",
"city": city,
"temperature": data["temp"],
"conditions": data["conditions"],
"humidity": data["humidity"],
},
state={"weather": {"city": city, **data}},
)
Args:
text: Text passed back to the LLM as the ``function_result`` content.
Defaults to an empty string for tools whose only output is a state
update.
state: A mapping merged into the AG-UI shared state via JSON-compatible
``dict.update`` semantics. Nested dicts are replaced, not deep-merged.
tool_result: JSON-safe payload emitted to AG-UI clients as
``ToolCallResultEvent.content`` for frontend rendering. The LLM
still receives ``text``. If ``text`` is empty, the serialized
display payload is also used as the LLM-bound text fallback.
Returns:
A ``Content`` object with ``type="text"``. The state payload rides in
``additional_properties`` under :data:`TOOL_RESULT_STATE_KEY`
(``"__ag_ui_tool_result_state__"``), and the display payload rides
under :data:`TOOL_RESULT_DISPLAY_KEY`
(``"__ag_ui_tool_result_display__"``). Both reserved keys are extracted
by the AG-UI emitter.
``additional_properties`` under :data:`TOOL_RESULT_STATE_KEY` and is
extracted by the AG-UI emitter.
Raises:
TypeError: If ``state`` is not a ``Mapping``.
"""
if state is not None and not isinstance(state, Mapping):
if not isinstance(state, Mapping):
raise TypeError(f"state_update() 'state' must be a Mapping, got {type(state).__name__}")
additional_properties: dict[str, Any] = {}
if state is not None:
additional_properties[TOOL_RESULT_STATE_KEY] = dict(state)
if tool_result is not _UNSET:
display_content = _serialize_tool_result(tool_result)
additional_properties[TOOL_RESULT_DISPLAY_KEY] = display_content
if not text:
text = display_content
return Content.from_text(
text,
additional_properties=additional_properties,
additional_properties={TOOL_RESULT_STATE_KEY: dict(state)},
)
@@ -68,19 +68,6 @@ def _tool_result_with_state(call_id: str, text: str, state: dict[str, Any]) -> A
)
def _tool_result_with_display(call_id: str, text: str, tool_result: Any, **kwargs: Any) -> AgentResponseUpdate:
"""Build a function_result update carrying an optional UI display marker."""
return AgentResponseUpdate(
contents=[
Content.from_function_result(
call_id=call_id,
result=[state_update(text=text, tool_result=tool_result, **kwargs)],
)
],
role="assistant",
)
# ── Golden stream tests ──
@@ -278,87 +265,3 @@ async def test_deterministic_state_coexists_with_predict_state_config() -> None:
# The final observed state must contain both the deterministic and predictive contributions.
final = stream.snapshot()
assert final["weather"] == {"city": "SF", "temp": 14}, f"Deterministic state missing from final snapshot: {final}"
async def test_tool_result_display_payload_reaches_ui_event_only() -> None:
"""Rich display payload overrides TOOL_CALL_RESULT without leaking marker keys."""
updates = [
_tool_call("call-1", "get_weather", '{"city": "SF"}'),
_tool_result_with_display(
"call-1",
text="Weather in SF: 14°C foggy",
tool_result={"city": "SF", "temp": 14, "conditions": "foggy"},
),
]
agent = _build_agent(updates)
stream = await _run(agent, PAYLOAD)
stream.assert_bookends()
stream.assert_no_run_error()
stream.assert_tool_calls_balanced()
result = stream.first("TOOL_CALL_RESULT")
assert result.content == '{"city": "SF", "temp": 14, "conditions": "foggy"}'
assert "__ag_ui_tool_result_display__" not in result.content
assert "__ag_ui_tool_result_state__" not in result.content
async def test_tool_result_display_falls_back_to_text_when_unset() -> None:
"""Without a display marker, the UI event keeps the existing text content."""
updates = [
_tool_call("call-1", "get_weather", '{"city": "SF"}'),
_tool_result_with_state(
"call-1",
text="Weather in SF: 14°C foggy",
state={"weather": {"city": "SF", "temp": 14}},
),
]
agent = _build_agent(updates)
stream = await _run(agent, PAYLOAD)
stream.assert_bookends()
stream.assert_no_run_error()
stream.assert_tool_calls_balanced()
result = stream.first("TOOL_CALL_RESULT")
assert result.content == "Weather in SF: 14°C foggy"
assert "__ag_ui_tool_result_display__" not in result.content
assert "__ag_ui_tool_result_state__" not in result.content
async def test_tool_result_display_coexists_with_state_snapshot() -> None:
"""Display and durable state markers produce one deterministic state snapshot."""
updates = [
_tool_call("call-1", "get_weather", '{"city": "SF"}'),
_tool_result_with_display(
"call-1",
text="Weather in SF: 14°C foggy",
tool_result={"city": "SF", "temp": 14, "conditions": "foggy"},
state={"weather": {"city": "SF", "temp": 14, "conditions": "foggy"}},
),
]
agent = _build_agent(updates)
stream = await _run(agent, PAYLOAD)
stream.assert_bookends()
stream.assert_no_run_error()
stream.assert_tool_calls_balanced()
stream.assert_ordered_types(["TOOL_CALL_RESULT", "STATE_SNAPSHOT", "RUN_FINISHED"])
result = stream.first("TOOL_CALL_RESULT")
assert result.content == '{"city": "SF", "temp": 14, "conditions": "foggy"}'
result_idx = stream.events.index(result)
deterministic_snapshots = [
event
for event in stream.events[result_idx + 1 :]
if getattr(getattr(event, "type", None), "value", getattr(event, "type", None)) == "STATE_SNAPSHOT"
]
assert len(deterministic_snapshots) == 1
assert deterministic_snapshots[0].snapshot["weather"] == {
"city": "SF",
"temp": 14,
"conditions": "foggy",
}
assert "__ag_ui_tool_result_display__" not in str(deterministic_snapshots[0].snapshot)
assert "__ag_ui_tool_result_state__" not in str(deterministic_snapshots[0].snapshot)
@@ -448,37 +448,3 @@ async def test_resolve_approval_responses_returns_only_approved() -> None:
rejection_results = [c for c in all_contents if c.type == "function_result" and c.call_id == rejected_call_id]
assert len(rejection_results) == 1
assert "rejected" in str(rejection_results[0].result).lower()
class TestApprovalToolResultDisplayChannel:
"""Approved tools using ``state_update(..., tool_result=...)`` must route the
display payload to the UI event while ``flow.tool_results`` still receives
the LLM-bound text. The HITL approval emitter is separate from the standard
streaming emitter, so it gets its own coverage.
"""
def test_approval_emits_display_payload_when_marker_present(self) -> None:
from agent_framework_ag_ui import state_update
from agent_framework_ag_ui._agent_run import _make_approval_tool_result_events
display_payload = {"city": "Seattle", "temp": 14, "conditions": "foggy"}
inner = state_update(text="14°C, foggy", tool_result=display_payload)
resolved = Content.from_function_result(call_id="call_disp", result=[inner])
events = _make_approval_tool_result_events([resolved])
assert len(events) == 1
# UI event must carry the serialized display payload, NOT the LLM text.
assert json.loads(events[0].content) == display_payload
assert events[0].content != "14°C, foggy"
def test_approval_falls_back_to_text_when_no_marker(self) -> None:
"""Backward compat: without a display marker, behaviour is unchanged."""
from agent_framework_ag_ui._agent_run import _make_approval_tool_result_events
resolved = Content.from_function_result(call_id="call_plain", result="Sunny in Seattle")
events = _make_approval_tool_result_events([resolved])
assert len(events) == 1
assert events[0].content == "Sunny in Seattle"
@@ -15,7 +15,7 @@ from agent_framework_ag_ui._run_common import (
_extract_tool_result_state,
_normalize_resume_interrupts,
)
from agent_framework_ag_ui._state import TOOL_RESULT_DISPLAY_KEY, TOOL_RESULT_STATE_KEY
from agent_framework_ag_ui._state import TOOL_RESULT_STATE_KEY
class TestNormalizeResumeInterrupts:
@@ -140,15 +140,6 @@ class TestStateUpdateHelper:
TOOL_RESULT_STATE_KEY: {"weather": {"temp": 14}},
}
def test_builds_text_content_with_display_marker(self):
"""state_update can carry a UI display payload without requiring state."""
c = state_update(text="14°C, foggy", tool_result={"temp": 14, "conditions": "foggy"})
assert c.type == "text"
assert c.text == "14°C, foggy"
assert c.additional_properties == {
TOOL_RESULT_DISPLAY_KEY: '{"temp": 14, "conditions": "foggy"}',
}
def test_empty_text_is_allowed(self):
"""State-only tools can omit the text argument."""
c = state_update(state={"steps": ["a", "b"]})
@@ -174,18 +165,6 @@ class TestStateUpdateHelper:
inner = c.additional_properties[TOOL_RESULT_STATE_KEY]
assert inner is not caller_state
def test_tool_result_without_text_falls_back_to_display_payload(self):
"""Display-only tools use the serialized display payload as LLM text."""
c = state_update(tool_result={"temp": 14, "conditions": "foggy"})
assert c.text == '{"temp": 14, "conditions": "foggy"}'
assert c.additional_properties[TOOL_RESULT_DISPLAY_KEY] == '{"temp": 14, "conditions": "foggy"}'
def test_string_tool_result_is_not_json_encoded_again(self):
"""A pre-serialized display string passes through verbatim."""
c = state_update(text="Weather summary", tool_result='{"temp":14}')
assert c.text == "Weather summary"
assert c.additional_properties[TOOL_RESULT_DISPLAY_KEY] == '{"temp":14}'
class TestExtractToolResultState:
"""Tests for ``_extract_tool_result_state``."""
@@ -286,60 +265,6 @@ class TestEmitToolResultWithState:
assert result_events[0].content == "Weather: 14°C"
assert TOOL_RESULT_STATE_KEY not in result_events[0].content
def test_display_payload_routes_to_ui_only(self):
"""A display marker overrides only the UI event, not the LLM-bound tool result."""
tool_return = state_update(
text="Weather: 14°C",
tool_result={"temp": 14, "conditions": "foggy"},
)
content = Content.from_function_result(call_id="c1", result=[tool_return])
flow = FlowState()
events = _emit_tool_result(content, flow)
result_events = [e for e in events if e.type == EventType.TOOL_CALL_RESULT]
assert len(result_events) == 1
assert result_events[0].content == '{"temp": 14, "conditions": "foggy"}'
assert flow.tool_results[-1]["content"] == "Weather: 14°C"
assert TOOL_RESULT_DISPLAY_KEY not in result_events[0].content
assert TOOL_RESULT_DISPLAY_KEY not in flow.tool_results[-1]["content"]
def test_plain_tool_result_uses_existing_content_for_both_channels(self):
"""Without a display marker, UI and LLM channels keep the existing derivation."""
content = Content.from_function_result(call_id="c1", result="plain result")
flow = FlowState()
events = _emit_tool_result(content, flow)
result_events = [e for e in events if e.type == EventType.TOOL_CALL_RESULT]
assert len(result_events) == 1
assert result_events[0].content == "plain result"
assert flow.tool_results[-1]["content"] == "plain result"
def test_display_only_payload_falls_back_to_llm_content(self):
"""When text is empty, both channels receive the serialized display payload."""
tool_return = state_update(tool_result={"temp": 14})
content = Content.from_function_result(call_id="c1", result=[tool_return])
flow = FlowState()
events = _emit_tool_result(content, flow)
result_events = [e for e in events if e.type == EventType.TOOL_CALL_RESULT]
assert result_events[0].content == '{"temp": 14}'
assert flow.tool_results[-1]["content"] == '{"temp": 14}'
def test_pre_serialized_display_string_routes_verbatim(self):
"""String display payloads pass through without JSON double-encoding."""
tool_return = state_update(text="Weather summary", tool_result='{"temp":14}')
content = Content.from_function_result(call_id="c1", result=[tool_return])
flow = FlowState()
events = _emit_tool_result(content, flow)
result_events = [e for e in events if e.type == EventType.TOOL_CALL_RESULT]
assert result_events[0].content == '{"temp":14}'
assert flow.tool_results[-1]["content"] == "Weather summary"
def test_coexists_with_active_predictive_state_handler(self):
"""Both predictive and deterministic state produce a single coalesced snapshot.
@@ -421,31 +346,3 @@ class TestEmitMcpToolResultWithState:
events = _emit_mcp_tool_result(content, flow)
assert all(e.type != EventType.STATE_SNAPSHOT for e in events)
class TestEmitMcpToolResultWithDisplay:
"""MCP tool results must honour the display marker so UI consumers can
render structured payloads while ``flow.tool_results`` keeps the LLM
string. MCP outputs do not pass through ``parse_result``; the marker
rides on the outer content's ``additional_properties``.
"""
def test_mcp_tool_result_routes_display_payload_to_ui_only(self):
import json as _json
display_payload = {"rows": [{"id": 1, "name": "alpha"}, {"id": 2, "name": "beta"}]}
content = Content.from_mcp_server_tool_result(
call_id="mcp_disp",
output="2 rows returned",
additional_properties={TOOL_RESULT_DISPLAY_KEY: display_payload},
)
flow = FlowState()
events = _emit_mcp_tool_result(content, flow)
result_events = [e for e in events if e.type == EventType.TOOL_CALL_RESULT]
assert len(result_events) == 1
# UI event carries the structured display payload.
assert _json.loads(result_events[0].content) == display_payload
# LLM-side accumulator keeps the short text.
assert flow.tool_results[-1]["content"] == "2 rows returned"
+1 -2
View File
@@ -69,8 +69,7 @@ agent_framework/
### Skills (`_skills.py`)
- **`Skill`** - Abstract base for a skill definition bundling instructions (`content`) with frontmatter metadata, resources, and scripts. Concrete subclasses (`InlineSkill`, `FileSkill`, `ClassSkill`) accept a `frontmatter=SkillFrontmatter(...)` argument carrying the spec fields. Adding new spec fields is done in one place — on `SkillFrontmatter` — keeping the subclass constructors stable.
- **`SkillFrontmatter`** - L1 discovery metadata for a skill (`name`, `description`, `license`, `compatibility`, `allowed_tools`, `metadata`). All fields are mutable plain attributes; the constructor validates `name`, `description`, and `compatibility` against the spec but post-construction assignments are not re-validated. Spec fields are reachable on every skill via `skill.frontmatter`.
- **`Skill`** - A skill definition bundling instructions (`content`) with metadata, resources, and scripts. Supports `@skill.resource` and `@skill.script` decorators for adding components.
- **`SkillResource`** - Named supplementary content attached to a skill; holds either static `content` or a dynamic `function` (sync or async). Exactly one must be provided.
- **`SkillScript`** - An executable script attached to a skill; holds either an inline `function` (code-defined, runs in-process) or a `path` to a file on disk (file-based, delegated to a runner). Exactly one must be provided.
- **`SkillScriptRunner`** - Protocol for file-based script execution. Any callable matching `(skill, script, args) -> Any` satisfies it. Code-defined scripts do not use a runner.
@@ -147,7 +147,6 @@ from ._skills import (
InlineSkillScript,
InMemorySkillsSource,
Skill,
SkillFrontmatter,
SkillResource,
SkillScript,
SkillScriptRunner,
@@ -433,7 +432,6 @@ __all__ = [
"SessionContext",
"SingleEdgeGroup",
"Skill",
"SkillFrontmatter",
"SkillResource",
"SkillScript",
"SkillScriptRunner",
+3 -47
View File
@@ -10,7 +10,7 @@ import logging
import re
import sys
from abc import abstractmethod
from collections.abc import Callable, Collection, Coroutine, Sequence
from collections.abc import Callable, Collection, Sequence
from contextlib import AsyncExitStack, _AsyncGeneratorContextManager # type: ignore
from datetime import timedelta
from functools import partial
@@ -264,7 +264,6 @@ class MCPTool:
self.is_connected: bool = False
self._tools_loaded: bool = False
self._prompts_loaded: bool = False
self._pending_reload_tasks: set[asyncio.Task[None]] = set()
def __str__(self) -> str:
return f"MCPTool(name={self.name}, description={self.description})"
@@ -906,47 +905,12 @@ class MCPTool:
if isinstance(message, types.ServerNotification):
match message.root.method:
case "notifications/tools/list_changed":
self._schedule_reload(self.load_tools())
await self.load_tools()
case "notifications/prompts/list_changed":
self._schedule_reload(self.load_prompts())
await self.load_prompts()
case _:
logger.debug("Unhandled notification: %s", message.root.method)
def _schedule_reload(self, coro: Coroutine[Any, Any, None]) -> None:
"""Schedule a reload coroutine as a background task.
Reloads (load_tools / load_prompts) triggered by MCP server
notifications must NOT be awaited inside the message handler because
the handler runs on the MCP SDK's single-threaded receive loop.
Awaiting a session request (e.g. ``list_tools``) from within that loop
deadlocks: the receive loop cannot read the response while it is
blocked waiting for the handler to return.
Instead we fire the reload as an independent ``asyncio.Task`` and keep
a strong reference in ``_pending_reload_tasks`` so it is not garbage-
collected before completion. Only one reload per kind (tools / prompts)
is kept in flight; a new notification cancels the previous pending task
for the same coroutine name to avoid unbounded growth.
"""
# Cancel-and-replace: only one reload per kind should be in flight.
reload_name = f"mcp-reload:{self.name}:{coro.__qualname__}"
for existing in list(self._pending_reload_tasks):
if existing.get_name() == reload_name and not existing.done():
logger.debug("Cancelling in-flight reload %s; superseded by new notification", reload_name)
existing.cancel()
async def _safe_reload() -> None:
try:
await coro
except asyncio.CancelledError:
raise
except Exception:
logger.warning("Background MCP reload failed", exc_info=True)
task = asyncio.create_task(_safe_reload(), name=reload_name)
self._pending_reload_tasks.add(task)
task.add_done_callback(self._pending_reload_tasks.discard)
def _determine_approval_mode(
self,
*candidate_names: str,
@@ -1083,14 +1047,6 @@ class MCPTool:
params = types.PaginatedRequestParams(cursor=tool_list.nextCursor)
async def _close_on_owner(self) -> None:
# Cancel any pending reload tasks before tearing down the session.
tasks = list(self._pending_reload_tasks)
for task in tasks:
task.cancel()
self._pending_reload_tasks.clear()
if tasks:
await asyncio.gather(*tasks, return_exceptions=True)
await self._safe_close_exit_stack()
self._exit_stack = AsyncExitStack()
self.session = None
File diff suppressed because it is too large Load Diff
+1 -111
View File
@@ -1,7 +1,6 @@
# Copyright (c) Microsoft. All rights reserved.
# type: ignore[reportPrivateUsage]
import asyncio
import contextlib
import json
import logging
import os
@@ -1616,7 +1615,7 @@ async def test_mcp_connection_reset_integration():
async def test_mcp_tool_message_handler_notification():
"""Test that message_handler correctly processes tools/list_changed and prompts/list_changed
notifications by scheduling reloads as background tasks."""
notifications."""
tool = MCPStdioTool(name="test_tool", command="python")
# Mock the load_tools and load_prompts methods
@@ -1630,8 +1629,6 @@ async def test_mcp_tool_message_handler_notification():
result = await tool.message_handler(tools_notification)
assert result is None
# The reload is scheduled as a background task; let it run.
await asyncio.sleep(0)
tool.load_tools.assert_called_once()
# Reset mock
@@ -1644,7 +1641,6 @@ async def test_mcp_tool_message_handler_notification():
result = await tool.message_handler(prompts_notification)
assert result is None
await asyncio.sleep(0)
tool.load_prompts.assert_called_once()
# Test unhandled notification
@@ -1668,112 +1664,6 @@ async def test_mcp_tool_message_handler_error():
assert result is None
async def test_mcp_tool_message_handler_does_not_block_receive_loop():
"""Test that message_handler does not deadlock the MCP receive loop.
Regression test for https://github.com/microsoft/agent-framework/issues/4828.
When the MCP server sends a ``notifications/tools/list_changed``
notification, the handler must NOT await ``load_tools()`` synchronously
because that would block the single-threaded MCP receive loop, preventing
it from delivering the ``list_tools`` response — a classic deadlock.
"""
tool = MCPStdioTool(name="test_tool", command="python")
# Use an event to make load_tools block until we release it.
# This simulates load_tools waiting for a session response that the
# receive loop would need to deliver.
release = asyncio.Event()
async def slow_load_tools():
await release.wait()
tool.load_tools = slow_load_tools # type: ignore[assignment]
tools_notification = Mock(spec=types.ServerNotification)
tools_notification.root = Mock()
tools_notification.root.method = "notifications/tools/list_changed"
# message_handler must return immediately even though load_tools blocks.
await tool.message_handler(tools_notification)
# If the handler had awaited load_tools synchronously, we would never
# reach this line (deadlock). Verify the reload task is pending.
assert len(tool._pending_reload_tasks) == 1
# Unblock the reload so the background task finishes cleanly.
release.set()
# Wait for the pending reload task(s) to complete so their done-callbacks
# have a chance to remove them from _pending_reload_tasks.
await asyncio.wait_for(asyncio.gather(*tool._pending_reload_tasks), timeout=1)
assert len(tool._pending_reload_tasks) == 0
async def test_mcp_tool_message_handler_reload_failure_is_logged(caplog: pytest.LogCaptureFixture):
"""Background reload errors are logged, not raised into the receive loop."""
tool = MCPStdioTool(name="test_tool", command="python")
tool.load_tools = AsyncMock(side_effect=RuntimeError("connection lost"))
tools_notification = Mock(spec=types.ServerNotification)
tools_notification.root = Mock()
tools_notification.root.method = "notifications/tools/list_changed"
await tool.message_handler(tools_notification)
# Let the background task run — it should not propagate the exception.
# Snapshot tasks and await them to ensure done-callbacks fire.
pending = list(tool._pending_reload_tasks)
if pending:
await asyncio.wait_for(asyncio.gather(*pending, return_exceptions=True), timeout=1)
tool.load_tools.assert_called_once()
assert len(tool._pending_reload_tasks) == 0
# Verify the warning was actually logged with exception info.
reload_warnings = [r for r in caplog.records if "Background MCP reload failed" in r.message]
assert len(reload_warnings) == 1
assert reload_warnings[0].levelname == "WARNING"
assert reload_warnings[0].exc_info is not None
async def test_mcp_tool_message_handler_cancel_and_replace():
"""Sending two notifications in quick succession cancels the first reload task."""
tool = MCPStdioTool(name="test_tool", command="python")
release = asyncio.Event()
call_count = 0
async def blocking_load_tools():
nonlocal call_count
call_count += 1
await release.wait()
tool.load_tools = blocking_load_tools # type: ignore[assignment]
notification = Mock(spec=types.ServerNotification)
notification.root = Mock()
notification.root.method = "notifications/tools/list_changed"
# First notification — starts a blocking reload task.
await tool.message_handler(notification)
assert len(tool._pending_reload_tasks) == 1
first_task = next(iter(tool._pending_reload_tasks))
# Second notification — should cancel the first and replace it.
await tool.message_handler(notification)
# Yield to the event loop so the cancellation is processed.
with contextlib.suppress(asyncio.CancelledError):
await first_task
assert first_task.cancelled()
assert len(tool._pending_reload_tasks) == 1
second_task = next(iter(tool._pending_reload_tasks))
assert second_task is not first_task
# Unblock and let the second task finish.
release.set()
await asyncio.wait_for(asyncio.gather(*tool._pending_reload_tasks), timeout=1)
assert len(tool._pending_reload_tasks) == 0
async def test_mcp_tool_sampling_callback_no_client():
"""Test sampling callback error path when no chat client is available."""
tool = MCPStdioTool(name="test_tool", command="python")
File diff suppressed because it is too large Load Diff
@@ -429,12 +429,7 @@ class _FullHistoryReplayCoordinator(Executor):
@pytest.mark.xfail(
reason=(
"Tracks the executor-layer half of #3295: AgentExecutor should clear service_session_id "
"when handed a full prior conversation. The wire-level 'Duplicate item' API error is "
"already closed by the chat-client strip in #3295; this xfail covers the defense-in-depth "
"follow-up that makes the executor wiring reflect intent."
),
reason="reset_service_session support not yet implemented — see #4047",
strict=True,
)
async def test_run_request_with_full_history_clears_service_session_id() -> None:
@@ -96,7 +96,7 @@ def serve(
ui_enabled: bool = True,
instrumentation_enabled: bool = False,
mode: str = "developer",
auth_enabled: bool = True,
auth_enabled: bool = False,
auth_token: str | None = None,
) -> None:
"""Launch Agent Framework DevUI with simple API.
@@ -126,29 +126,52 @@ def serve(
if not isinstance(port, int) or not (1 <= port <= 65535):
raise ValueError(f"Invalid port: {port}. Must be integer between 1 and 65535")
# Security check: warn loudly when network-exposed without authentication.
# Security check: Warn if network-exposed without authentication
if host not in ("127.0.0.1", "localhost") and not auth_enabled:
logger.warning("WARNING: Exposing DevUI to the network with --no-auth.")
logger.warning("Anyone on your network can read agent metadata and trigger requests.")
logger.warning("Drop --no-auth and DevUI will require Bearer tokens.")
logger.warning("⚠️ WARNING: Exposing DevUI to network without authentication!")
logger.warning("⚠️ This is INSECURE - anyone on your network can access your agents")
logger.warning("đź’ˇ For network exposure, add --auth flag: devui --host 0.0.0.0 --auth")
# Refuse to auto-generate a token for network-exposed binds. Auto-generated tokens
# are fine for localhost convenience; for anything else, require an explicit token.
if auth_enabled and not auth_token:
# Handle authentication configuration
if auth_enabled:
import os
import secrets
env_token = os.environ.get("DEVUI_AUTH_TOKEN")
if not env_token:
# Check if token is in environment variable first
if not auth_token:
auth_token = os.environ.get("DEVUI_AUTH_TOKEN")
# Auto-generate token if STILL not provided
if not auth_token:
# Check if we're in a production-like environment
is_production = (
host not in ("127.0.0.1", "localhost")
or os.environ.get("CI") == "true"
or os.environ.get("KUBERNETES_SERVICE_HOST")
host not in ("127.0.0.1", "localhost") # Exposed to network
or os.environ.get("CI") == "true" # Running in CI
or os.environ.get("KUBERNETES_SERVICE_HOST") # Running in k8s
)
if is_production:
logger.error("Authentication required but no token provided.")
logger.error("Set DEVUI_AUTH_TOKEN env var or pass auth_token='...' to serve().")
# REFUSE to start without explicit token
logger.error("❌ Authentication enabled but no token provided")
logger.error("❌ Auto-generated tokens are NOT secure for network-exposed deployments")
logger.error("đź’ˇ Set token: export DEVUI_AUTH_TOKEN=<your-secure-token>")
logger.error("đź’ˇ Or pass: serve(entities=[...], auth_token='your-token')")
raise ValueError("DEVUI_AUTH_TOKEN required when host is not localhost")
# Development mode: auto-generate and show
auth_token = secrets.token_urlsafe(32)
logger.info("đź”’ Authentication enabled with auto-generated token")
logger.info("\n" + "=" * 70)
logger.info("🔑 DEV TOKEN (localhost only, shown once):")
logger.info(f" {auth_token}")
logger.info("=" * 70 + "\n")
else:
logger.info("đź”’ Authentication enabled with provided token")
# Set environment variable for server to use
os.environ["AUTH_REQUIRED"] = "true"
os.environ["DEVUI_AUTH_TOKEN"] = auth_token
# Enable instrumentation if requested
if instrumentation_enabled:
from agent_framework.observability import enable_instrumentation
@@ -164,8 +187,6 @@ def serve(
cors_origins=cors_origins,
ui_enabled=ui_enabled,
mode=mode,
auth_enabled=auth_enabled,
auth_token=auth_token,
)
# Register in-memory entities if provided
@@ -79,15 +79,15 @@ Examples:
)
parser.add_argument(
"--no-auth",
"--auth",
action="store_true",
help="Disable Bearer token authentication. DevUI is auth-enabled by default; use this to opt out.",
help="Enable authentication via Bearer token (required for deployed environments)",
)
parser.add_argument(
"--auth-token",
type=str,
help="Custom Bearer token. Auto-generated and logged at startup when omitted.",
help="Custom authentication token (auto-generated if not provided with --auth)",
)
parser.add_argument("--version", action="version", version=f"Agent Framework DevUI {get_version()}")
@@ -184,7 +184,7 @@ def main() -> None:
ui_enabled=ui_enabled,
instrumentation_enabled=args.instrumentation,
mode=mode,
auth_enabled=not args.no_auth,
auth_enabled=args.auth,
auth_token=args.auth_token, # Pass through explicit token only
)
@@ -75,8 +75,6 @@ class DevServer:
cors_origins: list[str] | None = None,
ui_enabled: bool = True,
mode: str = "developer",
auth_enabled: bool = True,
auth_token: str | None = None,
) -> None:
"""Initialize the development server.
@@ -87,26 +85,20 @@ class DevServer:
cors_origins: List of allowed CORS origins
ui_enabled: Whether to enable the UI
mode: Server mode - 'developer' (full access, verbose errors) or 'user' (restricted APIs, generic errors)
auth_enabled: Whether to require Bearer token auth on /v1/* endpoints. Defaults to True.
auth_token: Bearer token. If None and auth_enabled, falls back to the DEVUI_AUTH_TOKEN
environment variable, then to an auto-generated token (logged at startup).
"""
self.entities_dir = entities_dir
self.port = port
self.host = host
# CORS default is same-origin only (empty allowlist) on every host. The
# previous wildcard-on-localhost default let any webpage the developer
# visited read DevUI's responses cross-origin. Callers who need a real
# cross-origin dev frontend pass an explicit allowlist.
# Smart CORS defaults: permissive for localhost, restrictive for network-exposed deployments
if cors_origins is None:
cors_origins = []
# Localhost development: allow cross-origin for dev tools (e.g., frontend dev server)
# Network-exposed: empty list (same-origin only, no CORS)
cors_origins = ["*"] if host in ("127.0.0.1", "localhost") else []
self.cors_origins = cors_origins
self.ui_enabled = ui_enabled
self.mode = mode
self.auth_enabled = auth_enabled
self.auth_token = self._resolve_auth_token(auth_enabled, auth_token)
self.executor: AgentFrameworkExecutor | None = None
self.openai_executor: OpenAIExecutor | None = None
self.deployment_manager = DeploymentManager()
@@ -118,37 +110,6 @@ class DevServer:
"""Set in-memory entities to register on startup."""
self._pending_entities = entities
_LOOPBACK_HOSTS = frozenset({"127.0.0.1", "localhost", "[::1]", "::1"})
def _loopback_allowed_hosts(self) -> frozenset[str] | None:
"""Return the Host-header allowlist when bound to a loopback interface, else None.
Returning None disables Host-header enforcement (e.g. for 0.0.0.0 / public binds,
where the operator is intentionally exposing the service).
"""
host = self.host.lower()
if host not in self._LOOPBACK_HOSTS:
return None
return self._LOOPBACK_HOSTS
@staticmethod
def _resolve_auth_token(auth_enabled: bool, auth_token: str | None) -> str | None:
"""Resolve the active Bearer token. Returns None when auth is disabled."""
if not auth_enabled:
return None
if auth_token:
return auth_token
env_token = os.getenv("DEVUI_AUTH_TOKEN")
if env_token:
return env_token
generated = secrets.token_urlsafe(32)
logger.info("=" * 70)
logger.info("DevUI authentication enabled with auto-generated token:")
logger.info(f" {generated}")
logger.info("Pass it as: Authorization: Bearer <token>")
logger.info("=" * 70)
return generated
def _is_dev_mode(self) -> bool:
"""Check if running in developer mode.
@@ -375,11 +336,6 @@ class DevServer:
lifespan=lifespan,
)
# Middleware registration order matters: Starlette wraps later-added
# middleware around earlier-added ones, so the LAST registered runs
# outermost (sees the request first). We want Host-header enforcement
# to run before CORS/auth, so it is registered last below.
# Add CORS middleware
# Note: allow_credentials cannot be True when allow_origins is ["*"]
# For localhost dev with wildcard origins, credentials are disabled
@@ -394,24 +350,29 @@ class DevServer:
allow_headers=["*"],
)
# Bearer-token authentication. Enabled by default; opt out via
# DevServer(auth_enabled=False) for embedded/test scenarios.
if self.auth_enabled and self.auth_token:
expected_token = self.auth_token
# Add authentication middleware using decorator pattern
# Auth is enabled by presence of DEVUI_AUTH_TOKEN
auth_token = os.getenv("DEVUI_AUTH_TOKEN", "")
auth_required = bool(auth_token)
if auth_required:
logger.info("Authentication middleware enabled")
@app.middleware("http")
async def auth_middleware(request: Request, call_next: Callable[[Request], Awaitable[Any]]) -> Any:
"""Validate Bearer token authentication.
Skips authentication for health, the UI shell, static assets, and OPTIONS preflight.
Skips authentication for health, meta, static UI endpoints, and OPTIONS requests.
"""
# Skip auth for OPTIONS (CORS preflight) requests
if request.method == "OPTIONS":
return await call_next(request)
if request.url.path in ["/health", "/"] or request.url.path.startswith("/assets"):
# Skip auth for health checks, meta endpoint, and static files
if request.url.path in ["/health", "/meta", "/"] or request.url.path.startswith("/assets"):
return await call_next(request)
# Check Authorization header
auth_header = request.headers.get("Authorization")
if not auth_header or not auth_header.startswith("Bearer "):
return JSONResponse(
@@ -427,8 +388,9 @@ class DevServer:
},
)
# Extract and validate token
token = auth_header.replace("Bearer ", "", 1).strip()
if not secrets.compare_digest(token, expected_token):
if not secrets.compare_digest(token, auth_token):
return JSONResponse(
status_code=401,
content={
@@ -440,40 +402,11 @@ class DevServer:
},
)
# Token valid, proceed
return await call_next(request)
_ = auth_middleware
# Host-header allowlist for loopback binds: on a loopback interface, only
# accept requests whose Host header names a loopback address. Registered LAST
# so it runs outermost, rejecting non-loopback Host values before CORS/auth
# (and before CORS can short-circuit a preflight on a rebound request).
allowed_hosts = self._loopback_allowed_hosts()
if allowed_hosts is not None:
expected_hosts = allowed_hosts
@app.middleware("http")
async def host_header_middleware(request: Request, call_next: Callable[[Request], Awaitable[Any]]) -> Any:
host_header = request.headers.get("host", "")
hostname = host_header.split(":", 1)[0].lower()
if hostname and hostname not in expected_hosts:
return JSONResponse(
status_code=400,
content={
"error": {
"message": (
f"Invalid Host header '{host_header}'. DevUI is bound to a "
"loopback interface and only accepts requests addressed to it."
),
"type": "invalid_host",
"code": "host_not_allowed",
}
},
)
return await call_next(request)
_ = host_header_middleware
self._register_routes(app)
self._mount_ui(app)
@@ -494,6 +427,8 @@ class DevServer:
@app.get("/meta", response_model=MetaResponse)
async def get_meta() -> MetaResponse:
"""Get server metadata and configuration."""
import os
# Ensure executors are initialized to check capabilities
openai_executor = await self._ensure_openai_executor()
@@ -507,7 +442,7 @@ class DevServer:
"openai_proxy": openai_executor.is_configured,
"deployment": True, # Deployment feature is available
},
auth_required=self.auth_enabled,
auth_required=bool(os.getenv("DEVUI_AUTH_TOKEN")),
)
@app.get("/v1/entities", response_model=DiscoveryResponse)
@@ -815,6 +750,7 @@ class DevServer:
headers={
"Cache-Control": "no-cache",
"Connection": "keep-alive",
"Access-Control-Allow-Origin": "*",
},
)
return await openai_executor.execute_sync(request)
@@ -858,6 +794,7 @@ class DevServer:
headers={
"Cache-Control": "no-cache",
"Connection": "keep-alive",
"Access-Control-Allow-Origin": "*",
"X-Response-ID": response_id, # Include ID for debugging/tracking
},
)
@@ -3,15 +3,11 @@
"""Focused tests for server functionality."""
import asyncio
import inspect
import tempfile
from pathlib import Path
import pytest
from conftest import MockAgent
from fastapi.testclient import TestClient
import agent_framework_devui
from agent_framework_devui import DevServer
from agent_framework_devui._utils import extract_executor_message_types, select_primary_input_type
from agent_framework_devui.models._openai_custom import AgentFrameworkRequest
@@ -103,11 +99,11 @@ async def test_server_execution_streaming(test_entities_dir):
def test_configuration():
"""Test basic configuration."""
server = DevServer(entities_dir="test", port=9000, host="localhost", auth_enabled=False)
server = DevServer(entities_dir="test", port=9000, host="localhost")
assert server.port == 9000
assert server.host == "localhost"
assert server.entities_dir == "test"
assert server.cors_origins == []
assert server.cors_origins == ["*"]
assert server.ui_enabled
@@ -256,18 +252,15 @@ async def test_api_restrictions_in_user_mode():
"""Test that developer APIs are restricted in user mode."""
from fastapi.testclient import TestClient
# Create servers with different modes. auth_enabled=False isolates this test
# to mode behavior — auth has its own dedicated suite.
dev_server = DevServer(mode="developer", auth_enabled=False)
user_server = DevServer(mode="user", auth_enabled=False)
# Create servers with different modes
dev_server = DevServer(mode="developer")
user_server = DevServer(mode="user")
dev_app = dev_server.create_app()
user_app = user_server.create_app()
# base_url sets the Host header to a loopback alias so the loopback
# host-header allowlist accepts the request.
dev_client = TestClient(dev_app, base_url="http://127.0.0.1")
user_client = TestClient(user_app, base_url="http://127.0.0.1")
dev_client = TestClient(dev_app)
user_client = TestClient(user_app)
# Test 1: Health endpoint should work in both modes
assert dev_client.get("/health").status_code == 200
@@ -410,171 +403,3 @@ async def test_checkpoint_api_endpoints(test_entities_dir):
# Test delete non-existent checkpoint
deleted = await storage.delete("nonexistent")
assert deleted is False
# =============================================================================
# Security posture: default CORS, auth, host-header, and streaming headers.
# =============================================================================
def _server_with_mock_agent(**kwargs) -> DevServer:
"""Build a DevServer with one in-memory mock agent registered."""
server = DevServer(**kwargs)
server.set_pending_entities([MockAgent(id="mock", name="Mock", response_text="hi")])
return server
def test_streaming_response_does_not_hardcode_acao_header():
"""A streaming /v1/responses must not set Access-Control-Allow-Origin itself.
The endpoint previously hardcoded `Access-Control-Allow-Origin: *` on the
StreamingResponse, bypassing CORSMiddleware. With no Origin header on the
request, CORSMiddleware never adds ACAO — so any ACAO we see proves the
streaming handler is still setting it.
"""
server = _server_with_mock_agent(auth_token="s3cret")
app = server.get_app()
with TestClient(app, base_url="http://127.0.0.1") as client:
response = client.post(
"/v1/responses",
json={"metadata": {"entity_id": "mock"}, "input": "hello", "stream": True},
headers={"Authorization": "Bearer s3cret"},
)
assert "access-control-allow-origin" not in {k.lower() for k in response.headers}, (
"Streaming response sets ACAO directly, bypassing CORSMiddleware"
)
def test_cors_default_does_not_allow_arbitrary_origin_even_on_localhost():
"""Default CORS must not echo Access-Control-Allow-Origin to arbitrary origins.
Previous default was `["*"]` on localhost binds, which let any webpage the
developer visited read DevUI's responses. Default is now `[]` — opt in by
passing `cors_origins=[...]` explicitly.
"""
server = _server_with_mock_agent(host="127.0.0.1", auth_token="s3cret")
app = server.get_app()
with TestClient(app, base_url="http://127.0.0.1") as client:
preflight = client.options(
"/v1/entities",
headers={
"Origin": "https://evil.example",
"Access-Control-Request-Method": "GET",
},
)
assert preflight.headers.get("access-control-allow-origin") not in ("*", "https://evil.example")
actual = client.get(
"/v1/entities",
headers={"Origin": "https://evil.example", "Authorization": "Bearer s3cret"},
)
assert actual.headers.get("access-control-allow-origin") not in ("*", "https://evil.example")
def test_devserver_requires_auth_by_default(monkeypatch):
"""A bare DevServer() must reject unauthenticated /v1/* requests.
Previously auth was opt-in via DEVUI_AUTH_TOKEN env var; the new default is
auth-on so a bare `devui ./agents` invocation does not expose an open API.
"""
monkeypatch.delenv("DEVUI_AUTH_TOKEN", raising=False)
server = DevServer()
app = server.get_app()
with TestClient(app, base_url="http://127.0.0.1") as client:
response = client.get("/v1/entities")
assert response.status_code == 401
def test_devserver_auth_can_be_explicitly_disabled(monkeypatch):
"""Callers can opt out of auth with auth_enabled=False (escape hatch for tests / trusted hosts)."""
monkeypatch.delenv("DEVUI_AUTH_TOKEN", raising=False)
server = _server_with_mock_agent(auth_enabled=False)
app = server.get_app()
with TestClient(app, base_url="http://127.0.0.1") as client:
response = client.get("/v1/entities")
assert response.status_code == 200
def test_devserver_accepts_request_with_valid_bearer_token(monkeypatch):
"""When auth is on, supplying the configured Bearer token grants access."""
monkeypatch.delenv("DEVUI_AUTH_TOKEN", raising=False)
server = DevServer(auth_token="s3cret")
app = server.get_app()
with TestClient(app, base_url="http://127.0.0.1") as client:
response = client.get("/v1/entities", headers={"Authorization": "Bearer s3cret"})
assert response.status_code == 200
def test_meta_endpoint_requires_auth(monkeypatch):
"""/meta exposes capability flags (deployment, instrumentation, version) — gate it behind auth.
Previously /meta was in the auth-bypass list alongside /health and /, so any
unauthenticated caller could read the deployment's capability flags.
"""
monkeypatch.delenv("DEVUI_AUTH_TOKEN", raising=False)
server = DevServer(auth_token="s3cret")
app = server.get_app()
with TestClient(app, base_url="http://127.0.0.1") as client:
unauth = client.get("/meta")
assert unauth.status_code == 401
ok = client.get("/meta", headers={"Authorization": "Bearer s3cret"})
assert ok.status_code == 200
def test_loopback_bind_rejects_non_allowlisted_host_header(monkeypatch):
"""A loopback-bound server must reject requests with a non-loopback Host header.
On a loopback bind, only Host values that name a loopback address are valid;
anything else (e.g. an external hostname that happens to resolve to 127.0.0.1)
is rejected before any handler runs.
"""
monkeypatch.delenv("DEVUI_AUTH_TOKEN", raising=False)
server = DevServer(host="127.0.0.1", auth_enabled=False)
app = server.get_app()
with TestClient(app, base_url="http://127.0.0.1") as client:
rebound = client.get("/health", headers={"Host": "evil.example"})
assert rebound.status_code == 400
ok = client.get("/health", headers={"Host": "127.0.0.1"})
assert ok.status_code == 200
ok_localhost = client.get("/health", headers={"Host": "localhost:8080"})
assert ok_localhost.status_code == 200
def test_serve_defaults_to_auth_enabled():
"""`serve()`'s public signature must default to auth_enabled=True."""
sig = inspect.signature(agent_framework_devui.serve)
assert sig.parameters["auth_enabled"].default is True, (
"serve() must default to auth_enabled=True so `devui ./agents` is secure out of the box"
)
def test_cli_enables_auth_by_default_and_supports_no_auth_optout():
"""`devui ./agents` must produce auth-enabled config; `--no-auth` is the explicit escape hatch."""
from agent_framework_devui._cli import create_cli_parser
parser = create_cli_parser()
default_args = parser.parse_args([])
assert default_args.no_auth is False, "Default CLI invocation should leave auth on"
optout_args = parser.parse_args(["--no-auth"])
assert optout_args.no_auth is True
@@ -582,7 +582,7 @@ def test_sample_peak_renderer_rss_mb_uses_browser_process_tree(
def memory_regression_server() -> Generator[tuple[str, str]]:
"""Start DevUI with a synthetic streaming agent and yield the base URL plus entity ID."""
server = DevServer(host="127.0.0.1", port=0, auth_enabled=False)
server = DevServer(host="127.0.0.1", port=0)
server.register_entities([
MemoryStressAgent(
id="memory-stream-agent",
@@ -435,7 +435,7 @@ async def test_chat_message_parsing_with_function_calls() -> None:
Message(role="tool", contents=[function_result]),
]
prepared_messages = client._prepare_messages_for_openai(messages, request_uses_service_side_storage=False)
prepared_messages = client._prepare_messages_for_openai(messages)
assert prepared_messages == [
{
@@ -1409,31 +1409,29 @@ class RawOpenAIChatClient( # type: ignore[misc]
}
additional_properties = message.additional_properties
replays_local_storage = "_attribution" in additional_properties
# Server-issued response item identities (function_call fc_*, reasoning rs_*, approval IDs,
# local-shell-call IDs) must not be re-sent inline when the request carries
# previous_response_id / conversation_id / conversation: the server already has them via
# the prior response and rejects duplicates with "Duplicate item found with id ...".
# function_result keeps its call_id and the server pairs it to the prior function_call via
# that key. See microsoft/agent-framework#3295. The strip is gated on the request-level
# flag, not a message-level one: HistoryProvider-attributed messages
# (replays_local_storage) still need stripping when the request also carries a continuation
# marker, since the server-stored items would otherwise duplicate the inline ones. Without
# storage, standalone reasoning items are invalid per the API ("reasoning was provided
# without its required following item"), so the reasoning branch always drops.
uses_service_side_storage = request_uses_service_side_storage and not replays_local_storage
# Reasoning items are only valid in input when they directly preceded a function_call
# in the same response. Including a reasoning item that preceded a text response
# (i.e. no function_call in the same message) causes an API error:
# "reasoning was provided without its required following item."
#
# Local storage is stricter: response-scoped reasoning items (rs_*) cannot be replayed
# back to the service unless that message is using service-side storage.
# In that mode we omit reasoning items and rely on function call + tool output replay.
has_function_call = any(c.type == "function_call" for c in message.contents)
for content in message.contents:
match content.type:
case "text_reasoning":
continue
if not uses_service_side_storage or not has_function_call:
continue # reasoning not followed by a function_call is invalid in input
reasoning = self._prepare_content_for_openai(
message.role,
content,
replays_local_storage=replays_local_storage,
)
if reasoning:
all_messages.append(reasoning)
case "function_result":
if request_uses_service_side_storage:
props = content.additional_properties or {}
# Local-shell variant serializes as `local_shell_call` carrying a server-issued id;
# plain function_call_output pairs by call_id and is safe under storage.
if (
props.get(OPENAI_SHELL_OUTPUT_TYPE_KEY) == OPENAI_SHELL_OUTPUT_TYPE_LOCAL_SHELL_CALL
and props.get(OPENAI_LOCAL_SHELL_CALL_ITEM_ID_KEY)
):
continue
new_args: dict[str, Any] = {}
new_args.update(
self._prepare_content_for_openai(
@@ -1445,8 +1443,6 @@ class RawOpenAIChatClient( # type: ignore[misc]
if new_args:
all_messages.append(new_args)
case "function_call":
if request_uses_service_side_storage:
continue
function_call = self._prepare_content_for_openai(
message.role,
content,
@@ -1455,8 +1451,6 @@ class RawOpenAIChatClient( # type: ignore[misc]
if function_call:
all_messages.append(function_call)
case "function_approval_response" | "function_approval_request":
if request_uses_service_side_storage:
continue
prepared = self._prepare_content_for_openai(
message.role,
content,
@@ -1469,12 +1463,6 @@ class RawOpenAIChatClient( # type: ignore[misc]
# top-level mcp_call input item; the result side emits an
# internal marker that `_prepare_messages_for_openai`
# coalesces onto the matching call (or drops if unmatched).
# The mcp_call item carries the model-emitted call_id as its
# server-side `id`, so under continuation it would duplicate
# the prior response's items (#3295). Drop the call here; the
# orphan result is dropped by the coalesce step that follows.
if request_uses_service_side_storage:
continue
prepared_mcp = self._prepare_content_for_openai(
message.role,
content,
@@ -1,5 +1,6 @@
# Copyright (c) Microsoft. All rights reserved.
import asyncio
import base64
import inspect
import json
@@ -120,6 +121,15 @@ async def create_vector_store(
if result.last_error is not None:
raise Exception(f"Vector store file processing failed with status: {result.last_error.message}")
# Wait for the vector store index to be fully searchable.
# create_and_poll confirms file processing, but the search index is eventually consistent.
for _ in range(10):
vs = await client.client.vector_stores.retrieve(vector_store.id)
if vs.file_counts.completed >= 1 and vs.file_counts.in_progress == 0:
break
await asyncio.sleep(1)
await asyncio.sleep(2)
return file.id, Content.from_hosted_vector_store(vector_store_id=vector_store.id)
@@ -333,6 +343,76 @@ async def test_get_response_with_all_parameters() -> None:
assert run_options["input"][1]["content"][0]["text"] == "Test message"
def test_openai_chat_options_declares_verbosity_field() -> None:
"""OpenAIChatOptions declares verbosity as a typed Literal field."""
from typing import get_args, get_type_hints
from agent_framework_openai import OpenAIChatOptions
annotations = get_type_hints(OpenAIChatOptions)
assert "verbosity" in annotations
assert {"low", "medium", "high"} <= set(get_args(annotations["verbosity"]))
async def test_verbosity_option_translates_to_text_field() -> None:
"""Top-level verbosity is translated to text.verbosity for the Responses API."""
client = OpenAIChatClient(model="test-model", api_key="test-key")
_, run_options, _ = await client._prepare_request(
messages=[Message(role="user", contents=["Test message"])],
options={"verbosity": "low"},
)
assert "verbosity" not in run_options
assert run_options["text"] == {"verbosity": "low"}
async def test_verbosity_option_merges_with_response_format() -> None:
"""Verbosity merges into text config alongside response_format-derived format."""
client = OpenAIChatClient(model="test-model", api_key="test-key")
_, run_options, _ = await client._prepare_request(
messages=[Message(role="user", contents=["Test message"])],
options={
"verbosity": "high",
"response_format": OutputStruct,
},
)
assert "verbosity" not in run_options
assert run_options["text"]["verbosity"] == "high"
assert run_options["text_format"] is OutputStruct
async def test_verbosity_option_top_level_overrides_nested_text_verbosity() -> None:
"""When both top-level and text['verbosity'] are set, the top-level value wins."""
client = OpenAIChatClient(model="test-model", api_key="test-key")
_, run_options, _ = await client._prepare_request(
messages=[Message(role="user", contents=["Test message"])],
options={
"verbosity": "high",
"text": {"verbosity": "low"},
},
)
assert "verbosity" not in run_options
assert run_options["text"]["verbosity"] == "high"
async def test_verbosity_option_merges_with_explicit_text_config() -> None:
"""Verbosity merges into a user-provided text config without overwriting other keys."""
client = OpenAIChatClient(model="test-model", api_key="test-key")
_, run_options, _ = await client._prepare_request(
messages=[Message(role="user", contents=["Test message"])],
options={
"verbosity": "medium",
"text": {"format": {"type": "text"}},
},
)
assert "verbosity" not in run_options
assert run_options["text"]["verbosity"] == "medium"
assert run_options["text"]["format"] == {"type": "text"}
@pytest.mark.asyncio
async def test_web_search_tool_with_location() -> None:
"""Test web search tool with location parameters."""
@@ -438,7 +518,7 @@ async def test_chat_message_parsing_with_function_calls() -> None:
Message(role="tool", contents=[function_result]),
]
prepared_messages = client._prepare_messages_for_openai(messages, request_uses_service_side_storage=False)
prepared_messages = client._prepare_messages_for_openai(messages)
assert prepared_messages == [
{
@@ -1754,7 +1834,7 @@ def test_prepare_message_for_openai_with_function_approval_response() -> None:
message = Message(role="user", contents=[approval_response])
result = client._prepare_message_for_openai(message, request_uses_service_side_storage=False)
result = client._prepare_message_for_openai(message)
# FunctionApprovalResponseContent is added directly, not nested in args with role
assert len(result) == 1
@@ -1786,20 +1866,16 @@ def test_prepare_message_for_openai_includes_reasoning_with_function_call() -> N
message = Message(role="assistant", contents=[reasoning, function_call])
# Storage-on path strips both server-issued reasoning (rs_*) and function_call items
# because the server already has them via previous_response_id (#3295).
storage_on_result = client._prepare_message_for_openai(message, request_uses_service_side_storage=True)
storage_on_types = [item["type"] for item in storage_on_result]
assert "reasoning" not in storage_on_types
assert "function_call" not in storage_on_types
result = client._prepare_message_for_openai(message)
# Storage-off path keeps function_call inline so the server sees the call. Reasoning items
# cannot be replayed inline against a server that has no record of the prior response, so
# they remain dropped on this path as well.
storage_off_result = client._prepare_message_for_openai(message, request_uses_service_side_storage=False)
storage_off_types = [item["type"] for item in storage_off_result]
assert "function_call" in storage_off_types
assert "reasoning" not in storage_off_types
# Both reasoning and function_call should be present as top-level items
types = [item["type"] for item in result]
assert "reasoning" in types, "Reasoning items must be included for reasoning models"
assert "function_call" in types
reasoning_item = next(item for item in result if item["type"] == "reasoning")
assert reasoning_item["summary"][0]["text"] == "Let me analyze the request"
assert reasoning_item["id"] == "rs_abc123", "Reasoning id must be preserved for the API"
def test_prepare_messages_for_openai_full_conversation_with_reasoning() -> None:
@@ -1844,20 +1920,27 @@ def test_prepare_messages_for_openai_full_conversation_with_reasoning() -> None:
),
]
# Storage-off path: function_call kept inline (server has no record of it),
# function_call_output kept. Reasoning is still dropped because rs_* response-scoped IDs
# cannot be replayed against a server that has no record of the originating response.
result = client._prepare_messages_for_openai(messages, request_uses_service_side_storage=False)
result = client._prepare_messages_for_openai(messages)
types = [item.get("type") for item in result]
assert "message" in types, "User/assistant messages should be present"
assert "function_call" in types, "Function call items must be present without storage"
assert "reasoning" in types, "Reasoning items must be present"
assert "function_call" in types, "Function call items must be present"
assert "function_call_output" in types, "Function call output must be present"
# Verify reasoning has id
reasoning_items = [item for item in result if item.get("type") == "reasoning"]
assert reasoning_items[0]["id"] == "rs_test123"
# Verify function_call has id
fc_items = [item for item in result if item.get("type") == "function_call"]
assert fc_items[0]["id"] == "fc_test456"
# Verify correct ordering: reasoning before function_call
reasoning_idx = types.index("reasoning")
fc_idx = types.index("function_call")
assert reasoning_idx < fc_idx, "Reasoning must come before function_call"
def test_prepare_message_for_openai_filters_error_content() -> None:
"""Test that error content in messages is handled properly."""
@@ -3999,13 +4082,7 @@ async def test_prepare_options_store_false_omits_reasoning_items_for_stateless_r
assert any(item.get("type") == "function_call_output" for item in options["input"])
async def test_prepare_options_with_conversation_id_strips_server_issued_items() -> None:
"""When the request continues via conversation_id / previous_response_id, server-issued
response items (reasoning rs_*, function_call fc_*) must not be re-sent inline. The server
already has them via the prior response and rejects duplicates with
'Duplicate item found with id ...'. The function_result keeps its call_id so the server
pairs result-to-call. See microsoft/agent-framework#3295. (Originally added in #5250 with
the opposite expectation; field reports proved that path 400s on the wire.)"""
async def test_prepare_options_with_conversation_id_keeps_reasoning_items() -> None:
client = OpenAIChatClient(model="test-model", api_key="test-key")
messages = [
Message(role="user", contents=[Content.from_text(text="search for hotels")]),
@@ -4041,16 +4118,13 @@ async def test_prepare_options_with_conversation_id_strips_server_issued_items()
ChatOptions(store=False, conversation_id="resp_prev123"), # type: ignore[arg-type]
)
types = [item.get("type") for item in options["input"]]
assert "reasoning" not in types
assert "function_call" not in types
assert "function_call_output" in types
output_item = next(item for item in options["input"] if item.get("type") == "function_call_output")
assert output_item["call_id"] == "call_1"
reasoning_items = [item for item in options["input"] if item.get("type") == "reasoning"]
assert len(reasoning_items) == 1
assert reasoning_items[0]["id"] == "rs_test123"
assert options["previous_response_id"] == "resp_prev123"
async def test_prepare_options_with_conversation_id_strips_server_items_for_mixed_history_and_live() -> None:
async def test_prepare_options_with_conversation_id_omits_reasoning_items_for_attributed_replay() -> None:
client = OpenAIChatClient(model="test-model", api_key="test-key")
messages = [
Message(role="user", contents=[Content.from_text(text="search for hotels")]),
@@ -4112,18 +4186,19 @@ async def test_prepare_options_with_conversation_id_strips_server_items_for_mixe
ChatOptions(store=False, conversation_id="resp_prev123"), # type: ignore[arg-type]
)
# Under continuation (request_uses_service_side_storage=True), the strip rule fires for
# every server-issued item type regardless of message attribution: history-attributed items
# would duplicate the prior response stored at resp_prev123, and live items would also
# eventually duplicate items stored on the response this request produces. Function results
# are kept; the server pairs them to prior function_calls via call_id (#3295).
types = [item.get("type") for item in options["input"]]
assert "reasoning" not in types
assert "function_call" not in types
output_call_ids = {
item["call_id"] for item in options["input"] if item.get("type") == "function_call_output"
}
assert output_call_ids == {"call_history", "call_live"}
reasoning_items = [item for item in options["input"] if item.get("type") == "reasoning"]
assert [item["id"] for item in reasoning_items] == ["rs_live123"]
assert any(
item.get("type") == "function_call" and item.get("call_id") == "call_history" for item in options["input"]
)
assert any(item.get("type") == "function_call" and item.get("call_id") == "call_live" for item in options["input"])
assert any(
item.get("type") == "function_call_output" and item.get("call_id") == "call_history"
for item in options["input"]
)
assert any(
item.get("type") == "function_call_output" and item.get("call_id") == "call_live" for item in options["input"]
)
assert options["previous_response_id"] == "resp_prev123"
@@ -4390,10 +4465,6 @@ async def test_integration_web_search() -> None:
assert response.text is not None
@pytest.mark.skip(
reason="Unreliable due to OpenAI vector store indexing potential "
"race condition. See https://github.com/microsoft/agent-framework/issues/1669"
)
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_openai_integration_tests_disabled
@@ -4403,31 +4474,29 @@ async def test_integration_file_search() -> None:
assert isinstance(openai_responses_client, SupportsChatGetResponse)
file_id, vector_store = await create_vector_store(openai_responses_client)
# Use static method for file search tool
file_search_tool = OpenAIChatClient.get_file_search_tool(vector_store_ids=[vector_store.vector_store_id])
# Test that the client will use the file search tool
response = await openai_responses_client.get_response(
messages=[
Message(
role="user",
contents=["What is the weather today? Do a file search to find the answer."],
)
],
options={
"tool_choice": "auto",
"tools": [file_search_tool],
},
)
try:
# Use static method for file search tool
file_search_tool = OpenAIChatClient.get_file_search_tool(vector_store_ids=[vector_store.vector_store_id])
# Test that the client will use the file search tool
response = await openai_responses_client.get_response(
messages=[
Message(
role="user",
contents=["What is the weather today? Do a file search to find the answer."],
)
],
options={
"tool_choice": "auto",
"tools": [file_search_tool],
},
)
await delete_vector_store(openai_responses_client, file_id, vector_store.vector_store_id)
assert "sunny" in response.text.lower()
assert "75" in response.text
assert "sunny" in response.text.lower()
assert "75" in response.text
finally:
await delete_vector_store(openai_responses_client, file_id, vector_store.vector_store_id)
@pytest.mark.skip(
reason="Unreliable due to OpenAI vector store indexing "
"potential race condition. See https://github.com/microsoft/agent-framework/issues/1669"
)
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_openai_integration_tests_disabled
@@ -4437,35 +4506,37 @@ async def test_integration_streaming_file_search() -> None:
assert isinstance(openai_responses_client, SupportsChatGetResponse)
file_id, vector_store = await create_vector_store(openai_responses_client)
# Use static method for file search tool
file_search_tool = OpenAIChatClient.get_file_search_tool(vector_store_ids=[vector_store.vector_store_id])
# Test that the client will use the web search tool
response = openai_responses_client.get_streaming_response(
messages=[
Message(
role="user",
contents=["What is the weather today? Do a file search to find the answer."],
)
],
options={
"tool_choice": "auto",
"tools": [file_search_tool],
},
)
try:
# Use static method for file search tool
file_search_tool = OpenAIChatClient.get_file_search_tool(vector_store_ids=[vector_store.vector_store_id])
# Test that the client will use the file search tool
response = openai_responses_client.get_response(
messages=[
Message(
role="user",
contents=["What is the weather today? Do a file search to find the answer."],
)
],
stream=True,
options={
"tool_choice": "auto",
"tools": [file_search_tool],
},
)
assert response is not None
full_message: str = ""
async for chunk in response:
assert chunk is not None
assert isinstance(chunk, ChatResponseUpdate)
for content in chunk.contents:
if content.type == "text" and content.text:
full_message += content.text
assert response is not None
full_message: str = ""
async for chunk in response:
assert chunk is not None
assert isinstance(chunk, ChatResponseUpdate)
for content in chunk.contents:
if content.type == "text" and content.text:
full_message += content.text
await delete_vector_store(openai_responses_client, file_id, vector_store.vector_store_id)
assert "sunny" in full_message.lower()
assert "75" in full_message
assert "sunny" in full_message.lower()
assert "75" in full_message
finally:
await delete_vector_store(openai_responses_client, file_id, vector_store.vector_store_id)
@pytest.mark.flaky
@@ -4988,10 +5059,7 @@ async def test_prepare_messages_for_openai_does_not_replay_fc_id_when_loaded_fro
next_turn_input = Message(role="user", contents=[Content.from_text(text="Book the cheapest one")])
live_result = client._prepare_messages_for_openai(
[*session.state[provider.source_id]["messages"], next_turn_input],
request_uses_service_side_storage=False,
)
live_result = client._prepare_messages_for_openai([*session.state[provider.source_id]["messages"], next_turn_input])
live_function_call = next(item for item in live_result if item.get("type") == "function_call")
assert live_function_call["id"] == "fc_provider123"
@@ -5004,8 +5072,7 @@ async def test_prepare_messages_for_openai_does_not_replay_fc_id_when_loaded_fro
) # type: ignore[arg-type]
loaded_result = client._prepare_messages_for_openai(
context.get_messages(sources={provider.source_id}, include_input=True),
request_uses_service_side_storage=False,
context.get_messages(sources={provider.source_id}, include_input=True)
)
loaded_function_call = next(item for item in loaded_result if item.get("type") == "function_call")
assert loaded_function_call["id"] == "fc_call_1"
@@ -5024,8 +5091,7 @@ async def test_prepare_messages_for_openai_does_not_replay_fc_id_when_loaded_fro
) # type: ignore[arg-type]
restored_result = client._prepare_messages_for_openai(
restored_context.get_messages(sources={provider.source_id}, include_input=True),
request_uses_service_side_storage=False,
restored_context.get_messages(sources={provider.source_id}, include_input=True)
)
restored_function_call = next(item for item in restored_result if item.get("type") == "function_call")
assert restored_function_call["id"] == "fc_call_1"
@@ -5059,9 +5125,7 @@ def test_prepare_messages_for_openai_keeps_live_fc_id_separate_from_replayed_his
],
)
result = client._prepare_messages_for_openai(
[history_message, live_message], request_uses_service_side_storage=False
)
result = client._prepare_messages_for_openai([history_message, live_message])
function_calls = [item for item in result if item.get("type") == "function_call"]
assert [item["id"] for item in function_calls] == ["fc_call_1", "fc_live123"]
@@ -5099,7 +5163,7 @@ def test_prepare_messages_for_openai_filters_empty_fc_id() -> None:
),
]
result = client._prepare_messages_for_openai(messages, request_uses_service_side_storage=False)
result = client._prepare_messages_for_openai(messages)
# Find the function_call items in the result
fc_items = [item for item in result if item.get("type") == "function_call"]
@@ -5134,7 +5198,7 @@ def test_prepare_messages_for_openai_filters_none_fc_id() -> None:
),
]
result = client._prepare_messages_for_openai(messages, request_uses_service_side_storage=False)
result = client._prepare_messages_for_openai(messages)
# Find the function_call item
fc_items = [item for item in result if item.get("type") == "function_call"]
@@ -5169,7 +5233,7 @@ def test_prepare_messages_for_openai_serializes_mcp_server_tool_call_as_mcp_call
),
]
result = client._prepare_messages_for_openai(messages, request_uses_service_side_storage=False)
result = client._prepare_messages_for_openai(messages)
mcp_items = [item for item in result if isinstance(item, dict) and item.get("type") == "mcp_call"]
assert len(mcp_items) == 1, f"expected exactly one mcp_call item; got result={result}"
@@ -5212,7 +5276,7 @@ def test_prepare_messages_for_openai_coalesces_mcp_call_and_result_into_single_i
),
]
result = client._prepare_messages_for_openai(messages, request_uses_service_side_storage=False)
result = client._prepare_messages_for_openai(messages)
mcp_items = [item for item in result if isinstance(item, dict) and item.get("type") == "mcp_call"]
assert len(mcp_items) == 1, f"expected one coalesced mcp_call item carrying both arguments and output; got {result}"
@@ -5246,7 +5310,7 @@ def test_prepare_messages_for_openai_drops_orphan_mcp_server_tool_result() -> No
),
]
result = client._prepare_messages_for_openai(messages, request_uses_service_side_storage=False)
result = client._prepare_messages_for_openai(messages)
fco_items = [item for item in result if isinstance(item, dict) and item.get("type") == "function_call_output"]
assert fco_items == [], f"orphan mcp_server_tool_result must not serialize as function_call_output; got {fco_items}"
@@ -5278,170 +5342,4 @@ def test_stringify_mcp_output_falls_back_to_json_for_non_text_dict_entries() ->
# endregion
# region: strip server-issued item IDs under storage (issue #3295)
def _strip_rule_messages() -> list[Message]:
return [
Message(role="user", contents=[Content.from_text(text="search hotels in Paris")]),
Message(
role="assistant",
contents=[
Content.from_function_call(
call_id="call_1",
name="search_hotels",
arguments='{"city": "Paris"}',
additional_properties={"fc_id": "fc_server_issued"},
),
],
),
Message(
role="tool",
contents=[Content.from_function_result(call_id="call_1", result="Found 3 hotels in Paris")],
),
]
def test_prepare_messages_strips_function_call_under_storage() -> None:
"""Regression for #3295: when previous_response_id / conversation_id is in flight, the chat
client must not re-send server-issued function_call items inline. The server already has them
via the prior response and rejects duplicates with 'Duplicate item found with id fc_...'.
The function_result keeps its call_id so the server can pair result-to-call."""
client = OpenAIChatClient(model="test-model", api_key="test-key")
result = client._prepare_messages_for_openai(_strip_rule_messages(), request_uses_service_side_storage=True)
types = [item.get("type") for item in result]
assert "function_call" not in types
assert "function_call_output" in types
output_item = next(item for item in result if item.get("type") == "function_call_output")
assert output_item["call_id"] == "call_1"
def test_prepare_messages_keeps_function_call_without_storage() -> None:
"""Without storage there is no previous_response_id, so inline function_call items are the
only source of truth for the server. Behavior is byte-identical to pre-#3295."""
client = OpenAIChatClient(model="test-model", api_key="test-key")
result = client._prepare_messages_for_openai(_strip_rule_messages(), request_uses_service_side_storage=False)
types = [item.get("type") for item in result]
assert "function_call" in types
assert "function_call_output" in types
fc_item = next(item for item in result if item.get("type") == "function_call")
assert fc_item["call_id"] == "call_1"
assert fc_item["id"] == "fc_server_issued"
output_item = next(item for item in result if item.get("type") == "function_call_output")
assert output_item["call_id"] == "call_1"
def test_prepare_messages_strips_approval_items_under_storage() -> None:
"""Approval request/response items also carry server-issued IDs and must be stripped under
storage. Without storage they are kept (#3295)."""
client = OpenAIChatClient(model="test-model", api_key="test-key")
function_call = Content.from_function_call(
call_id="mcp_1",
name="sensitive_action",
arguments='{"action": "delete"}',
)
approval_request = Content.from_function_approval_request(
id="approval_req_1",
function_call=function_call,
)
approval_response = Content.from_function_approval_response(
approved=True,
id="approval_req_1",
function_call=function_call,
)
messages = [
Message(role="assistant", contents=[approval_request]),
Message(role="user", contents=[approval_response]),
]
storage_on = client._prepare_messages_for_openai(messages, request_uses_service_side_storage=True)
storage_on_types = [item.get("type") for item in storage_on]
assert "mcp_approval_request" not in storage_on_types
assert "mcp_approval_response" not in storage_on_types
storage_off = client._prepare_messages_for_openai(messages, request_uses_service_side_storage=False)
storage_off_types = [item.get("type") for item in storage_off]
assert "mcp_approval_request" in storage_off_types
assert "mcp_approval_response" in storage_off_types
def test_prepare_messages_strips_local_shell_call_under_storage() -> None:
"""Local-shell-call function_results carry a server-issued local_shell_call_item_id and must
be stripped under storage. Plain function_results (no shell ID) are kept either way (#3295)."""
from agent_framework_openai._chat_client import (
OPENAI_LOCAL_SHELL_CALL_ITEM_ID_KEY,
OPENAI_SHELL_OUTPUT_TYPE_KEY,
OPENAI_SHELL_OUTPUT_TYPE_LOCAL_SHELL_CALL,
)
client = OpenAIChatClient(model="test-model", api_key="test-key")
shell_result = Content.from_function_result(
call_id="shell_1",
result="ok",
additional_properties={
OPENAI_SHELL_OUTPUT_TYPE_KEY: OPENAI_SHELL_OUTPUT_TYPE_LOCAL_SHELL_CALL,
OPENAI_LOCAL_SHELL_CALL_ITEM_ID_KEY: "lsh_server_issued",
},
)
plain_result = Content.from_function_result(call_id="plain_1", result="plain")
message = Message(role="tool", contents=[shell_result, plain_result])
storage_on = client._prepare_message_for_openai(message, request_uses_service_side_storage=True)
types_on = [item.get("type") for item in storage_on]
assert OPENAI_SHELL_OUTPUT_TYPE_LOCAL_SHELL_CALL not in types_on
assert "function_call_output" in types_on
storage_off = client._prepare_message_for_openai(message, request_uses_service_side_storage=False)
types_off = [item.get("type") for item in storage_off]
assert OPENAI_SHELL_OUTPUT_TYPE_LOCAL_SHELL_CALL in types_off
assert "function_call_output" in types_off
def test_prepare_messages_strips_mcp_items_under_storage() -> None:
"""Hosted-MCP tool call items carry server-issued IDs (the call_id surfaces as `id` on the
wire mcp_call item), so they must be stripped under storage. The orphan mcp_server_tool_result
is then dropped by the existing coalesce logic (#5581). Without storage, the call/result pair
coalesces normally into a single mcp_call wire item (#3295)."""
client = OpenAIChatClient(model="test-model", api_key="test-key")
messages = [
Message(
role="assistant",
contents=[
Content.from_mcp_server_tool_call(
call_id="mcp_abc123",
tool_name="search",
server_name="api_specs",
arguments='{"q": "cats"}',
)
],
),
Message(
role="tool",
contents=[
Content.from_mcp_server_tool_result(
call_id="mcp_abc123",
output=[Content.from_text(text="found 10 cats")],
)
],
),
]
storage_on = client._prepare_messages_for_openai(messages, request_uses_service_side_storage=True)
storage_on_types = [item.get("type") for item in storage_on]
assert "mcp_call" not in storage_on_types
storage_off = client._prepare_messages_for_openai(messages, request_uses_service_side_storage=False)
storage_off_types = [item.get("type") for item in storage_off]
assert "mcp_call" in storage_off_types
# endregion
# endregion
@@ -11,7 +11,7 @@ import os
from textwrap import dedent
from typing import Any
from agent_framework import Agent, InlineSkill, InlineSkillResource, SkillFrontmatter, SkillsProvider
from agent_framework import Agent, InlineSkill, InlineSkillResource, SkillsProvider
from agent_framework.foundry import FoundryChatClient
from azure.identity import AzureCliCredential
from dotenv import load_dotenv
@@ -47,9 +47,8 @@ load_dotenv()
# 1. Static Resources — inline content passed at construction time
# ---------------------------------------------------------------------------
unit_converter_skill = InlineSkill(
frontmatter=SkillFrontmatter(
name="unit-converter", description="Convert between common units using a conversion factor"
),
name="unit-converter",
description="Convert between common units using a conversion factor",
instructions=dedent("""\
Use this skill when the user asks to convert between units.
@@ -1,12 +1,6 @@
---
name: unit-converter
description: Convert between common units using a multiplication factor. Use when asked to convert miles, kilometers, pounds, or kilograms.
license: MIT
compatibility: Works with any model that supports tool use.
allowed-tools: convert
metadata:
author: agent-framework-samples
version: "1.0"
---
## Usage
@@ -21,7 +21,6 @@ from agent_framework import (
FileSkillsSource,
InlineSkill,
InMemorySkillsSource,
SkillFrontmatter,
SkillsProvider,
)
from agent_framework.foundry import FoundryChatClient
@@ -74,9 +73,8 @@ load_dotenv()
# ---------------------------------------------------------------------------
volume_converter_skill = InlineSkill(
frontmatter=SkillFrontmatter(
name="volume-converter", description="Convert between gallons and liters using a conversion factor"
),
name="volume-converter",
description="Convert between gallons and liters using a conversion factor",
instructions=dedent("""\
Use this skill when the user asks to convert between gallons and liters.
@@ -120,7 +118,6 @@ def convert_volume(value: float, factor: float) -> str:
# 2. Define a class-based skill for temperature conversion
# ---------------------------------------------------------------------------
class TemperatureConverterSkill(ClassSkill):
"""A temperature-converter skill defined as a Python class.
@@ -130,10 +127,8 @@ class TemperatureConverterSkill(ClassSkill):
def __init__(self) -> None:
super().__init__(
frontmatter=SkillFrontmatter(
name="temperature-converter",
description="Convert between temperature scales (Fahrenheit, Celsius, Kelvin).",
)
name="temperature-converter",
description="Convert between temperature scales (Fahrenheit, Celsius, Kelvin).",
)
@property
@@ -183,7 +178,6 @@ class TemperatureConverterSkill(ClassSkill):
# 3. Wire everything together and run the agent
# ---------------------------------------------------------------------------
async def main() -> None:
"""Run the combined skills demo."""
endpoint = os.environ["FOUNDRY_PROJECT_ENDPOINT"]
@@ -1,12 +1,6 @@
---
name: unit-converter
description: Convert between common units using a multiplication factor. Use when asked to convert miles, kilometers, pounds, or kilograms.
license: MIT
compatibility: Works with any model that supports tool use.
allowed-tools: convert
metadata:
author: agent-framework-samples
version: "1.0"
---
## Usage
@@ -9,7 +9,7 @@ import os
# warnings.filterwarnings("ignore", message=r"\[SKILLS\].*", category=FutureWarning)
from textwrap import dedent
from agent_framework import Agent, InlineSkill, SkillFrontmatter, SkillsProvider
from agent_framework import Agent, InlineSkill, SkillsProvider
from agent_framework.foundry import FoundryChatClient
from azure.identity import AzureCliCredential
from dotenv import load_dotenv
@@ -43,9 +43,8 @@ load_dotenv()
# Define a code skill with a script that performs a sensitive operation
deployment_skill = InlineSkill(
frontmatter=SkillFrontmatter(
name="deployment", description="Tools for deploying application versions to production"
),
name="deployment",
description="Tools for deploying application versions to production",
instructions=dedent("""\
Use this skill when the user asks to deploy an application.
@@ -75,7 +75,7 @@ async def main() -> None:
FilteringSkillsSource(
FileSkillsSource(str(skills_dir), script_runner=subprocess_script_runner),
# Only keep the volume-converter skill
predicate=lambda s: s.frontmatter.name != "length-converter",
predicate=lambda s: s.name != "length-converter",
)
)
@@ -1,12 +1,6 @@
---
name: length-converter
description: Convert between common length units (miles, km, feet, meters) using a multiplication factor.
license: MIT
compatibility: Works with any model that supports tool use.
allowed-tools: convert
metadata:
author: agent-framework-samples
version: "1.0"
---
## Usage
@@ -1,12 +1,6 @@
---
name: volume-converter
description: Convert between gallons and liters using a conversion factor.
license: MIT
compatibility: Works with any model that supports tool use.
allowed-tools: convert
metadata:
author: agent-framework-samples
version: "1.0"
---
## Usage
+1 -1
View File
@@ -602,7 +602,7 @@ dependencies = [
[package.metadata]
requires-dist = [
{ name = "agent-framework-core", editable = "packages/core" },
{ name = "github-copilot-sdk", marker = "python_full_version >= '3.11'", specifier = ">=1.0.0b2,<=1.0.0b2" },
{ name = "github-copilot-sdk", marker = "python_full_version >= '3.11'", specifier = "<=1.0.0b2,>=1.0.0b2" },
]
[[package]]