Compare commits

..
Author SHA1 Message Date
Jacob AlberandGitHub 0ecd13c43c Merge branch 'main' into copilot/add-unit-tests-workflows-routebuilder 2026-05-13 16:38:40 -04:00
7d23582e2b Python: fix: prevent MCP message_handler deadlock on notification reload (#4866)
* fix(python): prevent MCP message_handler deadlock on notification reload

When an MCP server sends a notifications/tools/list_changed or
notifications/prompts/list_changed notification, the message_handler
previously awaited load_tools()/load_prompts() directly. Since the
handler runs on the MCP SDK's single-threaded receive loop, this
caused a deadlock: load_tools() sends a list_tools request and waits
for its response, but the receive loop cannot deliver that response
while blocked in the handler.

This manifested as a timeout in call_tool(), which then surfaced as
"Error: Function failed." to the model instead of the real tool
output. The MATLAB MCP server reliably triggers this because it sends
a tools/list_changed notification during tool execution.

Fix: schedule reloads as background asyncio.Tasks via a new
_schedule_reload() helper, freeing the receive loop immediately.

Fixes #4828

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review feedback: fix exc_info, coalesce reloads, shutdown cleanup, tests

- Fix exc_info=exc -> exc_info=True in _schedule_reload and message_handler
- Tighten _schedule_reload param type from Any to Coroutine[Any, Any, None]
- Coalesce reloads: cancel-and-replace per reload kind to prevent unbounded growth
- Cancel pending reload tasks in _close_on_owner before tearing down session
- Re-raise CancelledError in _safe_reload to respect task cancellation
- Replace flaky asyncio.sleep(0) with asyncio.wait_for/gather in tests
- Add caplog assertions to verify reload failure is actually logged
- Assert _pending_reload_tasks cleanup on error path

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: address review comments on MCP reload handling

- Fix exc_info=True -> exc_info=message in message_handler error logging,
  since the handler is not called from an except block
- Await cancelled reload tasks in _close_on_owner before tearing down
  the session to avoid 'Task was destroyed but pending' warnings
- Add cancel-and-replace test verifying duplicate notifications cancel
  the first reload task and only keep one in flight

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: remove Task.cancelling() call for Python 3.10 compat

Task.cancelling() was added in Python 3.11. Replace with awaiting
the task and checking cancelled() instead.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add debug log when cancelling superseded reload task

Log at DEBUG level when a new notification cancels an in-flight reload
task, improving observability of the cancel-and-replace behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-13 20:09:59 +00:00
574631671d Update version for release. (#5789)
Co-authored-by: alliscode <25218250+alliscode@users.noreply.github.com>
2026-05-13 20:07:50 +00:00
beea1cf6b8 Fix IDE0001 format errors - simplify generic type names
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/8573214e-ec42-4969-ba94-76bdc8ad3e59

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>
2026-05-13 19:49:13 +00:00
Jacob AlberandGitHub 8d72a1c053 Merge branch 'main' into copilot/add-unit-tests-workflows-routebuilder 2026-05-13 15:26:47 -04:00
53787a354a Fix ValueTask compatibility with .NET Framework 4.7.2
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/a8437809-0898-43a6-a950-09eb3417f58a

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>
2026-05-13 19:24:08 +00:00
1a6e443789 Refactor overload int constants to HandlerOverload enum
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/19397f58-a88a-41cf-bd85-588f520e0d0f

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>
2026-05-13 19:10:41 +00:00
e59b9327db Refine RouteBuilder test helpers
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/012f3b3b-acb9-4869-9084-b767cbe1885b

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>
2026-05-13 19:03:44 +00:00
981726cc15 .NET: feat(evals): add ground_truth/expected_output support for workflow evaluation (#5755)
* .NET: feat(evals): add ground_truth/expected_output support for workflow eval

Brings .NET to parity with Python PR #5234 for issue #5135:

- Add expectedOutput parameter to Run.EvaluateAsync (workflow) and stamp on the overall EvalItem.ExpectedOutput.
- Map EvalItem.ExpectedOutput -> ground_truth in the Foundry JSONL payload, item_schema, and data_mapping for similarity.
- Add GroundTruthEvaluators set (currently builtin.similarity) and a FindMissingGroundTruthEvaluators helper.
- Fail fast with InvalidOperationException when a ground-truth evaluator is selected but no item provides an ExpectedOutput, instead of surfacing a remote provider error.
- Add tests in FoundryEvalConverterTests and WorkflowEvaluationTests.
- Add Evaluation_WorkflowExpectedOutputs sample (workflow + Foundry similarity).

Fixes microsoft/agent-framework#5135 (.NET side).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review: relax BuildOverallItem events to IReadOnlyList<WorkflowEvent>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Sample: disable per-agent breakdown when using reference-based evaluator

Per-agent EvalItems are intentionally left without ExpectedOutput, so the new fail-fast validation in FoundryEvals would throw when Similarity is invoked for per-agent items. Pass includePerAgent: false in the workflow + similarity sample, and document this gotcha in the EvaluateAsync XML doc.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix BuildOverallItem: fall back to last ExecutorCompletedEvent

AgentResponseEvent is only emitted when AIAgentHostOptions.EmitAgentResponseEvents is enabled, which is not the default for WorkflowBuilder(agent).AddEdge(...). When it is absent, fall back to the last non-internal ExecutorCompletedEvent whose Data is an AgentResponse / ChatMessage / string so the overall EvalItem (and any expectedOutput) is produced. Without this, samples wired up the standard way returned 0 evaluation items.

Update test to cover the fallback path.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Sample: enable EmitAgentResponseEvents; eval throws clear error when no overall response found

Root cause of '0 results': AIAgentHostExecutor only emits AgentResponseEvent when AIAgentHostOptions.EmitAgentResponseEvents is true (default false). For ordinary AIAgent executors the runtime's ExecutorCompletedEvent.Data is null, so the prior fallback couldn't find a final response either.

Sample now builds executors with EmitAgentResponseEvents=true via BindAsExecutor(hostOptions). EvaluateAsync now throws InvalidOperationException with a remediation hint when the user supplies expectedOutput but no overall final response can be located, instead of silently returning 0/0.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Guard against null sample/error/usage/datasource_item in ParseDetailedItem

Foundry eval responses can have these properties present with JSON null
or non-object values, which caused JsonElement.TryGetProperty to throw
'requires Object, has Null'. Check ValueKind == Object before drilling in.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review: reorder expectedOutput, tighten ground-truth check, add fail-fast test

* WorkflowEvaluationExtensions.EvaluateAsync: move 'expectedOutput' to
  after 'splitter' so the original positional contract of (splitter,
  cancellationToken) is preserved for existing callers.
* FoundryEvals: require ALL items to carry ExpectedOutput when a
  ground-truth evaluator is selected (e.g. similarity), not just any.
  Reference-based evaluators score per-item, so a single missing GT
  would still surface as a provider-side validation error. Updated
  fail-fast message accordingly.
* WorkflowEvaluationTests: add EvaluateAsync_WithExpectedOutputButNoFinalResponse_ThrowsAsync
  to verify the InvalidOperationException is thrown (and that the
  message mentions EmitAgentResponseEvents).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fail-fast on missing overall item regardless of expectedOutput; harden BuildOverallItem default

* EvaluateAsync now throws InvalidOperationException whenever 'includeOverall'
  is requested but BuildOverallItem cannot produce an item, instead of only
  when 'expectedOutput' is supplied. Same misconfiguration (agents not bound
  with EmitAgentResponseEvents) used to silently return empty results — now
  it surfaces a clear, actionable error in both cases.
* BuildOverallItem switch default now throws instead of returning null. The
  preceding for-loop already constrains Data to AgentResponse/ChatMessage/
  string, so reaching default would indicate a contract drift; throw to make
  the bug visible.
* Test renamed and broadened to verify the throw fires without expectedOutput.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: alliscode <25218250+alliscode@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-13 19:03:27 +00:00
2b66bad6b3 Fix RouteBuilder test nullability warning
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/012f3b3b-acb9-4869-9084-b767cbe1885b

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>
2026-05-13 19:02:21 +00:00
224d9ce6ee Address RouteBuilder test review feedback
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/012f3b3b-acb9-4869-9084-b767cbe1885b

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>
2026-05-13 19:01:06 +00:00
9ca8ef5891 Add RouteBuilder unit tests
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/012f3b3b-acb9-4869-9084-b767cbe1885b

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>
2026-05-13 18:59:35 +00:00
9b9604ce18 fix: avoid mutating handoff message roles (#5808)
Co-authored-by: Jacob Alber <jaalber@microsoft.com>
2026-05-13 18:52:19 +00:00
bd0d6070f1 Fixing FoundryToolboxMcp sample to use created toolbox. (#5786)
Co-authored-by: alliscode <bentho@microsoft.com>
2026-05-13 16:53:16 +00:00
CopilotGitHubrogerbarretocopilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
37a043a797 .NET: [Breaking Change] Auto-wire ChatClient with OpenTelemetryChatClient in OpenTelemetryAgent (#5750)
* Initial plan

* .NET: Auto-wire ChatClient with OpenTelemetryChatClient in OpenTelemetryAgent

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/96dd033a-0c48-4d3f-9148-324bfd436b75

Co-authored-by: rogerbarreto <19890735+rogerbarreto@users.noreply.github.com>

* Address review: remove extension overload; honor UseProvidedChatClientAsIs; drop redundant check

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/6ac3f75d-eeb7-4811-8043-9a27511b0a8b

Co-authored-by: rogerbarreto <19890735+rogerbarreto@users.noreply.github.com>

* Resolve ChatClientAgent via GetService before checking options/chat client

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/008d914d-8cbb-4e9f-81b6-f8c3c8bd8d04

Co-authored-by: rogerbarreto <19890735+rogerbarreto@users.noreply.github.com>

* Split OpenTelemetryAgent ctor to preserve original (innerAgent, sourceName) signature

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/a890c9a7-0b77-40ab-802c-dfbf09f8c260

Co-authored-by: rogerbarreto <19890735+rogerbarreto@users.noreply.github.com>

* Preserve base AgentRunOptions properties and avoid double-wrap on user factory

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/3afbf18c-de22-4236-a2f2-02ca1e98ae21

Co-authored-by: rogerbarreto <19890735+rogerbarreto@users.noreply.github.com>

* .NET: OpenTelemetryAgent normalize sourceName once and add OTEL wiring path coverage

Normalize the configured source name once in the constructor so the outer OpenTelemetryChatClient and the auto-wired inner OpenTelemetryChatClient always emit spans on the same ActivitySource. A caller passing an empty string previously produced agent-level spans on DefaultSourceName but auto-wired chat spans on the empty source, causing the chat spans to be silently dropped by exporters subscribed to the default source.

Tests added to cover the previously unexercised OTEL wiring branches:

- Ctor_NullOrEmptySourceName_AutoWiredChatClientUsesDefaultSource_Async (Theory: null and empty)

- AutoWireChatClient_PlainAgentRunOptions_PreservesContinuationToken_Async

- AutoWireChatClient_ChatClientAgentRunOptions_NoUserFactory_PreservesChatOptions_Async

- AutoWireChatClient_StreamingDisabled_DoesNotEmitChatSpan_Async

* .NET: Mark OpenTelemetryAgent autoWireChatClient ctor as [Experimental]

Annotate the new 3-arg OpenTelemetryAgent(AIAgent, string?, bool) constructor with [Experimental(DiagnosticIds.Experiments.AgentsAIExperiments)] (MAAI001) so callers must explicitly opt in to the auto-wire toggle. The original 2-arg constructor stays non-experimental and delegates with autoWireChatClient: true; the delegating call is locally suppressed so the existing source compatibility surface is preserved.

* .NET: OpenTelemetryAgent address westey-m PR review

- Use string.IsNullOrWhiteSpace (not IsNullOrEmpty) when normalizing the constructor sourceName, so callers passing whitespace-only strings still land on OpenTelemetryConsts.DefaultSourceName instead of an unsubscribed ActivitySource.

- Fix the misleading pragma comment on the 2-arg ctor delegating call: auto-wiring is the new default, it does not preserve the original (pre-PR) behavior.

- Expand the GetRunOptionsWithChatClientWiring XML doc to spell out that a base AgentRunOptions (not ChatClientAgentRunOptions) is also accepted: it is converted to ChatClientAgentRunOptions with the auto-wire factory installed and base properties copied.

- Tests: extend the source-name normalization Theory with whitespace cases ('   ' and '\t'); add end-to-end coverage for plain AgentRunOptions over a real ChatClientAgent (sync + streaming) asserting the inner chat client is invoked and both invoke_agent + chat spans are emitted.

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: rogerbarreto <19890735+rogerbarreto@users.noreply.github.com>
2026-05-13 13:06:45 +00:00
westeyandGitHub f16cb9a118 .NET: Add harness agent package (#5782)
* Add harness agent package

* Fix formatting.

* Fix formatting.

* Update release filter

* Address PR comments.
2026-05-13 10:58:05 +00:00
Evan MattsonandGitHub 9a301b8d4b Replace merge-gatekeeper Docker action with github-script polling (#5533)
The upsidr/merge-gatekeeper@v1 action is a Dockerfile-based action that
builds a golang image on every run. On merge_group events the run step
is conditioned out via `if: github.event_name == 'pull_request'`, so the
build happens but produces nothing.

Replace with an actions/github-script@v8 polling loop that mirrors the
action's behavior exactly: merges combined-statuses and check-runs for
the PR head SHA, with combined-status winning on name collisions, and
the same conclusion mapping (skipped → dropped, success/neutral →
success, anything else terminal → error). Same job name, triggers,
permissions, timeout (3600s), interval (30s), and ignored list, so
existing required-check rules stay valid.

PR runs now poll the API in seconds instead of waiting on a per-run
docker image build, and merge_group runs become near-instant no-ops.
2026-05-13 05:45:51 +00:00
Evan MattsonandGitHub 15a11a426a Python: add ag-ui tool result display channel (#5762)
* Python: add ag-ui tool result display channel

Key decisions:
- Add TOOL_RESULT_DISPLAY_KEY and make state_update accept optional state plus a tool_result display payload.
- Keep text as the LLM-bound tool result while using the display marker only for ToolCallResultEvent.content.
- Reuse one outer/inner Content additional_properties extraction helper for state and display markers, preserving fallback behavior when display is absent.

Files changed:
- python/packages/ag-ui/agent_framework_ag_ui/_state.py
- python/packages/ag-ui/agent_framework_ag_ui/_run_common.py
- python/packages/ag-ui/tests/ag_ui/test_run_common.py
- python/packages/ag-ui/tests/ag_ui/golden/test_scenario_deterministic_state.py
- python/issues/done/01-tool-result-display-channel.md

Blockers/notes:
- Slice 1 is complete and moved to issues/done.
- Slice 2 remains for docstring and README documentation.

* Python: document ag-ui tool result display channel

Key decisions:
- Document state_update as the single helper for LLM text, UI-only tool_result display content, and durable shared state.
- Keep the display guidance explicit that text remains LLM-bound while tool_result feeds ToolCallResultEvent.content.
- List both reserved additional_properties markers in the docstring return contract.

Files changed:
- python/packages/ag-ui/agent_framework_ag_ui/_state.py
- python/packages/ag-ui/README.md
- python/issues/done/02-docs-tool-result-display.md

Blockers/notes:
- Slice 2 is complete and moved to issues/done.
- Verification passed: uv run poe syntax -P ag-ui --check; uv run poe test -P ag-ui; uv run poe markdown-code-lint; uv run ruff check packages/ag-ui/agent_framework_ag_ui/_state.py.
- Commit hooks were skipped after poe-check repeatedly rewrote uv.lock ordering; the same checks were run manually and passed.

* Python: update gitignore
2026-05-12 22:12:04 +00:00
cfd3dfe40b .NET: CI hardening — split Functions tests, re-enable skipped integration tests (#5717)
* Split DurableTask/AzureFunctions integration tests into dedicated CI job

- Add -TestProjectNameExclude parameter to New-FilteredSolution.ps1
- Add 'functions' and 'core' path filters to paths-filter job
- Exclude DurableTask/AzureFunctions from main dotnet-test job
- Remove emulator setup from dotnet-test (no longer needed)
- Add new dotnet-test-functions job (ubuntu/net10.0 only, path-conditional)
- Update merge gate and report job to include dotnet-test-functions

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR feedback: add Workflows.Generators to core filter, drop dotnetChanges gate from functions job

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Re-enable Anthropic integration tests

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Upgrade Anthropic SDK 12.13.0 -> 12.20.0 to fix M.E.AI incompatibility

Fixes MissingMethodException on WebSearchToolResultContent.get_Results()
caused by Anthropic 12.13.0 being compiled against an older
Microsoft.Extensions.AI.Abstractions version.

Suppress RT0003 in AI.Abstractions.csproj as the transitive reference
from the upgraded Anthropic SDK conflicts with the explicit one.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix Anthropic unit test mocks for SDK 12.20.0 interface changes

Add missing interface members: IAnthropicClient.WebhookKey,
IBetaService.MemoryStores, IBetaService.Webhooks, IBetaService.UserProfiles

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Re-enable CheckSystem declarative integration tests

The CheckSystem.yaml tests were temporarily skipped in PR #4270 during
the Azure.AI.Projects 2.0.0-beta.1 SDK update. Since then, the system
variable plumbing (SystemScope, SetLastMessageAsync, conversation
initialization) has been significantly updated and stabilized. The
other tests in these same files pass reliably using the same
infrastructure.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix CheckSystem test case to expect 1 response

The CheckSystem workflow sends a 'PASSED!' SendActivity when all system
variables are populated, producing 1 AgentResponseEvent. The test case
had min_response_count: 0 with no max, so the assertion defaulted max
to 0 and failed with 'Response count greater than expected: 0 (Actual: 1)'.
Updated to expect exactly 1 response, matching the SendActivity pattern.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Re-enable Foundry OpenAPI server-side tool integration test

Remove Skip="For manual testing only" from
AsAIAgent_WithOpenAPITool_NativeSDKCreation_InvokesServerSideToolAsync.
The test already uses RetryFact(3 retries, 5s delay) to handle
transient failures from the external restcountries.com API.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Include workflow file in functions/core path filters

A PR editing only dotnet-build-and-test.yml would skip
dotnet-test-functions because the workflow path was missing
from both the functions and core path filter lists.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Rename filter parameters for consistency

TestProjectNameFilter  -> TestProjectNameIncludeFilter
TestProjectNameExclude -> TestProjectNameExcludeFilter

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove unnecessary RT0003 warning suppression

The RT0003 suppression was added during the Anthropic SDK 12.20.0
upgrade but the warning no longer fires. Removing it to keep the
NoWarn list minimal.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove duplicate WebhookKey properties from merge

Both our branch and main added WebhookKey to the Anthropic test
mock classes, resulting in CS0102 duplicate definition errors.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-12 17:56:31 +00:00
3b6a4574eb .NET: Fix OpenAIResponsesAgentClient to include agentName in endpoint path (#5748)
* Fix OpenAIResponsesAgentClient endpoint to include agentName in path (#5324)

The sample OpenAIResponsesAgentClient used '/v1/' as the endpoint, which
routes to the multi-agent endpoint requiring agent.name in the request body.
However, AsIChatClient(agentName) maps agentName to the model field, not
agent.name, causing HTTP 400 errors on OpenAI-compatible endpoints.

Changed the endpoint to '/{agentName}/v1/' to match the pattern used by
OpenAIChatCompletionsAgentClient, routing to the single-agent endpoint
where no agent.name body field is needed.

Added regression test verifying that the model field alone is insufficient
for agent resolution on the multi-agent endpoint.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review feedback for #5324

- URL-escape agentName in OpenAIResponsesAgentClient endpoint path to
  handle reserved characters safely
- Add per-agent MapOpenAIResponses() calls in AgentHost so the sample
  host serves the /{agentName}/v1/responses routes the client now targets
- Replace brittle Assert.Contains("agent.name") assertions with stable
  machine-readable error code assertion ("missing_required_parameter")

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address additional review feedback for #5324

- Apply Uri.EscapeDataString to OpenAIChatCompletionsAgentClient endpoint
  for consistency with OpenAIResponsesAgentClient
- Map OpenAI Responses and ChatCompletions endpoints for all builder-based
  agents (chemist, mathematician, literator, science workflows) so every
  discoverable agent is reachable via the single-agent endpoint path

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-12 17:16:47 +00:00
54 changed files with 3410 additions and 256 deletions
+95 -13
View File
@@ -2,7 +2,7 @@ name: Merge Gatekeeper
on:
pull_request:
branches: [ "main", "feature*" ]
branches: ["main", "feature*"]
merge_group:
branches: ["main"]
@@ -13,23 +13,105 @@ concurrency:
jobs:
merge-gatekeeper:
runs-on: ubuntu-latest
# Restrict permissions of the GITHUB_TOKEN.
# Docs: https://docs.github.com/en/actions/using-jobs/assigning-permissions-to-jobs
permissions:
checks: read
statuses: read
steps:
- name: Run Merge Gatekeeper
# NOTE: v1 is updated to reflect the latest v1.x.y. Please use any tag/branch that suits your needs:
# https://github.com/upsidr/merge-gatekeeper/tags
# https://github.com/upsidr/merge-gatekeeper/branches
uses: upsidr/merge-gatekeeper@v1
- name: Wait for required checks
if: github.event_name == 'pull_request'
with:
token: ${{ secrets.GITHUB_TOKEN }}
timeout: 3600
interval: 30
uses: actions/github-script@v8
env:
TIMEOUT_SECONDS: "3600"
INTERVAL_SECONDS: "30"
SELF_JOB_NAME: ${{ github.job }}
# "Cleanup artifacts", "Agent", "Prepare", and "Upload results" are check runs
# created by an org-level GitHub App (MSDO), not by any workflow in this repo.
# They are outside our control and their transient failures should not block merges.
ignored: CodeQL,CodeQL analysis (csharp),Cleanup artifacts,Agent,Prepare,Upload results
IGNORED_NAMES: "CodeQL,CodeQL analysis (csharp),Cleanup artifacts,Agent,Prepare,Upload results"
with:
script: |
const timeoutSeconds = Number(process.env.TIMEOUT_SECONDS);
const intervalSeconds = Number(process.env.INTERVAL_SECONDS);
const selfName = process.env.SELF_JOB_NAME;
const ignored = new Set(
process.env.IGNORED_NAMES.split(',').map((s) => s.trim()).filter(Boolean),
);
const sha = context.payload.pull_request.head.sha;
const { owner, repo } = context.repo;
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
// Mirrors upsidr/merge-gatekeeper: merge combined-statuses and check-runs
// for the PR head SHA, with combined-statuses winning on name collision.
async function collectChecks() {
const merged = new Map();
const combined = await github.rest.repos.getCombinedStatusForRef({
owner, repo, ref: sha, per_page: 100,
});
for (const s of combined.data.statuses ?? []) {
if (!merged.has(s.context)) {
// Combined-status states: success | pending | error | failure
merged.set(s.context, { name: s.context, state: s.state });
}
}
const runs = await github.paginate(github.rest.checks.listForRef, {
owner, repo, ref: sha, per_page: 100,
});
for (const r of runs) {
if (merged.has(r.name)) continue;
let state;
if (r.status !== 'completed') {
state = 'pending';
} else if (r.conclusion === 'skipped') {
continue; // Skipped runs are dropped, matching the original action.
} else if (r.conclusion === 'success' || r.conclusion === 'neutral') {
state = 'success';
} else {
// cancelled | timed_out | action_required | stale | failure
state = 'error';
}
merged.set(r.name, { name: r.name, state });
}
return [...merged.values()];
}
function evaluate(entries) {
const failed = [];
const pending = [];
const succeeded = [];
for (const e of entries) {
if (e.name === selfName || ignored.has(e.name)) continue;
if (e.state === 'success') succeeded.push(e.name);
else if (e.state === 'error' || e.state === 'failure') failed.push(e.name);
else pending.push(e.name);
}
return { failed, pending, succeeded };
}
const deadline = Date.now() + timeoutSeconds * 1000;
for (;;) {
const entries = await collectChecks();
const { failed, pending, succeeded } = evaluate(entries);
core.info(
`succeeded=${succeeded.length} pending=${pending.length} failed=${failed.length}`,
);
if (failed.length) {
core.setFailed(`Failing checks: ${failed.join(', ')}`);
return;
}
if (pending.length === 0) {
core.info(`All required checks passed: ${succeeded.join(', ') || '(none)'}`);
return;
}
if (Date.now() > deadline) {
core.setFailed(`Timed out waiting for: ${pending.join(', ')}`);
return;
}
core.info(`Waiting on (${pending.length}): ${pending.slice(0, 10).join(', ')}${pending.length > 10 ? ', …' : ''}`);
await sleep(intervalSeconds * 1000);
}
+2
View File
@@ -246,3 +246,5 @@ dotnet/filtered-*.slnx
# Local tool state
.omc/
.omx/
**/issues/
+3
View File
@@ -298,6 +298,7 @@
</Folder>
<Folder Name="/Samples/03-workflows/Evaluation/">
<Project Path="samples/03-workflows/Evaluation/Evaluation_WorkflowEval/Evaluation_WorkflowEval.csproj" />
<Project Path="samples/03-workflows/Evaluation/Evaluation_WorkflowExpectedOutputs/Evaluation_WorkflowExpectedOutputs.csproj" />
</Folder>
<Folder Name="/Samples/04-hosting/">
</Folder>
@@ -582,6 +583,7 @@
<Project Path="src/Microsoft.Agents.AI.DurableTask/Microsoft.Agents.AI.DurableTask.csproj" />
<Project Path="src/Microsoft.Agents.AI.Foundry.Hosting/Microsoft.Agents.AI.Foundry.Hosting.csproj" />
<Project Path="src/Microsoft.Agents.AI.Foundry/Microsoft.Agents.AI.Foundry.csproj" />
<Project Path="src/Microsoft.Agents.AI.Harness/Microsoft.Agents.AI.Harness.csproj" />
<Project Path="src/Microsoft.Agents.AI.GitHub.Copilot/Microsoft.Agents.AI.GitHub.Copilot.csproj" />
<Project Path="src/Microsoft.Agents.AI.Hosting.A2A.AspNetCore/Microsoft.Agents.AI.Hosting.A2A.AspNetCore.csproj" />
<Project Path="src/Microsoft.Agents.AI.Hosting.A2A/Microsoft.Agents.AI.Hosting.A2A.csproj" />
@@ -636,6 +638,7 @@
<Project Path="tests/Microsoft.Agents.AI.Foundry.UnitTests/Microsoft.Agents.AI.Foundry.UnitTests.csproj" />
<Project Path="tests/Microsoft.Agents.AI.Foundry.Hosting.UnitTests/Microsoft.Agents.AI.Foundry.Hosting.UnitTests.csproj" />
<Project Path="tests/Microsoft.Agents.AI.GitHub.Copilot.UnitTests/Microsoft.Agents.AI.GitHub.Copilot.UnitTests.csproj" />
<Project Path="tests/Microsoft.Agents.AI.Harness.UnitTests/Microsoft.Agents.AI.Harness.UnitTests.csproj" />
<Project Path="tests/Microsoft.Agents.AI.Hosting.A2A.UnitTests/Microsoft.Agents.AI.Hosting.A2A.UnitTests.csproj" />
<Project Path="tests/Microsoft.Agents.AI.Hosting.AGUI.AspNetCore.UnitTests/Microsoft.Agents.AI.Hosting.AGUI.AspNetCore.UnitTests.csproj" />
<Project Path="tests/Microsoft.Agents.AI.Hosting.AzureFunctions.UnitTests/Microsoft.Agents.AI.Hosting.AzureFunctions.UnitTests.csproj" />
+1
View File
@@ -7,6 +7,7 @@
"src\\Microsoft.Agents.AI.AGUI\\Microsoft.Agents.AI.AGUI.csproj",
"src\\Microsoft.Agents.AI.Anthropic\\Microsoft.Agents.AI.Anthropic.csproj",
"src\\Microsoft.Agents.AI.GitHub.Copilot\\Microsoft.Agents.AI.GitHub.Copilot.csproj",
"src\\Microsoft.Agents.AI.Harness\\Microsoft.Agents.AI.Harness.csproj",
"src\\Microsoft.Agents.AI.AzureAI.Persistent\\Microsoft.Agents.AI.AzureAI.Persistent.csproj",
"src\\Microsoft.Agents.AI.Foundry\\Microsoft.Agents.AI.Foundry.csproj",
"src\\Microsoft.Agents.AI.Foundry.Hosting\\Microsoft.Agents.AI.Foundry.Hosting.csproj",
+3 -3
View File
@@ -1,14 +1,14 @@
<Project>
<PropertyGroup>
<!-- Central version prefix - applies to all nuget packages. -->
<VersionPrefix>1.5.0</VersionPrefix>
<VersionPrefix>1.6.0</VersionPrefix>
<RCNumber>1</RCNumber>
<DateSuffix>260507</DateSuffix>
<DateSuffix>260512</DateSuffix>
<PackageVersion Condition="'$(IsReleaseCandidate)' == 'true'">$(VersionPrefix)-rc$(RCNumber)</PackageVersion>
<PackageVersion Condition="'$(IsReleaseCandidate)' != 'true' AND '$(VersionSuffix)' != ''">$(VersionPrefix)-$(VersionSuffix).$(DateSuffix).1</PackageVersion>
<PackageVersion Condition="'$(IsReleaseCandidate)' != 'true' AND '$(VersionSuffix)' == ''">$(VersionPrefix)-preview.$(DateSuffix).1</PackageVersion>
<PackageVersion Condition="'$(IsReleased)' == 'true'">$(VersionPrefix)</PackageVersion>
<GitTag>1.5.0</GitTag>
<GitTag>1.6.0</GitTag>
<Configurations>Debug;Release;Publish</Configurations>
<IsPackable>true</IsPackable>
@@ -20,22 +20,18 @@ using OpenAI.Responses;
#pragma warning disable OPENAI001 // Experimental API
#pragma warning disable AAIP001 // AgentToolboxes is experimental
// Must match the `<name>` segment of FOUNDRY_TOOLBOX_ENDPOINT.
// Name of the toolbox to create and connect to.
const string ToolboxName = "research_toolbox";
const string Query = "What tools do you have access to?";
string endpoint = Environment.GetEnvironmentVariable("AZURE_AI_PROJECT_ENDPOINT")
?? throw new InvalidOperationException("AZURE_AI_PROJECT_ENDPOINT is not set.");
string deploymentName = Environment.GetEnvironmentVariable("AZURE_AI_MODEL_DEPLOYMENT_NAME") ?? "gpt-5.4-mini";
string toolboxEndpoint = Environment.GetEnvironmentVariable("FOUNDRY_TOOLBOX_ENDPOINT")
?? throw new InvalidOperationException(
"FOUNDRY_TOOLBOX_ENDPOINT is not set. Example: " +
"https://<account>.services.ai.azure.com/api/projects/<project>/toolsets/<name>/mcp?api-version=2025-05-01-preview");
TokenCredential credential = new DefaultAzureCredential();
// Comment out if the toolbox already exists in your Foundry project.
await CreateSampleToolboxAsync(ToolboxName, endpoint, credential);
var toolboxEndpoint = await CreateSampleToolboxAsync(ToolboxName, endpoint, credential);
// Inject a fresh Azure AI bearer token on every MCP request.
using var httpClient = new HttpClient(new BearerTokenHandler(credential, "https://ai.azure.com/.default")
@@ -51,6 +47,11 @@ await using McpClient mcpClient = await McpClient.CreateAsync(
{
Endpoint = new Uri(toolboxEndpoint),
Name = "foundry_toolbox",
TransportMode = HttpTransportMode.StreamableHttp,
AdditionalHeaders = new Dictionary<string, string>
{
["Foundry-Features"] = "Toolboxes=V1Preview",
},
},
httpClient));
@@ -74,7 +75,7 @@ Console.WriteLine($"Assistant: {await agent.RunAsync(Query)}");
// ---------------------------------------------------------------------------
// Helper: create (or replace) a sample toolbox so the sample runs end-to-end
// ---------------------------------------------------------------------------
static async Task CreateSampleToolboxAsync(string name, string endpoint, TokenCredential credential)
static async Task<string> CreateSampleToolboxAsync(string name, string endpoint, TokenCredential credential)
{
// Toolboxes are normally configured in the Foundry portal or a deployment
// script, not the application itself. This helper exists so the sample can
@@ -103,12 +104,13 @@ static async Task CreateSampleToolboxAsync(string name, string endpoint, TokenCr
serverUri: new Uri("https://gitmcp.io/Azure/azure-rest-api-specs"),
toolCallApprovalPolicy: new McpToolCallApprovalPolicy(GlobalMcpToolCallApprovalPolicy.NeverRequireApproval)));
var created = (await toolboxClient.CreateToolboxVersionAsync(
ToolboxVersion created = (await toolboxClient.CreateToolboxVersionAsync(
name: name,
tools: [mcpTool],
description: "Sample toolbox with an MCP tool — created by Agent_Step25 sample.")).Value;
Console.WriteLine($"Created toolbox '{created.Name}' v{created.Version} ({created.Tools.Count} tool(s))");
return $"{endpoint}/toolboxes/{created.Name}/mcp?api-version=v{created.Version}";
}
// ---------------------------------------------------------------------------
@@ -19,10 +19,11 @@ Set the following environment variables:
```powershell
$env:AZURE_AI_PROJECT_ENDPOINT="https://your-foundry-service.services.ai.azure.com/api/projects/your-foundry-project"
$env:AZURE_AI_MODEL_DEPLOYMENT_NAME="gpt-5.4-mini"
$env:FOUNDRY_TOOLBOX_ENDPOINT="https://your-foundry-service.services.ai.azure.com/api/projects/your-foundry-project/toolsets/research_toolbox/mcp?api-version=2025-05-01-preview"
```
The `<name>` segment of `FOUNDRY_TOOLBOX_ENDPOINT` must match the `ToolboxName` constant in `Program.cs`.
The sample creates a toolbox named `research_toolbox` in your Foundry project on
startup, then connects to its MCP endpoint at
`{AZURE_AI_PROJECT_ENDPOINT}/toolboxes/research_toolbox/mcp?api-version=v{version}`.
## Run the sample
@@ -13,6 +13,7 @@
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\..\..\..\src\Microsoft.Agents.AI.Harness\Microsoft.Agents.AI.Harness.csproj" />
<ProjectReference Include="..\..\..\..\src\Microsoft.Agents.AI.OpenAI\Microsoft.Agents.AI.OpenAI.csproj" />
<ProjectReference Include="..\Harness_Shared_Console\Harness_Shared_Console.csproj" />
</ItemGroup>
@@ -1,6 +1,6 @@
// Copyright (c) Microsoft. All rights reserved.
// This sample demonstrates how to use a ChatClientAgent with the Harness AIContextProviders
// This sample demonstrates how to use a HarnessAgent with the Harness AIContextProviders
// (TodoProvider and AgentModeProvider) for interactive research tasks with web search
// capabilities powered by Azure AI Foundry.
// The agent plans research tasks, creates a todo list, gets user approval,
@@ -17,7 +17,6 @@ using System.ClientModel.Primitives;
using Azure.Identity;
using Harness.Shared.Console;
using Microsoft.Agents.AI;
using Microsoft.Agents.AI.Compaction;
using Microsoft.Extensions.AI;
using OpenAI;
using OpenAI.Responses;
@@ -29,7 +28,7 @@ var deploymentName = Environment.GetEnvironmentVariable("AZURE_AI_MODEL_DEPLOYME
const int MaxContextWindowTokens = 1_050_000;
const int MaxOutputTokens = 128_000;
// Create a ChatClientAgent with the Harness providers (TodoProvider and AgentModeProvider)
// Create a HarnessAgent with the Harness providers (TodoProvider and AgentModeProvider)
// and research-focused instructions including the mandatory planning workflow.
var instructions =
"""
@@ -110,13 +109,9 @@ var instructions =
- Check for relevant previously downloaded data / findings before starting new research.
""";
// Create a compaction strategy based on the model's context window.
// gpt-5.4: 1,050,000 token context window, 128,000 max output tokens.
// Defaults: tool result eviction at 50% of input budget, truncation at 80%.
var compactionStrategy = new ContextWindowCompactionStrategy(
maxContextWindowTokens: MaxContextWindowTokens,
maxOutputTokens: MaxOutputTokens);
// Create the agent using AsHarnessAgent, which pre-configures function invocation,
// per-service-call chat history persistence, and in-loop compaction.
// Then wrap with UseToolApproval to allow auto-approving tools once confirmed.
AIAgent agent =
// Create an OpenAIClient that communicates with the Foundry responses service.
new OpenAIClient(
@@ -130,49 +125,32 @@ AIAgent agent =
RetryPolicy = new ClientRetryPolicy(3) // Enable retries to improve resiliency.
})
.GetResponsesClient()
.AsIChatClientWithStoredOutputDisabled(deploymentName) // We want to manage chat history locally (not stored in the responses service), so that we can manage compaction ourselves.
// Build a ChatClient Pipeline
.AsBuilder()
.UseFunctionInvocation() // We are building our own stack from scratch so we need to include Function Invocation ourselves.
.UseMessageInjection() // Allow message injection during the function call loop.
.UsePerServiceCallChatHistoryPersistence() // Save chat history updates to the session after each service call, rather than only at the end of the run.
.UseAIContextProviders(new CompactionProvider(compactionStrategy)) // Add Compaction before each service call to responses so that long function invocation loops don't overflow the context.
// Build our agent on top of the ChatClient Pipeline
.BuildAIAgent(
new ChatClientAgentOptions
.AsIChatClientWithStoredOutputDisabled(deploymentName) // We want to manage chat history locally (not stored in the responses service), so that we can manage compaction ourselves.
.AsHarnessAgent(MaxContextWindowTokens, MaxOutputTokens, new HarnessAgentOptions
{
Name = "ResearchAgent",
Description = "A research assistant that plans and executes research tasks.",
AIContextProviders =
[
new TodoProvider(), // Add an AIContextProvider to allow the agent to create a TODO list, which is stored in the session.
new AgentModeProvider(), // Add an AIContextProvider that tracks the agent mode and allows switching mode. Current mode is stored in the session.
new FileMemoryProvider( // Add an AIContextProvider that can store memories in files under a session specific working folder.
new FileSystemAgentFileStore(Path.Combine(AppContext.BaseDirectory, "agent-files")),
(_) => new FileMemoryState() { WorkingFolder = DateTime.UtcNow.ToString("yyyyMMdd_HHmmss") + "_" + Guid.NewGuid().ToString() })
],
ChatOptions = new ChatOptions
{
Name = "ResearchAgent",
Description = "A research assistant that plans and executes research tasks.",
UseProvidedChatClientAsIs = true, // Since we built our own stack from scratch we need to tell the agent not to also add defaults like Function Invocation.
RequirePerServiceCallChatHistoryPersistence = true, // Since we are added the per service call persistence ChatClient, we need to tell the agent to not also store chat history at the end of the run.
ChatHistoryProvider = new InMemoryChatHistoryProvider( // Store chat history in memory in the session object. Will persist if the session is persisted.
new InMemoryChatHistoryProviderOptions
{
ChatReducer = compactionStrategy.AsChatReducer(), // Run compaction on the InMemory chat history when it gets too large.
}),
AIContextProviders =
Instructions = instructions,
Tools =
[
new TodoProvider(), // Add an AIContextProvider to allow the agent to create a TODO list, which is stored in the session.
new AgentModeProvider(), // Add an AIContextProvider that tracks the agent mode and allows switching mode. Current mode is stored in the session.
new FileMemoryProvider( // Add an AIContextProvider that can store memories in files under a session specific working folder.
new FileSystemAgentFileStore(Path.Combine(AppContext.BaseDirectory, "agent-files")),
(_) => new FileMemoryState() { WorkingFolder = DateTime.UtcNow.ToString("yyyyMMdd_HHmmss") + "_" + Guid.NewGuid().ToString() })
ResponseTool.CreateWebSearchTool().AsAITool(), // Add the foundry hosted web search tool that runs in the service.
new WebBrowsingTool( // Add a local web browsing tool that converts html to markdown.
new WebBrowsingToolOptions { AllowPublicNetworks = true }),
],
ChatOptions = new ChatOptions
{
Instructions = instructions,
Tools =
[
ResponseTool.CreateWebSearchTool().AsAITool(), // Add the foundry hosted web search tool that runs in the service.
new WebBrowsingTool( // Add a local web browsing tool that converts html to markdown.
new WebBrowsingToolOptions { AllowPublicNetworks = true }),
],
MaxOutputTokens = MaxOutputTokens, // Set a high token limit for long research tasks with many tool calls and long outputs.
Reasoning = new() { Effort = ReasoningEffort.Medium },
},
})
MaxOutputTokens = MaxOutputTokens, // Set a high token limit for long research tasks with many tool calls and long outputs.
Reasoning = new() { Effort = ReasoningEffort.Medium },
},
})
.AsBuilder()
.UseToolApproval() // Add the ability to auto approve tools once a user has said they don't want to be asked again. Approval rules are tied to the session.
.Build();
@@ -1,10 +1,11 @@
# What this sample demonstrates
This sample demonstrates how to use a `ChatClientAgent` with the Harness `AIContextProviders` (`TodoProvider` and `AgentModeProvider`) for interactive research tasks with web search capabilities powered by Azure AI Foundry.
This sample demonstrates how to use a `HarnessAgent` with the Harness `AIContextProviders` (`TodoProvider` and `AgentModeProvider`) for interactive research tasks with web search capabilities powered by Azure AI Foundry. The `HarnessAgent` pre-configures function invocation, per-service-call chat history persistence, and context-window compaction.
Key features showcased:
- **ChatClientAgent** — configured directly with Harness providers for planning and task management
- **HarnessAgent** — a pre-configured agent that wraps a `ChatClientAgent` with function invocation, per-service-call persistence, and context-window compaction
- **ToolApproval** — the agent is wrapped with `UseToolApproval()` to allow auto-approving tools once confirmed
- **Web Search** — the agent can search the web for current information via `ResponseTool.CreateWebSearchTool()`
- **TodoProvider** — the agent creates and manages a todo list to track research questions
- **AgentModeProvider** — the agent switches between "plan" mode (breaking down the topic) and "execute" mode (answering each research question)
@@ -13,6 +13,7 @@
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\..\..\..\src\Microsoft.Agents.AI.Harness\Microsoft.Agents.AI.Harness.csproj" />
<ProjectReference Include="..\..\..\..\src\Microsoft.Agents.AI.OpenAI\Microsoft.Agents.AI.OpenAI.csproj" />
<ProjectReference Include="..\Harness_Shared_Console\Harness_Shared_Console.csproj" />
</ItemGroup>
@@ -22,6 +22,9 @@ using OpenAI.Responses;
var endpoint = Environment.GetEnvironmentVariable("AZURE_FOUNDRY_OPENAI_ENDPOINT") ?? throw new InvalidOperationException("AZURE_FOUNDRY_OPENAI_ENDPOINT is not set.");
var deploymentName = Environment.GetEnvironmentVariable("AZURE_AI_MODEL_DEPLOYMENT_NAME") ?? "gpt-5.4";
const int MaxContextWindowTokens = 1_050_000;
const int MaxOutputTokens = 128_000;
// --- Sub-agent: Web Search Agent ---
// This agent can search the web and is used by the parent agent to look up stock prices.
AIAgent webSearchAgent =
@@ -34,20 +37,19 @@ AIAgent webSearchAgent =
})
.GetResponsesClient()
.AsIChatClientWithStoredOutputDisabled(deploymentName)
.AsAIAgent(
new ChatClientAgentOptions
.AsHarnessAgent(MaxContextWindowTokens, MaxOutputTokens, new HarnessAgentOptions
{
Name = "WebSearchAgent",
Description = "An agent that can search the web to find information.",
ChatOptions = new ChatOptions
{
Name = "WebSearchAgent",
Description = "An agent that can search the web to find information.",
ChatOptions = new ChatOptions
{
Instructions = "You are a web search assistant. When asked to find information, use the web search tool to look it up and return a concise, factual answer.",
Tools =
[
ResponseTool.CreateWebSearchTool().AsAITool(),
],
},
});
Instructions = "You are a web search assistant. When asked to find information, use the web search tool to look it up and return a concise, factual answer.",
Tools =
[
ResponseTool.CreateWebSearchTool().AsAITool(),
],
},
});
// --- Parent agent: Stock Price Researcher ---
// This agent orchestrates the sub-agent to look up stock prices in parallel.
@@ -83,21 +85,20 @@ AIAgent parentAgent =
})
.GetResponsesClient()
.AsIChatClientWithStoredOutputDisabled(deploymentName)
.AsAIAgent(
new ChatClientAgentOptions
.AsHarnessAgent(MaxContextWindowTokens, MaxOutputTokens, new HarnessAgentOptions
{
Name = "StockPriceResearcher",
Description = "An agent that researches stock prices using sub-agents.",
AIContextProviders =
[
new SubAgentsProvider([webSearchAgent]),
],
ChatOptions = new ChatOptions
{
Name = "StockPriceResearcher",
Description = "An agent that researches stock prices using sub-agents.",
AIContextProviders =
[
new SubAgentsProvider([webSearchAgent]),
],
ChatOptions = new ChatOptions
{
Instructions = parentInstructions,
MaxOutputTokens = 16_000,
},
});
Instructions = parentInstructions,
MaxOutputTokens = 16_000,
},
});
// Run the interactive console session.
await HarnessConsole.RunAgentAsync(
@@ -1,6 +1,6 @@
# Harness Step 02 — SubAgents (Stock Price Research)
This sample demonstrates how to use the **SubAgentsProvider** to delegate work from a parent agent to sub-agents.
This sample demonstrates how to use the **SubAgentsProvider** to delegate work from a parent agent to sub-agents. Both agents use `HarnessAgent` for pre-configured function invocation, per-service-call persistence, and context-window compaction.
## What It Does
@@ -13,6 +13,7 @@
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\..\..\..\src\Microsoft.Agents.AI.Harness\Microsoft.Agents.AI.Harness.csproj" />
<ProjectReference Include="..\..\..\..\src\Microsoft.Agents.AI.OpenAI\Microsoft.Agents.AI.OpenAI.csproj" />
<ProjectReference Include="..\Harness_Shared_Console\Harness_Shared_Console.csproj" />
</ItemGroup>
@@ -1,6 +1,6 @@
// Copyright (c) Microsoft. All rights reserved.
// This sample demonstrates how to use a ChatClientAgent with the FileAccessProvider
// This sample demonstrates how to use a HarnessAgent with the FileAccessProvider
// to give an agent access to a folder of CSV data files. The agent can read, analyze,
// and extract information from the data, then write results back as new files.
//
@@ -17,7 +17,6 @@ using System.ClientModel.Primitives;
using Azure.Identity;
using Harness.Shared.Console;
using Microsoft.Agents.AI;
using Microsoft.Agents.AI.Compaction;
using Microsoft.Extensions.AI;
using OpenAI;
using OpenAI.Responses;
@@ -57,11 +56,7 @@ var instructions =
- Always explain what you learned and what you are going to do next between tool calls, so the user can follow along with your thought process.
""";
// Create a compaction strategy based on the model's context window.
var compactionStrategy = new ContextWindowCompactionStrategy(
maxContextWindowTokens: MaxContextWindowTokens,
maxOutputTokens: MaxOutputTokens);
// Create the chat client from the OpenAI provider.
AIAgent agent =
new OpenAIClient(
new BearerTokenPolicy(new DefaultAzureCredential(), "https://ai.azure.com/.default"),
@@ -72,36 +67,20 @@ AIAgent agent =
})
.GetResponsesClient()
.AsIChatClientWithStoredOutputDisabled(deploymentName)
.AsBuilder()
.UseFunctionInvocation()
.UsePerServiceCallChatHistoryPersistence()
.UseAIContextProviders(new CompactionProvider(compactionStrategy))
.BuildAIAgent(
new ChatClientAgentOptions
.AsHarnessAgent(MaxContextWindowTokens, MaxOutputTokens, new HarnessAgentOptions
{
Name = "DataAnalyst",
Description = "A data analyst assistant that reads, analyzes, and processes data files.",
AIContextProviders =
[
new FileAccessProvider(fileStore),
],
ChatOptions = new ChatOptions
{
Name = "DataAnalyst",
Description = "A data analyst assistant that reads, analyzes, and processes data files.",
UseProvidedChatClientAsIs = true,
RequirePerServiceCallChatHistoryPersistence = true,
ChatHistoryProvider = new InMemoryChatHistoryProvider(
new InMemoryChatHistoryProviderOptions
{
ChatReducer = compactionStrategy.AsChatReducer(),
}),
AIContextProviders =
[
new FileAccessProvider(fileStore),
],
ChatOptions = new ChatOptions
{
Instructions = instructions,
MaxOutputTokens = MaxOutputTokens,
},
})
.AsBuilder()
.Build();
Instructions = instructions,
MaxOutputTokens = MaxOutputTokens,
},
});
// Run the interactive console session.
await HarnessConsole.RunAgentAsync(
@@ -1,9 +1,10 @@
# What this sample demonstrates
This sample demonstrates how to use a `ChatClientAgent` with the `FileAccessProvider` to give an agent access to a folder of data files for reading, analyzing, and writing results.
This sample demonstrates how to use a `HarnessAgent` with the `FileAccessProvider` to give an agent access to a folder of data files for reading, analyzing, and writing results. The `HarnessAgent` pre-configures function invocation, per-service-call chat history persistence, and in-loop compaction — so the sample only needs to supply the chat client, token limits, and application-specific options.
Key features showcased:
- **HarnessAgent** — a pre-configured agent that wraps a `ChatClientAgent` with function invocation, per-service-call persistence, and context-window compaction
- **FileAccessProvider** — gives the agent tools to read, write, list, search, and delete files in a shared data folder
- **CSV data processing** — the agent reads sales transaction data and performs analysis on demand
- **Output file creation** — the agent can write summaries, filtered data, or reports back to the data folder
@@ -0,0 +1,16 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFrameworks>net10.0</TargetFrameworks>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\..\..\..\src\Microsoft.Agents.AI.Foundry\Microsoft.Agents.AI.Foundry.csproj" />
<ProjectReference Include="..\..\..\..\src\Microsoft.Agents.AI.Workflows\Microsoft.Agents.AI.Workflows.csproj" />
</ItemGroup>
</Project>
@@ -0,0 +1,76 @@
// Copyright (c) Microsoft. All rights reserved.
// This sample demonstrates evaluating a multi-agent workflow against a
// golden answer using Foundry's reference-based Similarity evaluator.
using Azure.AI.Projects;
using Azure.Identity;
using Microsoft.Agents.AI;
using Microsoft.Agents.AI.Workflows;
using Microsoft.Extensions.AI;
using FoundryEvals = Microsoft.Agents.AI.Foundry.FoundryEvals;
string endpoint = Environment.GetEnvironmentVariable("AZURE_AI_PROJECT_ENDPOINT")
?? throw new InvalidOperationException("AZURE_AI_PROJECT_ENDPOINT is not set.");
string deploymentName = Environment.GetEnvironmentVariable("AZURE_AI_MODEL_DEPLOYMENT_NAME") ?? "gpt-4o-mini";
// WARNING: DefaultAzureCredential is convenient for development but requires careful consideration in production.
// In production, consider using a specific credential (e.g., ManagedIdentityCredential) to avoid
// latency issues, unintended credential probing, and potential security risks from fallback mechanisms.
AIProjectClient projectClient = new(new Uri(endpoint), new DefaultAzureCredential());
// Build a two-agent workflow: a researcher writes a draft answer, then an
// editor polishes it into the final response that we compare to ground truth.
// EmitAgentResponseEvents is enabled so the workflow surfaces an AgentResponseEvent
// for each agent — this is what EvaluateAsync uses to find the overall final answer.
var hostOptions = new AIAgentHostOptions { EmitAgentResponseEvents = true };
AIAgent researcher = projectClient.AsAIAgent(
model: deploymentName,
instructions: "You research questions and produce a short factual draft answer.",
name: "researcher");
AIAgent editor = projectClient.AsAIAgent(
model: deploymentName,
instructions: "You take a draft answer and produce the final concise response.",
name: "editor");
ExecutorBinding researcherExecutor = researcher.BindAsExecutor(hostOptions);
ExecutorBinding editorExecutor = editor.BindAsExecutor(hostOptions);
Workflow workflow = new WorkflowBuilder(researcherExecutor)
.AddEdge(researcherExecutor, editorExecutor)
.Build();
// Run the workflow against the user question.
const string Query = "What is the capital of France?";
const string GroundTruth = "Paris";
await using Run run = await InProcessExecution.RunAsync(
workflow,
new ChatMessage(ChatRole.User, Query));
// Evaluate the overall workflow output against a golden answer using the
// reference-based Similarity evaluator. The 'expectedOutput' value is stamped
// onto the overall EvalItem.ExpectedOutput and is surfaced to Foundry as
// `ground_truth` in the underlying JSONL payload.
//
// Per-agent breakdown is disabled here: ground truth applies to the workflow's
// final answer, not to each sub-agent's intermediate output. Without
// includePerAgent: false, the evaluator would be invoked for per-agent items
// (which have no ExpectedOutput) and Similarity would fail validation.
FoundryEvals similarity = new(projectClient, deploymentName, FoundryEvals.Similarity);
AgentEvaluationResults results = await run.EvaluateAsync(
similarity,
includePerAgent: false,
expectedOutput: GroundTruth);
Console.WriteLine($"Query: {Query}");
Console.WriteLine($"Expected: {GroundTruth}");
Console.WriteLine($"Provider: {results.ProviderName}");
Console.WriteLine($"Passed: {results.Passed}/{results.Total}");
if (results.ReportUrl is not null)
{
Console.WriteLine($"Report: {results.ReportUrl}");
}
@@ -0,0 +1,37 @@
# Evaluation - Workflow Expected Outputs
This sample demonstrates evaluating a multi-agent workflow's final answer
against a golden expected output using Foundry's reference-based **Similarity**
evaluator.
## What this sample demonstrates
- Building a small researcher → editor workflow
- Running the workflow and obtaining a `Run`
- Calling `run.EvaluateAsync(evaluator, expectedOutput: ...)` to attach a
ground-truth answer to the overall workflow item
- Using `FoundryEvals.Similarity`, which requires a `ground_truth` value
per item
The `expectedOutput` value is stamped onto the overall `EvalItem.ExpectedOutput`
and is surfaced to Foundry as `ground_truth` in the JSONL payload sent to the
Evals API.
## Prerequisites
- .NET 10 SDK or later
- Azure CLI installed and authenticated (`az login`)
Set the following environment variables:
```powershell
$env:AZURE_AI_PROJECT_ENDPOINT="https://your-foundry-service.services.ai.azure.com/api/projects/your-foundry-project"
$env:AZURE_AI_MODEL_DEPLOYMENT_NAME="gpt-4o-mini"
```
## Run the sample
```powershell
cd dotnet/samples/03-workflows/Evaluation
dotnet run --project .\Evaluation_WorkflowExpectedOutputs
```
@@ -163,10 +163,22 @@ app.MapA2AHttpJson(knightsKnavesAgentBuilder, path: "/a2a/knights-and-knaves");
app.MapDevUI();
app.MapOpenAIResponses();
app.MapOpenAIResponses(pirateAgentBuilder);
app.MapOpenAIResponses(knightsKnavesAgentBuilder);
app.MapOpenAIResponses(chemistryAgent);
app.MapOpenAIResponses(mathsAgent);
app.MapOpenAIResponses(literatureAgent);
app.MapOpenAIResponses(scienceSequentialWorkflow);
app.MapOpenAIResponses(scienceConcurrentWorkflow);
app.MapOpenAIConversations();
app.MapOpenAIChatCompletions(pirateAgentBuilder);
app.MapOpenAIChatCompletions(knightsKnavesAgentBuilder);
app.MapOpenAIChatCompletions(chemistryAgent);
app.MapOpenAIChatCompletions(mathsAgent);
app.MapOpenAIChatCompletions(literatureAgent);
app.MapOpenAIChatCompletions(scienceSequentialWorkflow);
app.MapOpenAIChatCompletions(scienceConcurrentWorkflow);
// Map the agents HTTP endpoints
app.MapAgentDiscovery("/agents");
@@ -24,7 +24,7 @@ internal sealed class OpenAIChatCompletionsAgentClient(HttpClient httpClient) :
{
OpenAIClientOptions options = new()
{
Endpoint = new Uri(httpClient.BaseAddress!, $"/{agentName}/v1/"),
Endpoint = new Uri(httpClient.BaseAddress!, $"/{Uri.EscapeDataString(agentName)}/v1/"),
Transport = new HttpClientPipelineTransport(httpClient)
};
@@ -23,7 +23,7 @@ internal sealed class OpenAIResponsesAgentClient(HttpClient httpClient) : AgentC
{
OpenAIClientOptions options = new()
{
Endpoint = new Uri(httpClient.BaseAddress!, "/v1/"),
Endpoint = new Uri(httpClient.BaseAddress!, $"/{Uri.EscapeDataString(agentName)}/v1/"),
Transport = new HttpClientPipelineTransport(httpClient)
};
@@ -130,6 +130,7 @@ internal static class FoundryEvalConverter
QueryMessages = ConvertMessages(queryMessages),
ResponseMessages = ConvertMessages(responseMessages),
Context = item.Context,
GroundTruth = item.ExpectedOutput,
ToolDefinitions = item.Tools is { Count: > 0 }
? item.Tools
.OfType<AIFunction>()
@@ -185,6 +186,11 @@ internal static class FoundryEvalConverter
dataMapping["context"] = "{{item.context}}";
}
if (GroundTruthEvaluators.Contains(qualified))
{
dataMapping["ground_truth"] = "{{item.ground_truth}}";
}
if (ToolEvaluators.Contains(qualified))
{
dataMapping["tool_definitions"] = "{{item.tool_definitions}}";
@@ -206,7 +212,7 @@ internal static class FoundryEvalConverter
/// <summary>
/// Builds the <c>item_schema</c> for custom JSONL eval definitions.
/// </summary>
internal static WireItemSchema BuildItemSchema(bool hasContext = false, bool hasTools = false)
internal static WireItemSchema BuildItemSchema(bool hasContext = false, bool hasTools = false, bool hasGroundTruth = false)
{
var properties = new Dictionary<string, WireSchemaProperty>
{
@@ -221,6 +227,11 @@ internal static class FoundryEvalConverter
properties["context"] = new WireSchemaProperty { Type = "string" };
}
if (hasGroundTruth)
{
properties["ground_truth"] = new WireSchemaProperty { Type = "string" };
}
if (hasTools)
{
properties["tool_definitions"] = new WireSchemaProperty { Type = "array" };
@@ -233,6 +244,31 @@ internal static class FoundryEvalConverter
};
}
/// <summary>
/// Returns the subset of <paramref name="evaluators"/> that require a ground-truth
/// (reference) value but cannot be evaluated because no item provided one.
/// </summary>
internal static List<string> FindMissingGroundTruthEvaluators(
IEnumerable<string> evaluators,
bool hasGroundTruth)
{
if (hasGroundTruth)
{
return [];
}
var missing = new List<string>();
foreach (var name in evaluators)
{
if (GroundTruthEvaluators.Contains(ResolveEvaluator(name)))
{
missing.Add(name);
}
}
return missing;
}
/// <summary>
/// Resolves a short evaluator name to its fully-qualified <c>builtin.*</c> form.
/// </summary>
@@ -277,6 +313,12 @@ internal static class FoundryEvalConverter
"builtin.tool_call_success",
};
// Evaluators that require a ground_truth (reference) value per item.
internal static readonly HashSet<string> GroundTruthEvaluators = new(StringComparer.OrdinalIgnoreCase)
{
"builtin.similarity",
};
// Short name → fully-qualified name mapping.
internal static readonly Dictionary<string, string> BuiltinEvaluators = new(StringComparer.OrdinalIgnoreCase)
{
@@ -103,6 +103,9 @@ internal sealed class WireEvalItemPayload
[JsonPropertyName("context")]
public string? Context { get; init; }
[JsonPropertyName("ground_truth")]
public string? GroundTruth { get; init; }
[JsonPropertyName("tool_definitions")]
public List<WireToolDefinition>? ToolDefinitions { get; init; }
}
@@ -145,6 +145,8 @@ public sealed class FoundryEvals : IAgentEvaluator
bool hasContext = payloads.Any(p => p.Context is not null);
bool hasTools = payloads.Any(p => p.ToolDefinitions is { Count: > 0 });
bool hasGroundTruth = payloads.Any(p => p.GroundTruth is not null);
bool allHaveGroundTruth = payloads.Count > 0 && payloads.All(p => p.GroundTruth is not null);
// Filter out tool evaluators if no items have tools; auto-add ToolCallAccuracy if tools present
var evaluators = FilterToolEvaluators(this._evaluatorNames, hasTools);
@@ -153,13 +155,27 @@ public sealed class FoundryEvals : IAgentEvaluator
evaluators = [.. evaluators, ToolCallAccuracy];
}
// Fail fast if a ground-truth evaluator (e.g. similarity) is requested but not
// every item carries an ExpectedOutput. Reference-based evaluators score each
// item against its own ground truth, so even one missing value will surface as
// a provider-side validation error. Catch it here with a clearer message.
var missingGroundTruth = FoundryEvalConverter.FindMissingGroundTruthEvaluators(evaluators, allHaveGroundTruth);
if (missingGroundTruth.Count > 0)
{
throw new InvalidOperationException(
"The following evaluator(s) require a ground-truth/expected output on every item but " +
$"at least one item is missing an {nameof(EvalItem.ExpectedOutput)}: {string.Join(", ", missingGroundTruth)}. " +
"Provide an expected output per item (for example via the 'expectedOutput' parameter on EvaluateAsync), " +
"or set 'includePerAgent: false' so the evaluator only runs on the overall item.");
}
// 2. Create the evaluation definition
var createEvalPayload = new WireCreateEvalRequest
{
Name = evalName,
DataSourceConfig = new WireCustomDataSourceConfig
{
ItemSchema = FoundryEvalConverter.BuildItemSchema(hasContext, hasTools),
ItemSchema = FoundryEvalConverter.BuildItemSchema(hasContext, hasTools, hasGroundTruth),
},
TestingCriteria = FoundryEvalConverter.BuildTestingCriteria(
evaluators, this._model, includeDataMapping: true),
@@ -822,15 +838,15 @@ public sealed class FoundryEvals : IAgentEvaluator
var result = new EvalItemResult(itemId, status, scores);
// Extract error info from sample
if (outputItem.TryGetProperty("sample", out var sample))
if (outputItem.TryGetProperty("sample", out var sample) && sample.ValueKind == JsonValueKind.Object)
{
if (sample.TryGetProperty("error", out var errObj))
if (sample.TryGetProperty("error", out var errObj) && errObj.ValueKind == JsonValueKind.Object)
{
result.ErrorCode = errObj.TryGetProperty("code", out var code) ? code.GetString() : null;
result.ErrorMessage = errObj.TryGetProperty("message", out var msg) ? msg.GetString() : null;
}
if (sample.TryGetProperty("usage", out var usage) && usage.TryGetProperty("total_tokens", out var tt) && tt.ValueKind == JsonValueKind.Number)
if (sample.TryGetProperty("usage", out var usage) && usage.ValueKind == JsonValueKind.Object && usage.TryGetProperty("total_tokens", out var tt) && tt.ValueKind == JsonValueKind.Number)
{
var tokenUsage = new Dictionary<string, int>();
if (usage.TryGetProperty("prompt_tokens", out var pt) && pt.ValueKind == JsonValueKind.Number)
@@ -886,7 +902,7 @@ public sealed class FoundryEvals : IAgentEvaluator
}
// Extract response_id from datasource_item
if (outputItem.TryGetProperty("datasource_item", out var dsItem))
if (outputItem.TryGetProperty("datasource_item", out var dsItem) && dsItem.ValueKind == JsonValueKind.Object)
{
if (dsItem.TryGetProperty("resp_id", out var respId))
{
@@ -0,0 +1,42 @@
// Copyright (c) Microsoft. All rights reserved.
using System.Diagnostics.CodeAnalysis;
using Microsoft.Agents.AI;
using Microsoft.Shared.DiagnosticIds;
namespace Microsoft.Extensions.AI;
/// <summary>
/// Provides extension methods for creating a <see cref="HarnessAgent"/> from an <see cref="IChatClient"/>.
/// </summary>
[Experimental(DiagnosticIds.Experiments.AgentsAIExperiments)]
public static class ChatClientHarnessExtensions
{
/// <summary>
/// Creates a new <see cref="HarnessAgent"/> that wraps this <see cref="IChatClient"/> with a pre-configured
/// pipeline including function invocation, per-service-call chat history persistence, and in-loop compaction.
/// </summary>
/// <param name="chatClient">
/// The <see cref="IChatClient"/> that provides access to the underlying AI model.
/// </param>
/// <param name="maxContextWindowTokens">
/// The maximum number of tokens the model's context window supports (e.g., 1,050,000 for gpt-5.4).
/// Used to configure the compaction strategy.
/// </param>
/// <param name="maxOutputTokens">
/// The maximum number of output tokens the model can generate per response (e.g., 128,000 for gpt-5.4).
/// Used to configure the compaction strategy.
/// </param>
/// <param name="options">
/// Optional configuration options for the agent, including instructions override, tools,
/// additional context providers, and chat history provider.
/// When <see langword="null"/>, the agent uses built-in default settings.
/// </param>
/// <returns>A new <see cref="HarnessAgent"/> instance.</returns>
public static HarnessAgent AsHarnessAgent(
this IChatClient chatClient,
int maxContextWindowTokens,
int maxOutputTokens,
HarnessAgentOptions? options = null) =>
new(chatClient, maxContextWindowTokens, maxOutputTokens, options);
}
@@ -0,0 +1,135 @@
// Copyright (c) Microsoft. All rights reserved.
using System.Diagnostics.CodeAnalysis;
using Microsoft.Agents.AI.Compaction;
using Microsoft.Extensions.AI;
using Microsoft.Shared.DiagnosticIds;
using Microsoft.Shared.Diagnostics;
namespace Microsoft.Agents.AI;
/// <summary>
/// A pre-configured <see cref="DelegatingAIAgent"/> that wraps a <see cref="ChatClientAgent"/> with
/// function invocation, per-service-call chat history persistence, and in-loop compaction.
/// </summary>
/// <remarks>
/// <para>
/// <see cref="HarnessAgent"/> assembles the following pipeline from a caller-supplied <see cref="IChatClient"/>:
/// <list type="number">
/// <item><description><see cref="FunctionInvokingChatClient"/> — automatic function/tool invocation.</description></item>
/// <item><description><see cref="PerServiceCallChatHistoryPersistingChatClient"/> — persists chat history after every individual service call within a function-invocation loop.</description></item>
/// <item><description><see cref="AIContextProviderChatClient"/> with a <see cref="CompactionProvider"/> — applies context-window compaction before each call so long function-invocation loops do not overflow the context window.</description></item>
/// </list>
/// </para>
/// <para>
/// The underlying <see cref="ChatClientAgent"/> is configured with
/// <see cref="ChatClientAgentOptions.UseProvidedChatClientAsIs"/> and
/// <see cref="ChatClientAgentOptions.RequirePerServiceCallChatHistoryPersistence"/> set to <see langword="true"/>
/// to match the manually-assembled pipeline.
/// </para>
/// <para>
/// When no <see cref="HarnessAgentOptions.ChatHistoryProvider"/> is supplied, the agent defaults to an
/// <see cref="InMemoryChatHistoryProvider"/> whose chat reducer applies the same compaction strategy,
/// keeping in-memory history from growing unboundedly across sessions.
/// </para>
/// </remarks>
[Experimental(DiagnosticIds.Experiments.AgentsAIExperiments)]
public sealed class HarnessAgent : DelegatingAIAgent
{
/// <summary>
/// The built-in default system instructions used when <see cref="ChatOptions.Instructions"/> is not set.
/// </summary>
public const string DefaultInstructions =
"""
You are a helpful AI assistant that uses tools to complete tasks.
## General guidelines
- Think through the task before acting. Break complex work into clear steps.
- Use the tools available to you to gather information, perform actions, and verify results.
- Explain your reasoning between tool calls so the user can follow your progress.
- If a tool call fails or returns unexpected results, adapt your approach rather than repeating the same call.
- When you have completed the task, present a clear and concise summary of what you did and what you found.
""";
/// <summary>
/// Initializes a new instance of the <see cref="HarnessAgent"/> class.
/// </summary>
/// <param name="chatClient">
/// The <see cref="IChatClient"/> that provides access to the underlying AI model.
/// The agent wraps this client in a function-invocation, per-service-call persistence,
/// and compaction pipeline automatically.
/// </param>
/// <param name="maxContextWindowTokens">
/// The maximum number of tokens the model's context window supports (e.g., 1,050,000 for gpt-5.4).
/// Used to configure the compaction strategy.
/// </param>
/// <param name="maxOutputTokens">
/// The maximum number of output tokens the model can generate per response (e.g., 128,000 for gpt-5.4).
/// Used to configure the compaction strategy and to limit the model's output.
/// </param>
/// <param name="options">
/// Optional configuration options for the agent, including instructions override, tools,
/// additional context providers, and chat history provider.
/// When <see langword="null"/>, the agent uses built-in default settings.
/// </param>
/// <exception cref="System.ArgumentNullException">
/// <paramref name="chatClient"/> is <see langword="null"/>.
/// </exception>
/// <exception cref="System.ArgumentOutOfRangeException">
/// <paramref name="maxContextWindowTokens"/> is not positive, or
/// <paramref name="maxOutputTokens"/> is negative or greater than or equal to <paramref name="maxContextWindowTokens"/>.
/// </exception>
public HarnessAgent(IChatClient chatClient, int maxContextWindowTokens, int maxOutputTokens, HarnessAgentOptions? options = null)
: base(BuildInnerAgent(
Throw.IfNull(chatClient),
maxContextWindowTokens,
maxOutputTokens,
options))
{
}
private static ChatClientAgent BuildInnerAgent(IChatClient chatClient, int maxContextWindowTokens, int maxOutputTokens, HarnessAgentOptions? options)
{
var compactionStrategy = new ContextWindowCompactionStrategy(
maxContextWindowTokens: maxContextWindowTokens,
maxOutputTokens: maxOutputTokens);
ChatHistoryProvider chatHistoryProvider = options?.ChatHistoryProvider
?? new InMemoryChatHistoryProvider(new InMemoryChatHistoryProviderOptions
{
ChatReducer = compactionStrategy.AsChatReducer(),
});
string instructions = options?.ChatOptions?.Instructions ?? DefaultInstructions;
ChatOptions chatOptions = BuildChatOptions(options?.ChatOptions, instructions, maxOutputTokens);
var compactionProvider = new CompactionProvider(compactionStrategy);
return chatClient
.AsBuilder()
.UseFunctionInvocation()
.UsePerServiceCallChatHistoryPersistence()
.UseAIContextProviders(compactionProvider)
.BuildAIAgent(new ChatClientAgentOptions
{
Id = options?.Id,
Name = options?.Name,
Description = options?.Description,
ChatOptions = chatOptions,
ChatHistoryProvider = chatHistoryProvider,
AIContextProviders = options?.AIContextProviders,
UseProvidedChatClientAsIs = true,
RequirePerServiceCallChatHistoryPersistence = true,
});
}
private static ChatOptions BuildChatOptions(ChatOptions? source, string instructions, int maxOutputTokens)
{
ChatOptions result = source?.Clone() ?? new ChatOptions();
result.Instructions = instructions;
result.MaxOutputTokens ??= maxOutputTokens;
return result;
}
}
@@ -0,0 +1,64 @@
// Copyright (c) Microsoft. All rights reserved.
using System.Collections.Generic;
using System.Diagnostics.CodeAnalysis;
using Microsoft.Extensions.AI;
using Microsoft.Shared.DiagnosticIds;
namespace Microsoft.Agents.AI;
/// <summary>
/// Represents configuration options for a <see cref="HarnessAgent"/>.
/// </summary>
[Experimental(DiagnosticIds.Experiments.AgentsAIExperiments)]
public sealed class HarnessAgentOptions
{
/// <summary>
/// Gets or sets the agent id.
/// </summary>
public string? Id { get; set; }
/// <summary>
/// Gets or sets the agent name.
/// </summary>
public string? Name { get; set; }
/// <summary>
/// Gets or sets the agent description.
/// </summary>
public string? Description { get; set; }
/// <summary>
/// Gets or sets additional chat options such as tools for the agent to use.
/// </summary>
/// <remarks>
/// <para>
/// Use <see cref="ChatOptions.Tools"/> to supply additional tools the agent can invoke.
/// </para>
/// <para>
/// Use <see cref="ChatOptions.Instructions"/> to override the <see cref="HarnessAgent"/>'s built-in
/// default instructions. When <see cref="ChatOptions.Instructions"/> is <see langword="null"/> or not set,
/// the default instructions are used.
/// </para>
/// </remarks>
public ChatOptions? ChatOptions { get; set; }
/// <summary>
/// Gets or sets the <see cref="ChatHistoryProvider"/> to use for storing chat history.
/// </summary>
/// <remarks>
/// When <see langword="null"/>, the agent defaults to an <see cref="InMemoryChatHistoryProvider"/>
/// configured with a compaction-based chat reducer derived from the <c>maxContextWindowTokens</c>
/// and <c>maxOutputTokens</c> constructor parameters of <see cref="HarnessAgent"/>.
/// </remarks>
public ChatHistoryProvider? ChatHistoryProvider { get; set; }
/// <summary>
/// Gets or sets additional <see cref="AIContextProvider"/> instances to include in the agent pipeline.
/// </summary>
/// <remarks>
/// These providers are passed to the underlying <see cref="ChatClientAgent"/> via
/// <see cref="ChatClientAgentOptions.AIContextProviders"/>.
/// </remarks>
public IEnumerable<AIContextProvider>? AIContextProviders { get; set; }
}
@@ -0,0 +1,28 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<IsReleaseCandidate>false</IsReleaseCandidate>
<InjectSharedThrow>true</InjectSharedThrow>
<InjectSharedDiagnosticIds>true</InjectSharedDiagnosticIds>
<InjectExperimentalAttributeOnLegacy>true</InjectExperimentalAttributeOnLegacy>
<InjectIsExternalInitOnLegacy>true</InjectIsExternalInitOnLegacy>
<InjectTrimAttributesOnLegacy>true</InjectTrimAttributesOnLegacy>
</PropertyGroup>
<Import Project="$(RepoRoot)/dotnet/nuget/nuget-package.props" />
<ItemGroup>
<ProjectReference Include="..\Microsoft.Agents.AI\Microsoft.Agents.AI.csproj" />
</ItemGroup>
<PropertyGroup>
<!-- NuGet Package Settings -->
<Title>Microsoft Agent Framework Harness</Title>
<Description>Provides the HarnessAgent, a pre-configured AI agent that can be used for long running tasks.</Description>
</PropertyGroup>
<ItemGroup>
<InternalsVisibleTo Include="Microsoft.Agents.AI.Harness.UnitTests" />
</ItemGroup>
</Project>
@@ -32,38 +32,8 @@ internal static class AIAgentsAbstractionsExtensions
return message;
}
/// <summary>
/// Iterates through <paramref name="messages"/> looking for <see cref="ChatRole.Assistant"/> messages and swapping
/// any that have a different <see cref="ChatMessage.AuthorName"/> from <paramref name="targetAgentName"/> to
/// <see cref="ChatRole.User"/>.
/// </summary>
public static List<ChatMessage>? ChangeAssistantToUserForOtherParticipants(this IEnumerable<ChatMessage> messages, string targetAgentName)
{
List<ChatMessage>? roleChanged = null;
foreach (var m in messages)
{
m.ChatAssistantToUserIfNotFromNamed(targetAgentName, out bool changed);
if (changed)
{
(roleChanged ??= []).Add(m);
}
}
return roleChanged;
}
/// <summary>
/// Undoes changes made by <see cref="ChangeAssistantToUserForOtherParticipants"/> when passed the list of changes
/// made by that method.
/// </summary>
public static void ResetUserToAssistantForChangedRoles(this List<ChatMessage>? roleChanged)
{
if (roleChanged is not null)
{
foreach (var m in roleChanged)
{
m.Role = ChatRole.Assistant;
}
}
}
public static List<ChatMessage> CopyWithAssistantToUserForOtherParticipants(
this IEnumerable<ChatMessage> messages,
string targetAgentName)
=> messages.Select(m => m.ChatAssistantToUserIfNotFromNamed(targetAgentName, out _, false)).ToList();
}
@@ -28,6 +28,17 @@ public static class WorkflowEvaluationExtensions
/// Use <see cref="ConversationSplitters.LastTurn"/>, <see cref="ConversationSplitters.Full"/>,
/// or a custom <see cref="IConversationSplitter"/> implementation.
/// </param>
/// <param name="expectedOutput">
/// Optional ground-truth/expected output for the workflow's overall final answer.
/// When provided, it is stamped onto the overall <see cref="EvalItem.ExpectedOutput"/>
/// so reference-based evaluators (for example, similarity) can compare the
/// workflow's response against a golden answer. Ground truth is only applied
/// to the overall item; per-agent items are intentionally left without an
/// expected output, since ground truth is defined against the final response.
/// When using a reference-based evaluator that requires ground truth, set
/// <paramref name="includePerAgent"/> to <see langword="false"/> to avoid
/// invoking the evaluator on per-agent items that have no expected output.
/// </param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>Evaluation results with optional per-agent sub-results.</returns>
public static async Task<AgentEvaluationResults> EvaluateAsync(
@@ -37,6 +48,7 @@ public static class WorkflowEvaluationExtensions
bool includePerAgent = true,
string evalName = "Workflow Eval",
IConversationSplitter? splitter = null,
string? expectedOutput = null,
CancellationToken cancellationToken = default)
{
var events = run.OutgoingEvents.ToList();
@@ -48,28 +60,26 @@ public static class WorkflowEvaluationExtensions
var overallItems = new List<EvalItem>();
if (includeOverall)
{
var finalResponse = events.OfType<AgentResponseEvent>().LastOrDefault();
if (finalResponse is not null)
var overallItem = BuildOverallItem(events, splitter, expectedOutput);
if (overallItem is not null)
{
var firstInvoked = events.OfType<ExecutorInvokedEvent>().FirstOrDefault();
var query = firstInvoked?.Data switch
{
ChatMessage cm => cm.Text ?? string.Empty,
IReadOnlyList<ChatMessage> msgs => msgs.LastOrDefault(m => m.Role == ChatRole.User)?.Text ?? string.Empty,
string s => s,
_ => firstInvoked?.Data?.ToString() ?? string.Empty,
};
var conversation = new List<ChatMessage>
{
new(ChatRole.User, query),
};
conversation.AddRange(finalResponse.Response.Messages);
overallItems.Add(new EvalItem(query, finalResponse.Response.Text, conversation)
{
Splitter = splitter,
});
overallItems.Add(overallItem);
}
else
{
// The caller asked for an overall evaluation but we couldn't find a final
// response to score — almost always because the workflow's agents weren't
// built with EmitAgentResponseEvents enabled (so no AgentResponseEvent was
// emitted) and no terminal ExecutorCompletedEvent carried an AgentResponse
// / ChatMessage / string payload. Fail loudly instead of silently returning
// 0/0 (or skipping evaluation against a supplied expectedOutput).
throw new InvalidOperationException(
"Cannot evaluate the overall workflow output: no AgentResponseEvent or " +
"ExecutorCompletedEvent with an AgentResponse/ChatMessage/string payload " +
"was found in the run. Bind agents with " +
"AIAgentHostOptions { EmitAgentResponseEvents = true } " +
"(for example via agent.BindAsExecutor(new AIAgentHostOptions { EmitAgentResponseEvents = true })) " +
"so the workflow surfaces the final agent response, or set 'includeOverall: false'.");
}
}
@@ -97,6 +107,86 @@ public static class WorkflowEvaluationExtensions
return overallResult;
}
internal static EvalItem? BuildOverallItem(
IReadOnlyList<WorkflowEvent> events,
IConversationSplitter? splitter,
string? expectedOutput)
{
var firstInvoked = events.OfType<ExecutorInvokedEvent>().FirstOrDefault();
var query = firstInvoked?.Data switch
{
ChatMessage cm => cm.Text ?? string.Empty,
IReadOnlyList<ChatMessage> msgs => msgs.LastOrDefault(m => m.Role == ChatRole.User)?.Text ?? string.Empty,
string s => s,
_ => firstInvoked?.Data?.ToString() ?? string.Empty,
};
var conversation = new List<ChatMessage>
{
new(ChatRole.User, query),
};
// Prefer AgentResponseEvent (only emitted when AIAgentHostOptions.EmitAgentResponseEvents
// is enabled). Otherwise fall back to the last ExecutorCompletedEvent that carries an
// AgentResponse / ChatMessage / string payload — these are always emitted by the runtime.
var finalResponse = events.OfType<AgentResponseEvent>().LastOrDefault();
string responseText;
if (finalResponse is not null)
{
responseText = finalResponse.Response.Text;
conversation.AddRange(finalResponse.Response.Messages);
}
else
{
ExecutorCompletedEvent? finalCompleted = null;
for (int i = events.Count - 1; i >= 0; i--)
{
if (events[i] is ExecutorCompletedEvent completed
&& !IsInternalExecutor(completed.ExecutorId)
&& completed.Data is AgentResponse or ChatMessage or string)
{
finalCompleted = completed;
break;
}
}
if (finalCompleted is null)
{
return null;
}
switch (finalCompleted.Data)
{
case AgentResponse ar:
responseText = ar.Text;
conversation.AddRange(ar.Messages);
break;
case ChatMessage cm:
responseText = cm.Text ?? string.Empty;
conversation.Add(cm);
break;
case string s:
responseText = s;
conversation.Add(new ChatMessage(ChatRole.Assistant, s));
break;
default:
// Unreachable — the for-loop above already constrains Data to one of the
// three handled types. Throw if the contract drifts so the bug is visible
// instead of silently dropping the overall item.
throw new InvalidOperationException(
"BuildOverallItem: unexpected ExecutorCompletedEvent.Data type " +
$"'{finalCompleted.Data?.GetType().FullName ?? "null"}'. Expected " +
$"{nameof(AgentResponse)}, {nameof(ChatMessage)}, or string.");
}
}
return new EvalItem(query, responseText, conversation)
{
Splitter = splitter,
ExpectedOutput = expectedOutput,
};
}
internal static Dictionary<string, List<EvalItem>> ExtractAgentData(
List<WorkflowEvent> events,
IConversationSplitter? splitter)
@@ -235,11 +235,10 @@ internal sealed class HandoffAgentExecutor :
// This will not filter out tool responses and approval responses that are part of this agent's turn, which is
// the expected behavior since those are part of the agent's reasoning process.
HandoffMessagesFilter handoffMessagesFilter = new(this._options.ToolCallFilteringBehavior);
IEnumerable<ChatMessage> messagesForAgent = state.IncomingState.RequestedHandoffTargetAgentId is not null
List<ChatMessage> messagesForAgent = (state.IncomingState.RequestedHandoffTargetAgentId is not null
? handoffMessagesFilter.FilterMessages(incomingMessages)
: incomingMessages;
List<ChatMessage>? roleChanges = messagesForAgent.ChangeAssistantToUserForOtherParticipants(this._agent.Name ?? this._agent.Id);
: incomingMessages)
.CopyWithAssistantToUserForOtherParticipants(this._agent.Name ?? this._agent.Id);
bool emitUpdateEvents = state.IncomingState!.ShouldEmitStreamingEvents(this._options.EmitAgentResponseUpdateEvents);
AgentInvocationResult result = await this.InvokeAgentAsync(messagesForAgent, context, emitUpdateEvents, cancellationToken)
@@ -250,8 +249,6 @@ internal sealed class HandoffAgentExecutor :
throw new InvalidOperationException("Cannot request a handoff while holding pending requests.");
}
roleChanges.ResetUserToAssistantForChangedRoles();
int newConversationBookmark = state.ConversationBookmark;
await this._sharedStateRef.InvokeWithStateAsync(
(sharedState, ctx, ct) =>
@@ -3,10 +3,12 @@
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Diagnostics.CodeAnalysis;
using System.Runtime.CompilerServices;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.AI;
using Microsoft.Shared.DiagnosticIds;
namespace Microsoft.Agents.AI;
@@ -32,6 +34,13 @@ public sealed class OpenTelemetryAgent : DelegatingAIAgent, IDisposable
private readonly OpenTelemetryChatClient _otelClient;
/// <summary>The provider name extracted from <see cref="AIAgentMetadata"/>.</summary>
private readonly string? _providerName;
/// <summary>The resolved source name for telemetry. Always non-empty; defaults to <see cref="OpenTelemetryConsts.DefaultSourceName"/>.</summary>
private readonly string _sourceName;
/// <summary>
/// Indicates whether the underlying <see cref="IChatClient"/> of a <see cref="ChatClientAgent"/> inner agent
/// should be automatically wrapped with <see cref="OpenTelemetryChatClient"/> on each invocation.
/// </summary>
private readonly bool _autoWireChatClient;
/// <summary>Initializes a new instance of the <see cref="OpenTelemetryAgent"/> class.</summary>
/// <param name="innerAgent">The underlying <see cref="AIAgent"/> to be augmented with telemetry capabilities.</param>
@@ -44,13 +53,44 @@ public sealed class OpenTelemetryAgent : DelegatingAIAgent, IDisposable
/// The constructor automatically extracts provider metadata from the inner agent and configures
/// telemetry collection according to OpenTelemetry semantic conventions for AI systems.
/// </remarks>
public OpenTelemetryAgent(AIAgent innerAgent, string? sourceName = null) : base(innerAgent)
public OpenTelemetryAgent(AIAgent innerAgent, string? sourceName = null)
#pragma warning disable MAAI001 // Auto-wiring is the new default; the experimental opt-out lives on the 3-arg overload.
: this(innerAgent, sourceName, autoWireChatClient: true)
#pragma warning restore MAAI001
{
}
/// <summary>Initializes a new instance of the <see cref="OpenTelemetryAgent"/> class.</summary>
/// <param name="innerAgent">The underlying <see cref="AIAgent"/> to be augmented with telemetry capabilities.</param>
/// <param name="sourceName">
/// An optional source name that will be used to identify telemetry data from this agent.
/// If not provided, a default source name will be used for telemetry identification.
/// </param>
/// <param name="autoWireChatClient">
/// When <see langword="true"/> and the inner agent is a <see cref="ChatClientAgent"/>, the underlying
/// <see cref="IChatClient"/> is automatically wrapped with <see cref="OpenTelemetryChatClient"/> for each invocation
/// so that chat-level telemetry flows alongside agent-level telemetry. If the underlying chat client is already
/// instrumented, no additional wrapping is applied. Set to <see langword="false"/> to opt-out of this behavior.
/// </param>
/// <exception cref="ArgumentNullException"><paramref name="innerAgent"/> is <see langword="null"/>.</exception>
/// <remarks>
/// The constructor automatically extracts provider metadata from the inner agent and configures
/// telemetry collection according to OpenTelemetry semantic conventions for AI systems.
/// </remarks>
[Experimental(DiagnosticIds.Experiments.AgentsAIExperiments)]
public OpenTelemetryAgent(AIAgent innerAgent, string? sourceName, bool autoWireChatClient) : base(innerAgent)
{
this._providerName = innerAgent.GetService<AIAgentMetadata>()?.ProviderName;
// Resolve once so the outer OpenTelemetryChatClient and the auto-wired inner
// OpenTelemetryChatClient always emit spans under the same ActivitySource, even when
// the caller passes "" or whitespace (which neither client should treat as a real source).
this._sourceName = string.IsNullOrWhiteSpace(sourceName) ? OpenTelemetryConsts.DefaultSourceName : sourceName!;
this._autoWireChatClient = autoWireChatClient;
this._otelClient = new OpenTelemetryChatClient(
new ForwardingChatClient(this),
sourceName: string.IsNullOrEmpty(sourceName) ? OpenTelemetryConsts.DefaultSourceName : sourceName!);
sourceName: this._sourceName);
}
/// <inheritdoc/>
@@ -163,6 +203,85 @@ public sealed class OpenTelemetryAgent : DelegatingAIAgent, IDisposable
public Activity? CurrentActivity { get; }
}
/// <summary>
/// If auto-wiring is enabled and the inner agent is a <see cref="ChatClientAgent"/> whose underlying
/// <see cref="IChatClient"/> is not already instrumented with <see cref="OpenTelemetryChatClient"/>, returns a
/// new <see cref="ChatClientAgentRunOptions"/> with a <see cref="ChatClientAgentRunOptions.ChatClientFactory"/>
/// that wraps the chat client with <see cref="OpenTelemetryChatClient"/>. When <paramref name="options"/> is a
/// plain <see cref="AgentRunOptions"/> (the base type, not <see cref="ChatClientAgentRunOptions"/>), the base
/// properties are copied onto the new <see cref="ChatClientAgentRunOptions"/> so high-level callers that pass
/// the abstract <see cref="AgentRunOptions"/> still benefit from auto-wiring and propagate their settings to
/// the inner agent. Otherwise, returns <paramref name="options"/> unchanged.
/// </summary>
private AgentRunOptions? GetRunOptionsWithChatClientWiring(AgentRunOptions? options)
{
if (!this._autoWireChatClient)
{
return options;
}
// The auto-wiring only applies when a ChatClientAgent is reachable from the inner agent. Otherwise, no-op.
// Use GetService rather than a type check so wrapping agents that expose a nested ChatClientAgent are supported.
var chatClientAgent = this.InnerAgent.GetService<ChatClientAgent>();
if (chatClientAgent is null)
{
return options;
}
// Respect ChatClientAgentOptions.UseProvidedChatClientAsIs: don't decorate the chat client when the user opted out.
if (chatClientAgent.GetService<ChatClientAgentOptions>()?.UseProvidedChatClientAsIs is true)
{
return options;
}
// Capture the underlying IChatClient and check whether it is already instrumented.
var chatClient = chatClientAgent.GetService<IChatClient>();
if (chatClient is null || chatClient.GetService(typeof(OpenTelemetryChatClient)) is not null)
{
return options;
}
string sourceName = this._sourceName;
static IChatClient WrapIfNeeded(IChatClient cc, string sourceName) =>
cc.GetService(typeof(OpenTelemetryChatClient)) is not null
? cc
: cc.AsBuilder().UseOpenTelemetry(sourceName: sourceName).Build();
if (options is ChatClientAgentRunOptions ccOptions)
{
// Don't mutate the caller's options; clone and chain any caller-provided factory.
// If the user factory already returns an OpenTelemetry-instrumented client, don't double-wrap.
var clone = (ChatClientAgentRunOptions)ccOptions.Clone();
var userFactory = clone.ChatClientFactory;
clone.ChatClientFactory = cc => WrapIfNeeded(userFactory is null ? cc : userFactory(cc), sourceName);
return clone;
}
// For a plain AgentRunOptions (or null), create a ChatClientAgentRunOptions and preserve
// any base AgentRunOptions properties from the caller so they reach the inner agent.
var newOptions = new ChatClientAgentRunOptions
{
ChatClientFactory = cc => WrapIfNeeded(cc, sourceName),
};
if (options is not null)
{
CopyBaseAgentRunOptions(options, newOptions);
}
return newOptions;
}
#pragma warning disable MEAI001 // ContinuationToken is experimental; copy it through to preserve caller-provided value.
private static void CopyBaseAgentRunOptions(AgentRunOptions source, AgentRunOptions target)
{
target.ContinuationToken = source.ContinuationToken;
target.AllowBackgroundResponses = source.AllowBackgroundResponses;
target.AdditionalProperties = source.AdditionalProperties?.Clone();
target.ResponseFormat = source.ResponseFormat;
}
#pragma warning restore MEAI001
/// <summary>The stub <see cref="IChatClient"/> used to delegate from the <see cref="OpenTelemetryChatClient"/> into the inner <see cref="AIAgent"/>.</summary>
/// <param name="parentAgent"></param>
private sealed class ForwardingChatClient(OpenTelemetryAgent parentAgent) : IChatClient
@@ -175,8 +294,11 @@ public sealed class OpenTelemetryAgent : DelegatingAIAgent, IDisposable
// Update the current activity to reflect the agent invocation.
parentAgent.UpdateCurrentActivity(fo?.CurrentActivity);
// If enabled, wire the underlying chat client with OpenTelemetryChatClient via ChatClientFactory.
var runOptions = parentAgent.GetRunOptionsWithChatClientWiring(fo?.Options);
// Invoke the inner agent.
var response = await parentAgent.InnerAgent.RunAsync(messages, fo?.Session, fo?.Options, cancellationToken).ConfigureAwait(false);
var response = await parentAgent.InnerAgent.RunAsync(messages, fo?.Session, runOptions, cancellationToken).ConfigureAwait(false);
// Wrap the response in a ChatResponse so we can pass it back through OpenTelemetryChatClient.
return response.AsChatResponse();
@@ -190,8 +312,11 @@ public sealed class OpenTelemetryAgent : DelegatingAIAgent, IDisposable
// Update the current activity to reflect the agent invocation.
parentAgent.UpdateCurrentActivity(fo?.CurrentActivity);
// If enabled, wire the underlying chat client with OpenTelemetryChatClient via ChatClientFactory.
var runOptions = parentAgent.GetRunOptionsWithChatClientWiring(fo?.Options);
// Invoke the inner agent.
await foreach (var update in parentAgent.InnerAgent.RunStreamingAsync(messages, fo?.Session, fo?.Options, cancellationToken).ConfigureAwait(false))
await foreach (var update in parentAgent.InnerAgent.RunStreamingAsync(messages, fo?.Session, runOptions, cancellationToken).ConfigureAwait(false))
{
// Wrap the response updates in ChatResponseUpdates so we can pass them back through OpenTelemetryChatClient.
yield return update.AsChatResponseUpdate();
@@ -179,6 +179,35 @@ public sealed class FoundryEvalConverterTests
Assert.Null(payload.Context);
}
[Fact]
public void ConvertEvalItem_WithExpectedOutput_PopulatesGroundTruth()
{
// Arrange
var item = new EvalItem(query: "q", response: "r")
{
ExpectedOutput = "the golden answer",
};
// Act
var payload = FoundryEvalConverter.ConvertEvalItem(item);
// Assert
Assert.Equal("the golden answer", payload.GroundTruth);
}
[Fact]
public void ConvertEvalItem_WithoutExpectedOutput_OmitsGroundTruth()
{
// Arrange
var item = new EvalItem(query: "q", response: "r");
// Act
var payload = FoundryEvalConverter.ConvertEvalItem(item);
// Assert
Assert.Null(payload.GroundTruth);
}
// ---------------------------------------------------------------
// FoundryEvalConverter.BuildTestingCriteria tests
// ---------------------------------------------------------------
@@ -239,6 +268,33 @@ public sealed class FoundryEvalConverterTests
Assert.Equal("{{item.context}}", mapping["context"]);
}
[Fact]
public void BuildTestingCriteria_SimilarityEvaluator_IncludesGroundTruth()
{
// Act
var criteria = FoundryEvalConverter.BuildTestingCriteria(
["similarity"], "gpt-4o-mini", includeDataMapping: true);
// Assert
Assert.Single(criteria);
Assert.Equal("builtin.similarity", criteria[0].EvaluatorName);
var mapping = criteria[0].DataMapping;
Assert.NotNull(mapping);
Assert.True(mapping.ContainsKey("ground_truth"));
Assert.Equal("{{item.ground_truth}}", mapping["ground_truth"]);
}
[Fact]
public void BuildTestingCriteria_NonGroundTruthEvaluator_OmitsGroundTruth()
{
var criteria = FoundryEvalConverter.BuildTestingCriteria(
["relevance"], "gpt-4o-mini", includeDataMapping: true);
var mapping = criteria[0].DataMapping;
Assert.NotNull(mapping);
Assert.False(mapping.ContainsKey("ground_truth"));
}
[Fact]
public void BuildTestingCriteria_WithoutDataMapping_OmitsMappingField()
{
@@ -282,6 +338,59 @@ public sealed class FoundryEvalConverterTests
Assert.True(schema.Properties.ContainsKey("tool_definitions"));
}
[Fact]
public void BuildItemSchema_WithGroundTruth_IncludesGroundTruthProperty()
{
// Act
var schema = FoundryEvalConverter.BuildItemSchema(hasGroundTruth: true);
// Assert
Assert.True(schema.Properties.ContainsKey("ground_truth"));
Assert.Equal("string", schema.Properties["ground_truth"].Type);
}
[Fact]
public void BuildItemSchema_WithoutGroundTruth_OmitsGroundTruthProperty()
{
var schema = FoundryEvalConverter.BuildItemSchema();
Assert.False(schema.Properties.ContainsKey("ground_truth"));
}
// ---------------------------------------------------------------
// FoundryEvalConverter.FindMissingGroundTruthEvaluators tests
// ---------------------------------------------------------------
[Fact]
public void FindMissingGroundTruthEvaluators_NoGroundTruth_ReturnsSimilarity()
{
// Act
var missing = FoundryEvalConverter.FindMissingGroundTruthEvaluators(
["similarity", "relevance"], hasGroundTruth: false);
// Assert
Assert.Single(missing);
Assert.Equal("similarity", missing[0]);
}
[Fact]
public void FindMissingGroundTruthEvaluators_HasGroundTruth_ReturnsEmpty()
{
var missing = FoundryEvalConverter.FindMissingGroundTruthEvaluators(
["similarity"], hasGroundTruth: true);
Assert.Empty(missing);
}
[Fact]
public void FindMissingGroundTruthEvaluators_NoGroundTruthEvaluators_ReturnsEmpty()
{
var missing = FoundryEvalConverter.FindMissingGroundTruthEvaluators(
["relevance", "coherence"], hasGroundTruth: false);
Assert.Empty(missing);
}
// ---------------------------------------------------------------
// FoundryEvalConverter.ConvertMessage DataContent test
// ---------------------------------------------------------------
@@ -0,0 +1,56 @@
// Copyright (c) Microsoft. All rights reserved.
namespace Microsoft.Agents.AI.UnitTests;
public class HarnessAgentOptionsTests
{
/// <summary>
/// Verify that default property values are as expected.
/// </summary>
[Fact]
public void DefaultPropertyValues()
{
// Arrange & Act
var options = new HarnessAgentOptions();
// Assert
Assert.Null(options.Id);
Assert.Null(options.Name);
Assert.Null(options.Description);
Assert.Null(options.ChatOptions);
Assert.Null(options.ChatHistoryProvider);
Assert.Null(options.AIContextProviders);
}
/// <summary>
/// Verify that all properties can be set and retrieved.
/// </summary>
[Fact]
public void PropertiesCanBeSetAndRetrieved()
{
// Arrange
var chatHistoryProvider = new InMemoryChatHistoryProvider();
var contextProviders = new AIContextProvider[] { new TodoProvider() };
// Act
var options = new HarnessAgentOptions
{
Id = "test-id",
Name = "test-name",
Description = "test-description",
ChatOptions = new() { Temperature = 0.5f, Instructions = "custom instructions" },
ChatHistoryProvider = chatHistoryProvider,
AIContextProviders = contextProviders,
};
// Assert
Assert.Equal("test-id", options.Id);
Assert.Equal("test-name", options.Name);
Assert.Equal("test-description", options.Description);
Assert.NotNull(options.ChatOptions);
Assert.Equal(0.5f, options.ChatOptions!.Temperature);
Assert.Equal("custom instructions", options.ChatOptions.Instructions);
Assert.Same(chatHistoryProvider, options.ChatHistoryProvider);
Assert.Same(contextProviders, options.AIContextProviders);
}
}
@@ -0,0 +1,516 @@
// Copyright (c) Microsoft. All rights reserved.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.AI;
using Moq;
namespace Microsoft.Agents.AI.UnitTests;
public class HarnessAgentTests
{
private const int TestMaxContextWindowTokens = 100_000;
private const int TestMaxOutputTokens = 10_000;
#region Constructor Validation
/// <summary>
/// Verify that the constructor throws when chatClient is null.
/// </summary>
[Fact]
public void Constructor_ThrowsWhenChatClientIsNull()
{
// Act & Assert
Assert.Throws<ArgumentNullException>(() => new HarnessAgent(null!, TestMaxContextWindowTokens, TestMaxOutputTokens));
}
/// <summary>
/// Verify that the constructor throws when MaxContextWindowTokens is invalid (zero).
/// </summary>
[Fact]
public void Constructor_ThrowsWhenMaxContextWindowTokensIsZero()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
// Act & Assert
Assert.Throws<ArgumentOutOfRangeException>(() => new HarnessAgent(chatClient, 0, TestMaxOutputTokens));
}
/// <summary>
/// Verify that the constructor throws when MaxOutputTokens equals MaxContextWindowTokens.
/// </summary>
[Fact]
public void Constructor_ThrowsWhenMaxOutputTokensEqualsContextWindow()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
// Act & Assert
Assert.Throws<ArgumentOutOfRangeException>(() => new HarnessAgent(chatClient, 100_000, 100_000));
}
/// <summary>
/// Verify that the constructor succeeds when options is null.
/// </summary>
[Fact]
public void Constructor_SucceedsWhenOptionsIsNull()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
// Act
var agent = new HarnessAgent(chatClient, TestMaxContextWindowTokens, TestMaxOutputTokens);
// Assert
Assert.NotNull(agent);
}
#endregion
#region Agent Identity
/// <summary>
/// Verify that Name and Description are passed through to the inner agent.
/// </summary>
[Fact]
public void NameAndDescription_ArePassedThrough()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
// Act
var agent = new HarnessAgent(chatClient, TestMaxContextWindowTokens, TestMaxOutputTokens, new HarnessAgentOptions
{
Name = "TestAgent",
Description = "A test agent",
});
// Assert
Assert.Equal("TestAgent", agent.Name);
Assert.Equal("A test agent", agent.Description);
}
/// <summary>
/// Verify that Id is passed through to the inner agent.
/// </summary>
[Fact]
public void Id_IsPassedThrough()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
// Act
var agent = new HarnessAgent(chatClient, TestMaxContextWindowTokens, TestMaxOutputTokens, new HarnessAgentOptions
{
Id = "my-agent-id",
});
// Assert
Assert.Equal("my-agent-id", agent.Id);
}
#endregion
#region Instructions
/// <summary>
/// Verify that default instructions are used when none are provided.
/// </summary>
[Fact]
public void Instructions_DefaultsToBuiltInInstructions()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
// Act
var agent = new HarnessAgent(chatClient, TestMaxContextWindowTokens, TestMaxOutputTokens);
var innerAgent = agent.GetService<ChatClientAgent>();
// Assert
Assert.NotNull(innerAgent);
Assert.Equal(HarnessAgent.DefaultInstructions, innerAgent!.Instructions);
}
/// <summary>
/// Verify that default instructions are used when options is provided but ChatOptions.Instructions is null.
/// </summary>
[Fact]
public void Instructions_DefaultsWhenChatOptionsInstructionsIsNull()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
// Act
var agent = new HarnessAgent(chatClient, TestMaxContextWindowTokens, TestMaxOutputTokens, new HarnessAgentOptions
{
ChatOptions = new ChatOptions { Temperature = 0.5f },
});
var innerAgent = agent.GetService<ChatClientAgent>();
// Assert
Assert.NotNull(innerAgent);
Assert.Equal(HarnessAgent.DefaultInstructions, innerAgent!.Instructions);
}
/// <summary>
/// Verify that ChatOptions.Instructions overrides the defaults.
/// </summary>
[Fact]
public void Instructions_CanBeOverriddenViaChatOptions()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
// Act
var agent = new HarnessAgent(chatClient, TestMaxContextWindowTokens, TestMaxOutputTokens, new HarnessAgentOptions
{
ChatOptions = new ChatOptions { Instructions = "You are a custom assistant." },
});
var innerAgent = agent.GetService<ChatClientAgent>();
// Assert
Assert.NotNull(innerAgent);
Assert.Equal("You are a custom assistant.", innerAgent!.Instructions);
}
#endregion
#region ChatHistoryProvider
/// <summary>
/// Verify that the default ChatHistoryProvider is InMemoryChatHistoryProvider when none is specified.
/// </summary>
[Fact]
public void ChatHistoryProvider_DefaultsToInMemory()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
// Act
var agent = new HarnessAgent(chatClient, TestMaxContextWindowTokens, TestMaxOutputTokens);
var innerAgent = agent.GetService<ChatClientAgent>();
// Assert
Assert.NotNull(innerAgent);
Assert.IsType<InMemoryChatHistoryProvider>(innerAgent!.ChatHistoryProvider);
}
/// <summary>
/// Verify that a custom ChatHistoryProvider is used when provided.
/// </summary>
[Fact]
public void ChatHistoryProvider_UsesCustomProviderWhenSpecified()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
var customProvider = new InMemoryChatHistoryProvider();
// Act
var agent = new HarnessAgent(chatClient, TestMaxContextWindowTokens, TestMaxOutputTokens, new HarnessAgentOptions
{
ChatHistoryProvider = customProvider,
});
var innerAgent = agent.GetService<ChatClientAgent>();
// Assert
Assert.NotNull(innerAgent);
Assert.Same(customProvider, innerAgent!.ChatHistoryProvider);
}
#endregion
#region ChatClient Pipeline
/// <summary>
/// Verify that the inner agent's ChatClient includes FunctionInvokingChatClient in the pipeline.
/// </summary>
[Fact]
public void Pipeline_IncludesFunctionInvokingChatClient()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
// Act
var agent = new HarnessAgent(chatClient, TestMaxContextWindowTokens, TestMaxOutputTokens);
var innerAgent = agent.GetService<ChatClientAgent>();
// Assert
Assert.NotNull(innerAgent);
var ficc = innerAgent!.ChatClient.GetService<FunctionInvokingChatClient>();
Assert.NotNull(ficc);
}
/// <summary>
/// Verify that the inner agent's ChatClient pipeline includes more than just the raw chat client,
/// confirming that per-service-call persistence and other decorators have been applied.
/// </summary>
[Fact]
public void Pipeline_HasDecoratedChatClient()
{
// Arrange
var mockClient = new Mock<IChatClient>();
var rawClient = mockClient.Object;
// Act
var agent = new HarnessAgent(rawClient, TestMaxContextWindowTokens, TestMaxOutputTokens);
var innerAgent = agent.GetService<ChatClientAgent>();
// Assert — the pipeline wraps the raw client, so the outer client is not the same object.
Assert.NotNull(innerAgent);
Assert.NotSame(rawClient, innerAgent!.ChatClient);
}
#endregion
#region AIContextProviders
/// <summary>
/// Verify that additional AIContextProviders from options are passed to the inner ChatClientAgent,
/// not merged into the chat client builder pipeline.
/// </summary>
[Fact]
public void AIContextProviders_ArePassedToInnerAgent()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
var todoProvider = new TodoProvider();
// Act
var agent = new HarnessAgent(chatClient, TestMaxContextWindowTokens, TestMaxOutputTokens, new HarnessAgentOptions
{
AIContextProviders = [todoProvider],
});
var innerAgent = agent.GetService<ChatClientAgent>();
// Assert — the TodoProvider should appear in the inner agent's AIContextProviders.
Assert.NotNull(innerAgent);
Assert.NotNull(innerAgent!.AIContextProviders);
Assert.Contains(todoProvider, innerAgent.AIContextProviders!);
}
/// <summary>
/// Verify that when no AIContextProviders are specified, the inner agent has no additional providers.
/// </summary>
[Fact]
public void AIContextProviders_IsNullWhenNoneSpecified()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
// Act
var agent = new HarnessAgent(chatClient, TestMaxContextWindowTokens, TestMaxOutputTokens);
var innerAgent = agent.GetService<ChatClientAgent>();
// Assert
Assert.NotNull(innerAgent);
Assert.Null(innerAgent!.AIContextProviders);
}
#endregion
#region ChatOptions and Tools
/// <summary>
/// Verify that tools from ChatOptions are passed to the model during invocation.
/// </summary>
[Fact]
public async Task ChatOptions_ToolsArePreservedAsync()
{
// Arrange
var tool = AIFunctionFactory.Create(() => "test", "TestTool");
var mockClient = new Mock<IChatClient>();
ChatOptions? capturedOptions = null;
mockClient
.Setup(c => c.GetResponseAsync(
It.IsAny<IEnumerable<ChatMessage>>(),
It.IsAny<ChatOptions>(),
It.IsAny<CancellationToken>()))
.Callback<IEnumerable<ChatMessage>, ChatOptions?, CancellationToken>((_, opts, _) => capturedOptions = opts)
.ReturnsAsync(new ChatResponse(new ChatMessage(ChatRole.Assistant, "Done")));
var agent = new HarnessAgent(mockClient.Object, TestMaxContextWindowTokens, TestMaxOutputTokens, new HarnessAgentOptions
{
ChatOptions = new ChatOptions
{
Tools = [tool],
},
});
var session = await agent.CreateSessionAsync();
// Act
await agent.RunAsync([new ChatMessage(ChatRole.User, "Hi")], session);
// Assert — verify the tool was included in the ChatOptions passed to the model.
Assert.NotNull(capturedOptions);
Assert.NotNull(capturedOptions!.Tools);
Assert.Contains(capturedOptions.Tools, t => t == tool);
}
/// <summary>
/// Verify that the source ChatOptions are cloned and not modified.
/// </summary>
[Fact]
public void ChatOptions_SourceIsNotModified()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
var sourceChatOptions = new ChatOptions
{
Instructions = "original instructions",
Temperature = 0.7f,
};
// Act
_ = new HarnessAgent(chatClient, TestMaxContextWindowTokens, TestMaxOutputTokens, new HarnessAgentOptions
{
ChatOptions = sourceChatOptions,
});
// Assert — source ChatOptions should not be mutated.
Assert.Equal("original instructions", sourceChatOptions.Instructions);
Assert.Equal(0.7f, sourceChatOptions.Temperature);
}
#endregion
#region GetService
/// <summary>
/// Verify that GetService returns the HarnessAgent for its own type.
/// </summary>
[Fact]
public void GetService_ReturnsSelfForHarnessAgentType()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
// Act
var agent = new HarnessAgent(chatClient, TestMaxContextWindowTokens, TestMaxOutputTokens);
// Assert
Assert.Same(agent, agent.GetService<HarnessAgent>());
}
/// <summary>
/// Verify that GetService returns the inner ChatClientAgent.
/// </summary>
[Fact]
public void GetService_ReturnsInnerChatClientAgent()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
// Act
var agent = new HarnessAgent(chatClient, TestMaxContextWindowTokens, TestMaxOutputTokens);
// Assert
Assert.NotNull(agent.GetService<ChatClientAgent>());
}
#endregion
#region RunAsync Delegation
/// <summary>
/// Verify that RunAsync delegates to the inner ChatClientAgent.
/// </summary>
[Fact]
public async Task RunAsync_DelegatesToInnerAgentAsync()
{
// Arrange
var mockClient = new Mock<IChatClient>();
mockClient
.Setup(c => c.GetResponseAsync(
It.IsAny<IEnumerable<ChatMessage>>(),
It.IsAny<ChatOptions>(),
It.IsAny<CancellationToken>()))
.ReturnsAsync(new ChatResponse(new ChatMessage(ChatRole.Assistant, "Hello!")));
var agent = new HarnessAgent(mockClient.Object, TestMaxContextWindowTokens, TestMaxOutputTokens);
var session = await agent.CreateSessionAsync();
// Act
var response = await agent.RunAsync(
[new ChatMessage(ChatRole.User, "Hi")],
session);
// Assert
Assert.NotNull(response);
Assert.True(response.Messages.Any());
}
#endregion
#region DefaultInstructions
/// <summary>
/// Verify that DefaultInstructions is a non-empty public constant.
/// </summary>
[Fact]
public void DefaultInstructions_IsNonEmpty()
{
// Assert
Assert.False(string.IsNullOrWhiteSpace(HarnessAgent.DefaultInstructions));
}
#endregion
#region AsHarnessAgent Extension Method
/// <summary>
/// Verify that AsHarnessAgent creates a HarnessAgent with default options.
/// </summary>
[Fact]
public void AsHarnessAgent_CreatesAgentWithDefaults()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
// Act
var agent = chatClient.AsHarnessAgent(TestMaxContextWindowTokens, TestMaxOutputTokens);
// Assert
Assert.NotNull(agent);
Assert.IsType<HarnessAgent>(agent);
Assert.Equal(HarnessAgent.DefaultInstructions, agent.GetService<ChatClientAgent>()!.Instructions);
}
/// <summary>
/// Verify that AsHarnessAgent passes options through to the HarnessAgent.
/// </summary>
[Fact]
public void AsHarnessAgent_PassesOptionsThrough()
{
// Arrange
var chatClient = new Mock<IChatClient>().Object;
// Act
var agent = chatClient.AsHarnessAgent(TestMaxContextWindowTokens, TestMaxOutputTokens, new HarnessAgentOptions
{
Name = "ExtensionAgent",
ChatOptions = new ChatOptions { Instructions = "Custom instructions" },
});
var innerAgent = agent.GetService<ChatClientAgent>();
// Assert
Assert.Equal("ExtensionAgent", agent.Name);
Assert.NotNull(innerAgent);
Assert.Equal("Custom instructions", innerAgent!.Instructions);
}
/// <summary>
/// Verify that AsHarnessAgent throws when chatClient is null.
/// </summary>
[Fact]
public void AsHarnessAgent_ThrowsWhenChatClientIsNull()
{
// Act & Assert
Assert.Throws<ArgumentNullException>(() => ((IChatClient)null!).AsHarnessAgent(TestMaxContextWindowTokens, TestMaxOutputTokens));
}
#endregion
}
@@ -0,0 +1,11 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<NoWarn>$(NoWarn);MAAI001</NoWarn>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\..\src\Microsoft.Agents.AI.Harness\Microsoft.Agents.AI.Harness.csproj" />
</ItemGroup>
</Project>
@@ -267,7 +267,43 @@ public sealed class OpenAIResponsesAgentResolutionIntegrationTests : IAsyncDispo
Assert.Equal(System.Net.HttpStatusCode.BadRequest, httpResponse.StatusCode);
string responseJson = await httpResponse.Content.ReadAsStringAsync();
Assert.Contains("agent.name", responseJson, StringComparison.OrdinalIgnoreCase);
using JsonDocument errorDoc1 = JsonDocument.Parse(responseJson);
string? errorCode = errorDoc1.RootElement.GetProperty("error").GetProperty("code").GetString();
Assert.Equal("missing_required_parameter", errorCode);
}
/// <summary>
/// Verifies that the model field alone is not used for agent resolution.
/// The multi-agent endpoint requires agent.name or metadata.entity_id; setting only model returns 400.
/// </summary>
[Fact]
public async Task CreateResponse_WithModelOnly_ReturnsBadRequestAsync()
{
// Arrange
const string AgentName = "test-agent";
this._httpClient = await this.CreateTestServerWithAgentResolutionAsync(
(AgentName, "Instructions", "Response"));
// Act - Send request with model=agentName but no agent.name or metadata.entity_id
using StringContent requestContent = new(JsonSerializer.Serialize(new
{
model = AgentName,
input = new[]
{
new { type = "message", role = "user", content = "Test message" }
}
}), Encoding.UTF8, "application/json");
using HttpResponseMessage httpResponse = await this._httpClient!.PostAsync(new Uri("/v1/responses", UriKind.Relative), requestContent);
// Assert - model is not used for agent resolution
Assert.Equal(System.Net.HttpStatusCode.BadRequest, httpResponse.StatusCode);
string responseJson = await httpResponse.Content.ReadAsStringAsync();
using JsonDocument errorDoc2 = JsonDocument.Parse(responseJson);
string? errorCode = errorDoc2.RootElement.GetProperty("error").GetProperty("code").GetString();
Assert.Equal("missing_required_parameter", errorCode);
}
/// <summary>
@@ -627,4 +627,455 @@ public class OpenTelemetryAgentTests
}
private static string ReplaceWhitespace(string? input) => Regex.Replace(input ?? "", @"\s+", "").Trim();
#region AutoWireChatClient
[Fact]
public async Task AutoWireChatClient_DefaultsToEnabled_EmitsChatSpan_Async()
{
var sourceName = Guid.NewGuid().ToString();
var activities = new List<Activity>();
using var tracerProvider = OpenTelemetry.Sdk.CreateTracerProviderBuilder()
.AddSource(sourceName)
.AddInMemoryExporter(activities)
.Build();
var fakeChatClient = new AutoWireTestChatClient();
var inner = new ChatClientAgent(fakeChatClient);
using var agent = new OpenTelemetryAgent(inner, sourceName);
_ = await agent.RunAsync("hi");
// Expect 2 activities: the inner chat span (from auto-wired OpenTelemetryChatClient) and the invoke_agent span.
Assert.Equal(2, activities.Count);
Assert.Contains(activities, a => a.DisplayName.StartsWith("invoke_agent", StringComparison.Ordinal));
Assert.Contains(activities, a => string.Equals(a.GetTagItem("gen_ai.operation.name") as string, "chat", StringComparison.Ordinal));
}
[Fact]
public async Task AutoWireChatClient_Streaming_EmitsChatSpan_Async()
{
var sourceName = Guid.NewGuid().ToString();
var activities = new List<Activity>();
using var tracerProvider = OpenTelemetry.Sdk.CreateTracerProviderBuilder()
.AddSource(sourceName)
.AddInMemoryExporter(activities)
.Build();
var fakeChatClient = new AutoWireTestChatClient();
var inner = new ChatClientAgent(fakeChatClient);
using var agent = new OpenTelemetryAgent(inner, sourceName);
await foreach (var _ in agent.RunStreamingAsync("hi"))
{
}
Assert.Equal(2, activities.Count);
Assert.Contains(activities, a => a.DisplayName.StartsWith("invoke_agent", StringComparison.Ordinal));
Assert.Contains(activities, a => string.Equals(a.GetTagItem("gen_ai.operation.name") as string, "chat", StringComparison.Ordinal));
}
[Fact]
public async Task AutoWireChatClient_Disabled_DoesNotEmitChatSpan_Async()
{
var sourceName = Guid.NewGuid().ToString();
var activities = new List<Activity>();
using var tracerProvider = OpenTelemetry.Sdk.CreateTracerProviderBuilder()
.AddSource(sourceName)
.AddInMemoryExporter(activities)
.Build();
var fakeChatClient = new AutoWireTestChatClient();
var inner = new ChatClientAgent(fakeChatClient);
using var agent = new OpenTelemetryAgent(inner, sourceName, autoWireChatClient: false);
_ = await agent.RunAsync("hi");
// Only the invoke_agent activity should be emitted; no chat span.
var activity = Assert.Single(activities);
Assert.StartsWith("invoke_agent", activity.DisplayName);
}
[Fact]
public async Task AutoWireChatClient_NonChatClientAgent_NoOp_Async()
{
// Inner is not a ChatClientAgent — auto-wiring must be a no-op and options must remain null.
AgentRunOptions? observedOptions = null;
var inner = new TestAIAgent
{
RunAsyncFunc = (messages, session, options, ct) =>
{
observedOptions = options;
return Task.FromResult(new AgentResponse(new ChatMessage(ChatRole.Assistant, "ok")));
},
};
using var agent = new OpenTelemetryAgent(inner);
_ = await agent.RunAsync("hi");
Assert.Null(observedOptions);
}
[Fact]
public async Task AutoWireChatClient_UseProvidedChatClientAsIs_DoesNotEmitChatSpan_Async()
{
var sourceName = Guid.NewGuid().ToString();
var activities = new List<Activity>();
using var tracerProvider = OpenTelemetry.Sdk.CreateTracerProviderBuilder()
.AddSource(sourceName)
.AddInMemoryExporter(activities)
.Build();
var fakeChatClient = new AutoWireTestChatClient();
var inner = new ChatClientAgent(fakeChatClient, new ChatClientAgentOptions { UseProvidedChatClientAsIs = true });
using var agent = new OpenTelemetryAgent(inner, sourceName);
_ = await agent.RunAsync("hi");
// UseProvidedChatClientAsIs opts out of auto-wiring, so only the invoke_agent span should be emitted.
var activity = Assert.Single(activities);
Assert.StartsWith("invoke_agent", activity.DisplayName);
}
[Fact]
public async Task AutoWireChatClient_AlreadyInstrumented_DoesNotDoubleWrap_Async()
{
var sourceName = Guid.NewGuid().ToString();
var activities = new List<Activity>();
using var tracerProvider = OpenTelemetry.Sdk.CreateTracerProviderBuilder()
.AddSource(sourceName)
.AddInMemoryExporter(activities)
.Build();
var fakeChatClient = new AutoWireTestChatClient();
// Pre-wrap with OpenTelemetryChatClient on the same source so spans flow through the tracer.
IChatClient preWrapped = fakeChatClient.AsBuilder().UseOpenTelemetry(sourceName: sourceName).Build();
var inner = new ChatClientAgent(preWrapped);
using var agent = new OpenTelemetryAgent(inner, sourceName);
_ = await agent.RunAsync("hi");
// Expect exactly 2 activities (one invoke_agent + one chat from the pre-existing wrapper). If we had double-wrapped, we would see 3.
Assert.Equal(2, activities.Count);
}
[Fact]
public async Task AutoWireChatClient_PreservesUserChatClientFactory_Async()
{
var sourceName = Guid.NewGuid().ToString();
var activities = new List<Activity>();
using var tracerProvider = OpenTelemetry.Sdk.CreateTracerProviderBuilder()
.AddSource(sourceName)
.AddInMemoryExporter(activities)
.Build();
bool userFactoryCalled = false;
var fakeChatClient = new AutoWireTestChatClient();
var inner = new ChatClientAgent(fakeChatClient);
using var agent = new OpenTelemetryAgent(inner, sourceName);
var runOptions = new ChatClientAgentRunOptions
{
ChatClientFactory = cc =>
{
userFactoryCalled = true;
return cc;
},
};
_ = await agent.RunAsync("hi", options: runOptions);
Assert.True(userFactoryCalled);
// Auto-wiring should still produce a chat span on top of the user's factory.
Assert.Equal(2, activities.Count);
Assert.Contains(activities, a => string.Equals(a.GetTagItem("gen_ai.operation.name") as string, "chat", StringComparison.Ordinal));
}
[Fact]
public async Task AutoWireChatClient_PlainAgentRunOptions_PreservesBaseProperties_Async()
{
// Auto-wiring converts a plain AgentRunOptions into a ChatClientAgentRunOptions. The base
// properties (ContinuationToken, AllowBackgroundResponses, AdditionalProperties, ResponseFormat)
// must be preserved so they reach the inner agent.
AgentRunOptions? observedOptions = null;
var fakeChatClient = new AutoWireTestChatClient();
var innerChatClientAgent = new ChatClientAgent(fakeChatClient);
// Wrapping agent: surfaces the ChatClientAgent via GetService (so auto-wiring activates),
// but captures the AgentRunOptions passed to RunAsync by the OpenTelemetryAgent.
var wrapper = new TestAIAgent
{
GetServiceFunc = (type, key) =>
type == typeof(ChatClientAgent) ? innerChatClientAgent : null,
RunAsyncFunc = (messages, session, options, ct) =>
{
observedOptions = options;
return Task.FromResult(new AgentResponse(new ChatMessage(ChatRole.Assistant, "ok")));
},
};
using var agent = new OpenTelemetryAgent(wrapper);
var additionalProps = new AdditionalPropertiesDictionary { ["customKey"] = "customValue" };
var inputOptions = new AgentRunOptions
{
AllowBackgroundResponses = true,
AdditionalProperties = additionalProps,
ResponseFormat = ChatResponseFormat.Json,
};
_ = await agent.RunAsync("hi", options: inputOptions);
Assert.NotNull(observedOptions);
Assert.IsType<ChatClientAgentRunOptions>(observedOptions);
Assert.Equal(true, observedOptions!.AllowBackgroundResponses);
Assert.Same(ChatResponseFormat.Json, observedOptions.ResponseFormat);
Assert.NotNull(observedOptions.AdditionalProperties);
Assert.Equal("customValue", observedOptions.AdditionalProperties!["customKey"]);
}
[Fact]
public async Task AutoWireChatClient_UserFactoryReturnsInstrumentedClient_DoesNotDoubleWrap_Async()
{
var sourceName = Guid.NewGuid().ToString();
var activities = new List<Activity>();
using var tracerProvider = OpenTelemetry.Sdk.CreateTracerProviderBuilder()
.AddSource(sourceName)
.AddInMemoryExporter(activities)
.Build();
var fakeChatClient = new AutoWireTestChatClient();
var inner = new ChatClientAgent(fakeChatClient);
using var agent = new OpenTelemetryAgent(inner, sourceName);
// User factory wraps the chat client with OpenTelemetryChatClient itself.
var runOptions = new ChatClientAgentRunOptions
{
ChatClientFactory = cc => cc.AsBuilder().UseOpenTelemetry(sourceName: sourceName).Build(),
};
_ = await agent.RunAsync("hi", options: runOptions);
// Expect 2 activities (invoke_agent + a single chat span). If we double-wrapped, we would see 3.
Assert.Equal(2, activities.Count);
}
[Theory]
[InlineData(null)]
[InlineData("")]
[InlineData(" ")]
[InlineData("\t")]
public async Task Ctor_NullOrWhitespaceSourceName_AutoWiredChatClientUsesDefaultSource_Async(string? sourceName)
{
// Both the agent-level invoke_agent span and the auto-wired chat span must be emitted under
// OpenTelemetryConsts.DefaultSourceName when the caller passes null, "", or whitespace, so they reach
// the same ActivitySource and are not silently dropped by the exporter.
var activities = new List<Activity>();
using var tracerProvider = OpenTelemetry.Sdk.CreateTracerProviderBuilder()
.AddSource("Experimental.Microsoft.Agents.AI")
.AddInMemoryExporter(activities)
.Build();
var fakeChatClient = new AutoWireTestChatClient();
var inner = new ChatClientAgent(fakeChatClient);
using var agent = new OpenTelemetryAgent(inner, sourceName);
_ = await agent.RunAsync("hi");
Assert.Equal(2, activities.Count);
Assert.All(activities, a => Assert.Equal("Experimental.Microsoft.Agents.AI", a.Source.Name));
Assert.Contains(activities, a => a.DisplayName.StartsWith("invoke_agent", StringComparison.Ordinal));
Assert.Contains(activities, a => string.Equals(a.GetTagItem("gen_ai.operation.name") as string, "chat", StringComparison.Ordinal));
}
#pragma warning disable MEAI001 // ResponseContinuationToken is experimental.
[Fact]
public async Task AutoWireChatClient_PlainAgentRunOptions_PreservesContinuationToken_Async()
{
// ContinuationToken is the fourth base AgentRunOptions property copied by CopyBaseAgentRunOptions
// and is not exercised by AutoWireChatClient_PlainAgentRunOptions_PreservesBaseProperties_Async.
AgentRunOptions? observedOptions = null;
var fakeChatClient = new AutoWireTestChatClient();
var innerChatClientAgent = new ChatClientAgent(fakeChatClient);
var wrapper = new TestAIAgent
{
GetServiceFunc = (type, key) =>
type == typeof(ChatClientAgent) ? innerChatClientAgent : null,
RunAsyncFunc = (messages, session, options, ct) =>
{
observedOptions = options;
return Task.FromResult(new AgentResponse(new ChatMessage(ChatRole.Assistant, "ok")));
},
};
using var agent = new OpenTelemetryAgent(wrapper);
var token = ResponseContinuationToken.FromBytes(new byte[] { 1, 2, 3 });
var inputOptions = new AgentRunOptions
{
ContinuationToken = token,
};
_ = await agent.RunAsync("hi", options: inputOptions);
Assert.NotNull(observedOptions);
Assert.IsType<ChatClientAgentRunOptions>(observedOptions);
Assert.Same(token, observedOptions!.ContinuationToken);
}
#pragma warning restore MEAI001
[Fact]
public async Task AutoWireChatClient_ChatClientAgentRunOptions_NoUserFactory_PreservesChatOptions_Async()
{
// When the caller passes a ChatClientAgentRunOptions without a ChatClientFactory, the auto-wiring
// must clone (not mutate) the caller's options, set the factory, and preserve nested ChatOptions.
var sourceName = Guid.NewGuid().ToString();
var activities = new List<Activity>();
using var tracerProvider = OpenTelemetry.Sdk.CreateTracerProviderBuilder()
.AddSource(sourceName)
.AddInMemoryExporter(activities)
.Build();
ChatOptions? observedChatOptions = null;
var fakeChatClient = new AutoWireTestChatClient
{
OnGetResponseAsync = (msgs, opts) => observedChatOptions = opts,
};
var inner = new ChatClientAgent(fakeChatClient);
using var agent = new OpenTelemetryAgent(inner, sourceName);
var inputChatOptions = new ChatOptions { Temperature = 0.42f, ModelId = "test-model" };
var inputOptions = new ChatClientAgentRunOptions(inputChatOptions);
_ = await agent.RunAsync("hi", options: inputOptions);
// Caller's options must not have been mutated (no factory installed on the caller's instance).
Assert.Null(inputOptions.ChatClientFactory);
// Inner chat client must observe the caller-supplied ChatOptions.
Assert.NotNull(observedChatOptions);
Assert.Equal(0.42f, observedChatOptions!.Temperature);
Assert.Equal("test-model", observedChatOptions.ModelId);
// Auto-wiring still produces a chat span.
Assert.Equal(2, activities.Count);
Assert.Contains(activities, a => string.Equals(a.GetTagItem("gen_ai.operation.name") as string, "chat", StringComparison.Ordinal));
}
[Fact]
public async Task AutoWireChatClient_StreamingDisabled_DoesNotEmitChatSpan_Async()
{
// Symmetry with AutoWireChatClient_Disabled_DoesNotEmitChatSpan_Async for the streaming path.
var sourceName = Guid.NewGuid().ToString();
var activities = new List<Activity>();
using var tracerProvider = OpenTelemetry.Sdk.CreateTracerProviderBuilder()
.AddSource(sourceName)
.AddInMemoryExporter(activities)
.Build();
var fakeChatClient = new AutoWireTestChatClient();
var inner = new ChatClientAgent(fakeChatClient);
using var agent = new OpenTelemetryAgent(inner, sourceName, autoWireChatClient: false);
await foreach (var _ in agent.RunStreamingAsync("hi"))
{
}
var activity = Assert.Single(activities);
Assert.StartsWith("invoke_agent", activity.DisplayName);
}
[Fact]
public async Task AutoWireChatClient_PlainAgentRunOptions_RealChatClientAgent_EmitsChatSpan_Async()
{
// High-level callers may pass the abstract base AgentRunOptions (not ChatClientAgentRunOptions) when
// wiring a ChatClientAgent. Auto-wiring must still kick in: convert to ChatClientAgentRunOptions,
// install the OTel-wrapping factory, and produce both the invoke_agent and chat spans end-to-end.
var sourceName = Guid.NewGuid().ToString();
var activities = new List<Activity>();
using var tracerProvider = OpenTelemetry.Sdk.CreateTracerProviderBuilder()
.AddSource(sourceName)
.AddInMemoryExporter(activities)
.Build();
ChatOptions? observedChatOptions = null;
var fakeChatClient = new AutoWireTestChatClient
{
OnGetResponseAsync = (_, opts) => observedChatOptions = opts,
};
var inner = new ChatClientAgent(fakeChatClient);
using var agent = new OpenTelemetryAgent(inner, sourceName);
// Pass the base AgentRunOptions, not ChatClientAgentRunOptions.
var inputOptions = new AgentRunOptions { AllowBackgroundResponses = false };
_ = await agent.RunAsync("hi", options: inputOptions);
// Inner chat client was actually invoked (auto-wired factory ran without breaking the pipeline).
Assert.NotNull(observedChatOptions);
Assert.Equal(2, activities.Count);
Assert.Contains(activities, a => a.DisplayName.StartsWith("invoke_agent", StringComparison.Ordinal));
Assert.Contains(activities, a => string.Equals(a.GetTagItem("gen_ai.operation.name") as string, "chat", StringComparison.Ordinal));
}
[Fact]
public async Task AutoWireChatClient_PlainAgentRunOptions_RealChatClientAgent_StreamingEmitsChatSpan_Async()
{
// Same as the sync test above but for the streaming path so both invocation paths
// are covered when callers pass a base AgentRunOptions.
var sourceName = Guid.NewGuid().ToString();
var activities = new List<Activity>();
using var tracerProvider = OpenTelemetry.Sdk.CreateTracerProviderBuilder()
.AddSource(sourceName)
.AddInMemoryExporter(activities)
.Build();
ChatOptions? observedChatOptions = null;
var fakeChatClient = new AutoWireTestChatClient
{
OnGetResponseAsync = (_, opts) => observedChatOptions = opts,
};
var inner = new ChatClientAgent(fakeChatClient);
using var agent = new OpenTelemetryAgent(inner, sourceName);
var inputOptions = new AgentRunOptions { AllowBackgroundResponses = false };
await foreach (var _ in agent.RunStreamingAsync("hi", options: inputOptions))
{
}
Assert.NotNull(observedChatOptions);
Assert.Equal(2, activities.Count);
Assert.Contains(activities, a => a.DisplayName.StartsWith("invoke_agent", StringComparison.Ordinal));
Assert.Contains(activities, a => string.Equals(a.GetTagItem("gen_ai.operation.name") as string, "chat", StringComparison.Ordinal));
}
private sealed class AutoWireTestChatClient : IChatClient
{
public Action<IEnumerable<ChatMessage>, ChatOptions?>? OnGetResponseAsync { get; set; }
public Task<ChatResponse> GetResponseAsync(IEnumerable<ChatMessage> messages, ChatOptions? options = null, CancellationToken cancellationToken = default)
{
this.OnGetResponseAsync?.Invoke(messages, options);
return Task.FromResult(new ChatResponse(new ChatMessage(ChatRole.Assistant, "ok")));
}
public async IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(IEnumerable<ChatMessage> messages, ChatOptions? options = null, [EnumeratorCancellation] CancellationToken cancellationToken = default)
{
this.OnGetResponseAsync?.Invoke(messages, options);
await Task.Yield();
yield return new ChatResponseUpdate(ChatRole.Assistant, "ok");
}
public object? GetService(Type serviceType, object? serviceKey = null) =>
serviceType?.IsInstanceOfType(this) == true ? this : null;
public void Dispose() { }
}
#endregion
}
@@ -0,0 +1,26 @@
// Copyright (c) Microsoft. All rights reserved.
using System.Collections.Generic;
using Microsoft.Extensions.AI;
namespace Microsoft.Agents.AI.Workflows.UnitTests;
public class AIAgentsAbstractionsExtensionsTests
{
[Fact]
public void CopyWithAssistantToUserForOtherParticipants_DoesNotMutateOriginalMessages()
{
ChatMessage original = new(ChatRole.Assistant, "from first agent")
{
AuthorName = "firstAgent"
};
List<ChatMessage> copied = new[] { original }
.CopyWithAssistantToUserForOtherParticipants("secondAgent");
Assert.Single(copied);
Assert.Equal(ChatRole.Assistant, original.Role);
Assert.Equal(ChatRole.User, copied[0].Role);
Assert.NotSame(original, copied[0]);
}
}
@@ -209,6 +209,36 @@ public class HandoffOrchestrationTests
Assert.DoesNotContain(capturedNextAgentMessages, m => m.Role == ChatRole.Tool && m.Contents.Any(c => c is FunctionResultContent frc && frc.Result?.ToString() == "Transferred."));
}
[Fact]
public async Task Handoffs_ReassignedMessagesDoNotMutateSharedConversationAsync()
{
var firstAgent = new ChatClientAgent(new MockChatClient((_, options) =>
{
string? transferFuncName = options?.Tools?.FirstOrDefault(t => t.Name.StartsWith("handoff_to_", StringComparison.Ordinal))?.Name;
Assert.NotNull(transferFuncName);
return new ChatResponse([
new ChatMessage(ChatRole.Assistant, "Context from first agent"),
new ChatMessage(ChatRole.Assistant, [new FunctionCallContent("call1", transferFuncName)]),
]);
}), name: "firstAgent");
CapturingAgent secondAgent = new("secondAgent", "The second agent", "Context from first agent");
var workflow =
AgentWorkflowBuilder.CreateHandoffBuilderWith(firstAgent)
.WithHandoff(firstAgent, secondAgent)
.Build();
(_, List<ChatMessage>? result, _, _) = await RunWorkflowAsync(workflow, [new ChatMessage(ChatRole.User, "start")]);
Assert.NotNull(result);
Assert.Equal(ChatRole.User, secondAgent.RoleSeenDuringRun);
ChatMessage sharedMessage = Assert.Single(result, m => m.Text == "Context from first agent");
Assert.Equal(ChatRole.Assistant, sharedMessage.Role);
Assert.NotSame(sharedMessage, secondAgent.MessageSeenDuringRun);
}
[Fact]
public async Task Handoffs_TwoTransfers_HandoffTargetsDoNotReceiveHandoffFunctionMessagesAsync()
{
@@ -1198,6 +1228,44 @@ public class HandoffOrchestrationTests
Workflow workflow, List<ChatMessage> input, ExecutionEnvironment executionEnvironment = ExecutionEnvironment.InProcess_Lockstep)
=> RunWorkflowCheckpointedAsync(workflow, input, executionEnvironment.ToWorkflowExecutionEnvironment());
private sealed class CapturingAgent(string name, string description, string textToCapture) : AIAgent
{
public override string Name => name;
public override string Description => description;
public ChatMessage? MessageSeenDuringRun { get; private set; }
public ChatRole? RoleSeenDuringRun { get; private set; }
protected override ValueTask<AgentSession> CreateSessionCoreAsync(CancellationToken cancellationToken = default)
=> new(new TestAgentSession());
protected override ValueTask<AgentSession> DeserializeSessionCoreAsync(JsonElement serializedState, JsonSerializerOptions? jsonSerializerOptions = null, CancellationToken cancellationToken = default)
=> new(new TestAgentSession());
protected override ValueTask<JsonElement> SerializeSessionCoreAsync(AgentSession session, JsonSerializerOptions? jsonSerializerOptions = null, CancellationToken cancellationToken = default)
=> default;
protected override Task<AgentResponse> RunCoreAsync(
IEnumerable<ChatMessage> messages, AgentSession? session = null, AgentRunOptions? options = null, CancellationToken cancellationToken = default) =>
throw new NotImplementedException();
protected override async IAsyncEnumerable<AgentResponseUpdate> RunCoreStreamingAsync(
IEnumerable<ChatMessage> messages, AgentSession? session = null, AgentRunOptions? options = null, [EnumeratorCancellation] CancellationToken cancellationToken = default)
{
await Task.Yield();
this.MessageSeenDuringRun = messages.Single(m => m.Text == textToCapture);
this.RoleSeenDuringRun = this.MessageSeenDuringRun.Role;
yield return new AgentResponseUpdate(ChatRole.Assistant, "Done")
{
AuthorName = this.Name,
MessageId = Guid.NewGuid().ToString("N"),
};
}
}
private sealed class TestAgentSession() : AgentSession();
private sealed class DoubleEchoAgent(string name) : AIAgent
{
public override string Name => name;
@@ -36,13 +36,18 @@ public sealed class InputWaiterTests : IDisposable
{
Task waitTask = this._waiter.WaitForInputAsync(TimeSpan.FromSeconds(5));
await Task.Delay(50);
waitTask.IsCompleted.Should().BeFalse("the waiter should block until input is signaled");
Task completedBeforeSignal = await Task.WhenAny(waitTask, Task.Delay(100));
completedBeforeSignal.Should().NotBeSameAs(
waitTask,
"the waiter should not complete before input is signaled");
this._waiter.SignalInput();
Task completed = await Task.WhenAny(waitTask, Task.Delay(TimeSpan.FromSeconds(1)));
completed.Should().BeSameAs(waitTask, "the wait task should complete after being signaled");
Task completedAfterSignal = await Task.WhenAny(waitTask, Task.Delay(TimeSpan.FromSeconds(1)));
completedAfterSignal.Should().BeSameAs(
waitTask,
"the wait task should complete after being signaled");
await waitTask;
}
@@ -0,0 +1,546 @@
// Copyright (c) Microsoft. All rights reserved.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using FluentAssertions;
using Microsoft.Agents.AI.Workflows.Execution;
namespace Microsoft.Agents.AI.Workflows.UnitTests;
public sealed class RouteBuilderTests
{
public enum HandlerOverload
{
SyncWithCancellation = 0,
SyncWithoutCancellation = 1,
AsyncWithCancellation = 2,
AsyncWithoutCancellation = 3,
}
private sealed record TestPayload(string Value);
private sealed class HandlerInvocation
{
public object? Message { get; private set; }
public IWorkflowContext? Context { get; private set; }
public CancellationToken CancellationToken { get; private set; }
public int InvocationCount { get; private set; }
public void Capture(object? message, IWorkflowContext context, CancellationToken cancellationToken = default)
{
this.Message = message;
this.Context = context;
this.CancellationToken = cancellationToken;
this.InvocationCount++;
}
}
private sealed class TestExternalRequestContext : IExternalRequestContext, IExternalRequestSink
{
public List<RequestPort> RegisteredPorts { get; } = [];
public List<ExternalRequest> PostedRequests { get; } = [];
public IExternalRequestSink RegisterPort(RequestPort port)
{
this.RegisteredPorts.Add(port);
return this;
}
public ValueTask PostAsync(ExternalRequest request)
{
this.PostedRequests.Add(request);
return default;
}
}
[Theory]
[InlineData(HandlerOverload.SyncWithCancellation)]
[InlineData(HandlerOverload.SyncWithoutCancellation)]
[InlineData(HandlerOverload.AsyncWithCancellation)]
[InlineData(HandlerOverload.AsyncWithoutCancellation)]
public async Task AddHandler_VoidOverloads_RouteExpectedMessageAsync(HandlerOverload overload)
{
// Arrange
RouteBuilder routeBuilder = new(null);
HandlerInvocation invocation = new();
CancellationToken cancellationToken = new CancellationTokenSource().Token;
RegisterVoidHandler(routeBuilder, invocation, overload);
MessageRouter router = routeBuilder.Build();
TestWorkflowContext context = new("executor");
// Act
CallResult? result = await router.RouteMessageAsync("hello", context, cancellationToken: cancellationToken);
// Assert
result.Should().NotBeNull();
result!.IsSuccess.Should().BeTrue();
result.IsVoid.Should().BeTrue();
result.Result.Should().BeNull();
invocation.InvocationCount.Should().Be(1);
invocation.Message.Should().Be("hello");
invocation.Context.Should().BeSameAs(context);
if (UsesCancellationToken(overload))
{
invocation.CancellationToken.Should().Be(cancellationToken);
}
}
[Theory]
[InlineData(HandlerOverload.SyncWithCancellation)]
[InlineData(HandlerOverload.SyncWithoutCancellation)]
[InlineData(HandlerOverload.AsyncWithCancellation)]
[InlineData(HandlerOverload.AsyncWithoutCancellation)]
public async Task AddHandler_ResultOverloads_RouteExpectedMessageAsync(HandlerOverload overload)
{
// Arrange
RouteBuilder routeBuilder = new(null);
HandlerInvocation invocation = new();
CancellationToken cancellationToken = new CancellationTokenSource().Token;
RegisterResultHandler(routeBuilder, invocation, overload);
MessageRouter router = routeBuilder.Build();
TestWorkflowContext context = new("executor");
// Act
CallResult? result = await router.RouteMessageAsync("hello", context, cancellationToken: cancellationToken);
// Assert
result.Should().NotBeNull();
result!.IsSuccess.Should().BeTrue();
result.IsVoid.Should().BeFalse();
result.Result.Should().Be("HELLO");
router.DefaultOutputTypes.Should().Contain(typeof(string));
invocation.InvocationCount.Should().Be(1);
invocation.Message.Should().Be("hello");
invocation.Context.Should().BeSameAs(context);
if (UsesCancellationToken(overload))
{
invocation.CancellationToken.Should().Be(cancellationToken);
}
}
[Theory]
[InlineData(HandlerOverload.SyncWithCancellation)]
[InlineData(HandlerOverload.SyncWithoutCancellation)]
[InlineData(HandlerOverload.AsyncWithCancellation)]
[InlineData(HandlerOverload.AsyncWithoutCancellation)]
public async Task AddCatchAll_VoidOverloads_RouteUnexpectedMessageAsync(HandlerOverload overload)
{
// Arrange
RouteBuilder routeBuilder = new(null);
HandlerInvocation invocation = new();
CancellationToken cancellationToken = new CancellationTokenSource().Token;
TestPayload payload = new("hello");
RegisterVoidCatchAll(routeBuilder, invocation, overload);
MessageRouter router = routeBuilder.Build();
TestWorkflowContext context = new("executor");
// Act
CallResult? result = await router.RouteMessageAsync(payload, context, cancellationToken: cancellationToken);
// Assert
result.Should().NotBeNull();
result!.IsSuccess.Should().BeTrue();
result.IsVoid.Should().BeTrue();
result.Result.Should().BeNull();
invocation.InvocationCount.Should().Be(1);
invocation.Message.Should().BeEquivalentTo(new PortableValue(payload));
invocation.Context.Should().BeSameAs(context);
if (UsesCancellationToken(overload))
{
invocation.CancellationToken.Should().Be(cancellationToken);
}
}
[Theory]
[InlineData(HandlerOverload.SyncWithCancellation)]
[InlineData(HandlerOverload.SyncWithoutCancellation)]
[InlineData(HandlerOverload.AsyncWithCancellation)]
[InlineData(HandlerOverload.AsyncWithoutCancellation)]
public async Task AddCatchAll_ResultOverloads_RouteUnexpectedMessageAsync(HandlerOverload overload)
{
// Arrange
RouteBuilder routeBuilder = new(null);
HandlerInvocation invocation = new();
CancellationToken cancellationToken = new CancellationTokenSource().Token;
TestPayload payload = new("hello");
RegisterResultCatchAll(routeBuilder, invocation, overload);
MessageRouter router = routeBuilder.Build();
TestWorkflowContext context = new("executor");
// Act
CallResult? result = await router.RouteMessageAsync(payload, context, cancellationToken: cancellationToken);
// Assert
result.Should().NotBeNull();
result!.IsSuccess.Should().BeTrue();
result.IsVoid.Should().BeFalse();
result.Result.Should().Be("HELLO");
invocation.InvocationCount.Should().Be(1);
invocation.Message.Should().BeEquivalentTo(new PortableValue(payload));
invocation.Context.Should().BeSameAs(context);
if (UsesCancellationToken(overload))
{
invocation.CancellationToken.Should().Be(cancellationToken);
}
}
[Fact]
public async Task AddHandlerUntyped_VoidAndResultOverloads_RouteExpectedMessageAsync()
{
// Arrange
RouteBuilder routeBuilder = new(null);
HandlerInvocation voidInvocation = new();
HandlerInvocation resultInvocation = new();
CancellationToken cancellationToken = new CancellationTokenSource().Token;
routeBuilder.AddHandlerUntyped(typeof(string), (message, context, token) =>
{
voidInvocation.Capture(message, context, token);
return default;
});
routeBuilder.AddHandlerUntyped<int>(typeof(int), (message, context, token) =>
{
resultInvocation.Capture(message, context, token);
return new((int)message + 1);
});
MessageRouter router = routeBuilder.Build();
TestWorkflowContext context = new("executor");
// Act
CallResult? voidResult = await router.RouteMessageAsync("hello", context, cancellationToken: cancellationToken);
CallResult? typedResult = await router.RouteMessageAsync(41, context, cancellationToken: cancellationToken);
// Assert
voidResult.Should().NotBeNull();
voidResult!.IsVoid.Should().BeTrue();
voidInvocation.Message.Should().Be("hello");
voidInvocation.Context.Should().BeSameAs(context);
voidInvocation.CancellationToken.Should().Be(cancellationToken);
typedResult.Should().NotBeNull();
typedResult!.Result.Should().Be(42);
router.DefaultOutputTypes.Should().Contain(typeof(int));
resultInvocation.Message.Should().Be(41);
resultInvocation.Context.Should().BeSameAs(context);
resultInvocation.CancellationToken.Should().Be(cancellationToken);
}
[Fact]
public void AddHandler_ForPortableValue_ThrowsInvalidOperationException()
{
// Arrange
RouteBuilder routeBuilder = new(null);
// Act
Action act = () => routeBuilder.AddHandler<PortableValue>((message, context) => { });
// Assert
act.Should().Throw<InvalidOperationException>()
.WithMessage("*Use AddCatchAll()*");
}
[Fact]
public void AddHandler_DuplicateRegistrationWithoutOverwrite_ThrowsArgumentException()
{
// Arrange
RouteBuilder routeBuilder = new(null);
routeBuilder.AddHandler<string>((message, context) => { });
// Act
Action act = () => routeBuilder.AddHandler<string>((message, context) => { });
// Assert
act.Should().Throw<ArgumentException>()
.WithMessage("*already registered*");
}
[Fact]
public void AddHandler_OverwriteWithoutExistingRegistration_ThrowsArgumentException()
{
// Arrange
RouteBuilder routeBuilder = new(null);
// Act
Action act = () => routeBuilder.AddHandler<string>((message, context) => { }, overwrite: true);
// Assert
act.Should().Throw<ArgumentException>()
.WithMessage("*has not yet been registered*");
}
[Fact]
public async Task AddHandler_OverwriteExistingRegistration_RoutesUpdatedHandlerAsync()
{
// Arrange
RouteBuilder routeBuilder = new(null);
routeBuilder.AddHandler<string>((message, context) => context.SendMessageAsync("first"));
routeBuilder.AddHandler<string>((message, context) => context.SendMessageAsync("second"), overwrite: true);
MessageRouter router = routeBuilder.Build();
TestWorkflowContext context = new("executor");
// Act
_ = await router.RouteMessageAsync("hello", context);
// Assert
context.SentMessages.Should().ContainSingle().Which.Should().Be("second");
}
[Fact]
public void AddCatchAll_DuplicateRegistrationWithoutOverwrite_ThrowsInvalidOperationException()
{
// Arrange
RouteBuilder routeBuilder = new(null);
routeBuilder.AddCatchAll((message, context) => { });
// Act
Action act = () => routeBuilder.AddCatchAll((message, context) => { });
// Assert
act.Should().Throw<InvalidOperationException>()
.WithMessage("*already registered*");
}
[Fact]
public async Task AddCatchAll_OverwriteExistingRegistration_RoutesUpdatedHandlerAsync()
{
// Arrange
RouteBuilder routeBuilder = new(null);
routeBuilder.AddCatchAll((message, context) => context.SendMessageAsync("first"));
routeBuilder.AddCatchAll((message, context) => context.SendMessageAsync("second"), overwrite: true);
MessageRouter router = routeBuilder.Build();
TestWorkflowContext context = new("executor");
// Act
_ = await router.RouteMessageAsync(new TestPayload("hello"), context);
// Assert
context.SentMessages.Should().ContainSingle().Which.Should().Be("second");
}
[Fact]
public void AddPortHandler_WithoutExternalRequestContext_ThrowsInvalidOperationException()
{
// Arrange
RouteBuilder routeBuilder = new(null);
// Act
Action act = () => routeBuilder.AddPortHandler<string, int>("port", (response, context, cancellationToken) => default, out _);
// Assert
act.Should().Throw<InvalidOperationException>()
.WithMessage("*external request context is required*");
}
[Fact]
public async Task AddPortHandler_RoutesMatchingExternalResponseAsync()
{
// Arrange
TestExternalRequestContext externalRequestContext = new();
RouteBuilder routeBuilder = new(externalRequestContext);
HandlerInvocation invocation = new();
routeBuilder.AddPortHandler<string, int>("port", (response, context, cancellationToken) =>
{
invocation.Capture(response, context, cancellationToken);
return default;
}, out PortBinding portBinding);
await portBinding.PostRequestAsync("request", requestId: "req-1");
MessageRouter router = routeBuilder.Build();
TestWorkflowContext context = new("executor");
CancellationToken cancellationToken = new CancellationTokenSource().Token;
ExternalResponse response = externalRequestContext.PostedRequests.Single().CreateResponse(42);
// Act
CallResult? result = await router.RouteMessageAsync(response, context, cancellationToken: cancellationToken);
// Assert
externalRequestContext.RegisteredPorts.Should().ContainSingle(port => port.Id == "port");
externalRequestContext.PostedRequests.Should().ContainSingle(request => request.RequestId == "req-1");
result.Should().NotBeNull();
result!.IsSuccess.Should().BeTrue();
result.Result.Should().BeSameAs(response);
invocation.InvocationCount.Should().Be(1);
invocation.Message.Should().Be(42);
invocation.Context.Should().BeSameAs(context);
invocation.CancellationToken.Should().Be(cancellationToken);
}
[Fact]
public async Task AddPortHandler_UnknownPort_ReturnsExceptionResultAsync()
{
// Arrange
TestExternalRequestContext externalRequestContext = new();
RouteBuilder routeBuilder = new(externalRequestContext);
routeBuilder.AddPortHandler<string, int>("port", (response, context, cancellationToken) => default, out _);
MessageRouter router = routeBuilder.Build();
ExternalRequest request = ExternalRequest.Create(RequestPort.Create<string, int>("other"), "request", requestId: "req-1");
// Act
CallResult? result = await router.RouteMessageAsync(request.CreateResponse(42), new TestWorkflowContext("executor"));
// Assert
result.Should().NotBeNull();
result!.IsSuccess.Should().BeFalse();
result.Exception.Should().BeOfType<InvalidOperationException>();
result.Exception!.Message.Should().Contain("Unknown port");
}
private static void RegisterVoidHandler(RouteBuilder routeBuilder, HandlerInvocation invocation, HandlerOverload overload)
{
switch (overload)
{
case HandlerOverload.SyncWithCancellation:
routeBuilder.AddHandler<string>((message, context, cancellationToken) => invocation.Capture(message, context, cancellationToken));
break;
case HandlerOverload.SyncWithoutCancellation:
routeBuilder.AddHandler<string>((message, context) => invocation.Capture(message, context));
break;
case HandlerOverload.AsyncWithCancellation:
routeBuilder.AddHandler<string>((message, context, cancellationToken) =>
{
invocation.Capture(message, context, cancellationToken);
return default;
});
break;
case HandlerOverload.AsyncWithoutCancellation:
routeBuilder.AddHandler<string>((message, context) =>
{
invocation.Capture(message, context);
return default;
});
break;
default:
throw new ArgumentOutOfRangeException(nameof(overload));
}
}
private static void RegisterResultHandler(RouteBuilder routeBuilder, HandlerInvocation invocation, HandlerOverload overload)
{
switch (overload)
{
case HandlerOverload.SyncWithCancellation:
routeBuilder.AddHandler<string, string>((message, context, cancellationToken) =>
{
invocation.Capture(message, context, cancellationToken);
return NormalizeHandlerResult(message);
});
break;
case HandlerOverload.SyncWithoutCancellation:
routeBuilder.AddHandler<string, string>((message, context) =>
{
invocation.Capture(message, context);
return NormalizeHandlerResult(message);
});
break;
case HandlerOverload.AsyncWithCancellation:
Func<string, IWorkflowContext, CancellationToken, ValueTask<string>> asyncHandlerWithCancellation = (message, context, cancellationToken) =>
{
invocation.Capture(message, context, cancellationToken);
return new ValueTask<string>(NormalizeHandlerResult(message));
};
routeBuilder.AddHandler(asyncHandlerWithCancellation);
break;
case HandlerOverload.AsyncWithoutCancellation:
Func<string, IWorkflowContext, ValueTask<string>> asyncHandler = (message, context) =>
{
invocation.Capture(message, context);
return new ValueTask<string>(NormalizeHandlerResult(message));
};
routeBuilder.AddHandler(asyncHandler);
break;
default:
throw new ArgumentOutOfRangeException(nameof(overload));
}
}
private static void RegisterVoidCatchAll(RouteBuilder routeBuilder, HandlerInvocation invocation, HandlerOverload overload)
{
switch (overload)
{
case HandlerOverload.SyncWithCancellation:
routeBuilder.AddCatchAll((message, context, cancellationToken) => invocation.Capture(message, context, cancellationToken));
break;
case HandlerOverload.SyncWithoutCancellation:
routeBuilder.AddCatchAll((message, context) => invocation.Capture(message, context));
break;
case HandlerOverload.AsyncWithCancellation:
routeBuilder.AddCatchAll((message, context, cancellationToken) =>
{
invocation.Capture(message, context, cancellationToken);
return default;
});
break;
case HandlerOverload.AsyncWithoutCancellation:
routeBuilder.AddCatchAll((message, context) =>
{
invocation.Capture(message, context);
return default;
});
break;
default:
throw new ArgumentOutOfRangeException(nameof(overload));
}
}
private static void RegisterResultCatchAll(RouteBuilder routeBuilder, HandlerInvocation invocation, HandlerOverload overload)
{
switch (overload)
{
case HandlerOverload.SyncWithCancellation:
routeBuilder.AddCatchAll((message, context, cancellationToken) =>
{
invocation.Capture(message, context, cancellationToken);
return NormalizeCatchAllResult(message);
});
break;
case HandlerOverload.SyncWithoutCancellation:
routeBuilder.AddCatchAll((message, context) =>
{
invocation.Capture(message, context);
return NormalizeCatchAllResult(message);
});
break;
case HandlerOverload.AsyncWithCancellation:
Func<PortableValue, IWorkflowContext, CancellationToken, ValueTask<string>> asyncCatchAllWithCancellation = (message, context, cancellationToken) =>
{
invocation.Capture(message, context, cancellationToken);
return new ValueTask<string>(NormalizeCatchAllResult(message));
};
routeBuilder.AddCatchAll(asyncCatchAllWithCancellation);
break;
case HandlerOverload.AsyncWithoutCancellation:
Func<PortableValue, IWorkflowContext, ValueTask<string>> asyncCatchAll = (message, context) =>
{
invocation.Capture(message, context);
return new ValueTask<string>(NormalizeCatchAllResult(message));
};
routeBuilder.AddCatchAll(asyncCatchAll);
break;
default:
throw new ArgumentOutOfRangeException(nameof(overload));
}
}
private static bool UsesCancellationToken(HandlerOverload overload) =>
overload is HandlerOverload.SyncWithCancellation or HandlerOverload.AsyncWithCancellation;
private static string NormalizeHandlerResult(string message) => message.ToUpperInvariant();
private static string NormalizeCatchAllResult(PortableValue message) => GetPayloadValue(message).ToUpperInvariant();
private static string GetPayloadValue(PortableValue message)
{
return message.As<TestPayload>() is TestPayload payload
? payload.Value
: throw new InvalidOperationException("Expected catch-all message payload to deserialize as TestPayload.");
}
}
@@ -1,5 +1,6 @@
// Copyright (c) Microsoft. All rights reserved.
using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Microsoft.Extensions.AI;
@@ -290,6 +291,121 @@ public sealed class WorkflowEvaluationTests
Assert.DoesNotContain("end", result.Keys);
}
// ---------------------------------------------------------------
// BuildOverallItem tests (expected output / ground truth)
// ---------------------------------------------------------------
[Fact]
public void BuildOverallItem_NoCompletedExecutorWithResponse_ReturnsNull()
{
// Arrange — no ExecutorCompletedEvent with usable response data and no AgentResponseEvent
var events = new List<WorkflowEvent>
{
new ExecutorInvokedEvent("agent-1", "query"),
};
// Act
var item = WorkflowEvaluationExtensions.BuildOverallItem(events, splitter: null, expectedOutput: null);
// Assert
Assert.Null(item);
}
[Fact]
public void BuildOverallItem_NoAgentResponseEvent_FallsBackToLastExecutorCompleted()
{
// Arrange — only ExecutorCompletedEvent (the default when EmitAgentResponseEvents is false)
var finalResponse = new AgentResponse(new ChatMessage(ChatRole.Assistant, "Paris"));
var events = new List<WorkflowEvent>
{
new ExecutorInvokedEvent("researcher", "What is the capital of France?"),
new ExecutorCompletedEvent("researcher", new AgentResponse(new ChatMessage(ChatRole.Assistant, "draft"))),
new ExecutorInvokedEvent("editor", "draft"),
new ExecutorCompletedEvent("editor", finalResponse),
};
// Act
var item = WorkflowEvaluationExtensions.BuildOverallItem(
events, splitter: null, expectedOutput: "Paris");
// Assert
Assert.NotNull(item);
Assert.Equal("What is the capital of France?", item.Query);
Assert.Equal("Paris", item.Response);
Assert.Equal("Paris", item.ExpectedOutput);
}
[Fact]
public void BuildOverallItem_WithFinalResponseAndExpectedOutput_StampsExpectedOutput()
{
// Arrange
var finalResponse = new AgentResponse(new ChatMessage(ChatRole.Assistant, "Ofrece 41 planes"));
var events = new List<WorkflowEvent>
{
new ExecutorInvokedEvent("agent-1", "How many plans does Netlife offer?"),
new ExecutorCompletedEvent("agent-1", finalResponse),
new AgentResponseEvent("agent-1", finalResponse),
};
// Act
var item = WorkflowEvaluationExtensions.BuildOverallItem(
events, splitter: null, expectedOutput: "Ofrece 41 planes");
// Assert
Assert.NotNull(item);
Assert.Equal("How many plans does Netlife offer?", item.Query);
Assert.Equal("Ofrece 41 planes", item.Response);
Assert.Equal("Ofrece 41 planes", item.ExpectedOutput);
}
[Fact]
public void BuildOverallItem_WithFinalResponseAndNoExpectedOutput_LeavesExpectedOutputNull()
{
// Arrange
var finalResponse = new AgentResponse(new ChatMessage(ChatRole.Assistant, "answer"));
var events = new List<WorkflowEvent>
{
new ExecutorInvokedEvent("agent-1", "query"),
new ExecutorCompletedEvent("agent-1", finalResponse),
new AgentResponseEvent("agent-1", finalResponse),
};
// Act
var item = WorkflowEvaluationExtensions.BuildOverallItem(events, splitter: null, expectedOutput: null);
// Assert
Assert.NotNull(item);
Assert.Null(item.ExpectedOutput);
}
[Fact]
public async Task EvaluateAsync_WithIncludeOverallButNoFinalResponse_ThrowsAsync()
{
// Arrange — build a workflow whose AIAgentHostExecutor is NOT bound with
// EmitAgentResponseEvents=true, so no AgentResponseEvent is emitted, and the
// ExecutorCompletedEvent for the host carries null Data. That is the scenario
// where BuildOverallItem returns null. When the caller asks for an overall
// evaluation (includeOverall: true), we should fail fast rather than silently
// returning empty results — regardless of whether expectedOutput was supplied.
var agent = new TestEchoAgent(name: "echo");
var workflow = AgentWorkflowBuilder.BuildSequential(agent);
var input = new List<ChatMessage> { new(ChatRole.User, "Hello") };
var evaluator = new LocalEvaluator(
FunctionEvaluator.Create("noop", (EvalItem _) => true));
await using var run = await InProcessExecution.RunAsync(workflow, input);
// Act + Assert — throws even without expectedOutput
var ex = await Assert.ThrowsAsync<InvalidOperationException>(() =>
run.EvaluateAsync(
evaluator,
includeOverall: true,
includePerAgent: false));
Assert.Contains("EmitAgentResponseEvents", ex.Message);
}
// ---------------------------------------------------------------
// EvaluateAsync integration test
// ---------------------------------------------------------------
+24
View File
@@ -99,6 +99,30 @@ The `AGUIChatClient` supports:
- Integration with `Agent` for client-side history management
- Interrupt metadata passthrough (`availableInterrupts` and `resume`)
## Tool Return Helpers
Use `state_update` when a backend tool needs to send different payloads to the model, the UI, and shared state. The `text` value remains the LLM-bound tool result, `tool_result` becomes the AG-UI `ToolCallResultEvent.content` for frontend rendering, and `state` is merged into durable shared state.
```python
from agent_framework import Content, tool
from agent_framework.ag_ui import state_update
@tool
async def get_weather(city: str) -> Content:
data = await fetch_weather(city)
return state_update(
text=f"{city}: {data['temp']}°C and {data['conditions']}",
tool_result={
"component": "weather-card",
"city": city,
"temperature": data["temp"],
"conditions": data["conditions"],
"humidity": data["humidity"],
},
state={"weather": {"city": city, **data}},
)
```
## Documentation
- **[Getting Started Tutorial](getting_started/)** - Step-by-step guide to building AG-UI servers and clients
@@ -49,8 +49,11 @@ from ._run_common import (
_close_reasoning_block, # type: ignore
_emit_content, # type: ignore
_extract_resume_payload, # type: ignore
_extract_tool_result_display, # type: ignore
_has_only_tool_calls, # type: ignore
_normalize_resume_interrupts, # type: ignore
_resolve_ui_payload, # type: ignore
_stringify_tool_result, # type: ignore
)
from ._utils import (
convert_agui_tools_to_agent_framework,
@@ -381,17 +384,23 @@ def _handle_step_based_approval(messages: list[Any]) -> list[BaseEvent]:
def _make_approval_tool_result_events(resolved_approval_results: list[Content]) -> list[ToolCallResultEvent]:
"""Build TOOL_CALL_RESULT events for tools executed during approval resolution."""
"""Build TOOL_CALL_RESULT events for tools executed during approval resolution.
Honors ``TOOL_RESULT_DISPLAY_KEY`` so tools returning
``state_update(..., tool_result=...)`` route the display payload to the UI
event even when gated by HITL approval.
"""
events: list[ToolCallResultEvent] = []
for resolved in resolved_approval_results:
if resolved.call_id:
raw = resolved.result if resolved.result is not None else ""
result_str = raw if isinstance(raw, str) else json.dumps(make_json_safe(raw))
llm_str = _stringify_tool_result(raw)
ui_str = _resolve_ui_payload(llm_str, _extract_tool_result_display(resolved))
events.append(
ToolCallResultEvent(
message_id=generate_event_id(),
tool_call_id=resolved.call_id,
content=result_str,
content=ui_str,
role="tool",
)
)
@@ -32,11 +32,14 @@ from ag_ui.core import (
from agent_framework import Content
from ._orchestration._predictive_state import PredictiveStateHandler
from ._state import TOOL_RESULT_STATE_KEY
from ._state import TOOL_RESULT_DISPLAY_KEY, TOOL_RESULT_STATE_KEY
from ._utils import generate_event_id, make_json_safe
logger = logging.getLogger(__name__)
# Sentinel for an unset display_result; distinguishes "caller didn't pass" from None/{}/"".
_UNSET = object()
def _has_only_tool_calls(contents: list[Any]) -> bool:
"""Check if contents have only tool calls (no text)."""
@@ -235,6 +238,22 @@ def _emit_tool_call(
return events
def _extract_tool_result_marker_values(content: Content, key: str) -> list[Any]:
"""Extract marker values from outer and inner tool-result content."""
values: list[Any] = []
outer_ap = getattr(content, "additional_properties", None) or {}
if key in outer_ap:
values.append(outer_ap[key])
for item in content.items or ():
item_ap = getattr(item, "additional_properties", None) or {}
if key in item_ap:
values.append(item_ap[key])
return values
def _extract_tool_result_state(content: Content) -> dict[str, Any] | None:
"""Extract a deterministic AG-UI state update from a tool-result ``Content``.
@@ -252,14 +271,7 @@ def _extract_tool_result_state(content: Content) -> dict[str, Any] | None:
"""
merged: dict[str, Any] | None = None
outer_ap = getattr(content, "additional_properties", None) or {}
outer_state = outer_ap.get(TOOL_RESULT_STATE_KEY)
if isinstance(outer_state, dict):
merged = dict(outer_state)
for item in content.items or ():
item_ap = getattr(item, "additional_properties", None) or {}
item_state = item_ap.get(TOOL_RESULT_STATE_KEY)
for item_state in _extract_tool_result_marker_values(content, TOOL_RESULT_STATE_KEY):
if isinstance(item_state, dict):
if merged is None:
merged = dict(item_state)
@@ -269,6 +281,21 @@ def _extract_tool_result_state(content: Content) -> dict[str, Any] | None:
return merged
def _extract_tool_result_display(content: Content) -> Any: # noqa: ANN401
"""Extract a UI-only AG-UI tool result display payload, if present."""
display_values = _extract_tool_result_marker_values(content, TOOL_RESULT_DISPLAY_KEY)
return display_values[-1] if display_values else _UNSET
def _stringify_tool_result(raw_result: Any) -> str: # noqa: ANN401
return raw_result if isinstance(raw_result, str) else json.dumps(make_json_safe(raw_result))
def _resolve_ui_payload(llm_str: str, display_result: Any) -> str: # noqa: ANN401
"""Pick the UI-bound string: the serialized display payload when set, else the LLM string."""
return llm_str if display_result is _UNSET else _stringify_tool_result(display_result)
def _emit_tool_result_common(
call_id: str,
raw_result: Any,
@@ -276,6 +303,7 @@ def _emit_tool_result_common(
predictive_handler: PredictiveStateHandler | None = None,
*,
state_update: Mapping[str, Any] | None = None,
display_result: Any = _UNSET, # noqa: ANN401
) -> list[BaseEvent]:
"""Shared helper for emitting ToolCallEnd + ToolCallResult events and performing FlowState cleanup.
@@ -301,13 +329,14 @@ def _emit_tool_result_common(
events.append(ToolCallEndEvent(tool_call_id=call_id))
flow.tool_calls_ended.add(call_id)
result_content = raw_result if isinstance(raw_result, str) else json.dumps(make_json_safe(raw_result))
result_content = _stringify_tool_result(raw_result)
ui_result_content = _resolve_ui_payload(result_content, display_result)
message_id = generate_event_id()
events.append(
ToolCallResultEvent(
message_id=message_id,
tool_call_id=call_id,
content=result_content,
content=ui_result_content,
role="tool",
)
)
@@ -358,12 +387,14 @@ def _emit_tool_result(
return []
raw_result = content.result if content.result is not None else ""
state_update = _extract_tool_result_state(content)
display_result = _extract_tool_result_display(content)
return _emit_tool_result_common(
content.call_id,
raw_result,
flow,
predictive_handler,
state_update=state_update,
display_result=display_result,
)
@@ -530,12 +561,14 @@ def _emit_mcp_tool_result(
return []
raw_output = content.output if content.output is not None else ""
state_update = _extract_tool_result_state(content)
display_result = _extract_tool_result_display(content)
return _emit_tool_result_common(
content.call_id,
raw_output,
flow,
predictive_handler,
state_update=state_update,
display_result=display_result,
)
@@ -1,12 +1,13 @@
# Copyright (c) Microsoft. All rights reserved.
"""Deterministic tool-driven AG-UI state updates.
"""Deterministic tool-driven AG-UI state updates and display payloads.
Tools wired into the :mod:`agent_framework_ag_ui` endpoint can push a
deterministic state update by returning :func:`state_update`. Unlike
``predict_state_config`` — which emits ``StateDeltaEvent``s optimistically from
LLM-predicted tool call arguments — ``state_update`` runs *after* the tool
executes, so the AG-UI state always reflects the tool's actual return value.
deterministic state update or a per-call tool result display payload by
returning :func:`state_update`. Unlike ``predict_state_config`` — which emits
``StateDeltaEvent``s optimistically from LLM-predicted tool call arguments —
``state_update`` runs *after* the tool executes, so AG-UI state and display
content always reflect the tool's actual return value.
See issue https://github.com/microsoft/agent-framework/issues/3167 for the
motivating discussion.
@@ -14,33 +15,48 @@ motivating discussion.
from __future__ import annotations
import json
from collections.abc import Mapping
from typing import Any
from agent_framework import Content
__all__ = ["TOOL_RESULT_STATE_KEY", "state_update"]
from ._utils import make_json_safe
__all__ = ["TOOL_RESULT_DISPLAY_KEY", "TOOL_RESULT_STATE_KEY", "state_update"]
TOOL_RESULT_STATE_KEY = "__ag_ui_tool_result_state__"
"""Reserved ``Content.additional_properties`` key used to carry a tool-driven
state snapshot from a tool return value through to the AG-UI emitter."""
TOOL_RESULT_DISPLAY_KEY = "__ag_ui_tool_result_display__"
"""Reserved ``Content.additional_properties`` key used to carry UI-only tool result display content from a tool return value through to the AG-UI emitter."""
_UNSET = object()
def _serialize_tool_result(value: Any) -> str: # noqa: ANN401
return value if isinstance(value, str) else json.dumps(make_json_safe(value))
def state_update(
text: str = "",
*,
state: Mapping[str, Any],
state: Mapping[str, Any] | None = None,
tool_result: Any = _UNSET, # noqa: ANN401
) -> Content:
"""Build a tool return value that deterministically updates AG-UI shared state.
"""Build a tool return value that updates AG-UI shared state or display content.
Return the result of this helper from an agent tool to push a state update
to AG-UI clients using the actual tool output, rather than LLM-predicted
tool arguments.
or UI-only display payload to AG-UI clients using the actual tool output,
rather than LLM-predicted tool arguments.
When the AG-UI endpoint emits the tool result, it will:
* Forward ``text`` to the LLM as the normal ``function_result`` content.
* Use ``tool_result`` as the ``ToolCallResultEvent.content`` payload shown
to AG-UI clients, falling back to ``text`` when no display payload is set.
* Merge ``state`` into ``FlowState.current_state``.
* Emit a deterministic ``StateSnapshotEvent`` after the ``ToolCallResult``
event so frontends observe the updated state deterministically. If
@@ -49,7 +65,7 @@ def state_update(
Example:
.. code-block:: python
from agent_framework import tool
from agent_framework import Content, tool
from agent_framework_ag_ui import state_update
@@ -61,24 +77,61 @@ def state_update(
state={"weather": {"city": city, **data}},
)
Example:
.. code-block:: python
from agent_framework import Content, tool
from agent_framework_ag_ui import state_update
@tool
async def get_weather(city: str) -> Content:
data = await _fetch_weather(city)
return state_update(
text=f"{city}: {data['temp']}°C and {data['conditions']}",
tool_result={
"component": "weather-card",
"city": city,
"temperature": data["temp"],
"conditions": data["conditions"],
"humidity": data["humidity"],
},
state={"weather": {"city": city, **data}},
)
Args:
text: Text passed back to the LLM as the ``function_result`` content.
Defaults to an empty string for tools whose only output is a state
update.
state: A mapping merged into the AG-UI shared state via JSON-compatible
``dict.update`` semantics. Nested dicts are replaced, not deep-merged.
tool_result: JSON-safe payload emitted to AG-UI clients as
``ToolCallResultEvent.content`` for frontend rendering. The LLM
still receives ``text``. If ``text`` is empty, the serialized
display payload is also used as the LLM-bound text fallback.
Returns:
A ``Content`` object with ``type="text"``. The state payload rides in
``additional_properties`` under :data:`TOOL_RESULT_STATE_KEY` and is
extracted by the AG-UI emitter.
``additional_properties`` under :data:`TOOL_RESULT_STATE_KEY`
(``"__ag_ui_tool_result_state__"``), and the display payload rides
under :data:`TOOL_RESULT_DISPLAY_KEY`
(``"__ag_ui_tool_result_display__"``). Both reserved keys are extracted
by the AG-UI emitter.
Raises:
TypeError: If ``state`` is not a ``Mapping``.
"""
if not isinstance(state, Mapping):
if state is not None and not isinstance(state, Mapping):
raise TypeError(f"state_update() 'state' must be a Mapping, got {type(state).__name__}")
additional_properties: dict[str, Any] = {}
if state is not None:
additional_properties[TOOL_RESULT_STATE_KEY] = dict(state)
if tool_result is not _UNSET:
display_content = _serialize_tool_result(tool_result)
additional_properties[TOOL_RESULT_DISPLAY_KEY] = display_content
if not text:
text = display_content
return Content.from_text(
text,
additional_properties={TOOL_RESULT_STATE_KEY: dict(state)},
additional_properties=additional_properties,
)
@@ -68,6 +68,19 @@ def _tool_result_with_state(call_id: str, text: str, state: dict[str, Any]) -> A
)
def _tool_result_with_display(call_id: str, text: str, tool_result: Any, **kwargs: Any) -> AgentResponseUpdate:
"""Build a function_result update carrying an optional UI display marker."""
return AgentResponseUpdate(
contents=[
Content.from_function_result(
call_id=call_id,
result=[state_update(text=text, tool_result=tool_result, **kwargs)],
)
],
role="assistant",
)
# ── Golden stream tests ──
@@ -265,3 +278,87 @@ async def test_deterministic_state_coexists_with_predict_state_config() -> None:
# The final observed state must contain both the deterministic and predictive contributions.
final = stream.snapshot()
assert final["weather"] == {"city": "SF", "temp": 14}, f"Deterministic state missing from final snapshot: {final}"
async def test_tool_result_display_payload_reaches_ui_event_only() -> None:
"""Rich display payload overrides TOOL_CALL_RESULT without leaking marker keys."""
updates = [
_tool_call("call-1", "get_weather", '{"city": "SF"}'),
_tool_result_with_display(
"call-1",
text="Weather in SF: 14°C foggy",
tool_result={"city": "SF", "temp": 14, "conditions": "foggy"},
),
]
agent = _build_agent(updates)
stream = await _run(agent, PAYLOAD)
stream.assert_bookends()
stream.assert_no_run_error()
stream.assert_tool_calls_balanced()
result = stream.first("TOOL_CALL_RESULT")
assert result.content == '{"city": "SF", "temp": 14, "conditions": "foggy"}'
assert "__ag_ui_tool_result_display__" not in result.content
assert "__ag_ui_tool_result_state__" not in result.content
async def test_tool_result_display_falls_back_to_text_when_unset() -> None:
"""Without a display marker, the UI event keeps the existing text content."""
updates = [
_tool_call("call-1", "get_weather", '{"city": "SF"}'),
_tool_result_with_state(
"call-1",
text="Weather in SF: 14°C foggy",
state={"weather": {"city": "SF", "temp": 14}},
),
]
agent = _build_agent(updates)
stream = await _run(agent, PAYLOAD)
stream.assert_bookends()
stream.assert_no_run_error()
stream.assert_tool_calls_balanced()
result = stream.first("TOOL_CALL_RESULT")
assert result.content == "Weather in SF: 14°C foggy"
assert "__ag_ui_tool_result_display__" not in result.content
assert "__ag_ui_tool_result_state__" not in result.content
async def test_tool_result_display_coexists_with_state_snapshot() -> None:
"""Display and durable state markers produce one deterministic state snapshot."""
updates = [
_tool_call("call-1", "get_weather", '{"city": "SF"}'),
_tool_result_with_display(
"call-1",
text="Weather in SF: 14°C foggy",
tool_result={"city": "SF", "temp": 14, "conditions": "foggy"},
state={"weather": {"city": "SF", "temp": 14, "conditions": "foggy"}},
),
]
agent = _build_agent(updates)
stream = await _run(agent, PAYLOAD)
stream.assert_bookends()
stream.assert_no_run_error()
stream.assert_tool_calls_balanced()
stream.assert_ordered_types(["TOOL_CALL_RESULT", "STATE_SNAPSHOT", "RUN_FINISHED"])
result = stream.first("TOOL_CALL_RESULT")
assert result.content == '{"city": "SF", "temp": 14, "conditions": "foggy"}'
result_idx = stream.events.index(result)
deterministic_snapshots = [
event
for event in stream.events[result_idx + 1 :]
if getattr(getattr(event, "type", None), "value", getattr(event, "type", None)) == "STATE_SNAPSHOT"
]
assert len(deterministic_snapshots) == 1
assert deterministic_snapshots[0].snapshot["weather"] == {
"city": "SF",
"temp": 14,
"conditions": "foggy",
}
assert "__ag_ui_tool_result_display__" not in str(deterministic_snapshots[0].snapshot)
assert "__ag_ui_tool_result_state__" not in str(deterministic_snapshots[0].snapshot)
@@ -448,3 +448,37 @@ async def test_resolve_approval_responses_returns_only_approved() -> None:
rejection_results = [c for c in all_contents if c.type == "function_result" and c.call_id == rejected_call_id]
assert len(rejection_results) == 1
assert "rejected" in str(rejection_results[0].result).lower()
class TestApprovalToolResultDisplayChannel:
"""Approved tools using ``state_update(..., tool_result=...)`` must route the
display payload to the UI event while ``flow.tool_results`` still receives
the LLM-bound text. The HITL approval emitter is separate from the standard
streaming emitter, so it gets its own coverage.
"""
def test_approval_emits_display_payload_when_marker_present(self) -> None:
from agent_framework_ag_ui import state_update
from agent_framework_ag_ui._agent_run import _make_approval_tool_result_events
display_payload = {"city": "Seattle", "temp": 14, "conditions": "foggy"}
inner = state_update(text="14°C, foggy", tool_result=display_payload)
resolved = Content.from_function_result(call_id="call_disp", result=[inner])
events = _make_approval_tool_result_events([resolved])
assert len(events) == 1
# UI event must carry the serialized display payload, NOT the LLM text.
assert json.loads(events[0].content) == display_payload
assert events[0].content != "14°C, foggy"
def test_approval_falls_back_to_text_when_no_marker(self) -> None:
"""Backward compat: without a display marker, behaviour is unchanged."""
from agent_framework_ag_ui._agent_run import _make_approval_tool_result_events
resolved = Content.from_function_result(call_id="call_plain", result="Sunny in Seattle")
events = _make_approval_tool_result_events([resolved])
assert len(events) == 1
assert events[0].content == "Sunny in Seattle"
@@ -15,7 +15,7 @@ from agent_framework_ag_ui._run_common import (
_extract_tool_result_state,
_normalize_resume_interrupts,
)
from agent_framework_ag_ui._state import TOOL_RESULT_STATE_KEY
from agent_framework_ag_ui._state import TOOL_RESULT_DISPLAY_KEY, TOOL_RESULT_STATE_KEY
class TestNormalizeResumeInterrupts:
@@ -140,6 +140,15 @@ class TestStateUpdateHelper:
TOOL_RESULT_STATE_KEY: {"weather": {"temp": 14}},
}
def test_builds_text_content_with_display_marker(self):
"""state_update can carry a UI display payload without requiring state."""
c = state_update(text="14°C, foggy", tool_result={"temp": 14, "conditions": "foggy"})
assert c.type == "text"
assert c.text == "14°C, foggy"
assert c.additional_properties == {
TOOL_RESULT_DISPLAY_KEY: '{"temp": 14, "conditions": "foggy"}',
}
def test_empty_text_is_allowed(self):
"""State-only tools can omit the text argument."""
c = state_update(state={"steps": ["a", "b"]})
@@ -165,6 +174,18 @@ class TestStateUpdateHelper:
inner = c.additional_properties[TOOL_RESULT_STATE_KEY]
assert inner is not caller_state
def test_tool_result_without_text_falls_back_to_display_payload(self):
"""Display-only tools use the serialized display payload as LLM text."""
c = state_update(tool_result={"temp": 14, "conditions": "foggy"})
assert c.text == '{"temp": 14, "conditions": "foggy"}'
assert c.additional_properties[TOOL_RESULT_DISPLAY_KEY] == '{"temp": 14, "conditions": "foggy"}'
def test_string_tool_result_is_not_json_encoded_again(self):
"""A pre-serialized display string passes through verbatim."""
c = state_update(text="Weather summary", tool_result='{"temp":14}')
assert c.text == "Weather summary"
assert c.additional_properties[TOOL_RESULT_DISPLAY_KEY] == '{"temp":14}'
class TestExtractToolResultState:
"""Tests for ``_extract_tool_result_state``."""
@@ -265,6 +286,60 @@ class TestEmitToolResultWithState:
assert result_events[0].content == "Weather: 14°C"
assert TOOL_RESULT_STATE_KEY not in result_events[0].content
def test_display_payload_routes_to_ui_only(self):
"""A display marker overrides only the UI event, not the LLM-bound tool result."""
tool_return = state_update(
text="Weather: 14°C",
tool_result={"temp": 14, "conditions": "foggy"},
)
content = Content.from_function_result(call_id="c1", result=[tool_return])
flow = FlowState()
events = _emit_tool_result(content, flow)
result_events = [e for e in events if e.type == EventType.TOOL_CALL_RESULT]
assert len(result_events) == 1
assert result_events[0].content == '{"temp": 14, "conditions": "foggy"}'
assert flow.tool_results[-1]["content"] == "Weather: 14°C"
assert TOOL_RESULT_DISPLAY_KEY not in result_events[0].content
assert TOOL_RESULT_DISPLAY_KEY not in flow.tool_results[-1]["content"]
def test_plain_tool_result_uses_existing_content_for_both_channels(self):
"""Without a display marker, UI and LLM channels keep the existing derivation."""
content = Content.from_function_result(call_id="c1", result="plain result")
flow = FlowState()
events = _emit_tool_result(content, flow)
result_events = [e for e in events if e.type == EventType.TOOL_CALL_RESULT]
assert len(result_events) == 1
assert result_events[0].content == "plain result"
assert flow.tool_results[-1]["content"] == "plain result"
def test_display_only_payload_falls_back_to_llm_content(self):
"""When text is empty, both channels receive the serialized display payload."""
tool_return = state_update(tool_result={"temp": 14})
content = Content.from_function_result(call_id="c1", result=[tool_return])
flow = FlowState()
events = _emit_tool_result(content, flow)
result_events = [e for e in events if e.type == EventType.TOOL_CALL_RESULT]
assert result_events[0].content == '{"temp": 14}'
assert flow.tool_results[-1]["content"] == '{"temp": 14}'
def test_pre_serialized_display_string_routes_verbatim(self):
"""String display payloads pass through without JSON double-encoding."""
tool_return = state_update(text="Weather summary", tool_result='{"temp":14}')
content = Content.from_function_result(call_id="c1", result=[tool_return])
flow = FlowState()
events = _emit_tool_result(content, flow)
result_events = [e for e in events if e.type == EventType.TOOL_CALL_RESULT]
assert result_events[0].content == '{"temp":14}'
assert flow.tool_results[-1]["content"] == "Weather summary"
def test_coexists_with_active_predictive_state_handler(self):
"""Both predictive and deterministic state produce a single coalesced snapshot.
@@ -346,3 +421,31 @@ class TestEmitMcpToolResultWithState:
events = _emit_mcp_tool_result(content, flow)
assert all(e.type != EventType.STATE_SNAPSHOT for e in events)
class TestEmitMcpToolResultWithDisplay:
"""MCP tool results must honour the display marker so UI consumers can
render structured payloads while ``flow.tool_results`` keeps the LLM
string. MCP outputs do not pass through ``parse_result``; the marker
rides on the outer content's ``additional_properties``.
"""
def test_mcp_tool_result_routes_display_payload_to_ui_only(self):
import json as _json
display_payload = {"rows": [{"id": 1, "name": "alpha"}, {"id": 2, "name": "beta"}]}
content = Content.from_mcp_server_tool_result(
call_id="mcp_disp",
output="2 rows returned",
additional_properties={TOOL_RESULT_DISPLAY_KEY: display_payload},
)
flow = FlowState()
events = _emit_mcp_tool_result(content, flow)
result_events = [e for e in events if e.type == EventType.TOOL_CALL_RESULT]
assert len(result_events) == 1
# UI event carries the structured display payload.
assert _json.loads(result_events[0].content) == display_payload
# LLM-side accumulator keeps the short text.
assert flow.tool_results[-1]["content"] == "2 rows returned"
+47 -3
View File
@@ -10,7 +10,7 @@ import logging
import re
import sys
from abc import abstractmethod
from collections.abc import Callable, Collection, Sequence
from collections.abc import Callable, Collection, Coroutine, Sequence
from contextlib import AsyncExitStack, _AsyncGeneratorContextManager # type: ignore
from datetime import timedelta
from functools import partial
@@ -264,6 +264,7 @@ class MCPTool:
self.is_connected: bool = False
self._tools_loaded: bool = False
self._prompts_loaded: bool = False
self._pending_reload_tasks: set[asyncio.Task[None]] = set()
def __str__(self) -> str:
return f"MCPTool(name={self.name}, description={self.description})"
@@ -905,12 +906,47 @@ class MCPTool:
if isinstance(message, types.ServerNotification):
match message.root.method:
case "notifications/tools/list_changed":
await self.load_tools()
self._schedule_reload(self.load_tools())
case "notifications/prompts/list_changed":
await self.load_prompts()
self._schedule_reload(self.load_prompts())
case _:
logger.debug("Unhandled notification: %s", message.root.method)
def _schedule_reload(self, coro: Coroutine[Any, Any, None]) -> None:
"""Schedule a reload coroutine as a background task.
Reloads (load_tools / load_prompts) triggered by MCP server
notifications must NOT be awaited inside the message handler because
the handler runs on the MCP SDK's single-threaded receive loop.
Awaiting a session request (e.g. ``list_tools``) from within that loop
deadlocks: the receive loop cannot read the response while it is
blocked waiting for the handler to return.
Instead we fire the reload as an independent ``asyncio.Task`` and keep
a strong reference in ``_pending_reload_tasks`` so it is not garbage-
collected before completion. Only one reload per kind (tools / prompts)
is kept in flight; a new notification cancels the previous pending task
for the same coroutine name to avoid unbounded growth.
"""
# Cancel-and-replace: only one reload per kind should be in flight.
reload_name = f"mcp-reload:{self.name}:{coro.__qualname__}"
for existing in list(self._pending_reload_tasks):
if existing.get_name() == reload_name and not existing.done():
logger.debug("Cancelling in-flight reload %s; superseded by new notification", reload_name)
existing.cancel()
async def _safe_reload() -> None:
try:
await coro
except asyncio.CancelledError:
raise
except Exception:
logger.warning("Background MCP reload failed", exc_info=True)
task = asyncio.create_task(_safe_reload(), name=reload_name)
self._pending_reload_tasks.add(task)
task.add_done_callback(self._pending_reload_tasks.discard)
def _determine_approval_mode(
self,
*candidate_names: str,
@@ -1047,6 +1083,14 @@ class MCPTool:
params = types.PaginatedRequestParams(cursor=tool_list.nextCursor)
async def _close_on_owner(self) -> None:
# Cancel any pending reload tasks before tearing down the session.
tasks = list(self._pending_reload_tasks)
for task in tasks:
task.cancel()
self._pending_reload_tasks.clear()
if tasks:
await asyncio.gather(*tasks, return_exceptions=True)
await self._safe_close_exit_stack()
self._exit_stack = AsyncExitStack()
self.session = None
+111 -1
View File
@@ -1,6 +1,7 @@
# Copyright (c) Microsoft. All rights reserved.
# type: ignore[reportPrivateUsage]
import asyncio
import contextlib
import json
import logging
import os
@@ -1615,7 +1616,7 @@ async def test_mcp_connection_reset_integration():
async def test_mcp_tool_message_handler_notification():
"""Test that message_handler correctly processes tools/list_changed and prompts/list_changed
notifications."""
notifications by scheduling reloads as background tasks."""
tool = MCPStdioTool(name="test_tool", command="python")
# Mock the load_tools and load_prompts methods
@@ -1629,6 +1630,8 @@ async def test_mcp_tool_message_handler_notification():
result = await tool.message_handler(tools_notification)
assert result is None
# The reload is scheduled as a background task; let it run.
await asyncio.sleep(0)
tool.load_tools.assert_called_once()
# Reset mock
@@ -1641,6 +1644,7 @@ async def test_mcp_tool_message_handler_notification():
result = await tool.message_handler(prompts_notification)
assert result is None
await asyncio.sleep(0)
tool.load_prompts.assert_called_once()
# Test unhandled notification
@@ -1664,6 +1668,112 @@ async def test_mcp_tool_message_handler_error():
assert result is None
async def test_mcp_tool_message_handler_does_not_block_receive_loop():
"""Test that message_handler does not deadlock the MCP receive loop.
Regression test for https://github.com/microsoft/agent-framework/issues/4828.
When the MCP server sends a ``notifications/tools/list_changed``
notification, the handler must NOT await ``load_tools()`` synchronously
because that would block the single-threaded MCP receive loop, preventing
it from delivering the ``list_tools`` response — a classic deadlock.
"""
tool = MCPStdioTool(name="test_tool", command="python")
# Use an event to make load_tools block until we release it.
# This simulates load_tools waiting for a session response that the
# receive loop would need to deliver.
release = asyncio.Event()
async def slow_load_tools():
await release.wait()
tool.load_tools = slow_load_tools # type: ignore[assignment]
tools_notification = Mock(spec=types.ServerNotification)
tools_notification.root = Mock()
tools_notification.root.method = "notifications/tools/list_changed"
# message_handler must return immediately even though load_tools blocks.
await tool.message_handler(tools_notification)
# If the handler had awaited load_tools synchronously, we would never
# reach this line (deadlock). Verify the reload task is pending.
assert len(tool._pending_reload_tasks) == 1
# Unblock the reload so the background task finishes cleanly.
release.set()
# Wait for the pending reload task(s) to complete so their done-callbacks
# have a chance to remove them from _pending_reload_tasks.
await asyncio.wait_for(asyncio.gather(*tool._pending_reload_tasks), timeout=1)
assert len(tool._pending_reload_tasks) == 0
async def test_mcp_tool_message_handler_reload_failure_is_logged(caplog: pytest.LogCaptureFixture):
"""Background reload errors are logged, not raised into the receive loop."""
tool = MCPStdioTool(name="test_tool", command="python")
tool.load_tools = AsyncMock(side_effect=RuntimeError("connection lost"))
tools_notification = Mock(spec=types.ServerNotification)
tools_notification.root = Mock()
tools_notification.root.method = "notifications/tools/list_changed"
await tool.message_handler(tools_notification)
# Let the background task run — it should not propagate the exception.
# Snapshot tasks and await them to ensure done-callbacks fire.
pending = list(tool._pending_reload_tasks)
if pending:
await asyncio.wait_for(asyncio.gather(*pending, return_exceptions=True), timeout=1)
tool.load_tools.assert_called_once()
assert len(tool._pending_reload_tasks) == 0
# Verify the warning was actually logged with exception info.
reload_warnings = [r for r in caplog.records if "Background MCP reload failed" in r.message]
assert len(reload_warnings) == 1
assert reload_warnings[0].levelname == "WARNING"
assert reload_warnings[0].exc_info is not None
async def test_mcp_tool_message_handler_cancel_and_replace():
"""Sending two notifications in quick succession cancels the first reload task."""
tool = MCPStdioTool(name="test_tool", command="python")
release = asyncio.Event()
call_count = 0
async def blocking_load_tools():
nonlocal call_count
call_count += 1
await release.wait()
tool.load_tools = blocking_load_tools # type: ignore[assignment]
notification = Mock(spec=types.ServerNotification)
notification.root = Mock()
notification.root.method = "notifications/tools/list_changed"
# First notification — starts a blocking reload task.
await tool.message_handler(notification)
assert len(tool._pending_reload_tasks) == 1
first_task = next(iter(tool._pending_reload_tasks))
# Second notification — should cancel the first and replace it.
await tool.message_handler(notification)
# Yield to the event loop so the cancellation is processed.
with contextlib.suppress(asyncio.CancelledError):
await first_task
assert first_task.cancelled()
assert len(tool._pending_reload_tasks) == 1
second_task = next(iter(tool._pending_reload_tasks))
assert second_task is not first_task
# Unblock and let the second task finish.
release.set()
await asyncio.wait_for(asyncio.gather(*tool._pending_reload_tasks), timeout=1)
assert len(tool._pending_reload_tasks) == 0
async def test_mcp_tool_sampling_callback_no_client():
"""Test sampling callback error path when no chat client is available."""
tool = MCPStdioTool(name="test_tool", command="python")
+1 -1
View File
@@ -602,7 +602,7 @@ dependencies = [
[package.metadata]
requires-dist = [
{ name = "agent-framework-core", editable = "packages/core" },
{ name = "github-copilot-sdk", marker = "python_full_version >= '3.11'", specifier = "<=1.0.0b2,>=1.0.0b2" },
{ name = "github-copilot-sdk", marker = "python_full_version >= '3.11'", specifier = ">=1.0.0b2,<=1.0.0b2" },
]
[[package]]