* Foundry Evals integration for .NET
- Core evaluation framework: EvalItem, LocalEvaluator, FunctionEvaluator, EvalChecks
- IAgentEvaluator interface with MeaiEvaluatorAdapter bridge
- AgentEvaluationExtensions for agent.EvaluateAsync() overloads
- FoundryEvals wrapping MEAI quality/safety evaluators
- ConversationSplitters (LastTurn, Full) and IConversationSplitter
- EvalItem.PerTurnItems() for multi-turn decomposition
- HasImageContent for multimodal content detection
- WorkflowEvaluationExtensions for per-agent workflow evaluation
- 7 eval samples mirroring Python parity:
02-agents/Evaluation: SimpleEval, ExpectedOutputs, Multimodal
03-workflows/Evaluation: WorkflowEval
05-end-to-end/Evaluation: FoundryQuality, MixedProviders, ConversationSplits
- Comprehensive unit tests (1958 passing)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Rewrite FoundryEvals to use real Foundry Evals API
Replace MEAI evaluator shim with actual OpenAI EvaluationClient protocol
methods. FoundryEvals now creates eval definitions, submits runs, polls
for completion, and fetches per-item results server-side.
- New constructor: FoundryEvals(AIProjectClient, model, evaluators)
- Add FoundryEvalConverter for MEAI ChatMessage -> Foundry JSON format
- Add EvalId, RunId, ReportUrl to AgentEvaluationResults
- All 20 built-in evaluator constants now work (agent, tool, quality, safety)
- Remove Microsoft.Extensions.AI.Evaluation.Quality/Safety dependencies
- Update all samples for new constructor (no more ChatConfiguration)
- Replace BuildEvaluators tests with ResolveEvaluator tests
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add response output to CustomEvals and ExpectedOutputs samples
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review: pagination, validation, error handling, tests
FoundryEvals fixes:
- Add pagination for output items (has_more/after cursor)
- Add guard clauses for pollIntervalSeconds/timeoutSeconds <= 0
- Fix double TryGetProperty for passed field parsing
- Throw on all-tool-evaluators with no tool definitions
- Fix XML doc (default 300s, not 180s)
New tests (30 added, 1989 total):
- EvalChecks: NonEmpty, ContainsExpected (pass/fail/skip/case),
HasImageContent, ToolCallsPresent
- FoundryEvalConverter: ConvertMessage (text, image, function call,
function results fan-out, empty fallback, mixed content),
ConvertEvalItem, BuildTestingCriteria (quality/agent/tool/groundedness
data mappings), BuildItemSchema
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix review: null-refs, Data.ToString() bug, ContainsExpected, add tests
- Fix NullReferenceException in sample Response display (pattern matching)
- Fix WorkflowEvaluationExtensions Data?.ToString() producing type names
instead of message text (pattern-match ChatMessage/AgentResponse/list)
- Change EvalChecks.ContainsExpected to return Passed=false when no
ExpectedOutput (was silently passing, masking misconfiguration)
- Add EvalItem constructor tests with LastTurn/Full/null splitters
- Add FoundryEvalConverter.ConvertMessage DataContent (base64 image) test
- Add ExtractAgentData tests with ChatMessage, list, and AgentResponse data
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix review: conversation fidelity, eval caching, fallback tests
- WorkflowEvaluationExtensions: preserve full response messages (tool calls,
intermediate) instead of synthetic 2-message conversation. Cast completed
Data to AgentResponse and use Messages when available, fallback to text.
- FoundryEvals: cache evalId per schema shape (hasContext, hasTools) so
subsequent EvaluateAsync calls create runs under the same eval definition.
- MeaiEvaluatorAdapter: code already correctly passes queryMessages (not full
conversation) to IEvaluator — no change needed, verified by inspection.
- Add tests: AgentResponse full messages preservation, unknown object
ToString() fallback for ExtractAgentData.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Rename AzureAI→Foundry: move eval files, update references
- Move FoundryEvals.cs and FoundryEvalConverter.cs from
Microsoft.Agents.AI.AzureAI to Microsoft.Agents.AI.Foundry
- Update namespace from AzureAI to Foundry in both files
- Add explicit usings required by Foundry project (no implicit usings)
- Move FoundryEvalConverter tests to Foundry.UnitTests project
(avoids ReplacingRedactor type conflict from dual project refs)
- Update all sample csproj references and using statements
- Remove Foundry project reference from AI UnitTests
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* PR review round 4: wire up tool extraction, remove eval cache, fix null safety
- BuildEvalItem: extract tools from agent via GetService<ChatOptions>() into EvalItem.Tools (Python parity)
- FoundryEvals: remove eval ID cache - each call creates fresh definition (matches Python behavior)
- FoundryEvals: replace null-forgiving operators with descriptive InvalidOperationException
- MixedProviders sample: remove unnecessary explicit PackageReferences (transitively provided)
- FoundryEvalConverter: document that tool results take precedence over text content
- Add LocalEvaluator zero-checks test documenting 0 metrics = failed behavior
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Python-dotnet parity: 9 feature gaps filled
New checks:
- ToolCallArgsMatch() — verify tool call names + argument subset match
- ToolCalledCheck(ToolCalledMode.Any, ...) — match any of the specified tools
- ToolCalledMode enum (All/Any)
FoundryEvals enhancements:
- Default evaluators now [Relevance, Coherence, TaskAdherence] (was Relevance, Coherence)
- Auto-add ToolCallAccuracy when items have tool definitions
- EvaluateTracesAsync — evaluate by response_ids, trace_ids, or agent_id
- EvaluateFoundryTargetAsync — evaluate deployed Foundry targets
Result type enrichment:
- AgentEvaluationResults: added Status, Error, PerEvaluator, DetailedItems
- New EvalItemResult/EvalScoreResult/PerEvaluatorResult types
- FoundryEvals populates all new fields from API responses
Workflow fix:
- Skip internal executors (_*, input-conversation, end-conversation, end)
Tests: 8 new tests covering ToolCallArgsMatch, ToolCalledMode.Any, internal executor filtering
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add MeaiEvaluatorAdapter and PerTurnItems edge case tests
- 3 tests for MeaiEvaluatorAdapter: query message forwarding, synthetic
response fallback, multiple items aggregation
- 3 tests for EvalItem.PerTurnItems: empty conversation, no user messages,
system+assistant only
- StubEvaluator and StubChatClient test helpers
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Blocking link check for outdated package in DevUI.
* Replace Dictionary<string, object> payloads with typed wire models
Introduce internal FoundryEvalWireModels.cs with compile-time-safe types
for the OpenAI Evals API wire format. The OpenAI .NET SDK (2.9.1) only
provides protocol-level methods with BinaryContent/ClientResult — no
typed request models. These internal models replace scattered dictionary
literals with [JsonPropertyName]-annotated classes, giving:
- Compile-time safety (typos become build errors)
- Single point of change when the API evolves
- IntelliSense discoverability
- Cleaner serialization via JsonPolymorphic for content items
Models: WireContentItem hierarchy (text, image, tool_call, tool_result),
WireMessage, WireEvalItemPayload, WireTestingCriterion, WireItemSchema,
WireCreateEvalRequest, WireCreateRunRequest, and data source variants.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Skip metric when Foundry returns neither score nor passed
When an evaluator returns no score and no passed value, the previous
code created BooleanMetric(name, false), which falsely failed items
via ItemPassed. Now we skip the MEAI metric entirely for indeterminate
results — the raw data remains available in DetailedItems for diagnostics.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR #4914 review comments: fix tool evaluator bug and add tests
- Fix duplicate ToolCallAccuracy: resolve evaluator names before checking
against ToolEvaluators set (Comment 2)
- Make FilterToolEvaluators internal for testability; add tests for the
ArgumentException edge case when all evaluators are tool-type (Comment 3)
- Add CancellationToken test for LocalEvaluator (Comment 4)
- Add EvaluateAsync integration test on Run with sequential workflow and
per-agent SubResults verification (Comment 5)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address Peter's review comments on PR #4914
- Add trailing newline to Evaluation_FoundryQuality.csproj (Comment 6)
- Make evaluator name lookups case-insensitive: switch BuiltinEvaluators,
ToolEvaluators, AgentEvaluators, and ResolveEvaluator's StartsWith check
from Ordinal to OrdinalIgnoreCase (Comment 7)
- Add Trace.TraceWarning when Foundry returns fewer results than submitted
items, indicating expected vs actual count before padding (Comment 8)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add Microsoft.Extensions.AI.Evaluation packages to Directory.Packages.props
These were removed in #5269 as unused, but are needed by the Foundry
and core evaluation integration added in this PR.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: alliscode <bentho@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* .NET: [Feature Branch] Add basic durable workflow support (#3648)
* Add basic durable workflow support.
* PR feedback fixes
* Add conditional edge sample.
* PR feedback fixes.
* Minor cleanup.
* Minor cleanup
* Minor formatting improvements.
* Improve comments/documentation on the execution flow.
* .NET: [Feature Branch] Add Azure Functions hosting support for durable workflows (#3935)
* Adding azure functions workflow support.
* - PR feedback fixes.
- Add example to demonstrate complex Object as payload.
* rename instanceId to runId.
* Use custom ITaskOrchestrator to run orchestrator function.
* .NET: [Feature Branch] Adding support for events & shared state in durable workflows (#4020)
* Adding support for events & shared state in durable workflows.
* PR feedback fixes
* PR feedback fixes.
* Add YieldOutputAsync calls to 05_WorkflowEvents sample executors
The integration test asserts that WorkflowOutputEvent is found in the
stream, but the sample executors only used AddEventAsync for custom
events and never called YieldOutputAsync. Since WorkflowOutputEvent is
only emitted via explicit YieldOutputAsync calls, the assertion would
fail. Added YieldOutputAsync to each executor to match the test
expectation and demonstrate the API in the sample.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix deserialization to use shared serializer options.
* PR feedback updates.
* Sample cleanup
* PR feedback fixes
* Addressing PR review feedback for DurableStreamingWorkflowRun
- Use -1 instead of 0 for taskId in TaskFailedException when task ID is not relevant.
- Add [NotNullWhen(true)] to TryParseWorkflowResult out parameter following .NET TryXXX conventions.
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* .NET: [Feature Branch] Add nested sub-workflow support for durable workflows (#4190)
* .NET: [Feature Branch] Add nested sub-workflow support for durable workflows
* fix readme path
* Switch Orchestration output from string to DurableWorkflowResult.
* PR feedback fixes
* Minor cleanup based on PR feedback.
* .NET: [Feature Branch] Add Human In the Loop support for durable workflows (#4358)
* Add Azure Functions HITL workflow sample
Add 06_WorkflowHITL Azure Functions sample demonstrating Human-in-the-Loop
workflow support with HTTP endpoints for status checking and approval responses.
The sample includes:
- ExpenseReimbursement workflow with RequestPort for manager approval
- Custom HTTP endpoint to check workflow status and pending approvals
- Custom HTTP endpoint to send approval responses via RaiseEventAsync
- demo.http file with step-by-step interaction examples
* PR feedback fixes
* Minor comment cleanup
* Minor comment clReverted the `!context.IsReplaying` guards on `PendingEvents.Add`/`RemoveAll` and `SetCustomStatus` in `ExecuteRequestPortAsync`. The guards broke fan-out scenarios where parallel RequestPorts need to be discoverable after replay. `SetCustomStatus` is idempotent metadata that doesn't affect replay determinism.eanup
* fix for PR feedback
* PR feedback updates
* Improvements to samples
* Improvements to README
* Update samples to use parallel request ports.
* Unit tests
* Introduce local variables to improve readability of Workflows.Workflows access patter
* Use GitHub-style callouts and add PowerShell command variants in HITL sample README
* Add changelog entries for durable workflow support (#4436)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Bump Microsoft.DurableTask.Worker to 1.19.1 to fix version downgrade
Microsoft.Azure.Functions.Worker.Extensions.DurableTask 1.13.1 requires
Microsoft.DurableTask.Worker >= 1.19.1 via its transitive dependency on
Microsoft.DurableTask.Worker.Grpc 1.19.1.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix broken markdown links in durable workflow sample READMEs
- Create Workflow/README.md with environment setup docs
- Fix ../README.md -> ../../README.md in ConsoleApps 01, 02, 03, 08
- Fix SubWorkflows relative path (3 levels -> 4 levels up)
- Fix dead Durable Task Scheduler URL
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix build errors from main merge: Throw conflict, ExecuteAsync rename, GetNewSessionAsync rename
- Remove InjectSharedThrow from DurableTask csproj (uses Workflows' internal Throw via InternalsVisibleTo)
- Update ExecuteAsync -> ExecuteCoreAsync with WorkflowTelemetryContext.Disabled
- Update GetNewSessionAsync -> CreateSessionAsync
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Move durable workflow samples to 04-hosting/DurableWorkflows
Aligns with main branch sample reorganization where durable samples
live under 04-hosting/ (alongside DurableAgents/).
- Move samples/Durable/Workflow/ -> samples/04-hosting/DurableWorkflows/
- Add Directory.Build.props matching DurableAgents pattern
- Update slnx project paths
- Update integration test sample paths
- Update README cd paths and cross-references
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix build errors: remove duplicate base class members, update renamed APIs
- Remove duplicate OutputLog, WriteInputAsync, CreateTestTimeoutCts, etc. from
ConsoleAppSamplesValidation (already in SamplesValidationBase)
- Update AddFanInEdge -> AddFanInBarrierEdge in workflow samples
- Update GetNewSessionAsync -> CreateSessionAsync in workflow samples
- Update SourceId -> ExecutorId (obsolete) in workflow samples
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix dotnet format issues: add UTF-8 BOM and remove unused using
- Add UTF-8 BOM to 20 .cs files across DurableTask, AzureFunctions,
unit tests, and workflow samples
- Remove unnecessary using directive in 07_SubWorkflows/Executors.cs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix typo PaymentProcesser -> PaymentProcessor and garbled arrows in README
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix GetExecutorName to handle agent names with underscores
Split on last underscore instead of first, and validate that the
suffix is a 32-char hex string (sanitized GUID) before stripping it.
This prevents truncation of agent names like 'my_agent' when the
executor ID is 'my_agent_<guid>'.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Align DurableTask.Client.AzureManaged to 1.19.1
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Bump DurableTask and Azure Functions extension package versions
- DurableTask.* packages: 1.19.1 -> 1.22.0
- Functions.Worker.Extensions.DurableTask: 1.13.1 -> 1.16.0
- Functions.Worker.Extensions.DurableTask.AzureManaged: 1.0.1 -> 1.5.0 (telemetry bug fix)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Bump DurableTask SDK packages to 1.22.0
- DurableTask.Client: 1.19.1 -> 1.22.0
- DurableTask.Client.AzureManaged: 1.19.1 -> 1.22.0
- DurableTask.Worker: 1.19.1 -> 1.22.0
- DurableTask.Worker.AzureManaged: 1.19.1 -> 1.22.0
- Azure Functions extensions kept at original versions (1.13.1/1.0.1) due to
host-side DurableTask.Core 3.7.0 incompatibility with newer extensions
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update Microsoft.Azure.Functions.Worker.Extensions.DurableTask to "1.16.0"
* Add the local.settings.json files to the sample which were previously ignored. This aligns with our other samples.
* Increase timeout for tests as CI has them failing transiently.
* increaset timeout value for azure functions integration tests.
* Add YieldsOutput(string) to workflow shared state sample executors
ValidateOrder and EnrichOrder call YieldOutputAsync with string messages,
but only their TOutput (OrderDetails) was in the allowed yield types.
This caused TargetInvocationException in the WorkflowSharedState sample
validation integration test.
* Downgrade the durable packages to 1.18.0
* Downgrading Worker.Extensions.DurableTask to 1.12.1
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* refactor: Rename AggregateTurnMessagesExecutor
* feat: Rework Agent Hosting for Configurability and HIL support
* Adds support for selecting whether updates and/or full responses are
emitted to events
* Adds support for HIL/FunctionCalls (including interception)
* Implements internal support for ExternalRequests from any executor
(not just RequestPort)
* test: Add tests for new AIAgentHostExecutor functionality
* feat: Unify non-Handoff Agent Hosting
* doc: More explicit documentation for `overwrite` in RouteBuilder
* Roslyn Source Generators for Workflow Executor Routing.
* Update dotnet/src/Microsoft.Agents.AI.Workflows.Generators/ExecutorRouteGenerator.cs
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* WIP.
* All fixed up except dangling sends/yields attriutes, working on that next.
* Add protocol-only generation for SendsMessage/YieldsOutput attributes
* Ensuring collections that can change order are sorted to enable pipeline caching.
* Improvents per PR feedback.
---------
Co-authored-by: alliscode <bentho@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Upgrade to .NET 10
- Require .NET 10 SDK
- Include net10.0 assets in all assemblies
- Move net9.0-only targets to net10.0
- Update LangVersion to latest
- Remove complicated distinctions between debug target TFMs and release target TFMs
- Remove unnecessary package dependencies when built into netcoreapp
- Clean up some ifdefs
- Clean up some analyzer warnings
* Fix CI
westeyandGitHub
Dmytro StrukandGitHub
Stephen ToubandGitHub
Tao ChenandGitHub
