Jacob Alber and GitHub
18d9ccc216
Merge branch 'main' into copilot/add-e2e-tests-for-magentic-again
2026-05-14 14:07:15 -04:00
3256550c55
.NET: fix: allow naming handoff workflows ( #5799 )
...
* fix: allow naming handoff workflows
* Only set name/description if not NullOrWhitespace
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com >
---------
Co-authored-by: Jacob Alber <jalber@fernir.com >
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com >
Co-authored-by: Jacob Alber <jaalber@microsoft.com >
2026-05-14 17:10:27 +00:00
190ca75b6a
.NET: Add Workflow Builder Specialized Edge tests ( #5826 )
...
* Add workflow builder edge tests
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/3c3d5324-cdcd-4a38-8c67-94e4e78e29c5
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
* Strengthen workflow edge helper tests
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/af831ee2-0a99-4427-9ffd-a3b5022c1b3b
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
* Normalize edge helper bad input validation
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/af831ee2-0a99-4427-9ffd-a3b5022c1b3b
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
* Clarify edge helper target validation
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/af831ee2-0a99-4427-9ffd-a3b5022c1b3b
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
* Use explicit target parameter names
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/af831ee2-0a99-4427-9ffd-a3b5022c1b3b
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
* Document workflow edge test helpers
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/af831ee2-0a99-4427-9ffd-a3b5022c1b3b
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
* Clarify null element validation messages
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/af831ee2-0a99-4427-9ffd-a3b5022c1b3b
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
* Add repeated chain executor coverage
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/af831ee2-0a99-4427-9ffd-a3b5022c1b3b
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
* Preserve Throw helper validation style
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/af831ee2-0a99-4427-9ffd-a3b5022c1b3b
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
* Cover empty switch case targets
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/af831ee2-0a99-4427-9ffd-a3b5022c1b3b
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
* Relax builder null assertion parameter checks
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/af831ee2-0a99-4427-9ffd-a3b5022c1b3b
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
* Inline ValidateTargets into call sites
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/cb9a6a6a-02c7-41a8-a4b4-da16ad62ef86
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
* Refactor ForwardExcept with TFM-specialized TryGetNonEnumeratedCount
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/b081f61f-93ce-45dc-abbd-82c465395470
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
* Use TFM-specialized count check: TryGetNonEnumeratedCount for NET6+, ICollection pattern for NETFX
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/8ec28a43-e7b7-456e-8d8e-921511b4accc
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
* Apply TFM-specialized count check to ForwardMessage as well
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/9238ea32-a3e8-4b83-9683-484ad400071f
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
* Address review feedback: simplify Throw.IfNull in SwitchBuilder per westey-m suggestion
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/299950fd-4457-47f3-a373-f65d601b7ea5
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
* Use indexed parameter name in SwitchBuilder Throw.IfNull: executors[index]
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/c5655707-5b0b-44f3-98a9-5f3961e32cfe
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
* Revert #if NET6_0_OR_GREATER back to #if NET; inline executorIndex++
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/c5655707-5b0b-44f3-98a9-5f3961e32cfe
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
* Add comment explaining unusual Throw.IfNull use for null elements inside collection
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/c5655707-5b0b-44f3-98a9-5f3961e32cfe
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com >
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
Co-authored-by: Jacob Alber <jaalber@microsoft.com >
2026-05-14 16:23:41 +00:00
8058fb1c5b
.NET: Fix flaky InputWaiter_WaitForInputAsync_BlocksUntilSignaledAsync ( #5835 )
...
* test: remove finite timeout in BlocksUntilSignaledAsync to fix race
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/962b7404-4266-4a16-906c-ba3e607c2764
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
* address review: clarify comment, add timeout test, cross-reference test names
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/e406a5f2-ad31-4d37-b090-69e10713f885
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com >
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
Co-authored-by: Jacob Alber <jaalber@microsoft.com >
2026-05-14 15:36:42 +00:00
Jacob Alber and GitHub
5be4d9f9f4
Merge branch 'main' into copilot/add-e2e-tests-for-magentic-again
2026-05-14 10:12:30 -04:00
eb40535436
.NET: Add Executor RouteBuilder Unit Tests ( #5824 )
...
* Add RouteBuilder unit tests
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/012f3b3b-acb9-4869-9084-b767cbe1885b
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
* Address RouteBuilder test review feedback
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/012f3b3b-acb9-4869-9084-b767cbe1885b
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
* Fix RouteBuilder test nullability warning
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/012f3b3b-acb9-4869-9084-b767cbe1885b
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
* Refine RouteBuilder test helpers
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/012f3b3b-acb9-4869-9084-b767cbe1885b
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
* Refactor overload int constants to HandlerOverload enum
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/19397f58-a88a-41cf-bd85-588f520e0d0f
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
* Fix ValueTask compatibility with .NET Framework 4.7.2
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/a8437809-0898-43a6-a950-09eb3417f58a
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
* Fix IDE0001 format errors - simplify generic type names
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/8573214e-ec42-4969-ba94-76bdc8ad3e59
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com >
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
Co-authored-by: Jacob Alber <jaalber@microsoft.com >
2026-05-14 13:36:38 +00:00
3f5b4fdb4e
Escape JSON string values in CreateProgressLedgerResponse test helper
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/ec610c61-0a14-44e2-82fd-1cf35e85d6cc
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-14 03:22:40 +00:00
3d16456bbe
Fix IDE1006: add Async suffix to async test methods in MagenticOrchestrationTests
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/629fcc07-865e-4832-9e59-ea13df561c5a
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-14 02:17:52 +00:00
7d062bf4f9
Remove temporary markdown files
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/b3e67553-a3a3-4282-98f2-afd8ad7a6b5d
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-14 01:54:32 +00:00
70abd53afd
Rewrite ImplementationReview.md with final 23-test status
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/e490fdf7-f107-4fde-ba1f-efdfd9a729c6
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-14 01:47:35 +00:00
eb5dec4162
Add IsTerminated guard to TakeTurnAsync and post-termination rejection test
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/e490fdf7-f107-4fde-ba1f-efdfd9a729c6
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-14 01:42:54 +00:00
d5bbf4ec9c
Add empty-team validation to MagenticWorkflowBuilder.Build() and E2E test
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/e490fdf7-f107-4fde-ba1f-efdfd9a729c6
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-14 01:32:44 +00:00
9cd4f2dc49
Rewrite Magentic implementation review
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/ed87670a-bf4d-4ba5-a2f3-395a2eead9de
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-14 01:21:56 +00:00
4dbe3598db
Update stall docs to use > semantics, skip checkpoint-state tests, simplify NextSpeaker fallback test
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/cc9ea5a8-84d8-4b6d-bb60-ac9619824d81
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-14 01:17:44 +00:00
964210730c
Rewrite Magentic E2E implementation review
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/3d15763b-3a68-488e-9412-3fa280e083c0
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-14 00:57:10 +00:00
5baa5a5f3f
Update implementation review doc for replan-fix and stall threshold alignment
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/43e46b0d-4263-4353-856a-c3730abb1734
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-13 23:42:16 +00:00
07e68604f1
Fix replan-on-every-turn: skip plan on agent return; align StallCount to > (match Python)
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/43e46b0d-4263-4353-856a-c3730abb1734
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-13 23:40:24 +00:00
e4538c34d3
Magentic E2E: preserve IsStalled on stall-triggered plan reviews, add routing/stall tests
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/9b34e409-61b8-4650-ae55-34efad034ed0
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-13 23:27:28 +00:00
1e242c2f9f
Add Consecutive_Stalls_Trigger_Reset E2E test for multi-stall threshold reset
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/9b34e409-61b8-4650-ae55-34efad034ed0
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-13 23:26:02 +00:00
e8cbd1cfec
Add Progress_Made_Decrements_StallCount E2E test verifying stall count decrement avoids reset
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/9b34e409-61b8-4650-ae55-34efad034ed0
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-13 23:23:29 +00:00
72c79aeac5
Add Task_Delegates_To_Correct_Agent E2E test with multi-participant routing assertion
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/9b34e409-61b8-4650-ae55-34efad034ed0
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-13 23:20:43 +00:00
c46ce45fdb
Preserve IsStalled on stall-triggered plan review requests
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/1b9e74e8-69e1-43f2-8467-c5ba963c2622
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-13 23:10:45 +00:00
1c8e16b332
Rewrite Magentic E2E implementation review
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/21f3b1ae-183e-4fea-99ad-14efc19f084d
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-13 22:58:34 +00:00
f566d49cf7
Update ImplementationReview.md to reflect 18 tests and new coverage
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/125f6628-6b3b-4c51-9a51-ae84baece6bb
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-13 22:51:28 +00:00
5063540e19
Add PlanReview_Multiple_Revisions E2E test
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/125f6628-6b3b-4c51-9a51-ae84baece6bb
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-13 22:49:20 +00:00
34b270937b
Add Stall_NoProgress_Increments_StallCount E2E test
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/125f6628-6b3b-4c51-9a51-ae84baece6bb
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-13 22:47:34 +00:00
600bc91c84
Add ProgressLedger_Max_Retries_Triggers_Reset E2E test
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/125f6628-6b3b-4c51-9a51-ae84baece6bb
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-13 22:46:21 +00:00
375356f765
Add ProgressLedger_Retry_On_Parse_Failure E2E test
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/125f6628-6b3b-4c51-9a51-ae84baece6bb
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-13 22:44:17 +00:00
584d9b1163
Rewrite Magentic E2E implementation review
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/6fe88a80-2e05-40d5-9539-ca7c59b9022b
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-13 22:36:18 +00:00
9a955c33a6
Update ImplementationReview.md to reflect 14 tests
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/aba19507-7c7e-40dd-850d-d1fabb5dfa65
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-13 22:30:33 +00:00
1ff632d365
Add Instruction_Message_Sent_When_Present E2E test
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/aba19507-7c7e-40dd-850d-d1fabb5dfa65
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-13 22:28:42 +00:00
2673054af2
Add PlanReview_On_Stall_Replan E2E test
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/aba19507-7c7e-40dd-850d-d1fabb5dfa65
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-13 22:26:31 +00:00
4601c1ac7b
Add MaxResetLimit_Terminates_Workflow E2E test
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/aba19507-7c7e-40dd-850d-d1fabb5dfa65
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-13 22:25:12 +00:00
eb3e0e476f
Rewrite Magentic E2E implementation review
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/1f878ef4-61b0-410a-a8bc-ebf618b3e5de
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-13 22:18:08 +00:00
4cb3010927
Update MagenticE2E_ImplementationReview.md with full coverage status
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/6e8bca46-448d-4f21-a7e9-240179571970
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-13 22:10:59 +00:00
d52de26759
Add MaxStallCount_Triggers_Reset E2E test
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/6e8bca46-448d-4f21-a7e9-240179571970
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-13 22:08:57 +00:00
b7fd35401b
Add MaxRoundLimit_Terminates_Workflow E2E test
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/6e8bca46-448d-4f21-a7e9-240179571970
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-13 22:07:32 +00:00
bbd94f7b05
Add PlanReview_Revised_Triggers_Replan E2E test
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/6e8bca46-448d-4f21-a7e9-240179571970
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-13 22:06:14 +00:00
1b609b230b
Add Task_Completes_After_Multiple_Rounds E2E test
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/6e8bca46-448d-4f21-a7e9-240179571970
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-13 22:03:25 +00:00
d1a66a8f3d
Add NextSpeaker_Empty_Falls_Back_To_First E2E test
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/6e8bca46-448d-4f21-a7e9-240179571970
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-13 22:02:11 +00:00
c0bd2a33eb
Add PlanSignoff_Disabled_Proceeds_Immediately E2E test
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/6e8bca46-448d-4f21-a7e9-240179571970
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-13 22:00:48 +00:00
8a6a5948ed
Add Magentic E2E implementation review
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/b2c60ce7-4d05-4a0d-b05d-d4284f5b7bb3
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-13 21:52:45 +00:00
00a132a095
Add next speaker validation test
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/322c9e2d-59bc-42ad-9a1e-f6fd4c866b26
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-13 21:28:35 +00:00
cd7e8c8822
Add plan review test and event emission tests
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/322c9e2d-59bc-42ad-9a1e-f6fd4c866b26
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-13 21:26:53 +00:00
b207921fcc
Fix MagenticOrchestrator output declaration and add first E2E test
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/322c9e2d-59bc-42ad-9a1e-f6fd4c866b26
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-13 21:23:17 +00:00
68fbead632
Add MagenticOrchestrationTests.cs scaffold for Magentic E2E tests
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/44a4fd8a-3828-40e5-9435-90381aeffdb8
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-13 20:52:51 +00:00
90ec8e4555
Add E2E test plan for Magentic orchestrator
...
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/96d76349-1ffd-482b-a3ee-ed208778b1bb
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com >
2026-05-13 20:22:46 +00:00
981726cc15
.NET: feat(evals): add ground_truth/expected_output support for workflow evaluation ( #5755 )
...
* .NET: feat(evals): add ground_truth/expected_output support for workflow eval
Brings .NET to parity with Python PR #5234 for issue #5135 :
- Add expectedOutput parameter to Run.EvaluateAsync (workflow) and stamp on the overall EvalItem.ExpectedOutput.
- Map EvalItem.ExpectedOutput -> ground_truth in the Foundry JSONL payload, item_schema, and data_mapping for similarity.
- Add GroundTruthEvaluators set (currently builtin.similarity) and a FindMissingGroundTruthEvaluators helper.
- Fail fast with InvalidOperationException when a ground-truth evaluator is selected but no item provides an ExpectedOutput, instead of surfacing a remote provider error.
- Add tests in FoundryEvalConverterTests and WorkflowEvaluationTests.
- Add Evaluation_WorkflowExpectedOutputs sample (workflow + Foundry similarity).
Fixes microsoft/agent-framework#5135 (.NET side).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com >
* Address review: relax BuildOverallItem events to IReadOnlyList<WorkflowEvent>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com >
* Sample: disable per-agent breakdown when using reference-based evaluator
Per-agent EvalItems are intentionally left without ExpectedOutput, so the new fail-fast validation in FoundryEvals would throw when Similarity is invoked for per-agent items. Pass includePerAgent: false in the workflow + similarity sample, and document this gotcha in the EvaluateAsync XML doc.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com >
* Fix BuildOverallItem: fall back to last ExecutorCompletedEvent
AgentResponseEvent is only emitted when AIAgentHostOptions.EmitAgentResponseEvents is enabled, which is not the default for WorkflowBuilder(agent).AddEdge(...). When it is absent, fall back to the last non-internal ExecutorCompletedEvent whose Data is an AgentResponse / ChatMessage / string so the overall EvalItem (and any expectedOutput) is produced. Without this, samples wired up the standard way returned 0 evaluation items.
Update test to cover the fallback path.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com >
* Sample: enable EmitAgentResponseEvents; eval throws clear error when no overall response found
Root cause of '0 results': AIAgentHostExecutor only emits AgentResponseEvent when AIAgentHostOptions.EmitAgentResponseEvents is true (default false). For ordinary AIAgent executors the runtime's ExecutorCompletedEvent.Data is null, so the prior fallback couldn't find a final response either.
Sample now builds executors with EmitAgentResponseEvents=true via BindAsExecutor(hostOptions). EvaluateAsync now throws InvalidOperationException with a remediation hint when the user supplies expectedOutput but no overall final response can be located, instead of silently returning 0/0.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com >
* Guard against null sample/error/usage/datasource_item in ParseDetailedItem
Foundry eval responses can have these properties present with JSON null
or non-object values, which caused JsonElement.TryGetProperty to throw
'requires Object, has Null'. Check ValueKind == Object before drilling in.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com >
* Address PR review: reorder expectedOutput, tighten ground-truth check, add fail-fast test
* WorkflowEvaluationExtensions.EvaluateAsync: move 'expectedOutput' to
after 'splitter' so the original positional contract of (splitter,
cancellationToken) is preserved for existing callers.
* FoundryEvals: require ALL items to carry ExpectedOutput when a
ground-truth evaluator is selected (e.g. similarity), not just any.
Reference-based evaluators score per-item, so a single missing GT
would still surface as a provider-side validation error. Updated
fail-fast message accordingly.
* WorkflowEvaluationTests: add EvaluateAsync_WithExpectedOutputButNoFinalResponse_ThrowsAsync
to verify the InvalidOperationException is thrown (and that the
message mentions EmitAgentResponseEvents).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com >
* Fail-fast on missing overall item regardless of expectedOutput; harden BuildOverallItem default
* EvaluateAsync now throws InvalidOperationException whenever 'includeOverall'
is requested but BuildOverallItem cannot produce an item, instead of only
when 'expectedOutput' is supplied. Same misconfiguration (agents not bound
with EmitAgentResponseEvents) used to silently return empty results — now
it surfaces a clear, actionable error in both cases.
* BuildOverallItem switch default now throws instead of returning null. The
preceding for-loop already constrains Data to AgentResponse/ChatMessage/
string, so reaching default would indicate a contract drift; throw to make
the bug visible.
* Test renamed and broadened to verify the throw fires without expectedOutput.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com >
---------
Co-authored-by: alliscode <25218250+alliscode@users.noreply.github.com >
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com >
2026-05-13 19:03:27 +00:00
9b9604ce18
fix: avoid mutating handoff message roles ( #5808 )
...
Co-authored-by: Jacob Alber <jaalber@microsoft.com >
2026-05-13 18:52:19 +00:00
818ae65b77
fix: declare Magentic protocol messages ( #5778 )
...
Co-authored-by: Jacob Alber <jaalber@microsoft.com >
2026-05-12 15:59:15 +00:00
Jacob AlberandGitHub
