diff --git a/dotnet/tests/Microsoft.Agents.AI.Workflows.UnitTests/MagenticE2E_TestPlan.md b/dotnet/tests/Microsoft.Agents.AI.Workflows.UnitTests/MagenticE2E_TestPlan.md new file mode 100644 index 0000000000..b4370de6ed --- /dev/null +++ b/dotnet/tests/Microsoft.Agents.AI.Workflows.UnitTests/MagenticE2E_TestPlan.md @@ -0,0 +1,228 @@ +# Magentic E2E Test Plan + +## Overview + +This document outlines a comprehensive plan for adding end-to-end (E2E) tests for the Magentic orchestrator, similar to the existing `HandoffOrchestrationTests.cs` and smoke test patterns. These tests will drive a fully-built workflow through `MagenticWorkflowBuilder.Build()` and verify correct behavior at every logical fork in the orchestrator logic. + +## Background + +### Magentic Orchestrator Logic Flow + +Based on the analysis of `MagenticOrchestrator.cs`, the orchestrator follows this decision tree: + +``` +TakeTurn (Initial) + └─> UpdatePlanAndDelegateAsync + ├─> Create/Update Plan via MagenticManager.UpdatePlanAsync + ├─> Emit MagenticPlanCreatedEvent or MagenticReplannedEvent + └─> If requirePlanSignoff: SubmitPlanReviewRequestAsync + └─> ProcessPlanReviewAsync (on human response) + ├─> If Approved: DelegateToTeamAsync + └─> If Revise: Add review to chat, UpdatePlanAndDelegateAsync + └─> If !requirePlanSignoff: DelegateToTeamAsync + +DelegateToTeamAsync + └─> RunCoordinationRoundAsync + ├─> CHECK: Hit round limit? → Yield termination message, set IsTerminated + ├─> CHECK: Hit reset limit? → Yield termination message, set IsTerminated + ├─> Increment RoundCount + ├─> UpdateProgressLedgerAsync (with retries) + │ └─> On exception (non-cancellation): ResetAndReplanAsync + ├─> Emit MagenticProgressLedgerUpdatedEvent + ├─> CHECK: IsRequestSatisfied? → PrepareFinalAnswerAsync + ├─> CHECK: IsInLoop OR !IsProgressBeingMade? → Increment StallCount + │ └─> Else: Decrement StallCount (min 0) + ├─> CHECK: IsStalled (StallCount >= MaxStallCount)? → ResetAndReplanAsync + ├─> Validate NextSpeaker → Fallback to first participant if empty + ├─> CHECK: Invalid NextSpeaker? → Warning + PrepareFinalAnswerAsync + └─> Send instruction + TurnToken to next agent + +PrepareFinalAnswerAsync + └─> Get final answer from manager, yield output, set IsTerminated + +ResetAndReplanAsync + └─> Reset context, send ResetChatSignal, UpdatePlanAndDelegateAsync +``` + +## Test Categories + +### 1. Happy Path Tests + +| Test Name | Description | Expected Behavior | +|-----------|-------------|-------------------| +| `Task_Completes_When_RequestSatisfied` | Manager reports task satisfied on first coordination round | Workflow yields final answer and terminates | +| `Task_Delegates_To_Correct_Agent` | Manager selects specific agent as next speaker | Selected agent receives TurnToken | +| `Task_Completes_After_Multiple_Rounds` | Task requires multiple coordination rounds before completion | Each round delegates to specified agent, final answer produced when satisfied | +| `PlanSignoff_Disabled_Proceeds_Immediately` | With `requirePlanSignoff=false` | Workflow proceeds to team delegation without plan review request | + +### 2. Plan Review Tests (Human-in-the-Loop) + +| Test Name | Description | Expected Behavior | +|-----------|-------------|-------------------| +| `PlanReview_Approved_Proceeds` | Human approves initial plan | Workflow continues to DelegateToTeamAsync | +| `PlanReview_Revised_Triggers_Replan` | Human requests plan revision | Revision added to chat, plan updated, MagenticReplannedEvent emitted | +| `PlanReview_Multiple_Revisions` | Human revises multiple times | Each revision triggers replan until approved | +| `PlanReview_On_Stall_Replan` | Stall triggers replan with plan signoff | Plan review request sent with IsStalled=true | + +### 3. Limit Enforcement Tests + +| Test Name | Description | Expected Behavior | +|-----------|-------------|-------------------| +| `MaxRoundLimit_Terminates_Workflow` | RoundCount exceeds MaxRoundCount | Workflow terminates with "maximum round count limit" message | +| `MaxResetLimit_Terminates_Workflow` | ResetCount exceeds MaxResetCount | Workflow terminates with "maximum reset count limit" message | +| `MaxStallCount_Triggers_Reset` | StallCount reaches MaxStallCount | ResetAndReplanAsync called, ResetChatSignal sent | + +### 4. Stall Detection Tests + +| Test Name | Description | Expected Behavior | +|-----------|-------------|-------------------| +| `Stall_IsInLoop_Increments_StallCount` | ProgressLedger reports IsInLoop=true | StallCount incremented | +| `Stall_NoProgress_Increments_StallCount` | ProgressLedger reports IsProgressBeingMade=false | StallCount incremented | +| `Progress_Made_Decrements_StallCount` | ProgressLedger reports progress being made | StallCount decremented (min 0) | +| `Consecutive_Stalls_Trigger_Reset` | Multiple stalls in a row reach MaxStallCount | Reset and replan triggered | + +### 5. Progress Ledger Validation Tests + +| Test Name | Description | Expected Behavior | +|-----------|-------------|-------------------| +| `ProgressLedger_Retry_On_Parse_Failure` | First attempt fails JSON parsing | Retry up to MaxProgressLedgerRetryCount times | +| `ProgressLedger_Max_Retries_Triggers_Reset` | All retry attempts fail | ResetAndReplanAsync called | +| `ProgressLedger_Updated_Event_Emitted` | Valid progress ledger update | MagenticProgressLedgerUpdatedEvent emitted | + +### 6. Next Speaker Validation Tests + +| Test Name | Description | Expected Behavior | +|-----------|-------------|-------------------| +| `NextSpeaker_Empty_Falls_Back_To_First` | ProgressLedger returns empty next_speaker | First team member selected, warning emitted | +| `NextSpeaker_Invalid_Triggers_FinalAnswer` | next_speaker doesn't match any team member | Warning emitted, final answer prepared | +| `NextSpeaker_Valid_Delegates_Correctly` | Valid team member specified | TurnToken sent to correct executor | + +### 7. Event Emission Tests + +| Test Name | Description | Expected Behavior | +|-----------|-------------|-------------------| +| `Initial_Plan_Emits_PlanCreatedEvent` | First plan creation | MagenticPlanCreatedEvent emitted with full task ledger | +| `Replan_Emits_ReplannedEvent` | Plan update after initial | MagenticReplannedEvent emitted | +| `Warning_Events_On_Errors` | Various warning conditions | WorkflowWarningEvent emitted with appropriate message | + +### 8. Checkpoint/Resume Tests + +| Test Name | Description | Expected Behavior | +|-----------|-------------|-------------------| +| `Checkpoint_Saves_TaskContext` | Workflow checkpointed mid-execution | MagenticTaskContext state serialized | +| `Checkpoint_Resume_Continues_Correctly` | Resume from checkpoint | TaskContext restored, execution continues from saved state | +| `Checkpoint_Preserves_ProgressLedger` | Resume preserves ledger state | ProgressLedger state correctly restored | + +### 9. Edge Cases + +| Test Name | Description | Expected Behavior | +|-----------|-------------|-------------------| +| `Empty_Team_Handling` | No participants added | Appropriate error or fallback behavior | +| `Single_Agent_Team` | Only one participant | Workflow functions correctly with single agent | +| `Instruction_Message_Sent_When_Present` | ProgressLedger has instruction_or_question | Instruction added to chat history and sent | +| `Terminated_Context_Rejects_New_Messages` | After termination, new messages arrive | InvalidOperationException thrown | + +## Implementation Approach + +### Test Infrastructure + +Following the patterns established in `HandoffOrchestrationTests.cs`: + +1. **TestReplayAgent for Manager**: Configure to return specific plan/progress ledger responses +2. **TestEchoAgent for Participants**: Simple agents that echo or return controlled responses +3. **MockChatClient**: For fine-grained control of LLM responses +4. **WorkflowRunResult**: Capture updates, outputs, checkpoints, and pending requests +5. **RunWorkflowAsync helper**: Execute workflow and collect results + +### Key Test Helpers Needed + +```csharp +// Helper to create manager agent with specific responses +internal static TestReplayAgent CreateManagerAgent( + List> planResponses, + List> progressLedgerResponses, + List> finalAnswerResponses); + +// Helper to create progress ledger JSON responses +internal static ChatMessage CreateProgressLedgerResponse( + bool isRequestSatisfied, + bool isInLoop, + bool isProgressBeingMade, + string nextSpeaker, + string instructionOrQuestion); + +// Helper to build and run Magentic workflow +internal static Task RunMagenticWorkflowAsync( + AIAgent manager, + List team, + List input, + bool requirePlanSignoff = false, + TaskLimits? limits = null, + ExecutionEnvironment environment = ExecutionEnvironment.InProcess_Lockstep); +``` + +### Test File Structure + +Create new test file: `MagenticOrchestrationTests.cs` + +``` +dotnet/tests/Microsoft.Agents.AI.Workflows.UnitTests/ +├── MagenticOrchestrationTests.cs (NEW - E2E tests) +├── MagenticOrchestratorTests.cs (existing - protocol tests) +├── MagenticManagerTests.cs (existing - manager unit tests) +├── MagenticProgressLedgerTests.cs (existing - ledger unit tests) +└── TestProgressLedgerState.cs (existing - reuse for E2E) +``` + +## Test Execution Order + +1. **Phase 1: Basic Flow Tests** + - Happy path completion + - Single agent delegation + - Multiple rounds + +2. **Phase 2: Plan Review Tests** + - Approval flow + - Revision flow + +3. **Phase 3: Limit and Stall Tests** + - Round limits + - Reset limits + - Stall detection and recovery + +4. **Phase 4: Error Handling Tests** + - Invalid next speaker + - Progress ledger failures + +5. **Phase 5: Checkpoint Tests** + - Save and restore + +## Success Criteria + +- [ ] All logical forks in `MagenticOrchestrator` are covered by at least one test +- [ ] Tests use the same patterns as `HandoffOrchestrationTests` +- [ ] Tests run against fully-built workflows (not isolated components) +- [ ] Each test verifies specific event emissions and state changes +- [ ] Tests cover both `requirePlanSignoff=true` and `false` paths +- [ ] Checkpoint/resume functionality is verified + +## Estimated Implementation + +| Component | Estimated Tests | Complexity | +|-----------|-----------------|------------| +| Happy Path | 4 | Low | +| Plan Review | 4 | Medium | +| Limits | 3 | Low | +| Stall Detection | 4 | Medium | +| Progress Ledger | 3 | Medium | +| Next Speaker | 3 | Low | +| Events | 3 | Low | +| Checkpoints | 3 | High | +| Edge Cases | 4 | Medium | +| **Total** | **~31 tests** | | + +## Dependencies + +- Existing test helpers: `TestReplayAgent`, `TestEchoAgent`, `TestProgressLedgerState` +- Existing execution infrastructure: `RunWorkflowAsync`, `CheckpointManager` +- Extensions may be needed for Magentic-specific assertions