# Magentic E2E Test Plan

## Overview

This document outlines a comprehensive plan for adding end-to-end (E2E) tests for the Magentic orchestrator, similar to the existing `HandoffOrchestrationTests.cs` and smoke test patterns. These tests will drive a fully-built workflow through `MagenticWorkflowBuilder.Build()` and verify correct behavior at every logical fork in the orchestrator logic.

## Background

### Magentic Orchestrator Logic Flow

Based on the analysis of `MagenticOrchestrator.cs`, the orchestrator follows this decision tree:

```
TakeTurn (Initial)
  └─> UpdatePlanAndDelegateAsync
       ├─> Create/Update Plan via MagenticManager.UpdatePlanAsync
       ├─> Emit MagenticPlanCreatedEvent or MagenticReplannedEvent
       └─> If requirePlanSignoff: SubmitPlanReviewRequestAsync
           └─> ProcessPlanReviewAsync (on human response)
               ├─> If Approved: DelegateToTeamAsync
               └─> If Revise: Add review to chat, UpdatePlanAndDelegateAsync
       └─> If !requirePlanSignoff: DelegateToTeamAsync

DelegateToTeamAsync
  └─> RunCoordinationRoundAsync
       ├─> CHECK: Hit round limit? → Yield termination message, set IsTerminated
       ├─> CHECK: Hit reset limit? → Yield termination message, set IsTerminated
       ├─> Increment RoundCount
       ├─> UpdateProgressLedgerAsync (with retries)
       │    └─> On exception (non-cancellation): ResetAndReplanAsync
       ├─> Emit MagenticProgressLedgerUpdatedEvent
       ├─> CHECK: IsRequestSatisfied? → PrepareFinalAnswerAsync
       ├─> CHECK: IsInLoop OR !IsProgressBeingMade? → Increment StallCount
       │    └─> Else: Decrement StallCount (min 0)
       ├─> CHECK: IsStalled (StallCount >= MaxStallCount)? → ResetAndReplanAsync
       ├─> Validate NextSpeaker → Fallback to first participant if empty
       ├─> CHECK: Invalid NextSpeaker? → Warning + PrepareFinalAnswerAsync
       └─> Send instruction + TurnToken to next agent

PrepareFinalAnswerAsync
  └─> Get final answer from manager, yield output, set IsTerminated

ResetAndReplanAsync
  └─> Reset context, send ResetChatSignal, UpdatePlanAndDelegateAsync
```

## Test Categories

### 1. Happy Path Tests

| Test Name | Description | Expected Behavior |
|-----------|-------------|-------------------|
| `Task_Completes_When_RequestSatisfied` | Manager reports task satisfied on first coordination round | Workflow yields final answer and terminates |
| `Task_Delegates_To_Correct_Agent` | Manager selects specific agent as next speaker | Selected agent receives TurnToken |
| `Task_Completes_After_Multiple_Rounds` | Task requires multiple coordination rounds before completion | Each round delegates to specified agent, final answer produced when satisfied |
| `PlanSignoff_Disabled_Proceeds_Immediately` | With `requirePlanSignoff=false` | Workflow proceeds to team delegation without plan review request |

### 2. Plan Review Tests (Human-in-the-Loop)

| Test Name | Description | Expected Behavior |
|-----------|-------------|-------------------|
| `PlanReview_Approved_Proceeds` | Human approves initial plan | Workflow continues to DelegateToTeamAsync |
| `PlanReview_Revised_Triggers_Replan` | Human requests plan revision | Revision added to chat, plan updated, MagenticReplannedEvent emitted |
| `PlanReview_Multiple_Revisions` | Human revises multiple times | Each revision triggers replan until approved |
| `PlanReview_On_Stall_Replan` | Stall triggers replan with plan signoff | Plan review request sent with IsStalled=true |

### 3. Limit Enforcement Tests

| Test Name | Description | Expected Behavior |
|-----------|-------------|-------------------|
| `MaxRoundLimit_Terminates_Workflow` | RoundCount exceeds MaxRoundCount | Workflow terminates with "maximum round count limit" message |
| `MaxResetLimit_Terminates_Workflow` | ResetCount exceeds MaxResetCount | Workflow terminates with "maximum reset count limit" message |
| `MaxStallCount_Triggers_Reset` | StallCount reaches MaxStallCount | ResetAndReplanAsync called, ResetChatSignal sent |

### 4. Stall Detection Tests

| Test Name | Description | Expected Behavior |
|-----------|-------------|-------------------|
| `Stall_IsInLoop_Increments_StallCount` | ProgressLedger reports IsInLoop=true | StallCount incremented |
| `Stall_NoProgress_Increments_StallCount` | ProgressLedger reports IsProgressBeingMade=false | StallCount incremented |
| `Progress_Made_Decrements_StallCount` | ProgressLedger reports progress being made | StallCount decremented (min 0) |
| `Consecutive_Stalls_Trigger_Reset` | Multiple stalls in a row reach MaxStallCount | Reset and replan triggered |

### 5. Progress Ledger Validation Tests

| Test Name | Description | Expected Behavior |
|-----------|-------------|-------------------|
| `ProgressLedger_Retry_On_Parse_Failure` | First attempt fails JSON parsing | Retry up to MaxProgressLedgerRetryCount times |
| `ProgressLedger_Max_Retries_Triggers_Reset` | All retry attempts fail | ResetAndReplanAsync called |
| `ProgressLedger_Updated_Event_Emitted` | Valid progress ledger update | MagenticProgressLedgerUpdatedEvent emitted |

### 6. Next Speaker Validation Tests

| Test Name | Description | Expected Behavior |
|-----------|-------------|-------------------|
| `NextSpeaker_Empty_Falls_Back_To_First` | ProgressLedger returns empty next_speaker | First team member selected, warning emitted |
| `NextSpeaker_Invalid_Triggers_FinalAnswer` | next_speaker doesn't match any team member | Warning emitted, final answer prepared |
| `NextSpeaker_Valid_Delegates_Correctly` | Valid team member specified | TurnToken sent to correct executor |

### 7. Event Emission Tests

| Test Name | Description | Expected Behavior |
|-----------|-------------|-------------------|
| `Initial_Plan_Emits_PlanCreatedEvent` | First plan creation | MagenticPlanCreatedEvent emitted with full task ledger |
| `Replan_Emits_ReplannedEvent` | Plan update after initial | MagenticReplannedEvent emitted |
| `Warning_Events_On_Errors` | Various warning conditions | WorkflowWarningEvent emitted with appropriate message |

### 8. Checkpoint/Resume Tests

| Test Name | Description | Expected Behavior |
|-----------|-------------|-------------------|
| `Checkpoint_Saves_TaskContext` | Workflow checkpointed mid-execution | MagenticTaskContext state serialized |
| `Checkpoint_Resume_Continues_Correctly` | Resume from checkpoint | TaskContext restored, execution continues from saved state |
| `Checkpoint_Preserves_ProgressLedger` | Resume preserves ledger state | ProgressLedger state correctly restored |

### 9. Edge Cases

| Test Name | Description | Expected Behavior |
|-----------|-------------|-------------------|
| `Empty_Team_Handling` | No participants added | Appropriate error or fallback behavior |
| `Single_Agent_Team` | Only one participant | Workflow functions correctly with single agent |
| `Instruction_Message_Sent_When_Present` | ProgressLedger has instruction_or_question | Instruction added to chat history and sent |
| `Terminated_Context_Rejects_New_Messages` | After termination, new messages arrive | InvalidOperationException thrown |

## Implementation Approach

### Test Infrastructure

Following the patterns established in `HandoffOrchestrationTests.cs`:

1. **TestReplayAgent for Manager**: Configure to return specific plan/progress ledger responses
2. **TestEchoAgent for Participants**: Simple agents that echo or return controlled responses
3. **MockChatClient**: For fine-grained control of LLM responses
4. **WorkflowRunResult**: Capture updates, outputs, checkpoints, and pending requests
5. **RunWorkflowAsync helper**: Execute workflow and collect results

### Key Test Helpers Needed

```csharp
// Helper to create manager agent with specific responses
internal static TestReplayAgent CreateManagerAgent(
    List<List<ChatMessage>> planResponses,
    List<List<ChatMessage>> progressLedgerResponses,
    List<List<ChatMessage>> finalAnswerResponses);

// Helper to create progress ledger JSON responses
internal static ChatMessage CreateProgressLedgerResponse(
    bool isRequestSatisfied,
    bool isInLoop,
    bool isProgressBeingMade,
    string nextSpeaker,
    string instructionOrQuestion);

// Helper to build and run Magentic workflow
internal static Task<WorkflowRunResult> RunMagenticWorkflowAsync(
    AIAgent manager,
    List<AIAgent> team,
    List<ChatMessage> input,
    bool requirePlanSignoff = false,
    TaskLimits? limits = null,
    ExecutionEnvironment environment = ExecutionEnvironment.InProcess_Lockstep);
```

### Test File Structure

Create new test file: `MagenticOrchestrationTests.cs`

```
dotnet/tests/Microsoft.Agents.AI.Workflows.UnitTests/
├── MagenticOrchestrationTests.cs  (NEW - E2E tests)
├── MagenticOrchestratorTests.cs   (existing - protocol tests)
├── MagenticManagerTests.cs        (existing - manager unit tests)
├── MagenticProgressLedgerTests.cs (existing - ledger unit tests)
└── TestProgressLedgerState.cs     (existing - reuse for E2E)
```

## Test Execution Order

1. **Phase 1: Basic Flow Tests**
   - Happy path completion
   - Single agent delegation
   - Multiple rounds
   
2. **Phase 2: Plan Review Tests**
   - Approval flow
   - Revision flow
   
3. **Phase 3: Limit and Stall Tests**
   - Round limits
   - Reset limits
   - Stall detection and recovery
   
4. **Phase 4: Error Handling Tests**
   - Invalid next speaker
   - Progress ledger failures
   
5. **Phase 5: Checkpoint Tests**
   - Save and restore

## Success Criteria

- [ ] All logical forks in `MagenticOrchestrator` are covered by at least one test
- [ ] Tests use the same patterns as `HandoffOrchestrationTests`
- [ ] Tests run against fully-built workflows (not isolated components)
- [ ] Each test verifies specific event emissions and state changes
- [ ] Tests cover both `requirePlanSignoff=true` and `false` paths
- [ ] Checkpoint/resume functionality is verified

## Estimated Implementation

| Component | Estimated Tests | Complexity |
|-----------|-----------------|------------|
| Happy Path | 4 | Low |
| Plan Review | 4 | Medium |
| Limits | 3 | Low |
| Stall Detection | 4 | Medium |
| Progress Ledger | 3 | Medium |
| Next Speaker | 3 | Low |
| Events | 3 | Low |
| Checkpoints | 3 | High |
| Edge Cases | 4 | Medium |
| **Total** | **~31 tests** | |

## Dependencies

- Existing test helpers: `TestReplayAgent`, `TestEchoAgent`, `TestProgressLedgerState`
- Existing execution infrastructure: `RunWorkflowAsync`, `CheckpointManager`
- Extensions may be needed for Magentic-specific assertions