.NET: Add a verify-samples tool and skill (#5005)

* Add a verify-samples tool and skill

* Address PR comments

* Move verify-samples to eng folder and improve definitions

dotnet/.github/skills/verify-samples-tool/SKILL.md

@@ -0,0 +1,213 @@
+---
+name: verify-samples-tool
+description: How to use the verify-samples tool to run, verify, and manage sample definitions in the Agent Framework repository. Use this when adding, updating, or running sample verification.
+---
+
+# verify-samples Tool
+
+The `verify-samples` project (`dotnet/eng/verify-samples/`) is an automated tool that runs sample projects and verifies their output using deterministic checks and AI-powered verification.
+
+## Running verify-samples
+
+```bash
+cd dotnet
+
+# Run all samples across all categories
+dotnet run --project eng/verify-samples -- --log results.log --csv results.csv
+
+# Run a specific category
+dotnet run --project eng/verify-samples -- --category 02-agents --log results.log
+
+# Run specific samples by name
+dotnet run --project eng/verify-samples -- Agent_Step02_StructuredOutput Agent_Step09_AsFunctionTool
+
+# Control parallelism (default 8)
+dotnet run --project eng/verify-samples -- --parallel 8 --log results.log
+
+# Combine options
+dotnet run --project eng/verify-samples -- --category 03-workflows --parallel 4 --log results.log --csv results.csv
+```
+
+### Required Environment Variables
+
+The tool itself needs:
+- `AZURE_OPENAI_ENDPOINT` — for the AI verification agent
+- `AZURE_OPENAI_DEPLOYMENT_NAME` (optional, defaults to `gpt-5-mini`)
+
+Individual samples require their own env vars (e.g., `AZURE_AI_PROJECT_ENDPOINT`). The tool automatically checks and skips samples with missing env vars.
+
+### Output Files
+
+- `--log results.log` — detailed per-sample log with stdout/stderr, AI reasoning, and a summary
+- `--csv results.csv` — tabular summary with Sample, ProjectPath, Status, FailedChecks, and Failures columns
+
+## Sample Categories
+
+Definitions are in the `dotnet/eng/verify-samples/` directory:
+
+| Category | Config File | Registered Key |
+|----------|-------------|----------------|
+| 01-get-started | `GetStartedSamples.cs` | `01-get-started` |
+| 02-agents | `AgentsSamples.cs` | `02-agents` |
+| 03-workflows | `WorkflowSamples.cs` | `03-workflows` |
+
+Categories are registered in `VerifyOptions.cs` in the `s_sampleSets` dictionary.
+
+## SampleDefinition Properties
+
+Each sample is defined as a `SampleDefinition` in the appropriate config file. Key properties:
+
+```csharp
+new SampleDefinition
+{
+    // Required: Display name for the sample
+    Name = "Agent_Step02_StructuredOutput",
+
+    // Required: Relative path from dotnet/ to the sample project directory
+    ProjectPath = "samples/02-agents/Agents/Agent_Step02_StructuredOutput",
+
+    // Environment variables the sample requires (throws if missing)
+    RequiredEnvironmentVariables = ["AZURE_OPENAI_ENDPOINT"],
+
+    // Environment variables with defaults that would prompt on console if unset
+    OptionalEnvironmentVariables = ["AZURE_OPENAI_DEPLOYMENT_NAME"],
+
+    // Skip this sample with a reason (for structural issues only)
+    SkipReason = null, // or "Requires external service X."
+
+    // Deterministic checks: substrings that must appear in stdout
+    MustContain = ["=== Section Header ==="],
+
+    // Substrings that must NOT appear in stdout
+    MustNotContain = [],
+
+    // If true, only MustContain checks are used (no AI verification)
+    IsDeterministic = false,
+
+    // AI verification: natural-language descriptions of expected output
+    // Each entry describes one aspect to verify independently
+    ExpectedOutputDescription =
+    [
+        "The output should show structured person information with Name, Age, and Occupation fields.",
+        "The output should not contain error messages or stack traces.",
+    ],
+
+    // Stdin inputs to feed to the sample (for interactive samples)
+    Inputs = ["Y", "Y", "Y"],
+
+    // Delay between stdin inputs in ms (default 2000, increase for LLM calls between inputs)
+    InputDelayMs = 3000,
+}
+```
+
+## How to Add a New Sample Definition
+
+1. **Check the sample's Program.cs** to understand:
+   - What environment variables it reads (look for `GetEnvironmentVariable`)
+   - Whether it needs stdin input (look for `Console.ReadLine`, `Application.GetInput`)
+   - Whether it has an external loop (look for `EXIT` patterns in YAML workflows)
+   - What output it produces (section headers, markers, expected behavior)
+   - Whether it exits on its own or runs as a server
+
+2. **Choose the right verification strategy:**
+   - **Deterministic** (`IsDeterministic = true`): Use `MustContain` for samples with fixed output strings. No AI verification.
+   - **AI-verified** (default): Use `ExpectedOutputDescription` with semantic descriptions. Write expectations that are flexible enough for non-deterministic LLM output.
+   - **Both**: Use `MustContain` for fixed markers AND `ExpectedOutputDescription` for LLM-generated content.
+
+3. **Set `SkipReason` only for structural issues:**
+   - Web servers that don't exit
+   - Multi-process client/server architectures
+   - Samples requiring external infrastructure (MCP servers you can't reach, Docker, etc.)
+   - Do NOT skip for missing env vars — the tool checks those dynamically.
+
+4. **For interactive samples, provide `Inputs`:**
+   - Samples using `Application.GetInput(args)` need one initial input
+   - Samples with `Console.ReadLine()` approval loops need `"Y"` inputs
+   - YAML workflows with `externalLoop` need `"EXIT"` as the last input
+   - Set `InputDelayMs` to 3000-8000ms for samples with LLM calls between inputs
+
+5. **Add the definition** to the appropriate config file (e.g., `AgentsSamples.cs`) in the `All` list.
+
+6. **Register new categories** (if needed) in `VerifyOptions.cs` `s_sampleSets` dictionary.
+
+### Writing Good ExpectedOutputDescription
+
+- Write descriptions that are **semantically flexible** — LLM output varies between runs
+- Each array entry should describe **one independent aspect** to verify
+- Always include `"The output should not contain error messages or stack traces."` as the last entry
+- Avoid exact wording expectations — use "should mention", "should contain information about", "should show"
+- Bad: `"The output should say 'The weather in Amsterdam is cloudy with a high of 15°C'"`
+- Good: `"The output should contain weather information about Amsterdam mentioning cloudy weather with a high of 15°C."`
+
+### Example: Simple LLM Sample
+
+```csharp
+new SampleDefinition
+{
+    Name = "Agent_With_AzureOpenAIChatCompletion",
+    ProjectPath = "samples/02-agents/AgentProviders/Agent_With_AzureOpenAIChatCompletion",
+    RequiredEnvironmentVariables = ["AZURE_OPENAI_ENDPOINT"],
+    OptionalEnvironmentVariables = ["AZURE_OPENAI_DEPLOYMENT_NAME"],
+    ExpectedOutputDescription =
+    [
+        "The output should contain a joke about a pirate.",
+        "The output should not contain error messages or stack traces.",
+    ],
+},
+```
+
+### Example: Deterministic Sample
+
+```csharp
+new SampleDefinition
+{
+    Name = "Workflow_Declarative_GenerateCode",
+    ProjectPath = "samples/03-workflows/Declarative/GenerateCode",
+    IsDeterministic = true,
+    MustContain = ["WORKFLOW: Parsing", "WORKFLOW: Defined"],
+    ExpectedOutputDescription = ["The output should show a YAML workflow being parsed and C# code being generated from it."],
+},
+```
+
+### Example: Interactive Sample with Approval Loop
+
+```csharp
+new SampleDefinition
+{
+    Name = "FoundryAgent_Hosted_MCP",
+    ProjectPath = "samples/02-agents/ModelContextProtocol/FoundryAgent_Hosted_MCP",
+    RequiredEnvironmentVariables = ["AZURE_AI_PROJECT_ENDPOINT"],
+    OptionalEnvironmentVariables = ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
+    Inputs = ["Y", "Y", "Y", "Y", "Y"],
+    InputDelayMs = 5000,
+    ExpectedOutputDescription = ["The output should show an agent using the Microsoft Learn MCP tool with approval prompts."],
+},
+```
+
+### Example: Declarative Workflow with External Loop
+
+```csharp
+new SampleDefinition
+{
+    Name = "Workflow_Declarative_FunctionTools",
+    ProjectPath = "samples/03-workflows/Declarative/FunctionTools",
+    RequiredEnvironmentVariables = ["AZURE_AI_PROJECT_ENDPOINT"],
+    OptionalEnvironmentVariables = ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
+    Inputs = ["What are today's specials?", "EXIT"],
+    InputDelayMs = 8000,
+    ExpectedOutputDescription = ["The output should show a workflow calling function tools to answer a question about restaurant specials."],
+},
+```
+
+### Example: Skipped Sample
+
+```csharp
+new SampleDefinition
+{
+    Name = "Agent_MCP_Server",
+    ProjectPath = "samples/02-agents/ModelContextProtocol/Agent_MCP_Server",
+    RequiredEnvironmentVariables = ["AZURE_OPENAI_ENDPOINT"],
+    OptionalEnvironmentVariables = ["AZURE_OPENAI_DEPLOYMENT_NAME"],
+    SkipReason = "Runs as an MCP stdio server that does not exit on its own.",
+},
+```

dotnet/agent-framework-dotnet.slnx

@@ -7,6 +7,7 @@
   <Folder Name="/Samples/">
     <File Path="samples/AGENTS.md" />
     <File Path="samples/README.md" />
+    <Project Path="eng/verify-samples/verify-samples.csproj" />
   </Folder>
   <Folder Name="/Samples/01-get-started/">
     <Project Path="samples/01-get-started/01_hello_agent/01_hello_agent.csproj" />

dotnet/eng/verify-samples/ConsoleReporter.cs

@@ -0,0 +1,95 @@
+// Copyright (c) Microsoft. All rights reserved.
+
+namespace VerifySamples;
+
+/// <summary>
+/// Thread-safe console output with sample-name prefixes and colored status.
+/// </summary>
+internal sealed class ConsoleReporter
+{
+    private readonly object _lock = new();
+
+    /// <summary>
+    /// Writes a complete prefixed line atomically to the console.
+    /// </summary>
+    public void WriteLineWithPrefix(string sampleName, string message, ConsoleColor? color = null)
+    {
+        lock (this._lock)
+        {
+            Console.ForegroundColor = ConsoleColor.Cyan;
+            Console.Write($"[{sampleName}] ");
+            if (color.HasValue)
+            {
+                Console.ForegroundColor = color.Value;
+            }
+            else
+            {
+                Console.ResetColor();
+            }
+
+            Console.WriteLine(message);
+            Console.ResetColor();
+        }
+    }
+
+    /// <summary>
+    /// Prints the final summary table and elapsed time to the console.
+    /// </summary>
+    public void PrintSummary(
+        IReadOnlyList<VerificationResult> orderedResults,
+        IReadOnlyList<(string Name, string Reason)> skipped,
+        TimeSpan elapsed)
+    {
+        var passCount = orderedResults.Count(r => r.Passed);
+        var failCount = orderedResults.Count(r => !r.Passed);
+
+        Console.WriteLine();
+        Console.WriteLine(new string('─', 60));
+        Console.ForegroundColor = ConsoleColor.White;
+        Console.WriteLine("SUMMARY");
+        Console.ResetColor();
+
+        foreach (var result in orderedResults)
+        {
+            Console.ForegroundColor = result.Passed ? ConsoleColor.Green : ConsoleColor.Red;
+            Console.Write(result.Passed ? "  ✓ " : "  ✗ ");
+            Console.ResetColor();
+            Console.WriteLine($"{result.SampleName}: {result.Summary}");
+        }
+
+        foreach (var (name, reason) in skipped)
+        {
+            Console.ForegroundColor = ConsoleColor.Yellow;
+            Console.Write("  ○ ");
+            Console.ResetColor();
+            Console.WriteLine($"{name}: Skipped — {reason}");
+        }
+
+        Console.WriteLine();
+        Console.Write("Results: ");
+        Console.ForegroundColor = ConsoleColor.Green;
+        Console.Write($"{passCount} passed");
+        Console.ResetColor();
+
+        if (failCount > 0)
+        {
+            Console.Write(", ");
+            Console.ForegroundColor = ConsoleColor.Red;
+            Console.Write($"{failCount} failed");
+            Console.ResetColor();
+        }
+
+        if (skipped.Count > 0)
+        {
+            Console.Write(", ");
+            Console.ForegroundColor = ConsoleColor.Yellow;
+            Console.Write($"{skipped.Count} skipped");
+            Console.ResetColor();
+        }
+
+        Console.WriteLine();
+        Console.ForegroundColor = ConsoleColor.DarkGray;
+        Console.WriteLine($"Elapsed: {elapsed.Hours:D2}:{elapsed.Minutes:D2}:{elapsed.Seconds:D2}");
+        Console.ResetColor();
+    }
+}

dotnet/eng/verify-samples/CsvResultWriter.cs

@@ -0,0 +1,56 @@
+// Copyright (c) Microsoft. All rights reserved.
+
+using System.Text;
+
+namespace VerifySamples;
+
+/// <summary>
+/// Writes a CSV summary of sample verification results.
+/// </summary>
+internal static class CsvResultWriter
+{
+    /// <summary>
+    /// Writes the results to a CSV file at the specified path.
+    /// </summary>
+    public static async Task WriteAsync(
+        string path,
+        IReadOnlyList<VerificationResult> orderedResults,
+        IReadOnlyList<(string Name, string Reason)> skipped,
+        IReadOnlyList<SampleDefinition> samples)
+    {
+        var pathLookup = samples.ToDictionary(s => s.Name, s => s.ProjectPath);
+
+        var sb = new StringBuilder();
+        sb.AppendLine("Sample,ProjectPath,Status,FailedChecks,Failures");
+
+        foreach (var result in orderedResults)
+        {
+            var status = result.Passed ? "PASSED" : "FAILED";
+            var failedChecks = result.Failures.Count;
+            var failures = string.Join("; ", result.Failures);
+            pathLookup.TryGetValue(result.SampleName, out var projectPath);
+            sb.AppendLine($"{CsvEscape(result.SampleName)},{CsvEscape(projectPath ?? "")},{status},{failedChecks},{CsvEscape(failures)}");
+        }
+
+        foreach (var (name, reason) in skipped)
+        {
+            pathLookup.TryGetValue(name, out var projectPath);
+            sb.AppendLine($"{CsvEscape(name)},{CsvEscape(projectPath ?? "")},SKIPPED,0,{CsvEscape(reason)}");
+        }
+
+        await File.WriteAllTextAsync(path, sb.ToString());
+    }
+
+    /// <summary>
+    /// Escapes a value for CSV: wraps in quotes if it contains commas, quotes, or newlines.
+    /// </summary>
+    private static string CsvEscape(string value)
+    {
+        if (value.Contains('"') || value.Contains(',') || value.Contains('\n') || value.Contains('\r'))
+        {
+            return $"\"{value.Replace("\"", "\"\"")}\"";
+        }
+
+        return value;
+    }
+}

dotnet/eng/verify-samples/GetStartedSamples.cs

@@ -0,0 +1,105 @@
+// Copyright (c) Microsoft. All rights reserved.
+
+namespace VerifySamples;
+
+/// <summary>
+/// Defines the expected behavior for each sample in 01-get-started.
+/// </summary>
+internal static class GetStartedSamples
+{
+    public static IReadOnlyList<SampleDefinition> All { get; } =
+    [
+        new SampleDefinition
+        {
+            Name = "05_first_workflow",
+            ProjectPath = "samples/01-get-started/05_first_workflow",
+            RequiredEnvironmentVariables = [],
+            IsDeterministic = true,
+            MustContain =
+            [
+                "UppercaseExecutor: HELLO, WORLD!",
+                "ReverseTextExecutor: !DLROW ,OLLEH",
+            ],
+        },
+
+        new SampleDefinition
+        {
+            Name = "01_hello_agent",
+            ProjectPath = "samples/01-get-started/01_hello_agent",
+            RequiredEnvironmentVariables = ["AZURE_OPENAI_ENDPOINT"],
+            OptionalEnvironmentVariables = ["AZURE_OPENAI_DEPLOYMENT_NAME"],
+            ExpectedOutputDescription =
+            [
+                "The output should contain a joke about a pirate.",
+                "There should be two separate joke responses — one from a non-streaming call and one from a streaming call.",
+                "The output should not contain error messages or stack traces.",
+            ],
+        },
+
+        new SampleDefinition
+        {
+            Name = "02_add_tools",
+            ProjectPath = "samples/01-get-started/02_add_tools",
+            RequiredEnvironmentVariables = ["AZURE_OPENAI_ENDPOINT"],
+            OptionalEnvironmentVariables = ["AZURE_OPENAI_DEPLOYMENT_NAME"],
+            MustContain = [],
+            ExpectedOutputDescription =
+            [
+                "The output should contain information about the weather in Amsterdam.",
+                "The response should mention that it is cloudy with a high of 15°C (or equivalent), since this comes from a tool that returns a canned response.",
+                "There should be two responses — one from a non-streaming call and one from a streaming call.",
+                "The output should not contain error messages or stack traces.",
+            ],
+        },
+
+        new SampleDefinition
+        {
+            Name = "03_multi_turn",
+            ProjectPath = "samples/01-get-started/03_multi_turn",
+            RequiredEnvironmentVariables = ["AZURE_OPENAI_ENDPOINT"],
+            OptionalEnvironmentVariables = ["AZURE_OPENAI_DEPLOYMENT_NAME"],
+            ExpectedOutputDescription =
+            [
+                "The output should contain a joke about a pirate.",
+                "After the initial joke, there should be a modified version that includes emojis and is told in the voice of a pirate's parrot.",
+                "The pattern repeats: first a non-streaming pirate joke + parrot version, then a streaming pirate joke + parrot version.",
+                "The output should not contain error messages or stack traces.",
+            ],
+        },
+
+        new SampleDefinition
+        {
+            Name = "04_memory",
+            ProjectPath = "samples/01-get-started/04_memory",
+            RequiredEnvironmentVariables = ["AZURE_OPENAI_ENDPOINT"],
+            OptionalEnvironmentVariables = ["AZURE_OPENAI_DEPLOYMENT_NAME"],
+            MustContain =
+            [
+                ">> Use session with blank memory",
+                ">> Use deserialized session with previously created memories",
+                ">> Read memories using memory component",
+                "MEMORY - User Name:",
+                "MEMORY - User Age:",
+                ">> Use new session with previously created memories",
+            ],
+            ExpectedOutputDescription =
+            [
+                "In the 'Use session with blank memory' section, the agent should respond to the user's messages. It may ask for the user's name or age if not yet known.",
+                "In the 'Use deserialized session with previously created memories' section, the agent should correctly recall that the user's name is Ruaidhrí and age is 20.",
+                "The 'MEMORY - User Name:' line should show 'Ruaidhrí' (or a close transliteration).",
+                "The 'MEMORY - User Age:' line should show '20'.",
+                "In the 'Use new session with previously created memories' section, the agent should know the user's name and age from the transferred memory.",
+                "The output should not contain error messages or stack traces.",
+            ],
+        },
+
+        new SampleDefinition
+        {
+            Name = "06_host_your_agent",
+            ProjectPath = "samples/01-get-started/06_host_your_agent",
+            RequiredEnvironmentVariables = ["AZURE_OPENAI_ENDPOINT"],
+            OptionalEnvironmentVariables = ["AZURE_OPENAI_DEPLOYMENT_NAME"],
+            SkipReason = "Requires Azure Functions Core Tools runtime and starts a web server.",
+        },
+    ];
+}

dotnet/eng/verify-samples/LogFileWriter.cs

@@ -0,0 +1,153 @@
+// Copyright (c) Microsoft. All rights reserved.
+
+using System.Text;
+
+namespace VerifySamples;
+
+/// <summary>
+/// Incrementally writes a sequential (non-interleaved) log file, appending after each sample completes.
+/// Thread-safe: multiple parallel tasks may call write methods concurrently.
+/// </summary>
+internal sealed class LogFileWriter : IDisposable
+{
+    private readonly string _path;
+    private readonly SemaphoreSlim _writeLock = new(1, 1);
+
+    public LogFileWriter(string path)
+    {
+        this._path = path;
+    }
+
+    /// <inheritdoc />
+    public void Dispose()
+    {
+        this._writeLock.Dispose();
+    }
+
+    /// <summary>
+    /// Writes the log file header. Call once at the start of the run.
+    /// </summary>
+    public async Task WriteHeaderAsync()
+    {
+        var sb = new StringBuilder();
+        sb.AppendLine($"Sample Verification Log — {DateTime.UtcNow:yyyy-MM-dd HH:mm:ss} UTC");
+        sb.AppendLine(new string('═', 72));
+        sb.AppendLine();
+
+        await File.WriteAllTextAsync(this._path, sb.ToString());
+    }
+
+    /// <summary>
+    /// Appends a skipped-sample entry to the log file.
+    /// </summary>
+    public async Task WriteSkippedAsync(string name, string reason)
+    {
+        var sb = new StringBuilder();
+        sb.AppendLine($"── {name} ──");
+        sb.AppendLine($"Status: SKIPPED — {reason}");
+        sb.AppendLine();
+
+        await this.AppendAsync(sb.ToString());
+    }
+
+    /// <summary>
+    /// Appends a completed sample's full output section to the log file.
+    /// </summary>
+    public async Task WriteSampleResultAsync(VerificationResult result)
+    {
+        var sb = new StringBuilder();
+        sb.AppendLine(new string('─', 72));
+        sb.AppendLine($"── {result.SampleName} ──");
+        sb.AppendLine($"Status: {(result.Passed ? "PASSED" : "FAILED")}");
+        sb.AppendLine();
+
+        foreach (var line in result.LogLines)
+        {
+            sb.AppendLine(line);
+        }
+
+        sb.AppendLine();
+
+        if (!string.IsNullOrWhiteSpace(result.Stdout))
+        {
+            sb.AppendLine("--- stdout ---");
+            sb.AppendLine(result.Stdout.TrimEnd());
+            sb.AppendLine("--- end stdout ---");
+            sb.AppendLine();
+        }
+
+        if (!string.IsNullOrWhiteSpace(result.Stderr))
+        {
+            sb.AppendLine("--- stderr ---");
+            sb.AppendLine(result.Stderr.TrimEnd());
+            sb.AppendLine("--- end stderr ---");
+            sb.AppendLine();
+        }
+
+        if (result.Failures.Count > 0)
+        {
+            sb.AppendLine("Failures:");
+            foreach (var failure in result.Failures)
+            {
+                sb.AppendLine($"  ✗ {failure}");
+            }
+
+            sb.AppendLine();
+        }
+
+        if (result.AIReasoning is not null)
+        {
+            sb.AppendLine("AI Reasoning:");
+            sb.AppendLine(result.AIReasoning);
+            sb.AppendLine();
+        }
+
+        await this.AppendAsync(sb.ToString());
+    }
+
+    /// <summary>
+    /// Appends the final summary section and elapsed time to the log file.
+    /// </summary>
+    public async Task WriteSummaryAsync(
+        IReadOnlyList<VerificationResult> orderedResults,
+        IReadOnlyList<(string Name, string Reason)> skipped,
+        TimeSpan elapsed)
+    {
+        var passCount = orderedResults.Count(r => r.Passed);
+        var failCount = orderedResults.Count(r => !r.Passed);
+
+        var sb = new StringBuilder();
+        sb.AppendLine(new string('═', 72));
+        sb.AppendLine("SUMMARY");
+        sb.AppendLine();
+
+        foreach (var result in orderedResults)
+        {
+            sb.AppendLine($"  {(result.Passed ? "✓" : "✗")} {result.SampleName}: {result.Summary}");
+        }
+
+        foreach (var (name, reason) in skipped)
+        {
+            sb.AppendLine($"  ○ {name}: Skipped — {reason}");
+        }
+
+        sb.AppendLine();
+        sb.AppendLine($"Results: {passCount} passed{(failCount > 0 ? $", {failCount} failed" : "")}{(skipped.Count > 0 ? $", {skipped.Count} skipped" : "")}");
+        sb.AppendLine($"Elapsed: {elapsed.Hours:D2}:{elapsed.Minutes:D2}:{elapsed.Seconds:D2}");
+
+        await this.AppendAsync(sb.ToString());
+    }
+
+    private async Task AppendAsync(string text)
+    {
+        await this._writeLock.WaitAsync();
+        try
+        {
+            await File.AppendAllTextAsync(this._path, text);
+        }
+        finally
+        {
+            this._writeLock.Release();
+        }
+    }
+}

dotnet/eng/verify-samples/Program.cs

@@ -0,0 +1,98 @@
+// Copyright (c) Microsoft. All rights reserved.
+
+// This tool runs the 01-get-started, 02-agents, and 03-workflows samples and verifies their output.
+// Deterministic samples are verified with exact string matching.
+// Non-deterministic (LLM) samples are verified using an agent-framework agent.
+//
+// Usage:
+//   dotnet run                                          # Run all samples
+//   dotnet run -- 01_hello_agent 05_first_workflow      # Run specific samples by name
+//   dotnet run -- --category 01-get-started             # Run the 01-get-started category
+//   dotnet run -- --category 02-agents                  # Run the 02-agents category
+//   dotnet run -- --category 03-workflows               # Run the 03-workflows category
+//   dotnet run -- --parallel 16                         # Run up to 16 samples concurrently
+//   dotnet run -- --log results.log                     # Write sequential log to file
+//   dotnet run -- --csv results.csv                     # Write CSV summary to file
+//
+// Required environment variables (for AI-powered samples):
+//   AZURE_OPENAI_ENDPOINT
+//   AZURE_OPENAI_DEPLOYMENT_NAME (optional, defaults to gpt-5-mini)
+
+using System.Diagnostics;
+using Azure.AI.OpenAI;
+using Azure.Identity;
+using VerifySamples;
+
+var options = VerifyOptions.Parse(args);
+if (options is null)
+{
+    return 1;
+}
+
+var stopwatch = Stopwatch.StartNew();
+
+// Resolve the dotnet/ root directory (verify-samples is at dotnet/eng/verify-samples/)
+var dotnetRoot = Path.GetFullPath(Path.Combine(AppContext.BaseDirectory, "..", "..", "..", "..", ".."));
+if (!File.Exists(Path.Combine(dotnetRoot, "agent-framework-dotnet.slnx")))
+{
+    dotnetRoot = Path.GetFullPath(Path.Combine(Directory.GetCurrentDirectory(), "..", ".."));
+}
+
+// Set up the AI verifier
+var endpoint = Environment.GetEnvironmentVariable("AZURE_OPENAI_ENDPOINT");
+var deploymentName = Environment.GetEnvironmentVariable("AZURE_OPENAI_DEPLOYMENT_NAME") ?? "gpt-5-mini";
+
+OpenAI.Chat.ChatClient? chatClient = null;
+if (!string.IsNullOrEmpty(endpoint))
+{
+    chatClient = new AzureOpenAIClient(new Uri(endpoint), new DefaultAzureCredential())
+        .GetChatClient(deploymentName);
+}
+
+// Set up optional log file writer
+LogFileWriter? logWriter = null;
+if (options.LogFilePath is not null)
+{
+    logWriter = new LogFileWriter(options.LogFilePath);
+    await logWriter.WriteHeaderAsync();
+}
+
+try
+{
+    // Run all samples
+    var reporter = new ConsoleReporter();
+    var verifier = new SampleVerifier(chatClient);
+    var orchestrator = new VerificationOrchestrator(verifier, reporter, dotnetRoot, TimeSpan.FromMinutes(3), logWriter);
+
+    var run = await orchestrator.RunAllAsync(options.Samples, options.MaxParallelism);
+
+    stopwatch.Stop();
+
+    // Print summary
+    var orderedResults = run.SampleOrder
+        .Where(run.Results.ContainsKey)
+        .Select(name => run.Results[name])
+        .ToList();
+
+    reporter.PrintSummary(orderedResults, run.Skipped, stopwatch.Elapsed);
+
+    // Write log file summary
+    if (logWriter is not null)
+    {
+        await logWriter.WriteSummaryAsync(orderedResults, run.Skipped, stopwatch.Elapsed);
+        Console.WriteLine($"Log written to: {options.LogFilePath}");
+    }
+
+    // Write CSV summary
+    if (options.CsvFilePath is not null)
+    {
+        await CsvResultWriter.WriteAsync(options.CsvFilePath, orderedResults, run.Skipped, options.Samples);
+        Console.WriteLine($"CSV written to: {options.CsvFilePath}");
+    }
+
+    return orderedResults.Any(r => !r.Passed) ? 1 : 0;
+}
+finally
+{
+    logWriter?.Dispose();
+}

dotnet/eng/verify-samples/SampleDefinition.cs

@@ -0,0 +1,79 @@
+// Copyright (c) Microsoft. All rights reserved.
+
+namespace VerifySamples;
+
+/// <summary>
+/// Describes a sample to verify, including its expected output.
+/// </summary>
+internal sealed class SampleDefinition
+{
+    /// <summary>
+    /// Display name for the sample (e.g., "01_hello_agent").
+    /// </summary>
+    public required string Name { get; init; }
+
+    /// <summary>
+    /// Relative path from the dotnet/ directory to the sample project directory.
+    /// </summary>
+    public required string ProjectPath { get; init; }
+
+    /// <summary>
+    /// Environment variables that the sample requires for a meaningful run.
+    /// The runner checks these before running and will skip the sample if any are unset,
+    /// recording a skip reason that indicates which required variables are missing.
+    /// </summary>
+    public string[] RequiredEnvironmentVariables { get; init; } = [];
+
+    /// <summary>
+    /// Environment variables that the sample can use but typically has fallbacks or defaults for.
+    /// If these are not set, the sample might prompt or behave interactively, which could cause
+    /// automated verification to hang. The runner checks these and skips the sample if they are unset
+    /// to avoid non-deterministic or blocking behavior in automated runs.
+    /// </summary>
+    public string[] OptionalEnvironmentVariables { get; init; } = [];
+
+    /// <summary>
+    /// If set, the sample is skipped with this reason.
+    /// Use only for structural reasons (e.g., web server, multi-process, needs external service).
+    /// Do NOT use for missing environment variables — those are checked dynamically.
+    /// </summary>
+    public string? SkipReason { get; init; }
+
+    /// <summary>
+    /// Substrings that must appear in stdout for the sample to pass.
+    /// Used for deterministic verification.
+    /// </summary>
+    public string[] MustContain { get; init; } = [];
+
+    /// <summary>
+    /// Substrings that must not appear in stdout for the sample to pass.
+    /// </summary>
+    public string[] MustNotContain { get; init; } = [];
+
+    /// <summary>
+    /// If true, <see cref="MustContain"/> entries cover the entire expected output —
+    /// no AI verification is needed.
+    /// </summary>
+    public bool IsDeterministic { get; init; }
+
+    /// <summary>
+    /// Natural-language description of what the sample output should look like.
+    /// Used by the AI verifier for non-deterministic samples.
+    /// Each entry describes one aspect of the expected output that should be verified.
+    /// </summary>
+    public string[] ExpectedOutputDescription { get; init; } = [];
+
+    /// <summary>
+    /// Sequence of stdin inputs to feed to the sample process.
+    /// Each entry is written as a line (followed by newline) to the process stdin.
+    /// A <c>null</c> entry inserts a delay without writing anything.
+    /// Inputs are sent with a short delay between each to allow the process to prompt.
+    /// </summary>
+    public string?[] Inputs { get; init; } = [];
+
+    /// <summary>
+    /// Delay in milliseconds between each input line. Default is 2000ms.
+    /// Increase for samples that need more time between prompts (e.g., LLM calls between inputs).
+    /// </summary>
+    public int InputDelayMs { get; init; } = 2000;
+}

dotnet/eng/verify-samples/SampleRunner.cs

@@ -0,0 +1,132 @@
+// Copyright (c) Microsoft. All rights reserved.
+
+using System.Diagnostics;
+
+namespace VerifySamples;
+
+/// <summary>
+/// Result of running a sample process.
+/// </summary>
+internal sealed record SampleRunResult(
+    string Stdout,
+    string Stderr,
+    int ExitCode,
+    TimeSpan Elapsed);
+
+/// <summary>
+/// Runs a sample project via <c>dotnet run</c> and captures its output.
+/// </summary>
+internal static class SampleRunner
+{
+    /// <summary>
+    /// Runs <c>dotnet run --framework net10.0</c> in the given project directory.
+    /// </summary>
+    public static Task<SampleRunResult> RunAsync(
+        string projectPath,
+        TimeSpan timeout,
+        CancellationToken cancellationToken = default)
+        => RunAsync(projectPath, "run --framework net10.0", timeout, inputs: null, inputDelayMs: 0, cancellationToken: cancellationToken);
+
+    /// <summary>
+    /// Runs <c>dotnet run --framework net10.0</c> with stdin inputs.
+    /// </summary>
+    public static Task<SampleRunResult> RunAsync(
+        string projectPath,
+        TimeSpan timeout,
+        string?[]? inputs,
+        int inputDelayMs = 2000,
+        CancellationToken cancellationToken = default)
+        => RunAsync(projectPath, "run --framework net10.0", timeout, inputs, inputDelayMs, cancellationToken);
+
+    /// <summary>
+    /// Runs an arbitrary <c>dotnet</c> command in the given working directory.
+    /// </summary>
+    public static async Task<SampleRunResult> RunAsync(
+        string workingDirectory,
+        string dotnetArgs,
+        TimeSpan timeout,
+        string?[]? inputs = null,
+        int inputDelayMs = 0,
+        CancellationToken cancellationToken = default)
+    {
+        var psi = new ProcessStartInfo
+        {
+            FileName = "dotnet",
+            Arguments = dotnetArgs,
+            WorkingDirectory = workingDirectory,
+            RedirectStandardOutput = true,
+            RedirectStandardError = true,
+            RedirectStandardInput = inputs is { Length: > 0 },
+            UseShellExecute = false,
+            CreateNoWindow = true,
+        };
+
+        var sw = Stopwatch.StartNew();
+
+        using var process = new Process { StartInfo = psi };
+        process.Start();
+
+        var stdoutTask = process.StandardOutput.ReadToEndAsync(cancellationToken);
+        var stderrTask = process.StandardError.ReadToEndAsync(cancellationToken);
+
+        // Feed stdin inputs with delays if configured
+        if (inputs is { Length: > 0 })
+        {
+            _ = Task.Run(async () =>
+            {
+                try
+                {
+                    foreach (var input in inputs)
+                    {
+                        await Task.Delay(inputDelayMs, cancellationToken);
+                        if (input is not null)
+                        {
+                            await process.StandardInput.WriteLineAsync(input.AsMemory(), cancellationToken);
+                            await process.StandardInput.FlushAsync(cancellationToken);
+                        }
+                    }
+
+                    process.StandardInput.Close();
+                }
+                catch (Exception ex) when (ex is IOException or ObjectDisposedException or OperationCanceledException)
+                {
+                    // Process may have exited before all inputs were sent
+                }
+            }, cancellationToken);
+        }
+
+        using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
+        cts.CancelAfter(timeout);
+
+        try
+        {
+            await process.WaitForExitAsync(cts.Token);
+        }
+        catch (OperationCanceledException) when (!cancellationToken.IsCancellationRequested)
+        {
+            // Timeout — kill the process
+            try
+            {
+                process.Kill(entireProcessTree: true);
+            }
+            catch
+            {
+                // Best effort
+            }
+
+            sw.Stop();
+            return new SampleRunResult(
+                Stdout: await stdoutTask,
+                Stderr: $"TIMEOUT: Sample did not complete within {timeout.TotalSeconds}s.\n{await stderrTask}",
+                ExitCode: -1,
+                Elapsed: sw.Elapsed);
+        }
+
+        sw.Stop();
+        return new SampleRunResult(
+            Stdout: await stdoutTask,
+            Stderr: await stderrTask,
+            ExitCode: process.ExitCode,
+            Elapsed: sw.Elapsed);
+    }
+}

dotnet/eng/verify-samples/SampleVerifier.cs

@@ -0,0 +1,202 @@
+// Copyright (c) Microsoft. All rights reserved.
+
+using System.Text.Json.Serialization;
+using Microsoft.Agents.AI;
+using Microsoft.Extensions.AI;
+using OpenAI.Chat;
+
+namespace VerifySamples;
+
+/// <summary>
+/// Verifies sample output using deterministic checks and an AI agent
+/// for non-deterministic output validation.
+/// </summary>
+internal sealed class SampleVerifier
+{
+    private readonly AIAgent? _verifierAgent;
+
+    /// <summary>
+    /// Creates a verifier. If <paramref name="chatClient"/> is provided,
+    /// AI-based verification is available for non-deterministic samples.
+    /// </summary>
+    public SampleVerifier(ChatClient? chatClient = null)
+    {
+        if (chatClient is not null)
+        {
+            this._verifierAgent = chatClient.AsAIAgent(
+                instructions: """
+                    You are a test output verifier. You will be given:
+                    1. The actual stdout output of a program
+                    2. A list of expectations about what the output should contain or demonstrate
+
+                    Your job is to determine whether the actual output satisfies each expectation.
+                    Be reasonable — the output comes from an LLM so exact wording won't match, but the
+                    semantic intent should be clearly satisfied.
+                    """,
+                name: "OutputVerifier");
+        }
+    }
+
+    /// <summary>
+    /// Verifies the output of a sample run against its definition.
+    /// </summary>
+    public async Task<VerificationResult> VerifyAsync(SampleDefinition sample, SampleRunResult run)
+    {
+        var failures = new List<string>();
+
+        // 1. Exit code check
+        if (run.ExitCode != 0)
+        {
+            failures.Add($"Exit code was {run.ExitCode}, expected 0. Stderr: {Truncate(run.Stderr, 500)}");
+        }
+
+        // 2. Must-contain checks
+        foreach (var expected in sample.MustContain)
+        {
+            if (!run.Stdout.Contains(expected, StringComparison.Ordinal))
+            {
+                failures.Add($"Output missing expected substring: \"{expected}\"");
+            }
+        }
+
+        // 3. Must-not-contain checks
+        foreach (var unexpected in sample.MustNotContain)
+        {
+            if (run.Stdout.Contains(unexpected, StringComparison.Ordinal))
+            {
+                failures.Add($"Output contains unexpected substring: \"{unexpected}\"");
+            }
+        }
+
+        // 4. AI verification for non-deterministic samples
+        string? aiReasoning = null;
+        if (!sample.IsDeterministic && sample.ExpectedOutputDescription.Length > 0)
+        {
+            if (this._verifierAgent is null)
+            {
+                failures.Add("AI verification required but no AI agent configured (missing AZURE_OPENAI_ENDPOINT).");
+            }
+            else
+            {
+                var aiResult = await this.VerifyWithAIAsync(run.Stdout, sample.ExpectedOutputDescription);
+                aiReasoning = aiResult.Reasoning;
+
+                foreach (var unmet in aiResult.UnmetExpectations)
+                {
+                    failures.Add($"AI expectation not met: {unmet}");
+                }
+            }
+        }
+
+        bool passed = failures.Count == 0;
+        return new VerificationResult
+        {
+            SampleName = sample.Name,
+            Passed = passed,
+            Summary = passed ? "All checks passed" : $"{failures.Count} check(s) failed",
+            Failures = failures,
+            AIReasoning = aiReasoning,
+        };
+    }
+
+    private async Task<(string Reasoning, List<string> UnmetExpectations)> VerifyWithAIAsync(
+        string actualOutput,
+        string[] expectations)
+    {
+        var expectationList = string.Join("\n", expectations.Select((e, i) => $"  {i + 1}. {e}"));
+        var prompt = $"""
+            Actual program output:
+            ---
+            {Truncate(actualOutput, 4000)}
+            ---
+
+            Expectations to verify:
+            {expectationList}
+
+            Does the output satisfy all expectations?
+            """;
+
+        try
+        {
+            var response = await this._verifierAgent!.RunAsync<AIVerificationResponse>(prompt);
+            var result = response.Result;
+
+            if (result is null)
+            {
+                return ($"AI verification returned null result. Raw: {response.Text}", ["AI verification returned null result."]);
+            }
+
+            var reasoning = result.Reasoning ?? "(no reasoning provided)";
+
+            // Collect unmet expectations as individual failures
+            var unmet = new List<string>();
+            if (result.ExpectationResults is { Count: > 0 })
+            {
+                foreach (var er in result.ExpectationResults.Where(er => !er.Met))
+                {
+                    var detail = string.IsNullOrWhiteSpace(er.Detail) ? er.Expectation : $"{er.Expectation} — {er.Detail}";
+                    unmet.Add(detail ?? "Unknown expectation");
+                }
+
+                // If the model flagged overall failure but all individual expectations were met,
+                // still treat as failure using the overall reasoning.
+                if (unmet.Count == 0 && !result.Pass)
+                {
+                    unmet.Add(reasoning);
+                }
+            }
+            else if (!result.Pass)
+            {
+                // Fallback: no per-expectation detail but overall pass is false
+                unmet.Add(reasoning);
+            }
+
+            return (reasoning, unmet);
+        }
+        catch (Exception ex)
+        {
+            return ($"AI verification error: {ex.Message}", [$"AI verification error: {ex.Message}"]);
+        }
+    }
+
+    private static string Truncate(string text, int maxLength)
+        => text.Length <= maxLength ? text : text[..maxLength] + "... (truncated)";
+}
+
+/// <summary>
+/// Structured response from the AI verification agent.
+/// </summary>
+[System.Diagnostics.CodeAnalysis.SuppressMessage("Performance", "CA1812:Avoid uninstantiated internal classes", Justification = "Instantiated by JSON deserialization via RunAsync<T>.")]
+internal sealed class AIVerificationResponse
+{
+    /// <summary>Whether all expectations were met.</summary>
+    [JsonPropertyName("pass")]
+    public bool Pass { get; set; }
+
+    /// <summary>Brief explanation of the overall assessment.</summary>
+    [JsonPropertyName("reasoning")]
+    public string? Reasoning { get; set; }
+
+    /// <summary>Per-expectation results.</summary>
+    [JsonPropertyName("expectation_results")]
+    public List<ExpectationResult>? ExpectationResults { get; set; }
+}
+
+/// <summary>
+/// Result for an individual expectation check.
+/// </summary>
+[System.Diagnostics.CodeAnalysis.SuppressMessage("Performance", "CA1812:Avoid uninstantiated internal classes", Justification = "Instantiated by JSON deserialization via RunAsync<T>.")]
+internal sealed class ExpectationResult
+{
+    /// <summary>The expectation text that was evaluated.</summary>
+    [JsonPropertyName("expectation")]
+    public string? Expectation { get; set; }
+
+    /// <summary>Whether this expectation was met.</summary>
+    [JsonPropertyName("met")]
+    public bool Met { get; set; }
+
+    /// <summary>Detail about how the expectation was or was not met.</summary>
+    [JsonPropertyName("detail")]
+    public string? Detail { get; set; }
+}

dotnet/eng/verify-samples/VerificationOrchestrator.cs

@@ -0,0 +1,197 @@
+// Copyright (c) Microsoft. All rights reserved.
+
+using System.Collections.Concurrent;
+
+namespace VerifySamples;
+
+/// <summary>
+/// Orchestrates sample verification: filters, runs in parallel, and collects results.
+/// </summary>
+internal sealed class VerificationOrchestrator
+{
+    private readonly SampleVerifier _verifier;
+    private readonly ConsoleReporter _reporter;
+    private readonly LogFileWriter? _logWriter;
+    private readonly string _dotnetRoot;
+    private readonly TimeSpan _timeout;
+
+    public VerificationOrchestrator(
+        SampleVerifier verifier,
+        ConsoleReporter reporter,
+        string dotnetRoot,
+        TimeSpan timeout,
+        LogFileWriter? logWriter = null)
+    {
+        this._verifier = verifier;
+        this._reporter = reporter;
+        this._logWriter = logWriter;
+        this._dotnetRoot = dotnetRoot;
+        this._timeout = timeout;
+    }
+
+    /// <summary>
+    /// The result of running all samples through the orchestrator.
+    /// </summary>
+    internal sealed record RunAllResult(
+        ConcurrentDictionary<string, VerificationResult> Results,
+        List<(string Name, string Reason)> Skipped,
+        List<string> SampleOrder);
+
+    /// <summary>
+    /// Filters samples, runs the runnable ones in parallel, and returns all results.
+    /// </summary>
+    public async Task<RunAllResult> RunAllAsync(
+        IReadOnlyList<SampleDefinition> samples,
+        int maxParallelism)
+    {
+        var skipped = new List<(string Name, string Reason)>();
+        var runnableSamples = new List<SampleDefinition>();
+        var sampleOrder = new List<string>();
+
+        // Separate samples into skipped and runnable
+        foreach (var sample in samples)
+        {
+            sampleOrder.Add(sample.Name);
+
+            if (sample.SkipReason is not null)
+            {
+                skipped.Add((sample.Name, sample.SkipReason));
+                this._reporter.WriteLineWithPrefix(sample.Name, $"SKIPPED — {sample.SkipReason}", ConsoleColor.Yellow);
+
+                if (this._logWriter is not null)
+                {
+                    await this._logWriter.WriteSkippedAsync(sample.Name, sample.SkipReason);
+                }
+
+                continue;
+            }
+
+            var missingRequired = sample.RequiredEnvironmentVariables
+                .Where(v => string.IsNullOrEmpty(Environment.GetEnvironmentVariable(v)))
+                .ToList();
+
+            var missingOptional = sample.OptionalEnvironmentVariables
+                .Where(v => string.IsNullOrEmpty(Environment.GetEnvironmentVariable(v)))
+                .ToList();
+
+            if (missingRequired.Count > 0 || missingOptional.Count > 0)
+            {
+                var reasons = new List<string>();
+                if (missingRequired.Count > 0)
+                {
+                    reasons.Add($"Missing required: {string.Join(", ", missingRequired)}");
+                }
+
+                if (missingOptional.Count > 0)
+                {
+                    reasons.Add($"Missing optional (would cause console prompt hang): {string.Join(", ", missingOptional)}");
+                }
+
+                var skipReason = string.Join("; ", reasons);
+                skipped.Add((sample.Name, skipReason));
+                this._reporter.WriteLineWithPrefix(sample.Name, $"SKIPPED — {skipReason}", ConsoleColor.Yellow);
+
+                if (this._logWriter is not null)
+                {
+                    await this._logWriter.WriteSkippedAsync(sample.Name, skipReason);
+                }
+
+                continue;
+            }
+
+            runnableSamples.Add(sample);
+        }
+
+        // Run samples in parallel
+        var results = new ConcurrentDictionary<string, VerificationResult>();
+        var semaphore = new SemaphoreSlim(maxParallelism);
+
+        this._reporter.WriteLineWithPrefix(
+            "runner", $"Running {runnableSamples.Count} samples (max {maxParallelism} parallel)...");
+
+        try
+        {
+            var tasks = runnableSamples.Select(sample => this.RunSingleAsync(sample, results, semaphore)).ToArray();
+            await Task.WhenAll(tasks);
+        }
+        finally
+        {
+            semaphore.Dispose();
+        }
+
+        return new RunAllResult(results, skipped, sampleOrder);
+    }
+
+    private async Task RunSingleAsync(
+        SampleDefinition sample,
+        ConcurrentDictionary<string, VerificationResult> results,
+        SemaphoreSlim semaphore)
+    {
+        await semaphore.WaitAsync();
+        try
+        {
+            var log = new List<string>();
+            log.Add($"[{sample.Name}] Running...");
+            this._reporter.WriteLineWithPrefix(sample.Name, "Running...");
+
+            var projectPath = Path.Combine(this._dotnetRoot, sample.ProjectPath);
+            var run = sample.Inputs.Length > 0
+                ? await SampleRunner.RunAsync(projectPath, this._timeout, sample.Inputs, sample.InputDelayMs)
+                : await SampleRunner.RunAsync(projectPath, this._timeout);
+
+            log.Add($"[{sample.Name}] Completed ({run.Elapsed.TotalSeconds:F1}s, exit={run.ExitCode})");
+            this._reporter.WriteLineWithPrefix(
+                sample.Name, $"Completed ({run.Elapsed.TotalSeconds:F1}s, exit={run.ExitCode}). Verifying...");
+
+            var result = await this._verifier.VerifyAsync(sample, run);
+
+            if (result.Passed)
+            {
+                log.Add($"[{sample.Name}] PASSED");
+                this._reporter.WriteLineWithPrefix(sample.Name, "PASSED", ConsoleColor.Green);
+            }
+            else
+            {
+                log.Add($"[{sample.Name}] FAILED");
+                this._reporter.WriteLineWithPrefix(sample.Name, "FAILED", ConsoleColor.Red);
+                foreach (var failure in result.Failures)
+                {
+                    log.Add($"[{sample.Name}]   ✗ {failure}");
+                    this._reporter.WriteLineWithPrefix(sample.Name, $"  ✗ {failure}", ConsoleColor.Red);
+                }
+            }
+
+            if (result.AIReasoning is not null)
+            {
+                log.Add($"[{sample.Name}]   AI: {result.AIReasoning}");
+                this._reporter.WriteLineWithPrefix(
+                    sample.Name, $"  AI: {Truncate(result.AIReasoning, 300)}", ConsoleColor.DarkGray);
+            }
+
+            var verificationResult = new VerificationResult
+            {
+                SampleName = result.SampleName,
+                Passed = result.Passed,
+                Summary = result.Summary,
+                Failures = result.Failures,
+                AIReasoning = result.AIReasoning,
+                Stdout = run.Stdout,
+                Stderr = run.Stderr,
+                LogLines = log,
+            };
+            results[sample.Name] = verificationResult;
+
+            if (this._logWriter is not null)
+            {
+                await this._logWriter.WriteSampleResultAsync(verificationResult);
+            }
+        }
+        finally
+        {
+            semaphore.Release();
+        }
+    }
+
+    private static string Truncate(string text, int maxLength)
+        => text.Length <= maxLength ? text : text[..maxLength] + "...";
+}

dotnet/eng/verify-samples/VerificationResult.cs

@@ -0,0 +1,31 @@
+// Copyright (c) Microsoft. All rights reserved.
+
+namespace VerifySamples;
+
+/// <summary>
+/// The result of verifying a single sample.
+/// </summary>
+internal sealed class VerificationResult
+{
+    public required string SampleName { get; init; }
+    public required bool Passed { get; init; }
+    public required string Summary { get; init; }
+    public List<string> Failures { get; init; } = [];
+    public string? AIReasoning { get; init; }
+
+    /// <summary>
+    /// The sample's stdout output, captured for log file output.
+    /// </summary>
+    public string? Stdout { get; init; }
+
+    /// <summary>
+    /// The sample's stderr output, captured for log file output.
+    /// </summary>
+    public string? Stderr { get; init; }
+
+    /// <summary>
+    /// Per-sample log lines, buffered during parallel execution
+    /// and written sequentially to the log file.
+    /// </summary>
+    public List<string> LogLines { get; init; } = [];
+}

dotnet/eng/verify-samples/VerifyOptions.cs

@@ -0,0 +1,124 @@
+// Copyright (c) Microsoft. All rights reserved.
+
+namespace VerifySamples;
+
+/// <summary>
+/// Parsed command-line options for the sample verification tool.
+/// </summary>
+internal sealed class VerifyOptions
+{
+    /// <summary>
+    /// Maximum number of samples to run concurrently.
+    /// </summary>
+    public int MaxParallelism { get; init; } = 8;
+
+    /// <summary>
+    /// Path to write a CSV summary file, or <c>null</c> to skip.
+    /// </summary>
+    public string? CsvFilePath { get; init; }
+
+    /// <summary>
+    /// Path to write a sequential log file, or <c>null</c> to skip.
+    /// </summary>
+    public string? LogFilePath { get; init; }
+
+    /// <summary>
+    /// The filtered list of samples to process.
+    /// </summary>
+    public required IReadOnlyList<SampleDefinition> Samples { get; init; }
+
+    /// <summary>
+    /// All known sample set registries, keyed by category name.
+    /// </summary>
+    private static readonly Dictionary<string, IReadOnlyList<SampleDefinition>> s_sampleSets =
+        new(StringComparer.OrdinalIgnoreCase)
+        {
+            ["01-get-started"] = GetStartedSamples.All,
+            ["02-agents"] = AgentsSamples.All,
+            ["03-workflows"] = WorkflowSamples.All,
+        };
+
+    /// <summary>
+    /// Parses command-line arguments and resolves the sample list.
+    /// Returns <c>null</c> and writes to stderr if the arguments are invalid.
+    /// </summary>
+    public static VerifyOptions? Parse(string[] args)
+    {
+        var argList = args.ToList();
+
+        var categoryFilter = ExtractArg(argList, "--category");
+        var logFilePath = ExtractArg(argList, "--log");
+        var csvFilePath = ExtractArg(argList, "--csv");
+
+        int maxParallelism = 8;
+        var parallelArg = ExtractArg(argList, "--parallel");
+        if (parallelArg is not null && int.TryParse(parallelArg, out var p) && p > 0)
+        {
+            maxParallelism = p;
+        }
+
+        HashSet<string>? nameFilter = null;
+        if (argList.Count > 0)
+        {
+            nameFilter = argList.ToHashSet(StringComparer.OrdinalIgnoreCase);
+        }
+
+        // Build the sample list
+        IReadOnlyList<SampleDefinition> samples;
+        if (categoryFilter is not null)
+        {
+            if (!s_sampleSets.TryGetValue(categoryFilter, out var categoryList))
+            {
+                Console.Error.WriteLine(
+                    $"Unknown category '{categoryFilter}'. Available: {string.Join(", ", s_sampleSets.Keys)}");
+                return null;
+            }
+
+            samples = categoryList;
+        }
+        else
+        {
+            samples = s_sampleSets.Values.SelectMany(s => s).ToList();
+        }
+
+        if (nameFilter is not null)
+        {
+            samples = samples.Where(s => nameFilter.Contains(s.Name)).ToList();
+        }
+
+        if (samples.Count == 0)
+        {
+            var allNames = s_sampleSets.Values.SelectMany(s => s).Select(s => s.Name);
+            Console.Error.WriteLine($"No matching samples found. Available: {string.Join(", ", allNames)}");
+            return null;
+        }
+
+        return new VerifyOptions
+        {
+            MaxParallelism = maxParallelism,
+            LogFilePath = logFilePath,
+            CsvFilePath = csvFilePath,
+            Samples = samples,
+        };
+    }
+
+    private static string? ExtractArg(List<string> list, string flag)
+    {
+        var idx = list.IndexOf(flag);
+        if (idx < 0)
+        {
+            return null;
+        }
+
+        if (idx + 1 >= list.Count)
+        {
+            Console.Error.WriteLine($"Missing value for {flag}.");
+            list.RemoveAt(idx);
+            return null;
+        }
+
+        var value = list[idx + 1];
+        list.RemoveRange(idx, 2);
+        return value;
+    }
+}

dotnet/eng/verify-samples/WorkflowSamples.cs

@@ -0,0 +1,525 @@
+// Copyright (c) Microsoft. All rights reserved.
+
+namespace VerifySamples;
+
+/// <summary>
+/// Defines the expected behavior for each sample in 03-workflows.
+/// </summary>
+internal static class WorkflowSamples
+{
+    public static IReadOnlyList<SampleDefinition> All { get; } =
+    [
+        // ───────────────────────────────────────────────────────────────────
+        // _StartHere
+        // ───────────────────────────────────────────────────────────────────
+
+        new SampleDefinition
+        {
+            Name = "Workflow_StartHere_01_Streaming",
+            ProjectPath = "samples/03-workflows/_StartHere/01_Streaming",
+            RequiredEnvironmentVariables = [],
+            IsDeterministic = true,
+            MustContain =
+            [
+                "UppercaseExecutor: HELLO, WORLD!",
+                "ReverseTextExecutor: !DLROW ,OLLEH",
+            ],
+        },
+
+        new SampleDefinition
+        {
+            Name = "Workflow_StartHere_02_AgentsInWorkflows",
+            ProjectPath = "samples/03-workflows/_StartHere/02_AgentsInWorkflows",
+            RequiredEnvironmentVariables = ["AZURE_OPENAI_ENDPOINT"],
+            OptionalEnvironmentVariables = ["AZURE_OPENAI_DEPLOYMENT_NAME"],
+            ExpectedOutputDescription =
+            [
+                "The output should show agent responses from a translation workflow.",
+                "The output should not contain error messages or stack traces.",
+            ],
+        },
+
+        new SampleDefinition
+        {
+            Name = "Workflow_StartHere_03_AgentWorkflowPatterns",
+            ProjectPath = "samples/03-workflows/_StartHere/03_AgentWorkflowPatterns",
+            RequiredEnvironmentVariables = ["AZURE_OPENAI_ENDPOINT"],
+            OptionalEnvironmentVariables = ["AZURE_OPENAI_DEPLOYMENT_NAME"],
+            Inputs = ["sequential"],
+            InputDelayMs = 3000,
+            ExpectedOutputDescription =
+            [
+                "The output should show a sequential workflow pattern with multiple agents executing tasks in order.",
+                "The output should not contain error messages or stack traces.",
+            ],
+        },
+
+        new SampleDefinition
+        {
+            Name = "Workflow_StartHere_04_MultiModelService",
+            ProjectPath = "samples/03-workflows/_StartHere/04_MultiModelService",
+            RequiredEnvironmentVariables = ["BEDROCK_ACCESS_KEY", "BEDROCK_SECRET_KEY", "ANTHROPIC_API_KEY", "OPENAI_API_KEY"],
+            SkipReason = "Requires multiple external provider API keys (Bedrock, Anthropic, OpenAI).",
+        },
+
+        new SampleDefinition
+        {
+            Name = "Workflow_StartHere_05_SubWorkflows",
+            ProjectPath = "samples/03-workflows/_StartHere/05_SubWorkflows",
+            RequiredEnvironmentVariables = [],
+            IsDeterministic = true,
+            MustContain =
+            [
+                "=== Sub-Workflow Demonstration ===",
+                "Final Output:",
+                "=== Main Workflow Completed ===",
+                "Sample Complete: Workflows can be composed hierarchically using sub-workflows",
+            ],
+        },
+
+        new SampleDefinition
+        {
+            Name = "Workflow_StartHere_06_MixedWorkflowAgentsAndExecutors",
+            ProjectPath = "samples/03-workflows/_StartHere/06_MixedWorkflowAgentsAndExecutors",
+            RequiredEnvironmentVariables = ["AZURE_OPENAI_ENDPOINT"],
+            OptionalEnvironmentVariables = ["AZURE_OPENAI_DEPLOYMENT_NAME"],
+            Inputs = ["What is 2 plus 2?"],
+            InputDelayMs = 3000,
+            ExpectedOutputDescription =
+            [
+                "The output should show agents and executors working together to process a user question.",
+                "The output should not contain error messages or stack traces.",
+            ],
+        },
+
+        new SampleDefinition
+        {
+            Name = "Workflow_StartHere_07_WriterCriticWorkflow",
+            ProjectPath = "samples/03-workflows/_StartHere/07_WriterCriticWorkflow",
+            RequiredEnvironmentVariables = ["AZURE_OPENAI_ENDPOINT"],
+            OptionalEnvironmentVariables = ["AZURE_OPENAI_DEPLOYMENT_NAME"],
+            MustContain = ["=== Writer-Critic Iteration Workflow ==="],
+            ExpectedOutputDescription =
+            [
+                "The output should show a writer-critic iteration workflow with writer and critic sections.",
+                "The critic should either approve or request revisions.",
+                "The output should not contain error messages or stack traces.",
+            ],
+        },
+
+        // ───────────────────────────────────────────────────────────────────
+        // Agents
+        // ───────────────────────────────────────────────────────────────────
+
+        new SampleDefinition
+        {
+            Name = "Workflow_Agents_CustomAgentExecutors",
+            ProjectPath = "samples/03-workflows/Agents/CustomAgentExecutors",
+            RequiredEnvironmentVariables = ["AZURE_OPENAI_ENDPOINT"],
+            OptionalEnvironmentVariables = ["AZURE_OPENAI_DEPLOYMENT_NAME"],
+            ExpectedOutputDescription =
+            [
+                "The output should show custom workflow events including slogan generation and feedback.",
+                "The output should not contain error messages or stack traces.",
+            ],
+        },
+
+        new SampleDefinition
+        {
+            Name = "Workflow_Agents_FoundryAgent",
+            ProjectPath = "samples/03-workflows/Agents/FoundryAgent",
+            RequiredEnvironmentVariables = ["AZURE_AI_PROJECT_ENDPOINT"],
+            OptionalEnvironmentVariables = ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
+            SkipReason = "Requires Azure AI Foundry project endpoint.",
+        },
+
+        new SampleDefinition
+        {
+            Name = "Workflow_Agents_GroupChatToolApproval",
+            ProjectPath = "samples/03-workflows/Agents/GroupChatToolApproval",
+            RequiredEnvironmentVariables = ["AZURE_OPENAI_ENDPOINT"],
+            OptionalEnvironmentVariables = ["AZURE_OPENAI_DEPLOYMENT_NAME"],
+            MustContain = ["Starting group chat workflow for software deployment..."],
+            ExpectedOutputDescription =
+            [
+                "The output should show a group chat workflow with QA and DevOps agents for software deployment.",
+                "There should be approval requests for tool calls.",
+                "The workflow should show interaction between QA and DevOps agents toward deployment.",
+                "The output should not contain error messages or stack traces.",
+            ],
+        },
+
+        new SampleDefinition
+        {
+            Name = "Workflow_Agents_WorkflowAsAnAgent",
+            ProjectPath = "samples/03-workflows/Agents/WorkflowAsAnAgent",
+            RequiredEnvironmentVariables = ["AZURE_OPENAI_ENDPOINT"],
+            OptionalEnvironmentVariables = ["AZURE_OPENAI_DEPLOYMENT_NAME"],
+            Inputs = ["hello", "exit"],
+            InputDelayMs = 5000,
+            ExpectedOutputDescription =
+            [
+                "The output should show a conversational workflow responding to the user's hello message.",
+                "The output should not contain error messages or stack traces.",
+            ],
+        },
+
+        // ───────────────────────────────────────────────────────────────────
+        // Checkpoint
+        // ───────────────────────────────────────────────────────────────────
+
+        new SampleDefinition
+        {
+            Name = "Workflow_Checkpoint_CheckpointAndRehydrate",
+            ProjectPath = "samples/03-workflows/Checkpoint/CheckpointAndRehydrate",
+            RequiredEnvironmentVariables = [],
+            IsDeterministic = true,
+            MustContain =
+            [
+                "Workflow completed with result:",
+                "Number of checkpoints created:",
+                "Hydrating a new workflow instance from the 6th checkpoint.",
+            ],
+        },
+
+        new SampleDefinition
+        {
+            Name = "Workflow_Checkpoint_CheckpointAndResume",
+            ProjectPath = "samples/03-workflows/Checkpoint/CheckpointAndResume",
+            RequiredEnvironmentVariables = [],
+            IsDeterministic = true,
+            MustContain =
+            [
+                "Workflow completed with result:",
+                "Number of checkpoints created:",
+                "Restoring from the 6th checkpoint.",
+            ],
+        },
+
+        new SampleDefinition
+        {
+            Name = "Workflow_Checkpoint_CheckpointWithHumanInTheLoop",
+            ProjectPath = "samples/03-workflows/Checkpoint/CheckpointWithHumanInTheLoop",
+            RequiredEnvironmentVariables = [],
+            Inputs = ["50", "25", "40", "45", "42", "50", "25", "40", "45", "42"],
+            InputDelayMs = 1000,
+            MustContain = ["found in"],
+            ExpectedOutputDescription =
+            [
+                "The output should show a number guessing game with higher/lower hints that eventually reaches the correct number.",
+                "The output should demonstrate checkpoint save and restore behavior.",
+            ],
+        },
+
+        // ───────────────────────────────────────────────────────────────────
+        // Concurrent
+        // ───────────────────────────────────────────────────────────────────
+
+        new SampleDefinition
+        {
+            Name = "Workflow_Concurrent_Concurrent",
+            ProjectPath = "samples/03-workflows/Concurrent/Concurrent",
+            RequiredEnvironmentVariables = ["AZURE_OPENAI_ENDPOINT"],
+            OptionalEnvironmentVariables = ["AZURE_OPENAI_DEPLOYMENT_NAME"],
+            ExpectedOutputDescription =
+            [
+                "The output should show results from concurrent agent processing.",
+                "The output should not contain error messages or stack traces.",
+            ],
+        },
+
+        new SampleDefinition
+        {
+            Name = "Workflow_Concurrent_MapReduce",
+            ProjectPath = "samples/03-workflows/Concurrent/MapReduce",
+            RequiredEnvironmentVariables = [],
+            MustContain =
+            [
+                "=== RUNNING WORKFLOW ===",
+            ],
+        },
+
+        // ───────────────────────────────────────────────────────────────────
+        // ConditionalEdges
+        // ───────────────────────────────────────────────────────────────────
+
+        new SampleDefinition
+        {
+            Name = "Workflow_ConditionalEdges_01_EdgeCondition",
+            ProjectPath = "samples/03-workflows/ConditionalEdges/01_EdgeCondition",
+            RequiredEnvironmentVariables = ["AZURE_OPENAI_ENDPOINT"],
+            OptionalEnvironmentVariables = ["AZURE_OPENAI_DEPLOYMENT_NAME"],
+            ExpectedOutputDescription =
+            [
+                "The output should show an email being classified as spam or not spam and processed accordingly.",
+                "The output should not contain error messages or stack traces.",
+            ],
+        },
+
+        new SampleDefinition
+        {
+            Name = "Workflow_ConditionalEdges_02_SwitchCase",
+            ProjectPath = "samples/03-workflows/ConditionalEdges/02_SwitchCase",
+            RequiredEnvironmentVariables = ["AZURE_OPENAI_ENDPOINT"],
+            OptionalEnvironmentVariables = ["AZURE_OPENAI_DEPLOYMENT_NAME"],
+            ExpectedOutputDescription =
+            [
+                "The output should show an ambiguous email being classified as spam, not spam, or uncertain.",
+                "The output should not contain error messages or stack traces.",
+            ],
+        },
+
+        new SampleDefinition
+        {
+            Name = "Workflow_ConditionalEdges_03_MultiSelection",
+            ProjectPath = "samples/03-workflows/ConditionalEdges/03_MultiSelection",
+            RequiredEnvironmentVariables = ["AZURE_OPENAI_ENDPOINT"],
+            OptionalEnvironmentVariables = ["AZURE_OPENAI_DEPLOYMENT_NAME"],
+            ExpectedOutputDescription =
+            [
+                "The output should show an email being classified and potentially routed to multiple handlers.",
+                "The output should not contain error messages or stack traces.",
+            ],
+        },
+
+        // ───────────────────────────────────────────────────────────────────
+        // HumanInTheLoop
+        // ───────────────────────────────────────────────────────────────────
+
+        new SampleDefinition
+        {
+            Name = "Workflow_HumanInTheLoop_Basic",
+            ProjectPath = "samples/03-workflows/HumanInTheLoop/HumanInTheLoopBasic",
+            RequiredEnvironmentVariables = [],
+            Inputs = ["50", "25", "40", "45", "42"],
+            InputDelayMs = 1000,
+            MustContain = ["found in"],
+            ExpectedOutputDescription =
+            [
+                "The output should show a number guessing game with higher/lower hints that eventually reaches the correct number 42.",
+            ],
+        },
+
+        // ───────────────────────────────────────────────────────────────────
+        // Loop
+        // ───────────────────────────────────────────────────────────────────
+
+        new SampleDefinition
+        {
+            Name = "Workflow_Loop",
+            ProjectPath = "samples/03-workflows/Loop",
+            RequiredEnvironmentVariables = [],
+            MustContain = ["Result:"],
+        },
+
+        // ───────────────────────────────────────────────────────────────────
+        // SharedStates
+        // ───────────────────────────────────────────────────────────────────
+
+        new SampleDefinition
+        {
+            Name = "Workflow_SharedStates",
+            ProjectPath = "samples/03-workflows/SharedStates",
+            RequiredEnvironmentVariables = [],
+            IsDeterministic = true,
+            MustContain =
+            [
+                "Total Paragraphs:",
+                "Total Words:",
+            ],
+        },
+
+        // ───────────────────────────────────────────────────────────────────
+        // Visualization
+        // ───────────────────────────────────────────────────────────────────
+
+        new SampleDefinition
+        {
+            Name = "Workflow_Visualization",
+            ProjectPath = "samples/03-workflows/Visualization",
+            RequiredEnvironmentVariables = [],
+            IsDeterministic = true,
+            MustContain =
+            [
+                "Generating workflow visualization...",
+                "Mermaid string:",
+                "DiGraph string:",
+            ],
+        },
+
+        // ───────────────────────────────────────────────────────────────────
+        // Observability
+        // ───────────────────────────────────────────────────────────────────
+
+        new SampleDefinition
+        {
+            Name = "Workflow_Observability_ApplicationInsights",
+            ProjectPath = "samples/03-workflows/Observability/ApplicationInsights",
+            RequiredEnvironmentVariables = ["APPLICATIONINSIGHTS_CONNECTION_STRING"],
+            SkipReason = "Requires Application Insights connection string.",
+        },
+
+        new SampleDefinition
+        {
+            Name = "Workflow_Observability_AspireDashboard",
+            ProjectPath = "samples/03-workflows/Observability/AspireDashboard",
+            RequiredEnvironmentVariables = [],
+            SkipReason = "Requires Aspire Dashboard / OTLP endpoint.",
+        },
+
+        new SampleDefinition
+        {
+            Name = "Workflow_Observability_WorkflowAsAnAgent",
+            ProjectPath = "samples/03-workflows/Observability/WorkflowAsAnAgent",
+            RequiredEnvironmentVariables = ["AZURE_OPENAI_ENDPOINT"],
+            OptionalEnvironmentVariables = ["AZURE_OPENAI_DEPLOYMENT_NAME"],
+            SkipReason = "Interactive console with ReadLine loop; requires OTLP endpoint.",
+        },
+
+        // ───────────────────────────────────────────────────────────────────
+        // Declarative
+        // ───────────────────────────────────────────────────────────────────
+
+        new SampleDefinition
+        {
+            Name = "Workflow_Declarative_ConfirmInput",
+            ProjectPath = "samples/03-workflows/Declarative/ConfirmInput",
+            RequiredEnvironmentVariables = ["AZURE_AI_PROJECT_ENDPOINT"],
+            Inputs = ["hello", "hello"],
+            InputDelayMs = 8000,
+            ExpectedOutputDescription = ["The output should show a confirmation prompt and a user response."],
+        },
+
+        new SampleDefinition
+        {
+            Name = "Workflow_Declarative_CustomerSupport",
+            ProjectPath = "samples/03-workflows/Declarative/CustomerSupport",
+            RequiredEnvironmentVariables = ["AZURE_AI_PROJECT_ENDPOINT"],
+            OptionalEnvironmentVariables = ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
+            Inputs = ["My laptop won't start"],
+            InputDelayMs = 3000,
+            ExpectedOutputDescription = ["The output should show a customer support workflow processing a laptop issue, with agent responses providing troubleshooting or support."],
+        },
+
+        new SampleDefinition
+        {
+            Name = "Workflow_Declarative_DeepResearch",
+            ProjectPath = "samples/03-workflows/Declarative/DeepResearch",
+            RequiredEnvironmentVariables = ["AZURE_AI_PROJECT_ENDPOINT"],
+            OptionalEnvironmentVariables = ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
+            SkipReason = "Requires external weather API (wttr.in).",
+        },
+
+        new SampleDefinition
+        {
+            Name = "Workflow_Declarative_ExecuteCode",
+            ProjectPath = "samples/03-workflows/Declarative/ExecuteCode",
+            RequiredEnvironmentVariables = ["AZURE_AI_PROJECT_ENDPOINT"],
+            Inputs = ["What is 12 * 34?"],
+            InputDelayMs = 5000,
+            ExpectedOutputDescription = ["The output should show a declarative workflow executing generated code, processing a math question and producing a result."],
+        },
+
+        new SampleDefinition
+        {
+            Name = "Workflow_Declarative_ExecuteWorkflow",
+            ProjectPath = "samples/03-workflows/Declarative/ExecuteWorkflow",
+            RequiredEnvironmentVariables = ["AZURE_AI_PROJECT_ENDPOINT"],
+            SkipReason = "Requires a workflow file path as a CLI argument.",
+        },
+
+        new SampleDefinition
+        {
+            Name = "Workflow_Declarative_FunctionTools",
+            ProjectPath = "samples/03-workflows/Declarative/FunctionTools",
+            RequiredEnvironmentVariables = ["AZURE_AI_PROJECT_ENDPOINT"],
+            OptionalEnvironmentVariables = ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
+            Inputs = ["What are today's specials?", "EXIT"],
+            InputDelayMs = 8000,
+            ExpectedOutputDescription = ["The output should show a workflow calling function tools (e.g. a menu plugin) to answer a question about restaurant specials."],
+        },
+
+        new SampleDefinition
+        {
+            Name = "Workflow_Declarative_GenerateCode",
+            ProjectPath = "samples/03-workflows/Declarative/GenerateCode",
+            IsDeterministic = true,
+            MustContain = ["WORKFLOW: Parsing", "WORKFLOW: Defined"],
+            ExpectedOutputDescription = ["The output should show a YAML workflow being parsed and C# code being generated from it."],
+        },
+
+        new SampleDefinition
+        {
+            Name = "Workflow_Declarative_HostedWorkflow",
+            ProjectPath = "samples/03-workflows/Declarative/HostedWorkflow",
+            RequiredEnvironmentVariables = ["AZURE_AI_PROJECT_ENDPOINT"],
+            SkipReason = "Hosts a persistent workflow server that does not exit.",
+        },
+
+        new SampleDefinition
+        {
+            Name = "Workflow_Declarative_InputArguments",
+            ProjectPath = "samples/03-workflows/Declarative/InputArguments",
+            RequiredEnvironmentVariables = ["AZURE_AI_PROJECT_ENDPOINT"],
+            OptionalEnvironmentVariables = ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
+            Inputs = ["I'd like to visit Seattle", "EXIT"],
+            InputDelayMs = 8000,
+            ExpectedOutputDescription = ["The output should show a workflow capturing location input and providing travel-related information about Seattle."],
+        },
+
+        new SampleDefinition
+        {
+            Name = "Workflow_Declarative_InvokeFunctionTool",
+            ProjectPath = "samples/03-workflows/Declarative/InvokeFunctionTool",
+            RequiredEnvironmentVariables = ["AZURE_AI_PROJECT_ENDPOINT"],
+            OptionalEnvironmentVariables = ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
+            Inputs = ["What's the soup of the day?", "EXIT"],
+            InputDelayMs = 8000,
+            ExpectedOutputDescription = ["The output should show a workflow invoking a function tool (e.g. a menu plugin) to answer a question about the soup of the day."],
+        },
+
+        new SampleDefinition
+        {
+            Name = "Workflow_Declarative_InvokeMcpTool",
+            ProjectPath = "samples/03-workflows/Declarative/InvokeMcpTool",
+            RequiredEnvironmentVariables = ["AZURE_AI_PROJECT_ENDPOINT"],
+            OptionalEnvironmentVariables = ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
+            Inputs = ["Search for .NET tutorials on Microsoft Learn"],
+            InputDelayMs = 3000,
+            ExpectedOutputDescription = ["The output should show a workflow using MCP tools to search Microsoft Learn documentation and provide a summary of results."],
+        },
+
+        new SampleDefinition
+        {
+            Name = "Workflow_Declarative_Marketing",
+            ProjectPath = "samples/03-workflows/Declarative/Marketing",
+            RequiredEnvironmentVariables = ["AZURE_AI_PROJECT_ENDPOINT"],
+            OptionalEnvironmentVariables = ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
+            Inputs = ["A smart water bottle that tracks hydration"],
+            InputDelayMs = 3000,
+            ExpectedOutputDescription = ["The output should show a marketing workflow generating content about a smart water bottle product."],
+        },
+
+        new SampleDefinition
+        {
+            Name = "Workflow_Declarative_StudentTeacher",
+            ProjectPath = "samples/03-workflows/Declarative/StudentTeacher",
+            RequiredEnvironmentVariables = ["AZURE_AI_PROJECT_ENDPOINT"],
+            OptionalEnvironmentVariables = ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
+            Inputs = ["What is 18 + 27?"],
+            InputDelayMs = 3000,
+            ExpectedOutputDescription = ["The output should show a student-teacher workflow where a student asks a math question and a teacher provides the answer."],
+        },
+
+        new SampleDefinition
+        {
+            Name = "Workflow_Declarative_ToolApproval",
+            ProjectPath = "samples/03-workflows/Declarative/ToolApproval",
+            RequiredEnvironmentVariables = ["AZURE_AI_PROJECT_ENDPOINT"],
+            OptionalEnvironmentVariables = ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
+            Inputs = ["Search for .NET tutorials", "EXIT"],
+            InputDelayMs = 8000,
+            ExpectedOutputDescription = ["The output should show a workflow using an MCP tool with approval to search Microsoft Learn, followed by an exit from the input loop."],
+        },
+    ];
+}

dotnet/eng/verify-samples/verify-samples.csproj

@@ -0,0 +1,24 @@
+<Project Sdk="Microsoft.NET.Sdk">
+
+  <PropertyGroup>
+    <OutputType>Exe</OutputType>
+    <TargetFrameworks>net10.0</TargetFrameworks>
+    <Nullable>enable</Nullable>
+    <ImplicitUsings>enable</ImplicitUsings>
+    <IsPackable>false</IsPackable>
+    <IsAotCompatible>false</IsAotCompatible>
+    <!-- This is a top-level console app; ConfigureAwait is unnecessary -->
+    <NoWarn>$(NoWarn);CA2007</NoWarn>
+  </PropertyGroup>
+
+  <ItemGroup>
+    <PackageReference Include="Azure.AI.OpenAI" />
+    <PackageReference Include="Azure.Identity" />
+    <PackageReference Include="Microsoft.Extensions.AI.OpenAI" />
+  </ItemGroup>
+
+  <ItemGroup>
+    <ProjectReference Include="..\..\src\Microsoft.Agents.AI.OpenAI\Microsoft.Agents.AI.OpenAI.csproj" />
+  </ItemGroup>
+
+</Project>