mirror of
https://github.com/microsoft/agent-framework.git
synced 2026-06-16 21:04:09 +08:00
.NET: Foundry Evals integration for .NET (#4914)
* Foundry Evals integration for .NET - Core evaluation framework: EvalItem, LocalEvaluator, FunctionEvaluator, EvalChecks - IAgentEvaluator interface with MeaiEvaluatorAdapter bridge - AgentEvaluationExtensions for agent.EvaluateAsync() overloads - FoundryEvals wrapping MEAI quality/safety evaluators - ConversationSplitters (LastTurn, Full) and IConversationSplitter - EvalItem.PerTurnItems() for multi-turn decomposition - HasImageContent for multimodal content detection - WorkflowEvaluationExtensions for per-agent workflow evaluation - 7 eval samples mirroring Python parity: 02-agents/Evaluation: SimpleEval, ExpectedOutputs, Multimodal 03-workflows/Evaluation: WorkflowEval 05-end-to-end/Evaluation: FoundryQuality, MixedProviders, ConversationSplits - Comprehensive unit tests (1958 passing) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Rewrite FoundryEvals to use real Foundry Evals API Replace MEAI evaluator shim with actual OpenAI EvaluationClient protocol methods. FoundryEvals now creates eval definitions, submits runs, polls for completion, and fetches per-item results server-side. - New constructor: FoundryEvals(AIProjectClient, model, evaluators) - Add FoundryEvalConverter for MEAI ChatMessage -> Foundry JSON format - Add EvalId, RunId, ReportUrl to AgentEvaluationResults - All 20 built-in evaluator constants now work (agent, tool, quality, safety) - Remove Microsoft.Extensions.AI.Evaluation.Quality/Safety dependencies - Update all samples for new constructor (no more ChatConfiguration) - Replace BuildEvaluators tests with ResolveEvaluator tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add response output to CustomEvals and ExpectedOutputs samples Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: pagination, validation, error handling, tests FoundryEvals fixes: - Add pagination for output items (has_more/after cursor) - Add guard clauses for pollIntervalSeconds/timeoutSeconds <= 0 - Fix double TryGetProperty for passed field parsing - Throw on all-tool-evaluators with no tool definitions - Fix XML doc (default 300s, not 180s) New tests (30 added, 1989 total): - EvalChecks: NonEmpty, ContainsExpected (pass/fail/skip/case), HasImageContent, ToolCallsPresent - FoundryEvalConverter: ConvertMessage (text, image, function call, function results fan-out, empty fallback, mixed content), ConvertEvalItem, BuildTestingCriteria (quality/agent/tool/groundedness data mappings), BuildItemSchema Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix review: null-refs, Data.ToString() bug, ContainsExpected, add tests - Fix NullReferenceException in sample Response display (pattern matching) - Fix WorkflowEvaluationExtensions Data?.ToString() producing type names instead of message text (pattern-match ChatMessage/AgentResponse/list) - Change EvalChecks.ContainsExpected to return Passed=false when no ExpectedOutput (was silently passing, masking misconfiguration) - Add EvalItem constructor tests with LastTurn/Full/null splitters - Add FoundryEvalConverter.ConvertMessage DataContent (base64 image) test - Add ExtractAgentData tests with ChatMessage, list, and AgentResponse data Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix review: conversation fidelity, eval caching, fallback tests - WorkflowEvaluationExtensions: preserve full response messages (tool calls, intermediate) instead of synthetic 2-message conversation. Cast completed Data to AgentResponse and use Messages when available, fallback to text. - FoundryEvals: cache evalId per schema shape (hasContext, hasTools) so subsequent EvaluateAsync calls create runs under the same eval definition. - MeaiEvaluatorAdapter: code already correctly passes queryMessages (not full conversation) to IEvaluator — no change needed, verified by inspection. - Add tests: AgentResponse full messages preservation, unknown object ToString() fallback for ExtractAgentData. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Rename AzureAI→Foundry: move eval files, update references - Move FoundryEvals.cs and FoundryEvalConverter.cs from Microsoft.Agents.AI.AzureAI to Microsoft.Agents.AI.Foundry - Update namespace from AzureAI to Foundry in both files - Add explicit usings required by Foundry project (no implicit usings) - Move FoundryEvalConverter tests to Foundry.UnitTests project (avoids ReplacingRedactor type conflict from dual project refs) - Update all sample csproj references and using statements - Remove Foundry project reference from AI UnitTests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * PR review round 4: wire up tool extraction, remove eval cache, fix null safety - BuildEvalItem: extract tools from agent via GetService<ChatOptions>() into EvalItem.Tools (Python parity) - FoundryEvals: remove eval ID cache - each call creates fresh definition (matches Python behavior) - FoundryEvals: replace null-forgiving operators with descriptive InvalidOperationException - MixedProviders sample: remove unnecessary explicit PackageReferences (transitively provided) - FoundryEvalConverter: document that tool results take precedence over text content - Add LocalEvaluator zero-checks test documenting 0 metrics = failed behavior Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Python-dotnet parity: 9 feature gaps filled New checks: - ToolCallArgsMatch() — verify tool call names + argument subset match - ToolCalledCheck(ToolCalledMode.Any, ...) — match any of the specified tools - ToolCalledMode enum (All/Any) FoundryEvals enhancements: - Default evaluators now [Relevance, Coherence, TaskAdherence] (was Relevance, Coherence) - Auto-add ToolCallAccuracy when items have tool definitions - EvaluateTracesAsync — evaluate by response_ids, trace_ids, or agent_id - EvaluateFoundryTargetAsync — evaluate deployed Foundry targets Result type enrichment: - AgentEvaluationResults: added Status, Error, PerEvaluator, DetailedItems - New EvalItemResult/EvalScoreResult/PerEvaluatorResult types - FoundryEvals populates all new fields from API responses Workflow fix: - Skip internal executors (_*, input-conversation, end-conversation, end) Tests: 8 new tests covering ToolCallArgsMatch, ToolCalledMode.Any, internal executor filtering Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add MeaiEvaluatorAdapter and PerTurnItems edge case tests - 3 tests for MeaiEvaluatorAdapter: query message forwarding, synthetic response fallback, multiple items aggregation - 3 tests for EvalItem.PerTurnItems: empty conversation, no user messages, system+assistant only - StubEvaluator and StubChatClient test helpers Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Blocking link check for outdated package in DevUI. * Replace Dictionary<string, object> payloads with typed wire models Introduce internal FoundryEvalWireModels.cs with compile-time-safe types for the OpenAI Evals API wire format. The OpenAI .NET SDK (2.9.1) only provides protocol-level methods with BinaryContent/ClientResult — no typed request models. These internal models replace scattered dictionary literals with [JsonPropertyName]-annotated classes, giving: - Compile-time safety (typos become build errors) - Single point of change when the API evolves - IntelliSense discoverability - Cleaner serialization via JsonPolymorphic for content items Models: WireContentItem hierarchy (text, image, tool_call, tool_result), WireMessage, WireEvalItemPayload, WireTestingCriterion, WireItemSchema, WireCreateEvalRequest, WireCreateRunRequest, and data source variants. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Skip metric when Foundry returns neither score nor passed When an evaluator returns no score and no passed value, the previous code created BooleanMetric(name, false), which falsely failed items via ItemPassed. Now we skip the MEAI metric entirely for indeterminate results — the raw data remains available in DetailedItems for diagnostics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #4914 review comments: fix tool evaluator bug and add tests - Fix duplicate ToolCallAccuracy: resolve evaluator names before checking against ToolEvaluators set (Comment 2) - Make FilterToolEvaluators internal for testability; add tests for the ArgumentException edge case when all evaluators are tool-type (Comment 3) - Add CancellationToken test for LocalEvaluator (Comment 4) - Add EvaluateAsync integration test on Run with sequential workflow and per-agent SubResults verification (Comment 5) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address Peter's review comments on PR #4914 - Add trailing newline to Evaluation_FoundryQuality.csproj (Comment 6) - Make evaluator name lookups case-insensitive: switch BuiltinEvaluators, ToolEvaluators, AgentEvaluators, and ResolveEvaluator's StartsWith check from Ordinal to OrdinalIgnoreCase (Comment 7) - Add Trace.TraceWarning when Foundry returns fewer results than submitted items, indicating expected vs actual count before padding (Comment 8) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add Microsoft.Extensions.AI.Evaluation packages to Directory.Packages.props These were removed in #5269 as unused, but are needed by the Foundry and core evaluation integration added in this PR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: alliscode <bentho@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
Ben Thomas
co-authored by
Copilot
alliscode
Copilot
alliscode
parent
91e34358eb
commit
aee1acbf8b
@@ -0,0 +1,307 @@
|
||||
// Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using Microsoft.Extensions.AI;
|
||||
|
||||
namespace Microsoft.Agents.AI.Foundry;
|
||||
|
||||
/// <summary>
|
||||
/// Converts MEAI <see cref="ChatMessage"/> objects to the Foundry evaluator JSON format.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Handles the type gap between MEAI's <see cref="ChatMessage"/> / <see cref="AIContent"/> types
|
||||
/// and the OpenAI-style agent message schema used by Foundry evaluation providers.
|
||||
/// </remarks>
|
||||
internal static class FoundryEvalConverter
|
||||
{
|
||||
/// <summary>
|
||||
/// Converts a single <see cref="ChatMessage"/> to one or more Foundry evaluator wire messages.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// A single message with multiple <see cref="FunctionResultContent"/> entries produces
|
||||
/// multiple output messages (one per tool result), matching the Foundry evaluator schema.
|
||||
/// </remarks>
|
||||
internal static List<WireMessage> ConvertMessage(ChatMessage message)
|
||||
{
|
||||
var role = message.Role.Value;
|
||||
var contentItems = new List<WireContentItem>();
|
||||
var toolResults = new List<(string CallId, object Result)>();
|
||||
|
||||
foreach (var content in message.Contents)
|
||||
{
|
||||
switch (content)
|
||||
{
|
||||
case TextContent tc when !string.IsNullOrEmpty(tc.Text):
|
||||
contentItems.Add(new WireTextContent { Text = tc.Text });
|
||||
break;
|
||||
|
||||
case UriContent uc when uc.HasTopLevelMediaType("image"):
|
||||
contentItems.Add(new WireImageContent { ImageUrl = uc.Uri.ToString() });
|
||||
break;
|
||||
|
||||
case DataContent dc when dc.HasTopLevelMediaType("image"):
|
||||
contentItems.Add(new WireImageContent { ImageUrl = dc.Uri });
|
||||
break;
|
||||
|
||||
case FunctionCallContent fc:
|
||||
contentItems.Add(new WireToolCallContent
|
||||
{
|
||||
ToolCallId = fc.CallId ?? string.Empty,
|
||||
Name = fc.Name ?? string.Empty,
|
||||
Arguments = fc.Arguments is { Count: > 0 } ? fc.Arguments : null,
|
||||
});
|
||||
break;
|
||||
|
||||
case FunctionResultContent fr:
|
||||
toolResults.Add((fr.CallId ?? string.Empty, fr.Result ?? string.Empty));
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
var output = new List<WireMessage>();
|
||||
|
||||
if (toolResults.Count > 0)
|
||||
{
|
||||
// Tool results take precedence — the Foundry Evals API expects tool messages
|
||||
// to have role=tool with a single tool_result content. Any text content in the
|
||||
// same message is omitted since the API format doesn't support mixed content.
|
||||
foreach (var (callId, result) in toolResults)
|
||||
{
|
||||
output.Add(new WireMessage
|
||||
{
|
||||
Role = "tool",
|
||||
ToolCallId = callId,
|
||||
Content = [new WireToolResultContent { ToolResult = result }],
|
||||
});
|
||||
}
|
||||
}
|
||||
else if (contentItems.Count > 0)
|
||||
{
|
||||
output.Add(new WireMessage
|
||||
{
|
||||
Role = role,
|
||||
Content = contentItems,
|
||||
});
|
||||
}
|
||||
else
|
||||
{
|
||||
output.Add(new WireMessage
|
||||
{
|
||||
Role = role,
|
||||
Content = [new WireTextContent { Text = string.Empty }],
|
||||
});
|
||||
}
|
||||
|
||||
return output;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Converts a sequence of <see cref="ChatMessage"/> objects to Foundry evaluator format.
|
||||
/// </summary>
|
||||
internal static List<WireMessage> ConvertMessages(IEnumerable<ChatMessage> messages)
|
||||
{
|
||||
var result = new List<WireMessage>();
|
||||
foreach (var msg in messages)
|
||||
{
|
||||
result.AddRange(ConvertMessage(msg));
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Converts an <see cref="EvalItem"/> to a wire-format payload for the Foundry Evals API.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Produces both string fields (query, response) for quality evaluators and
|
||||
/// conversation arrays (query_messages, response_messages) for agent evaluators.
|
||||
/// </remarks>
|
||||
internal static WireEvalItemPayload ConvertEvalItem(EvalItem item, IConversationSplitter? defaultSplitter = null)
|
||||
{
|
||||
var splitter = item.Splitter ?? defaultSplitter ?? ConversationSplitters.LastTurn;
|
||||
var (queryMessages, responseMessages) = splitter.Split(item.Conversation);
|
||||
|
||||
return new WireEvalItemPayload
|
||||
{
|
||||
Query = item.Query,
|
||||
Response = item.Response,
|
||||
QueryMessages = ConvertMessages(queryMessages),
|
||||
ResponseMessages = ConvertMessages(responseMessages),
|
||||
Context = item.Context,
|
||||
ToolDefinitions = item.Tools is { Count: > 0 }
|
||||
? item.Tools
|
||||
.OfType<AIFunction>()
|
||||
.Select(t => new WireToolDefinition
|
||||
{
|
||||
Name = t.Name,
|
||||
Description = t.Description,
|
||||
Parameters = t.JsonSchema,
|
||||
})
|
||||
.ToList()
|
||||
: null,
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Builds the <c>testing_criteria</c> array for <c>evals.create()</c>.
|
||||
/// </summary>
|
||||
/// <param name="evaluators">Evaluator names (short or fully-qualified).</param>
|
||||
/// <param name="model">Model deployment name for the LLM judge.</param>
|
||||
/// <param name="includeDataMapping">
|
||||
/// Whether to include field-level data mapping (required for JSONL data source).
|
||||
/// </param>
|
||||
internal static List<WireTestingCriterion> BuildTestingCriteria(
|
||||
IEnumerable<string> evaluators,
|
||||
string model,
|
||||
bool includeDataMapping = false)
|
||||
{
|
||||
var criteria = new List<WireTestingCriterion>();
|
||||
foreach (var name in evaluators)
|
||||
{
|
||||
var qualified = ResolveEvaluator(name);
|
||||
var shortName = name.StartsWith("builtin.", StringComparison.Ordinal)
|
||||
? name.Substring("builtin.".Length)
|
||||
: name;
|
||||
|
||||
Dictionary<string, string>? dataMapping = null;
|
||||
if (includeDataMapping)
|
||||
{
|
||||
dataMapping = new Dictionary<string, string>();
|
||||
if (AgentEvaluators.Contains(qualified))
|
||||
{
|
||||
dataMapping["query"] = "{{item.query_messages}}";
|
||||
dataMapping["response"] = "{{item.response_messages}}";
|
||||
}
|
||||
else
|
||||
{
|
||||
dataMapping["query"] = "{{item.query}}";
|
||||
dataMapping["response"] = "{{item.response}}";
|
||||
}
|
||||
|
||||
if (qualified == "builtin.groundedness")
|
||||
{
|
||||
dataMapping["context"] = "{{item.context}}";
|
||||
}
|
||||
|
||||
if (ToolEvaluators.Contains(qualified))
|
||||
{
|
||||
dataMapping["tool_definitions"] = "{{item.tool_definitions}}";
|
||||
}
|
||||
}
|
||||
|
||||
criteria.Add(new WireTestingCriterion
|
||||
{
|
||||
Name = shortName,
|
||||
EvaluatorName = qualified,
|
||||
InitializationParameters = new WireInitParams { DeploymentName = model },
|
||||
DataMapping = dataMapping,
|
||||
});
|
||||
}
|
||||
|
||||
return criteria;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Builds the <c>item_schema</c> for custom JSONL eval definitions.
|
||||
/// </summary>
|
||||
internal static WireItemSchema BuildItemSchema(bool hasContext = false, bool hasTools = false)
|
||||
{
|
||||
var properties = new Dictionary<string, WireSchemaProperty>
|
||||
{
|
||||
["query"] = new() { Type = "string" },
|
||||
["response"] = new() { Type = "string" },
|
||||
["query_messages"] = new() { Type = "array" },
|
||||
["response_messages"] = new() { Type = "array" },
|
||||
};
|
||||
|
||||
if (hasContext)
|
||||
{
|
||||
properties["context"] = new WireSchemaProperty { Type = "string" };
|
||||
}
|
||||
|
||||
if (hasTools)
|
||||
{
|
||||
properties["tool_definitions"] = new WireSchemaProperty { Type = "array" };
|
||||
}
|
||||
|
||||
return new WireItemSchema
|
||||
{
|
||||
Properties = properties,
|
||||
Required = ["query", "response"],
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Resolves a short evaluator name to its fully-qualified <c>builtin.*</c> form.
|
||||
/// </summary>
|
||||
internal static string ResolveEvaluator(string name)
|
||||
{
|
||||
if (name.StartsWith("builtin.", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
return name;
|
||||
}
|
||||
|
||||
if (BuiltinEvaluators.TryGetValue(name, out var qualified))
|
||||
{
|
||||
return qualified;
|
||||
}
|
||||
|
||||
throw new ArgumentException(
|
||||
$"Unknown evaluator '{name}'. Available: {string.Join(", ", BuiltinEvaluators.Keys.Order())}",
|
||||
nameof(name));
|
||||
}
|
||||
|
||||
// Agent evaluators that accept query/response as conversation arrays.
|
||||
internal static readonly HashSet<string> AgentEvaluators = new(StringComparer.OrdinalIgnoreCase)
|
||||
{
|
||||
"builtin.intent_resolution",
|
||||
"builtin.task_adherence",
|
||||
"builtin.task_completion",
|
||||
"builtin.task_navigation_efficiency",
|
||||
"builtin.tool_call_accuracy",
|
||||
"builtin.tool_selection",
|
||||
"builtin.tool_input_accuracy",
|
||||
"builtin.tool_output_utilization",
|
||||
"builtin.tool_call_success",
|
||||
};
|
||||
|
||||
// Evaluators that additionally require tool_definitions.
|
||||
internal static readonly HashSet<string> ToolEvaluators = new(StringComparer.OrdinalIgnoreCase)
|
||||
{
|
||||
"builtin.tool_call_accuracy",
|
||||
"builtin.tool_selection",
|
||||
"builtin.tool_input_accuracy",
|
||||
"builtin.tool_output_utilization",
|
||||
"builtin.tool_call_success",
|
||||
};
|
||||
|
||||
// Short name → fully-qualified name mapping.
|
||||
internal static readonly Dictionary<string, string> BuiltinEvaluators = new(StringComparer.OrdinalIgnoreCase)
|
||||
{
|
||||
// Agent behavior
|
||||
["intent_resolution"] = "builtin.intent_resolution",
|
||||
["task_adherence"] = "builtin.task_adherence",
|
||||
["task_completion"] = "builtin.task_completion",
|
||||
["task_navigation_efficiency"] = "builtin.task_navigation_efficiency",
|
||||
// Tool usage
|
||||
["tool_call_accuracy"] = "builtin.tool_call_accuracy",
|
||||
["tool_selection"] = "builtin.tool_selection",
|
||||
["tool_input_accuracy"] = "builtin.tool_input_accuracy",
|
||||
["tool_output_utilization"] = "builtin.tool_output_utilization",
|
||||
["tool_call_success"] = "builtin.tool_call_success",
|
||||
// Quality
|
||||
["coherence"] = "builtin.coherence",
|
||||
["fluency"] = "builtin.fluency",
|
||||
["relevance"] = "builtin.relevance",
|
||||
["groundedness"] = "builtin.groundedness",
|
||||
["response_completeness"] = "builtin.response_completeness",
|
||||
["similarity"] = "builtin.similarity",
|
||||
// Safety
|
||||
["violence"] = "builtin.violence",
|
||||
["sexual"] = "builtin.sexual",
|
||||
["self_harm"] = "builtin.self_harm",
|
||||
["hate_unfairness"] = "builtin.hate_unfairness",
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,314 @@
|
||||
// Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
using System.Collections.Generic;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace Microsoft.Agents.AI.Foundry;
|
||||
|
||||
/// <summary>
|
||||
/// Internal wire-format models for the OpenAI Evals API.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// The OpenAI .NET SDK (as of 2.9.1) marks its <c>EvaluationClient</c> as experimental
|
||||
/// and exposes only protocol-level methods that accept <c>BinaryContent</c> and return
|
||||
/// <c>ClientResult</c> — no strongly typed request or response models are provided.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// These internal models replace hand-built <c>Dictionary<string, object></c> payloads
|
||||
/// with compile-time–safe types that are serialized via <see cref="System.Text.Json"/>.
|
||||
/// When the SDK ships typed models, these should be replaced.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
// -----------------------------------------------------------------------
|
||||
// Message content items (polymorphic by "type" discriminator)
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
[JsonPolymorphic(TypeDiscriminatorPropertyName = "type")]
|
||||
[JsonDerivedType(typeof(WireTextContent), "text")]
|
||||
[JsonDerivedType(typeof(WireImageContent), "input_image")]
|
||||
[JsonDerivedType(typeof(WireToolCallContent), "tool_call")]
|
||||
[JsonDerivedType(typeof(WireToolResultContent), "tool_result")]
|
||||
internal abstract class WireContentItem
|
||||
{
|
||||
}
|
||||
|
||||
internal sealed class WireTextContent : WireContentItem
|
||||
{
|
||||
[JsonPropertyName("text")]
|
||||
public required string Text { get; init; }
|
||||
}
|
||||
|
||||
internal sealed class WireImageContent : WireContentItem
|
||||
{
|
||||
[JsonPropertyName("image_url")]
|
||||
public required string ImageUrl { get; init; }
|
||||
|
||||
[JsonPropertyName("detail")]
|
||||
public string Detail { get; init; } = "auto";
|
||||
}
|
||||
|
||||
internal sealed class WireToolCallContent : WireContentItem
|
||||
{
|
||||
[JsonPropertyName("tool_call_id")]
|
||||
public required string ToolCallId { get; init; }
|
||||
|
||||
[JsonPropertyName("name")]
|
||||
public required string Name { get; init; }
|
||||
|
||||
[JsonPropertyName("arguments")]
|
||||
public IDictionary<string, object?>? Arguments { get; init; }
|
||||
}
|
||||
|
||||
internal sealed class WireToolResultContent : WireContentItem
|
||||
{
|
||||
[JsonPropertyName("tool_result")]
|
||||
public required object ToolResult { get; init; }
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Message
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
internal sealed class WireMessage
|
||||
{
|
||||
[JsonPropertyName("role")]
|
||||
public required string Role { get; init; }
|
||||
|
||||
[JsonPropertyName("content")]
|
||||
public required List<WireContentItem> Content { get; init; }
|
||||
|
||||
[JsonPropertyName("tool_call_id")]
|
||||
public string? ToolCallId { get; init; }
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Eval item payload (a single JSONL row sent to the Evals API)
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
internal sealed class WireEvalItemPayload
|
||||
{
|
||||
[JsonPropertyName("query")]
|
||||
public required string Query { get; init; }
|
||||
|
||||
[JsonPropertyName("response")]
|
||||
public required string Response { get; init; }
|
||||
|
||||
[JsonPropertyName("query_messages")]
|
||||
public required List<WireMessage> QueryMessages { get; init; }
|
||||
|
||||
[JsonPropertyName("response_messages")]
|
||||
public required List<WireMessage> ResponseMessages { get; init; }
|
||||
|
||||
[JsonPropertyName("context")]
|
||||
public string? Context { get; init; }
|
||||
|
||||
[JsonPropertyName("tool_definitions")]
|
||||
public List<WireToolDefinition>? ToolDefinitions { get; init; }
|
||||
}
|
||||
|
||||
internal sealed class WireToolDefinition
|
||||
{
|
||||
[JsonPropertyName("name")]
|
||||
public string? Name { get; init; }
|
||||
|
||||
[JsonPropertyName("description")]
|
||||
public string? Description { get; init; }
|
||||
|
||||
[JsonPropertyName("parameters")]
|
||||
public object? Parameters { get; init; }
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Testing criteria (evaluator definitions within an eval)
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
internal sealed class WireTestingCriterion
|
||||
{
|
||||
[JsonPropertyName("type")]
|
||||
public string Type { get; init; } = "azure_ai_evaluator";
|
||||
|
||||
[JsonPropertyName("name")]
|
||||
public required string Name { get; init; }
|
||||
|
||||
[JsonPropertyName("evaluator_name")]
|
||||
public required string EvaluatorName { get; init; }
|
||||
|
||||
[JsonPropertyName("initialization_parameters")]
|
||||
public required WireInitParams InitializationParameters { get; init; }
|
||||
|
||||
[JsonPropertyName("data_mapping")]
|
||||
public Dictionary<string, string>? DataMapping { get; init; }
|
||||
}
|
||||
|
||||
internal sealed class WireInitParams
|
||||
{
|
||||
[JsonPropertyName("deployment_name")]
|
||||
public required string DeploymentName { get; init; }
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Item schema (for custom JSONL data source definitions)
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
internal sealed class WireItemSchema
|
||||
{
|
||||
[JsonPropertyName("type")]
|
||||
public string Type { get; init; } = "object";
|
||||
|
||||
[JsonPropertyName("properties")]
|
||||
public required Dictionary<string, WireSchemaProperty> Properties { get; init; }
|
||||
|
||||
[JsonPropertyName("required")]
|
||||
public required List<string> Required { get; init; }
|
||||
}
|
||||
|
||||
internal sealed class WireSchemaProperty
|
||||
{
|
||||
[JsonPropertyName("type")]
|
||||
public required string Type { get; init; }
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Create evaluation request
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
internal sealed class WireCreateEvalRequest
|
||||
{
|
||||
[JsonPropertyName("name")]
|
||||
public required string Name { get; init; }
|
||||
|
||||
[JsonPropertyName("data_source_config")]
|
||||
public required object DataSourceConfig { get; init; }
|
||||
|
||||
[JsonPropertyName("testing_criteria")]
|
||||
public required List<WireTestingCriterion> TestingCriteria { get; init; }
|
||||
}
|
||||
|
||||
// Data source configuration variants
|
||||
|
||||
internal sealed class WireCustomDataSourceConfig
|
||||
{
|
||||
[JsonPropertyName("type")]
|
||||
public string Type { get; init; } = "custom";
|
||||
|
||||
[JsonPropertyName("item_schema")]
|
||||
public required WireItemSchema ItemSchema { get; init; }
|
||||
|
||||
[JsonPropertyName("include_sample_schema")]
|
||||
public bool IncludeSampleSchema { get; init; } = true;
|
||||
}
|
||||
|
||||
internal sealed class WireAzureAiDataSourceConfig
|
||||
{
|
||||
[JsonPropertyName("type")]
|
||||
public string Type { get; init; } = "azure_ai_source";
|
||||
|
||||
[JsonPropertyName("scenario")]
|
||||
public required string Scenario { get; init; }
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Create evaluation run request
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
internal sealed class WireCreateRunRequest
|
||||
{
|
||||
[JsonPropertyName("name")]
|
||||
public required string Name { get; init; }
|
||||
|
||||
[JsonPropertyName("data_source")]
|
||||
public required object DataSource { get; init; }
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Data source variants (used in run requests)
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
internal sealed class WireJsonlDataSource
|
||||
{
|
||||
[JsonPropertyName("type")]
|
||||
public string Type { get; init; } = "jsonl";
|
||||
|
||||
[JsonPropertyName("source")]
|
||||
public required WireFileContentSource Source { get; init; }
|
||||
}
|
||||
|
||||
internal sealed class WireFileContentSource
|
||||
{
|
||||
[JsonPropertyName("type")]
|
||||
public string Type { get; init; } = "file_content";
|
||||
|
||||
[JsonPropertyName("content")]
|
||||
public required List<WireItemWrapper> Content { get; init; }
|
||||
}
|
||||
|
||||
internal sealed class WireItemWrapper
|
||||
{
|
||||
[JsonPropertyName("item")]
|
||||
public required object Item { get; init; }
|
||||
}
|
||||
|
||||
internal sealed class WireResponsesDataSource
|
||||
{
|
||||
[JsonPropertyName("type")]
|
||||
public string Type { get; init; } = "azure_ai_responses";
|
||||
|
||||
[JsonPropertyName("item_generation_params")]
|
||||
public required WireResponseRetrievalParams ItemGenerationParams { get; init; }
|
||||
}
|
||||
|
||||
internal sealed class WireResponseRetrievalParams
|
||||
{
|
||||
[JsonPropertyName("type")]
|
||||
public string Type { get; init; } = "response_retrieval";
|
||||
|
||||
[JsonPropertyName("data_mapping")]
|
||||
public required Dictionary<string, string> DataMapping { get; init; }
|
||||
|
||||
[JsonPropertyName("source")]
|
||||
public required WireFileContentSource Source { get; init; }
|
||||
}
|
||||
|
||||
internal sealed class WireTracesDataSource
|
||||
{
|
||||
[JsonPropertyName("type")]
|
||||
public string Type { get; init; } = "azure_ai_traces";
|
||||
|
||||
[JsonPropertyName("lookback_hours")]
|
||||
public int LookbackHours { get; init; }
|
||||
|
||||
[JsonPropertyName("trace_ids")]
|
||||
public List<string>? TraceIds { get; init; }
|
||||
|
||||
[JsonPropertyName("agent_id")]
|
||||
public string? AgentId { get; init; }
|
||||
}
|
||||
|
||||
internal sealed class WireTargetCompletionsDataSource
|
||||
{
|
||||
[JsonPropertyName("type")]
|
||||
public string Type { get; init; } = "azure_ai_target_completions";
|
||||
|
||||
[JsonPropertyName("target")]
|
||||
public required IDictionary<string, object> Target { get; init; }
|
||||
|
||||
[JsonPropertyName("source")]
|
||||
public required WireFileContentSource Source { get; init; }
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Small item payloads used inside WireItemWrapper
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
internal sealed class WireResponseIdItem
|
||||
{
|
||||
[JsonPropertyName("resp_id")]
|
||||
public required string RespId { get; init; }
|
||||
}
|
||||
|
||||
internal sealed class WireQueryItem
|
||||
{
|
||||
[JsonPropertyName("query")]
|
||||
public required string Query { get; init; }
|
||||
}
|
||||
@@ -0,0 +1,920 @@
|
||||
// Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
using System;
|
||||
using System.ClientModel;
|
||||
using System.ClientModel.Primitives;
|
||||
using System.Collections.Generic;
|
||||
using System.Diagnostics;
|
||||
using System.Diagnostics.CodeAnalysis;
|
||||
using System.Linq;
|
||||
using System.Text.Json;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using Azure.AI.Projects;
|
||||
using Microsoft.Extensions.AI.Evaluation;
|
||||
using OpenAI.Evals;
|
||||
|
||||
#pragma warning disable OPENAI001 // EvaluationClient is experimental
|
||||
|
||||
namespace Microsoft.Agents.AI.Foundry;
|
||||
|
||||
/// <summary>
|
||||
/// Azure AI Foundry evaluator provider that calls the Foundry Evals API.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// Uses the OpenAI Evals API (<c>evals.create</c> / <c>evals.runs.create</c>) via the
|
||||
/// project endpoint to run evaluations server-side. All built-in Foundry evaluators
|
||||
/// (quality, safety, agent behavior, tool usage) are supported.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Results appear in the Azure AI Foundry portal with a report URL for detailed analysis.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
[UnconditionalSuppressMessage("Trimming", "IL2026", Justification = "Serializing Dictionary<string, object> for eval API payloads.")]
|
||||
[UnconditionalSuppressMessage("AOT", "IL3050", Justification = "Serializing Dictionary<string, object> for eval API payloads.")]
|
||||
public sealed class FoundryEvals : IAgentEvaluator
|
||||
{
|
||||
private static readonly JsonSerializerOptions s_jsonOptions = new()
|
||||
{
|
||||
PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseLower,
|
||||
DefaultIgnoreCondition = System.Text.Json.Serialization.JsonIgnoreCondition.WhenWritingNull,
|
||||
};
|
||||
|
||||
private readonly EvaluationClient _evaluationClient;
|
||||
private readonly string _model;
|
||||
private readonly string[] _evaluatorNames;
|
||||
private readonly IConversationSplitter? _splitter;
|
||||
private readonly double _pollIntervalSeconds = 5.0;
|
||||
private readonly double _timeoutSeconds = 300.0;
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Constructors
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
/// <summary>
|
||||
/// Initializes a new instance of the <see cref="FoundryEvals"/> class.
|
||||
/// </summary>
|
||||
/// <param name="projectClient">The Azure AI Foundry project client.</param>
|
||||
/// <param name="model">Model deployment name for the LLM judge evaluator.</param>
|
||||
/// <param name="evaluators">
|
||||
/// Names of evaluators to use (e.g., <see cref="Relevance"/>, <see cref="Coherence"/>).
|
||||
/// When empty, defaults to relevance and coherence.
|
||||
/// </param>
|
||||
public FoundryEvals(AIProjectClient projectClient, string model, params string[] evaluators)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(projectClient);
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(model);
|
||||
|
||||
this._evaluationClient = projectClient.GetProjectOpenAIClient().GetEvaluationClient();
|
||||
this._model = model;
|
||||
this._evaluatorNames = evaluators.Length > 0
|
||||
? evaluators
|
||||
: [Relevance, Coherence, TaskAdherence];
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Initializes a new instance of the <see cref="FoundryEvals"/> class with a conversation splitter.
|
||||
/// </summary>
|
||||
/// <param name="projectClient">The Azure AI Foundry project client.</param>
|
||||
/// <param name="model">Model deployment name for the LLM judge evaluator.</param>
|
||||
/// <param name="splitter">
|
||||
/// Default conversation splitter for multi-turn conversations.
|
||||
/// Use <see cref="ConversationSplitters.LastTurn"/>, <see cref="ConversationSplitters.Full"/>,
|
||||
/// or a custom <see cref="IConversationSplitter"/> implementation.
|
||||
/// </param>
|
||||
/// <param name="evaluators">
|
||||
/// Names of evaluators to use (e.g., <see cref="Relevance"/>, <see cref="Coherence"/>).
|
||||
/// When empty, defaults to relevance and coherence.
|
||||
/// </param>
|
||||
public FoundryEvals(
|
||||
AIProjectClient projectClient,
|
||||
string model,
|
||||
IConversationSplitter? splitter,
|
||||
params string[] evaluators)
|
||||
: this(projectClient, model, evaluators)
|
||||
{
|
||||
this._splitter = splitter;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Initializes a new instance of the <see cref="FoundryEvals"/> class with full configuration.
|
||||
/// </summary>
|
||||
/// <param name="projectClient">The Azure AI Foundry project client.</param>
|
||||
/// <param name="model">Model deployment name for the LLM judge evaluator.</param>
|
||||
/// <param name="splitter">
|
||||
/// Default conversation splitter for multi-turn conversations.
|
||||
/// </param>
|
||||
/// <param name="pollIntervalSeconds">Seconds between status polls (default 5).</param>
|
||||
/// <param name="timeoutSeconds">Maximum seconds to wait for completion (default 300).</param>
|
||||
/// <param name="evaluators">Evaluator names to use.</param>
|
||||
public FoundryEvals(
|
||||
AIProjectClient projectClient,
|
||||
string model,
|
||||
IConversationSplitter? splitter,
|
||||
double pollIntervalSeconds,
|
||||
double timeoutSeconds,
|
||||
params string[] evaluators)
|
||||
: this(projectClient, model, splitter, evaluators)
|
||||
{
|
||||
ArgumentOutOfRangeException.ThrowIfLessThanOrEqual(pollIntervalSeconds, 0);
|
||||
ArgumentOutOfRangeException.ThrowIfLessThanOrEqual(timeoutSeconds, 0);
|
||||
this._pollIntervalSeconds = pollIntervalSeconds;
|
||||
this._timeoutSeconds = timeoutSeconds;
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// IAgentEvaluator
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
/// <inheritdoc />
|
||||
public string Name => "FoundryEvals";
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<AgentEvaluationResults> EvaluateAsync(
|
||||
IReadOnlyList<EvalItem> items,
|
||||
string evalName = "Agent Framework Eval",
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
// 1. Convert EvalItems to typed payloads
|
||||
var payloads = new List<WireEvalItemPayload>(items.Count);
|
||||
foreach (var item in items)
|
||||
{
|
||||
payloads.Add(FoundryEvalConverter.ConvertEvalItem(item, this._splitter));
|
||||
}
|
||||
|
||||
bool hasContext = payloads.Any(p => p.Context is not null);
|
||||
bool hasTools = payloads.Any(p => p.ToolDefinitions is { Count: > 0 });
|
||||
|
||||
// Filter out tool evaluators if no items have tools; auto-add ToolCallAccuracy if tools present
|
||||
var evaluators = FilterToolEvaluators(this._evaluatorNames, hasTools);
|
||||
if (hasTools && !evaluators.Any(e => FoundryEvalConverter.ToolEvaluators.Contains(FoundryEvalConverter.ResolveEvaluator(e))))
|
||||
{
|
||||
evaluators = [.. evaluators, ToolCallAccuracy];
|
||||
}
|
||||
|
||||
// 2. Create the evaluation definition
|
||||
var createEvalPayload = new WireCreateEvalRequest
|
||||
{
|
||||
Name = evalName,
|
||||
DataSourceConfig = new WireCustomDataSourceConfig
|
||||
{
|
||||
ItemSchema = FoundryEvalConverter.BuildItemSchema(hasContext, hasTools),
|
||||
},
|
||||
TestingCriteria = FoundryEvalConverter.BuildTestingCriteria(
|
||||
evaluators, this._model, includeDataMapping: true),
|
||||
};
|
||||
|
||||
var createEvalJson = JsonSerializer.Serialize(createEvalPayload, s_jsonOptions);
|
||||
var createEvalResult = await this._evaluationClient.CreateEvaluationAsync(
|
||||
BinaryContent.Create(BinaryData.FromString(createEvalJson)),
|
||||
new RequestOptions { CancellationToken = cancellationToken }).ConfigureAwait(false);
|
||||
|
||||
string evalId;
|
||||
using (var evalResponse = JsonDocument.Parse(createEvalResult.GetRawResponse().Content))
|
||||
{
|
||||
evalId = evalResponse.RootElement.GetProperty("id").GetString()
|
||||
?? throw new InvalidOperationException("Foundry eval creation returned a null ID.");
|
||||
}
|
||||
|
||||
// 3. Create the evaluation run with inline JSONL data
|
||||
var createRunPayload = new WireCreateRunRequest
|
||||
{
|
||||
Name = $"{evalName} Run",
|
||||
DataSource = new WireJsonlDataSource
|
||||
{
|
||||
Source = new WireFileContentSource
|
||||
{
|
||||
Content = payloads.ConvertAll(p => new WireItemWrapper { Item = p }),
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
var createRunJson = JsonSerializer.Serialize(createRunPayload, s_jsonOptions);
|
||||
var createRunResult = await this._evaluationClient.CreateEvaluationRunAsync(
|
||||
evalId,
|
||||
BinaryContent.Create(BinaryData.FromString(createRunJson)),
|
||||
new RequestOptions { CancellationToken = cancellationToken }).ConfigureAwait(false);
|
||||
|
||||
string runId;
|
||||
using (var runResponse = JsonDocument.Parse(createRunResult.GetRawResponse().Content))
|
||||
{
|
||||
runId = runResponse.RootElement.GetProperty("id").GetString()
|
||||
?? throw new InvalidOperationException("Foundry eval run creation returned a null run ID.");
|
||||
}
|
||||
|
||||
// 4. Poll until complete
|
||||
var pollResult = await this.PollEvalRunAsync(evalId, runId, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
if (pollResult.Status is "failed" or "canceled")
|
||||
{
|
||||
throw new InvalidOperationException(
|
||||
$"Foundry evaluation run {runId} {pollResult.Status}: {pollResult.ErrorMessage ?? "no details available"}");
|
||||
}
|
||||
|
||||
if (pollResult.Status == "timeout")
|
||||
{
|
||||
throw new TimeoutException(
|
||||
$"Foundry evaluation run {runId} did not complete within {this._timeoutSeconds}s. " +
|
||||
"Increase timeoutSeconds or check the run status in the Foundry portal.");
|
||||
}
|
||||
|
||||
// 5. Fetch output items and build results
|
||||
var fetchResult = await this.FetchOutputItemResultsAsync(evalId, runId, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
// Pad MEAI results if we got fewer than items (e.g. partial output)
|
||||
if (fetchResult.MeaiResults.Count < items.Count)
|
||||
{
|
||||
Trace.TraceWarning(
|
||||
"Foundry returned {0} result(s) but {1} item(s) were submitted. " +
|
||||
"Padding {2} missing item(s) with empty results — these items will count as failed.",
|
||||
fetchResult.MeaiResults.Count,
|
||||
items.Count,
|
||||
items.Count - fetchResult.MeaiResults.Count);
|
||||
}
|
||||
|
||||
while (fetchResult.MeaiResults.Count < items.Count)
|
||||
{
|
||||
fetchResult.MeaiResults.Add(new EvaluationResult());
|
||||
}
|
||||
|
||||
return new AgentEvaluationResults(this.Name, fetchResult.MeaiResults, inputItems: items)
|
||||
{
|
||||
ReportUrl = pollResult.ReportUrl is not null ? new Uri(pollResult.ReportUrl) : null,
|
||||
EvalId = evalId,
|
||||
RunId = runId,
|
||||
Status = pollResult.Status,
|
||||
Error = pollResult.ErrorMessage,
|
||||
PerEvaluator = pollResult.PerEvaluator,
|
||||
DetailedItems = fetchResult.DetailedItems,
|
||||
};
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Static evaluation methods (traces and targets)
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
/// <summary>
|
||||
/// Evaluates agent behavior from Responses API response IDs, OTel traces, or agent activity.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// Foundry-specific method that works with any agent emitting OTel traces to App Insights.
|
||||
/// Provide <paramref name="responseIds"/> for specific Responses API responses,
|
||||
/// <paramref name="traceIds"/> for specific traces, or <paramref name="agentId"/> with
|
||||
/// <paramref name="lookbackHours"/> to evaluate recent activity.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
/// <param name="projectClient">The Azure AI Foundry project client.</param>
|
||||
/// <param name="model">Model deployment name for the LLM judge evaluator.</param>
|
||||
/// <param name="responseIds">Evaluate specific Responses API response IDs.</param>
|
||||
/// <param name="traceIds">Evaluate specific OTel trace IDs from App Insights.</param>
|
||||
/// <param name="agentId">Filter traces by agent ID (used with <paramref name="lookbackHours"/>).</param>
|
||||
/// <param name="lookbackHours">Hours of trace history to evaluate (default 24).</param>
|
||||
/// <param name="evaluators">Evaluator names. Defaults to relevance, coherence, and task adherence.</param>
|
||||
/// <param name="evalName">Display name for the evaluation.</param>
|
||||
/// <param name="pollIntervalSeconds">Seconds between status polls (default 5).</param>
|
||||
/// <param name="timeoutSeconds">Maximum seconds to wait for completion (default 300).</param>
|
||||
/// <param name="cancellationToken">Cancellation token.</param>
|
||||
/// <returns>Evaluation results with status, report URL, and per-item details.</returns>
|
||||
public static async Task<AgentEvaluationResults> EvaluateTracesAsync(
|
||||
AIProjectClient projectClient,
|
||||
string model,
|
||||
IEnumerable<string>? responseIds = null,
|
||||
IEnumerable<string>? traceIds = null,
|
||||
string? agentId = null,
|
||||
int lookbackHours = 24,
|
||||
string[]? evaluators = null,
|
||||
string evalName = "Agent Framework Trace Eval",
|
||||
double pollIntervalSeconds = 5.0,
|
||||
double timeoutSeconds = 300.0,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(projectClient);
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(model);
|
||||
|
||||
var responseIdList = responseIds?.ToList();
|
||||
var traceIdList = traceIds?.ToList();
|
||||
|
||||
if ((responseIdList is null || responseIdList.Count == 0)
|
||||
&& (traceIdList is null || traceIdList.Count == 0)
|
||||
&& string.IsNullOrEmpty(agentId))
|
||||
{
|
||||
throw new ArgumentException("Provide at least one of: responseIds, traceIds, or agentId.");
|
||||
}
|
||||
|
||||
var evalClient = projectClient.GetProjectOpenAIClient().GetEvaluationClient();
|
||||
var resolvedEvaluators = evaluators is { Length: > 0 }
|
||||
? evaluators
|
||||
: [Relevance, Coherence, TaskAdherence];
|
||||
|
||||
// Create the evaluation definition with the appropriate data source scenario
|
||||
object dataSourceConfig;
|
||||
object runDataSource;
|
||||
|
||||
if (responseIdList is { Count: > 0 })
|
||||
{
|
||||
// Responses API path
|
||||
dataSourceConfig = new WireAzureAiDataSourceConfig { Scenario = "responses" };
|
||||
|
||||
runDataSource = new WireResponsesDataSource
|
||||
{
|
||||
ItemGenerationParams = new WireResponseRetrievalParams
|
||||
{
|
||||
DataMapping = new Dictionary<string, string> { ["response_id"] = "{{item.resp_id}}" },
|
||||
Source = new WireFileContentSource
|
||||
{
|
||||
Content = responseIdList.ConvertAll(id => new WireItemWrapper
|
||||
{
|
||||
Item = new WireResponseIdItem { RespId = id },
|
||||
}),
|
||||
},
|
||||
},
|
||||
};
|
||||
}
|
||||
else
|
||||
{
|
||||
// Traces path
|
||||
dataSourceConfig = new WireAzureAiDataSourceConfig { Scenario = "traces" };
|
||||
|
||||
runDataSource = new WireTracesDataSource
|
||||
{
|
||||
LookbackHours = lookbackHours,
|
||||
TraceIds = traceIdList is { Count: > 0 } ? traceIdList : null,
|
||||
AgentId = !string.IsNullOrEmpty(agentId) ? agentId : null,
|
||||
};
|
||||
}
|
||||
|
||||
var createEvalPayload = new WireCreateEvalRequest
|
||||
{
|
||||
Name = evalName,
|
||||
DataSourceConfig = dataSourceConfig,
|
||||
TestingCriteria = FoundryEvalConverter.BuildTestingCriteria(resolvedEvaluators, model),
|
||||
};
|
||||
|
||||
var createEvalJson = JsonSerializer.Serialize(createEvalPayload, s_jsonOptions);
|
||||
var createEvalResult = await evalClient.CreateEvaluationAsync(
|
||||
BinaryContent.Create(BinaryData.FromString(createEvalJson)),
|
||||
new RequestOptions { CancellationToken = cancellationToken }).ConfigureAwait(false);
|
||||
|
||||
string evalId;
|
||||
using (var evalResponse = JsonDocument.Parse(createEvalResult.GetRawResponse().Content))
|
||||
{
|
||||
evalId = evalResponse.RootElement.GetProperty("id").GetString()
|
||||
?? throw new InvalidOperationException("Foundry eval creation returned a null ID.");
|
||||
}
|
||||
|
||||
var createRunPayload = new WireCreateRunRequest
|
||||
{
|
||||
Name = $"{evalName} Run",
|
||||
DataSource = runDataSource,
|
||||
};
|
||||
|
||||
var createRunJson = JsonSerializer.Serialize(createRunPayload, s_jsonOptions);
|
||||
var createRunResult = await evalClient.CreateEvaluationRunAsync(
|
||||
evalId,
|
||||
BinaryContent.Create(BinaryData.FromString(createRunJson)),
|
||||
new RequestOptions { CancellationToken = cancellationToken }).ConfigureAwait(false);
|
||||
|
||||
string runId;
|
||||
using (var runResponse = JsonDocument.Parse(createRunResult.GetRawResponse().Content))
|
||||
{
|
||||
runId = runResponse.RootElement.GetProperty("id").GetString()
|
||||
?? throw new InvalidOperationException("Foundry eval run creation returned a null run ID.");
|
||||
}
|
||||
|
||||
// Poll and fetch
|
||||
var instance = new FoundryEvals(projectClient, model, null, pollIntervalSeconds, timeoutSeconds, resolvedEvaluators);
|
||||
var pollResult = await instance.PollEvalRunAsync(evalId, runId, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
if (pollResult.Status is "failed" or "canceled")
|
||||
{
|
||||
throw new InvalidOperationException(
|
||||
$"Foundry trace evaluation run {runId} {pollResult.Status}: {pollResult.ErrorMessage ?? "no details available"}");
|
||||
}
|
||||
|
||||
if (pollResult.Status == "timeout")
|
||||
{
|
||||
throw new TimeoutException(
|
||||
$"Foundry trace evaluation run {runId} did not complete within {timeoutSeconds}s.");
|
||||
}
|
||||
|
||||
var fetchResult = await instance.FetchOutputItemResultsAsync(evalId, runId, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
return new AgentEvaluationResults("FoundryEvals", fetchResult.MeaiResults)
|
||||
{
|
||||
ReportUrl = pollResult.ReportUrl is not null ? new Uri(pollResult.ReportUrl) : null,
|
||||
EvalId = evalId,
|
||||
RunId = runId,
|
||||
Status = pollResult.Status,
|
||||
Error = pollResult.ErrorMessage,
|
||||
PerEvaluator = pollResult.PerEvaluator,
|
||||
DetailedItems = fetchResult.DetailedItems,
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Evaluates a Foundry-registered agent or model deployment.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Foundry invokes the target, captures the output, and evaluates it.
|
||||
/// Use this for scheduled evaluations, red teaming, and CI/CD quality gates.
|
||||
/// </remarks>
|
||||
/// <param name="projectClient">The Azure AI Foundry project client.</param>
|
||||
/// <param name="model">Model deployment name for the LLM judge evaluator.</param>
|
||||
/// <param name="target">Target configuration (must include a "type" key, e.g. "azure_ai_agent").</param>
|
||||
/// <param name="testQueries">Queries for Foundry to send to the target.</param>
|
||||
/// <param name="evaluators">Evaluator names. Defaults to relevance, coherence, and task adherence.</param>
|
||||
/// <param name="evalName">Display name for the evaluation.</param>
|
||||
/// <param name="pollIntervalSeconds">Seconds between status polls (default 5).</param>
|
||||
/// <param name="timeoutSeconds">Maximum seconds to wait for completion (default 300).</param>
|
||||
/// <param name="cancellationToken">Cancellation token.</param>
|
||||
/// <returns>Evaluation results with status, report URL, and per-item details.</returns>
|
||||
public static async Task<AgentEvaluationResults> EvaluateFoundryTargetAsync(
|
||||
AIProjectClient projectClient,
|
||||
string model,
|
||||
IDictionary<string, object> target,
|
||||
IEnumerable<string> testQueries,
|
||||
string[]? evaluators = null,
|
||||
string evalName = "Agent Framework Target Eval",
|
||||
double pollIntervalSeconds = 5.0,
|
||||
double timeoutSeconds = 300.0,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(projectClient);
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(model);
|
||||
ArgumentNullException.ThrowIfNull(target);
|
||||
|
||||
if (!target.ContainsKey("type"))
|
||||
{
|
||||
throw new ArgumentException("Target must include a 'type' key (e.g., 'azure_ai_agent').", nameof(target));
|
||||
}
|
||||
|
||||
var queryList = testQueries.ToList();
|
||||
if (queryList.Count == 0)
|
||||
{
|
||||
throw new ArgumentException("At least one test query is required.", nameof(testQueries));
|
||||
}
|
||||
|
||||
var evalClient = projectClient.GetProjectOpenAIClient().GetEvaluationClient();
|
||||
var resolvedEvaluators = evaluators is { Length: > 0 }
|
||||
? evaluators
|
||||
: [Relevance, Coherence, TaskAdherence];
|
||||
|
||||
var createEvalPayload = new WireCreateEvalRequest
|
||||
{
|
||||
Name = evalName,
|
||||
DataSourceConfig = new WireAzureAiDataSourceConfig { Scenario = "target_completions" },
|
||||
TestingCriteria = FoundryEvalConverter.BuildTestingCriteria(resolvedEvaluators, model),
|
||||
};
|
||||
|
||||
var createEvalJson = JsonSerializer.Serialize(createEvalPayload, s_jsonOptions);
|
||||
var createEvalResult = await evalClient.CreateEvaluationAsync(
|
||||
BinaryContent.Create(BinaryData.FromString(createEvalJson)),
|
||||
new RequestOptions { CancellationToken = cancellationToken }).ConfigureAwait(false);
|
||||
|
||||
string evalId;
|
||||
using (var evalResponse = JsonDocument.Parse(createEvalResult.GetRawResponse().Content))
|
||||
{
|
||||
evalId = evalResponse.RootElement.GetProperty("id").GetString()
|
||||
?? throw new InvalidOperationException("Foundry eval creation returned a null ID.");
|
||||
}
|
||||
|
||||
var createRunPayload = new WireCreateRunRequest
|
||||
{
|
||||
Name = $"{evalName} Run",
|
||||
DataSource = new WireTargetCompletionsDataSource
|
||||
{
|
||||
Target = target,
|
||||
Source = new WireFileContentSource
|
||||
{
|
||||
Content = queryList.ConvertAll(q => new WireItemWrapper
|
||||
{
|
||||
Item = new WireQueryItem { Query = q },
|
||||
}),
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
var createRunJson = JsonSerializer.Serialize(createRunPayload, s_jsonOptions);
|
||||
var createRunResult = await evalClient.CreateEvaluationRunAsync(
|
||||
evalId,
|
||||
BinaryContent.Create(BinaryData.FromString(createRunJson)),
|
||||
new RequestOptions { CancellationToken = cancellationToken }).ConfigureAwait(false);
|
||||
|
||||
string runId;
|
||||
using (var runResponse = JsonDocument.Parse(createRunResult.GetRawResponse().Content))
|
||||
{
|
||||
runId = runResponse.RootElement.GetProperty("id").GetString()
|
||||
?? throw new InvalidOperationException("Foundry eval run creation returned a null run ID.");
|
||||
}
|
||||
|
||||
var instance = new FoundryEvals(projectClient, model, null, pollIntervalSeconds, timeoutSeconds, resolvedEvaluators);
|
||||
var pollResult = await instance.PollEvalRunAsync(evalId, runId, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
if (pollResult.Status is "failed" or "canceled")
|
||||
{
|
||||
throw new InvalidOperationException(
|
||||
$"Foundry target evaluation run {runId} {pollResult.Status}: {pollResult.ErrorMessage ?? "no details available"}");
|
||||
}
|
||||
|
||||
if (pollResult.Status == "timeout")
|
||||
{
|
||||
throw new TimeoutException(
|
||||
$"Foundry target evaluation run {runId} did not complete within {timeoutSeconds}s.");
|
||||
}
|
||||
|
||||
var fetchResult = await instance.FetchOutputItemResultsAsync(evalId, runId, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
return new AgentEvaluationResults("FoundryEvals", fetchResult.MeaiResults)
|
||||
{
|
||||
ReportUrl = pollResult.ReportUrl is not null ? new Uri(pollResult.ReportUrl) : null,
|
||||
EvalId = evalId,
|
||||
RunId = runId,
|
||||
Status = pollResult.Status,
|
||||
Error = pollResult.ErrorMessage,
|
||||
PerEvaluator = pollResult.PerEvaluator,
|
||||
DetailedItems = fetchResult.DetailedItems,
|
||||
};
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Evaluator name constants
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
// Agent behavior
|
||||
|
||||
/// <summary>Evaluates whether the agent correctly resolves user intent.</summary>
|
||||
public const string IntentResolution = "intent_resolution";
|
||||
|
||||
/// <summary>Evaluates whether the agent adheres to its task instructions.</summary>
|
||||
public const string TaskAdherence = "task_adherence";
|
||||
|
||||
/// <summary>Evaluates whether the agent completes the requested task.</summary>
|
||||
public const string TaskCompletion = "task_completion";
|
||||
|
||||
/// <summary>Evaluates the efficiency of the agent's navigation to complete the task.</summary>
|
||||
public const string TaskNavigationEfficiency = "task_navigation_efficiency";
|
||||
|
||||
// Tool usage
|
||||
|
||||
/// <summary>Evaluates the accuracy of tool calls made by the agent.</summary>
|
||||
public const string ToolCallAccuracy = "tool_call_accuracy";
|
||||
|
||||
/// <summary>Evaluates whether the agent selects the correct tools.</summary>
|
||||
public const string ToolSelection = "tool_selection";
|
||||
|
||||
/// <summary>Evaluates the accuracy of inputs provided to tools.</summary>
|
||||
public const string ToolInputAccuracy = "tool_input_accuracy";
|
||||
|
||||
/// <summary>Evaluates how well the agent uses tool outputs.</summary>
|
||||
public const string ToolOutputUtilization = "tool_output_utilization";
|
||||
|
||||
/// <summary>Evaluates whether tool calls succeed.</summary>
|
||||
public const string ToolCallSuccess = "tool_call_success";
|
||||
|
||||
// Quality
|
||||
|
||||
/// <summary>Evaluates the coherence of the response.</summary>
|
||||
public const string Coherence = "coherence";
|
||||
|
||||
/// <summary>Evaluates the fluency of the response.</summary>
|
||||
public const string Fluency = "fluency";
|
||||
|
||||
/// <summary>Evaluates the relevance of the response to the query.</summary>
|
||||
public const string Relevance = "relevance";
|
||||
|
||||
/// <summary>Evaluates whether the response is grounded in the provided context.</summary>
|
||||
public const string Groundedness = "groundedness";
|
||||
|
||||
/// <summary>Evaluates the completeness of the response.</summary>
|
||||
public const string ResponseCompleteness = "response_completeness";
|
||||
|
||||
/// <summary>Evaluates the similarity between the response and the expected output.</summary>
|
||||
public const string Similarity = "similarity";
|
||||
|
||||
// Safety
|
||||
|
||||
/// <summary>Evaluates the response for violent content.</summary>
|
||||
public const string Violence = "violence";
|
||||
|
||||
/// <summary>Evaluates the response for sexual content.</summary>
|
||||
public const string Sexual = "sexual";
|
||||
|
||||
/// <summary>Evaluates the response for self-harm content.</summary>
|
||||
public const string SelfHarm = "self_harm";
|
||||
|
||||
/// <summary>Evaluates the response for hate or unfairness.</summary>
|
||||
public const string HateUnfairness = "hate_unfairness";
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Internal helpers
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
private async Task<PollResult> PollEvalRunAsync(
|
||||
string evalId,
|
||||
string runId,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var deadline = DateTime.UtcNow.AddSeconds(this._timeoutSeconds);
|
||||
|
||||
while (true)
|
||||
{
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
|
||||
var result = await this._evaluationClient.GetEvaluationRunAsync(
|
||||
evalId,
|
||||
runId,
|
||||
new RequestOptions { CancellationToken = cancellationToken }).ConfigureAwait(false);
|
||||
|
||||
using var runDoc = JsonDocument.Parse(result.GetRawResponse().Content);
|
||||
var root = runDoc.RootElement;
|
||||
var status = root.GetProperty("status").GetString()!;
|
||||
|
||||
if (status is "completed" or "failed" or "canceled")
|
||||
{
|
||||
string? reportUrl = root.TryGetProperty("report_url", out var urlProp) ? urlProp.GetString() : null;
|
||||
string? errorMessage = root.TryGetProperty("error", out var errProp) ? errProp.ToString() : null;
|
||||
|
||||
// Extract per-evaluator breakdown
|
||||
Dictionary<string, PerEvaluatorResult>? perEvaluator = null;
|
||||
if (root.TryGetProperty("per_testing_criteria_results", out var criteriaArray)
|
||||
&& criteriaArray.ValueKind == JsonValueKind.Array)
|
||||
{
|
||||
perEvaluator = new Dictionary<string, PerEvaluatorResult>();
|
||||
foreach (var item in criteriaArray.EnumerateArray())
|
||||
{
|
||||
var name = item.TryGetProperty("testing_criteria", out var tcProp)
|
||||
? tcProp.GetString()
|
||||
: null;
|
||||
if (name is not null)
|
||||
{
|
||||
int passed = item.TryGetProperty("passed", out var pp) && pp.ValueKind == JsonValueKind.Number
|
||||
? pp.GetInt32() : 0;
|
||||
int failed = item.TryGetProperty("failed", out var fp) && fp.ValueKind == JsonValueKind.Number
|
||||
? fp.GetInt32() : 0;
|
||||
perEvaluator[name] = new PerEvaluatorResult(passed, failed);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return new PollResult(status, reportUrl, errorMessage, perEvaluator);
|
||||
}
|
||||
|
||||
if (DateTime.UtcNow >= deadline)
|
||||
{
|
||||
return new PollResult("timeout", null, null, null);
|
||||
}
|
||||
|
||||
await Task.Delay(TimeSpan.FromSeconds(this._pollIntervalSeconds), cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
|
||||
private sealed record PollResult(
|
||||
string Status,
|
||||
string? ReportUrl,
|
||||
string? ErrorMessage,
|
||||
Dictionary<string, PerEvaluatorResult>? PerEvaluator);
|
||||
|
||||
private async Task<FetchResult> FetchOutputItemResultsAsync(
|
||||
string evalId,
|
||||
string runId,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var meaiResults = new List<EvaluationResult>();
|
||||
var detailedItems = new List<EvalItemResult>();
|
||||
string? afterCursor = null;
|
||||
|
||||
while (true)
|
||||
{
|
||||
var response = await this._evaluationClient.GetEvaluationRunOutputItemsAsync(
|
||||
evalId,
|
||||
runId,
|
||||
limit: 100,
|
||||
order: null,
|
||||
after: afterCursor,
|
||||
outputItemStatus: null,
|
||||
new RequestOptions { CancellationToken = cancellationToken }).ConfigureAwait(false);
|
||||
|
||||
using var doc = JsonDocument.Parse(response.GetRawResponse().Content);
|
||||
|
||||
if (doc.RootElement.TryGetProperty("data", out var dataArray))
|
||||
{
|
||||
foreach (var outputItem in dataArray.EnumerateArray())
|
||||
{
|
||||
meaiResults.Add(ParseOutputItem(outputItem));
|
||||
detailedItems.Add(ParseDetailedItem(outputItem));
|
||||
}
|
||||
}
|
||||
|
||||
// Check for more pages
|
||||
bool hasMore = doc.RootElement.TryGetProperty("has_more", out var hasMoreProp)
|
||||
&& hasMoreProp.ValueKind == JsonValueKind.True;
|
||||
|
||||
if (!hasMore)
|
||||
{
|
||||
break;
|
||||
}
|
||||
|
||||
// Get cursor for next page — use last_id or last item's id
|
||||
if (doc.RootElement.TryGetProperty("last_id", out var lastIdProp))
|
||||
{
|
||||
afterCursor = lastIdProp.GetString();
|
||||
}
|
||||
else if (doc.RootElement.TryGetProperty("data", out var data2) && data2.GetArrayLength() > 0)
|
||||
{
|
||||
var lastItem = data2[data2.GetArrayLength() - 1];
|
||||
afterCursor = lastItem.TryGetProperty("id", out var idProp) ? idProp.GetString() : null;
|
||||
}
|
||||
|
||||
if (afterCursor is null)
|
||||
{
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
return new FetchResult(meaiResults, detailedItems);
|
||||
}
|
||||
|
||||
private sealed record FetchResult(
|
||||
List<EvaluationResult> MeaiResults,
|
||||
List<EvalItemResult> DetailedItems);
|
||||
|
||||
private static EvaluationResult ParseOutputItem(JsonElement outputItem)
|
||||
{
|
||||
var evalResult = new EvaluationResult();
|
||||
|
||||
if (outputItem.TryGetProperty("results", out var itemResults))
|
||||
{
|
||||
foreach (var r in itemResults.EnumerateArray())
|
||||
{
|
||||
var metricName = r.TryGetProperty("name", out var nameProp)
|
||||
? nameProp.GetString() ?? "unknown"
|
||||
: "unknown";
|
||||
|
||||
bool? passed = null;
|
||||
if (r.TryGetProperty("passed", out var passedProp)
|
||||
&& passedProp.ValueKind is JsonValueKind.True or JsonValueKind.False)
|
||||
{
|
||||
passed = passedProp.ValueKind == JsonValueKind.True;
|
||||
}
|
||||
|
||||
double? score = r.TryGetProperty("score", out var scoreProp) && scoreProp.ValueKind == JsonValueKind.Number
|
||||
? scoreProp.GetDouble()
|
||||
: null;
|
||||
|
||||
EvaluationMetricInterpretation? interpretation = passed.HasValue
|
||||
? new EvaluationMetricInterpretation
|
||||
{
|
||||
Rating = passed.Value ? EvaluationRating.Good : EvaluationRating.Unacceptable,
|
||||
Failed = !passed.Value,
|
||||
}
|
||||
: null;
|
||||
|
||||
if (score.HasValue)
|
||||
{
|
||||
evalResult.Metrics[metricName] = new NumericMetric(metricName, score.Value)
|
||||
{
|
||||
Interpretation = interpretation,
|
||||
};
|
||||
}
|
||||
else if (passed.HasValue)
|
||||
{
|
||||
evalResult.Metrics[metricName] = new BooleanMetric(metricName, passed.Value)
|
||||
{
|
||||
Interpretation = interpretation,
|
||||
};
|
||||
}
|
||||
|
||||
// When neither score nor passed is present, the evaluator returned no
|
||||
// actionable data (e.g. an error or informational entry). Skip the metric
|
||||
// so it doesn't falsely influence ItemPassed. The raw data is still
|
||||
// available in DetailedItems for diagnostics.
|
||||
}
|
||||
}
|
||||
|
||||
return evalResult;
|
||||
}
|
||||
|
||||
private static EvalItemResult ParseDetailedItem(JsonElement outputItem)
|
||||
{
|
||||
var itemId = outputItem.TryGetProperty("id", out var idProp) ? idProp.GetString() ?? "" : "";
|
||||
var status = outputItem.TryGetProperty("status", out var statusProp) ? statusProp.GetString() ?? "" : "";
|
||||
|
||||
var scores = new List<EvalScoreResult>();
|
||||
if (outputItem.TryGetProperty("results", out var itemResults))
|
||||
{
|
||||
foreach (var r in itemResults.EnumerateArray())
|
||||
{
|
||||
var name = r.TryGetProperty("name", out var np) ? np.GetString() ?? "unknown" : "unknown";
|
||||
double score = r.TryGetProperty("score", out var sp) && sp.ValueKind == JsonValueKind.Number
|
||||
? sp.GetDouble() : 0.0;
|
||||
bool? passed = null;
|
||||
if (r.TryGetProperty("passed", out var pp) && pp.ValueKind is JsonValueKind.True or JsonValueKind.False)
|
||||
{
|
||||
passed = pp.ValueKind == JsonValueKind.True;
|
||||
}
|
||||
|
||||
scores.Add(new EvalScoreResult(name, score, passed));
|
||||
}
|
||||
}
|
||||
|
||||
var result = new EvalItemResult(itemId, status, scores);
|
||||
|
||||
// Extract error info from sample
|
||||
if (outputItem.TryGetProperty("sample", out var sample))
|
||||
{
|
||||
if (sample.TryGetProperty("error", out var errObj))
|
||||
{
|
||||
result.ErrorCode = errObj.TryGetProperty("code", out var code) ? code.GetString() : null;
|
||||
result.ErrorMessage = errObj.TryGetProperty("message", out var msg) ? msg.GetString() : null;
|
||||
}
|
||||
|
||||
if (sample.TryGetProperty("usage", out var usage) && usage.TryGetProperty("total_tokens", out var tt) && tt.ValueKind == JsonValueKind.Number)
|
||||
{
|
||||
var tokenUsage = new Dictionary<string, int>();
|
||||
if (usage.TryGetProperty("prompt_tokens", out var pt) && pt.ValueKind == JsonValueKind.Number)
|
||||
{
|
||||
tokenUsage["prompt_tokens"] = pt.GetInt32();
|
||||
}
|
||||
|
||||
if (usage.TryGetProperty("completion_tokens", out var ct) && ct.ValueKind == JsonValueKind.Number)
|
||||
{
|
||||
tokenUsage["completion_tokens"] = ct.GetInt32();
|
||||
}
|
||||
|
||||
tokenUsage["total_tokens"] = tt.GetInt32();
|
||||
result.TokenUsage = tokenUsage;
|
||||
}
|
||||
|
||||
// Extract input/output text
|
||||
if (sample.TryGetProperty("input", out var inputArr) && inputArr.ValueKind == JsonValueKind.Array)
|
||||
{
|
||||
var parts = new List<string>();
|
||||
foreach (var si in inputArr.EnumerateArray())
|
||||
{
|
||||
if (si.TryGetProperty("role", out var role) && role.GetString() == "user"
|
||||
&& si.TryGetProperty("content", out var content))
|
||||
{
|
||||
parts.Add(content.GetString() ?? "");
|
||||
}
|
||||
}
|
||||
|
||||
if (parts.Count > 0)
|
||||
{
|
||||
result.InputText = string.Join(" ", parts);
|
||||
}
|
||||
}
|
||||
|
||||
if (sample.TryGetProperty("output", out var outputArr) && outputArr.ValueKind == JsonValueKind.Array)
|
||||
{
|
||||
var parts = new List<string>();
|
||||
foreach (var so in outputArr.EnumerateArray())
|
||||
{
|
||||
if (so.TryGetProperty("role", out var role) && role.GetString() == "assistant"
|
||||
&& so.TryGetProperty("content", out var content))
|
||||
{
|
||||
parts.Add(content.GetString() ?? "");
|
||||
}
|
||||
}
|
||||
|
||||
if (parts.Count > 0)
|
||||
{
|
||||
result.OutputText = string.Join(" ", parts);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Extract response_id from datasource_item
|
||||
if (outputItem.TryGetProperty("datasource_item", out var dsItem))
|
||||
{
|
||||
if (dsItem.TryGetProperty("resp_id", out var respId))
|
||||
{
|
||||
result.ResponseId = respId.GetString();
|
||||
}
|
||||
else if (dsItem.TryGetProperty("response_id", out var responseId))
|
||||
{
|
||||
result.ResponseId = responseId.GetString();
|
||||
}
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
internal static string[] FilterToolEvaluators(string[] evaluators, bool hasTools)
|
||||
{
|
||||
if (hasTools)
|
||||
{
|
||||
return evaluators;
|
||||
}
|
||||
|
||||
var filtered = Array.FindAll(evaluators, e =>
|
||||
!FoundryEvalConverter.ToolEvaluators.Contains(FoundryEvalConverter.ResolveEvaluator(e)));
|
||||
|
||||
return filtered.Length > 0
|
||||
? filtered
|
||||
: throw new ArgumentException(
|
||||
"All configured evaluators require tool definitions, but no tool calls were found in the eval items. "
|
||||
+ $"Tool evaluators: {string.Join(", ", evaluators)}. Either add tool call content to your EvalItems or remove tool-type evaluators.");
|
||||
}
|
||||
}
|
||||
@@ -28,6 +28,18 @@
|
||||
<PackageReference Include="OpenAI" />
|
||||
</ItemGroup>
|
||||
|
||||
<!-- Evaluation support requires net8.0+ (MEAI.Evaluation does not support legacy TFMs) -->
|
||||
<ItemGroup Condition="$([MSBuild]::IsTargetFrameworkCompatible('$(TargetFramework)', 'net8.0'))">
|
||||
<PackageReference Include="Microsoft.Extensions.AI.Evaluation" />
|
||||
</ItemGroup>
|
||||
<ItemGroup Condition="!$([MSBuild]::IsTargetFrameworkCompatible('$(TargetFramework)', 'net8.0'))">
|
||||
<Compile Remove="Evaluation\**\*.cs" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<InternalsVisibleTo Include="Microsoft.Agents.AI.UnitTests" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\Microsoft.Agents.AI\Microsoft.Agents.AI.csproj" />
|
||||
</ItemGroup>
|
||||
|
||||
@@ -0,0 +1,175 @@
|
||||
// Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using Microsoft.Extensions.AI;
|
||||
using Microsoft.Extensions.AI.Evaluation;
|
||||
|
||||
namespace Microsoft.Agents.AI.Workflows;
|
||||
|
||||
/// <summary>
|
||||
/// Extension methods for evaluating workflow runs.
|
||||
/// </summary>
|
||||
public static class WorkflowEvaluationExtensions
|
||||
{
|
||||
/// <summary>
|
||||
/// Evaluates a completed workflow run.
|
||||
/// </summary>
|
||||
/// <param name="run">The completed workflow run.</param>
|
||||
/// <param name="evaluator">The evaluator to score results.</param>
|
||||
/// <param name="includeOverall">Whether to include an overall evaluation.</param>
|
||||
/// <param name="includePerAgent">Whether to include per-agent breakdowns.</param>
|
||||
/// <param name="evalName">Display name for this evaluation run.</param>
|
||||
/// <param name="splitter">
|
||||
/// Optional conversation splitter to apply to all items.
|
||||
/// Use <see cref="ConversationSplitters.LastTurn"/>, <see cref="ConversationSplitters.Full"/>,
|
||||
/// or a custom <see cref="IConversationSplitter"/> implementation.
|
||||
/// </param>
|
||||
/// <param name="cancellationToken">Cancellation token.</param>
|
||||
/// <returns>Evaluation results with optional per-agent sub-results.</returns>
|
||||
public static async Task<AgentEvaluationResults> EvaluateAsync(
|
||||
this Run run,
|
||||
IAgentEvaluator evaluator,
|
||||
bool includeOverall = true,
|
||||
bool includePerAgent = true,
|
||||
string evalName = "Workflow Eval",
|
||||
IConversationSplitter? splitter = null,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
var events = run.OutgoingEvents.ToList();
|
||||
|
||||
// Extract per-agent data
|
||||
var agentData = ExtractAgentData(events, splitter);
|
||||
|
||||
// Build overall items from final output
|
||||
var overallItems = new List<EvalItem>();
|
||||
if (includeOverall)
|
||||
{
|
||||
var finalResponse = events.OfType<AgentResponseEvent>().LastOrDefault();
|
||||
if (finalResponse is not null)
|
||||
{
|
||||
var firstInvoked = events.OfType<ExecutorInvokedEvent>().FirstOrDefault();
|
||||
var query = firstInvoked?.Data switch
|
||||
{
|
||||
ChatMessage cm => cm.Text ?? string.Empty,
|
||||
IReadOnlyList<ChatMessage> msgs => msgs.LastOrDefault(m => m.Role == ChatRole.User)?.Text ?? string.Empty,
|
||||
string s => s,
|
||||
_ => firstInvoked?.Data?.ToString() ?? string.Empty,
|
||||
};
|
||||
var conversation = new List<ChatMessage>
|
||||
{
|
||||
new(ChatRole.User, query),
|
||||
};
|
||||
|
||||
conversation.AddRange(finalResponse.Response.Messages);
|
||||
|
||||
overallItems.Add(new EvalItem(query, finalResponse.Response.Text, conversation)
|
||||
{
|
||||
Splitter = splitter,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Evaluate overall
|
||||
var overallResult = overallItems.Count > 0
|
||||
? await evaluator.EvaluateAsync(overallItems, evalName, cancellationToken).ConfigureAwait(false)
|
||||
: new AgentEvaluationResults(evaluator.Name, Array.Empty<EvaluationResult>());
|
||||
|
||||
// Per-agent breakdown
|
||||
if (includePerAgent && agentData.Count > 0)
|
||||
{
|
||||
var subResults = new Dictionary<string, AgentEvaluationResults>();
|
||||
|
||||
foreach (var kvp in agentData)
|
||||
{
|
||||
subResults[kvp.Key] = await evaluator.EvaluateAsync(
|
||||
kvp.Value,
|
||||
$"{evalName} - {kvp.Key}",
|
||||
cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
overallResult.SubResults = subResults;
|
||||
}
|
||||
|
||||
return overallResult;
|
||||
}
|
||||
|
||||
internal static Dictionary<string, List<EvalItem>> ExtractAgentData(
|
||||
List<WorkflowEvent> events,
|
||||
IConversationSplitter? splitter)
|
||||
{
|
||||
var invoked = new Dictionary<string, ExecutorInvokedEvent>();
|
||||
var agentData = new Dictionary<string, List<EvalItem>>();
|
||||
|
||||
foreach (var evt in events)
|
||||
{
|
||||
if (evt is ExecutorInvokedEvent invokedEvent)
|
||||
{
|
||||
if (IsInternalExecutor(invokedEvent.ExecutorId))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
invoked[invokedEvent.ExecutorId] = invokedEvent;
|
||||
}
|
||||
else if (evt is ExecutorCompletedEvent completedEvent
|
||||
&& invoked.TryGetValue(completedEvent.ExecutorId, out var matchingInvoked))
|
||||
{
|
||||
var query = matchingInvoked.Data switch
|
||||
{
|
||||
ChatMessage cm => cm.Text ?? string.Empty,
|
||||
IReadOnlyList<ChatMessage> msgs => msgs.LastOrDefault(m => m.Role == ChatRole.User)?.Text ?? string.Empty,
|
||||
string s => s,
|
||||
_ => matchingInvoked.Data?.ToString() ?? string.Empty,
|
||||
};
|
||||
|
||||
var responseText = completedEvent.Data switch
|
||||
{
|
||||
AgentResponse ar => ar.Text,
|
||||
ChatMessage cm => cm.Text ?? string.Empty,
|
||||
string s => s,
|
||||
_ => completedEvent.Data?.ToString() ?? string.Empty,
|
||||
};
|
||||
var agentResponse = completedEvent.Data as AgentResponse;
|
||||
var conversation = new List<ChatMessage>
|
||||
{
|
||||
new(ChatRole.User, query),
|
||||
};
|
||||
|
||||
if (agentResponse is not null)
|
||||
{
|
||||
conversation.AddRange(agentResponse.Messages);
|
||||
}
|
||||
else
|
||||
{
|
||||
conversation.Add(new(ChatRole.Assistant, responseText));
|
||||
}
|
||||
|
||||
var item = new EvalItem(query, responseText, conversation)
|
||||
{
|
||||
Splitter = splitter,
|
||||
};
|
||||
|
||||
if (!agentData.TryGetValue(completedEvent.ExecutorId, out var items))
|
||||
{
|
||||
items = new List<EvalItem>();
|
||||
agentData[completedEvent.ExecutorId] = items;
|
||||
}
|
||||
|
||||
items.Add(item);
|
||||
invoked.Remove(completedEvent.ExecutorId);
|
||||
}
|
||||
}
|
||||
|
||||
return agentData;
|
||||
}
|
||||
|
||||
private static bool IsInternalExecutor(string executorId)
|
||||
{
|
||||
return executorId.StartsWith('_')
|
||||
|| executorId is "input-conversation" or "end-conversation" or "end";
|
||||
}
|
||||
}
|
||||
@@ -55,4 +55,9 @@
|
||||
<PackageReference Include="System.Diagnostics.DiagnosticSource" />
|
||||
</ItemGroup>
|
||||
|
||||
<!-- Evaluation support requires net8.0+ (MEAI.Evaluation does not support legacy TFMs) -->
|
||||
<ItemGroup Condition="!$([MSBuild]::IsTargetFrameworkCompatible('$(TargetFramework)', 'net8.0'))">
|
||||
<Compile Remove="Evaluation\**\*.cs" />
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
|
||||
@@ -0,0 +1,369 @@
|
||||
// Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using Microsoft.Extensions.AI;
|
||||
using Microsoft.Extensions.AI.Evaluation;
|
||||
|
||||
namespace Microsoft.Agents.AI;
|
||||
|
||||
/// <summary>
|
||||
/// Extension methods for evaluating agents, responses, and workflow runs.
|
||||
/// </summary>
|
||||
public static partial class AgentEvaluationExtensions
|
||||
{
|
||||
private const string DefaultEvalName = "AgentFrameworkEval";
|
||||
|
||||
/// <summary>
|
||||
/// Evaluates an agent by running it against test queries and scoring the responses.
|
||||
/// </summary>
|
||||
/// <param name="agent">The agent to evaluate.</param>
|
||||
/// <param name="queries">Test queries to send to the agent.</param>
|
||||
/// <param name="evaluator">The evaluator to score responses.</param>
|
||||
/// <param name="evalName">Display name for this evaluation run.</param>
|
||||
/// <param name="expectedOutput">
|
||||
/// Optional ground-truth expected outputs, one per query. When provided,
|
||||
/// must be the same length as <paramref name="queries"/>. Each value is
|
||||
/// stamped on the corresponding <see cref="EvalItem.ExpectedOutput"/>.
|
||||
/// </param>
|
||||
/// <param name="expectedToolCalls">
|
||||
/// Optional expected tool calls, one list per query. When provided,
|
||||
/// must be the same length as <paramref name="queries"/>. Each list is
|
||||
/// stamped on the corresponding <see cref="EvalItem.ExpectedToolCalls"/>.
|
||||
/// </param>
|
||||
/// <param name="splitter">
|
||||
/// Optional conversation splitter to apply to all items.
|
||||
/// Use <see cref="ConversationSplitters.LastTurn"/>, <see cref="ConversationSplitters.Full"/>,
|
||||
/// or a custom <see cref="IConversationSplitter"/> implementation.
|
||||
/// </param>
|
||||
/// <param name="numRepetitions">
|
||||
/// Number of times to run each query (default 1). When greater than 1, each query is invoked
|
||||
/// independently N times to measure consistency. Results contain all N × queries.Count items.
|
||||
/// </param>
|
||||
/// <param name="cancellationToken">Cancellation token.</param>
|
||||
/// <returns>Evaluation results.</returns>
|
||||
public static async Task<AgentEvaluationResults> EvaluateAsync(
|
||||
this AIAgent agent,
|
||||
IEnumerable<string> queries,
|
||||
IAgentEvaluator evaluator,
|
||||
string evalName = DefaultEvalName,
|
||||
IEnumerable<string>? expectedOutput = null,
|
||||
IEnumerable<IEnumerable<ExpectedToolCall>>? expectedToolCalls = null,
|
||||
IConversationSplitter? splitter = null,
|
||||
int numRepetitions = 1,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
var items = await RunAgentForEvalAsync(agent, queries, expectedOutput, expectedToolCalls, splitter, numRepetitions, cancellationToken).ConfigureAwait(false);
|
||||
return await evaluator.EvaluateAsync(items, evalName, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Evaluates an agent using an MEAI evaluator directly.
|
||||
/// </summary>
|
||||
/// <param name="agent">The agent to evaluate.</param>
|
||||
/// <param name="queries">Test queries to send to the agent.</param>
|
||||
/// <param name="evaluator">The MEAI evaluator (e.g., <c>RelevanceEvaluator</c>, <c>CompositeEvaluator</c>).</param>
|
||||
/// <param name="chatConfiguration">Chat configuration for the MEAI evaluator (includes the judge model).</param>
|
||||
/// <param name="evalName">Display name for this evaluation run.</param>
|
||||
/// <param name="expectedOutput">
|
||||
/// Optional ground-truth expected outputs, one per query.
|
||||
/// </param>
|
||||
/// <param name="expectedToolCalls">
|
||||
/// Optional expected tool calls, one list per query.
|
||||
/// </param>
|
||||
/// <param name="splitter">
|
||||
/// Optional conversation splitter to apply to all items.
|
||||
/// Use <see cref="ConversationSplitters.LastTurn"/>, <see cref="ConversationSplitters.Full"/>,
|
||||
/// or a custom <see cref="IConversationSplitter"/> implementation.
|
||||
/// </param>
|
||||
/// <param name="numRepetitions">
|
||||
/// Number of times to run each query (default 1). When greater than 1, each query is invoked
|
||||
/// independently N times to measure consistency.
|
||||
/// </param>
|
||||
/// <param name="cancellationToken">Cancellation token.</param>
|
||||
/// <returns>Evaluation results.</returns>
|
||||
public static async Task<AgentEvaluationResults> EvaluateAsync(
|
||||
this AIAgent agent,
|
||||
IEnumerable<string> queries,
|
||||
IEvaluator evaluator,
|
||||
ChatConfiguration chatConfiguration,
|
||||
string evalName = DefaultEvalName,
|
||||
IEnumerable<string>? expectedOutput = null,
|
||||
IEnumerable<IEnumerable<ExpectedToolCall>>? expectedToolCalls = null,
|
||||
IConversationSplitter? splitter = null,
|
||||
int numRepetitions = 1,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
var wrapped = new MeaiEvaluatorAdapter(evaluator, chatConfiguration);
|
||||
return await agent.EvaluateAsync(queries, wrapped, evalName, expectedOutput, expectedToolCalls, splitter, numRepetitions, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Evaluates an agent by running it against test queries with multiple evaluators.
|
||||
/// </summary>
|
||||
/// <param name="agent">The agent to evaluate.</param>
|
||||
/// <param name="queries">Test queries to send to the agent.</param>
|
||||
/// <param name="evaluators">The evaluators to score responses.</param>
|
||||
/// <param name="evalName">Display name for this evaluation run.</param>
|
||||
/// <param name="expectedOutput">
|
||||
/// Optional ground-truth expected outputs, one per query.
|
||||
/// </param>
|
||||
/// <param name="expectedToolCalls">
|
||||
/// Optional expected tool calls, one list per query.
|
||||
/// </param>
|
||||
/// <param name="splitter">
|
||||
/// Optional conversation splitter to apply to all items.
|
||||
/// Use <see cref="ConversationSplitters.LastTurn"/>, <see cref="ConversationSplitters.Full"/>,
|
||||
/// or a custom <see cref="IConversationSplitter"/> implementation.
|
||||
/// </param>
|
||||
/// <param name="numRepetitions">
|
||||
/// Number of times to run each query (default 1). When greater than 1, each query is invoked
|
||||
/// independently N times to measure consistency.
|
||||
/// </param>
|
||||
/// <param name="cancellationToken">Cancellation token.</param>
|
||||
/// <returns>One result per evaluator.</returns>
|
||||
public static async Task<IReadOnlyList<AgentEvaluationResults>> EvaluateAsync(
|
||||
this AIAgent agent,
|
||||
IEnumerable<string> queries,
|
||||
IEnumerable<IAgentEvaluator> evaluators,
|
||||
string evalName = DefaultEvalName,
|
||||
IEnumerable<string>? expectedOutput = null,
|
||||
IEnumerable<IEnumerable<ExpectedToolCall>>? expectedToolCalls = null,
|
||||
IConversationSplitter? splitter = null,
|
||||
int numRepetitions = 1,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
var items = await RunAgentForEvalAsync(agent, queries, expectedOutput, expectedToolCalls, splitter, numRepetitions, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
var results = new List<AgentEvaluationResults>();
|
||||
foreach (var evaluator in evaluators)
|
||||
{
|
||||
var result = await evaluator.EvaluateAsync(items, evalName, cancellationToken).ConfigureAwait(false);
|
||||
results.Add(result);
|
||||
}
|
||||
|
||||
return results;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Evaluates pre-existing agent responses without re-running the agent.
|
||||
/// </summary>
|
||||
/// <param name="agent">The agent (used for tool definitions).</param>
|
||||
/// <param name="responses">Pre-existing agent responses.</param>
|
||||
/// <param name="queries">The queries that produced each response (must match count).</param>
|
||||
/// <param name="evaluator">The evaluator to score responses.</param>
|
||||
/// <param name="evalName">Display name for this evaluation run.</param>
|
||||
/// <param name="expectedOutput">
|
||||
/// Optional ground-truth expected outputs, one per query.
|
||||
/// </param>
|
||||
/// <param name="expectedToolCalls">
|
||||
/// Optional expected tool calls, one list per query.
|
||||
/// </param>
|
||||
/// <param name="cancellationToken">Cancellation token.</param>
|
||||
/// <returns>Evaluation results.</returns>
|
||||
public static async Task<AgentEvaluationResults> EvaluateAsync(
|
||||
this AIAgent agent,
|
||||
IEnumerable<AgentResponse> responses,
|
||||
IEnumerable<string> queries,
|
||||
IAgentEvaluator evaluator,
|
||||
string evalName = DefaultEvalName,
|
||||
IEnumerable<string>? expectedOutput = null,
|
||||
IEnumerable<IEnumerable<ExpectedToolCall>>? expectedToolCalls = null,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
var items = BuildItemsFromResponses(agent, responses, queries, expectedOutput, expectedToolCalls);
|
||||
return await evaluator.EvaluateAsync(items, evalName, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Evaluates pre-existing agent responses using an MEAI evaluator directly.
|
||||
/// </summary>
|
||||
/// <param name="agent">The agent (used for tool definitions).</param>
|
||||
/// <param name="responses">Pre-existing agent responses.</param>
|
||||
/// <param name="queries">The queries that produced each response (must match count).</param>
|
||||
/// <param name="evaluator">The MEAI evaluator.</param>
|
||||
/// <param name="chatConfiguration">Chat configuration for the MEAI evaluator.</param>
|
||||
/// <param name="evalName">Display name for this evaluation run.</param>
|
||||
/// <param name="expectedOutput">
|
||||
/// Optional ground-truth expected outputs, one per query.
|
||||
/// </param>
|
||||
/// <param name="expectedToolCalls">
|
||||
/// Optional expected tool calls, one list per query.
|
||||
/// </param>
|
||||
/// <param name="cancellationToken">Cancellation token.</param>
|
||||
/// <returns>Evaluation results.</returns>
|
||||
public static async Task<AgentEvaluationResults> EvaluateAsync(
|
||||
this AIAgent agent,
|
||||
IEnumerable<AgentResponse> responses,
|
||||
IEnumerable<string> queries,
|
||||
IEvaluator evaluator,
|
||||
ChatConfiguration chatConfiguration,
|
||||
string evalName = DefaultEvalName,
|
||||
IEnumerable<string>? expectedOutput = null,
|
||||
IEnumerable<IEnumerable<ExpectedToolCall>>? expectedToolCalls = null,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
var wrapped = new MeaiEvaluatorAdapter(evaluator, chatConfiguration);
|
||||
return await agent.EvaluateAsync(responses, queries, wrapped, evalName, expectedOutput, expectedToolCalls, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
internal static List<EvalItem> BuildItemsFromResponses(
|
||||
AIAgent agent,
|
||||
IEnumerable<AgentResponse> responses,
|
||||
IEnumerable<string> queries,
|
||||
IEnumerable<string>? expectedOutput,
|
||||
IEnumerable<IEnumerable<ExpectedToolCall>>? expectedToolCalls)
|
||||
{
|
||||
var responseList = responses.ToList();
|
||||
var queryList = queries.ToList();
|
||||
var expectedList = expectedOutput?.ToList();
|
||||
var expectedToolCallsList = expectedToolCalls?.ToList();
|
||||
|
||||
if (responseList.Count != queryList.Count)
|
||||
{
|
||||
throw new ArgumentException(
|
||||
$"Found {queryList.Count} queries but {responseList.Count} responses. Counts must match.");
|
||||
}
|
||||
|
||||
if (expectedList != null && expectedList.Count != queryList.Count)
|
||||
{
|
||||
throw new ArgumentException(
|
||||
$"Found {queryList.Count} queries but {expectedList.Count} expectedOutput values. Counts must match.");
|
||||
}
|
||||
|
||||
if (expectedToolCallsList != null && expectedToolCallsList.Count != queryList.Count)
|
||||
{
|
||||
throw new ArgumentException(
|
||||
$"Found {queryList.Count} queries but {expectedToolCallsList.Count} expectedToolCalls lists. Counts must match.");
|
||||
}
|
||||
|
||||
var items = new List<EvalItem>();
|
||||
for (int i = 0; i < responseList.Count; i++)
|
||||
{
|
||||
var query = queryList[i];
|
||||
var response = responseList[i];
|
||||
|
||||
var messages = new List<ChatMessage>
|
||||
{
|
||||
new(ChatRole.User, query),
|
||||
};
|
||||
messages.AddRange(response.Messages);
|
||||
|
||||
var item = BuildEvalItem(query, response, messages, agent);
|
||||
if (expectedList != null)
|
||||
{
|
||||
item.ExpectedOutput = expectedList[i];
|
||||
}
|
||||
|
||||
if (expectedToolCallsList != null)
|
||||
{
|
||||
item.ExpectedToolCalls = expectedToolCallsList[i].ToList();
|
||||
}
|
||||
|
||||
items.Add(item);
|
||||
}
|
||||
|
||||
return items;
|
||||
}
|
||||
|
||||
private static async Task<List<EvalItem>> RunAgentForEvalAsync(
|
||||
AIAgent agent,
|
||||
IEnumerable<string> queries,
|
||||
IEnumerable<string>? expectedOutput,
|
||||
IEnumerable<IEnumerable<ExpectedToolCall>>? expectedToolCalls,
|
||||
IConversationSplitter? splitter,
|
||||
int numRepetitions,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
if (numRepetitions < 1)
|
||||
{
|
||||
throw new ArgumentException($"numRepetitions must be >= 1, got {numRepetitions}.", nameof(numRepetitions));
|
||||
}
|
||||
|
||||
var items = new List<EvalItem>();
|
||||
var queryList = queries.ToList();
|
||||
var expectedList = expectedOutput?.ToList();
|
||||
var expectedToolCallsList = expectedToolCalls?.ToList();
|
||||
|
||||
if (expectedList != null && expectedList.Count != queryList.Count)
|
||||
{
|
||||
throw new ArgumentException(
|
||||
$"Got {queryList.Count} queries but {expectedList.Count} expectedOutput values. Counts must match.");
|
||||
}
|
||||
|
||||
if (expectedToolCallsList != null && expectedToolCallsList.Count != queryList.Count)
|
||||
{
|
||||
throw new ArgumentException(
|
||||
$"Got {queryList.Count} queries but {expectedToolCallsList.Count} expectedToolCalls lists. Counts must match.");
|
||||
}
|
||||
|
||||
for (int rep = 0; rep < numRepetitions; rep++)
|
||||
{
|
||||
for (int i = 0; i < queryList.Count; i++)
|
||||
{
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
|
||||
var query = queryList[i];
|
||||
var messages = new List<ChatMessage>
|
||||
{
|
||||
new(ChatRole.User, query),
|
||||
};
|
||||
|
||||
var response = await agent.RunAsync(messages, cancellationToken: cancellationToken).ConfigureAwait(false);
|
||||
var item = BuildEvalItem(query, response, messages, agent);
|
||||
item.Splitter = splitter;
|
||||
if (expectedList != null)
|
||||
{
|
||||
item.ExpectedOutput = expectedList[i];
|
||||
}
|
||||
|
||||
if (expectedToolCallsList != null)
|
||||
{
|
||||
item.ExpectedToolCalls = expectedToolCallsList[i].ToList();
|
||||
}
|
||||
|
||||
items.Add(item);
|
||||
}
|
||||
}
|
||||
|
||||
return items;
|
||||
}
|
||||
|
||||
internal static EvalItem BuildEvalItem(
|
||||
string query,
|
||||
AgentResponse response,
|
||||
List<ChatMessage> messages,
|
||||
AIAgent? agent)
|
||||
{
|
||||
// Build conversation from existing messages plus any new response messages
|
||||
var conversation = new List<ChatMessage>(messages);
|
||||
foreach (var msg in response.Messages)
|
||||
{
|
||||
if (!conversation.Contains(msg))
|
||||
{
|
||||
conversation.Add(msg);
|
||||
}
|
||||
}
|
||||
|
||||
var item = new EvalItem(query, response.Text, conversation)
|
||||
{
|
||||
RawResponse = new ChatResponse(response.Messages.LastOrDefault()
|
||||
?? new ChatMessage(ChatRole.Assistant, response.Text)),
|
||||
};
|
||||
|
||||
// Extract tool definitions from the agent (mirrors Python's to_eval_item(agent=...))
|
||||
if (agent is not null)
|
||||
{
|
||||
var chatOptions = agent.GetService<ChatOptions>();
|
||||
if (chatOptions?.Tools is { Count: > 0 } tools)
|
||||
{
|
||||
item.Tools = tools.ToList().AsReadOnly();
|
||||
}
|
||||
}
|
||||
|
||||
return item;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,143 @@
|
||||
// Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using Microsoft.Extensions.AI.Evaluation;
|
||||
|
||||
namespace Microsoft.Agents.AI;
|
||||
|
||||
/// <summary>
|
||||
/// Aggregate evaluation results across multiple items.
|
||||
/// </summary>
|
||||
public sealed class AgentEvaluationResults
|
||||
{
|
||||
private readonly List<EvaluationResult> _items;
|
||||
|
||||
/// <summary>
|
||||
/// Initializes a new instance of the <see cref="AgentEvaluationResults"/> class.
|
||||
/// </summary>
|
||||
/// <param name="providerName">Name of the evaluation provider.</param>
|
||||
/// <param name="items">Per-item MEAI evaluation results.</param>
|
||||
/// <param name="inputItems">The original eval items that were evaluated, for auditing.</param>
|
||||
public AgentEvaluationResults(string providerName, IEnumerable<EvaluationResult> items, IReadOnlyList<EvalItem>? inputItems = null)
|
||||
{
|
||||
this.ProviderName = providerName;
|
||||
this._items = new List<EvaluationResult>(items);
|
||||
this.InputItems = inputItems;
|
||||
}
|
||||
|
||||
/// <summary>Gets the evaluation provider name.</summary>
|
||||
public string ProviderName { get; }
|
||||
|
||||
/// <summary>Gets the portal URL for viewing results (Foundry only).</summary>
|
||||
public Uri? ReportUrl { get; set; }
|
||||
|
||||
/// <summary>Gets the Foundry evaluation ID (Foundry only).</summary>
|
||||
public string? EvalId { get; set; }
|
||||
|
||||
/// <summary>Gets the Foundry evaluation run ID (Foundry only).</summary>
|
||||
public string? RunId { get; set; }
|
||||
|
||||
/// <summary>Gets the evaluation run status (e.g., "completed", "failed", "canceled", "timeout").</summary>
|
||||
public string? Status { get; set; }
|
||||
|
||||
/// <summary>Gets error details when the evaluation run failed.</summary>
|
||||
public string? Error { get; set; }
|
||||
|
||||
/// <summary>Gets the per-item MEAI evaluation results.</summary>
|
||||
public IReadOnlyList<EvaluationResult> Items => this._items;
|
||||
|
||||
/// <summary>
|
||||
/// Gets the original eval items that produced these results, for auditing.
|
||||
/// Each entry corresponds positionally to <see cref="Items"/> — <c>InputItems[i]</c>
|
||||
/// is the query/response that produced <c>Items[i]</c>.
|
||||
/// </summary>
|
||||
public IReadOnlyList<EvalItem>? InputItems { get; }
|
||||
|
||||
/// <summary>Gets per-agent results for workflow evaluations.</summary>
|
||||
public IReadOnlyDictionary<string, AgentEvaluationResults>? SubResults { get; set; }
|
||||
|
||||
/// <summary>Gets per-evaluator pass/fail breakdown (Foundry only).</summary>
|
||||
public IReadOnlyDictionary<string, PerEvaluatorResult>? PerEvaluator { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Gets detailed per-item results from the Foundry output_items API,
|
||||
/// including individual evaluator scores, error info, and token usage.
|
||||
/// </summary>
|
||||
public IReadOnlyList<EvalItemResult>? DetailedItems { get; set; }
|
||||
|
||||
/// <summary>Gets the number of items that passed.</summary>
|
||||
public int Passed => this._items.Count(ItemPassed);
|
||||
|
||||
/// <summary>Gets the number of items that failed.</summary>
|
||||
public int Failed => this._items.Count(i => !ItemPassed(i));
|
||||
|
||||
/// <summary>Gets the total number of items evaluated.</summary>
|
||||
public int Total => this._items.Count;
|
||||
|
||||
/// <summary>Gets whether all items passed.</summary>
|
||||
public bool AllPassed
|
||||
{
|
||||
get
|
||||
{
|
||||
if (this.SubResults is not null)
|
||||
{
|
||||
return this.SubResults.Values.All(s => s.AllPassed)
|
||||
&& (this.Total == 0 || this.Failed == 0);
|
||||
}
|
||||
|
||||
return this.Total > 0 && this.Failed == 0;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Asserts that all items passed. Throws <see cref="InvalidOperationException"/> on failure.
|
||||
/// </summary>
|
||||
/// <param name="message">Optional custom failure message.</param>
|
||||
/// <exception cref="InvalidOperationException">Thrown when any items failed.</exception>
|
||||
public void AssertAllPassed(string? message = null)
|
||||
{
|
||||
if (!this.AllPassed)
|
||||
{
|
||||
var detail = message ?? $"{this.ProviderName}: {this.Passed} passed, {this.Failed} failed out of {this.Total}.";
|
||||
if (this.ReportUrl is not null)
|
||||
{
|
||||
detail += $" See {this.ReportUrl} for details.";
|
||||
}
|
||||
|
||||
if (this.SubResults is not null)
|
||||
{
|
||||
var failedAgents = this.SubResults
|
||||
.Where(kvp => !kvp.Value.AllPassed)
|
||||
.Select(kvp => kvp.Key);
|
||||
detail += $" Failed agents: {string.Join(", ", failedAgents)}.";
|
||||
}
|
||||
|
||||
throw new InvalidOperationException(detail);
|
||||
}
|
||||
}
|
||||
|
||||
private static bool ItemPassed(EvaluationResult result)
|
||||
{
|
||||
foreach (var metric in result.Metrics.Values)
|
||||
{
|
||||
// Trust the evaluator's own pass/fail determination first.
|
||||
if (metric.Interpretation?.Failed == true)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
// A boolean false is unambiguous — the check failed.
|
||||
if (metric is BooleanMetric boolean && boolean.Value == false)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
// Numeric metrics without Interpretation are informational scores;
|
||||
// the evaluator should set Interpretation if it wants pass/fail semantics.
|
||||
}
|
||||
|
||||
return result.Metrics.Count > 0;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
// Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
namespace Microsoft.Agents.AI;
|
||||
|
||||
/// <summary>
|
||||
/// Result of a single check on a single evaluation item.
|
||||
/// </summary>
|
||||
/// <param name="Passed">Whether the check passed.</param>
|
||||
/// <param name="Reason">Human-readable explanation.</param>
|
||||
/// <param name="CheckName">Name of the check that produced this result.</param>
|
||||
public sealed record EvalCheckResult(bool Passed, string Reason, string CheckName);
|
||||
@@ -0,0 +1,10 @@
|
||||
// Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
namespace Microsoft.Agents.AI;
|
||||
|
||||
/// <summary>
|
||||
/// Delegate for a synchronous evaluation check on a single item.
|
||||
/// </summary>
|
||||
/// <param name="item">The evaluation item.</param>
|
||||
/// <returns>The check result.</returns>
|
||||
public delegate EvalCheckResult EvalCheck(EvalItem item);
|
||||
@@ -0,0 +1,328 @@
|
||||
// Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using System.Text.Json;
|
||||
using Microsoft.Extensions.AI;
|
||||
|
||||
namespace Microsoft.Agents.AI;
|
||||
|
||||
/// <summary>
|
||||
/// Specifies how <see cref="EvalChecks.ToolCalledCheck(ToolCalledMode, string[])"/> matches tool names.
|
||||
/// </summary>
|
||||
public enum ToolCalledMode
|
||||
{
|
||||
/// <summary>All specified tools must have been called.</summary>
|
||||
All,
|
||||
|
||||
/// <summary>At least one of the specified tools must have been called.</summary>
|
||||
Any,
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Built-in check functions for common evaluation patterns.
|
||||
/// </summary>
|
||||
public static class EvalChecks
|
||||
{
|
||||
/// <summary>
|
||||
/// Creates a check that verifies the response contains all specified keywords.
|
||||
/// </summary>
|
||||
/// <param name="keywords">Keywords that must appear in the response.</param>
|
||||
/// <returns>An <see cref="EvalCheck"/> delegate.</returns>
|
||||
public static EvalCheck KeywordCheck(params string[] keywords)
|
||||
{
|
||||
return KeywordCheck(caseSensitive: false, keywords);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Creates a check that verifies the response contains all specified keywords.
|
||||
/// </summary>
|
||||
/// <param name="caseSensitive">Whether the comparison is case-sensitive.</param>
|
||||
/// <param name="keywords">Keywords that must appear in the response.</param>
|
||||
/// <returns>An <see cref="EvalCheck"/> delegate.</returns>
|
||||
public static EvalCheck KeywordCheck(bool caseSensitive, params string[] keywords)
|
||||
{
|
||||
return (EvalItem item) =>
|
||||
{
|
||||
var comparison = caseSensitive
|
||||
? StringComparison.Ordinal
|
||||
: StringComparison.OrdinalIgnoreCase;
|
||||
|
||||
var missing = keywords
|
||||
.Where(kw => !item.Response.Contains(kw, comparison))
|
||||
.ToList();
|
||||
|
||||
var passed = missing.Count == 0;
|
||||
var reason = passed
|
||||
? $"All keywords found: {string.Join(", ", keywords)}"
|
||||
: $"Missing keywords: {string.Join(", ", missing)}";
|
||||
|
||||
return new EvalCheckResult(passed, reason, "keyword_check");
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Creates a check that verifies specific tools were called in the conversation.
|
||||
/// All specified tools must have been called.
|
||||
/// </summary>
|
||||
/// <param name="toolNames">Tool names that must appear in the conversation.</param>
|
||||
/// <returns>An <see cref="EvalCheck"/> delegate.</returns>
|
||||
public static EvalCheck ToolCalledCheck(params string[] toolNames)
|
||||
{
|
||||
return ToolCalledCheck(ToolCalledMode.All, toolNames);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Creates a check that verifies specific tools were called in the conversation.
|
||||
/// </summary>
|
||||
/// <param name="mode">Whether <see cref="ToolCalledMode.All"/> or <see cref="ToolCalledMode.Any"/> of the specified tools must be called.</param>
|
||||
/// <param name="toolNames">Tool names to check for.</param>
|
||||
/// <returns>An <see cref="EvalCheck"/> delegate.</returns>
|
||||
public static EvalCheck ToolCalledCheck(ToolCalledMode mode, params string[] toolNames)
|
||||
{
|
||||
return (EvalItem item) =>
|
||||
{
|
||||
var calledTools = GetCalledTools(item);
|
||||
|
||||
if (mode == ToolCalledMode.Any)
|
||||
{
|
||||
var found = toolNames.Where(t => calledTools.Contains(t)).ToList();
|
||||
var passed = found.Count > 0;
|
||||
var reason = passed
|
||||
? $"Called: {string.Join(", ", found)}"
|
||||
: $"None of expected tools called: {string.Join(", ", toolNames)}";
|
||||
return new EvalCheckResult(passed, reason, "tool_called_check");
|
||||
}
|
||||
|
||||
var missing = toolNames.Where(t => !calledTools.Contains(t)).ToList();
|
||||
var allPassed = missing.Count == 0;
|
||||
var allReason = allPassed
|
||||
? $"All tools called: {string.Join(", ", toolNames)}"
|
||||
: $"Missing tool calls: {string.Join(", ", missing)}";
|
||||
|
||||
return new EvalCheckResult(allPassed, allReason, "tool_called_check");
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A check that verifies at least one tool was called in the conversation.
|
||||
/// </summary>
|
||||
/// <returns>An <see cref="EvalCheck"/> delegate.</returns>
|
||||
public static EvalCheck ToolCallsPresent()
|
||||
{
|
||||
return (EvalItem item) =>
|
||||
{
|
||||
var calledTools = GetCalledTools(item);
|
||||
var passed = calledTools.Count > 0;
|
||||
var reason = passed
|
||||
? $"Tools called: {string.Join(", ", calledTools)}"
|
||||
: "No tool calls found in conversation";
|
||||
|
||||
return new EvalCheckResult(passed, reason, "tool_calls_present");
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A check that verifies expected tool calls match on name and optionally arguments.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// For each expected tool call, finds matching calls in the conversation by name.
|
||||
/// If <see cref="ExpectedToolCall.Arguments"/> is provided, checks that the actual
|
||||
/// arguments contain all expected key-value pairs (subset match — extra actual arguments are OK).
|
||||
/// </para>
|
||||
/// <para>If no expected tool calls are set on the item, the check passes.</para>
|
||||
/// </remarks>
|
||||
/// <returns>An <see cref="EvalCheck"/> delegate.</returns>
|
||||
public static EvalCheck ToolCallArgsMatch()
|
||||
{
|
||||
return (EvalItem item) =>
|
||||
{
|
||||
var expected = item.ExpectedToolCalls;
|
||||
if (expected is null || expected.Count == 0)
|
||||
{
|
||||
return new EvalCheckResult(true, "No expected tool calls specified.", "tool_call_args_match");
|
||||
}
|
||||
|
||||
var actualCalls = GetCalledToolsWithArgs(item);
|
||||
int matched = 0;
|
||||
var details = new List<string>();
|
||||
|
||||
foreach (var exp in expected)
|
||||
{
|
||||
var matching = actualCalls.Where(c => string.Equals(c.Name, exp.Name, StringComparison.OrdinalIgnoreCase)).ToList();
|
||||
|
||||
if (matching.Count == 0)
|
||||
{
|
||||
details.Add($" {exp.Name}: not called");
|
||||
continue;
|
||||
}
|
||||
|
||||
if (exp.Arguments is null)
|
||||
{
|
||||
matched++;
|
||||
details.Add($" {exp.Name}: called (args not checked)");
|
||||
continue;
|
||||
}
|
||||
|
||||
// Subset match — all expected keys present with expected values
|
||||
bool found = false;
|
||||
foreach (var call in matching)
|
||||
{
|
||||
if (call.Arguments is not null
|
||||
&& exp.Arguments.All(kvp =>
|
||||
call.Arguments.TryGetValue(kvp.Key, out var actual)
|
||||
&& Equals(actual, kvp.Value)))
|
||||
{
|
||||
found = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if (found)
|
||||
{
|
||||
matched++;
|
||||
details.Add($" {exp.Name}: args match");
|
||||
}
|
||||
else
|
||||
{
|
||||
details.Add($" {exp.Name}: args mismatch");
|
||||
}
|
||||
}
|
||||
|
||||
var passed = matched == expected.Count;
|
||||
var reason = $"Tool call args match: {matched}/{expected.Count}\n{string.Join("\n", details)}";
|
||||
return new EvalCheckResult(passed, reason, "tool_call_args_match");
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Creates a check that verifies the response is non-empty and meets a minimum length.
|
||||
/// </summary>
|
||||
/// <param name="minLength">Minimum response length (default 1).</param>
|
||||
/// <returns>An <see cref="EvalCheck"/> delegate.</returns>
|
||||
public static EvalCheck NonEmpty(int minLength = 1)
|
||||
{
|
||||
return (EvalItem item) =>
|
||||
{
|
||||
var trimmed = item.Response.Trim();
|
||||
var passed = trimmed.Length >= minLength;
|
||||
var reason = passed
|
||||
? $"Response length {trimmed.Length} meets minimum {minLength}"
|
||||
: $"Response length {trimmed.Length} is below minimum {minLength}";
|
||||
|
||||
return new EvalCheckResult(passed, reason, "non_empty");
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Creates a check that verifies the response contains the expected output text.
|
||||
/// </summary>
|
||||
/// <param name="caseSensitive">Whether the comparison is case-sensitive (default false).</param>
|
||||
/// <returns>An <see cref="EvalCheck"/> delegate.</returns>
|
||||
public static EvalCheck ContainsExpected(bool caseSensitive = false)
|
||||
{
|
||||
return (EvalItem item) =>
|
||||
{
|
||||
if (string.IsNullOrEmpty(item.ExpectedOutput))
|
||||
{
|
||||
return new EvalCheckResult(false, "ExpectedOutput is not set; check cannot be applied.", "contains_expected");
|
||||
}
|
||||
|
||||
var comparison = caseSensitive
|
||||
? StringComparison.Ordinal
|
||||
: StringComparison.OrdinalIgnoreCase;
|
||||
|
||||
var passed = item.Response.Contains(item.ExpectedOutput, comparison);
|
||||
var reason = passed
|
||||
? $"Response contains expected output: \"{item.ExpectedOutput}\""
|
||||
: $"Response does not contain expected output: \"{item.ExpectedOutput}\"";
|
||||
|
||||
return new EvalCheckResult(passed, reason, "contains_expected");
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A check that verifies the conversation contains at least one image
|
||||
/// (<see cref="DataContent"/> or <see cref="UriContent"/> with an image media type).
|
||||
/// </summary>
|
||||
/// <returns>An <see cref="EvalCheck"/> delegate.</returns>
|
||||
public static EvalCheck HasImageContent()
|
||||
{
|
||||
return (EvalItem item) =>
|
||||
{
|
||||
var passed = item.HasImageContent;
|
||||
var reason = passed
|
||||
? "Conversation contains image content"
|
||||
: "No image content found in conversation";
|
||||
|
||||
return new EvalCheckResult(passed, reason, "has_image_content");
|
||||
};
|
||||
}
|
||||
|
||||
private static HashSet<string> GetCalledTools(EvalItem item)
|
||||
{
|
||||
var calledTools = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
foreach (var message in item.Conversation)
|
||||
{
|
||||
foreach (var content in message.Contents)
|
||||
{
|
||||
if (content is FunctionCallContent functionCall)
|
||||
{
|
||||
calledTools.Add(functionCall.Name);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return calledTools;
|
||||
}
|
||||
|
||||
private static List<(string Name, IReadOnlyDictionary<string, object>? Arguments)> GetCalledToolsWithArgs(EvalItem item)
|
||||
{
|
||||
var calls = new List<(string Name, IReadOnlyDictionary<string, object>? Arguments)>();
|
||||
|
||||
foreach (var message in item.Conversation)
|
||||
{
|
||||
foreach (var content in message.Contents)
|
||||
{
|
||||
if (content is FunctionCallContent functionCall)
|
||||
{
|
||||
IDictionary<string, object?>? rawArgs = functionCall.Arguments;
|
||||
IReadOnlyDictionary<string, object>? args = null;
|
||||
if (rawArgs is not null)
|
||||
{
|
||||
var dict = new Dictionary<string, object>(StringComparer.OrdinalIgnoreCase);
|
||||
foreach (var kvp in rawArgs)
|
||||
{
|
||||
if (kvp.Value is not null)
|
||||
{
|
||||
// Normalize JsonElement values to their .NET equivalents for comparison
|
||||
dict[kvp.Key] = kvp.Value is JsonElement je ? UnwrapJsonElement(je) : kvp.Value;
|
||||
}
|
||||
}
|
||||
|
||||
args = dict;
|
||||
}
|
||||
|
||||
calls.Add((functionCall.Name, args));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return calls;
|
||||
}
|
||||
|
||||
private static object UnwrapJsonElement(JsonElement element)
|
||||
{
|
||||
return element.ValueKind switch
|
||||
{
|
||||
JsonValueKind.String => element.GetString()!,
|
||||
JsonValueKind.Number => element.TryGetInt64(out var l) ? l : element.GetDouble(),
|
||||
JsonValueKind.True => true,
|
||||
JsonValueKind.False => false,
|
||||
_ => element.ToString(),
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,211 @@
|
||||
// Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using Microsoft.Extensions.AI;
|
||||
|
||||
namespace Microsoft.Agents.AI;
|
||||
|
||||
/// <summary>
|
||||
/// Provider-agnostic data for a single evaluation item.
|
||||
/// </summary>
|
||||
public sealed class EvalItem
|
||||
{
|
||||
/// <summary>
|
||||
/// Initializes a new instance of the <see cref="EvalItem"/> class.
|
||||
/// </summary>
|
||||
/// <param name="query">The user query.</param>
|
||||
/// <param name="response">The agent response text.</param>
|
||||
/// <param name="conversation">The full conversation as <see cref="ChatMessage"/> list.</param>
|
||||
public EvalItem(string query, string response, IReadOnlyList<ChatMessage> conversation)
|
||||
{
|
||||
this.Query = query;
|
||||
this.Response = response;
|
||||
this.Conversation = conversation;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Initializes a new instance of the <see cref="EvalItem"/> class from a conversation,
|
||||
/// deriving query and response text via the default splitter.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Use this constructor when the conversation contains multimodal content (images, etc.)
|
||||
/// that can't be represented as plain text. The query is extracted from the last user
|
||||
/// message text, and the response from the last assistant message text.
|
||||
/// </remarks>
|
||||
/// <param name="conversation">The full conversation as <see cref="ChatMessage"/> list.</param>
|
||||
/// <param name="splitter">
|
||||
/// Optional splitter to determine query/response boundaries.
|
||||
/// Defaults to <see cref="ConversationSplitters.LastTurn"/>.
|
||||
/// </param>
|
||||
public EvalItem(IReadOnlyList<ChatMessage> conversation, IConversationSplitter? splitter = null)
|
||||
{
|
||||
this.Conversation = conversation;
|
||||
this.Splitter = splitter;
|
||||
|
||||
var effective = splitter ?? ConversationSplitters.LastTurn;
|
||||
var (queryMessages, responseMessages) = effective.Split(conversation);
|
||||
|
||||
this.Query = queryMessages.LastOrDefault(m => m.Role == ChatRole.User)?.Text ?? string.Empty;
|
||||
this.Response = string.Join(
|
||||
" ",
|
||||
responseMessages
|
||||
.Where(m => m.Role == ChatRole.Assistant && !string.IsNullOrEmpty(m.Text))
|
||||
.Select(m => m.Text));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Initializes a new instance of the <see cref="EvalItem"/> class from query and response
|
||||
/// strings, automatically building a minimal conversation.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Use this constructor for simple text-only evaluations where you don't need
|
||||
/// a full conversation history.
|
||||
/// </remarks>
|
||||
/// <param name="query">The user query.</param>
|
||||
/// <param name="response">The agent response text.</param>
|
||||
public EvalItem(string query, string response)
|
||||
{
|
||||
this.Query = query;
|
||||
this.Response = response;
|
||||
this.Conversation = new List<ChatMessage>
|
||||
{
|
||||
new(ChatRole.User, query),
|
||||
new(ChatRole.Assistant, response),
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>Gets the user query.</summary>
|
||||
public string Query { get; }
|
||||
|
||||
/// <summary>Gets the agent response text.</summary>
|
||||
public string Response { get; }
|
||||
|
||||
/// <summary>Gets the full conversation history.</summary>
|
||||
/// <remarks>
|
||||
/// The conversation preserves all content types including images
|
||||
/// (<see cref="DataContent"/>, <see cref="UriContent"/> with image media types).
|
||||
/// Use this property in custom <see cref="EvalCheck"/> functions
|
||||
/// to inspect multimodal content that isn't captured in the
|
||||
/// text-only <see cref="Query"/> and <see cref="Response"/> properties.
|
||||
/// </remarks>
|
||||
public IReadOnlyList<ChatMessage> Conversation { get; }
|
||||
|
||||
/// <summary>
|
||||
/// Gets whether any message in the conversation contains image content.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Checks for <see cref="DataContent"/> or <see cref="UriContent"/> with an image media type.
|
||||
/// Useful in <see cref="EvalCheck"/> functions to verify multimodal content is present.
|
||||
/// </remarks>
|
||||
public bool HasImageContent =>
|
||||
this.Conversation.Any(m =>
|
||||
m.Contents.Any(c =>
|
||||
(c is DataContent dc && dc.HasTopLevelMediaType("image"))
|
||||
|| (c is UriContent uc && uc.HasTopLevelMediaType("image"))));
|
||||
|
||||
/// <summary>Gets or sets the tools available to the agent.</summary>
|
||||
public IReadOnlyList<AITool>? Tools { get; set; }
|
||||
|
||||
/// <summary>Gets or sets grounding context for evaluation.</summary>
|
||||
public string? Context { get; set; }
|
||||
|
||||
/// <summary>Gets or sets the expected output for ground-truth comparison.</summary>
|
||||
public string? ExpectedOutput { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Gets or sets the expected tool calls for tool-correctness evaluation.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Each entry describes a tool call the agent should make. The evaluator
|
||||
/// decides matching semantics (ordering, extras, argument checking).
|
||||
/// See <see cref="ExpectedToolCall"/>.
|
||||
/// </remarks>
|
||||
public IReadOnlyList<ExpectedToolCall>? ExpectedToolCalls { get; set; }
|
||||
|
||||
/// <summary>Gets or sets the raw chat response for MEAI evaluators.</summary>
|
||||
public ChatResponse? RawResponse { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Gets or sets the conversation splitter for this item.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// When set by orchestration functions (e.g. <c>EvaluateAsync(splitter: ...)</c>),
|
||||
/// this is used as the default by <see cref="Split(IConversationSplitter?)"/>.
|
||||
/// Priority: explicit <c>Split(splitter)</c> argument >
|
||||
/// <see cref="Splitter"/> > <see cref="ConversationSplitters.LastTurn"/>.
|
||||
/// </remarks>
|
||||
public IConversationSplitter? Splitter { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Splits the conversation into query messages and response messages.
|
||||
/// </summary>
|
||||
/// <param name="splitter">
|
||||
/// The splitter to use. When <c>null</c>, uses <see cref="Splitter"/>
|
||||
/// if set, otherwise <see cref="ConversationSplitters.LastTurn"/>.
|
||||
/// </param>
|
||||
/// <returns>A tuple of (query messages, response messages).</returns>
|
||||
public (IReadOnlyList<ChatMessage> QueryMessages, IReadOnlyList<ChatMessage> ResponseMessages) Split(
|
||||
IConversationSplitter? splitter = null)
|
||||
{
|
||||
var effective = splitter ?? this.Splitter ?? ConversationSplitters.LastTurn;
|
||||
return effective.Split(this.Conversation);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Splits a multi-turn conversation into one <see cref="EvalItem"/> per user turn.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Each user message starts a new turn. The resulting item has cumulative context:
|
||||
/// query messages contain the full conversation up to and including that user message,
|
||||
/// and the response is everything up to the next user message.
|
||||
/// </remarks>
|
||||
/// <param name="conversation">The full conversation to split.</param>
|
||||
/// <param name="tools">Optional tools available to the agent.</param>
|
||||
/// <param name="context">Optional grounding context.</param>
|
||||
/// <returns>A list of eval items, one per user turn.</returns>
|
||||
public static IReadOnlyList<EvalItem> PerTurnItems(
|
||||
IReadOnlyList<ChatMessage> conversation,
|
||||
IReadOnlyList<AITool>? tools = null,
|
||||
string? context = null)
|
||||
{
|
||||
var items = new List<EvalItem>();
|
||||
var userIndices = new List<int>();
|
||||
|
||||
for (int i = 0; i < conversation.Count; i++)
|
||||
{
|
||||
if (conversation[i].Role == ChatRole.User)
|
||||
{
|
||||
userIndices.Add(i);
|
||||
}
|
||||
}
|
||||
|
||||
for (int t = 0; t < userIndices.Count; t++)
|
||||
{
|
||||
int userIdx = userIndices[t];
|
||||
int nextBoundary = t + 1 < userIndices.Count
|
||||
? userIndices[t + 1]
|
||||
: conversation.Count;
|
||||
|
||||
var responseMessages = conversation.Skip(userIdx + 1).Take(nextBoundary - userIdx - 1).ToList();
|
||||
|
||||
var query = conversation[userIdx].Text ?? string.Empty;
|
||||
var responseText = string.Join(
|
||||
" ",
|
||||
responseMessages
|
||||
.Where(m => m.Role == ChatRole.Assistant && !string.IsNullOrEmpty(m.Text))
|
||||
.Select(m => m.Text));
|
||||
|
||||
var fullSlice = conversation.Take(nextBoundary).ToList();
|
||||
var item = new EvalItem(query, responseText, fullSlice)
|
||||
{
|
||||
Tools = tools,
|
||||
Context = context,
|
||||
};
|
||||
|
||||
items.Add(item);
|
||||
}
|
||||
|
||||
return items;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,76 @@
|
||||
// Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
|
||||
namespace Microsoft.Agents.AI;
|
||||
|
||||
/// <summary>
|
||||
/// Per-item result from a Foundry evaluation run, with individual evaluator scores and error details.
|
||||
/// </summary>
|
||||
public sealed class EvalItemResult
|
||||
{
|
||||
/// <summary>
|
||||
/// Initializes a new instance of the <see cref="EvalItemResult"/> class.
|
||||
/// </summary>
|
||||
/// <param name="itemId">The output item ID from the evaluation API.</param>
|
||||
/// <param name="status">The item evaluation status (e.g., "pass", "fail", "error").</param>
|
||||
/// <param name="scores">Per-evaluator score results.</param>
|
||||
public EvalItemResult(string itemId, string status, IReadOnlyList<EvalScoreResult> scores)
|
||||
{
|
||||
this.ItemId = itemId;
|
||||
this.Status = status;
|
||||
this.Scores = scores;
|
||||
}
|
||||
|
||||
/// <summary>Gets the output item ID from the evaluation API.</summary>
|
||||
public string ItemId { get; }
|
||||
|
||||
/// <summary>Gets the item evaluation status (e.g., "pass", "fail", "error", "errored").</summary>
|
||||
public string Status { get; }
|
||||
|
||||
/// <summary>Gets the per-evaluator score results.</summary>
|
||||
public IReadOnlyList<EvalScoreResult> Scores { get; }
|
||||
|
||||
/// <summary>Gets or sets an error code when the item evaluation errored.</summary>
|
||||
public string? ErrorCode { get; set; }
|
||||
|
||||
/// <summary>Gets or sets an error message when the item evaluation errored.</summary>
|
||||
public string? ErrorMessage { get; set; }
|
||||
|
||||
/// <summary>Gets or sets the response ID from the evaluation API (e.g., for response-based evals).</summary>
|
||||
public string? ResponseId { get; set; }
|
||||
|
||||
/// <summary>Gets or sets the input text echoed back by the evaluation API.</summary>
|
||||
public string? InputText { get; set; }
|
||||
|
||||
/// <summary>Gets or sets the output text echoed back by the evaluation API.</summary>
|
||||
public string? OutputText { get; set; }
|
||||
|
||||
/// <summary>Gets or sets token usage information from the evaluation.</summary>
|
||||
public IReadOnlyDictionary<string, int>? TokenUsage { get; set; }
|
||||
|
||||
/// <summary>Gets whether this item is in an error state.</summary>
|
||||
public bool IsError => this.Status is "error" or "errored";
|
||||
|
||||
/// <summary>Gets whether this item passed all evaluators.</summary>
|
||||
public bool IsPassed => this.Scores.Count > 0 && this.Scores.All(s => s.Passed == true);
|
||||
|
||||
/// <summary>Gets whether this item failed any evaluator.</summary>
|
||||
public bool IsFailed => this.Scores.Any(s => s.Passed == false);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A single evaluator's score on one evaluation item.
|
||||
/// </summary>
|
||||
/// <param name="Name">The evaluator name that produced this score.</param>
|
||||
/// <param name="Score">The numeric score value.</param>
|
||||
/// <param name="Passed">Whether the evaluator considered this a pass, or null if not determined.</param>
|
||||
public record EvalScoreResult(string Name, double Score, bool? Passed = null);
|
||||
|
||||
/// <summary>
|
||||
/// Per-evaluator pass/fail breakdown from an evaluation run.
|
||||
/// </summary>
|
||||
/// <param name="Passed">Number of items that passed for this evaluator.</param>
|
||||
/// <param name="Failed">Number of items that failed for this evaluator.</param>
|
||||
public record PerEvaluatorResult(int Passed, int Failed);
|
||||
@@ -0,0 +1,20 @@
|
||||
// Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
using System.Collections.Generic;
|
||||
|
||||
namespace Microsoft.Agents.AI;
|
||||
|
||||
/// <summary>
|
||||
/// A tool call that an agent is expected to make.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Used with <c>EvaluateAsync</c> to assert that the agent called the correct tools.
|
||||
/// The evaluator decides matching semantics (order, extras, argument checking);
|
||||
/// this type is pure data.
|
||||
/// </remarks>
|
||||
/// <param name="Name">The tool/function name (e.g. <c>"get_weather"</c>).</param>
|
||||
/// <param name="Arguments">
|
||||
/// Expected arguments. <c>null</c> means "don't check arguments".
|
||||
/// When provided, evaluators typically do subset matching (all expected keys must be present).
|
||||
/// </param>
|
||||
public record ExpectedToolCall(string Name, IReadOnlyDictionary<string, object>? Arguments = null);
|
||||
@@ -0,0 +1,68 @@
|
||||
// Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
using System;
|
||||
|
||||
namespace Microsoft.Agents.AI;
|
||||
|
||||
/// <summary>
|
||||
/// Factory for creating <see cref="EvalCheck"/> delegates from typed lambda functions.
|
||||
/// </summary>
|
||||
public static class FunctionEvaluator
|
||||
{
|
||||
/// <summary>
|
||||
/// Creates a check from a function that takes the response text and returns a bool.
|
||||
/// </summary>
|
||||
/// <param name="name">Check name for reporting.</param>
|
||||
/// <param name="check">Function that returns true if the response passes.</param>
|
||||
public static EvalCheck Create(string name, Func<string, bool> check)
|
||||
{
|
||||
return (EvalItem item) =>
|
||||
{
|
||||
var passed = check(item.Response);
|
||||
return new EvalCheckResult(passed, passed ? "Passed" : "Failed", name);
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Creates a check from a function that takes response and expected text.
|
||||
/// </summary>
|
||||
/// <param name="name">Check name for reporting.</param>
|
||||
/// <param name="check">Function that returns true if the response passes.</param>
|
||||
public static EvalCheck Create(string name, Func<string, string?, bool> check)
|
||||
{
|
||||
return (EvalItem item) =>
|
||||
{
|
||||
var passed = check(item.Response, item.ExpectedOutput);
|
||||
return new EvalCheckResult(passed, passed ? "Passed" : "Failed", name);
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Creates a check from a function that takes the full <see cref="EvalItem"/>.
|
||||
/// </summary>
|
||||
/// <param name="name">Check name for reporting.</param>
|
||||
/// <param name="check">Function that returns true if the item passes.</param>
|
||||
public static EvalCheck Create(string name, Func<EvalItem, bool> check)
|
||||
{
|
||||
return (EvalItem item) =>
|
||||
{
|
||||
var passed = check(item);
|
||||
return new EvalCheckResult(passed, passed ? "Passed" : "Failed", name);
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Creates a check from a function that takes the full <see cref="EvalItem"/>
|
||||
/// and returns a <see cref="EvalCheckResult"/>.
|
||||
/// </summary>
|
||||
/// <param name="name">Check name (used as fallback if the result has no name).</param>
|
||||
/// <param name="check">Function that returns a full check result.</param>
|
||||
public static EvalCheck Create(string name, Func<EvalItem, EvalCheckResult> check)
|
||||
{
|
||||
return (EvalItem item) =>
|
||||
{
|
||||
var result = check(item);
|
||||
return result with { CheckName = result.CheckName ?? name };
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
// Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
using System.Collections.Generic;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
|
||||
namespace Microsoft.Agents.AI;
|
||||
|
||||
/// <summary>
|
||||
/// Batch-oriented evaluator interface for agent evaluation.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Unlike MEAI's <c>IEvaluator</c> which evaluates one item at a time,
|
||||
/// <see cref="IAgentEvaluator"/> evaluates a batch of items. This enables
|
||||
/// efficient cloud-based evaluation (e.g., Foundry) and aggregate result computation.
|
||||
/// </remarks>
|
||||
public interface IAgentEvaluator
|
||||
{
|
||||
/// <summary>Gets the evaluator name.</summary>
|
||||
string Name { get; }
|
||||
|
||||
/// <summary>
|
||||
/// Evaluates a batch of items and returns aggregate results.
|
||||
/// </summary>
|
||||
/// <param name="items">The items to evaluate.</param>
|
||||
/// <param name="evalName">A display name for this evaluation run.</param>
|
||||
/// <param name="cancellationToken">Cancellation token.</param>
|
||||
/// <returns>Aggregate evaluation results.</returns>
|
||||
Task<AgentEvaluationResults> EvaluateAsync(
|
||||
IReadOnlyList<EvalItem> items,
|
||||
string evalName = "Agent Framework Eval",
|
||||
CancellationToken cancellationToken = default);
|
||||
}
|
||||
@@ -0,0 +1,103 @@
|
||||
// Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using Microsoft.Extensions.AI;
|
||||
|
||||
namespace Microsoft.Agents.AI;
|
||||
|
||||
/// <summary>
|
||||
/// Strategy for splitting a conversation into query and response halves for evaluation.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Use one of the built-in splitters from <see cref="ConversationSplitters"/> or implement
|
||||
/// your own for domain-specific splitting logic (e.g., splitting before a memory-retrieval
|
||||
/// tool call to evaluate recall quality).
|
||||
/// </remarks>
|
||||
public interface IConversationSplitter
|
||||
{
|
||||
/// <summary>
|
||||
/// Splits a conversation into query messages and response messages.
|
||||
/// </summary>
|
||||
/// <param name="conversation">The full conversation to split.</param>
|
||||
/// <returns>A tuple of (query messages, response messages).</returns>
|
||||
(IReadOnlyList<ChatMessage> QueryMessages, IReadOnlyList<ChatMessage> ResponseMessages) Split(
|
||||
IReadOnlyList<ChatMessage> conversation);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Built-in conversation splitters for common evaluation patterns.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <list type="bullet">
|
||||
/// <item><see cref="LastTurn"/>: Evaluates whether the agent answered the <em>latest</em> question well.</item>
|
||||
/// <item><see cref="Full"/>: Evaluates whether the <em>whole conversation trajectory</em> served the original request.</item>
|
||||
/// </list>
|
||||
/// For custom splits, implement <see cref="IConversationSplitter"/> directly.
|
||||
/// </remarks>
|
||||
public static class ConversationSplitters
|
||||
{
|
||||
/// <summary>
|
||||
/// Split at the last user message. Everything up to and including that message
|
||||
/// is the query; everything after is the response. This is the default strategy.
|
||||
/// </summary>
|
||||
public static IConversationSplitter LastTurn { get; } = new LastTurnSplitter();
|
||||
|
||||
/// <summary>
|
||||
/// The first user message (and any preceding system messages) is the query;
|
||||
/// the entire remainder of the conversation is the response.
|
||||
/// Evaluates overall conversation trajectory.
|
||||
/// </summary>
|
||||
public static IConversationSplitter Full { get; } = new FullSplitter();
|
||||
|
||||
private sealed class LastTurnSplitter : IConversationSplitter
|
||||
{
|
||||
public (IReadOnlyList<ChatMessage>, IReadOnlyList<ChatMessage>) Split(
|
||||
IReadOnlyList<ChatMessage> conversation)
|
||||
{
|
||||
int lastUserIdx = -1;
|
||||
for (int i = 0; i < conversation.Count; i++)
|
||||
{
|
||||
if (conversation[i].Role == ChatRole.User)
|
||||
{
|
||||
lastUserIdx = i;
|
||||
}
|
||||
}
|
||||
|
||||
if (lastUserIdx >= 0)
|
||||
{
|
||||
return (
|
||||
conversation.Take(lastUserIdx + 1).ToList(),
|
||||
conversation.Skip(lastUserIdx + 1).ToList());
|
||||
}
|
||||
|
||||
return (new List<ChatMessage>(), conversation.ToList());
|
||||
}
|
||||
}
|
||||
|
||||
private sealed class FullSplitter : IConversationSplitter
|
||||
{
|
||||
public (IReadOnlyList<ChatMessage>, IReadOnlyList<ChatMessage>) Split(
|
||||
IReadOnlyList<ChatMessage> conversation)
|
||||
{
|
||||
int firstUserIdx = -1;
|
||||
for (int i = 0; i < conversation.Count; i++)
|
||||
{
|
||||
if (conversation[i].Role == ChatRole.User)
|
||||
{
|
||||
firstUserIdx = i;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if (firstUserIdx >= 0)
|
||||
{
|
||||
return (
|
||||
conversation.Take(firstUserIdx + 1).ToList(),
|
||||
conversation.Skip(firstUserIdx + 1).ToList());
|
||||
}
|
||||
|
||||
return (new List<ChatMessage>(), conversation.ToList());
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
// Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
using System.Collections.Generic;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using Microsoft.Extensions.AI.Evaluation;
|
||||
|
||||
namespace Microsoft.Agents.AI;
|
||||
|
||||
/// <summary>
|
||||
/// Evaluator that runs check functions locally without API calls.
|
||||
/// </summary>
|
||||
public sealed class LocalEvaluator : IAgentEvaluator
|
||||
{
|
||||
private readonly EvalCheck[] _checks;
|
||||
|
||||
/// <summary>
|
||||
/// Initializes a new instance of the <see cref="LocalEvaluator"/> class.
|
||||
/// </summary>
|
||||
/// <param name="checks">The check functions to run on each item.</param>
|
||||
public LocalEvaluator(params EvalCheck[] checks)
|
||||
{
|
||||
this._checks = checks;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public string Name => "LocalEvaluator";
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task<AgentEvaluationResults> EvaluateAsync(
|
||||
IReadOnlyList<EvalItem> items,
|
||||
string evalName = "Local Eval",
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
var results = new List<EvaluationResult>(items.Count);
|
||||
|
||||
foreach (var item in items)
|
||||
{
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
|
||||
var evalResult = new EvaluationResult();
|
||||
|
||||
foreach (var check in this._checks)
|
||||
{
|
||||
var EvalCheckResult = check(item);
|
||||
evalResult.Metrics[EvalCheckResult.CheckName] = new BooleanMetric(
|
||||
EvalCheckResult.CheckName,
|
||||
EvalCheckResult.Passed,
|
||||
reason: EvalCheckResult.Reason)
|
||||
{
|
||||
Interpretation = new EvaluationMetricInterpretation
|
||||
{
|
||||
Rating = EvalCheckResult.Passed
|
||||
? EvaluationRating.Good
|
||||
: EvaluationRating.Unacceptable,
|
||||
Failed = !EvalCheckResult.Passed,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
results.Add(evalResult);
|
||||
}
|
||||
|
||||
return Task.FromResult(new AgentEvaluationResults(this.Name, results, inputItems: items));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
// Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using Microsoft.Extensions.AI;
|
||||
using Microsoft.Extensions.AI.Evaluation;
|
||||
|
||||
namespace Microsoft.Agents.AI;
|
||||
|
||||
/// <summary>
|
||||
/// Adapter that wraps an MEAI <see cref="IEvaluator"/> into an <see cref="IAgentEvaluator"/>.
|
||||
/// Runs the MEAI evaluator per-item and aggregates results.
|
||||
/// </summary>
|
||||
internal sealed class MeaiEvaluatorAdapter : IAgentEvaluator
|
||||
{
|
||||
private readonly IEvaluator _evaluator;
|
||||
private readonly ChatConfiguration _chatConfiguration;
|
||||
|
||||
/// <summary>
|
||||
/// Initializes a new instance of the <see cref="MeaiEvaluatorAdapter"/> class.
|
||||
/// </summary>
|
||||
/// <param name="evaluator">The MEAI evaluator to wrap.</param>
|
||||
/// <param name="chatConfiguration">Chat configuration for the evaluator (includes the judge model).</param>
|
||||
public MeaiEvaluatorAdapter(IEvaluator evaluator, ChatConfiguration chatConfiguration)
|
||||
{
|
||||
this._evaluator = evaluator;
|
||||
this._chatConfiguration = chatConfiguration;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public string Name => this._evaluator.GetType().Name;
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<AgentEvaluationResults> EvaluateAsync(
|
||||
IReadOnlyList<EvalItem> items,
|
||||
string evalName = "MEAI Eval",
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
var results = new List<EvaluationResult>(items.Count);
|
||||
|
||||
foreach (var item in items)
|
||||
{
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
|
||||
var (queryMessages, _) = item.Split();
|
||||
var messages = queryMessages.ToList();
|
||||
var chatResponse = item.RawResponse
|
||||
?? new ChatResponse(new ChatMessage(ChatRole.Assistant, item.Response));
|
||||
|
||||
var result = await this._evaluator.EvaluateAsync(
|
||||
messages,
|
||||
chatResponse,
|
||||
this._chatConfiguration,
|
||||
cancellationToken: cancellationToken).ConfigureAwait(false);
|
||||
|
||||
results.Add(result);
|
||||
}
|
||||
|
||||
return new AgentEvaluationResults(this.Name, results, inputItems: items);
|
||||
}
|
||||
}
|
||||
@@ -31,6 +31,14 @@
|
||||
<PackageReference Include="System.Diagnostics.DiagnosticSource" />
|
||||
</ItemGroup>
|
||||
|
||||
<!-- Evaluation support requires net8.0+ (MEAI.Evaluation does not support legacy TFMs) -->
|
||||
<ItemGroup Condition="$([MSBuild]::IsTargetFrameworkCompatible('$(TargetFramework)', 'net8.0'))">
|
||||
<PackageReference Include="Microsoft.Extensions.AI.Evaluation" />
|
||||
</ItemGroup>
|
||||
<ItemGroup Condition="!$([MSBuild]::IsTargetFrameworkCompatible('$(TargetFramework)', 'net8.0'))">
|
||||
<Compile Remove="Evaluation\**\*.cs" />
|
||||
</ItemGroup>
|
||||
|
||||
<PropertyGroup>
|
||||
<!-- NuGet Package Settings -->
|
||||
<Title>Microsoft Agent Framework</Title>
|
||||
|
||||
Reference in New Issue
Block a user