mirror of
https://github.com/microsoft/agent-framework.git
synced 2026-06-16 21:04:09 +08:00
feature/python-harness
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
   
|
aee1acbf8b |
.NET: Foundry Evals integration for .NET (#4914)
* Foundry Evals integration for .NET - Core evaluation framework: EvalItem, LocalEvaluator, FunctionEvaluator, EvalChecks - IAgentEvaluator interface with MeaiEvaluatorAdapter bridge - AgentEvaluationExtensions for agent.EvaluateAsync() overloads - FoundryEvals wrapping MEAI quality/safety evaluators - ConversationSplitters (LastTurn, Full) and IConversationSplitter - EvalItem.PerTurnItems() for multi-turn decomposition - HasImageContent for multimodal content detection - WorkflowEvaluationExtensions for per-agent workflow evaluation - 7 eval samples mirroring Python parity: 02-agents/Evaluation: SimpleEval, ExpectedOutputs, Multimodal 03-workflows/Evaluation: WorkflowEval 05-end-to-end/Evaluation: FoundryQuality, MixedProviders, ConversationSplits - Comprehensive unit tests (1958 passing) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Rewrite FoundryEvals to use real Foundry Evals API Replace MEAI evaluator shim with actual OpenAI EvaluationClient protocol methods. FoundryEvals now creates eval definitions, submits runs, polls for completion, and fetches per-item results server-side. - New constructor: FoundryEvals(AIProjectClient, model, evaluators) - Add FoundryEvalConverter for MEAI ChatMessage -> Foundry JSON format - Add EvalId, RunId, ReportUrl to AgentEvaluationResults - All 20 built-in evaluator constants now work (agent, tool, quality, safety) - Remove Microsoft.Extensions.AI.Evaluation.Quality/Safety dependencies - Update all samples for new constructor (no more ChatConfiguration) - Replace BuildEvaluators tests with ResolveEvaluator tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add response output to CustomEvals and ExpectedOutputs samples Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: pagination, validation, error handling, tests FoundryEvals fixes: - Add pagination for output items (has_more/after cursor) - Add guard clauses for pollIntervalSeconds/timeoutSeconds <= 0 - Fix double TryGetProperty for passed field parsing - Throw on all-tool-evaluators with no tool definitions - Fix XML doc (default 300s, not 180s) New tests (30 added, 1989 total): - EvalChecks: NonEmpty, ContainsExpected (pass/fail/skip/case), HasImageContent, ToolCallsPresent - FoundryEvalConverter: ConvertMessage (text, image, function call, function results fan-out, empty fallback, mixed content), ConvertEvalItem, BuildTestingCriteria (quality/agent/tool/groundedness data mappings), BuildItemSchema Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix review: null-refs, Data.ToString() bug, ContainsExpected, add tests - Fix NullReferenceException in sample Response display (pattern matching) - Fix WorkflowEvaluationExtensions Data?.ToString() producing type names instead of message text (pattern-match ChatMessage/AgentResponse/list) - Change EvalChecks.ContainsExpected to return Passed=false when no ExpectedOutput (was silently passing, masking misconfiguration) - Add EvalItem constructor tests with LastTurn/Full/null splitters - Add FoundryEvalConverter.ConvertMessage DataContent (base64 image) test - Add ExtractAgentData tests with ChatMessage, list, and AgentResponse data Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix review: conversation fidelity, eval caching, fallback tests - WorkflowEvaluationExtensions: preserve full response messages (tool calls, intermediate) instead of synthetic 2-message conversation. Cast completed Data to AgentResponse and use Messages when available, fallback to text. - FoundryEvals: cache evalId per schema shape (hasContext, hasTools) so subsequent EvaluateAsync calls create runs under the same eval definition. - MeaiEvaluatorAdapter: code already correctly passes queryMessages (not full conversation) to IEvaluator — no change needed, verified by inspection. - Add tests: AgentResponse full messages preservation, unknown object ToString() fallback for ExtractAgentData. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Rename AzureAI→Foundry: move eval files, update references - Move FoundryEvals.cs and FoundryEvalConverter.cs from Microsoft.Agents.AI.AzureAI to Microsoft.Agents.AI.Foundry - Update namespace from AzureAI to Foundry in both files - Add explicit usings required by Foundry project (no implicit usings) - Move FoundryEvalConverter tests to Foundry.UnitTests project (avoids ReplacingRedactor type conflict from dual project refs) - Update all sample csproj references and using statements - Remove Foundry project reference from AI UnitTests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * PR review round 4: wire up tool extraction, remove eval cache, fix null safety - BuildEvalItem: extract tools from agent via GetService<ChatOptions>() into EvalItem.Tools (Python parity) - FoundryEvals: remove eval ID cache - each call creates fresh definition (matches Python behavior) - FoundryEvals: replace null-forgiving operators with descriptive InvalidOperationException - MixedProviders sample: remove unnecessary explicit PackageReferences (transitively provided) - FoundryEvalConverter: document that tool results take precedence over text content - Add LocalEvaluator zero-checks test documenting 0 metrics = failed behavior Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Python-dotnet parity: 9 feature gaps filled New checks: - ToolCallArgsMatch() — verify tool call names + argument subset match - ToolCalledCheck(ToolCalledMode.Any, ...) — match any of the specified tools - ToolCalledMode enum (All/Any) FoundryEvals enhancements: - Default evaluators now [Relevance, Coherence, TaskAdherence] (was Relevance, Coherence) - Auto-add ToolCallAccuracy when items have tool definitions - EvaluateTracesAsync — evaluate by response_ids, trace_ids, or agent_id - EvaluateFoundryTargetAsync — evaluate deployed Foundry targets Result type enrichment: - AgentEvaluationResults: added Status, Error, PerEvaluator, DetailedItems - New EvalItemResult/EvalScoreResult/PerEvaluatorResult types - FoundryEvals populates all new fields from API responses Workflow fix: - Skip internal executors (_*, input-conversation, end-conversation, end) Tests: 8 new tests covering ToolCallArgsMatch, ToolCalledMode.Any, internal executor filtering Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add MeaiEvaluatorAdapter and PerTurnItems edge case tests - 3 tests for MeaiEvaluatorAdapter: query message forwarding, synthetic response fallback, multiple items aggregation - 3 tests for EvalItem.PerTurnItems: empty conversation, no user messages, system+assistant only - StubEvaluator and StubChatClient test helpers Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Blocking link check for outdated package in DevUI. * Replace Dictionary<string, object> payloads with typed wire models Introduce internal FoundryEvalWireModels.cs with compile-time-safe types for the OpenAI Evals API wire format. The OpenAI .NET SDK (2.9.1) only provides protocol-level methods with BinaryContent/ClientResult — no typed request models. These internal models replace scattered dictionary literals with [JsonPropertyName]-annotated classes, giving: - Compile-time safety (typos become build errors) - Single point of change when the API evolves - IntelliSense discoverability - Cleaner serialization via JsonPolymorphic for content items Models: WireContentItem hierarchy (text, image, tool_call, tool_result), WireMessage, WireEvalItemPayload, WireTestingCriterion, WireItemSchema, WireCreateEvalRequest, WireCreateRunRequest, and data source variants. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Skip metric when Foundry returns neither score nor passed When an evaluator returns no score and no passed value, the previous code created BooleanMetric(name, false), which falsely failed items via ItemPassed. Now we skip the MEAI metric entirely for indeterminate results — the raw data remains available in DetailedItems for diagnostics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #4914 review comments: fix tool evaluator bug and add tests - Fix duplicate ToolCallAccuracy: resolve evaluator names before checking against ToolEvaluators set (Comment 2) - Make FilterToolEvaluators internal for testability; add tests for the ArgumentException edge case when all evaluators are tool-type (Comment 3) - Add CancellationToken test for LocalEvaluator (Comment 4) - Add EvaluateAsync integration test on Run with sequential workflow and per-agent SubResults verification (Comment 5) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address Peter's review comments on PR #4914 - Add trailing newline to Evaluation_FoundryQuality.csproj (Comment 6) - Make evaluator name lookups case-insensitive: switch BuiltinEvaluators, ToolEvaluators, AgentEvaluators, and ResolveEvaluator's StartsWith check from Ordinal to OrdinalIgnoreCase (Comment 7) - Add Trace.TraceWarning when Foundry returns fewer results than submitted items, indicating expected vs actual count before padding (Comment 8) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add Microsoft.Extensions.AI.Evaluation packages to Directory.Packages.props These were removed in #5269 as unused, but are needed by the Foundry and core evaluation integration added in this PR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: alliscode <bentho@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
 Roger BarretoandGitHub
|
4134c74060 |
Add CreateSessionAsync(conversationId) to FoundryAgent (#5144)
Adds a public CreateSessionAsync(string conversationId, CancellationToken) method to FoundryAgent that delegates to the inner ChatClientAgent, allowing users to create sessions with existing server-side conversation IDs. Fixes #5138 |
||
|
Roger BarretoandGitHub
|
b0613a8ceb |
.NET: Bump Azure.AI.Projects to 2.0.0 GA (#5060)
* Bump Azure.AI.Projects to 2.0.0 GA - Update Azure.AI.Projects from 2.0.0-beta.2 to 2.0.0 in CPM - Update Azure.Identity from 1.19.0 to 1.20.0 (transitive dep) - Update System.ClientModel from 1.9.0 to 1.10.0 (transitive dep) - Rename types per Azure.AI.Projects.Agents 2.0.0 breaking changes: - AgentVersion -> ProjectsAgentVersion - AgentRecord -> ProjectsAgentRecord - AgentDefinition -> ProjectsAgentDefinition - AgentVersionCreationOptions -> ProjectsAgentVersionCreationOptions - PromptAgentDefinition -> DeclarativeAgentDefinition - AgentTool -> ProjectsAgentTool - AgentsClient -> AgentAdministrationClient - .Agents property -> .AgentAdministrationClient - Add using Azure.AI.Projects.Memory namespace (types moved) - Update AGENTS.md with BOM and output capture conventions * Address PR review feedback - Rename AIProjectClient parameter to aiProjectClient in AsChatClientAgent overloads - Fix XML doc: ProjectsAgentTool namespace from Azure.AI.Projects.OpenAI to Azure.AI.Projects.Agents - Rename test method to reflect DeclarativeAgentDefinition terminology |
||
|
Roger BarretoandGitHub
|
628bb1af48 |
.NET: Rename Microsoft.Agents.AI.AzureAI to Microsoft.Agents.AI.Foundry and consolidate FoundryMemory (#5042)
* Update Foundry Responses as ChatClientAgent * Migrate obsolete AzureAI integration tests to versioned agent pattern Replace obsolete CreateAIAgentAsync/GetAIAgentAsync calls with Agents.CreateAgentVersionAsync() + AsAIAgent(AgentVersion) in all AzureAI integration tests. - Rename AIProjectClient* test files to FoundryVersionedAgent* - Register AIFunction tools in PromptAgentDefinition.Tools for server-side visibility via AsOpenAIResponseTool() - Skip structured output tests (AzureAIProjectChatClient clears ResponseFormat for versioned agents) - Remove all [Obsolete] attributes and #pragma warning disable CS0618 * Merge FoundryMemory package into AzureAI under Memory/ folder Move all FoundryMemory source, unit tests, and integration tests into the Microsoft.Agents.AI.AzureAI package. Change namespace from Microsoft.Agents.AI.FoundryMemory to Microsoft.Agents.AI.AzureAI. - Add [Experimental] to FoundryMemoryProviderOptions and Scope - Rename internal AIProjectClientExtensions to MemoryStoreExtensions - Update AzureAI .csproj with Compliance.Abstractions, Redaction - Remove FoundryMemory from solution and release filter - Update sample to reference AzureAI instead of FoundryMemory - Delete old Microsoft.Agents.AI.FoundryMemory project and tests * Add EnsureMemoryStoreCreatedAsync and memory existence checks to integration tests - Ensure memory store is created before testing memory operations - Add AZURE_AI_EMBEDDING_DEPLOYMENT_NAME config setting - Assert memories exist in store via SearchMemoriesAsync before cleanup - Verify scope isolation with direct memory store queries * Fix and rename AzureAI unit tests for RAPI vs Versioned clarity - Rename AsAIAgentAsync_* to AsAIAgent_* (drop Async from method group) - Add _Rapi_ prefix to non-versioned (Responses API) tests - Add _Versioned_ prefix to versioned agent tests where needed - Fix RAPI tests: assert GetService<AIProjectClient>() is null - Fix Versioned tests: assert IsType<FoundryAgent> and GetService<AIProjectClient>() returns the client instance - Fix UserAgent header tests: proper HTTP handler routing - Fix ChatClient_UsesDefaultConversationIdAsync test setup - All 153 unit tests pass with 0 failures * Rename Microsoft.Agents.AI.AzureAI to Microsoft.Agents.AI.Foundry Rename the project, namespace, folder, and all references from Microsoft.Agents.AI.AzureAI to Microsoft.Agents.AI.Foundry. Also rename Workflows.Declarative.AzureAI to .Foundry. - Rename src, unit test, integration test, and workflow folders - Update namespaces in all source and test .cs files - Update ProjectReferences in ~47 sample and test .csproj files - Update solution files (.slnx, .slnf) - Update sample using statements - Update READMEs, SKILL.md, ADRs in docs/ - Disable package validation baseline for renamed packages - Fix UTF-8 BOM encoding on all affected .cs files - AzureAI.Persistent left completely unchanged * Fix format: remove ImplicitUsings, add explicit usings, fix BOM encoding - Remove ImplicitUsings=enable from Foundry csproj to resolve IDE0005 on shared ReplacingRedactor.cs - Add explicit System usings to all source files that relied on them - Sort usings alphabetically per editorconfig rules - Fix UTF-8 BOM on 12 sample Program.cs files - Rename Azure AI Foundry Agents to Microsoft Foundry Agents in docs |