agent/agent-framework
1
0
Fork 0
mirror of https://github.com/microsoft/agent-framework.git synced 2026-06-16 21:04:09 +08:00
Code Issues Packages Projects Releases Wiki Activity
2,155 Commits 249 Branches 163 Tags
feature/python-hosting
Commit Graph

6 Commits

  • .NET: Foundry Evals integration for .NET (#4914) 
    * Foundry Evals integration for .NET

- Core evaluation framework: EvalItem, LocalEvaluator, FunctionEvaluator, EvalChecks
- IAgentEvaluator interface with MeaiEvaluatorAdapter bridge
- AgentEvaluationExtensions for agent.EvaluateAsync() overloads
- FoundryEvals wrapping MEAI quality/safety evaluators
- ConversationSplitters (LastTurn, Full) and IConversationSplitter
- EvalItem.PerTurnItems() for multi-turn decomposition
- HasImageContent for multimodal content detection
- WorkflowEvaluationExtensions for per-agent workflow evaluation
- 7 eval samples mirroring Python parity:
  02-agents/Evaluation: SimpleEval, ExpectedOutputs, Multimodal
  03-workflows/Evaluation: WorkflowEval
  05-end-to-end/Evaluation: FoundryQuality, MixedProviders, ConversationSplits
- Comprehensive unit tests (1958 passing)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Rewrite FoundryEvals to use real Foundry Evals API

Replace MEAI evaluator shim with actual OpenAI EvaluationClient protocol
methods. FoundryEvals now creates eval definitions, submits runs, polls
for completion, and fetches per-item results server-side.

- New constructor: FoundryEvals(AIProjectClient, model, evaluators)
- Add FoundryEvalConverter for MEAI ChatMessage -> Foundry JSON format
- Add EvalId, RunId, ReportUrl to AgentEvaluationResults
- All 20 built-in evaluator constants now work (agent, tool, quality, safety)
- Remove Microsoft.Extensions.AI.Evaluation.Quality/Safety dependencies
- Update all samples for new constructor (no more ChatConfiguration)
- Replace BuildEvaluators tests with ResolveEvaluator tests

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add response output to CustomEvals and ExpectedOutputs samples

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review: pagination, validation, error handling, tests

FoundryEvals fixes:
- Add pagination for output items (has_more/after cursor)
- Add guard clauses for pollIntervalSeconds/timeoutSeconds <= 0
- Fix double TryGetProperty for passed field parsing
- Throw on all-tool-evaluators with no tool definitions
- Fix XML doc (default 300s, not 180s)

New tests (30 added, 1989 total):
- EvalChecks: NonEmpty, ContainsExpected (pass/fail/skip/case),
  HasImageContent, ToolCallsPresent
- FoundryEvalConverter: ConvertMessage (text, image, function call,
  function results fan-out, empty fallback, mixed content),
  ConvertEvalItem, BuildTestingCriteria (quality/agent/tool/groundedness
  data mappings), BuildItemSchema

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix review: null-refs, Data.ToString() bug, ContainsExpected, add tests

- Fix NullReferenceException in sample Response display (pattern matching)
- Fix WorkflowEvaluationExtensions Data?.ToString() producing type names
  instead of message text (pattern-match ChatMessage/AgentResponse/list)
- Change EvalChecks.ContainsExpected to return Passed=false when no
  ExpectedOutput (was silently passing, masking misconfiguration)
- Add EvalItem constructor tests with LastTurn/Full/null splitters
- Add FoundryEvalConverter.ConvertMessage DataContent (base64 image) test
- Add ExtractAgentData tests with ChatMessage, list, and AgentResponse data

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix review: conversation fidelity, eval caching, fallback tests

- WorkflowEvaluationExtensions: preserve full response messages (tool calls,
  intermediate) instead of synthetic 2-message conversation. Cast completed
  Data to AgentResponse and use Messages when available, fallback to text.
- FoundryEvals: cache evalId per schema shape (hasContext, hasTools) so
  subsequent EvaluateAsync calls create runs under the same eval definition.
- MeaiEvaluatorAdapter: code already correctly passes queryMessages (not full
  conversation) to IEvaluator — no change needed, verified by inspection.
- Add tests: AgentResponse full messages preservation, unknown object
  ToString() fallback for ExtractAgentData.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Rename AzureAI→Foundry: move eval files, update references

- Move FoundryEvals.cs and FoundryEvalConverter.cs from
  Microsoft.Agents.AI.AzureAI to Microsoft.Agents.AI.Foundry
- Update namespace from AzureAI to Foundry in both files
- Add explicit usings required by Foundry project (no implicit usings)
- Move FoundryEvalConverter tests to Foundry.UnitTests project
  (avoids ReplacingRedactor type conflict from dual project refs)
- Update all sample csproj references and using statements
- Remove Foundry project reference from AI UnitTests

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* PR review round 4: wire up tool extraction, remove eval cache, fix null safety

- BuildEvalItem: extract tools from agent via GetService<ChatOptions>() into EvalItem.Tools (Python parity)
- FoundryEvals: remove eval ID cache - each call creates fresh definition (matches Python behavior)
- FoundryEvals: replace null-forgiving operators with descriptive InvalidOperationException
- MixedProviders sample: remove unnecessary explicit PackageReferences (transitively provided)
- FoundryEvalConverter: document that tool results take precedence over text content
- Add LocalEvaluator zero-checks test documenting 0 metrics = failed behavior

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Python-dotnet parity: 9 feature gaps filled

New checks:
- ToolCallArgsMatch() — verify tool call names + argument subset match
- ToolCalledCheck(ToolCalledMode.Any, ...) — match any of the specified tools
- ToolCalledMode enum (All/Any)

FoundryEvals enhancements:
- Default evaluators now [Relevance, Coherence, TaskAdherence] (was Relevance, Coherence)
- Auto-add ToolCallAccuracy when items have tool definitions
- EvaluateTracesAsync — evaluate by response_ids, trace_ids, or agent_id
- EvaluateFoundryTargetAsync — evaluate deployed Foundry targets

Result type enrichment:
- AgentEvaluationResults: added Status, Error, PerEvaluator, DetailedItems
- New EvalItemResult/EvalScoreResult/PerEvaluatorResult types
- FoundryEvals populates all new fields from API responses

Workflow fix:
- Skip internal executors (_*, input-conversation, end-conversation, end)

Tests: 8 new tests covering ToolCallArgsMatch, ToolCalledMode.Any, internal executor filtering

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add MeaiEvaluatorAdapter and PerTurnItems edge case tests

- 3 tests for MeaiEvaluatorAdapter: query message forwarding, synthetic
  response fallback, multiple items aggregation
- 3 tests for EvalItem.PerTurnItems: empty conversation, no user messages,
  system+assistant only
- StubEvaluator and StubChatClient test helpers

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Blocking link check for outdated package in DevUI.

* Replace Dictionary<string, object> payloads with typed wire models

Introduce internal FoundryEvalWireModels.cs with compile-time-safe types
for the OpenAI Evals API wire format. The OpenAI .NET SDK (2.9.1) only
provides protocol-level methods with BinaryContent/ClientResult — no
typed request models. These internal models replace scattered dictionary
literals with [JsonPropertyName]-annotated classes, giving:

- Compile-time safety (typos become build errors)
- Single point of change when the API evolves
- IntelliSense discoverability
- Cleaner serialization via JsonPolymorphic for content items

Models: WireContentItem hierarchy (text, image, tool_call, tool_result),
WireMessage, WireEvalItemPayload, WireTestingCriterion, WireItemSchema,
WireCreateEvalRequest, WireCreateRunRequest, and data source variants.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Skip metric when Foundry returns neither score nor passed

When an evaluator returns no score and no passed value, the previous
code created BooleanMetric(name, false), which falsely failed items
via ItemPassed. Now we skip the MEAI metric entirely for indeterminate
results — the raw data remains available in DetailedItems for diagnostics.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR #4914 review comments: fix tool evaluator bug and add tests

- Fix duplicate ToolCallAccuracy: resolve evaluator names before checking
  against ToolEvaluators set (Comment 2)
- Make FilterToolEvaluators internal for testability; add tests for the
  ArgumentException edge case when all evaluators are tool-type (Comment 3)
- Add CancellationToken test for LocalEvaluator (Comment 4)
- Add EvaluateAsync integration test on Run with sequential workflow and
  per-agent SubResults verification (Comment 5)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address Peter's review comments on PR #4914

- Add trailing newline to Evaluation_FoundryQuality.csproj (Comment 6)
- Make evaluator name lookups case-insensitive: switch BuiltinEvaluators,
  ToolEvaluators, AgentEvaluators, and ResolveEvaluator's StartsWith check
  from Ordinal to OrdinalIgnoreCase (Comment 7)
- Add Trace.TraceWarning when Foundry returns fewer results than submitted
  items, indicating expected vs actual count before padding (Comment 8)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add Microsoft.Extensions.AI.Evaluation packages to Directory.Packages.props

These were removed in #5269 as unused, but are needed by the Foundry
and core evaluation integration added in this PR.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: alliscode <bentho@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
    Ben Thomas · 2026-04-16 19:40:07 +00:00
    aee1acbf8b
  • .NET: Add foundry extension samples for python and dotnet (#4359) 
    * Add foundry extension samples for python and dotnet

* Align foundry extension samples with existing hosted agent patterns

- Fix Python multiagent indentation bug (from_agent_framework ran in both modes)
- Remove hardcoded personal endpoint from appsettings.Development.json
- Rename .NET folders/projects to PascalCase (FoundryMultiAgent, FoundrySingleAgent)
- Upgrade .NET multiagent from net9.0 to net10.0
- Add ManagePackageVersionsCentrally=false and analyzer blocks to .csproj files
- Replace wildcard package versions with fixed versions
- Use alpine Docker images and standard build pattern
- Align agent.yaml structure (template nesting, displayName, resources, authors)
- Convert .NET multiagent from namespace/class to top-level statements
- Add run-requests.http for multiagent sample
- Fix Python requirements.txt (remove dev deps, add agent-framework)
- Add proper copyright headers

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Align foundry samples: fix builds, upgrade AgentServer to beta.8

- Fix TargetFrameworks (plural) to override inherited net472 from Directory.Build.props
- Upgrade Azure.AI.AgentServer.AgentFramework to 1.0.0-beta.8 (latest)
- Bump OpenTelemetry packages to 1.12.0 (required by beta.8)
- Fix Roslynator/format errors (imports ordering, BOM, sealed record, target-typed new)
- Verified with docker dotnet format (matching CI pipeline)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Refactor hosted samples to use AIProjectClient.CreateAIAgentAsync

Replace PersistentAgentsClient and manual AzureOpenAIClient setup with
AIProjectClient.CreateAIAgentAsync() from Microsoft.Agents.AI.AzureAI.

- FoundryMultiAgent: Remove Azure.AI.Agents.Persistent, use CreateAIAgentAsync
  for Writer and Reviewer agents with cleanup in finally block
- FoundrySingleAgent: Remove manual GetConnection/AzureOpenAIClient chain,
  use CreateAIAgentAsync with hotel search tool
- Update csproj: add Microsoft.Agents.AI.AzureAI, remove unused packages

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update READMEs to reflect AIProjectClient.CreateAIAgentAsync usage

- Reference Microsoft.Agents.AI.AzureAI and Microsoft.Agents.AI.Workflows packages
- Add Azure AI Developer role requirement for agents/write data action
- Replace PersistentAgentsClient references

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add HostedAgents READMEs and Foundry samples to solution

- Create dotnet/samples/05-end-to-end/HostedAgents/README.md with sample index
- Create python/samples/05-end-to-end/hosted_agents/README.md with sample index
- Add FoundryMultiAgent and FoundrySingleAgent to agent-framework-dotnet.slnx

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix Python linting: reorder imports before load_dotenv, remove trailing whitespace

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update uv.lock to match latest package versions

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix trailing whitespace in foundry_single_agent agent.yaml

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Exclude dotnet.microsoft.com from link checker

This domain intermittently times out in CI, causing flaky markdown
link check failures unrelated to PR changes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Align env vars to AZURE_AI_PROJECT_ENDPOINT and default model to gpt-4o-mini

Addresses PR review feedback:
- Rename PROJECT_ENDPOINT to AZURE_AI_PROJECT_ENDPOINT across all
  Foundry samples (dotnet + python) to match existing samples
- Change default model from gpt-4.1-mini to gpt-4o-mini consistently

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Skip flaky test CreatesWorkflowEndToEndActivities_WithCorrectName_DefaultAsync

Tracked in #4398

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove Python foundry samples from PR scope

Python hosted agent samples need further alignment with the azure-ai
package conventions. Removing from this PR to ship .NET samples first.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Narrow linkspector exclusion to dotnet.microsoft.com/download only

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Leo Yao <leoyao@Leos-MacBook-Pro.local>
Co-authored-by: Roger Barreto <19890735+rogerbarreto@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
    Leo Yao · 2026-03-05 11:43:24 +00:00
    56bba795cb
  • .NET: Add DevUI package for .NET (#1603) 
    * Implement DevUI

* Review feedback

* Fix build
    Reuben Bond · 2025-11-05 20:02:48 +00:00
    8855bfb065
  • Fix broken .md link (#1818)
    westey · 2025-10-30 20:03:29 +00:00
    947e9eb1dc
  • Python: bump version and update changelog (#1277) 
    * bump version and update changelog

* undo lab package version bump

* add changelog to exclude pattern for links

* fix for ignore path

* exclude file
    Eduard van Valkenburg · 2025-10-07 21:41:59 +00:00
    8b73b5b1be
  • .NET: Add link inspector (#1062) 
    * Add link inspector

* Comment out excludedirs while it's empty

* Fix broken links

* More links fixes

* Push further fixes

* Fix more links
    westey · 2025-10-01 16:43:17 +00:00
    364188c44f