- Add [Experimental(OPENAI001)] to all 7 public Hosting types per reviewer request
- Fix McpConsentContext XML doc: 'Thread-static' -> 'Async-local' (AsyncLocal
flows with ExecutionContext, not thread-static)
- Expand UserAgentMiddleware test regex to match prerelease versions (e.g. 1.0.0-rc.4)
- Propagate CancellationToken in AgentFrameworkResponseHandler session save
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Extract hosted agent functionality from Microsoft.Agents.AI.Foundry into a
new Microsoft.Agents.AI.Foundry.Hosting preview package. This resolves NU5104
build errors caused by the stable Foundry package depending on prerelease
Azure SDK packages (Azure.AI.AgentServer.Responses, Azure.AI.Projects beta).
Changes:
- Create Microsoft.Agents.AI.Foundry.Hosting with VersionSuffix=preview,
targeting .NET Core only (net8.0/9.0/10.0)
- Move all Hosting/ source files to the new project
- Move ToolboxRecord/ToolboxVersion overloads to FoundryAIToolExtensions
- Revert Azure.AI.Projects to 2.0.0 in Directory.Packages.props;
Hosting uses VersionOverride for 2.1.0-beta.1
- Clean V1 Foundry csproj: remove beta deps, ASP.NET Core ref, hosting conditionals
- Update 8 hosted agent sample projects to reference Foundry.Hosting
- Split unit tests: ToolboxRecord/ToolboxVersion tests moved to Hosting/
- Add Foundry.Hosting to solution file
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Bump preview version to 260420.1 and fix AgentServer package deps
- Bump PackageVersion to 0.0.1-preview.260420.1
- Bump Azure.AI.AgentServer.Core beta.21 -> beta.22 (required by
Azure.AI.AgentServer.Responses beta.3)
- Replace AgentHostTelemetry.ResponsesSourceName with local constant
(type made internal in AgentServer.Core beta.22)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix CA1873: guard LogError with IsEnabled check in FoundryToolboxService
Wrap the LogError call with an IsEnabled(LogLevel.Error) guard to satisfy
the CA1873 analyzer rule which flags potentially expensive argument
evaluation when logging is disabled.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Bumping NuGet version
* Restore conditional versioning, remove dev feed, bump Azure.AI.Projects to beta.1
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR #5312 review comments
- Add comment explaining NU1903 suppression (Microsoft.Bcl.Memory transitive vuln)
- Remove NU1903 from sample/test projects where not needed
- Fix Dockerfile ENTRYPOINT mismatch in Hosted-Workflow-Simple
- Align agent name to 'hosted-workflow-simple' in agent.yaml and README
- Fix Hosted-McpTools README: replace GitHub PAT refs with Microsoft Learn
- Fix session persistence: only persist when client provides conversation ID
- Upgrade IsNullOrEmpty to IsNullOrWhiteSpace for session ID checks
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: alliscode <bentho@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* feat: Add Foundry Toolbox (MCP) support to AgentFrameworkResponseHandler
Adds support for Foundry Toolsets MCP proxy integration in the hosted agent
response handler. Toolsets connect at startup via IHostedService, gating the
readiness probe per spec §3.1. MCP tools are injected into every request's
ChatOptions and OAuth consent errors (-32006) are intercepted and surfaced as
mcp_approval_request + incomplete SSE events.
New files:
- FoundryToolboxOptions.cs: configuration POCO for toolset names and API version
- FoundryToolboxBearerTokenHandler.cs: DelegatingHandler with Azure Bearer token
auth, Foundry-Features header injection, and 3x exponential backoff on 429/5xx
- McpConsentContext.cs: AsyncLocal-based per-request consent state shared between
the tool wrapper and the response handler
- ConsentAwareMcpClientTool.cs: AIFunction wrapper that catches -32006 errors and
signals consent via shared state and linked CancellationTokenSource
- FoundryToolboxService.cs: IHostedService that creates McpClient per toolset at
startup and exposes cached tools
Modified files:
- AgentFrameworkResponseHandler.cs: injects toolbox tools into ChatOptions, sets
up linked CTS consent interception, emits mcp_approval_request on -32006
- ServiceCollectionExtensions.cs: adds AddFoundryToolboxes(params string[]) extension
- Microsoft.Agents.AI.Foundry.csproj: adds ModelContextProtocol and Azure.Identity
dependencies under NETCoreApp condition
Sample:
- Hosted-Toolbox: minimal hosted agent sample using AddFoundryToolboxes
* Rename toolset to toolbox in user-facing API; rename ConsentAwareMcpClientTool to ConsentAwareMcpClientAIFunction
* Add HostedMcpToolboxAITool for client-selectable Foundry toolboxes
Introduces HostedMcpToolboxAITool, a marker tool subclassing HostedMcpServerTool that rides the OpenAI Responses 'mcp' wire format to let clients request a specific Foundry toolbox per request.
- New FoundryAITool.CreateHostedMcpToolbox(name, version?) factory.
- FoundryToolboxOptions.StrictMode (default true) rejects unregistered toolboxes; set to false to allow lazy-open on first use.
- FoundryToolboxService.GetToolboxToolsAsync(name, version?) resolves cached or lazy-opened MCP tools.
- AgentFrameworkResponseHandler parses request.Tools for foundry-toolbox://name[?version=v] markers and injects resolved tools per request, merging with pre-registered ones.
- Unit tests for marker parsing and strict-mode resolution.
* Bump Azure.AI.Projects to 2.1.0-alpha; add ToolboxRecord/ToolboxVersion factory overloads + tests
* Fix PR review issues: retry off-by-one, URI encoding, docs, tests, build
- Fix off-by-one in FoundryToolboxBearerTokenHandler retry loop (4 attempts → 3)
- URI-encode version parameter in HostedMcpToolboxAITool.BuildAddress
- Add XML doc clarifying version pinning is reserved for future use
- Add comment clarifying AddHostedService deduplication safety
- Fix DevTemporaryTokenCredential expiry to use DateTimeOffset.MaxValue
- Fix AgentCard ambiguity in A2AServer sample with using alias
- Add 18 new unit tests for retry handler and ReadMcpToolboxMarkers
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Roger Barreto <19890735+rogerbarreto@users.noreply.github.com>
Co-authored-by: alliscode <bentho@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Foundry Evals integration for .NET
- Core evaluation framework: EvalItem, LocalEvaluator, FunctionEvaluator, EvalChecks
- IAgentEvaluator interface with MeaiEvaluatorAdapter bridge
- AgentEvaluationExtensions for agent.EvaluateAsync() overloads
- FoundryEvals wrapping MEAI quality/safety evaluators
- ConversationSplitters (LastTurn, Full) and IConversationSplitter
- EvalItem.PerTurnItems() for multi-turn decomposition
- HasImageContent for multimodal content detection
- WorkflowEvaluationExtensions for per-agent workflow evaluation
- 7 eval samples mirroring Python parity:
02-agents/Evaluation: SimpleEval, ExpectedOutputs, Multimodal
03-workflows/Evaluation: WorkflowEval
05-end-to-end/Evaluation: FoundryQuality, MixedProviders, ConversationSplits
- Comprehensive unit tests (1958 passing)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Rewrite FoundryEvals to use real Foundry Evals API
Replace MEAI evaluator shim with actual OpenAI EvaluationClient protocol
methods. FoundryEvals now creates eval definitions, submits runs, polls
for completion, and fetches per-item results server-side.
- New constructor: FoundryEvals(AIProjectClient, model, evaluators)
- Add FoundryEvalConverter for MEAI ChatMessage -> Foundry JSON format
- Add EvalId, RunId, ReportUrl to AgentEvaluationResults
- All 20 built-in evaluator constants now work (agent, tool, quality, safety)
- Remove Microsoft.Extensions.AI.Evaluation.Quality/Safety dependencies
- Update all samples for new constructor (no more ChatConfiguration)
- Replace BuildEvaluators tests with ResolveEvaluator tests
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add response output to CustomEvals and ExpectedOutputs samples
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review: pagination, validation, error handling, tests
FoundryEvals fixes:
- Add pagination for output items (has_more/after cursor)
- Add guard clauses for pollIntervalSeconds/timeoutSeconds <= 0
- Fix double TryGetProperty for passed field parsing
- Throw on all-tool-evaluators with no tool definitions
- Fix XML doc (default 300s, not 180s)
New tests (30 added, 1989 total):
- EvalChecks: NonEmpty, ContainsExpected (pass/fail/skip/case),
HasImageContent, ToolCallsPresent
- FoundryEvalConverter: ConvertMessage (text, image, function call,
function results fan-out, empty fallback, mixed content),
ConvertEvalItem, BuildTestingCriteria (quality/agent/tool/groundedness
data mappings), BuildItemSchema
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix review: null-refs, Data.ToString() bug, ContainsExpected, add tests
- Fix NullReferenceException in sample Response display (pattern matching)
- Fix WorkflowEvaluationExtensions Data?.ToString() producing type names
instead of message text (pattern-match ChatMessage/AgentResponse/list)
- Change EvalChecks.ContainsExpected to return Passed=false when no
ExpectedOutput (was silently passing, masking misconfiguration)
- Add EvalItem constructor tests with LastTurn/Full/null splitters
- Add FoundryEvalConverter.ConvertMessage DataContent (base64 image) test
- Add ExtractAgentData tests with ChatMessage, list, and AgentResponse data
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix review: conversation fidelity, eval caching, fallback tests
- WorkflowEvaluationExtensions: preserve full response messages (tool calls,
intermediate) instead of synthetic 2-message conversation. Cast completed
Data to AgentResponse and use Messages when available, fallback to text.
- FoundryEvals: cache evalId per schema shape (hasContext, hasTools) so
subsequent EvaluateAsync calls create runs under the same eval definition.
- MeaiEvaluatorAdapter: code already correctly passes queryMessages (not full
conversation) to IEvaluator — no change needed, verified by inspection.
- Add tests: AgentResponse full messages preservation, unknown object
ToString() fallback for ExtractAgentData.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Rename AzureAI→Foundry: move eval files, update references
- Move FoundryEvals.cs and FoundryEvalConverter.cs from
Microsoft.Agents.AI.AzureAI to Microsoft.Agents.AI.Foundry
- Update namespace from AzureAI to Foundry in both files
- Add explicit usings required by Foundry project (no implicit usings)
- Move FoundryEvalConverter tests to Foundry.UnitTests project
(avoids ReplacingRedactor type conflict from dual project refs)
- Update all sample csproj references and using statements
- Remove Foundry project reference from AI UnitTests
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* PR review round 4: wire up tool extraction, remove eval cache, fix null safety
- BuildEvalItem: extract tools from agent via GetService<ChatOptions>() into EvalItem.Tools (Python parity)
- FoundryEvals: remove eval ID cache - each call creates fresh definition (matches Python behavior)
- FoundryEvals: replace null-forgiving operators with descriptive InvalidOperationException
- MixedProviders sample: remove unnecessary explicit PackageReferences (transitively provided)
- FoundryEvalConverter: document that tool results take precedence over text content
- Add LocalEvaluator zero-checks test documenting 0 metrics = failed behavior
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Python-dotnet parity: 9 feature gaps filled
New checks:
- ToolCallArgsMatch() — verify tool call names + argument subset match
- ToolCalledCheck(ToolCalledMode.Any, ...) — match any of the specified tools
- ToolCalledMode enum (All/Any)
FoundryEvals enhancements:
- Default evaluators now [Relevance, Coherence, TaskAdherence] (was Relevance, Coherence)
- Auto-add ToolCallAccuracy when items have tool definitions
- EvaluateTracesAsync — evaluate by response_ids, trace_ids, or agent_id
- EvaluateFoundryTargetAsync — evaluate deployed Foundry targets
Result type enrichment:
- AgentEvaluationResults: added Status, Error, PerEvaluator, DetailedItems
- New EvalItemResult/EvalScoreResult/PerEvaluatorResult types
- FoundryEvals populates all new fields from API responses
Workflow fix:
- Skip internal executors (_*, input-conversation, end-conversation, end)
Tests: 8 new tests covering ToolCallArgsMatch, ToolCalledMode.Any, internal executor filtering
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add MeaiEvaluatorAdapter and PerTurnItems edge case tests
- 3 tests for MeaiEvaluatorAdapter: query message forwarding, synthetic
response fallback, multiple items aggregation
- 3 tests for EvalItem.PerTurnItems: empty conversation, no user messages,
system+assistant only
- StubEvaluator and StubChatClient test helpers
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Blocking link check for outdated package in DevUI.
* Replace Dictionary<string, object> payloads with typed wire models
Introduce internal FoundryEvalWireModels.cs with compile-time-safe types
for the OpenAI Evals API wire format. The OpenAI .NET SDK (2.9.1) only
provides protocol-level methods with BinaryContent/ClientResult — no
typed request models. These internal models replace scattered dictionary
literals with [JsonPropertyName]-annotated classes, giving:
- Compile-time safety (typos become build errors)
- Single point of change when the API evolves
- IntelliSense discoverability
- Cleaner serialization via JsonPolymorphic for content items
Models: WireContentItem hierarchy (text, image, tool_call, tool_result),
WireMessage, WireEvalItemPayload, WireTestingCriterion, WireItemSchema,
WireCreateEvalRequest, WireCreateRunRequest, and data source variants.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Skip metric when Foundry returns neither score nor passed
When an evaluator returns no score and no passed value, the previous
code created BooleanMetric(name, false), which falsely failed items
via ItemPassed. Now we skip the MEAI metric entirely for indeterminate
results — the raw data remains available in DetailedItems for diagnostics.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR #4914 review comments: fix tool evaluator bug and add tests
- Fix duplicate ToolCallAccuracy: resolve evaluator names before checking
against ToolEvaluators set (Comment 2)
- Make FilterToolEvaluators internal for testability; add tests for the
ArgumentException edge case when all evaluators are tool-type (Comment 3)
- Add CancellationToken test for LocalEvaluator (Comment 4)
- Add EvaluateAsync integration test on Run with sequential workflow and
per-agent SubResults verification (Comment 5)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address Peter's review comments on PR #4914
- Add trailing newline to Evaluation_FoundryQuality.csproj (Comment 6)
- Make evaluator name lookups case-insensitive: switch BuiltinEvaluators,
ToolEvaluators, AgentEvaluators, and ResolveEvaluator's StartsWith check
from Ordinal to OrdinalIgnoreCase (Comment 7)
- Add Trace.TraceWarning when Foundry returns fewer results than submitted
items, indicating expected vs actual count before padding (Comment 8)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add Microsoft.Extensions.AI.Evaluation packages to Directory.Packages.props
These were removed in #5269 as unused, but are needed by the Foundry
and core evaluation integration added in this PR.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: alliscode <bentho@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add InMemoryAgentSessionStore registration to all ServiceCollection
setups in AgentFrameworkResponseHandlerTests and WorkflowIntegrationTests.
This is needed after the AgentSessionStore infrastructure was introduced
in the responses-hosting feature. Tests still have NotImplementedException
stubs for CreateSessionCoreAsync which will be fixed when the session
infrastructure is fully available.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Previously, unhandled exceptions from agent execution would bubble up
to the SDK orchestrator, which emits a generic 'An internal server
error occurred.' message — hiding the actual cause (e.g., 401 auth
failures, model not found, etc.).
Now AgentFrameworkResponseHandler catches non-cancellation exceptions
and emits a proper response.failed event containing the real error
message, making it visible to clients and in logs.
OperationCanceledException still propagates for proper cancellation
handling by the SDK.
Also bumps package version to 0.9.0-hosted.260403.2.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- InputConverter: stop propagating request.Model to ChatOptions.ModelId
Hosted agents use their own model; client-provided model values like
'hosted-agent' were being passed through and causing server errors.
- Add FoundryResponsesRepl sample: interactive CLI client that connects
to a Foundry Responses endpoint using ResponsesClient.AsAIAgent()
- Bump package version to 0.9.0-hosted.260403.1
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move source and test files from the standalone Hosting.AzureAIResponses project
into the Foundry package under a Hosting/ subfolder. This consolidates the
Foundry-specific hosting adapter into the main Foundry package.
- Source: Microsoft.Agents.AI.Foundry.Hosting namespace
- Tests: merged into Foundry.UnitTests/Hosting/
- Conditionally compiled for .NETCoreApp TFMs only (net8.0+)
- Deleted standalone Hosting.AzureAIResponses project and test project
- Updated sample and solution references
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds a public CreateSessionAsync(string conversationId, CancellationToken)
method to FoundryAgent that delegates to the inner ChatClientAgent,
allowing users to create sessions with existing server-side conversation IDs.
Fixes#5138
* Update Foundry Responses as ChatClientAgent
* Migrate obsolete AzureAI integration tests to versioned agent pattern
Replace obsolete CreateAIAgentAsync/GetAIAgentAsync calls with
Agents.CreateAgentVersionAsync() + AsAIAgent(AgentVersion) in all
AzureAI integration tests.
- Rename AIProjectClient* test files to FoundryVersionedAgent*
- Register AIFunction tools in PromptAgentDefinition.Tools for
server-side visibility via AsOpenAIResponseTool()
- Skip structured output tests (AzureAIProjectChatClient clears
ResponseFormat for versioned agents)
- Remove all [Obsolete] attributes and #pragma warning disable CS0618
* Merge FoundryMemory package into AzureAI under Memory/ folder
Move all FoundryMemory source, unit tests, and integration tests into
the Microsoft.Agents.AI.AzureAI package. Change namespace from
Microsoft.Agents.AI.FoundryMemory to Microsoft.Agents.AI.AzureAI.
- Add [Experimental] to FoundryMemoryProviderOptions and Scope
- Rename internal AIProjectClientExtensions to MemoryStoreExtensions
- Update AzureAI .csproj with Compliance.Abstractions, Redaction
- Remove FoundryMemory from solution and release filter
- Update sample to reference AzureAI instead of FoundryMemory
- Delete old Microsoft.Agents.AI.FoundryMemory project and tests
* Add EnsureMemoryStoreCreatedAsync and memory existence checks to integration tests
- Ensure memory store is created before testing memory operations
- Add AZURE_AI_EMBEDDING_DEPLOYMENT_NAME config setting
- Assert memories exist in store via SearchMemoriesAsync before cleanup
- Verify scope isolation with direct memory store queries
* Fix and rename AzureAI unit tests for RAPI vs Versioned clarity
- Rename AsAIAgentAsync_* to AsAIAgent_* (drop Async from method group)
- Add _Rapi_ prefix to non-versioned (Responses API) tests
- Add _Versioned_ prefix to versioned agent tests where needed
- Fix RAPI tests: assert GetService<AIProjectClient>() is null
- Fix Versioned tests: assert IsType<FoundryAgent> and
GetService<AIProjectClient>() returns the client instance
- Fix UserAgent header tests: proper HTTP handler routing
- Fix ChatClient_UsesDefaultConversationIdAsync test setup
- All 153 unit tests pass with 0 failures
* Rename Microsoft.Agents.AI.AzureAI to Microsoft.Agents.AI.Foundry
Rename the project, namespace, folder, and all references from
Microsoft.Agents.AI.AzureAI to Microsoft.Agents.AI.Foundry.
Also rename Workflows.Declarative.AzureAI to .Foundry.
- Rename src, unit test, integration test, and workflow folders
- Update namespaces in all source and test .cs files
- Update ProjectReferences in ~47 sample and test .csproj files
- Update solution files (.slnx, .slnf)
- Update sample using statements
- Update READMEs, SKILL.md, ADRs in docs/
- Disable package validation baseline for renamed packages
- Fix UTF-8 BOM encoding on all affected .cs files
- AzureAI.Persistent left completely unchanged
* Fix format: remove ImplicitUsings, add explicit usings, fix BOM encoding
- Remove ImplicitUsings=enable from Foundry csproj to resolve IDE0005
on shared ReplacingRedactor.cs
- Add explicit System usings to all source files that relied on them
- Sort usings alphabetically per editorconfig rules
- Fix UTF-8 BOM on 12 sample Program.cs files
- Rename Azure AI Foundry Agents to Microsoft Foundry Agents in docs
alliscodeandCopilot
