* Initial plan
* .NET: Auto-wire ChatClient with OpenTelemetryChatClient in OpenTelemetryAgent
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/96dd033a-0c48-4d3f-9148-324bfd436b75
Co-authored-by: rogerbarreto <19890735+rogerbarreto@users.noreply.github.com>
* Address review: remove extension overload; honor UseProvidedChatClientAsIs; drop redundant check
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/6ac3f75d-eeb7-4811-8043-9a27511b0a8b
Co-authored-by: rogerbarreto <19890735+rogerbarreto@users.noreply.github.com>
* Resolve ChatClientAgent via GetService before checking options/chat client
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/008d914d-8cbb-4e9f-81b6-f8c3c8bd8d04
Co-authored-by: rogerbarreto <19890735+rogerbarreto@users.noreply.github.com>
* Split OpenTelemetryAgent ctor to preserve original (innerAgent, sourceName) signature
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/a890c9a7-0b77-40ab-802c-dfbf09f8c260
Co-authored-by: rogerbarreto <19890735+rogerbarreto@users.noreply.github.com>
* Preserve base AgentRunOptions properties and avoid double-wrap on user factory
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/3afbf18c-de22-4236-a2f2-02ca1e98ae21
Co-authored-by: rogerbarreto <19890735+rogerbarreto@users.noreply.github.com>
* .NET: OpenTelemetryAgent normalize sourceName once and add OTEL wiring path coverage
Normalize the configured source name once in the constructor so the outer OpenTelemetryChatClient and the auto-wired inner OpenTelemetryChatClient always emit spans on the same ActivitySource. A caller passing an empty string previously produced agent-level spans on DefaultSourceName but auto-wired chat spans on the empty source, causing the chat spans to be silently dropped by exporters subscribed to the default source.
Tests added to cover the previously unexercised OTEL wiring branches:
- Ctor_NullOrEmptySourceName_AutoWiredChatClientUsesDefaultSource_Async (Theory: null and empty)
- AutoWireChatClient_PlainAgentRunOptions_PreservesContinuationToken_Async
- AutoWireChatClient_ChatClientAgentRunOptions_NoUserFactory_PreservesChatOptions_Async
- AutoWireChatClient_StreamingDisabled_DoesNotEmitChatSpan_Async
* .NET: Mark OpenTelemetryAgent autoWireChatClient ctor as [Experimental]
Annotate the new 3-arg OpenTelemetryAgent(AIAgent, string?, bool) constructor with [Experimental(DiagnosticIds.Experiments.AgentsAIExperiments)] (MAAI001) so callers must explicitly opt in to the auto-wire toggle. The original 2-arg constructor stays non-experimental and delegates with autoWireChatClient: true; the delegating call is locally suppressed so the existing source compatibility surface is preserved.
* .NET: OpenTelemetryAgent address westey-m PR review
- Use string.IsNullOrWhiteSpace (not IsNullOrEmpty) when normalizing the constructor sourceName, so callers passing whitespace-only strings still land on OpenTelemetryConsts.DefaultSourceName instead of an unsubscribed ActivitySource.
- Fix the misleading pragma comment on the 2-arg ctor delegating call: auto-wiring is the new default, it does not preserve the original (pre-PR) behavior.
- Expand the GetRunOptionsWithChatClientWiring XML doc to spell out that a base AgentRunOptions (not ChatClientAgentRunOptions) is also accepted: it is converted to ChatClientAgentRunOptions with the auto-wire factory installed and base properties copied.
- Tests: extend the source-name normalization Theory with whitespace cases (' ' and '\t'); add end-to-end coverage for plain AgentRunOptions over a real ChatClientAgent (sync + streaming) asserting the inner chat client is invoked and both invoke_agent + chat spans are emitted.
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: rogerbarreto <19890735+rogerbarreto@users.noreply.github.com>
* .NET: Persist input messages on streaming errors in PerServiceCallChatHistoryPersistingChatClient
When the underlying chat service emits an in-stream error (for example a
`response.error` SSE event from the OpenAI Responses API on rate limit),
the OpenAI client surfaces it as an `ErrorContent` update and ends the
stream without throwing. Previously, `PerServiceCallChatHistoryPersistingChatClient`
only persisted history when the streaming loop completed successfully and
`NotifyProvidersOfNewMessagesAsync` was called at the end. On the
in-stream-error path, the input messages handed to that iteration -
typically `FunctionResultContent` produced by `FunctionInvokingChatClient`
in the previous iteration - were never persisted. The next run would
replay session history with a dangling `FunctionCallContent` and the
service would reject the request with `No tool output found for function
call <id>`.
This change:
- Adds a `PersistInputOnErrorAsync` helper that persists the input
messages (with no response messages) so function-call/function-result
pairings are not split across failures.
- Calls the helper from every error path: pre-loop enumerator creation,
the first `MoveNextAsync`, the in-loop `MoveNextAsync`, and a new
`finally` that handles abnormal iterator disposal.
- After the streaming loop, scans the assembled response for any
`ErrorContent` and, if present, persists the input, notifies
providers of failure, and throws `InvalidOperationException` so the
error is surfaced to the caller instead of silently corrupting history.
- Hardens `InMemoryChatHistoryProvider.StoreChatHistoryAsync` to treat
a null `RequestMessages` as empty, since the new error path can
invoke it with no response messages.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix dropped FunctionResultContent on streaming pipeline early-disposal
When a consumer of ChatClientAgent.RunStreamingAsync stops iterating early
(e.g. ToolApprovalAgent yields the approval request and then `yield break`),
the framework cascades DisposeAsync down the stream. C# async iterators do
not auto-dispose IAsyncDisposable locals, so the inner enumerator returned
by IChatClient.GetStreamingResponseAsync(...).GetAsyncEnumerator(ct) was
left suspended. That suspended FunctionInvokingChatClient downstream, which
suspended PerServiceCallChatHistoryPersistingChatClient at its `yield
return`, so its finally block never ran and the in-flight
FunctionResultContent for the just-completed tool call was not persisted
to chat history. The next turn then loaded a session that contained a
FunctionCallContent with no matching FunctionResultContent and the model
returned HTTP 400 `No tool output found for function call`.
Fixes:
* ChatClientAgent.RunStreamingAsync: wrap the iteration in
try/finally that disposes the inner enumerator. Disposal now cascades
through the pipeline and PerService's finally runs on early exit.
* PerServiceCallChatHistoryPersistingChatClient: in the streaming path,
snapshot input messages with `messages.ToList()` (the caller, FICC,
reuses a single mutable buffer across iterations and may mutate it
before our finally / error path persists), wrap GetAsyncEnumerator,
the first MoveNextAsync, and in-loop MoveNextAsync in try/catch each
calling PersistInputOnErrorAsync + NotifyProvidersOfFailureAsync, and
add a finally that calls PersistInputOnErrorAsync when the loop did
not exit normally so per-iteration FRCs are persisted on early
disposal as well as on errors.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* .NET: Add tests for PerService streaming error/dispose persistence paths
Adds five regression tests covering the new error-path persistence in
PerServiceCallChatHistoryPersistingChatClient.GetStreamingResponseInnerAsync:
- Persists input messages when GetStreamingResponseAsync throws synchronously.
- Persists input messages when the first MoveNextAsync throws.
- Persists input messages when a mid-stream MoveNextAsync throws.
- Persists input messages when the consumer abandons enumeration early
(the ToolApprovalAgent yield-break / disposal-cascade case).
- Throws and persists input when the stream emits an in-band ErrorContent.
All 66 tests in the class pass on net10.0 and net472.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* .NET: Address PR feedback on PerService streaming error persistence
Two follow-ups from PR #5744 review:
1. Prevent duplicate persistence on the in-loop MoveNextAsync catch path.
The inner catch persists input messages, then rethrows, which propagates
through the surrounding try/finally where loopExitedNormally is still false,
causing the finally to persist again. Introduced an inputPersisted flag
that the inner catch sets after persisting; the finally now skips when
inputPersisted is true.
2. Use the caller's CancellationToken in the abnormal-exit finally instead
of CancellationToken.None, so cleanup remains responsive to cancellation.
Fall back to CancellationToken.None only when the caller's token is
already canceled (otherwise the persist call would observe the
cancellation, throw, and mask the original early-exit reason).
Tightened all five new streaming-error tests from Times.AtLeastOnce to
Times.Once on the input-persistence matcher to regression-guard against
duplicate persistence. All 66 tests in the class still pass (net10.0 + net472).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* .NET: Scope PerService streaming changes to cooperative early-exit only
Per discussion on PR #5744, scope this PR back to fix only the original
ToolApprovalAgent dropped-FunctionResultContent bug and address the
enumerator-disposal review comment. Specifically:
- Remove input-message persistence from the GetAsyncEnumerator and
MoveNextAsync error paths. Routing failed service calls through the
success notification channel was breaking the provider contract; we
will instead rely on inner-agent retries for transient errors. Failure
paths still call NotifyProvidersOfFailureAsync as before.
- Remove the in-stream ErrorContent detection block (same rationale).
- Keep the try/finally that calls the (now narrower) early-exit input
notification on cooperative disposal (e.g. ToolApprovalAgent yield
break). A new serviceErrorOccurred flag ensures we do NOT renotify
on exception paths.
- Always DisposeAsync the underlying enumerator on every exit path,
addressing the copilot-reviewer comment about leaked HTTP/streams.
- Rename PersistInputOnErrorAsync -> NotifyProvidersOfEarlyExitInputAsync
to better reflect what it does and when it runs (rogerbarreto nit).
- Apply rogerbarreto nit on InMemoryChatHistoryProvider null-coalescing.
- Drop the four tests that covered the removed error-path behavior;
keep RunStreamingAsync_PersistsInputMessages_WhenConsumerAbandons
EnumerationAsync (regression guard for the cooperative-pause path).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: alliscode <bentho@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Adding the ability to inject messages during the function call loop
* Split message injection functionality
* Remove interface, since it is not required not that we split the chat client.
* Address conversation id propogation
* Fix formatting issue
* Bump MEAI to 10.5.1 and add per-call x-client header support
Replaces the brittle UserAgentResponsesClient subclass with a clean
per-call x-client-* header pipeline built on the new Microsoft.Extensions.AI
10.5.1 OpenAIRequestPolicies hook.
Public surface (Microsoft.Agents.AI.Foundry, [Experimental(MAAI001)]):
* chatOptions.WithClientHeader(name, value) and .WithClientHeaders(IEnumerable)
validate the x-client- prefix (case-insensitive), apply all-or-nothing on
bulk, and throw InvalidOperationException on foreign-typed slot collision
* myAgent.AsBuilder().UseClientHeaders().Build() opts a customer-built agent
into the pipeline; idempotent via agent.GetService<ClientHeadersAgent>()
* Foundry-built agents (FoundryAgent.Create*) pre-wire automatically
Internals:
* ClientHeadersAgent decorator snapshots the dict at scope-push time so
concurrent runs sharing a ChatOptions reference do not leak headers
* ClientHeadersScope is an AsyncLocal<IReadOnlyDictionary<string,string>?>
with LIFO push/dispose semantics
* ClientHeadersPolicy singleton stamps headers via Headers.Set so per-call
values overwrite any same-name header from earlier policies and so
duplicate registration is value-stable
* OpenAIRequestPoliciesReflection dedups against MEAI's private _entries
field and falls back to AddPolicy on any reflection failure; a CI test
asserts the field shape on every MEAI bump
Hosting cleanup:
* Deleted UserAgentResponsesClient and its dummy throwing pipeline
* HostedAgentUserAgentPolicy is now registered via OpenAIRequestPolicies
in FoundryHostingExtensions.TryApplyUserAgent
Tests:
* 19 new unit tests in ClientHeadersExtensionsTests.cs covering validation,
AsyncLocal isolation, snapshot semantics, end-to-end wire stamping, and
shared-chat-client dedup
* Updated OpenTelemetryAgentTests for MEAI 10.5.1 changes to web_search
serialization and the reduced tool definition payload when sensitive
data capture is disabled
Microsoft.Extensions.Compliance.Abstractions stays at 10.5.0 because no
10.5.1 release exists on nuget.org.
* Address PR review: pre-wire AsAIAgent path and dedup TryApplyUserAgent
* FoundryAgent: extract WireClientHeaders helper and call it from the
internal (AIProjectClient, ChatClientAgent) constructor used by
AzureAIProjectChatClientExtensions.AsAIAgent so those Foundry-built
agents also pre-wire the x-client header pipeline.
* Foundry.Hosting TryApplyUserAgent: dedup HostedAgentUserAgentPolicy
registration per OpenAIRequestPolicies instance via
ConditionalWeakTable so per-request resolution does not grow the
policy list unboundedly on singleton agents.
* Add tests covering AsAIAgent pre-wire and TryApplyUserAgent dedup
Backs the PR review fixes from a4c8f91 with regression tests:
* ClientHeadersExtensionsTests: AsAIAgent_FoundryAgent_HasPreWiredClientHeadersAgent
asserts the FoundryAgent built via AzureAIProjectChatClientExtensions.AsAIAgent
contains a ClientHeadersAgent in its delegating chain (catches future
regressions of the bypass).
* ClientHeadersExtensionsTests: FoundryAgent_PublicConstructor_HasPreWiredClientHeadersAgent
covers the public constructor path the same way.
* ClientHeadersExtensionsTests: UseClientHeaders_RepeatedRegistrations_OnSameChatClient_OnlyRegistersOnce
invokes UseClientHeaders 25 times on a shared chat client and asserts via
reflection that OpenAIRequestPolicies._entries length is exactly 1.
* HostedTryApplyUserAgentDedupTests: two tests asserting
FoundryHostingExtensions.TryApplyUserAgent stays at one entry per
OpenAIRequestPolicies instance after 50 calls on the same agent and across
distinct agents on different chat clients.
* Move tests next to their SUT
Removes the dedicated HostedTryApplyUserAgentDedupTests.cs test class.
Tests are co-located with the SUT they exercise:
* FoundryAgentTests.cs gains the Constructor_PreWiresClientHeadersAgent
and Constructor_FromAsAIAgentExtension_PreWiresClientHeadersAgent
cases, since FoundryAgent is the SUT for the pre-wire behavior.
* HostedOutboundUserAgentTests.cs gains the two TryApplyUserAgent dedup
cases, since FoundryHostingExtensions.TryApplyUserAgent is the SUT
it already covers.
* ClientHeadersExtensionsTests.cs keeps only the
UseClientHeaders_RepeatedRegistrations_OnSameChatClient_OnlyRegistersOnce
case, which exercises the public ClientHeadersExtensions surface.
* Remove redundant WithCancellation on inner streaming call
ct is already passed to InnerAgent.RunStreamingAsync, so
.WithCancellation(ct) on the resulting IAsyncEnumerable is a no-op.
Caught by Sergey on PR review.
* Address PR review: surface downstream MEAI experimental ID
* Add AIOpenAIRequestPolicies = MEAIExperiments alias to
DiagnosticIds.Experiments (matches the existing AIResponseContinuations,
AIMcpServers, AIFunctionApprovals pattern).
* Mark public ClientHeadersExtensions with [Experimental(AIOpenAIRequestPolicies)]
instead of AgentsAIExperiments. Consumers now see the MEAI001 warning,
surfacing the dependency on MEAI's experimental OpenAIRequestPolicies hook.
* Mark internal OpenAIRequestPoliciesReflection with the same alias to
suppress warnings at the source rather than via project-wide NoWarn.
* Remove MEAI001 from Foundry csproj NoWarn (kept on Foundry.Hosting where
pre-PR usages remain).
* Clarify ClientHeadersScope XML doc: AsyncLocal flows values forward but
does NOT auto-restore on method return; explicit using/Dispose is what
gives stack-style LIFO semantics.
* Foundry Evals integration for .NET
- Core evaluation framework: EvalItem, LocalEvaluator, FunctionEvaluator, EvalChecks
- IAgentEvaluator interface with MeaiEvaluatorAdapter bridge
- AgentEvaluationExtensions for agent.EvaluateAsync() overloads
- FoundryEvals wrapping MEAI quality/safety evaluators
- ConversationSplitters (LastTurn, Full) and IConversationSplitter
- EvalItem.PerTurnItems() for multi-turn decomposition
- HasImageContent for multimodal content detection
- WorkflowEvaluationExtensions for per-agent workflow evaluation
- 7 eval samples mirroring Python parity:
02-agents/Evaluation: SimpleEval, ExpectedOutputs, Multimodal
03-workflows/Evaluation: WorkflowEval
05-end-to-end/Evaluation: FoundryQuality, MixedProviders, ConversationSplits
- Comprehensive unit tests (1958 passing)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Rewrite FoundryEvals to use real Foundry Evals API
Replace MEAI evaluator shim with actual OpenAI EvaluationClient protocol
methods. FoundryEvals now creates eval definitions, submits runs, polls
for completion, and fetches per-item results server-side.
- New constructor: FoundryEvals(AIProjectClient, model, evaluators)
- Add FoundryEvalConverter for MEAI ChatMessage -> Foundry JSON format
- Add EvalId, RunId, ReportUrl to AgentEvaluationResults
- All 20 built-in evaluator constants now work (agent, tool, quality, safety)
- Remove Microsoft.Extensions.AI.Evaluation.Quality/Safety dependencies
- Update all samples for new constructor (no more ChatConfiguration)
- Replace BuildEvaluators tests with ResolveEvaluator tests
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add response output to CustomEvals and ExpectedOutputs samples
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review: pagination, validation, error handling, tests
FoundryEvals fixes:
- Add pagination for output items (has_more/after cursor)
- Add guard clauses for pollIntervalSeconds/timeoutSeconds <= 0
- Fix double TryGetProperty for passed field parsing
- Throw on all-tool-evaluators with no tool definitions
- Fix XML doc (default 300s, not 180s)
New tests (30 added, 1989 total):
- EvalChecks: NonEmpty, ContainsExpected (pass/fail/skip/case),
HasImageContent, ToolCallsPresent
- FoundryEvalConverter: ConvertMessage (text, image, function call,
function results fan-out, empty fallback, mixed content),
ConvertEvalItem, BuildTestingCriteria (quality/agent/tool/groundedness
data mappings), BuildItemSchema
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix review: null-refs, Data.ToString() bug, ContainsExpected, add tests
- Fix NullReferenceException in sample Response display (pattern matching)
- Fix WorkflowEvaluationExtensions Data?.ToString() producing type names
instead of message text (pattern-match ChatMessage/AgentResponse/list)
- Change EvalChecks.ContainsExpected to return Passed=false when no
ExpectedOutput (was silently passing, masking misconfiguration)
- Add EvalItem constructor tests with LastTurn/Full/null splitters
- Add FoundryEvalConverter.ConvertMessage DataContent (base64 image) test
- Add ExtractAgentData tests with ChatMessage, list, and AgentResponse data
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix review: conversation fidelity, eval caching, fallback tests
- WorkflowEvaluationExtensions: preserve full response messages (tool calls,
intermediate) instead of synthetic 2-message conversation. Cast completed
Data to AgentResponse and use Messages when available, fallback to text.
- FoundryEvals: cache evalId per schema shape (hasContext, hasTools) so
subsequent EvaluateAsync calls create runs under the same eval definition.
- MeaiEvaluatorAdapter: code already correctly passes queryMessages (not full
conversation) to IEvaluator — no change needed, verified by inspection.
- Add tests: AgentResponse full messages preservation, unknown object
ToString() fallback for ExtractAgentData.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Rename AzureAI→Foundry: move eval files, update references
- Move FoundryEvals.cs and FoundryEvalConverter.cs from
Microsoft.Agents.AI.AzureAI to Microsoft.Agents.AI.Foundry
- Update namespace from AzureAI to Foundry in both files
- Add explicit usings required by Foundry project (no implicit usings)
- Move FoundryEvalConverter tests to Foundry.UnitTests project
(avoids ReplacingRedactor type conflict from dual project refs)
- Update all sample csproj references and using statements
- Remove Foundry project reference from AI UnitTests
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* PR review round 4: wire up tool extraction, remove eval cache, fix null safety
- BuildEvalItem: extract tools from agent via GetService<ChatOptions>() into EvalItem.Tools (Python parity)
- FoundryEvals: remove eval ID cache - each call creates fresh definition (matches Python behavior)
- FoundryEvals: replace null-forgiving operators with descriptive InvalidOperationException
- MixedProviders sample: remove unnecessary explicit PackageReferences (transitively provided)
- FoundryEvalConverter: document that tool results take precedence over text content
- Add LocalEvaluator zero-checks test documenting 0 metrics = failed behavior
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Python-dotnet parity: 9 feature gaps filled
New checks:
- ToolCallArgsMatch() — verify tool call names + argument subset match
- ToolCalledCheck(ToolCalledMode.Any, ...) — match any of the specified tools
- ToolCalledMode enum (All/Any)
FoundryEvals enhancements:
- Default evaluators now [Relevance, Coherence, TaskAdherence] (was Relevance, Coherence)
- Auto-add ToolCallAccuracy when items have tool definitions
- EvaluateTracesAsync — evaluate by response_ids, trace_ids, or agent_id
- EvaluateFoundryTargetAsync — evaluate deployed Foundry targets
Result type enrichment:
- AgentEvaluationResults: added Status, Error, PerEvaluator, DetailedItems
- New EvalItemResult/EvalScoreResult/PerEvaluatorResult types
- FoundryEvals populates all new fields from API responses
Workflow fix:
- Skip internal executors (_*, input-conversation, end-conversation, end)
Tests: 8 new tests covering ToolCallArgsMatch, ToolCalledMode.Any, internal executor filtering
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add MeaiEvaluatorAdapter and PerTurnItems edge case tests
- 3 tests for MeaiEvaluatorAdapter: query message forwarding, synthetic
response fallback, multiple items aggregation
- 3 tests for EvalItem.PerTurnItems: empty conversation, no user messages,
system+assistant only
- StubEvaluator and StubChatClient test helpers
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Blocking link check for outdated package in DevUI.
* Replace Dictionary<string, object> payloads with typed wire models
Introduce internal FoundryEvalWireModels.cs with compile-time-safe types
for the OpenAI Evals API wire format. The OpenAI .NET SDK (2.9.1) only
provides protocol-level methods with BinaryContent/ClientResult — no
typed request models. These internal models replace scattered dictionary
literals with [JsonPropertyName]-annotated classes, giving:
- Compile-time safety (typos become build errors)
- Single point of change when the API evolves
- IntelliSense discoverability
- Cleaner serialization via JsonPolymorphic for content items
Models: WireContentItem hierarchy (text, image, tool_call, tool_result),
WireMessage, WireEvalItemPayload, WireTestingCriterion, WireItemSchema,
WireCreateEvalRequest, WireCreateRunRequest, and data source variants.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Skip metric when Foundry returns neither score nor passed
When an evaluator returns no score and no passed value, the previous
code created BooleanMetric(name, false), which falsely failed items
via ItemPassed. Now we skip the MEAI metric entirely for indeterminate
results — the raw data remains available in DetailedItems for diagnostics.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR #4914 review comments: fix tool evaluator bug and add tests
- Fix duplicate ToolCallAccuracy: resolve evaluator names before checking
against ToolEvaluators set (Comment 2)
- Make FilterToolEvaluators internal for testability; add tests for the
ArgumentException edge case when all evaluators are tool-type (Comment 3)
- Add CancellationToken test for LocalEvaluator (Comment 4)
- Add EvaluateAsync integration test on Run with sequential workflow and
per-agent SubResults verification (Comment 5)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address Peter's review comments on PR #4914
- Add trailing newline to Evaluation_FoundryQuality.csproj (Comment 6)
- Make evaluator name lookups case-insensitive: switch BuiltinEvaluators,
ToolEvaluators, AgentEvaluators, and ResolveEvaluator's StartsWith check
from Ordinal to OrdinalIgnoreCase (Comment 7)
- Add Trace.TraceWarning when Foundry returns fewer results than submitted
items, indicating expected vs actual count before padding (Comment 8)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add Microsoft.Extensions.AI.Evaluation packages to Directory.Packages.props
These were removed in #5269 as unused, but are needed by the Foundry
and core evaluation integration added in this PR.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: alliscode <bentho@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rename authored identifiers, XML docs, log messages, and comments
from 'folder' to 'directory' across the file skills codebase for
consistency with the agentskills.io specification and .NET conventions.
Public API changes (experimental):
- ScriptFolders → ScriptDirectories
- ResourceFolders → ResourceDirectories
.NET BCL API calls (Directory.Exists, Path.GetDirectoryName, etc.)
were already using 'directory' and are unchanged.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* support reflection for discovery of resources and scripts in class-based skills
* fix format issues
* refactor samples to use reflection
* Validate resource member signatures during discovery
Add discovery-time validation in AgentClassSkill.DiscoverResources() to
fail fast when [AgentSkillResource] is applied to members with incompatible
signatures:
- Reject indexer properties (getter has parameters)
- Reject methods with parameters other than IServiceProvider or
CancellationToken
Throws InvalidOperationException with actionable error messages instead of
allowing silent runtime failures when ReadAsync invokes the AIFunction with
no named arguments.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* prevent duplicates
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* .NET: Add JsonSerializerOptions support to programmatic skill APIs
Allow callers to pass custom JsonSerializerOptions when creating inline
resources and scripts via AgentInlineSkill, AgentClassSkill,
AgentInlineSkillResource, and AgentInlineSkillScript. A skill-level
default can be set on AgentInlineSkill and overridden per-resource/
script call.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update dotnet/tests/Microsoft.Agents.AI.UnitTests/AgentSkills/TestSkillTypes.cs
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* add class-based skills
* address formating issues
* Remove generated filtered-unit.slnx and add to .gitignore
The filtered solution file is generated dynamically by
eng/scripts/New-FilteredSolution.ps1 during CI. Checking it in
risks it becoming stale and out-of-sync with the real solution.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove generated filtered-unit.slnx and add to .gitignore
The filtered solution file is generated dynamically by
eng/scripts/New-FilteredSolution.ps1 during CI. Checking it in
risks it becoming stale and out-of-sync with the real solution.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* discover scripts and resource from folders defined in spec
* Remove Step05 and Step06 DI skill samples
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* address review comments
* fix build error
* Fix mixed path separators in skill folder discovery on .NET Framework
Path.Combine with forward-slash folder names (e.g. "scripts/f1") produces
mixed separators on Windows, causing the StartsWith containment check to
fail against Path.GetFullPath-resolved file paths. Wrap in Path.GetFullPath
to canonicalize separators before the containment comparison.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* address comment
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* add class-based skills
* address formating issues
* Remove generated filtered-unit.slnx and add to .gitignore
The filtered solution file is generated dynamically by
eng/scripts/New-FilteredSolution.ps1 during CI. Checking it in
risks it becoming stale and out-of-sync with the real solution.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove generated filtered-unit.slnx and add to .gitignore
The filtered solution file is generated dynamically by
eng/scripts/New-FilteredSolution.ps1 during CI. Checking it in
risks it becoming stale and out-of-sync with the real solution.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* consolidate DI samples into one
* fix file encoding
* suppress compatibility warning
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* add inline skills
* Fix IDE1006 and IDE0004 formatting errors in test files
- Add 'Async' suffix to async test methods in FilteringAgentSkillsSourceTests,
DeduplicatingAgentSkillsSourceTests, and AgentInMemorySkillsSourceTests
- Use pragma to suppress false-positive IDE0004 on casts needed for overload
disambiguation in AgentInlineSkillTests and AgentInlineSkillResourceTests
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* address issues
* address comments
* make inline skills script and resource model classes internal
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Changes
* Fix ChatClientAgent streaming responses missing MessageId
Generate fallback MessageId in ChatClientAgent.RunCoreStreamingAsync when
the underlying LLM provider does not set ChatResponseUpdate.MessageId.
Without a MessageId the AGUI converter's null==null check silently drops
all text content, causing CopilotKit Zod validation errors.
Changes:
- ChatClientAgent: generate msg_{Guid} fallback via ??= in streaming loop
- AgentResponseExtensions: sync wrapper MessageId back to RawRepresentation
in AsChatResponseUpdate() so downstream consumers see the value
- Add unit tests for both fixes and AGUI streaming MessageId scenarios
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR #4615 review comments
- Fix MessageId seeding: use first-seen provider MessageId (or generate
fallback) and apply consistently to all chunks in the stream, preventing
message splitting when providers set MessageId only on the first chunk
- Add test for mixed MessageId scenario (first chunk only)
- Fix skipped TextStreaming test: assert Empty (not NotEmpty) to match
actual null==null behavior
- Fix skipped ToolCalls test: assert empty ParentMessageId to match
actual empty-string passthrough behavior
* Handle empty MessageId in AsChatResponseUpdate sync
Treat empty/whitespace MessageId the same as null when syncing from
the AgentResponseUpdate wrapper back to RawRepresentation. Providers
that return empty string MessageId (e.g. tool call responses) now get
the wrapper value recovered correctly.
Add test for empty string MessageId recovery scenario.
* Move MessageId fallback generation to AGUI layer
Move fallback MessageId generation from ChatClientAgent to
AsAGUIEventStreamAsync, addressing the architectural concern that
MessageId is nullable in the AIAgent abstraction and the requirement
for non-null values is specific to the AGUI protocol.
The AGUI layer now generates a fallback MessageId for null or
empty/whitespace values, covering all agent types (not just
ChatClientAgent) including external implementations.
Changes:
- Revert MessageId generation from ChatClientAgent.RunCoreStreamingAsync
- Add fallback MessageId generation in AsAGUIEventStreamAsync for
null/empty MessageId values (handles both null and whitespace)
- Unskip and update AGUI tests to verify fallback generation
- Update ChatClientAgent tests to reflect passthrough behavior
* Revert AsChatResponseUpdate MessageId sync-back
Remove the MessageId sync-back logic from AsChatResponseUpdate() as it
is no longer needed. With fallback generation moved to the AGUI layer,
the abstraction layer should not mutate the RawRepresentation object.
Revert to the original passthrough behavior for AsChatResponseUpdate()
and update tests accordingly.
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Persist messages during the Function Call Loop
* Revert version reset
* Fix bugs and improve sample
* Fix formatting issues
* Also updating conversation id during run
* Update based on ADR feedback
* Fix FileAgentSkillsProvider accepting SkillsInstructionPrompt without {0} placeholder (#4638)
BuildSkillsInstructionPrompt validated only format-string syntax via
string.Format(template, ""), which silently accepted templates without a
{0} placeholder. The generated skills list was then dropped from the final
instructions.
Tighten validation to format with a sentinel string and verify it appears
in the output, rejecting templates that do not reference argument 0 with
an ArgumentException.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix netstandard2.0 compat and simplify prompt template validation (#4638)
- Replace string.Contains(string, StringComparison) with IndexOf for
netstandard2.0/net472 compatibility
- Remove sentinel round-trip check; validate {0} directly on the raw
template string using IndexOf
- Add positive test verifying custom SkillsInstructionPrompt with {0}
is accepted and applied to output
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix filter combine logic for ChatHistoryMemoryProvider
* Replace var with explicit types in filter building code and test
Address PR review nit: use explicit types instead of var for better
readability in the filter-building logic and the new combined filter
compilation test.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix style issues
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* support script execution by code interpretor
* improve the instruction prompt
* Add DefaultAzureCredential production warning to AgentSkills samples
Add the standard three-line WARNING comment about DefaultAzureCredential
production considerations to both AgentSkills sample Program.cs files,
matching the convention used in all other GettingStarted/Agents samples.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* address pr review comments
* address feedback
* rename Skill* types to FileAgentSkill* prefix for consistency
- Rename SkillFrontmatter -> FileAgentSkillFrontmatter
- Rename SkillScriptExecutor -> FileAgentSkillScriptExecutor
- Add FileAgentSkillScriptExecutionContext and FileAgentSkillScriptExecutionDetails
- Update sample, provider, loader, and tests accordingly
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* reorder usings
* use set for props initialization instead of init
* rename HostedCodeInterpreterSkillScriptExecutor
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When converting base AgentRunOptions to ChatClientAgentRunOptions, the middleware
now preserves AllowBackgroundResponses, ContinuationToken, and AdditionalProperties
in addition to ResponseFormat.
Added unit test verifying all properties are preserved during the conversion.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add ChatClient decorator for calling AIContextProviders
* Format new files
* Address PR comments
* Revert problematic change
* Rename Use to UseAIContextProvider
* .NET: [BREAKING] Add session statebag to use for state storage instead of inside providers (#3737)
* Add a StateBag to AgentSession and pass Agent and AgentSession to AIContextProvider and ChatHistoryProviders
* Convert all AIContextProviders to use the statebag
* Update InMemoryChatHistoryProvider to use StateBag
* Update Comsos and Workflow ChatHistoryProviders
* Update 3rd party chat history storage sample.
* Remove serialize method from providers
* Replacing provider factories with properties
* Remove Providers from Session and flatten state bag serialization
* Update samples to use getservice on agent
* Updated additional session types to serialize statebag
* Fix regression
* Address PR comments
* Address PR comments.
* Fix formatting
* Fix unit tests
* Remove InMemoryAgentSession since it is not required anymore.
* Address PR comments
* Convert sessions for A2AAgent, ChatClientAgent, CopilotStudioAgent and GithubCopilotAgent to use regular json serialization.
* Fix durable agent session jso usgae
* Add jso to InMemory and Workflow ChatHistoryProviders
* Update InMemoryChatHistoryProvider to use an options class for it's many optional settings.
* Apply suggestions from code review
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Address PR feedback
* Fix verification bug.
* Improve state bag thread safety
* Address PR comments and fix unit tests
* Address PR comments
* Fix unit test
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Add a public StateKey property to providers (#3810)
* .NET: [BREAKING] Update providers in such a way that they can participate in a pipeline (#3846)
* Make providers pipeline capable
* Fix unit tests
* Move source stamping to providers from base class
* Also update samples.
* Address PR comments
* Rename AsAgentRequestMessageSourcedMessage to WithAgentRequestMessageSource
* .NET: [BREAKING] Add consistent message filtering to all providers. (#3851)
* Add consistent message filtering to all providers.
* Remove old chat history filtering classes
* Fix merge issues
* Fix unit test
* Enforce non-nullable property
* Fix merging bug and make troubleshooting source info easier by adding tostring implementation
* .NET: [BREAKING] Add support for multiple AIContextProviders on a ChatClientAgent (#3863)
* Add support for multiple AIContextProviders on a ChatClientAgent
* Address PR comments and fix tests
* Address PR comments.
* .NET: [BREAKING]Delay AIContext Materialization until the end of the pipeline is reached. (#3883)
* Delay AIContext Materialization until the end of the pipeline is reached.
* Address PR comments.
* Address PR comments
* Modify InvokedContext to be immutable (#3888)
* .NET: Address Feedback on StateBag feature branch PR (#3910)
* Address Feedback on statebag feature branch PR
* Update dotnet/src/Microsoft.Agents.AI.DurableTask/CHANGELOG.md
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Address PR comments
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
RunCoreStreamingAsync was passing inputMessagesForProviders (which lacks
chat history) to GetStreamingResponseAsync instead of
inputMessagesForChatClient (which includes chat history). This caused
streaming runs to lose conversation context on subsequent calls.
The non-streaming path (RunCoreAsync) already correctly used
inputMessagesForChatClient. This aligns the streaming path to match.
Also adds a unit test that validates chat history is included in
messages sent to the chat client during streaming on subsequent calls.
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>
westey ·
Copilot ·
Ben Thomas ·
Roger Barreto ·
Teja Kusireddy ·
SergeyMenshykh ·
chetantoshniwal ·
Evan Mattson ·
Chris ·
Stephen Toub ·