Compare commits

...
Author SHA1 Message Date
Giles Odigwe e33d3e5bc6 remove load_dotenv from test file 2026-04-30 14:02:42 -07:00
Giles OdigweandCopilot 097095c1ea Fix Ollama pull failure propagation and Azure OpenAI vector store readiness
- Ollama CI: fail the step immediately if model pull fails after 3
  retries instead of silently proceeding to tests
- Azure OpenAI file search: add the same vector-store readiness polling
  that was applied to the non-Azure OpenAI tests, preventing eventual
  consistency race conditions

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-30 11:42:18 -07:00
Giles Odigwe 0edd5f1b32 Merge branch 'main' into flaky-test-report 2026-04-30 10:28:28 -07:00
Giles OdigweandCopilot 52589ab474 Rename flaky_report to integration_test_report and add try/finally cleanup
- Rename scripts/flaky_report/ to scripts/integration_test_report/ to
  reflect expanded scope beyond flaky-test detection
- Update workflow references in both CI files
- Wrap file search integration tests in try/finally to ensure vector
  store cleanup runs even on test failure or timeout

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-30 10:22:00 -07:00
Giles Odigwe d2de5ba1b5 Revert "Re-enable workflow parallel tests with xdist_group marker"
This reverts commit 455c28da62.
2026-04-30 09:43:06 -07:00
6cd81286a9 .NET: dotnet: Add hosted-agent User-Agent supplement to outgoing requests (#5453)
* dotnet: Add hosted-agent User-Agent supplement to outgoing requests

When an agent runs inside a Foundry Hosted Agent, the outgoing
User-Agent header now includes 'agent-framework-hosted/{version}'
alongside the existing 'MEAI/{version}' segment.

- Add HostedAgentContext with AsyncLocal<string?> property
- MeaiUserAgentPolicy reads the supplement per-call
- AgentFrameworkResponseHandler sets/restores the context

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chore: update hosted UA format to foundry-hosting/agent-framework-dotnet/{version}

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Trying to get UA flowing, no luck yet.

* .NET: Polyfill MEAI OpenAIResponsesChatClient to add hosted-agent User-Agent supplement

When AgentFrameworkResponseHandler resolves an agent (i.e. we are running in a
hosted context), TryApplyUserAgent walks the agent's IChatClient decorator chain
to find MEAI's internal OpenAIResponsesChatClient and reflectively swaps its
inner _responseClient field with a DelegatingResponsesClient wrapper. The
wrapper overrides the public-virtual protocol methods to add a per-call
HostedAgentUserAgentPolicy to the RequestOptions and delegate to the inner
ResponsesClient. The OpenAI SDK's internal streaming overloads bottom out in
calls to the public-virtual non-streaming overloads via virtual dispatch on
this, so streaming is covered without overriding any non-virtual member.

The wrapper accepts any ResponsesClient-derived inner — both the Foundry
ProjectResponsesClient and the native OpenAI ResponsesClient — and preserves
the inner client's full pipeline (Transport, RetryPolicy, NetworkTimeout,
OrganizationId / ProjectId / UserAgentApplicationId, custom policies).

- Add DelegatingResponsesClient + HostedAgentUserAgentPolicy in Microsoft.Agents.AI.Foundry.Hosting.
- Add TryApplyUserAgent next to ApplyOpenTelemetry in FoundryHostingExtensions; wire it into AgentFrameworkResponseHandler.GetAgent for both keyed and default-agent paths.
- Drop earlier-iteration dead code: AddHostedAgentTelemetry extension, HostedUserAgentPolicy class, HostedAgentContext.cs, and the never-called ToRequestOptions helper.
- Revert RequestOptionsExtensions.MeaiUserAgentPolicy to MEAI-only (the supplement is now injected by the polyfill).
- Revert unrelated whitespace change in Agent_Step25_ToolboxServerSideTools sample.
- Tests cover streaming AND non-streaming, retry policy preservation, OrganizationId/ProjectId/UserAgentApplicationId pass-through, idempotency, native OpenAI ResponsesClient, and reflection guards for MEAI/OpenAI shape drift.

* .NET: Address review feedback on hosted-agent User-Agent polyfill

- TryApplyUserAgent: replace silent null-return with ArgumentNullException to match the codebase's convention.
- Add idempotency test (TryApplyUserAgent_CalledTwiceOnSameAgent_DoesNotDoubleWrap) — runs the polyfill twice on the same agent and asserts the wire UA contains exactly one foundry-hosting segment, proving the 'current is DelegatingResponsesClient' guard prevents nested wrapping.
- Add retry-double-append test (Polyfill_RetryWithinCall_DoesNotDuplicateSupplementInUserAgent) — exercises the HostedAgentUserAgentPolicy Contains-guard via a custom retry policy that re-runs the inner pipeline on the same message.
- Replace TryApplyUserAgent_NullAgent_ReturnsNullWithoutThrowing with TryApplyUserAgent_NullAgent_ThrowsArgumentNullException to match the new contract.

* .NET: Drop null check from TryApplyUserAgent and its now-redundant test

The two call sites in AgentFrameworkResponseHandler.GetAgent already null-check the agent before invoking TryApplyUserAgent, so the defensive ArgumentNullException is unreachable. Remove it and the corresponding test.

* .NET: Remove unused Microsoft.Shared.Diagnostics import in ServiceCollectionExtensions

The Throw.IfNull helper from this namespace was used by the now-removed null check in TryApplyUserAgent. Drop the unused import to satisfy IDE0005 in CI's full-project dotnet format run.

---------

Co-authored-by: alliscode <bentho@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Roger Barreto <19890735+rogerbarreto@users.noreply.github.com>
2026-04-30 16:37:54 +00:00
Giles OdigweandCopilot 455c28da62 Re-enable workflow parallel tests with xdist_group marker
The tests were skipped because xdist distributes module tests across
workers, each spawning their own func process (port conflicts). Adding
xdist_group forces all tests in this module onto a single worker so
the module-scoped function_app_for_test fixture works correctly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-30 09:04:46 -07:00
Giles OdigweandCopilot 7ce27ddda3 Increase reliable streaming test timeouts from 30s to 60s
The LLM call through Azure OpenAI + Redis streaming pipeline can exceed
30s in CI due to cold starts or throttling. Raise to 60s to reduce
flaky timeouts while still bounded by pytest's 120s per-test limit.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-30 08:42:08 -07:00
Giles OdigweandCopilot acf24ea2e4 Stabilize Ollama tool call integration tests with no-arg function
Use a no-argument greet() function instead of hello_world(arg1) for
integration tests. The 1.5B model in CI is unreliable at generating
correct tool call arguments, causing 'Argument parsing failed' errors.
A no-arg function eliminates this flakiness entirely.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-30 00:36:09 -07:00
Giles OdigweandCopilot 3ab3370a8e Remove temperature from foundry hosting test (unsupported by CI model)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-29 15:31:57 -07:00
Giles OdigweandCopilot 072123a8f1 Fix flaky integration tests and re-enable skipped tests
- Foundry agent: add allow_preview=True to custom client test
- Foundry hosting: raise max_output_tokens 50->200, add temperature,
  relax assertion in test_temperature_and_max_tokens
- Foundry embedding: update skip reason with root cause (endpoint mismatch)
- OpenAI file search: fix vector store indexing race condition by polling
  file_counts before querying; fix get_streaming_response -> get_response(stream=True)
- Azure OpenAI file search: remove skip (transient 500 resolved)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-29 15:06:17 -07:00
Giles Odigwe 2e6b999bd2 Merge branch 'main' into flaky-test-report 2026-04-29 08:55:29 -07:00
Giles OdigweandCopilot e2eba0bacc Add retry logic and port-conflict fix for Ollama CI setup
- Kill any auto-started Ollama before launching serve (fixes port
  conflict: 'address already in use')
- Retry ollama pull up to 3 times with 15s backoff (fixes 429 rate
  limit failures)
- Applied to both python-merge-tests.yml and python-integration-tests.yml

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-28 16:51:07 -07:00
Giles OdigweandCopilot 386e08ed64 Fix E501 line-too-long in azurefunctions parallel test skip reasons
Wrap skip reason strings to stay within 120 char line limit.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-28 16:48:35 -07:00
Giles OdigweandCopilot 374526515d Re-skip parallel workflow tests: xdist worker distribution issue
The 4 parallel workflow tests crash because xdist worksteal distributes
them across separate workers, each spawning its own func process against
shared emulators. Auth fix (api_key->credential) was valid and stays.
test_conditional_branching now passes with the auth fix.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-27 06:38:19 -07:00
Giles OdigweandCopilot a6e0ab5603 Fix auth routing in samples 06/11: api_key -> credential for Azure OpenAI
Both samples passed a bearer token provider via api_key= which caused the
client to route to api.openai.com instead of Azure OpenAI, resulting in
401 Unauthorized. Changed to credential= which correctly triggers Azure
routing and picks up AZURE_OPENAI_ENDPOINT from the environment.

- samples/azure_functions/11_workflow_parallel/function_app.py: 1 fix
- samples/durabletask/06_multi_agent_orchestration_conditionals/worker.py: 2 fixes
- Re-enable 4 parallel workflow tests and 1 conditional branching test

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-27 06:12:28 -07:00
Giles OdigweandCopilot f6f87477c9 Re-skip failing Functions/DurableTask tests with specific root causes
- test_11_workflow_parallel (4 tests): xdist worker crashes during execution
- test_conditional_branching: orchestration fails with RuntimeError, not a timeout
- Keep 480s timeout bump for remaining Functions tests

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-27 05:37:50 -07:00
Giles OdigweandCopilot 9316f2c2f8 Re-enable skipped Functions/DurableTask tests and bump timeout to 480s
- Remove hard skips from 4 tests in test_11_workflow_parallel.py
- Remove hard skip from test_conditional_branching in test_06_dt_multi_agent_orchestration_conditionals.py
- Increase pytest --timeout from 360 to 480 for Functions+DurableTask CI job
- Updated in both python-merge-tests.yml and python-integration-tests.yml

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-24 11:00:29 -07:00
Giles OdigweandCopilot dc64d63a2a Re-enable reliable streaming integration tests
Remove the hard skip on test_03_reliable_streaming tests that was
temporarily disabled for instability investigation. CI infrastructure
(Azurite, DTS emulator, Redis, func CLI) is already in place.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-24 07:35:39 -07:00
Giles OdigweandCopilot 733bfb9bfe Bump Ollama model to qwen2.5:1.5b for better instruction following
The 0.5b model was too small to reliably follow simple prompts like
'Say Hello World', causing test assertion failures. The 1.5b model
follows instructions more reliably while still being small enough
for fast CI pulls (~1GB).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-23 14:51:16 -07:00
Giles OdigweandCopilot 101f50134c Enable Ollama integration tests in CI and rename report to Integration Test Report
- Install Ollama, cache models (qwen2.5:0.5b + nomic-embed-text), and start
  server in the Misc integration job for both workflow files
- Set OLLAMA_MODEL and OLLAMA_EMBEDDING_MODEL env vars so the 5 Ollama tests
  are no longer skipped
- Rename Flaky Test Report to Integration Test Report throughout (job names,
  artifact names, cache keys, file names, script titles/docstrings)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-23 14:25:38 -07:00
26 changed files with 1287 additions and 339 deletions
+57 -18
View File
@@ -157,6 +157,8 @@ jobs:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
ANTHROPIC_CHAT_MODEL: ${{ vars.ANTHROPIC_CHAT_MODEL_ID }}
LOCAL_MCP_URL: ${{ vars.LOCAL_MCP__URL }}
OLLAMA_MODEL: qwen2.5:1.5b
OLLAMA_EMBEDDING_MODEL: nomic-embed-text
defaults:
run:
working-directory: python
@@ -171,6 +173,43 @@ jobs:
with:
python-version: ${{ env.UV_PYTHON }}
os: ${{ runner.os }}
- name: Install Ollama
run: curl -fsSL https://ollama.com/install.sh | sh
working-directory: .
- name: Cache Ollama models
uses: actions/cache@v4
with:
path: ~/.ollama/models
key: ollama-models-qwen2.5-1.5b-nomic-embed-text-v1
- name: Start Ollama and pull models
run: |
# Stop any Ollama instance auto-started by the install script
pkill ollama || true
sleep 2
ollama serve &
for i in $(seq 1 30); do
if curl -sf http://localhost:11434/api/tags > /dev/null 2>&1; then
break
fi
sleep 1
done
# Pull models with retry for transient 429 rate limits
for model in qwen2.5:1.5b nomic-embed-text; do
pulled=false
for attempt in 1 2 3; do
if ollama pull "$model"; then
pulled=true
break
fi
echo "Retry $attempt for $model (waiting 15s)..."
sleep 15
done
if [ "$pulled" != "true" ]; then
echo "ERROR: Failed to pull $model after 3 attempts"
exit 1
fi
done
working-directory: .
- name: Start local MCP server
id: local-mcp
uses: ./.github/actions/setup-local-mcp-server
@@ -271,7 +310,7 @@ jobs:
-m integration
-n logical --dist worksteal
-x
--timeout=360 --session-timeout=900 --timeout_method thread
--timeout=480 --session-timeout=900 --timeout_method thread
--retries 2 --retry-delay 5
--junitxml=pytest.xml
- name: Upload test results
@@ -435,9 +474,9 @@ jobs:
path: ./python/pytest.xml
if-no-files-found: ignore
# Flaky test trend report (aggregates per-job JUnit XML results)
python-flaky-test-report:
name: Flaky Test Report
# Integration test trend report (aggregates per-job JUnit XML results)
python-integration-test-report:
name: Integration Test Report
if: >
always() &&
(contains(join(needs.*.result, ','), 'success') ||
@@ -471,36 +510,36 @@ jobs:
with:
pattern: test-results-*
path: test-results/
- name: Restore flaky report history cache
- name: Restore report history cache
uses: actions/cache/restore@v4
with:
path: python/flaky-report-history.json
key: flaky-report-history-integration-${{ github.run_id }}
path: python/integration-report-history.json
key: integration-report-history-integration-${{ github.run_id }}
restore-keys: |
flaky-report-history-integration-
integration-report-history-integration-
- name: Generate trend report
run: >
uv run python scripts/flaky_report/aggregate.py
uv run python scripts/integration_test_report/aggregate.py
../test-results/
flaky-report-history.json
flaky-test-report.md
integration-report-history.json
integration-test-report.md
- name: Post to Job Summary
if: always()
run: cat flaky-test-report.md >> $GITHUB_STEP_SUMMARY
- name: Save flaky report history cache
run: cat integration-test-report.md >> $GITHUB_STEP_SUMMARY
- name: Save report history cache
if: always()
uses: actions/cache/save@v4
with:
path: python/flaky-report-history.json
key: flaky-report-history-integration-${{ github.run_id }}
path: python/integration-report-history.json
key: integration-report-history-integration-${{ github.run_id }}
- name: Upload unified trend report
if: always()
uses: actions/upload-artifact@v7
with:
name: flaky-test-report
name: integration-test-report
path: |
python/flaky-test-report.md
python/flaky-report-history.json
python/integration-test-report.md
python/integration-report-history.json
python-integration-tests-check:
if: always()
+57 -18
View File
@@ -278,6 +278,8 @@ jobs:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
ANTHROPIC_CHAT_MODEL: ${{ vars.ANTHROPIC_CHAT_MODEL_ID }}
LOCAL_MCP_URL: ${{ vars.LOCAL_MCP__URL }}
OLLAMA_MODEL: qwen2.5:1.5b
OLLAMA_EMBEDDING_MODEL: nomic-embed-text
defaults:
run:
working-directory: python
@@ -289,6 +291,43 @@ jobs:
with:
python-version: ${{ env.UV_PYTHON }}
os: ${{ runner.os }}
- name: Install Ollama
run: curl -fsSL https://ollama.com/install.sh | sh
working-directory: .
- name: Cache Ollama models
uses: actions/cache@v4
with:
path: ~/.ollama/models
key: ollama-models-qwen2.5-1.5b-nomic-embed-text-v1
- name: Start Ollama and pull models
run: |
# Stop any Ollama instance auto-started by the install script
pkill ollama || true
sleep 2
ollama serve &
for i in $(seq 1 30); do
if curl -sf http://localhost:11434/api/tags > /dev/null 2>&1; then
break
fi
sleep 1
done
# Pull models with retry for transient 429 rate limits
for model in qwen2.5:1.5b nomic-embed-text; do
pulled=false
for attempt in 1 2 3; do
if ollama pull "$model"; then
pulled=true
break
fi
echo "Retry $attempt for $model (waiting 15s)..."
sleep 15
done
if [ "$pulled" != "true" ]; then
echo "ERROR: Failed to pull $model after 3 attempts"
exit 1
fi
done
working-directory: .
- name: Start local MCP server
id: local-mcp
uses: ./.github/actions/setup-local-mcp-server
@@ -403,7 +442,7 @@ jobs:
-m integration
-n logical --dist worksteal
-x
--timeout=360 --session-timeout=900 --timeout_method thread
--timeout=480 --session-timeout=900 --timeout_method thread
--retries 2 --retry-delay 5
--junitxml=pytest.xml
working-directory: ./python
@@ -619,9 +658,9 @@ jobs:
path: ./python/pytest.xml
if-no-files-found: ignore
# Flaky test trend report (aggregates per-job JUnit XML results)
python-flaky-test-report:
name: Flaky Test Report
# Integration test trend report (aggregates per-job JUnit XML results)
python-integration-test-report:
name: Integration Test Report
if: >
always() &&
(contains(join(needs.*.result, ','), 'success') ||
@@ -652,36 +691,36 @@ jobs:
with:
pattern: test-results-*
path: test-results/
- name: Restore flaky report history cache
- name: Restore report history cache
uses: actions/cache/restore@v4
with:
path: python/flaky-report-history.json
key: flaky-report-history-merge-${{ github.run_id }}
path: python/integration-report-history.json
key: integration-report-history-merge-${{ github.run_id }}
restore-keys: |
flaky-report-history-merge-
integration-report-history-merge-
- name: Generate trend report
run: >
uv run python scripts/flaky_report/aggregate.py
uv run python scripts/integration_test_report/aggregate.py
../test-results/
flaky-report-history.json
flaky-test-report.md
integration-report-history.json
integration-test-report.md
- name: Post to Job Summary
if: always()
run: cat flaky-test-report.md >> $GITHUB_STEP_SUMMARY
- name: Save flaky report history cache
run: cat integration-test-report.md >> $GITHUB_STEP_SUMMARY
- name: Save report history cache
if: always()
uses: actions/cache/save@v4
with:
path: python/flaky-report-history.json
key: flaky-report-history-merge-${{ github.run_id }}
path: python/integration-report-history.json
key: integration-report-history-merge-${{ github.run_id }}
- name: Upload unified trend report
if: always()
uses: actions/upload-artifact@v7
with:
name: flaky-test-report
name: integration-test-report
path: |
python/flaky-test-report.md
python/flaky-report-history.json
python/integration-test-report.md
python/integration-report-history.json
python-integration-tests-check:
if: always()
@@ -297,6 +297,7 @@ public class AgentFrameworkResponseHandler : ResponseHandler
var agent = this._serviceProvider.GetKeyedService<AIAgent>(agentName);
if (agent is not null)
{
FoundryHostingExtensions.TryApplyUserAgent(agent);
return FoundryHostingExtensions.ApplyOpenTelemetry(agent);
}
@@ -310,12 +311,13 @@ public class AgentFrameworkResponseHandler : ResponseHandler
var defaultAgent = this._serviceProvider.GetService<AIAgent>();
if (defaultAgent is not null)
{
FoundryHostingExtensions.TryApplyUserAgent(defaultAgent);
return FoundryHostingExtensions.ApplyOpenTelemetry(defaultAgent);
}
var errorMessage = string.IsNullOrEmpty(agentName)
? "No agent name specified in the request (via agent.name or metadata[\"entity_id\"]) and no default AIAgent is registered."
: $"Agent '{agentName}' not found. Ensure it is registered via AddAIAgent(\"{agentName}\", ...) or as a default AIAgent.";
: $"Agent '{agentName}' not found. Ensure it is registered via AddFoundryResponses(services, agent) or services.AddKeyedSingleton<AIAgent>(\"{agentName}\", ...).";
throw new InvalidOperationException(errorMessage);
}
@@ -352,7 +354,7 @@ public class AgentFrameworkResponseHandler : ResponseHandler
var errorMessage = string.IsNullOrEmpty(agentName)
? "No agent name specified in the request (via agent.name or metadata[\"entity_id\"]) and no default AgentSessionStore is registered."
: $"Agent '{agentName}' not found. Ensure it is registered via AddAIAgent(\"{agentName}\", ...) or as a default AgentSessionStore.";
: $"AgentSessionStore for agent '{agentName}' not found. Ensure it is registered via AddFoundryResponses(services, agent, agentSessionStore) or services.AddKeyedSingleton<AgentSessionStore>(\"{agentName}\", ...).";
throw new InvalidOperationException(errorMessage);
}
@@ -0,0 +1,113 @@
// Copyright (c) Microsoft. All rights reserved.
using System;
using System.ClientModel;
using System.ClientModel.Primitives;
using System.Collections.Generic;
using System.Threading.Tasks;
using OpenAI;
using OpenAI.Responses;
#pragma warning disable OPENAI001, SCME0001
namespace Microsoft.Agents.AI.Foundry.Hosting;
/// <summary>
/// A <see cref="ResponsesClient"/> subclass that delegates every protocol-level request to a
/// wrapped <see cref="ResponsesClient"/>. Before each call, a
/// <see cref="HostedAgentUserAgentPolicy"/> is added to the per-call
/// <see cref="RequestOptions"/> so the wrapped client's pipeline appends the hosted-agent
/// <c>User-Agent</c> segment on the wire.
/// </summary>
/// <remarks>
/// <para>
/// The streaming overloads MEAI binds via reflection (<c>internal CreateResponseStreamingAsync(CreateResponseOptions, RequestOptions)</c>
/// and <c>internal GetResponseStreamingAsync(GetResponseOptions, RequestOptions)</c>) bottom out
/// in calls to the public-virtual non-streaming protocol overloads on <see langword="this"/>. Overriding those
/// non-streaming overloads is therefore sufficient to intercept both streaming and non-streaming traffic.
/// </para>
/// <para>
/// The base pipeline supplied to <see cref="ResponsesClient(ClientPipeline, OpenAIClientOptions)"/>
/// is a dummy pipeline whose terminal transport throws if invoked. Every override on this class
/// delegates to the inner client BEFORE any code path reaches <see cref="ResponsesClient.Pipeline"/>, so the dummy is
/// never expected to run; the throwing transport surfaces any unexpected escape route loudly.
/// </para>
/// </remarks>
internal sealed class DelegatingResponsesClient : ResponsesClient
{
private readonly ResponsesClient _inner;
public DelegatingResponsesClient(ResponsesClient inner)
: base(BuildDummyPipeline(), new OpenAIClientOptions { Endpoint = inner?.Endpoint })
{
this._inner = inner ?? throw new ArgumentNullException(nameof(inner));
}
public override async Task<ClientResult> CreateResponseAsync(BinaryContent content, RequestOptions? options = null)
=> await this._inner.CreateResponseAsync(content, AddUserAgentPolicy(options)).ConfigureAwait(false);
public override ClientResult CreateResponse(BinaryContent content, RequestOptions? options = null)
=> this._inner.CreateResponse(content, AddUserAgentPolicy(options));
public override async Task<ClientResult> GetResponseAsync(string responseId, IEnumerable<IncludedResponseProperty>? include, bool? stream, int? startingAfter, bool? includeObfuscation, RequestOptions options)
=> await this._inner.GetResponseAsync(responseId, include, stream, startingAfter, includeObfuscation, AddUserAgentPolicy(options)).ConfigureAwait(false);
public override ClientResult GetResponse(string responseId, IEnumerable<IncludedResponseProperty>? include, bool? stream, int? startingAfter, bool? includeObfuscation, RequestOptions options)
=> this._inner.GetResponse(responseId, include, stream, startingAfter, includeObfuscation, AddUserAgentPolicy(options));
public override async Task<ClientResult> DeleteResponseAsync(string responseId, RequestOptions options)
=> await this._inner.DeleteResponseAsync(responseId, AddUserAgentPolicy(options)).ConfigureAwait(false);
public override ClientResult DeleteResponse(string responseId, RequestOptions options)
=> this._inner.DeleteResponse(responseId, AddUserAgentPolicy(options));
public override async Task<ClientResult> CancelResponseAsync(string responseId, RequestOptions options)
=> await this._inner.CancelResponseAsync(responseId, AddUserAgentPolicy(options)).ConfigureAwait(false);
public override ClientResult CancelResponse(string responseId, RequestOptions options)
=> this._inner.CancelResponse(responseId, AddUserAgentPolicy(options));
public override async Task<ClientResult> GetInputTokenCountAsync(string contentType, BinaryContent content, RequestOptions? options = null)
=> await this._inner.GetInputTokenCountAsync(contentType, content, AddUserAgentPolicy(options)).ConfigureAwait(false);
public override ClientResult GetInputTokenCount(string contentType, BinaryContent content, RequestOptions? options = null)
=> this._inner.GetInputTokenCount(contentType, content, AddUserAgentPolicy(options));
public override async Task<ClientResult> CompactResponseAsync(string contentType, BinaryContent content, RequestOptions? options = null)
=> await this._inner.CompactResponseAsync(contentType, content, AddUserAgentPolicy(options)).ConfigureAwait(false);
public override ClientResult CompactResponse(string contentType, BinaryContent content, RequestOptions? options = null)
=> this._inner.CompactResponse(contentType, content, AddUserAgentPolicy(options));
public override async Task<ClientResult> GetResponseInputItemCollectionPageAsync(string responseId, int? limit, string order, string after, string before, RequestOptions options)
=> await this._inner.GetResponseInputItemCollectionPageAsync(responseId, limit, order, after, before, AddUserAgentPolicy(options)).ConfigureAwait(false);
public override ClientResult GetResponseInputItemCollectionPage(string responseId, int? limit, string order, string after, string before, RequestOptions options)
=> this._inner.GetResponseInputItemCollectionPage(responseId, limit, order, after, before, AddUserAgentPolicy(options));
private static RequestOptions AddUserAgentPolicy(RequestOptions? options)
{
options ??= new RequestOptions();
options.AddPolicy(HostedAgentUserAgentPolicy.Instance, PipelinePosition.PerCall);
return options;
}
private static ClientPipeline BuildDummyPipeline()
{
var options = new ClientPipelineOptions
{
Transport = new ThrowingTransport(),
};
return ClientPipeline.Create(options, default, default, default);
}
private sealed class ThrowingTransport : PipelineTransport
{
private const string Message =
"DelegatingResponsesClient transport invoked bypassed the override-and-delegate design. This exception should be unreachable and should never be thrown following the correct usage of DelegatingResponsesClient.";
protected override PipelineMessage CreateMessageCore() => throw new InvalidOperationException(Message);
protected override void ProcessCore(PipelineMessage message) => throw new InvalidOperationException(Message);
protected override ValueTask ProcessCoreAsync(PipelineMessage message) => throw new InvalidOperationException(Message);
}
}
@@ -0,0 +1,84 @@
// Copyright (c) Microsoft. All rights reserved.
using System.ClientModel.Primitives;
using System.Collections.Generic;
using System.Reflection;
using System.Threading.Tasks;
namespace Microsoft.Agents.AI.Foundry.Hosting;
/// <summary>
/// Pipeline policy that appends the hosted-agent <c>User-Agent</c> segment
/// (e.g. <c>"foundry-hosting/agent-framework-dotnet/{version}"</c>) to outgoing requests.
/// </summary>
/// <remarks>
/// <para>
/// The supplement value is computed once from the Microsoft.Agents.AI.Foundry.Hosting
/// assembly's informational version. The policy is idempotent on retries: if the segment
/// is already present in the <c>User-Agent</c> header, the policy does not append it again.
/// </para>
/// <para>
/// This policy is added at request time (per-call <see cref="PipelinePosition"/>)
/// by <see cref="DelegatingResponsesClient"/> when invoking the wrapped
/// <see cref="OpenAI.Responses.ResponsesClient"/>. It is only registered when an agent is
/// resolved by the Foundry hosting layer.
/// </para>
/// </remarks>
internal sealed class HostedAgentUserAgentPolicy : PipelinePolicy
{
public static HostedAgentUserAgentPolicy Instance { get; } = new HostedAgentUserAgentPolicy();
private static readonly string s_supplementValue = CreateSupplementValue();
public override void Process(PipelineMessage message, IReadOnlyList<PipelinePolicy> pipeline, int currentIndex)
{
AppendHeader(message);
ProcessNext(message, pipeline, currentIndex);
}
public override async ValueTask ProcessAsync(PipelineMessage message, IReadOnlyList<PipelinePolicy> pipeline, int currentIndex)
{
AppendHeader(message);
await ProcessNextAsync(message, pipeline, currentIndex).ConfigureAwait(false);
}
private static void AppendHeader(PipelineMessage message)
{
if (message.Request.Headers.TryGetValue("User-Agent", out var existing) && !string.IsNullOrEmpty(existing))
{
// Guard against double-append on retries or when the policy
// is registered on multiple pipeline positions.
if (existing.Contains(s_supplementValue))
{
return;
}
message.Request.Headers.Set("User-Agent", $"{existing} {s_supplementValue}");
}
else
{
message.Request.Headers.Set("User-Agent", s_supplementValue);
}
}
private static string CreateSupplementValue()
{
const string Name = "foundry-hosting/agent-framework-dotnet";
if (typeof(HostedAgentUserAgentPolicy).Assembly.GetCustomAttribute<AssemblyInformationalVersionAttribute>()?.InformationalVersion is string version)
{
int pos = version.IndexOf('+');
if (pos >= 0)
{
version = version.Substring(0, pos);
}
if (version.Length > 0)
{
return $"{Name}/{version}";
}
}
return Name;
}
}
@@ -3,16 +3,15 @@
using System;
using System.Diagnostics.CodeAnalysis;
using System.Reflection;
using System.Threading.Tasks;
using Azure.AI.AgentServer.Responses;
using Azure.Core;
using Azure.Identity;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Routing;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.DependencyInjection.Extensions;
using Microsoft.Shared.DiagnosticIds;
using OpenAI.Responses;
namespace Microsoft.Agents.AI.Foundry.Hosting;
@@ -36,7 +35,7 @@ public static class FoundryHostingExtensions
/// <para>
/// Example:
/// <code>
/// builder.AddAIAgent("my-agent", ...);
/// builder.Services.AddKeyedSingleton&lt;AIAgent&gt;("my-agent", myAgent);
/// builder.Services.AddFoundryResponses();
///
/// var app = builder.Build();
@@ -181,13 +180,6 @@ public static class FoundryHostingExtensions
{
ArgumentNullException.ThrowIfNull(endpoints);
endpoints.MapResponsesServer(prefix);
if (endpoints is IApplicationBuilder app)
{
// Ensure the middleware is added to the pipeline
app.UseMiddleware<AgentFrameworkUserAgentMiddleware>();
}
return endpoints;
}
@@ -216,46 +208,85 @@ public static class FoundryHostingExtensions
.Build();
}
private sealed class AgentFrameworkUserAgentMiddleware(RequestDelegate next)
/// <summary>
/// Attempts to wrap the agent's underlying <see cref="ResponsesClient"/>
/// with a <see cref="DelegatingResponsesClient"/> so every outgoing Responses-API request
/// carries the hosted-agent <c>User-Agent</c> segment.
/// </summary>
/// <remarks>
/// <para>
/// Best-effort and idempotent. The method is a no-op when:
/// <list type="bullet">
/// <item><description><paramref name="agent"/> exposes no <see cref="IChatClient"/>;</description></item>
/// <item><description>the chat client is not backed by MEAI's internal <c>OpenAIResponsesChatClient</c> (e.g., a non-OpenAI provider or a custom impl);</description></item>
/// <item><description>the inner <see cref="ResponsesClient"/> is already a <see cref="DelegatingResponsesClient"/>.</description></item>
/// </list>
/// </para>
/// <para>
/// Works for any <see cref="ResponsesClient"/>-derived inner client — both the Foundry-specific
/// <see cref="Azure.AI.Extensions.OpenAI.ProjectResponsesClient"/> and the native OpenAI
/// <see cref="ResponsesClient"/> obtained from <see cref="OpenAI.OpenAIClient"/>. The wrapper preserves
/// the inner client's pipeline (Transport, RetryPolicy, NetworkTimeout, OrganizationId / ProjectId /
/// UserAgentApplicationId, custom policies) because every override delegates to the inner instance.
/// </para>
/// <para>
/// Returns the same <paramref name="agent"/> instance unchanged. Mutation happens via
/// reflection on MEAI's private <c>_responseClient</c> field; the agent itself is not wrapped.
/// </para>
/// </remarks>
internal static AIAgent TryApplyUserAgent(AIAgent agent)
{
private static readonly string s_userAgentValue = CreateUserAgentValue();
public async Task InvokeAsync(HttpContext context)
var chatClient = agent.GetService<IChatClient>();
if (chatClient is null)
{
var headers = context.Request.Headers;
var userAgent = headers.UserAgent.ToString();
if (string.IsNullOrEmpty(userAgent))
{
headers.UserAgent = s_userAgentValue;
}
else if (!userAgent.Contains(s_userAgentValue, StringComparison.OrdinalIgnoreCase))
{
headers.UserAgent = $"{userAgent} {s_userAgentValue}";
}
await next(context).ConfigureAwait(false);
return agent;
}
private static string CreateUserAgentValue()
var meaiType = s_meaiResponsesChatClientType;
if (meaiType is null)
{
const string Name = "agent-framework-dotnet";
if (typeof(AgentFrameworkUserAgentMiddleware).Assembly.GetCustomAttribute<AssemblyInformationalVersionAttribute>()?.InformationalVersion is string version)
{
int pos = version.IndexOf('+');
if (pos >= 0)
{
version = version.Substring(0, pos);
}
if (version.Length > 0)
{
return $"{Name}/{version}";
}
}
return Name;
return agent;
}
var meaiInstance = chatClient.GetService(meaiType);
if (meaiInstance is null)
{
return agent;
}
var field = s_meaiResponseClientField;
if (field is null)
{
return agent;
}
var current = field.GetValue(meaiInstance) as ResponsesClient;
if (current is null or DelegatingResponsesClient)
{
return agent;
}
field.SetValue(meaiInstance, new DelegatingResponsesClient(current));
return agent;
}
/// <summary>
/// MEAI's internal <c>OpenAIResponsesChatClient</c> type, resolved once via reflection.
/// <see langword="null"/> if the type cannot be found (e.g., MEAI version drift).
/// </summary>
[UnconditionalSuppressMessage("Trimming", "IL2026:RequiresUnreferencedCode",
Justification = "MEAI's OpenAIResponsesChatClient is referenced through MicrosoftExtensionsAIResponsesExtensions and survives trimming.")]
[UnconditionalSuppressMessage("Trimming", "IL2073:RequiresUnreferencedCode",
Justification = "MEAI's OpenAIResponsesChatClient is referenced through MicrosoftExtensionsAIResponsesExtensions and survives trimming.")]
private static readonly Type? s_meaiResponsesChatClientType =
typeof(MicrosoftExtensionsAIResponsesExtensions).Assembly.GetType("Microsoft.Extensions.AI.OpenAIResponsesChatClient");
/// <summary>
/// MEAI's internal <c>_responseClient</c> field on <c>OpenAIResponsesChatClient</c>,
/// resolved once via reflection. <see langword="null"/> if the field cannot be found.
/// </summary>
[UnconditionalSuppressMessage("Trimming", "IL2080:RequiresDynamicallyAccessedMembers",
Justification = "OpenAIResponsesChatClient and its private fields are preserved by the polyfill design; MEAI does the same reflection internally.")]
private static readonly FieldInfo? s_meaiResponseClientField =
s_meaiResponsesChatClientType?.GetField("_responseClient", BindingFlags.NonPublic | BindingFlags.Instance);
}
@@ -3,7 +3,6 @@
using System.ClientModel.Primitives;
using System.Collections.Generic;
using System.Reflection;
using System.Threading;
using System.Threading.Tasks;
namespace Microsoft.Agents.AI;
@@ -13,20 +12,6 @@ internal static class RequestOptionsExtensions
/// <summary>Gets the singleton <see cref="PipelinePolicy"/> that adds a MEAI user-agent header.</summary>
internal static PipelinePolicy UserAgentPolicy => MeaiUserAgentPolicy.Instance;
/// <summary>Creates a <see cref="RequestOptions"/> configured for use with Foundry Agents.</summary>
public static RequestOptions ToRequestOptions(this CancellationToken cancellationToken, bool streaming)
{
RequestOptions requestOptions = new()
{
CancellationToken = cancellationToken,
BufferResponse = !streaming
};
requestOptions.AddPolicy(MeaiUserAgentPolicy.Instance, PipelinePosition.PerCall);
return requestOptions;
}
/// <summary>Provides a pipeline policy that adds a "MEAI/x.y.z" user-agent header.</summary>
private sealed class MeaiUserAgentPolicy : PipelinePolicy
{
@@ -0,0 +1,453 @@
// Copyright (c) Microsoft. All rights reserved.
using System;
using System.ClientModel;
using System.ClientModel.Primitives;
using System.Collections.Generic;
using System.Net;
using System.Net.Http;
using System.Reflection;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using Azure.AI.Extensions.OpenAI;
using Microsoft.Agents.AI.Foundry.Hosting;
using Microsoft.Extensions.AI;
using OpenAI;
using OpenAI.Responses;
#pragma warning disable OPENAI001, SCME0001, SCME0002, MEAI001
namespace Microsoft.Agents.AI.Foundry.UnitTests.Hosting;
/// <summary>
/// Verifies that <see cref="DelegatingResponsesClient"/> preserves user-supplied client options
/// (Transport, RetryPolicy, UserAgentApplicationId, OrganizationId, ProjectId) and adds the
/// hosted-agent User-Agent supplement on every outgoing request, including streaming.
/// Covers both the Azure-flavored <see cref="ProjectResponsesClient"/> and the native OpenAI
/// <see cref="ResponsesClient"/>.
/// </summary>
public sealed partial class DelegatingResponsesClientTests
{
private const string TestEndpoint = "https://fake-foundry.example.com/api/projects/fake-prj";
private const string OpenAIEndpoint = "https://fake-openai.example.com/v1";
private const string Deployment = "fake-deployment";
[System.Text.RegularExpressions.GeneratedRegex("foundry-hosting/agent-framework-dotnet")]
private static partial System.Text.RegularExpressions.Regex SupplementRegex();
[Fact]
public async Task Polyfill_NonStreaming_PreservesAppId_ThroughCustomTransport_AddsSupplementAsync()
{
// Arrange
using var handler = new RecordingHandler(MinimalResponseJson());
#pragma warning disable CA5399
using var httpClient = new HttpClient(handler);
#pragma warning restore CA5399
var inner = BuildInner(httpClient, userAgentApplicationId: "MY_APP_ID");
var chat = MakeWithDelegating(inner);
// Act
_ = await chat.GetResponseAsync("hello");
// Assert
var req = Assert.Single(handler.Requests);
Assert.Contains("MY_APP_ID", req.UserAgent);
Assert.Contains("MEAI/", req.UserAgent);
Assert.Contains("foundry-hosting/agent-framework-dotnet", req.UserAgent);
Assert.StartsWith(TestEndpoint, req.Uri);
}
[Fact]
public async Task Polyfill_Streaming_PreservesAppId_ThroughCustomTransport_AddsSupplementAsync()
{
// Arrange
using var handler = new RecordingHandler(MinimalSseResponse());
#pragma warning disable CA5399
using var httpClient = new HttpClient(handler);
#pragma warning restore CA5399
var inner = BuildInner(httpClient, userAgentApplicationId: "MY_APP_ID");
var chat = MakeWithDelegating(inner);
// Act
await foreach (var _ in chat.GetStreamingResponseAsync("hello"))
{
}
// Assert
var req = Assert.Single(handler.Requests);
Assert.Contains("MY_APP_ID", req.UserAgent);
Assert.Contains("MEAI/", req.UserAgent);
Assert.Contains("foundry-hosting/agent-framework-dotnet", req.UserAgent);
Assert.StartsWith(TestEndpoint, req.Uri);
}
[Fact]
public async Task Polyfill_PreservesOrganizationAndProjectHeadersAsync()
{
// Arrange
using var handler = new RecordingHandler(MinimalResponseJson());
#pragma warning disable CA5399
using var httpClient = new HttpClient(handler);
#pragma warning restore CA5399
var inner = BuildInner(httpClient,
userAgentApplicationId: "MY_APP_ID",
organizationId: "org_xyz",
projectId: "proj_abc");
var chat = MakeWithDelegating(inner);
// Act
_ = await chat.GetResponseAsync("hello");
// Assert
var req = Assert.Single(handler.Requests);
Assert.Contains("MY_APP_ID", req.UserAgent);
Assert.Contains("foundry-hosting/agent-framework-dotnet", req.UserAgent);
}
[Fact]
public async Task Polyfill_HonorsUserSuppliedRetryPolicy_ByCountingRetriesAsync()
{
// Arrange
var retryPolicy = new CountingRetryPolicy(extraAttempts: 2);
using var handler = new RecordingHandler(MinimalResponseJson());
#pragma warning disable CA5399
using var httpClient = new HttpClient(handler);
#pragma warning restore CA5399
var inner = BuildInner(httpClient, userAgentApplicationId: "MY_APP_ID", retryPolicy: retryPolicy);
var chat = MakeWithDelegating(inner);
// Act
_ = await chat.GetResponseAsync("hello");
// Assert: retry policy ran (1 + 2 extras = 3 attempts).
Assert.Equal(3, handler.Requests.Count);
Assert.Equal(3, retryPolicy.InvocationCount);
foreach (var req in handler.Requests)
{
Assert.Contains("MY_APP_ID", req.UserAgent);
Assert.Contains("MEAI/", req.UserAgent);
Assert.Contains("foundry-hosting/agent-framework-dotnet", req.UserAgent);
}
}
[Fact]
public async Task Baseline_NonStreaming_DoesNotInjectSupplementAsync()
{
// Arrange
using var handler = new RecordingHandler(MinimalResponseJson());
#pragma warning disable CA5399
using var httpClient = new HttpClient(handler);
#pragma warning restore CA5399
var inner = BuildInner(httpClient, userAgentApplicationId: "MY_APP_ID");
var chat = inner.AsIChatClient(Deployment);
// Act
_ = await chat.GetResponseAsync("hello");
// Assert
var req = Assert.Single(handler.Requests);
Assert.Contains("MY_APP_ID", req.UserAgent);
Assert.Contains("MEAI/", req.UserAgent);
Assert.DoesNotContain("foundry-hosting/agent-framework-dotnet", req.UserAgent);
}
[Fact]
public async Task Polyfill_NativeOpenAIResponsesClient_NonStreaming_AddsSupplementAsync()
{
// Arrange: use the NATIVE OpenAI SDK ResponsesClient (no Foundry / Azure project involved).
using var handler = new RecordingHandler(MinimalResponseJson());
#pragma warning disable CA5399
using var httpClient = new HttpClient(handler);
#pragma warning restore CA5399
var inner = BuildOpenAIInner(httpClient, userAgentApplicationId: "MY_APP_ID");
var chat = MakeWithDelegating(inner);
// Act
_ = await chat.GetResponseAsync("hello");
// Assert
var req = Assert.Single(handler.Requests);
Assert.Contains("MY_APP_ID", req.UserAgent);
Assert.Contains("MEAI/", req.UserAgent);
Assert.Contains("foundry-hosting/agent-framework-dotnet", req.UserAgent);
Assert.StartsWith(OpenAIEndpoint, req.Uri);
}
[Fact]
public async Task Polyfill_NativeOpenAIResponsesClient_Streaming_AddsSupplementAsync()
{
// Arrange
using var handler = new RecordingHandler(MinimalSseResponse());
#pragma warning disable CA5399
using var httpClient = new HttpClient(handler);
#pragma warning restore CA5399
var inner = BuildOpenAIInner(httpClient, userAgentApplicationId: "MY_APP_ID");
var chat = MakeWithDelegating(inner);
// Act
await foreach (var _ in chat.GetStreamingResponseAsync("hello"))
{
}
// Assert
var req = Assert.Single(handler.Requests);
Assert.Contains("MY_APP_ID", req.UserAgent);
Assert.Contains("MEAI/", req.UserAgent);
Assert.Contains("foundry-hosting/agent-framework-dotnet", req.UserAgent);
Assert.StartsWith(OpenAIEndpoint, req.Uri);
}
[Theory]
[InlineData("DeleteResponseAsync")]
[InlineData("CancelResponseAsync")]
[InlineData("GetInputTokenCountAsync")]
[InlineData("CompactResponseAsync")]
[InlineData("GetResponseInputItemCollectionPageAsync")]
public async Task Polyfill_AncillaryProtocolMethod_AddsSupplementAsync(string method)
{
// Arrange: hit the wrapper DIRECTLY (no MEAI in the chain) to simulate user code that
// grabs the underlying ResponsesClient via chat.GetService<ResponsesClient>() and invokes
// a non-Create/Get protocol method. This is the regression path: without overriding these,
// the wrapper's dummy throwing pipeline would fire.
using var handler = new RecordingHandler(MinimalResponseJson());
#pragma warning disable CA5399
using var httpClient = new HttpClient(handler);
#pragma warning restore CA5399
var inner = BuildOpenAIInner(httpClient, userAgentApplicationId: "MY_APP_ID");
var wrapper = new DelegatingResponsesClient(inner);
// Act
switch (method)
{
case "DeleteResponseAsync":
_ = await wrapper.DeleteResponseAsync("resp_1", options: null!);
break;
case "CancelResponseAsync":
_ = await wrapper.CancelResponseAsync("resp_1", options: null!);
break;
case "GetInputTokenCountAsync":
_ = await wrapper.GetInputTokenCountAsync("application/json", BinaryContent.Create(BinaryData.FromString("{}")));
break;
case "CompactResponseAsync":
_ = await wrapper.CompactResponseAsync("application/json", BinaryContent.Create(BinaryData.FromString("{}")));
break;
case "GetResponseInputItemCollectionPageAsync":
_ = await wrapper.GetResponseInputItemCollectionPageAsync("resp_1", limit: null, order: "asc", after: "a", before: "b", options: null!);
break;
default:
Assert.Fail($"Unhandled method: {method}");
break;
}
// Assert
var req = Assert.Single(handler.Requests);
Assert.Contains("MY_APP_ID", req.UserAgent);
Assert.Contains("foundry-hosting/agent-framework-dotnet", req.UserAgent);
}
[Fact]
public async Task Polyfill_RetryWithinCall_DoesNotDuplicateSupplementInUserAgentAsync()
{
// Arrange: a custom retry policy that re-runs the inner pipeline on the SAME message,
// so the per-call HostedAgentUserAgentPolicy fires multiple times against the same headers.
// The policy's Contains-guard must prevent the supplement from appearing twice.
var retryPolicy = new CountingRetryPolicy(extraAttempts: 2);
using var handler = new RecordingHandler(MinimalResponseJson());
#pragma warning disable CA5399
using var httpClient = new HttpClient(handler);
#pragma warning restore CA5399
var inner = BuildInner(httpClient, userAgentApplicationId: "MY_APP_ID", retryPolicy: retryPolicy);
var chat = MakeWithDelegating(inner);
// Act
_ = await chat.GetResponseAsync("hello");
// Assert: each retry attempt must have exactly ONE foundry-hosting segment, never two.
Assert.Equal(3, handler.Requests.Count);
foreach (var req in handler.Requests)
{
int matches = SupplementRegex().Matches(req.UserAgent).Count;
Assert.True(matches == 1, $"Expected exactly one foundry-hosting segment per retry attempt, got {matches}. UA: {req.UserAgent}");
}
}
[Fact]
public async Task TryApplyUserAgent_CalledTwiceOnSameAgent_DoesNotDoubleWrapAsync()
{
// Arrange: build a real ChatClientAgent whose IChatClient resolves to MEAI's
// OpenAIResponsesChatClient → ProjectResponsesClient (with a fake transport).
using var handler = new RecordingHandler(MinimalResponseJson());
#pragma warning disable CA5399
using var httpClient = new HttpClient(handler);
#pragma warning restore CA5399
var inner = BuildInner(httpClient, userAgentApplicationId: "MY_APP_ID");
IChatClient chatClient = inner.AsIChatClient(Deployment);
AIAgent agent = new ChatClientAgent(chatClient);
// Act: apply twice.
FoundryHostingExtensions.TryApplyUserAgent(agent);
FoundryHostingExtensions.TryApplyUserAgent(agent);
// Assert: invoking the agent produces exactly ONE outbound request whose UA contains
// the supplement EXACTLY ONCE (would be twice if the wrapper were nested).
_ = await chatClient.GetResponseAsync("hello");
var req = Assert.Single(handler.Requests);
int matches = SupplementRegex().Matches(req.UserAgent).Count;
Assert.True(matches == 1, $"Expected exactly one foundry-hosting segment, got {matches}. UA: {req.UserAgent}");
}
[Fact]
public void OpenAIResponsesChatClient_ResponseClientField_ReflectionGuard()
{
// Guards the polyfill's reflection target. Failure here means MEAI internals
// changed and the polyfill needs updating.
var meaiType = typeof(MicrosoftExtensionsAIResponsesExtensions).Assembly
.GetType("Microsoft.Extensions.AI.OpenAIResponsesChatClient");
Assert.NotNull(meaiType);
var field = meaiType!.GetField("_responseClient", BindingFlags.NonPublic | BindingFlags.Instance);
Assert.NotNull(field);
Assert.True(typeof(ResponsesClient).IsAssignableFrom(field!.FieldType),
$"Expected _responseClient to be assignable to ResponsesClient but was {field.FieldType}.");
}
[Fact]
public void ResponsesClient_PipelineProperty_ReflectionGuard()
{
// The polyfill design assumes ResponsesClient.Pipeline remains accessible.
var pipelineProp = typeof(ResponsesClient).GetProperty("Pipeline", BindingFlags.Public | BindingFlags.Instance);
Assert.NotNull(pipelineProp);
Assert.Equal(typeof(ClientPipeline), pipelineProp!.PropertyType);
}
private static IChatClient MakeWithDelegating(ResponsesClient inner)
{
IChatClient meai = inner.AsIChatClient(Deployment);
var meaiType = meai.GetType();
var field = meaiType.GetField("_responseClient", BindingFlags.NonPublic | BindingFlags.Instance)!;
field.SetValue(meai, new DelegatingResponsesClient(inner));
return meai;
}
private static ProjectResponsesClient BuildInner(
HttpClient httpClient,
string? userAgentApplicationId = null,
string? organizationId = null,
string? projectId = null,
PipelinePolicy? retryPolicy = null)
{
var options = new ProjectResponsesClientOptions
{
Transport = new HttpClientPipelineTransport(httpClient),
};
if (userAgentApplicationId is not null)
{
options.UserAgentApplicationId = userAgentApplicationId;
}
if (organizationId is not null)
{
options.OrganizationId = organizationId;
}
if (projectId is not null)
{
options.ProjectId = projectId;
}
if (retryPolicy is not null)
{
options.RetryPolicy = retryPolicy;
}
return new ProjectResponsesClient(new Uri(TestEndpoint), new FakeAuthenticationTokenProvider(), options);
}
private static ResponsesClient BuildOpenAIInner(
HttpClient httpClient,
string? userAgentApplicationId = null)
{
var options = new OpenAIClientOptions
{
Transport = new HttpClientPipelineTransport(httpClient),
Endpoint = new Uri(OpenAIEndpoint),
};
if (userAgentApplicationId is not null)
{
options.UserAgentApplicationId = userAgentApplicationId;
}
return new ResponsesClient(new ApiKeyCredential("test-key"), options);
}
private static string MinimalResponseJson() => """
{
"id":"resp_1","object":"response","created_at":1700000000,"status":"completed",
"model":"fake","output":[],"usage":{"input_tokens":1,"output_tokens":1,"total_tokens":2}
}
""";
private static string MinimalSseResponse()
{
var sb = new StringBuilder();
sb.Append("event: response.completed\n");
sb.Append("data: ").Append("""{"type":"response.completed","response":{"id":"resp_1","object":"response","created_at":1700000000,"status":"completed","model":"fake","output":[],"usage":{"input_tokens":1,"output_tokens":1,"total_tokens":2}}}""").Append("\n\n");
sb.Append("data: [DONE]\n\n");
return sb.ToString();
}
private sealed class RecordingHandler : HttpClientHandler
{
private readonly string _body;
public List<RecordedRequest> Requests { get; } = [];
public RecordingHandler(string body)
{
this._body = body;
}
protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
string ua = request.Headers.TryGetValues("User-Agent", out var values)
? string.Join(",", values)
: "(none)";
this.Requests.Add(new RecordedRequest(request.Method.Method, request.RequestUri?.ToString() ?? "?", ua));
var resp = new HttpResponseMessage(HttpStatusCode.OK)
{
Content = new StringContent(this._body, Encoding.UTF8, "application/json"),
RequestMessage = request,
};
return Task.FromResult(resp);
}
}
private readonly record struct RecordedRequest(string Method, string Uri, string UserAgent);
private sealed class CountingRetryPolicy : PipelinePolicy
{
private readonly int _extraAttempts;
public int InvocationCount { get; private set; }
public CountingRetryPolicy(int extraAttempts)
{
this._extraAttempts = extraAttempts;
}
public override void Process(PipelineMessage message, IReadOnlyList<PipelinePolicy> pipeline, int currentIndex)
{
for (int i = 0; i <= this._extraAttempts; i++)
{
this.InvocationCount++;
ProcessNext(message, pipeline, currentIndex);
}
}
public override async ValueTask ProcessAsync(PipelineMessage message, IReadOnlyList<PipelinePolicy> pipeline, int currentIndex)
{
for (int i = 0; i <= this._extraAttempts; i++)
{
this.InvocationCount++;
await ProcessNextAsync(message, pipeline, currentIndex).ConfigureAwait(false);
}
}
}
}
@@ -0,0 +1,165 @@
// Copyright (c) Microsoft. All rights reserved.
using System;
using System.ClientModel.Primitives;
using System.Collections.Generic;
using System.Net;
using System.Net.Http;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using Azure.AI.Extensions.OpenAI;
using Microsoft.Agents.AI.Foundry.Hosting;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting.Server;
using Microsoft.AspNetCore.TestHost;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.DependencyInjection;
#pragma warning disable OPENAI001, SCME0001, SCME0002, MEAI001
namespace Microsoft.Agents.AI.Foundry.UnitTests.Hosting;
/// <summary>
/// End-to-end tests that exercise the FULL hosted ASP.NET Core pipeline:
/// inbound HTTP → MapFoundryResponses → AgentFrameworkResponseHandler → TryApplyUserAgent →
/// agent invocation → outbound HTTP from inside the hosted environment.
/// Verifies that the hosted-agent <c>User-Agent</c> supplement reaches the outbound wire,
/// not just the inbound request.
/// </summary>
public sealed class HostedOutboundUserAgentTests : IAsyncDisposable
{
private const string TestEndpoint = "https://fake-foundry.example.com/api/projects/fake-prj";
private const string Deployment = "fake-deployment";
private WebApplication? _app;
private HttpClient? _inboundClient;
private RecordingHandler? _outboundHandler;
public async ValueTask DisposeAsync()
{
this._inboundClient?.Dispose();
this._outboundHandler?.Dispose();
if (this._app is not null)
{
await this._app.DisposeAsync();
}
}
[Fact]
public async Task Hosted_InboundResponsesRequest_TriggersOutboundCall_WithFoundryHostingSupplementAsync()
{
// Arrange: spin up a real ASP.NET Core TestServer that hosts an AIAgent backed by MEAI's
// OpenAIResponsesChatClient → ProjectResponsesClient → fake HTTP transport. This is the
// exact production stack minus the network: the only thing not real is the wire transport.
await this.StartHostedServerAsync();
// Act: send an inbound /openai/v1/responses request as the Foundry runtime would.
using var inboundRequest = new HttpRequestMessage(HttpMethod.Post, "/responses")
{
Content = new StringContent(InboundResponsesRequestJson(), Encoding.UTF8, "application/json"),
};
using var inboundResponse = await this._inboundClient!.SendAsync(inboundRequest);
var inboundBody = await inboundResponse.Content.ReadAsStringAsync();
// Assert: at least one OUTBOUND request reached the fake transport, AND it carries the
// foundry-hosting/agent-framework-dotnet/{version} supplement on its User-Agent.
// (We don't care about the inbound response shape — only that the agent's call to MEAI
// triggered an outbound request whose UA reaches the sandbox boundary correctly.)
Assert.True(this._outboundHandler!.Requests.Count > 0,
$"Expected at least one outbound request. Inbound status: {(int)inboundResponse.StatusCode}, body: {inboundBody}");
var outbound = this._outboundHandler.Requests[0];
Assert.StartsWith(TestEndpoint, outbound.Uri);
Assert.Contains("MEAI/", outbound.UserAgent);
Assert.Contains("foundry-hosting/agent-framework-dotnet", outbound.UserAgent);
}
private async Task StartHostedServerAsync()
{
var builder = WebApplication.CreateBuilder();
builder.WebHost.UseTestServer();
// Build a real ChatClientAgent whose IChatClient is MEAI's OpenAIResponsesChatClient
// wrapping a ProjectResponsesClient backed by a fake HTTP handler. After AgentFrameworkResponseHandler
// resolves this agent, TryApplyUserAgent will swap the inner _responseClient with our wrapper.
this._outboundHandler = new RecordingHandler(MinimalResponseJson());
#pragma warning disable CA5399
var outboundHttpClient = new HttpClient(this._outboundHandler);
#pragma warning restore CA5399
var projectOptions = new ProjectResponsesClientOptions
{
Transport = new HttpClientPipelineTransport(outboundHttpClient),
};
var projectResponsesClient = new ProjectResponsesClient(
new Uri(TestEndpoint),
new FakeAuthenticationTokenProvider(),
projectOptions);
IChatClient chatClient = projectResponsesClient.AsIChatClient(Deployment);
AIAgent agent = new ChatClientAgent(chatClient);
builder.Services.AddFoundryResponses(agent);
builder.Services.AddLogging();
this._app = builder.Build();
this._app.MapFoundryResponses();
await this._app.StartAsync();
var testServer = this._app.Services.GetRequiredService<IServer>() as TestServer
?? throw new InvalidOperationException("TestServer not found");
this._inboundClient = testServer.CreateClient();
}
private static string InboundResponsesRequestJson() => """
{
"model": "fake-deployment",
"input": [
{
"type": "message",
"id": "msg_1",
"status": "completed",
"role": "user",
"content": [{ "type": "input_text", "text": "Hello" }]
}
]
}
""";
private static string MinimalResponseJson() => """
{
"id":"resp_1","object":"response","created_at":1700000000,"status":"completed",
"model":"fake","output":[],"usage":{"input_tokens":1,"output_tokens":1,"total_tokens":2}
}
""";
private sealed class RecordingHandler : HttpClientHandler
{
private readonly string _body;
public List<RecordedRequest> Requests { get; } = [];
public RecordingHandler(string body)
{
this._body = body;
}
protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
string ua = request.Headers.TryGetValues("User-Agent", out var values)
? string.Join(",", values)
: "(none)";
this.Requests.Add(new RecordedRequest(request.RequestUri?.ToString() ?? "?", ua));
var resp = new HttpResponseMessage(HttpStatusCode.OK)
{
Content = new StringContent(this._body, Encoding.UTF8, "application/json"),
RequestMessage = request,
};
return Task.FromResult(resp);
}
}
private readonly record struct RecordedRequest(string Uri, string UserAgent);
}
@@ -4,8 +4,10 @@ using System;
using System.Linq;
using Azure.AI.AgentServer.Responses;
using Microsoft.Agents.AI.Foundry.Hosting;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.DependencyInjection;
using Moq;
using OpenAI.Responses;
namespace Microsoft.Agents.AI.Foundry.UnitTests.Hosting;
@@ -93,4 +95,45 @@ public class ServiceCollectionExtensionsTests
Assert.Same(instrumented, result);
}
[Fact]
public void TryApplyUserAgent_AgentWithoutChatClient_NoOp()
{
// Arrange: agent.GetService<IChatClient>() returns null.
var mockAgent = new Mock<AIAgent>();
// Act
var result = FoundryHostingExtensions.TryApplyUserAgent(mockAgent.Object);
// Assert
Assert.Same(mockAgent.Object, result);
}
[Fact]
public void TryApplyUserAgent_AgentWithNonMeaiChatClient_NoOp()
{
// Arrange: chat client that does not return MEAI's OpenAIResponsesChatClient via GetService.
var mockChatClient = new Mock<IChatClient>();
mockChatClient.Setup(c => c.GetService(It.IsAny<Type>(), It.IsAny<object?>())).Returns(null!);
var mockAgent = new Mock<AIAgent>();
mockAgent.Setup(a => a.GetService(typeof(IChatClient), It.IsAny<object?>())).Returns(mockChatClient.Object);
// Act
var result = FoundryHostingExtensions.TryApplyUserAgent(mockAgent.Object);
// Assert
Assert.Same(mockAgent.Object, result);
}
[Fact]
public void MeaiOpenAIResponsesChatClient_TypeFullName_ReflectionGuard()
{
// Guards the polyfill's reflection target type-name.
var meaiType = typeof(MicrosoftExtensionsAIResponsesExtensions).Assembly
.GetType("Microsoft.Extensions.AI.OpenAIResponsesChatClient");
Assert.NotNull(meaiType);
Assert.True(typeof(IChatClient).IsAssignableFrom(meaiType!),
$"Expected MEAI {meaiType!.FullName} to implement IChatClient.");
}
}
@@ -1,134 +0,0 @@
// Copyright (c) Microsoft. All rights reserved.
using System;
using System.Net.Http;
using System.Text.RegularExpressions;
using System.Threading.Tasks;
using Microsoft.Agents.AI.Foundry.Hosting;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting.Server;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.TestHost;
using Microsoft.Extensions.DependencyInjection;
using Moq;
namespace Microsoft.Agents.AI.Foundry.UnitTests.Hosting;
/// <summary>
/// Tests for the <c>AgentFrameworkUserAgentMiddleware</c> registered by
/// <see cref="FoundryHostingExtensions.MapFoundryResponses"/>.
/// </summary>
public sealed partial class UserAgentMiddlewareTests : IAsyncDisposable
{
private const string VersionedUserAgentPattern = @"agent-framework-dotnet/\d+\.\d+\.\d+(-[\w.]+)?";
private WebApplication? _app;
private HttpClient? _httpClient;
public async ValueTask DisposeAsync()
{
this._httpClient?.Dispose();
if (this._app != null)
{
await this._app.DisposeAsync();
}
}
[Fact]
public async Task MapFoundryResponses_NoUserAgentHeader_SetsAgentFrameworkUserAgentAsync()
{
// Arrange
await this.CreateTestServerAsync();
using var request = new HttpRequestMessage(HttpMethod.Get, "/test-ua");
// Act
var response = await this._httpClient!.SendAsync(request);
var userAgent = await response.Content.ReadAsStringAsync();
// Assert
Assert.Matches(VersionedUserAgentPattern, userAgent);
}
[Fact]
public async Task MapFoundryResponses_WithExistingUserAgent_AppendsAgentFrameworkUserAgentAsync()
{
// Arrange
await this.CreateTestServerAsync();
using var request = new HttpRequestMessage(HttpMethod.Get, "/test-ua");
request.Headers.TryAddWithoutValidation("User-Agent", "MyApp/1.0");
// Act
var response = await this._httpClient!.SendAsync(request);
var userAgent = await response.Content.ReadAsStringAsync();
// Assert
Assert.StartsWith("MyApp/1.0", userAgent);
Assert.Matches(VersionedUserAgentPattern, userAgent);
}
[Fact]
public async Task MapFoundryResponses_AlreadyContainsUserAgent_DoesNotDuplicateAsync()
{
// Arrange
await this.CreateTestServerAsync();
// First request to capture the actual middleware-generated value
using var firstRequest = new HttpRequestMessage(HttpMethod.Get, "/test-ua");
var firstResponse = await this._httpClient!.SendAsync(firstRequest);
var middlewareValue = await firstResponse.Content.ReadAsStringAsync();
// Act: send a second request that already contains the middleware value
using var secondRequest = new HttpRequestMessage(HttpMethod.Get, "/test-ua");
secondRequest.Headers.TryAddWithoutValidation("User-Agent", $"MyApp/2.0 {middlewareValue}");
var secondResponse = await this._httpClient!.SendAsync(secondRequest);
var userAgent = await secondResponse.Content.ReadAsStringAsync();
// Assert: should remain unchanged (no duplication)
Assert.Equal($"MyApp/2.0 {middlewareValue}", userAgent);
Assert.Single(VersionedUserAgentRegex().Matches(userAgent));
}
[Fact]
public async Task MapFoundryResponses_UserAgentValue_ContainsVersionAsync()
{
// Arrange
await this.CreateTestServerAsync();
using var request = new HttpRequestMessage(HttpMethod.Get, "/test-ua");
// Act
var response = await this._httpClient!.SendAsync(request);
var userAgent = await response.Content.ReadAsStringAsync();
// Assert: should match "agent-framework-dotnet/x.y.z" pattern
Assert.Matches(VersionedUserAgentPattern, userAgent);
}
private async Task CreateTestServerAsync()
{
var builder = WebApplication.CreateBuilder();
builder.WebHost.UseTestServer();
var mockAgent = new Mock<AIAgent>();
builder.Services.AddFoundryResponses(mockAgent.Object);
this._app = builder.Build();
this._app.MapFoundryResponses();
// Test endpoint that echoes the User-Agent header after middleware processing
this._app.MapGet("/test-ua", (HttpContext ctx) =>
Results.Text(ctx.Request.Headers.UserAgent.ToString()));
await this._app.StartAsync();
var testServer = this._app.Services.GetRequiredService<IServer>() as TestServer
?? throw new InvalidOperationException("TestServer not found");
this._httpClient = testServer.CreateClient();
}
[GeneratedRegex(VersionedUserAgentPattern)]
private static partial Regex VersionedUserAgentRegex();
}
@@ -0,0 +1,115 @@
// Copyright (c) Microsoft. All rights reserved.
using System.ClientModel.Primitives;
using System.Net;
using System.Net.Http;
using System.Reflection;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
namespace Microsoft.Agents.AI.Foundry.UnitTests;
/// <summary>
/// Verifies the per-call <c>MeaiUserAgentPolicy</c> exposed via
/// <see cref="RequestOptionsExtensions.UserAgentPolicy"/>. The policy is reachable through the
/// public <see cref="FoundryAgent"/> constructors (which add it to the internally-built
/// <see cref="Azure.AI.Projects.AIProjectClient"/>'s pipeline), so its behavior is part of the
/// public API surface.
/// </summary>
public sealed class RequestOptionsExtensionsTests
{
[Fact]
public async Task MeaiUserAgentPolicy_AddsMeaiSegment_ToOutgoingRequestAsync()
{
// Arrange
using var handler = new RecordingHandler();
#pragma warning disable CA5399
using var httpClient = new HttpClient(handler);
#pragma warning restore CA5399
var pipeline = ClientPipeline.Create(
new ClientPipelineOptions { Transport = new HttpClientPipelineTransport(httpClient) },
perCallPolicies: [RequestOptionsExtensions.UserAgentPolicy],
perTryPolicies: default,
beforeTransportPolicies: default);
// Act
var message = pipeline.CreateMessage();
message.Request.Method = "POST";
message.Request.Uri = new System.Uri("https://example.test/anything");
await pipeline.SendAsync(message);
// Assert
Assert.Equal(1, handler.Count);
Assert.NotNull(handler.LastUserAgent);
Assert.Contains("MEAI/", handler.LastUserAgent);
}
[Fact]
public async Task MeaiUserAgentPolicy_DoesNotAddFoundryHostingSegmentAsync()
{
// Arrange
using var handler = new RecordingHandler();
#pragma warning disable CA5399
using var httpClient = new HttpClient(handler);
#pragma warning restore CA5399
var pipeline = ClientPipeline.Create(
new ClientPipelineOptions { Transport = new HttpClientPipelineTransport(httpClient) },
perCallPolicies: [RequestOptionsExtensions.UserAgentPolicy],
perTryPolicies: default,
beforeTransportPolicies: default);
// Act
var message = pipeline.CreateMessage();
message.Request.Method = "POST";
message.Request.Uri = new System.Uri("https://example.test/anything");
await pipeline.SendAsync(message);
// Assert: the policy is MEAI-only; the foundry-hosting supplement is added elsewhere
// (by the polyfill DelegatingResponsesClient → HostedAgentUserAgentPolicy).
Assert.NotNull(handler.LastUserAgent);
Assert.DoesNotContain("foundry-hosting/agent-framework-dotnet", handler.LastUserAgent);
}
[Fact]
public void UserAgentPolicy_ExposesSingletonInstance()
{
// Two reads of the static property must return the same instance — the policy is stateless and shared.
var first = RequestOptionsExtensions.UserAgentPolicy;
var second = RequestOptionsExtensions.UserAgentPolicy;
Assert.Same(first, second);
}
[Fact]
public void MeaiUserAgentPolicy_ValueIncludesAFFoundryAssemblyVersion_ReflectionGuard()
{
// The policy emits "MEAI/{Microsoft.Agents.AI.Foundry assembly InformationalVersion}".
// If the assembly metadata stops being readable, the policy falls back to "MEAI" without a version,
// which is a measurable telemetry regression.
var attr = typeof(RequestOptionsExtensions).Assembly
.GetCustomAttribute<AssemblyInformationalVersionAttribute>();
Assert.NotNull(attr);
Assert.False(string.IsNullOrEmpty(attr!.InformationalVersion));
}
private sealed class RecordingHandler : HttpClientHandler
{
public int Count { get; private set; }
public string? LastUserAgent { get; private set; }
protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
this.Count++;
this.LastUserAgent = request.Headers.TryGetValues("User-Agent", out var values)
? string.Join(",", values)
: null;
var resp = new HttpResponseMessage(HttpStatusCode.OK)
{
Content = new StringContent("{}", Encoding.UTF8, "application/json"),
RequestMessage = request,
};
return Task.FromResult(resp);
}
}
}
@@ -26,7 +26,6 @@ pytestmark = [
pytest.mark.integration,
pytest.mark.sample("03_reliable_streaming"),
pytest.mark.usefixtures("function_app_for_test"),
pytest.mark.skip(reason="Temp disabled to fix test instability - needs investigation into root cause"),
]
@@ -56,12 +55,11 @@ class TestSampleReliableStreaming:
# Wait a moment for the agent to start writing to Redis
time.sleep(2)
# Stream response from Redis with shorter timeout
# Note: We use text/plain to avoid SSE parsing complexity
# Stream response from Redis with longer timeout to account for LLM latency
stream_response = requests.get(
f"{self.stream_url}/{thread_id}",
headers={"Accept": "text/plain"},
timeout=30, # Shorter timeout for test
timeout=60,
)
assert stream_response.status_code == 200
@@ -83,7 +81,7 @@ class TestSampleReliableStreaming:
stream_response = requests.get(
f"{self.stream_url}/{thread_id}",
headers={"Accept": "text/event-stream"},
timeout=30, # Shorter timeout
timeout=60,
)
assert stream_response.status_code == 200
content_type = stream_response.headers.get("content-type", "")
@@ -42,7 +42,7 @@ class TestWorkflowParallel:
self.base_url = base_url
self.helper = sample_helper
@pytest.mark.skip(reason="Causes timeouts.")
@pytest.mark.skip(reason="xdist distributes module tests across workers, each spawning a func process")
def test_parallel_workflow_document_analysis(self) -> None:
"""Test parallel workflow with a standard document."""
payload = {
@@ -71,7 +71,7 @@ class TestWorkflowParallel:
assert status["runtimeStatus"] == "Completed"
assert "output" in status
@pytest.mark.skip(reason="Causes timeouts.")
@pytest.mark.skip(reason="xdist distributes module tests across workers, each spawning a func process")
def test_parallel_workflow_short_document(self) -> None:
"""Test parallel workflow with a short document."""
payload = {
@@ -91,7 +91,7 @@ class TestWorkflowParallel:
assert status["runtimeStatus"] == "Completed"
assert "output" in status
@pytest.mark.skip(reason="Causes timeouts.")
@pytest.mark.skip(reason="xdist distributes module tests across workers, each spawning a func process")
def test_parallel_workflow_technical_document(self) -> None:
"""Test parallel workflow with a technical document."""
payload = {
@@ -115,7 +115,7 @@ class TestWorkflowParallel:
status = self.helper.wait_for_orchestration_with_output(data["statusQueryGetUri"], max_wait=300)
assert status["runtimeStatus"] == "Completed"
@pytest.mark.skip(reason="Causes timeouts.")
@pytest.mark.skip(reason="xdist distributes module tests across workers, each spawning a func process")
def test_workflow_status_endpoint(self) -> None:
"""Test that the workflow status endpoint works correctly."""
payload = {
@@ -52,7 +52,6 @@ class TestMultiAgentOrchestrationConditionals:
assert email_agent is not None
assert email_agent.name == EMAIL_AGENT_NAME
@pytest.mark.skip(reason="Consistently fails due to orchestration timeouts - needs investigation")
def test_conditional_branching(self):
"""Test that conditional branching works correctly."""
# Test with obvious spam
@@ -634,7 +634,6 @@ async def test_foundry_agent_configure_azure_monitor_import_error() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_foundry_agent_integration_tests_disabled
@pytest.mark.skip(reason="Test agent seems to have disappeared from the test environment; needs investigation.")
async def test_foundry_agent_basic_run() -> None:
"""Smoke-test FoundryAgent against a real configured agent."""
async with FoundryAgent(credential=AzureCliCredential(), allow_preview=True) as agent:
@@ -648,10 +647,11 @@ async def test_foundry_agent_basic_run() -> None:
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_foundry_agent_integration_tests_disabled
@pytest.mark.skip(reason="Test agent seems to have disappeared from the test environment; needs investigation.")
async def test_foundry_agent_custom_client_run() -> None:
"""Smoke-test FoundryAgent against a real configured agent."""
async with FoundryAgent(credential=AzureCliCredential(), client_type=RawFoundryAgentChatClient) as agent:
async with FoundryAgent(
credential=AzureCliCredential(), client_type=RawFoundryAgentChatClient, allow_preview=True
) as agent:
response = await agent.run("Please respond with exactly: 'This is a response test.'")
assert isinstance(response, AgentResponse)
@@ -559,25 +559,21 @@ class TestToolCalling:
class TestOptions:
"""Verify chat options are passed through to the model."""
@pytest.mark.skip(reason="Flaky in merge queue, blocking unrelated PRs. Tracked in #5553.")
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_foundry_hosting_integration_tests_disabled
async def test_temperature_and_max_tokens(self, server: ResponsesHostServer) -> None:
"""Set temperature and max_output_tokens and verify the response succeeds."""
"""Set max_output_tokens and verify the response succeeds."""
resp = await _post_json(
server,
{
"input": "Say hello briefly.",
"stream": False,
"max_output_tokens": 50,
"max_output_tokens": 200,
},
)
assert resp.status_code == 200
body = resp.json()
assert body["status"] == "completed"
output_messages = [o for o in body["output"] if o["type"] == "message"]
assert len(output_messages) == 1
output_text = output_messages[0]["content"][0]["text"]
assert len(output_text) > 0
assert len(body["output"]) > 0
@@ -150,6 +150,12 @@ def hello_world(arg1: str) -> str:
return "Hello World"
@tool(approval_mode="never_require")
def greet() -> str:
"""Say hello to the world. No-arg tool for integration tests to avoid argument parsing flakiness."""
return "Hello World"
def test_init(ollama_unit_test_env: dict[str, str]) -> None:
# Test successful initialization
ollama_chat_client = OllamaChatClient()
@@ -500,10 +506,10 @@ async def test_cmc_with_invalid_content_type(
async def test_cmc_integration_with_tool_call(
chat_history: list[Message],
) -> None:
chat_history.append(Message(contents=["Call the hello world function and repeat what it says"], role="user"))
chat_history.append(Message(contents=["Call the greet function and repeat what it says"], role="user"))
ollama_client = OllamaChatClient()
result = await ollama_client.get_response(messages=chat_history, options={"tools": [hello_world]})
result = await ollama_client.get_response(messages=chat_history, options={"tools": [greet]})
assert "hello" in result.text.lower() and "world" in result.text.lower()
assert result.messages[-2].contents[0].type == "function_result"
@@ -531,11 +537,11 @@ async def test_cmc_integration_with_chat_completion(
async def test_cmc_streaming_integration_with_tool_call(
chat_history: list[Message],
) -> None:
chat_history.append(Message(contents=["Call the hello world function and repeat what it says"], role="user"))
chat_history.append(Message(contents=["Call the greet function and repeat what it says"], role="user"))
ollama_client = OllamaChatClient()
result: AsyncIterable[ChatResponseUpdate] = ollama_client.get_response(
messages=chat_history, stream=True, options={"tools": [hello_world]}
messages=chat_history, stream=True, options={"tools": [greet]}
)
chunks: list[ChatResponseUpdate] = []
@@ -549,7 +555,7 @@ async def test_cmc_streaming_integration_with_tool_call(
assert tool_result.result == "Hello World"
if c.contents[0].type == "function_call":
tool_call = c.contents[0]
assert tool_call.name == "hello_world"
assert tool_call.name == "greet"
@pytest.mark.flaky
@@ -1,5 +1,6 @@
# Copyright (c) Microsoft. All rights reserved.
import asyncio
import base64
import inspect
import json
@@ -120,6 +121,15 @@ async def create_vector_store(
if result.last_error is not None:
raise Exception(f"Vector store file processing failed with status: {result.last_error.message}")
# Wait for the vector store index to be fully searchable.
# create_and_poll confirms file processing, but the search index is eventually consistent.
for _ in range(10):
vs = await client.client.vector_stores.retrieve(vector_store.id)
if vs.file_counts.completed >= 1 and vs.file_counts.in_progress == 0:
break
await asyncio.sleep(1)
await asyncio.sleep(2)
return file.id, Content.from_hosted_vector_store(vector_store_id=vector_store.id)
@@ -4385,10 +4395,6 @@ async def test_integration_web_search() -> None:
assert response.text is not None
@pytest.mark.skip(
reason="Unreliable due to OpenAI vector store indexing potential "
"race condition. See https://github.com/microsoft/agent-framework/issues/1669"
)
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_openai_integration_tests_disabled
@@ -4398,31 +4404,29 @@ async def test_integration_file_search() -> None:
assert isinstance(openai_responses_client, SupportsChatGetResponse)
file_id, vector_store = await create_vector_store(openai_responses_client)
# Use static method for file search tool
file_search_tool = OpenAIChatClient.get_file_search_tool(vector_store_ids=[vector_store.vector_store_id])
# Test that the client will use the file search tool
response = await openai_responses_client.get_response(
messages=[
Message(
role="user",
contents=["What is the weather today? Do a file search to find the answer."],
)
],
options={
"tool_choice": "auto",
"tools": [file_search_tool],
},
)
try:
# Use static method for file search tool
file_search_tool = OpenAIChatClient.get_file_search_tool(vector_store_ids=[vector_store.vector_store_id])
# Test that the client will use the file search tool
response = await openai_responses_client.get_response(
messages=[
Message(
role="user",
contents=["What is the weather today? Do a file search to find the answer."],
)
],
options={
"tool_choice": "auto",
"tools": [file_search_tool],
},
)
await delete_vector_store(openai_responses_client, file_id, vector_store.vector_store_id)
assert "sunny" in response.text.lower()
assert "75" in response.text
assert "sunny" in response.text.lower()
assert "75" in response.text
finally:
await delete_vector_store(openai_responses_client, file_id, vector_store.vector_store_id)
@pytest.mark.skip(
reason="Unreliable due to OpenAI vector store indexing "
"potential race condition. See https://github.com/microsoft/agent-framework/issues/1669"
)
@pytest.mark.flaky
@pytest.mark.integration
@skip_if_openai_integration_tests_disabled
@@ -4432,35 +4436,37 @@ async def test_integration_streaming_file_search() -> None:
assert isinstance(openai_responses_client, SupportsChatGetResponse)
file_id, vector_store = await create_vector_store(openai_responses_client)
# Use static method for file search tool
file_search_tool = OpenAIChatClient.get_file_search_tool(vector_store_ids=[vector_store.vector_store_id])
# Test that the client will use the web search tool
response = openai_responses_client.get_streaming_response(
messages=[
Message(
role="user",
contents=["What is the weather today? Do a file search to find the answer."],
)
],
options={
"tool_choice": "auto",
"tools": [file_search_tool],
},
)
try:
# Use static method for file search tool
file_search_tool = OpenAIChatClient.get_file_search_tool(vector_store_ids=[vector_store.vector_store_id])
# Test that the client will use the file search tool
response = openai_responses_client.get_response(
messages=[
Message(
role="user",
contents=["What is the weather today? Do a file search to find the answer."],
)
],
stream=True,
options={
"tool_choice": "auto",
"tools": [file_search_tool],
},
)
assert response is not None
full_message: str = ""
async for chunk in response:
assert chunk is not None
assert isinstance(chunk, ChatResponseUpdate)
for content in chunk.contents:
if content.type == "text" and content.text:
full_message += content.text
assert response is not None
full_message: str = ""
async for chunk in response:
assert chunk is not None
assert isinstance(chunk, ChatResponseUpdate)
for content in chunk.contents:
if content.type == "text" and content.text:
full_message += content.text
await delete_vector_store(openai_responses_client, file_id, vector_store.vector_store_id)
assert "sunny" in full_message.lower()
assert "75" in full_message
assert "sunny" in full_message.lower()
assert "75" in full_message
finally:
await delete_vector_store(openai_responses_client, file_id, vector_store.vector_store_id)
@pytest.mark.flaky
@@ -2,6 +2,7 @@
from __future__ import annotations
import asyncio
import os
from functools import wraps
from pathlib import Path
@@ -77,6 +78,15 @@ async def create_vector_store(client: OpenAIChatClient) -> tuple[str, Content]:
if result.last_error is not None:
raise RuntimeError(f"Vector store file processing failed with status: {result.last_error.message}")
# Wait for the vector store index to be fully searchable.
# create_and_poll confirms file processing, but the search index is eventually consistent.
for _ in range(10):
vs = await client.client.vector_stores.retrieve(vector_store.id)
if vs.file_counts.completed >= 1 and vs.file_counts.in_progress == 0:
break
await asyncio.sleep(1)
await asyncio.sleep(2)
return file.id, Content.from_hosted_vector_store(vector_store_id=vector_store.id)
@@ -355,7 +365,6 @@ async def test_integration_web_search() -> None:
@pytest.mark.integration
@skip_if_azure_openai_integration_tests_disabled
@_with_azure_openai_debug()
@pytest.mark.skip(reason="Azure OpenAI with files raises 500 error. Needs investigation.")
async def test_integration_client_file_search() -> None:
async with AzureCliCredential() as credential:
client = OpenAIChatClient(credential=credential)
@@ -381,7 +390,6 @@ async def test_integration_client_file_search() -> None:
@pytest.mark.integration
@skip_if_azure_openai_integration_tests_disabled
@_with_azure_openai_debug()
@pytest.mark.skip(reason="Azure OpenAI with files raises 500 error. Needs investigation.")
async def test_integration_client_file_search_streaming() -> None:
async with AzureCliCredential() as credential:
client = OpenAIChatClient(credential=credential)
@@ -363,7 +363,7 @@ def _create_workflow() -> Workflow:
chat_client = OpenAIChatCompletionClient(
model=os.environ["AZURE_OPENAI_MODEL"],
api_key=get_bearer_token_provider(credential, "https://cognitiveservices.azure.com/.default"),
credential=get_bearer_token_provider(credential, "https://cognitiveservices.azure.com/.default"),
)
# Create agents for parallel analysis
@@ -70,7 +70,7 @@ def create_spam_agent() -> "Agent":
return Agent(
client=OpenAIChatCompletionClient(
model=os.environ["AZURE_OPENAI_MODEL"],
api_key=get_async_bearer_token_provider(
credential=get_async_bearer_token_provider(
AsyncAzureCliCredential(), "https://cognitiveservices.azure.com/.default"
),
),
@@ -88,7 +88,7 @@ def create_email_agent() -> "Agent":
return Agent(
client=OpenAIChatCompletionClient(
model=os.environ["AZURE_OPENAI_MODEL"],
api_key=get_async_bearer_token_provider(
credential=get_async_bearer_token_provider(
AsyncAzureCliCredential(), "https://cognitiveservices.azure.com/.default"
),
),
-20
View File
@@ -1,20 +0,0 @@
# Copyright (c) Microsoft. All rights reserved.
"""CLI entry point for the flaky test report tool.
Usage:
uv run python -m scripts.flaky_report <reports-dir> <history-file> <output-file>
Example (from python/ directory):
uv run python -m scripts.flaky_report \\
../flaky-reports/ \\
flaky-report-history.json \\
flaky-test-report.md
"""
import sys
from scripts.flaky_report.aggregate import main
if __name__ == "__main__":
sys.exit(main())
@@ -1,11 +1,11 @@
# Copyright (c) Microsoft. All rights reserved.
"""Flaky test report aggregation and trend generation.
"""Integration test report aggregation and trend generation.
Parses JUnit XML (``pytest.xml``) files produced by each CI job, merges
them with historical data, and generates a markdown trend report showing
per-test status across the last N runs.
Usage:
uv run python -m scripts.flaky_report <reports-dir> <history-file> <output-file>
uv run python -m scripts.integration_test_report <reports-dir> <history-file> <output-file>
"""
@@ -0,0 +1,20 @@
# Copyright (c) Microsoft. All rights reserved.
"""CLI entry point for the integration test report tool.
Usage:
uv run python -m scripts.integration_test_report <reports-dir> <history-file> <output-file>
Example (from python/ directory):
uv run python -m scripts.integration_test_report \\
../test-results/ \\
integration-report-history.json \\
integration-test-report.md
"""
import sys
from scripts.integration_test_report.aggregate import main
if __name__ == "__main__":
sys.exit(main())
@@ -247,7 +247,7 @@ def _short_name(nodeid: str) -> str:
def generate_trend_report(runs: list[dict[str, Any]]) -> str:
"""Generate a markdown trend report from run history."""
lines = [
"# 🔬 Flaky Test Report",
"# 🔬 Integration Test Report",
"",
f"*Generated: {datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M UTC')}*",
"",