* Improve PR template and breaking-change label automation
- Add a structured "Related Issue" section using GitHub closing keywords
- Add a Review Guide prompt (major changes, impact, reviewer focus) with a
note that the focus item is for human reviewers only
- Add checklist items for issue linkage / no duplicate PRs and invert the
breaking-change item (checked = not breaking)
- Extend label-title-prefix to prepend [BREAKING] when the "breaking change"
label is added
- Add label-breaking-change workflow to apply the "breaking change" label
when a PR title contains [BREAKING]
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add pull-requests agent skill with dotnet/python links
- Add root .github/skills/pull-requests/SKILL.md covering PR description
authoring (following the PR template) and the review-comment workflow
(review -> plan -> user review -> implement -> reply to all -> resolve)
- Symlink the skill from python/.github/skills and dotnet/.github/skills
- Reference the skill from python/AGENTS.md and dotnet/AGENTS.md
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fold breaking-change labeling into label-pr workflow
Move the title -> 'breaking change' label logic into the existing label-pr
workflow (which already applies the python/.NET labels) and drop the separate
label-breaking-change workflow.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR title prefix review feedback
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Pin patched MessagePack for .NET restore
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Revert MessagePack central pin
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Move title prefix tests out of tracked GitHub tests
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Exclude skill docs from CI path filters
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Match skill symlinks in CI path exclusions
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Exclude AGENTS docs from CI path filters
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Scope title-prefix normalization to a real prefix
The normalization branch in addTitlePrefix matched ^Python (no colon), so
titles like "Python samples improvements" or "Pythonic refactor" were treated
as already-prefixed and only re-cased, never receiving the "Python: " prefix.
Scope the match to ^<prefix>:\s* so only an actual existing prefix is
normalized; otherwise the prefix is prepended. Same fix applies to the .NET
prefix (e.g. ".NETStandard bump").
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Eduard van Valkenburg
·
2026-06-15 10:55:23 +00:00
* Fix .NET Copilot integration tests for SDK v1.0.0
- Remove hard-skip in favor of runtime Assert.Skip when COPILOT_GITHUB_TOKEN is not set
- Add [Trait("Category", "Integration")] for CI filtering
- Fix FunctionTool test: use explicit SessionConfig with Tools, OnPermissionRequest, and SystemMessage
- Mark RemoteMcp test as IntegrationDisabled (requires OAuth flow)
- Create explicit sessions in all tests and delete after each (cleanup)
- Remove unused System.Diagnostics import
- Simplify SkipIfCopilotNotConfigured to only check env var
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review: use try/finally for session cleanup, IsNullOrWhiteSpace
- Wrap act/assert in try/finally so sessions are always deleted even on failure
- Use IsNullOrWhiteSpace instead of IsNullOrEmpty for token check
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add COPILOT_GITHUB_TOKEN to .NET integration test workflow
The Copilot SDK runtime reads this env var directly for authentication.
No Node.js/npm install needed - the SDK downloads the CLI binary at build time.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a dedicated integration test job for the github_copilot package to both
python-integration-tests.yml and python-merge-tests.yml.
The job:
- Runs 6 integration tests marked with @pytest.mark.integration
- Uses COPILOT_GITHUB_TOKEN secret from the integration environment
- Follows the same pattern as other provider integration jobs
- Includes path filtering in merge-tests (github_copilot package + core changes)
- Added to needs lists in report and check jobs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix missing id on function_call_output in Foundry Hosting
The Foundry storage layer was rejecting responses with
"ID cannot be null or empty (Parameter 'id')" because
function_call_output items emitted by OutputConverter had no id on
the wire.
OutputItemFunctionToolCallOutput's public ctor only sets CallId and
Output; Id is read-only and only the SDK's internal ctor populates
it. OutputItemBuilder<T>.ApplyAutoStamps fills ResponseId and
AgentReference but not Id, so the itemId passed to
AddOutputItem<T>(itemId) was used only for event sequencing and the
serialized item went out with id=null.
Switch to stream.OutputItemFunctionCallOutput(callId, output), the
SDK convenience method that uses the internal ctor and stamps the
id. Add a regression test asserting the added/done events carry a
non-empty matching Id.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* ci: free disk space and relocate NuGet cache on ubuntu runners
The ubuntu-latest dotnet-build/test jobs were hitting No space left on device because the runner image only ships ~14 GB free on /. The full multi-TFM build plus the dotnet pack + console-app install-check exhausts that easily.
Add a reusable composite action .github/actions/free-runner-disk-space that runs on Linux runners only and:
* removes pre-installed toolchains we never use here (Android SDK, GHC/Haskell, CodeQL, PyPy, Ruby, Go, boost, vcpkg, etc.), prunes docker images, and disables swap (reclaims ~25-30 GB on /)
* relocates the NuGet package cache to /mnt/nuget via NUGET_PACKAGES env, since /mnt has ~75 GB free on hosted runners
Wire the action into the four ubuntu-touching jobs in dotnet-build-and-test.yml (dotnet-build, dotnet-test, dotnet-foundry-hosted-it, dotnet-test-functions). The action self-guards with runner.os == 'Linux' so the matrix legs that run on windows are unaffected.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: alliscode <25218250+alliscode@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Adding AgentFileStore and FileAccessProvider to support file ased operations for agents.
* Address PR review feedback on FileAccessProvider
- Probe symlinks on the unresolved candidate path so in-root symlinks
cannot silently pass and out-of-root symlinks surface the correct
error message.
- Validate matching_lines elements in FileSearchResult.from_dict and
raise a clean ValueError for non-mapping entries.
- Cap search regex pattern length (256 chars) via a new
_compile_search_regex helper to mitigate ReDoS, and surface the cap
in the file_access_search_files tool description.
- Skip non-UTF-8 files during filesystem search instead of aborting
the entire directory walk.
- Replace the module-scope trailing string in the data-processing
sample with comments to avoid Ruff B018.
- Remove the checked-in working/region_totals.md sample artifact so
the save flow works from a clean checkout.
- Expand the Windows stdout reconfiguration comment in task_runner.py
for clarity.
- Add tests for invalid/oversize regex, non-UTF-8 file search, and
in-root symlink rejection.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix mypy redundant-cast in FileSearchResult.from_dict
Use cast(list[object], ...) instead of cast(list[Any], ...) so the
cast represents a real type change (lists are invariant) and is no
longer flagged by mypy as redundant, while still satisfying pyright's
reportUnknownVariableType. Matches the existing pattern in _memory.py.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Tighten path normalization and directory resolution in FileAccess
- _normalize_relative_path now strips surrounding whitespace up front
so leading/trailing spaces never leak into file segments, and
rejects trailing path separators for file paths so 'foo/' is no
longer silently coerced to 'foo'.
- FileSystemAgentFileStore._resolve_safe_directory_path normalizes
with is_directory=True and maps an empty normalized result to the
root. This matches InMemoryAgentFileStore so whitespace-only
directory inputs resolve to the root instead of raising.
- Added tests for whitespace stripping, trailing-separator rejection,
and whitespace-only directory listing on the filesystem store.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Harden FileAccess search and atomic save in store API
- Add wall-clock timeout (10s) around regex scans so a pathological pattern (e.g. `(a+)+`) below the length cap cannot stall the event loop.
- Offload the InMemoryAgentFileStore regex scan to a worker thread, matching the filesystem store.
- Fail closed when `Path.is_symlink` raises during the safe-path probe so a permission error cannot silently bypass the symlink/reparse-point rejection.
- Add `overwrite: bool = True` to `AgentFileStore.write_file`; the in-memory store performs the check under the existing lock and the filesystem store uses `open(mode='x')` so concurrent callers cannot race past `overwrite=False`.
- `file_access_save_file` now relies on the atomic store call instead of a separate `file_exists` round-trip.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix Python 3.10 timeout handling and add directory arg to list/search tools
- Catch asyncio.TimeoutError in _run_search_with_timeout. In Python 3.10
asyncio.wait_for raises asyncio.exceptions.TimeoutError, which is
distinct from the builtin TimeoutError (the two were unified in 3.11).
Catching the asyncio alias works on every supported version.
- Add an optional directory parameter to file_access_list_files and
file_access_search_files so agents can enumerate / scope searches to
nested folders, not just the store root.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address FileAccess review feedback: case, errors, signal, TOCTOU
- InMemoryAgentFileStore now stores (display_name, content) so list_files
and search_files return the original-case names callers wrote, matching
the behaviour of FileSystemAgentFileStore on case-preserving filesystems
and removing the silent in-memory vs. on-disk contract divergence.
- FileSystemAgentFileStore.read_file raises ValueError instead of letting
UnicodeDecodeError bubble for binary / non-UTF-8 input, restoring
symmetry with search_files (which still skips) and giving the tool
layer a recoverable type to translate.
- Tool wrappers now catch ValueError and OSError around every operation
and surface them as readable strings, so 'you used ..' and 'the file
already exists' are both reported to the model the same way instead of
the former crashing out as an unhandled exception.
- _search_files_sync logs per skipped non-UTF-8 file at WARNING and an
aggregate INFO summary so operators can distinguish 'no matches' from
'half the corpus was unreadable'.
- FileSystemAgentFileStore softens its docstrings to acknowledge the
inherent probe-then-open TOCTOU window. On POSIX both read and write
now pass O_NOFOLLOW so the kernel refuses if the leaf segment becomes
a symlink between the probe and the open. Windows has no equivalent
flag; the limitation is documented.
- Tests cover: case preservation on list/search, ValueError on non-UTF-8
read at the store and tool layer, tool-layer string responses for
path-traversal and oversized-regex inputs, search-skip log output,
symlink rejection on delete/search/list, and symlinked intermediate
directory rejection.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address FileAccess nit comments: docstrings, enumerate, opt-in delete approval
- Expand FileSearchMatch/FileSearchResult.to_dict docstrings to explain why
the override is needed (__slots__ defeats the mixin's __dict__ iteration)
and why exclude/exclude_none are accepted-but-ignored (mixin signature
compatibility for callers like to_json).
- Use enumerate(lines, start=1) in _search_file_content so the +1 below is
no longer needed; rename loop variable to line_number for clarity.
- Add opt-in require_delete_approval: bool = False on FileAccessProvider.
When True, file_access_delete_file is registered with approval_mode
'always_require' so the host must approve every delete. Default False
preserves current behaviour and matches the .NET reference, but
deployments that want a safer-by-default posture can enable it.
- Add tests covering both delete approval modes.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* FileAccess: require delete approval by default
Flip the default for FileAccessProvider(require_delete_approval=...) from
False to True so destructive deletes are gated by host approval out of the
box. Callers that want the previous autonomous behaviour (which matches the
.NET reference) can pass require_delete_approval=False.
Tests updated accordingly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fixing linkinspector by installing Chrome for puppeteer first.
---------
Co-authored-by: Ben Thomas <25218250+alliscode@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replaces every floating tag in our workflow and composite action files
with an immutable 40-character commit SHA, keeping the original `# vX`
comment so Dependabot can still propose version bumps. 186 occurrences
across 25 workflows and 2 composite actions.
Also widens the github-actions Dependabot entry to use the plural
`directories` key with `/.github/actions/*` so composite actions under
`.github/actions/<name>/action.yml` are kept up to date. Previously
Dependabot only scanned `.github/workflows` and the repo-root
`action.yml`, leaving our `python-setup` and `sample-validation-setup`
composite actions unmaintained.
The upsidr/merge-gatekeeper@v1 action is a Dockerfile-based action that
builds a golang image on every run. On merge_group events the run step
is conditioned out via `if: github.event_name == 'pull_request'`, so the
build happens but produces nothing.
Replace with an actions/github-script@v8 polling loop that mirrors the
action's behavior exactly: merges combined-statuses and check-runs for
the PR head SHA, with combined-status winning on name collisions, and
the same conclusion mapping (skipped → dropped, success/neutral →
success, anything else terminal → error). Same job name, triggers,
permissions, timeout (3600s), interval (30s), and ignored list, so
existing required-check rules stay valid.
PR runs now poll the API in seconds instead of waiting on a per-run
docker image build, and merge_group runs become near-instant no-ops.
* Split DurableTask/AzureFunctions integration tests into dedicated CI job
- Add -TestProjectNameExclude parameter to New-FilteredSolution.ps1
- Add 'functions' and 'core' path filters to paths-filter job
- Exclude DurableTask/AzureFunctions from main dotnet-test job
- Remove emulator setup from dotnet-test (no longer needed)
- Add new dotnet-test-functions job (ubuntu/net10.0 only, path-conditional)
- Update merge gate and report job to include dotnet-test-functions
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR feedback: add Workflows.Generators to core filter, drop dotnetChanges gate from functions job
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Re-enable Anthropic integration tests
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Upgrade Anthropic SDK 12.13.0 -> 12.20.0 to fix M.E.AI incompatibility
Fixes MissingMethodException on WebSearchToolResultContent.get_Results()
caused by Anthropic 12.13.0 being compiled against an older
Microsoft.Extensions.AI.Abstractions version.
Suppress RT0003 in AI.Abstractions.csproj as the transitive reference
from the upgraded Anthropic SDK conflicts with the explicit one.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix Anthropic unit test mocks for SDK 12.20.0 interface changes
Add missing interface members: IAnthropicClient.WebhookKey,
IBetaService.MemoryStores, IBetaService.Webhooks, IBetaService.UserProfiles
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Re-enable CheckSystem declarative integration tests
The CheckSystem.yaml tests were temporarily skipped in PR #4270 during
the Azure.AI.Projects 2.0.0-beta.1 SDK update. Since then, the system
variable plumbing (SystemScope, SetLastMessageAsync, conversation
initialization) has been significantly updated and stabilized. The
other tests in these same files pass reliably using the same
infrastructure.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix CheckSystem test case to expect 1 response
The CheckSystem workflow sends a 'PASSED!' SendActivity when all system
variables are populated, producing 1 AgentResponseEvent. The test case
had min_response_count: 0 with no max, so the assertion defaulted max
to 0 and failed with 'Response count greater than expected: 0 (Actual: 1)'.
Updated to expect exactly 1 response, matching the SendActivity pattern.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Re-enable Foundry OpenAPI server-side tool integration test
Remove Skip="For manual testing only" from
AsAIAgent_WithOpenAPITool_NativeSDKCreation_InvokesServerSideToolAsync.
The test already uses RetryFact(3 retries, 5s delay) to handle
transient failures from the external restcountries.com API.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Include workflow file in functions/core path filters
A PR editing only dotnet-build-and-test.yml would skip
dotnet-test-functions because the workflow path was missing
from both the functions and core path filter lists.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Rename filter parameters for consistency
TestProjectNameFilter -> TestProjectNameIncludeFilter
TestProjectNameExclude -> TestProjectNameExcludeFilter
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove unnecessary RT0003 warning suppression
The RT0003 suppression was added during the Anthropic SDK 12.20.0
upgrade but the warning no longer fires. Removing it to keep the
NoWarn list minimal.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove duplicate WebhookKey properties from merge
Both our branch and main added WebhookKey to the Anthropic test
mock classes, resulting in CS0102 duplicate definition errors.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* .NET: Hosted Agents - RAG Sample with Azure AI Search (#5693)
Adds a Hosted-AzureSearchRag sample plus a live Foundry.Hosting integration
test scenario backed by a real Azure AI Search index.
Sample (Hosted-AzureSearchRag): keyword-only Azure AI Search via
SearchClient adapter into TextSearchProvider, scope-aware
DevTemporaryTokenCredential consuming AZURE_BEARER_TOKEN_FOUNDRY +
AZURE_BEARER_TOKEN_SEARCH for local Docker, Dockerfile + contributor
Dockerfile mirroring Hosted-TextRag.
Integration test: AzureSearchRagHostedAgentFixture extends the PR #5598
HostedAgentFixture with the new azure-search-rag scenario branch in the
shared test container; AzureSearchRagHostedAgentTests asserts the model
returns canary tokens (TR-CANARY-7821, SHIP-CANARY-4493) that exist only
in the seeded documents - real proof the agent grounded its answer in
retrieved content rather than training data.
* Address PR 5701 Copilot review feedback
- Sample README: drop stale 'bootstraps the index on first run' line; index is pre-provisioned out of band
- Sample + TestContainer search adapters: propagate CancellationToken to await foreach via .WithCancellation()
* .NET: Foundry.Hosted IT - fix MSBuild parallel-output races
Two surgical changes inside the dotnet-foundry-hosted-it job:
1. Replace dotnet build <slnx> -f net10.0 with dotnet build <test.csproj>. The test csproj pins TargetFrameworks=net10.0 and its ProjectReference closure gives MSBuild a single-rooted graph, eliminating the duplicate inner-builds that race on bin/obj. Drops the two New-FilteredSolution.ps1 steps.
2. In it-build-image.ps1, drop the -UsePrebuiltProjectReferences switch and always pass --no-dependencies to dotnet publish. Publish now resolves TestContainer's framework refs by reading prebuilt DLLs and never re-touches them. Replaces the partial-mitigation in PR #5689 with a structural fix.
Local validation confirmed published Foundry.dll has identical mtime and bytes as the prebuild output.
* .NET: dotnet test - use --project flag for Microsoft Testing Platform
* Add dotnet integration test report to CI
- Add --report-junit flag to dotnet integration test step to generate
JUnit XML alongside TRX, with explicit --results-directory to
centralize output in IntegrationTestResults/
- Upload JUnit XML artifacts from each matrix leg (net10.0/ubuntu,
net472/windows) as dotnet-test-results-{framework}-{os}
- Add dotnet-integration-test-report job that downloads artifacts,
runs the existing aggregate.py script, posts markdown to Job Summary,
and saves trend history via actions/cache
- Refactor aggregate.py to discover JUnit XML files recursively,
supporting both pytest (pytest.xml) and xunit (*.junit.xml) layouts
- Handle provider name derivation for dotnet artifact naming convention
- Fix nodeid collision when same test runs under multiple frameworks
by qualifying keys with provider when collisions are detected
- Improve module extraction for dotnet C# classnames (recognizes
IntegrationTests/UnitTests namespace segments)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* chore: trigger dotnet CI for report validation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: use .junit extension (not .junit.xml) for xunit v3 output
xUnit v3 generates files with .junit extension, not .junit.xml.
Update upload glob and aggregate.py discovery to match.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: use deterministic provider-qualified keys for dotnet tests
Always prefix dotnet test keys with provider (e.g. net10.0 (ubuntu)::TestName)
to ensure stable, comparable counts across runs regardless of file parse order.
Also show Executed (passed+failed) instead of Total in summary table.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: match Python report summary format (Total, passed/total, etc.)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* feat: split dotnet report into per-framework tables
Dotnet tests run on multiple frameworks (net10.0, net472). Instead of
one combined table with unstable totals, show separate sections per
framework — each with its own summary row and per-test table. Python
reports retain the original single-table format.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Re-enable 7 flaky dotnet integration tests with increased timeouts
Increase timeouts to reduce timing-related flakiness in LLM-backed
integration tests (issue #4971):
- ExternalClientTests: 60s -> 120s default timeout
- SamplesValidationBase: 60s -> 120s default timeout
- ConsoleAppSamplesValidation: 90s -> 150s for long-running tests
- AzureFunctions SamplesValidation: 2min -> 3min orchestration timeout,
60s -> 90s per-step WaitForConditionAsync timeouts
Remove all Skip=Flaky annotations and unused SkipFlakyTimingTest constants.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Re-skip LLM non-determinism flaky tests, keep timeout fixes
Re-skip SingleAgentOrchestrationHITLSampleValidationAsync and
LongRunningToolsSampleValidationAsync - these fail due to LLM producing
extra review notifications, not timeouts. Updated skip reasons to
accurately describe the root cause. Reverted unnecessary timeout change
on the skipped LongRunningTools test.
The remaining 5 re-enabled tests with timeout increases are stable.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Enable Anthropic integration tests in CI
Replace hardcoded skip with conditional skip pattern (matching
CopilotStudio approach): tests gracefully skip when ANTHROPIC_API_KEY
is missing, and run when present.
Changes:
- AnthropicChatCompletionFixture: try/catch in InitializeAsync with
Assert.Skip on missing config (replaces hardcoded SkipReason)
- AnthropicSkillsIntegrationTests: same pattern per test method
- dotnet-build-and-test.yml: wire up ANTHROPIC_API_KEY,
ANTHROPIC_CHAT_MODEL_NAME, and ANTHROPIC_REASONING_MODEL_NAME
env vars to the integration test step
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix missing System using in AnthropicSkillsIntegrationTests
Add 'using System;' for InvalidOperationException in try/catch blocks.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Skip flaky SingleAgentOrchestrationChainingSampleValidationAsync
LLM non-determinism causes Assert.NotNull failures on orchestration
results. Skip until test logic is hardened against non-deterministic
LLM responses.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Re-enable HITL and LongRunningTools tests with timeout and flexibility fixes
- Remove Skip attribute from SingleAgentOrchestrationHITLSampleValidationAsync
- Remove Skip attribute from LongRunningToolsSampleValidationAsync
- Increase timeout from 120s/90s to 180s to accommodate 2+ LLM round-trips
- Replace rigid 2-cycle assertion with flexible approval logic that handles
extra review cycles from LLM non-determinism
Fixes the two failure modes identified in #4971:
1. Timeout: 120s/90s was insufficient for multiple LLM calls under CI load
2. Extra notifications: Assert.Fail on 3rd+ review cycle was too rigid
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Increase AzureFunctions LongRunningTools test timeouts from 90s to 180s
The LongRunningToolsSampleValidationAsync test in the AzureFunctions integration
tests was failing in CI with TimeoutException at the 'Content published
notification is logged' step. The 90-second timeouts are too tight for CI
environments where LLM calls and orchestration overhead can be slow.
Increased all three WaitForConditionAsync timeouts from 90s to 180s:
- Waiting for human feedback notification
- Waiting for publish notification (the step that was failing)
- Waiting for orchestration completion
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Merge main and fix dotnet report path after flaky_report rename
Merge upstream/main which renamed scripts/flaky_report/ to
scripts/integration_test_report/ (from Python PR #5454). Update the
dotnet-build-and-test workflow to reference the new path.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add RetryFact to DurableTask and AzureFunctions integration tests
These tests interact with LLMs via stdin/stdout (DurableTask) or HTTP
(AzureFunctions) and are inherently non-deterministic. Unlike the Python
side which uses pytest-retry, the dotnet tests had no retry mechanism
and a single transient failure would fail the entire CI run.
Changes:
- Switch [Fact] to [RetryFact(2, 5000)] on all LLM-dependent tests
across ConsoleAppSamplesValidation, ExternalClientTests,
WorkflowConsoleAppSamplesValidation, and AzureFunctions SamplesValidation
- Add re-prompt mechanism to LongRunningToolsSampleValidationAsync:
if the LLM doesn't invoke the tool within 60s, re-send the prompt
(up to 2 retries) instead of burning the full timeout
- Reduce LongRunningTools timeout from 240s to 180s (re-prompt makes
the extra buffer unnecessary)
- Leave simple/deterministic tests as [Fact] (SingleAgent, unit tests)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add persist-credentials: false to Integration Test Report checkout step
Matches the convention used by other checkout steps in this workflow
to avoid leaving GITHUB_TOKEN credentials in the local git config.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* small fixes
* disable anthropic failing tests
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CI publish step: gate the BuildProjectReferences=false fast-path on an explicit -UsePrebuiltProjectReferences switch (passed by the workflow) instead of marker detection. Adds a preflight error when stale obj/Release/net10.0 outputs would cause CS0579, with actionable recovery instructions.
Telemetry UT flake: AgentFrameworkResponseHandlerTelemetryTests was using a plain List<Activity> for OTel's InMemoryExporter. The exporter writes from background Activity completion callbacks while parallel tests on the same global ActivitySource feed every listener, racing against the assertion's enumeration and throwing 'Collection was modified'. Replaced with a small thread-safe ConcurrentActivityList that locks add/enumerate and returns a snapshot for assertions.
* Foundry.Hosting.IntegrationTests: scaffold project, fixtures, and 24 tests
Add a new integration test project for Foundry hosted agents alongside the existing Foundry.IntegrationTests project. The project provisions a real Foundry hosted agent per scenario via AgentAdministrationClient.CreateAgentVersionAsync, points it at a single test container image (built and pushed out of band by scripts/it-build-image.ps1 in a follow up commit), and exercises the agent through AIProjectClient.AsAIAgent.
Six scenario fixtures are introduced, each pointing at the same image but selecting behavior via the IT_SCENARIO environment variable on the HostedAgentDefinition:
- HappyPathHostedAgentFixture (round trip, multi turn, stored=false flag)
- ToolCallingHostedAgentFixture (server side AIFunctions)
- ToolCallingApprovalHostedAgentFixture (approval flow)
- ToolboxHostedAgentFixture (Foundry toolbox)
- McpToolboxHostedAgentFixture (MCP backed toolbox)
- CustomStorageHostedAgentFixture (custom storage provider)
24 tests across 6 test classes are scaffolded. All are tagged Skip pending the test container build and the end to end smoke iteration in follow up commits. Once the container is in place the Skip annotations can be removed scenario by scenario.
Adds an IT_HOSTED_AGENT_IMAGE constant to the shared TestSettings so every IT project agrees on the env var name the build script emits.
* Foundry.Hosting.IntegrationTests: add TestContainer, build script, slnx, README
Adds the rest of the integration test infrastructure on top of the previous scaffolding commit:
* Foundry.Hosting.IntegrationTests.TestContainer csproj and Program.cs implementing the multi scenario container (one image, IT_SCENARIO env var dispatches between happy-path, tool-calling, tool-calling-approval, toolbox, mcp-toolbox, and custom-storage). The toolbox, mcp-toolbox, and custom-storage branches are placeholders pending API surface stabilization.
* Dockerfile and dockerignore in the test container project, using the contributor pattern matching the investigation work (host side dotnet publish, container only does COPY out/).
* scripts/it-build-image.ps1 with mandatory Registry parameter (no hardcoded ACR), content hashed tags so unchanged source results in a no op push, and emits IT_HOSTED_AGENT_IMAGE for shells and CI to consume.
* slnx entry for both new projects.
* README in the IT project covering env vars, image build, scenario table, and current placeholder status.
Steps still pending: end to end smoke (step 5) and CI workflow integration (step 6) require a live Foundry deployment and ACR push, so they land in follow up commits.
* Foundry.Hosting.IntegrationTests: address PR 5598 review feedback
Fix issues raised by Copilot review:
* it-build-image.ps1: hash file contents, not the path list, so any source edit produces a fresh tag. Normalize Registry input by stripping scheme and trailing slash before deriving the ACR short name. Validate the short name is non empty.
* HostedAgentFixture: route GetAgentAsync through _adminClient (which has the FoundryFeaturesPolicy attached) instead of through _projectClient.AgentAdministrationClient (which does not).
* HostedAgentFixture FoundryFeaturesPolicy: replace Headers.Add with Remove plus Add so retries cannot accumulate duplicate headers.
* HappyPath, ToolCalling, ToolCallingApproval, CustomStorage tests: create the AgentSession before turn 1 and reuse it for both turns. The previous pattern created the session after turn 1 so turn 2 had no link to turn 1, defeating the multi turn assertion.
* .NET: Foundry.Hosting.IntegrationTests: constrain to net10.0 + dotnet format autofix
- Set <TargetFrameworks>net10.0</TargetFrameworks>: the project references both
Microsoft.Agents.AI.Foundry.Hosting (net8/9/10 only) and AgentConformance.IntegrationTests
(net10.0;net472 — inherits the tests-default TFM list). The intersection is net10.0;
the previous $(TargetFrameworksCore) triple caused NU1702 + System.Text.Json version
conflicts on the net8.0/net9.0 builds because AgentConformance had no matching asset.
- Apply `dotnet format` autofix on the test files (IDE0005, IDE0009, IDE0032, IMPORTS).
* .NET: Foundry.Hosting.IntegrationTests.TestContainer/Program.cs: add UTF-8 BOM
CI's check-format requires charset=utf-8-bom per .editorconfig.
* Foundry.Hosting IntegrationTests: wire end-to-end CI flow against hosted agents
Make the integration tests usable end-to-end against a live Foundry deployment, including
a per-run rebuild of the test container so framework code changes are exercised.
Fixture (HostedAgentFixture.cs)
* Switch from per-run unique agent names to stable scenario-keyed names (it-happy-path,
it-tool-calling, ...). The agent's managed identity carries the Azure AI User role on
the project scope, which is required for inbound inference; deleting the agent recycles
the MI and breaks that role assignment, so we keep the agent across runs and only churn
versions.
* Add IT_RUN_ID env var to defeat Foundry's content-addressed version dedup; otherwise a
rerun just receives the existing version and Dispose deletes it.
* PATCH the per-agent endpoint with AgentEndpointConfig (Responses protocol, version
selector at 100% to the new version). Without this, /agents/{name}/endpoint/protocols/
openai/responses returns HTTP 400.
* Build a per-agent ProjectOpenAIClient (not the cached projectClient.ProjectOpenAIClient,
which is bound to the project-level URL); set AgentName in options so the URL routes
through the agent endpoint, and add the Foundry-Features header to the inference
pipeline.
* Use Versions (which serializes to container_protocol_versions) instead of the
deprecated ProtocolVersions; the server now rejects the legacy field.
* On Dispose, delete only the version this fixture created. Never delete the agent.
Tests
* Tag every HostedAgentTests class with [Trait("Category", "FoundryHostedAgents")] so the
CI workflow can route them to a separate Foundry project than the rest of the
integration suite.
CI workflow (.github/workflows/dotnet-build-and-test.yml)
* Add a foundryHosting paths-filter covering Microsoft.Agents.AI.Foundry.Hosting and its
in-repo dependency chain (Foundry, Agents.AI, Agents.AI.Abstractions), the test
container, the test fixture, Directory.Packages.props, the build script, and this
workflow file. Skip the costly hosted-agent steps when none of those changed.
* Add "Build and push Foundry Hosted Agents test container" step that invokes
scripts/it-build-image.ps1 against vars.IT_HOSTED_AGENT_REGISTRY and pipes the resulting
IT_HOSTED_AGENT_IMAGE=<tag> into GITHUB_ENV.
* Add "Run Foundry Hosted Agents Integration Tests" step that filters in only the new
trait, with AZURE_AI_PROJECT_ENDPOINT/AZURE_AI_MODEL_DEPLOYMENT_NAME pointed at
IT_HOSTED_AGENT_PROJECT_ENDPOINT/IT_HOSTED_AGENT_MODEL_DEPLOYMENT_NAME (Tao project,
East US 2; the SK IT project's region does not yet support hosted agents preview).
* Exclude the new trait from the existing "Run Integration Tests" step.
* TEMP: drop the != 'pull_request' guard on the new steps and on Azure CLI Login when the
paths-filter triggers, so PR #5598 can validate the wiring before promoting to merge
queue only. Restore the original guard after one green PR run.
Build script (scripts/it-build-image.ps1)
* Hash now spans TestContainer source AND its referenced framework projects so any
framework code change forces a fresh tag and a real docker push; the previous
TestContainer-only hash silently reused stale images on framework edits.
Bootstrap script (dotnet/tests/Foundry.Hosting.IntegrationTests/scripts/it-bootstrap-agents.ps1)
* New idempotent script that creates the six stable scenario agents and grants Azure AI
User on the project scope to each agent's MI. Run once per Foundry project. Includes
AAD-graph propagation retries because newly created MIs take time to appear there.
README (dotnet/tests/Foundry.Hosting.IntegrationTests/README.md)
* Document the bootstrap prerequisite, the regional caveat (East US 2 is the only region
we have validated; East US returned "Unsupported region" at the time of writing), the
per-run image rebuild, and the CI wiring including the SP RBAC requirements.
SDK pin (TEMP)
* Bump Microsoft.Agents.AI.Foundry.Hosting's Azure.AI.Projects VersionOverride to
2.1.0-alpha.20260505.1 from the azure-sdk public daily feed (added to nuget.config).
This release is the first that builds the per-agent inference URL as
/agents/{name}/endpoint/protocols/openai (the 2.1.0-beta.1 release builds
.../openai/openai/v1, which the server rejects). Revert both the feed and the override
once the URL fix lands in a stable Azure.AI.Projects release.
* Foundry.Hosting IntegrationTests: revert alpha SDK pin; move endpoint PATCH to bootstrap
The alpha SDK pin (Azure.AI.Projects 2.1.0-alpha.20260505.1 from the azure-sdk public
daily feed) was needed only for the URL routing fix and the strongly-typed
AgentEndpointConfig/PatchAgentOptions wrapper. We do not need either right now: the
fixture stays compatible with the public 2.1.0-beta.1 by moving the one-time endpoint
PATCH to the bootstrap script (it sets version_selector to FixedRatio @latest, so each
new fixture run becomes the served version automatically without a per-run PATCH from
the test code). The hosted-agent invocation path will start working end-to-end once the
URL routing fix lands in a stable Azure.AI.Projects release; until then the tests stay
[Fact(Skip = ...)] as documented.
* Revert dotnet/nuget.config: drop the azure-sdk-for-net public feed.
* Revert Microsoft.Agents.AI.Foundry.Hosting.csproj VersionOverride to 2.1.0-beta.1.
* Revert Microsoft.Agents.AI.Foundry.UnitTests and Microsoft.Agents.AI.Foundry.Hosting.UnitTests
Azure.AI.Projects pin (they had been bumped to align Azure.Core 1.54 transitive).
* Drop the AgentEndpointConfig PATCH block from HostedAgentFixture.cs (the type is
alpha-only). Replace with a comment pointing at the bootstrap script.
* Bootstrap script (it-bootstrap-agents.ps1) now also PATCHes each agent's endpoint
with version_selector=@latest if not already set. Idempotent.
* Foundry.Hosting IntegrationTests: drop accidentally committed filtered.slnx
* Foundry.Hosting IntegrationTests: revert TEMP PR override on Azure CLI Login + IT steps
The previous attempt to validate the new hosted-agent IT wiring on PR #5598 failed
because the PR is from a fork (rogerbarreto/agent-framework-public). GitHub never passes
environment secrets to fork PRs regardless of event-name guards on individual steps,
so 'azure/login@v2' fails with 'client-id and tenant-id are not supplied'. Restore the
original github.event_name != 'pull_request' guard. The new steps will execute on
push to main and on merge_group runs.
* Foundry.Hosting IntegrationTests: invoke build-and-push script with absolute path
The pwsh shell on the GitHub Actions runner couldn't resolve ./scripts/it-build-image.ps1
when the step had no working-directory set; the step inherits the runner's PWD which is
not always the repo root after preceding steps. Use github.workspace explicitly to remove
the ambiguity.
* Foundry.Hosting IntegrationTests: move it-build-image.ps1 inside the IT project tree
The previous location at scripts/it-build-image.ps1 lived outside the sparse-checkout
paths the workflow uses (.github, dotnet, python, declarative-agents), so the runner
never had the file when the new step tried to invoke it. Move the script next to its
sibling it-bootstrap-agents.ps1 inside the IT project tree, and anchor its relative
paths to the repo root via so callers can invoke it from any PWD.
* Move scripts/it-build-image.ps1 -> dotnet/tests/Foundry.Hosting.IntegrationTests/scripts/it-build-image.ps1
* Add Push-Location to the resolved repo root inside the script (Pop-Location in finally)
so the existing relative paths (TestContainerProject, hashed src dirs) keep working
no matter where the script is invoked from.
* Update the workflow path filter and the step's invocation path to the new location.
* Foundry.Hosting IntegrationTests: enable 5 HappyPath tests on the live Foundry endpoint
The fixture already constructs ProjectOpenAIClient via the per-agent path that beta.1
supports (new ProjectOpenAIClient(uri, cred, opts { AgentName })), so no SDK pin bump
is required to run the smoke tests end-to-end. Un-skip the 5 tests that pass against
the live test container.
Tests un-skipped (verified passing locally against tao-foundry-prj):
* RunAsync_ReturnsNonEmptyTextAsync
* RunStreamingAsync_YieldsAtLeastOneUpdateAsync
* MultiTurn_WithPreviousResponseId_PreservesContextAsync
* StoredFalse_Baseline_DoesNotPersistResponseAsync
* Instructions_FromContainerDefinition_AreObeyedAsync
Tests still skipped with a more specific reason (4 of 9 in HappyPath plus all
ToolCalling*, McpToolbox, Toolbox, CustomStorage) because the test container does not
yet emit usable response_id / conversation_id chains, and the placeholder scenarios are
not implemented in the test container's Program.cs. These are test container limitations,
not infra bugs, and can be un-skipped as the container surfaces stabilize.
* Foundry.Hosting IntegrationTests: extract hosted IT into parallel job, add Workflows dep
Address Wesley's review feedback on PR #5598:
1. Pull Foundry hosted-agent IT into its own dotnet-foundry-hosted-it job that runs in parallel to dotnet-build and dotnet-test. Same path-filter gate keeps it skipped on unrelated edits. Builds only the filtered solution containing Foundry.Hosting.IntegrationTests and src deps. dotnet-build-and-test-check now waits on it too.
2. Add Microsoft.Agents.AI.Workflows to the foundryHosting paths-filter and to hashedDirs in it-build-image.ps1 since Foundry.Hosting transitively depends on it.
TFM constraint on the IT csproj stays at net10.0 because AgentConformance.IntegrationTests targets net10/net472 and is consumed by ~12 other IT projects on net472.
---------
Co-authored-by: Roger Barreto <rbarreto@microsoft.com>
* Enable Ollama integration tests in CI and rename report to Integration Test Report
- Install Ollama, cache models (qwen2.5:0.5b + nomic-embed-text), and start
server in the Misc integration job for both workflow files
- Set OLLAMA_MODEL and OLLAMA_EMBEDDING_MODEL env vars so the 5 Ollama tests
are no longer skipped
- Rename Flaky Test Report to Integration Test Report throughout (job names,
artifact names, cache keys, file names, script titles/docstrings)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Bump Ollama model to qwen2.5:1.5b for better instruction following
The 0.5b model was too small to reliably follow simple prompts like
'Say Hello World', causing test assertion failures. The 1.5b model
follows instructions more reliably while still being small enough
for fast CI pulls (~1GB).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Re-enable reliable streaming integration tests
Remove the hard skip on test_03_reliable_streaming tests that was
temporarily disabled for instability investigation. CI infrastructure
(Azurite, DTS emulator, Redis, func CLI) is already in place.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Re-enable skipped Functions/DurableTask tests and bump timeout to 480s
- Remove hard skips from 4 tests in test_11_workflow_parallel.py
- Remove hard skip from test_conditional_branching in test_06_dt_multi_agent_orchestration_conditionals.py
- Increase pytest --timeout from 360 to 480 for Functions+DurableTask CI job
- Updated in both python-merge-tests.yml and python-integration-tests.yml
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Re-skip failing Functions/DurableTask tests with specific root causes
- test_11_workflow_parallel (4 tests): xdist worker crashes during execution
- test_conditional_branching: orchestration fails with RuntimeError, not a timeout
- Keep 480s timeout bump for remaining Functions tests
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix auth routing in samples 06/11: api_key -> credential for Azure OpenAI
Both samples passed a bearer token provider via api_key= which caused the
client to route to api.openai.com instead of Azure OpenAI, resulting in
401 Unauthorized. Changed to credential= which correctly triggers Azure
routing and picks up AZURE_OPENAI_ENDPOINT from the environment.
- samples/azure_functions/11_workflow_parallel/function_app.py: 1 fix
- samples/durabletask/06_multi_agent_orchestration_conditionals/worker.py: 2 fixes
- Re-enable 4 parallel workflow tests and 1 conditional branching test
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Re-skip parallel workflow tests: xdist worker distribution issue
The 4 parallel workflow tests crash because xdist worksteal distributes
them across separate workers, each spawning its own func process against
shared emulators. Auth fix (api_key->credential) was valid and stays.
test_conditional_branching now passes with the auth fix.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix E501 line-too-long in azurefunctions parallel test skip reasons
Wrap skip reason strings to stay within 120 char line limit.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add retry logic and port-conflict fix for Ollama CI setup
- Kill any auto-started Ollama before launching serve (fixes port
conflict: 'address already in use')
- Retry ollama pull up to 3 times with 15s backoff (fixes 429 rate
limit failures)
- Applied to both python-merge-tests.yml and python-integration-tests.yml
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix flaky integration tests and re-enable skipped tests
- Foundry agent: add allow_preview=True to custom client test
- Foundry hosting: raise max_output_tokens 50->200, add temperature,
relax assertion in test_temperature_and_max_tokens
- Foundry embedding: update skip reason with root cause (endpoint mismatch)
- OpenAI file search: fix vector store indexing race condition by polling
file_counts before querying; fix get_streaming_response -> get_response(stream=True)
- Azure OpenAI file search: remove skip (transient 500 resolved)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove temperature from foundry hosting test (unsupported by CI model)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Stabilize Ollama tool call integration tests with no-arg function
Use a no-argument greet() function instead of hello_world(arg1) for
integration tests. The 1.5B model in CI is unreliable at generating
correct tool call arguments, causing 'Argument parsing failed' errors.
A no-arg function eliminates this flakiness entirely.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Increase reliable streaming test timeouts from 30s to 60s
The LLM call through Azure OpenAI + Redis streaming pipeline can exceed
30s in CI due to cold starts or throttling. Raise to 60s to reduce
flaky timeouts while still bounded by pytest's 120s per-test limit.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Re-enable workflow parallel tests with xdist_group marker
The tests were skipped because xdist distributes module tests across
workers, each spawning their own func process (port conflicts). Adding
xdist_group forces all tests in this module onto a single worker so
the module-scoped function_app_for_test fixture works correctly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Revert "Re-enable workflow parallel tests with xdist_group marker"
This reverts commit 455c28da62.
* Rename flaky_report to integration_test_report and add try/finally cleanup
- Rename scripts/flaky_report/ to scripts/integration_test_report/ to
reflect expanded scope beyond flaky-test detection
- Update workflow references in both CI files
- Wrap file search integration tests in try/finally to ensure vector
store cleanup runs even on test failure or timeout
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix Ollama pull failure propagation and Azure OpenAI vector store readiness
- Ollama CI: fail the step immediately if model pull fails after 3
retries instead of silently proceeding to tests
- Azure OpenAI file search: add the same vector-store readiness polling
that was applied to the non-Azure OpenAI tests, preventing eventual
consistency race conditions
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* remove load_dotenv from test file
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Scopes the triage job to the integration GitHub Environment, adds
the azure/login OIDC step, and exposes the same OpenAI / Azure
OpenAI / Foundry / Anthropic env vars the integration test
workflow uses. This lets the triage agent write repro code that
constructs model clients from the environment without any secrets
entering the agent prompt or generated-code literals.
Azure OpenAI and Foundry continue to authenticate via AAD
(DefaultAzureCredential), so there is no API key to leak for
those providers.
* Automated issue triage workflow
* Bump dependencies
* Fix issue-triage workflow: security, reliability, and testability
Address six review comments on the issue-triage workflow:
1. Change trigger from issues:opened to issues:labeled so the
secret-backed triage flow is only triggered by a maintainer-
controlled signal.
2. Include inputs.issue_number in the concurrency group so
workflow_dispatch runs for the same issue are properly
de-duplicated.
3. Improve team membership error handling to fail closed: verify
the team exists before checking membership, and only treat a
404 as 'not a member' (all other errors fail the job).
4. Use optional chaining (issue.user?.login) for the API-fetched
issue to handle deleted GitHub accounts without crashing.
5. Extract the inline github-script into a testable module at
.github/scripts/check_team_membership.js with 10 tests in
.github/tests/test_check_team_membership.js covering all
code paths (payload/API author resolution, deleted accounts,
team lookup failure, 404 vs non-404 membership errors).
6. Make the spam gate actually stop the job by exiting non-zero
instead of just logging, so future steps cannot accidentally
run for spam issues.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Make issue-triage workflow manually triggered only for initial testing
Remove the 'issues' event trigger, keeping only 'workflow_dispatch' so the
workflow can be tested manually before enabling automatic triggers.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add flaky test trend reporting to CI workflows
Parse JUnit XML (pytest.xml) from each integration test job and
aggregate results into a markdown trend report showing per-test
pass/fail/skip status across the last 5 runs.
Changes:
- Add python/scripts/flaky_report/ package (JUnit XML parser + trend
report generator following the sample_validation pattern)
- Add upload-artifact steps to all 6 integration test jobs in both
python-merge-tests.yml and python-integration-tests.yml
- Add python-flaky-test-report aggregation job with history caching
- Add --junitxml=pytest.xml to integration-tests.yml jobs (already
present in merge-tests.yml)
- Fix Cosmos job --junitxml path (use absolute path since uv run
--directory changes cwd)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix flaky report: handle missing test results gracefully
- Guard against missing reports directory in load_current_run()
- Only run report job when at least one integration test job completed
(skip when all jobs are skipped, e.g. on pull_request events)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR review: fix provider names and if-expression precedence
- Use explicit provider name mapping in _derive_provider() so OpenAI
renders correctly instead of 'Openai'
- Fix operator precedence in workflow if-expressions by wrapping
success/failure checks in parentheses
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add File column and xfail detection to flaky test report
- Add File column showing module name (e.g., test_openai_chat_client)
to disambiguate tests with the same function name across files
- Detect pytest xfail tests in JUnit XML (type=pytest.xfail) and
show them with a distinct warning emoji instead of skip emoji
- Update legend to include xfail explanation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add Foundry embedding env vars to merge-tests workflow
Sync the Foundry integration job in python-merge-tests.yml with
python-integration-tests.yml by adding FOUNDRY_MODELS_ENDPOINT,
FOUNDRY_MODELS_API_KEY, FOUNDRY_EMBEDDING_MODEL, and
FOUNDRY_IMAGE_EMBEDDING_MODEL. Once the repo variables/secrets
are configured, the embedding integration test will run in CI.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix File column showing class name instead of module name
When a test is inside a class, pytest writes the classname as e.g.
'pkg.test_file.TestClass'. The previous rsplit logic extracted
'TestClass' instead of 'test_file'. Now detect uppercase-starting
segments as class names and use the preceding segment instead.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR review: UTC timestamps, XML error handling, summary fix, docstring
- Use datetime.now(timezone.utc) for accurate UTC timestamps
- Catch ET.ParseError per-file so corrupt XML doesn't crash the report
- Remove separate 'error' key from summary (errors folded into 'failed')
- Fix _short_name docstring to show actual dotted classname::name format
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The misc-integration job (Anthropic, Ollama, MCP) frequently fails on merge to main when the upstream MCP server (e.g. learn.microsoft.com/api/mcp) returns a transient rate-limit error. The previous 5s retry delay is too short to ride out the upstream backoff window, so all retries fail and the merge queue is blocked. Bumping to 30s gives the upstream a chance to recover before pytest-retry re-runs the test.
* Fix reasoning text done events duplicating streamed delta content (#5157)
The OpenAI Responses API sends both reasoning_text.delta (incremental
chunks) and reasoning_text.done (full accumulated text) events. The
chat client was emitting Content for both, causing ag-ui to append the
full done text onto already-accumulated delta text, producing
duplicated reasoning output.
Stop emitting Content for reasoning_text.done and
reasoning_summary_text.done events, matching how output_text.done is
already handled (not emitted). The deltas contain all the content;
the done event is redundant.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix(openai): emit reasoning done content as fallback when no deltas observed (#5157)
Address PR review feedback:
- Track item_ids that received reasoning deltas via seen_reasoning_delta_item_ids set
- Emit content from done events only when no deltas were received for the
item_id, preventing silent content loss on stream resumption
- Add comment documenting code_interpreter done event asymmetry
- Replace redundant ag-ui test with deduplication-focused test
- Add integration test for delta+done sequence in OpenAI chat client tests
- Add fallback path tests for done events without preceding deltas
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback for #5157: Python: [Bug]: "type": "response.reasoning_text.delta" and "response.reasoning_text.done" both get exposed as "text_reasoning"
* Fix AG-UI reasoning streaming to use proper Start/End pattern (#5157)
_emit_text_reasoning now follows the same streaming pattern as _emit_text:
- Emits ReasoningStartEvent/ReasoningMessageStartEvent only on the first
delta for a given message_id
- Emits only ReasoningMessageContentEvent for subsequent deltas
- Defers ReasoningMessageEndEvent/ReasoningEndEvent until
_close_reasoning_block is called (on content type switch or end-of-run)
This produces the correct protocol pattern:
ReasoningStartEvent
ReasoningMessageStartEvent
ReasoningMessageContentEvent(delta1)
ReasoningMessageContentEvent(delta2)
ReasoningMessageEndEvent
ReasoningEndEvent
Instead of wrapping every delta in a full Start→End sequence.
Backward compatibility is preserved: calling _emit_text_reasoning without
a flow argument still produces the full sequence per call.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix import ordering lint error in AG-UI test file (#5157)
Move inline import of TextMessageContentEvent to the top-level import
block and ensure alphabetical ordering to satisfy ruff I001 rule.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix mypy error: rename loop variable to avoid type conflict with WorkflowEvent
The 'event' variable was already typed as WorkflowEvent[Any] from the
async for loop at line 590. Reusing it in the _close_reasoning_block
loop (which returns list[BaseEvent]) caused an incompatible assignment
error. Renamed to 'reasoning_evt' to avoid the conflict.
Fixes#5162
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback for #5157: review comment fixes
* narrow test result reporting to explicit pytest JUnit XML
* Fix test args
* Fix pytest-results-action in merge workflow and remove committed test artifacts
Apply the same JUnit XML fix from python-tests.yml to python-merge-tests.yml:
add --junitxml=pytest.xml to all test commands and narrow the results action
path from ./python/**.xml to ./python/pytest.xml. Also remove accidentally
committed pytest.xml and python-coverage.xml and add them to .gitignore.
---------
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add github actions workflow for verify-samples
* Make workflow run as part of PR (for now)
* Update workflow to remove pr trigger
* Address PR comments
* Refactor Anthropic model option and provider clients
Rename the Anthropic client model option from model_id to model, add provider-specific Anthropic wrappers for Foundry, Bedrock, and Vertex, and expose them through the Anthropic, Foundry, Amazon, and Google namespaces. Update core option handling, docs, samples, and tests accordingly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix Anthropic skills sample typing
Cast the Anthropic beta client to Any in the skills sample so the pre-commit sample pyright check no longer fails on beta skills and files endpoints that are not exposed by the current SDK stubs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* undo sample mypy
* Retry CI after transient external failures
Retrigger PR validation after an unrelated Copilot review workflow SAML failure and a transient external tau2 git fetch failure in the Windows Python test setup.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback on model option merging
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address Anthropic compatibility review feedback
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* moved all to `model`
* fixes for azure ai search
* Python: standardize remaining sample env var names
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Python: fix foundry-local pyright compatibility
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* updated env vars in cicd
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Eduard van Valkenburg
·
2026-04-01 19:00:18 +00:00
* Add script to ping on stale issues/PRs
* Add script to ping on stale issues/PRs
* Fix stale issue/PR ping script review comments
- Rename TEAM_NAME env var to TEAM_SLUG for clarity
- Add actionable error messages for 403/404 team lookup failures
- Add contents:read permission for actions/checkout
- Use github.event.inputs context with fallback for scheduled runs
- Pin PyGithub to 2.6.0 for reproducible builds
- Fetch comments once in should_ping() to reduce API calls
- Make ping() retry loop idempotent (track comment/label state)
- Validate DAYS_THRESHOLD with helpful error for non-numeric input
- Fix timezone bug: use astimezone() instead of replace(tzinfo=)
- Add comprehensive unit tests (29 tests)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* updated automation tasks and commands, with alias for the time being
* Restore aggregate test exclusions
Preserve the legacy all-tests scope for test --all by excluding lab and devui from the default aggregate sweep, while still allowing explicit package selection. Also ignore hidden/generated test directories such as .mypy_cache during aggregate discovery.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* updated versions in pre-commit
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Eduard van Valkenburg
·
2026-03-18 18:39:11 +00:00
Eduard van Valkenburg ·
Giles Odigwe ·
Copilot ·
Ben Thomas ·
Evan Mattson ·
Roger Barreto ·
Tao Chen ·
westey ·
dependabot[bot] ·