2 Commits

• .NET: Python: Add dotnet integration test report to CI (#5515)

  * Add dotnet integration test report to CI
  
  - Add --report-junit flag to dotnet integration test step to generate
    JUnit XML alongside TRX, with explicit --results-directory to
    centralize output in IntegrationTestResults/
  - Upload JUnit XML artifacts from each matrix leg (net10.0/ubuntu,
    net472/windows) as dotnet-test-results-{framework}-{os}
  - Add dotnet-integration-test-report job that downloads artifacts,
    runs the existing aggregate.py script, posts markdown to Job Summary,
    and saves trend history via actions/cache
  - Refactor aggregate.py to discover JUnit XML files recursively,
    supporting both pytest (pytest.xml) and xunit (*.junit.xml) layouts
  - Handle provider name derivation for dotnet artifact naming convention
  - Fix nodeid collision when same test runs under multiple frameworks
    by qualifying keys with provider when collisions are detected
  - Improve module extraction for dotnet C# classnames (recognizes
    IntegrationTests/UnitTests namespace segments)
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * chore: trigger dotnet CI for report validation
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * fix: use .junit extension (not .junit.xml) for xunit v3 output
  
  xUnit v3 generates files with .junit extension, not .junit.xml.
  Update upload glob and aggregate.py discovery to match.
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * fix: use deterministic provider-qualified keys for dotnet tests
  
  Always prefix dotnet test keys with provider (e.g. net10.0 (ubuntu)::TestName)
  to ensure stable, comparable counts across runs regardless of file parse order.
  Also show Executed (passed+failed) instead of Total in summary table.
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * fix: match Python report summary format (Total, passed/total, etc.)
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * feat: split dotnet report into per-framework tables
  
  Dotnet tests run on multiple frameworks (net10.0, net472). Instead of
  one combined table with unstable totals, show separate sections per
  framework — each with its own summary row and per-test table. Python
  reports retain the original single-table format.
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * Re-enable 7 flaky dotnet integration tests with increased timeouts
  
  Increase timeouts to reduce timing-related flakiness in LLM-backed
  integration tests (issue #4971):
  
  - ExternalClientTests: 60s -> 120s default timeout
  - SamplesValidationBase: 60s -> 120s default timeout
  - ConsoleAppSamplesValidation: 90s -> 150s for long-running tests
  - AzureFunctions SamplesValidation: 2min -> 3min orchestration timeout,
    60s -> 90s per-step WaitForConditionAsync timeouts
  
  Remove all Skip=Flaky annotations and unused SkipFlakyTimingTest constants.
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * Re-skip LLM non-determinism flaky tests, keep timeout fixes
  
  Re-skip SingleAgentOrchestrationHITLSampleValidationAsync and
  LongRunningToolsSampleValidationAsync - these fail due to LLM producing
  extra review notifications, not timeouts. Updated skip reasons to
  accurately describe the root cause. Reverted unnecessary timeout change
  on the skipped LongRunningTools test.
  
  The remaining 5 re-enabled tests with timeout increases are stable.
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * Enable Anthropic integration tests in CI
  
  Replace hardcoded skip with conditional skip pattern (matching
  CopilotStudio approach): tests gracefully skip when ANTHROPIC_API_KEY
  is missing, and run when present.
  
  Changes:
  - AnthropicChatCompletionFixture: try/catch in InitializeAsync with
    Assert.Skip on missing config (replaces hardcoded SkipReason)
  - AnthropicSkillsIntegrationTests: same pattern per test method
  - dotnet-build-and-test.yml: wire up ANTHROPIC_API_KEY,
    ANTHROPIC_CHAT_MODEL_NAME, and ANTHROPIC_REASONING_MODEL_NAME
    env vars to the integration test step
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * Fix missing System using in AnthropicSkillsIntegrationTests
  
  Add 'using System;' for InvalidOperationException in try/catch blocks.
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * Skip flaky SingleAgentOrchestrationChainingSampleValidationAsync
  
  LLM non-determinism causes Assert.NotNull failures on orchestration
  results. Skip until test logic is hardened against non-deterministic
  LLM responses.
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * Re-enable HITL and LongRunningTools tests with timeout and flexibility fixes
  
  - Remove Skip attribute from SingleAgentOrchestrationHITLSampleValidationAsync
  - Remove Skip attribute from LongRunningToolsSampleValidationAsync
  - Increase timeout from 120s/90s to 180s to accommodate 2+ LLM round-trips
  - Replace rigid 2-cycle assertion with flexible approval logic that handles
    extra review cycles from LLM non-determinism
  
  Fixes the two failure modes identified in #4971:
  1. Timeout: 120s/90s was insufficient for multiple LLM calls under CI load
  2. Extra notifications: Assert.Fail on 3rd+ review cycle was too rigid
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * Increase AzureFunctions LongRunningTools test timeouts from 90s to 180s
  
  The LongRunningToolsSampleValidationAsync test in the AzureFunctions integration
  tests was failing in CI with TimeoutException at the 'Content published
  notification is logged' step. The 90-second timeouts are too tight for CI
  environments where LLM calls and orchestration overhead can be slow.
  
  Increased all three WaitForConditionAsync timeouts from 90s to 180s:
  - Waiting for human feedback notification
  - Waiting for publish notification (the step that was failing)
  - Waiting for orchestration completion
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * Merge main and fix dotnet report path after flaky_report rename
  
  Merge upstream/main which renamed scripts/flaky_report/ to
  scripts/integration_test_report/ (from Python PR #5454). Update the
  dotnet-build-and-test workflow to reference the new path.
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * Add RetryFact to DurableTask and AzureFunctions integration tests
  
  These tests interact with LLMs via stdin/stdout (DurableTask) or HTTP
  (AzureFunctions) and are inherently non-deterministic. Unlike the Python
  side which uses pytest-retry, the dotnet tests had no retry mechanism
  and a single transient failure would fail the entire CI run.
  
  Changes:
  - Switch [Fact] to [RetryFact(2, 5000)] on all LLM-dependent tests
    across ConsoleAppSamplesValidation, ExternalClientTests,
    WorkflowConsoleAppSamplesValidation, and AzureFunctions SamplesValidation
  - Add re-prompt mechanism to LongRunningToolsSampleValidationAsync:
    if the LLM doesn't invoke the tool within 60s, re-send the prompt
    (up to 2 retries) instead of burning the full timeout
  - Reduce LongRunningTools timeout from 240s to 180s (re-prompt makes
    the extra buffer unnecessary)
  - Leave simple/deterministic tests as [Fact] (SingleAgent, unit tests)
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * Add persist-credentials: false to Integration Test Report checkout step
  
  Matches the convention used by other checkout steps in this workflow
  to avoid leaving GITHUB_TOKEN credentials in the local git config.
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * small fixes
  
  * disable anthropic failing tests
  
  ---------
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

  Giles Odigwe · 2026-05-07 20:39:32 +00:00
  c06af9a1b3
• Python: Reduce flaky integration tests and improve CI signal quality (#5454)

  * Enable Ollama integration tests in CI and rename report to Integration Test Report
  
  - Install Ollama, cache models (qwen2.5:0.5b + nomic-embed-text), and start
    server in the Misc integration job for both workflow files
  - Set OLLAMA_MODEL and OLLAMA_EMBEDDING_MODEL env vars so the 5 Ollama tests
    are no longer skipped
  - Rename Flaky Test Report to Integration Test Report throughout (job names,
    artifact names, cache keys, file names, script titles/docstrings)
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * Bump Ollama model to qwen2.5:1.5b for better instruction following
  
  The 0.5b model was too small to reliably follow simple prompts like
  'Say Hello World', causing test assertion failures. The 1.5b model
  follows instructions more reliably while still being small enough
  for fast CI pulls (~1GB).
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * Re-enable reliable streaming integration tests
  
  Remove the hard skip on test_03_reliable_streaming tests that was
  temporarily disabled for instability investigation. CI infrastructure
  (Azurite, DTS emulator, Redis, func CLI) is already in place.
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * Re-enable skipped Functions/DurableTask tests and bump timeout to 480s
  
  - Remove hard skips from 4 tests in test_11_workflow_parallel.py
  - Remove hard skip from test_conditional_branching in test_06_dt_multi_agent_orchestration_conditionals.py
  - Increase pytest --timeout from 360 to 480 for Functions+DurableTask CI job
  - Updated in both python-merge-tests.yml and python-integration-tests.yml
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * Re-skip failing Functions/DurableTask tests with specific root causes
  
  - test_11_workflow_parallel (4 tests): xdist worker crashes during execution
  - test_conditional_branching: orchestration fails with RuntimeError, not a timeout
  - Keep 480s timeout bump for remaining Functions tests
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * Fix auth routing in samples 06/11: api_key -> credential for Azure OpenAI
  
  Both samples passed a bearer token provider via api_key= which caused the
  client to route to api.openai.com instead of Azure OpenAI, resulting in
  401 Unauthorized. Changed to credential= which correctly triggers Azure
  routing and picks up AZURE_OPENAI_ENDPOINT from the environment.
  
  - samples/azure_functions/11_workflow_parallel/function_app.py: 1 fix
  - samples/durabletask/06_multi_agent_orchestration_conditionals/worker.py: 2 fixes
  - Re-enable 4 parallel workflow tests and 1 conditional branching test
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * Re-skip parallel workflow tests: xdist worker distribution issue
  
  The 4 parallel workflow tests crash because xdist worksteal distributes
  them across separate workers, each spawning its own func process against
  shared emulators. Auth fix (api_key->credential) was valid and stays.
  test_conditional_branching now passes with the auth fix.
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * Fix E501 line-too-long in azurefunctions parallel test skip reasons
  
  Wrap skip reason strings to stay within 120 char line limit.
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * Add retry logic and port-conflict fix for Ollama CI setup
  
  - Kill any auto-started Ollama before launching serve (fixes port
    conflict: 'address already in use')
  - Retry ollama pull up to 3 times with 15s backoff (fixes 429 rate
    limit failures)
  - Applied to both python-merge-tests.yml and python-integration-tests.yml
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * Fix flaky integration tests and re-enable skipped tests
  
  - Foundry agent: add allow_preview=True to custom client test
  - Foundry hosting: raise max_output_tokens 50->200, add temperature,
    relax assertion in test_temperature_and_max_tokens
  - Foundry embedding: update skip reason with root cause (endpoint mismatch)
  - OpenAI file search: fix vector store indexing race condition by polling
    file_counts before querying; fix get_streaming_response -> get_response(stream=True)
  - Azure OpenAI file search: remove skip (transient 500 resolved)
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * Remove temperature from foundry hosting test (unsupported by CI model)
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * Stabilize Ollama tool call integration tests with no-arg function
  
  Use a no-argument greet() function instead of hello_world(arg1) for
  integration tests. The 1.5B model in CI is unreliable at generating
  correct tool call arguments, causing 'Argument parsing failed' errors.
  A no-arg function eliminates this flakiness entirely.
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * Increase reliable streaming test timeouts from 30s to 60s
  
  The LLM call through Azure OpenAI + Redis streaming pipeline can exceed
  30s in CI due to cold starts or throttling. Raise to 60s to reduce
  flaky timeouts while still bounded by pytest's 120s per-test limit.
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * Re-enable workflow parallel tests with xdist_group marker
  
  The tests were skipped because xdist distributes module tests across
  workers, each spawning their own func process (port conflicts). Adding
  xdist_group forces all tests in this module onto a single worker so
  the module-scoped function_app_for_test fixture works correctly.
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * Revert "Re-enable workflow parallel tests with xdist_group marker"
  
  This reverts commit 455c28da62.
  
  * Rename flaky_report to integration_test_report and add try/finally cleanup
  
  - Rename scripts/flaky_report/ to scripts/integration_test_report/ to
    reflect expanded scope beyond flaky-test detection
  - Update workflow references in both CI files
  - Wrap file search integration tests in try/finally to ensure vector
    store cleanup runs even on test failure or timeout
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * Fix Ollama pull failure propagation and Azure OpenAI vector store readiness
  
  - Ollama CI: fail the step immediately if model pull fails after 3
    retries instead of silently proceeding to tests
  - Azure OpenAI file search: add the same vector-store readiness polling
    that was applied to the non-Azure OpenAI tests, preventing eventual
    consistency race conditions
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * remove load_dotenv from test file
  
  ---------
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

  Giles Odigwe · 2026-05-01 00:41:39 +00:00
  540193ccef