Bump actions/cache from 4 to 5

Bumps [actions/cache](https://github.com/actions/cache) from 4 to 5. - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](https://github.com/actions/cache/compare/v4...v5) --- updated-dependencies: - dependency-name: actions/cache dependency-version: '5' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
Python: Flaky test report (#5342 )
2026-06-16 21:04:09 +08:00 · 2026-04-22 20:20:51 +00:00 · 2026-04-22 20:16:50 +00:00 · 2026-04-22 17:48:02 +00:00 · 2026-04-22 17:45:25 +00:00 · 2026-04-22 17:44:41 +00:00
22 changed files with 1222 additions and 27 deletions
@@ -0,0 +1,161 @@
+name: DevFlow PR Review
+
+on:
+  pull_request_target:
+    types:
+      - opened
+      - reopened
+      - ready_for_review
+  workflow_dispatch:
+    inputs:
+      pr_number:
+        description: Pull request number to review
+        required: true
+        type: string
+
+permissions:
+  contents: read
+  issues: write
+  pull-requests: write
+
+concurrency:
+  group: devflow-pr-review-${{ github.repository }}-${{ github.event.pull_request.number || inputs.pr_number || github.run_id }}
+  cancel-in-progress: true
+
+env:
+  DEVFLOW_REPOSITORY: ${{ vars.DF_REPO }}
+  DEVFLOW_REF: main
+  TARGET_REPO_PATH: ${{ github.workspace }}/target-repo
+  DEVFLOW_PATH: ${{ github.workspace }}/devflow
+
+jobs:
+  team_check:
+    runs-on: ubuntu-latest
+    outputs:
+      is_team_member: ${{ steps.check.outputs.is_team_member }}
+      pr_number: ${{ steps.pr.outputs.pr_number }}
+      pr_url: ${{ steps.pr.outputs.pr_url }}
+      repo: ${{ steps.pr.outputs.repo }}
+    steps:
+      - name: Resolve PR metadata
+        id: pr
+        shell: bash
+        env:
+          PR_HTML_URL: ${{ github.event.pull_request.html_url }}
+          PR_NUMBER_EVENT: ${{ github.event.pull_request.number }}
+          PR_NUMBER_INPUT: ${{ inputs.pr_number }}
+        run: |
+          set -euo pipefail
+
+          if [[ "${GITHUB_EVENT_NAME}" == "pull_request_target" ]]; then
+            pr_number="${PR_NUMBER_EVENT}"
+            pr_url="${PR_HTML_URL}"
+          else
+            pr_number="${PR_NUMBER_INPUT}"
+            pr_url="https://github.com/${GITHUB_REPOSITORY}/pull/${pr_number}"
+          fi
+
+          if [[ ! "$pr_number" =~ ^[1-9][0-9]*$ ]]; then
+            echo "Could not determine PR number; for workflow_dispatch runs, the 'pr_number' input is required when not running on pull_request_target." >&2
+            exit 1
+          fi
+
+          echo "pr_url=${pr_url}" >> "$GITHUB_OUTPUT"
+          echo "pr_number=${pr_number}" >> "$GITHUB_OUTPUT"
+          echo "repo=${GITHUB_REPOSITORY}" >> "$GITHUB_OUTPUT"
+
+      - name: Check PR author team membership
+        id: check
+        uses: actions/github-script@v8
+        env:
+          TEAM_NAME: ${{ secrets.DEVELOPER_TEAM }}
+          PR_NUMBER: ${{ steps.pr.outputs.pr_number }}
+        with:
+          github-token: ${{ secrets.GH_ACTIONS_PR_WRITE }}
+          script: |
+            let author = context.payload.pull_request?.user?.login;
+            if (!author) {
+              const { data: pr } = await github.rest.pulls.get({
+                owner: context.repo.owner,
+                repo: context.repo.repo,
+                pull_number: Number(process.env.PR_NUMBER),
+              });
+              author = pr.user.login;
+            }
+
+            let isTeamMember = false;
+            try {
+              const teamMembership = await github.rest.teams.getMembershipForUserInOrg({
+                org: context.repo.owner,
+                team_slug: process.env.TEAM_NAME,
+                username: author,
+              });
+              isTeamMember = teamMembership.data.state === 'active';
+            } catch (error) {
+              console.log(`Team membership lookup failed for ${author}: ${error.message}`);
+              isTeamMember = false;
+            }
+
+            core.setOutput('is_team_member', isTeamMember ? 'true' : 'false');
+            if (isTeamMember) {
+              core.info(`Author ${author} is a team member; proceeding with review.`);
+            } else {
+              core.info(`Author ${author} is not a member of ${process.env.TEAM_NAME}; skipping review.`);
+            }
+
+  review:
+    runs-on: ubuntu-latest
+    needs: team_check
+    if: ${{ needs.team_check.outputs.is_team_member == 'true' }}
+    timeout-minutes: 60
+
+    steps:
+      # Safe checkout: base repo only, not the untrusted PR head.
+      - name: Checkout target repo base
+        uses: actions/checkout@v6
+        with:
+          ref: ${{ github.event_name == 'pull_request_target' && github.event.pull_request.base.sha || github.sha }}
+          fetch-depth: 0
+          persist-credentials: false
+          path: target-repo
+
+      # Private DevFlow checkout: the PAT/token grants access to this repo's code.
+      - name: Checkout DevFlow
+        uses: actions/checkout@v6
+        with:
+          repository: ${{ env.DEVFLOW_REPOSITORY }}
+          ref: ${{ env.DEVFLOW_REF }}
+          token: ${{ secrets.DEVFLOW_TOKEN }}
+          fetch-depth: 1
+          persist-credentials: false
+          path: devflow
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+
+      - name: Set up uv
+        uses: astral-sh/setup-uv@v7
+        with:
+          version: "0.11.x"
+          enable-cache: true
+
+      - name: Install DevFlow dependencies
+        working-directory: ${{ env.DEVFLOW_PATH }}
+        run: uv sync --frozen
+
+      - name: Run PR review
+        id: review
+        working-directory: ${{ env.DEVFLOW_PATH }}
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          GH_COPILOT_TOKEN: ${{ secrets.GH_COPILOT_TOKEN }}
+          SK_REPO_PATH: ${{ env.TARGET_REPO_PATH }}
+          AGENT_REPO_PATH: ${{ env.TARGET_REPO_PATH }}
+          PR_URL: ${{ needs.team_check.outputs.pr_url }}
+        run: |
+          uv run python scripts/trigger_pr_review.py \
+            --pr-url "$PR_URL" \
+            --github-username "$GITHUB_ACTOR" \
+            --no-require-comment-selection
@@ -87,6 +87,14 @@ jobs:
          -n logical --dist worksteal
          --timeout=120 --session-timeout=900 --timeout_method thread
          --retries 2 --retry-delay 5
+          --junitxml=pytest.xml
+      - name: Upload test results
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: test-results-openai
+          path: ./python/pytest.xml
+          if-no-files-found: ignore

  # Azure OpenAI integration tests
  python-tests-azure-openai:
@@ -130,6 +138,14 @@ jobs:
          -n logical --dist worksteal
          --timeout=120 --session-timeout=900 --timeout_method thread
          --retries 2 --retry-delay 5
+          --junitxml=pytest.xml
+      - name: Upload test results
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: test-results-azure-openai
+          path: ./python/pytest.xml
+          if-no-files-found: ignore

  # Misc integration tests (Anthropic, Hyperlight, Ollama, MCP)
  python-tests-misc-integration:
@@ -173,6 +189,14 @@ jobs:
          -n logical --dist worksteal
          --timeout=120 --session-timeout=900 --timeout_method thread
          --retries 2 --retry-delay 30
+          --junitxml=pytest.xml
+      - name: Upload test results
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: test-results-misc
+          path: ./python/pytest.xml
+          if-no-files-found: ignore
      - name: Stop local MCP server
        if: always()
        shell: bash
@@ -249,6 +273,14 @@ jobs:
          -x
          --timeout=360 --session-timeout=900 --timeout_method thread
          --retries 2 --retry-delay 5
+          --junitxml=pytest.xml
+      - name: Upload test results
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: test-results-functions
+          path: ./python/pytest.xml
+          if-no-files-found: ignore

  # Foundry integration tests
  python-tests-foundry:
@@ -295,6 +327,14 @@ jobs:
          -n logical --dist worksteal
          --timeout=120 --session-timeout=900 --timeout_method thread
          --retries 2 --retry-delay 5
+          --junitxml=pytest.xml
+      - name: Upload test results
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: test-results-foundry
+          path: ./python/pytest.xml
+          if-no-files-found: ignore

  # Azure Cosmos integration tests
  python-tests-cosmos:
@@ -339,7 +379,80 @@ jobs:
          echo "Cosmos DB emulator did not become ready in time." >&2
          exit 1
      - name: Test with pytest (Cosmos integration)
-        run: uv run --directory packages/azure-cosmos poe integration-tests -n logical --dist worksteal --timeout=120 --session-timeout=900 --timeout_method thread --retries 2 --retry-delay 5
+        run: uv run --directory packages/azure-cosmos poe integration-tests -n logical --dist worksteal --timeout=120 --session-timeout=900 --timeout_method thread --retries 2 --retry-delay 5 --junitxml=${{ github.workspace }}/python/pytest.xml
+      - name: Upload test results
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: test-results-cosmos
+          path: ./python/pytest.xml
+          if-no-files-found: ignore
+
+  # Flaky test trend report (aggregates per-job JUnit XML results)
+  python-flaky-test-report:
+    name: Flaky Test Report
+    if: >
+      always() &&
+      (contains(join(needs.*.result, ','), 'success') ||
+       contains(join(needs.*.result, ','), 'failure'))
+    needs:
+      [
+        python-tests-openai,
+        python-tests-azure-openai,
+        python-tests-misc-integration,
+        python-tests-functions,
+        python-tests-foundry,
+        python-tests-cosmos,
+      ]
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: python
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          ref: ${{ inputs.checkout-ref }}
+          persist-credentials: false
+      - name: Set up python and install the project
+        uses: ./.github/actions/python-setup
+        with:
+          python-version: ${{ env.UV_PYTHON }}
+          os: ${{ runner.os }}
+      - name: Download all test results from current run
+        uses: actions/download-artifact@v4
+        with:
+          pattern: test-results-*
+          path: test-results/
+      - name: Restore flaky report history cache
+        uses: actions/cache/restore@v5
+        with:
+          path: python/flaky-report-history.json
+          key: flaky-report-history-integration-${{ github.run_id }}
+          restore-keys: |
+            flaky-report-history-integration-
+      - name: Generate trend report
+        run: >
+          uv run python scripts/flaky_report/aggregate.py
+          ../test-results/
+          flaky-report-history.json
+          flaky-test-report.md
+      - name: Post to Job Summary
+        if: always()
+        run: cat flaky-test-report.md >> $GITHUB_STEP_SUMMARY
+      - name: Save flaky report history cache
+        if: always()
+        uses: actions/cache/save@v5
+        with:
+          path: python/flaky-report-history.json
+          key: flaky-report-history-integration-${{ github.run_id }}
+      - name: Upload unified trend report
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: flaky-test-report
+          path: |
+            python/flaky-test-report.md
+            python/flaky-report-history.json

  python-integration-tests-check:
    if: always()
@@ -181,6 +181,13 @@ jobs:
          display-options: fEX
          fail-on-empty: false
          title: OpenAI integration test results
+      - name: Upload test results
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: test-results-openai
+          path: ./python/pytest.xml
+          if-no-files-found: ignore

  # Azure OpenAI integration tests
  python-tests-azure-openai:
@@ -244,6 +251,13 @@ jobs:
          display-options: fEX
          fail-on-empty: false
          title: Azure OpenAI integration test results
+      - name: Upload test results
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: test-results-azure-openai
+          path: ./python/pytest.xml
+          if-no-files-found: ignore

  # Misc integration tests (Anthropic, Ollama, MCP)
  python-tests-misc-integration:
@@ -321,6 +335,13 @@ jobs:
          display-options: fEX
          fail-on-empty: false
          title: Misc integration test results
+      - name: Upload test results
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: test-results-misc
+          path: ./python/pytest.xml
+          if-no-files-found: ignore

  # Azure Functions + Durable Task integration tests
  python-tests-functions:
@@ -392,6 +413,13 @@ jobs:
          display-options: fEX
          fail-on-empty: false
          title: Functions integration test results
+      - name: Upload test results
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: test-results-functions
+          path: ./python/pytest.xml
+          if-no-files-found: ignore

  python-tests-foundry:
    name: Python Integration Tests - Foundry
@@ -409,6 +437,10 @@ jobs:
      FOUNDRY_MODEL: ${{ vars.FOUNDRY_MODEL }}
      FOUNDRY_AGENT_NAME: ${{ vars.FOUNDRY_AGENT_NAME }}
      FOUNDRY_AGENT_VERSION: ${{ vars.FOUNDRY_AGENT_VERSION }}
+      FOUNDRY_MODELS_ENDPOINT: ${{ vars.FOUNDRY_MODELS_ENDPOINT || '' }}
+      FOUNDRY_MODELS_API_KEY: ${{ secrets.FOUNDRY_MODELS_API_KEY || '' }}
+      FOUNDRY_EMBEDDING_MODEL: ${{ vars.FOUNDRY_EMBEDDING_MODEL || '' }}
+      FOUNDRY_IMAGE_EMBEDDING_MODEL: ${{ vars.FOUNDRY_IMAGE_EMBEDDING_MODEL || '' }}
      LOCAL_MCP_URL: ${{ vars.LOCAL_MCP__URL }}
    defaults:
      run:
@@ -448,6 +480,13 @@ jobs:
          display-options: fEX
          fail-on-empty: false
          title: Test results
+      - name: Upload test results
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: test-results-foundry
+          path: ./python/pytest.xml
+          if-no-files-found: ignore

  # TODO: Add python-tests-lab

@@ -497,7 +536,7 @@ jobs:
          echo "Cosmos DB emulator did not become ready in time." >&2
          exit 1
      - name: Test with pytest (Cosmos integration)
-        run: uv run --directory packages/azure-cosmos poe integration-tests -n logical --dist worksteal --timeout=120 --session-timeout=900 --timeout_method thread --retries 2 --retry-delay 5 --junitxml=pytest.xml
+        run: uv run --directory packages/azure-cosmos poe integration-tests -n logical --dist worksteal --timeout=120 --session-timeout=900 --timeout_method thread --retries 2 --retry-delay 5 --junitxml=${{ github.workspace }}/python/pytest.xml
        working-directory: ./python
      - name: Surface failing tests
        if: always()
@@ -508,6 +547,76 @@ jobs:
          display-options: fEX
          fail-on-empty: false
          title: Cosmos integration test results
+      - name: Upload test results
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: test-results-cosmos
+          path: ./python/pytest.xml
+          if-no-files-found: ignore
+
+  # Flaky test trend report (aggregates per-job JUnit XML results)
+  python-flaky-test-report:
+    name: Flaky Test Report
+    if: >
+      always() &&
+      (contains(join(needs.*.result, ','), 'success') ||
+       contains(join(needs.*.result, ','), 'failure'))
+    needs:
+      [
+        python-tests-openai,
+        python-tests-azure-openai,
+        python-tests-misc-integration,
+        python-tests-functions,
+        python-tests-foundry,
+        python-tests-cosmos,
+      ]
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: python
+    steps:
+      - uses: actions/checkout@v6
+      - name: Set up python and install the project
+        uses: ./.github/actions/python-setup
+        with:
+          python-version: ${{ env.UV_PYTHON }}
+          os: ${{ runner.os }}
+      - name: Download all test results from current run
+        uses: actions/download-artifact@v4
+        with:
+          pattern: test-results-*
+          path: test-results/
+      - name: Restore flaky report history cache
+        uses: actions/cache/restore@v5
+        with:
+          path: python/flaky-report-history.json
+          key: flaky-report-history-merge-${{ github.run_id }}
+          restore-keys: |
+            flaky-report-history-merge-
+      - name: Generate trend report
+        run: >
+          uv run python scripts/flaky_report/aggregate.py
+          ../test-results/
+          flaky-report-history.json
+          flaky-test-report.md
+      - name: Post to Job Summary
+        if: always()
+        run: cat flaky-test-report.md >> $GITHUB_STEP_SUMMARY
+      - name: Save flaky report history cache
+        if: always()
+        uses: actions/cache/save@v5
+        with:
+          path: python/flaky-report-history.json
+          key: flaky-report-history-merge-${{ github.run_id }}
+      - name: Upload unified trend report
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: flaky-test-report
+          path: |
+            python/flaky-test-report.md
+            python/flaky-report-history.json

  python-integration-tests-check:
    if: always()
@@ -701,7 +701,7 @@ jobs:

      - name: Restore validation history
        id: cache-restore
-        uses: actions/cache/restore@v4
+        uses: actions/cache/restore@v5
        with:
          path: validation-history/
          key: validation-history-${{ github.run_id }}
@@ -719,7 +719,7 @@ jobs:
        run: cat trend-report.md >> "$GITHUB_STEP_SUMMARY"

      - name: Save validation history
-        uses: actions/cache/save@v4
+        uses: actions/cache/save@v5
        with:
          path: validation-history/
          key: validation-history-${{ github.run_id }}
@@ -295,7 +295,10 @@ class A2AAgent(AgentTelemetryLayer, BaseAgent):
        else:
            if not normalized_messages:
                raise ValueError("At least one message is required when starting a new task (no continuation_token).")
-            a2a_message = self._prepare_message_for_a2a(normalized_messages[-1])
+            a2a_message = self._prepare_message_for_a2a(
+                normalized_messages[-1],
+                context_id=session.service_session_id if session else None,
+            )
            a2a_stream = self.client.send_message(a2a_message)

        provider_session = session
@@ -584,7 +587,7 @@ class A2AAgent(AgentTelemetryLayer, BaseAgent):
            return AgentResponse.from_updates(updates)
        return AgentResponse(messages=[], response_id=task.id, raw_representation=task)

-    def _prepare_message_for_a2a(self, message: Message) -> A2AMessage:
+    def _prepare_message_for_a2a(self, message: Message, *, context_id: str | None = None) -> A2AMessage:
        """Prepare a Message for the A2A protocol.

        Transforms Agent Framework Message objects into A2A protocol Messages by:
@@ -593,6 +596,13 @@ class A2AAgent(AgentTelemetryLayer, BaseAgent):
        - Converting file references (URI/data/hosted_file) to FilePart objects
        - Preserving metadata and additional properties from the original message
        - Setting the role to 'user' as framework messages are treated as user input
+
+        Args:
+            message: The framework Message to convert.
+            context_id: Optional fallback context identifier (e.g. derived from
+                ``AgentSession.service_session_id``). When the *message* already
+                carries a ``context_id`` in its ``additional_properties`` that
+                value takes precedence; otherwise this fallback is used.
        """
        parts: list[A2APart] = []
        if not message.contents:
@@ -672,7 +682,7 @@ class A2AAgent(AgentTelemetryLayer, BaseAgent):
            role=A2ARole("user"),
            parts=parts,
            message_id=message.message_id or uuid.uuid4().hex,
-            context_id=message.additional_properties.get("context_id"),
+            context_id=message.additional_properties.get("context_id") or context_id,
            metadata=metadata,
        )

@@ -46,6 +46,7 @@ class MockA2AClient:
        self.responses: list[Any] = []
        self.resubscribe_responses: list[Any] = []
        self.get_task_response: Task | None = None
+        self.last_message: Any = None

    def add_message_response(self, message_id: str, text: str, role: str = "agent") -> None:
        """Add a mock Message response."""
@@ -111,6 +112,7 @@ class MockA2AClient:

    async def send_message(self, message: Any) -> AsyncIterator[Any]:
        """Mock send_message method that yields responses."""
+        self.last_message = message
        self.call_count += 1

        # All queued responses are delivered as a single streaming batch per call.
@@ -539,6 +541,37 @@ def test_prepare_message_for_a2a_forwards_context_id() -> None:
    assert result.metadata == {"trace_id": "trace-456"}


+def test_prepare_message_for_a2a_uses_fallback_context_id() -> None:
+    """Test that context_id kwarg is used when message has no context_id property."""
+
+    agent = A2AAgent(client=MagicMock(), http_client=None)
+
+    message = Message(
+        role="user",
+        contents=[Content.from_text(text="Hello")],
+    )
+
+    result = agent._prepare_message_for_a2a(message, context_id="session-ctx-1")
+
+    assert result.context_id == "session-ctx-1"
+
+
+def test_prepare_message_for_a2a_message_context_id_takes_precedence() -> None:
+    """Test that message.additional_properties context_id wins over the fallback."""
+
+    agent = A2AAgent(client=MagicMock(), http_client=None)
+
+    message = Message(
+        role="user",
+        contents=[Content.from_text(text="Hello")],
+        additional_properties={"context_id": "explicit-ctx"},
+    )
+
+    result = agent._prepare_message_for_a2a(message, context_id="session-ctx-1")
+
+    assert result.context_id == "explicit-ctx"
+
+
 def test_parse_contents_from_a2a_with_data_part() -> None:
    """Test conversion of A2A DataPart."""

@@ -868,6 +901,43 @@ async def test_poll_task_completed(a2a_agent: A2AAgent, mock_a2a_client: MockA2A
 # endregion


+# region Session context_id Integration Tests
+
+
+@mark.asyncio
+async def test_run_passes_session_service_session_id_as_context_id(mock_a2a_client: MockA2AClient) -> None:
+    """Test that run() wires session.service_session_id to the A2A message context_id."""
+    agent = A2AAgent(name="Test Agent", id="test-agent", client=mock_a2a_client, http_client=None)
+    mock_a2a_client.add_message_response("msg-ctx", "reply")
+
+    session = AgentSession(service_session_id="svc-session-42")
+    await agent.run("Hello", session=session)
+
+    assert mock_a2a_client.last_message is not None
+    assert mock_a2a_client.last_message.context_id == "svc-session-42"
+
+
+@mark.asyncio
+async def test_run_message_context_id_takes_precedence_over_session(mock_a2a_client: MockA2AClient) -> None:
+    """Test that an explicit context_id on the message wins over session.service_session_id."""
+    agent = A2AAgent(name="Test Agent", id="test-agent", client=mock_a2a_client, http_client=None)
+    mock_a2a_client.add_message_response("msg-ctx2", "reply")
+
+    session = AgentSession(service_session_id="svc-session-42")
+    message = Message(
+        role="user",
+        contents=[Content.from_text(text="Hello")],
+        additional_properties={"context_id": "explicit-ctx"},
+    )
+    await agent.run(messages=[message], session=session)
+
+    assert mock_a2a_client.last_message is not None
+    assert mock_a2a_client.last_message.context_id == "explicit-ctx"
+
+
+# endregion
+
+
 # region Context Provider Tests


@@ -790,9 +790,9 @@ async def run_agent_stream(
    # Create session (with service session support)
    if config.use_service_session:
        supplied_thread_id = input_data.get("thread_id") or input_data.get("threadId")
-        session = AgentSession(service_session_id=supplied_thread_id)
+        session = AgentSession(session_id=thread_id, service_session_id=supplied_thread_id)
    else:
-        session = AgentSession()
+        session = AgentSession(session_id=thread_id)

    # Inject metadata for AG-UI orchestration (Feature #2: Azure-safe truncation)
    base_metadata: dict[str, Any] = {
@@ -183,6 +183,7 @@ class StubAgent(SupportsAgentRun):
        self.client = client or SimpleNamespace(function_invocation_configuration=None)
        self.messages_received: list[Any] = []
        self.tools_received: list[Any] | None = None
+        self.last_session: AgentSession | None = None

    @overload
    def run(
@@ -216,6 +217,7 @@ class StubAgent(SupportsAgentRun):

            async def _stream() -> AsyncIterator[AgentResponseUpdate]:
                self.messages_received = [] if messages is None else list(messages)  # type: ignore[arg-type]
+                self.last_session = session
                self.tools_received = kwargs.get("tools")
                for update in self.updates:
                    yield update
@@ -1640,3 +1640,115 @@ class TestReasoningInSnapshot:
        # close: MsgEnd(block2) + End(block2)
        assert isinstance(close[0], ReasoningMessageEndEvent)
        assert close[0].message_id == "block2"
+
+
+async def test_session_id_matches_thread_id():
+    """Session created by run_agent_stream uses the client thread_id as session_id."""
+    from conftest import StubAgent
+
+    from agent_framework_ag_ui import AgentFrameworkAgent
+
+    stub = StubAgent()
+    agent = AgentFrameworkAgent(agent=stub)
+
+    payload = {
+        "thread_id": "my-thread-123",
+        "run_id": "run-1",
+        "messages": [{"role": "user", "content": "Hello"}],
+    }
+
+    _ = [event async for event in agent.run(payload)]
+
+    assert stub.last_session is not None
+    assert stub.last_session.session_id == "my-thread-123"
+
+
+async def test_session_id_matches_camel_case_thread_id():
+    """Session uses threadId (camelCase) as session_id when snake_case is absent."""
+    from conftest import StubAgent
+
+    from agent_framework_ag_ui import AgentFrameworkAgent
+
+    stub = StubAgent()
+    agent = AgentFrameworkAgent(agent=stub)
+
+    payload = {
+        "threadId": "camel-thread-456",
+        "run_id": "run-2",
+        "messages": [{"role": "user", "content": "Hello"}],
+    }
+
+    _ = [event async for event in agent.run(payload)]
+
+    assert stub.last_session is not None
+    assert stub.last_session.session_id == "camel-thread-456"
+
+
+async def test_session_id_matches_thread_id_with_service_session():
+    """Session uses thread_id as session_id even when use_service_session is enabled."""
+    from conftest import StubAgent
+
+    from agent_framework_ag_ui import AgentFrameworkAgent
+
+    stub = StubAgent()
+    agent = AgentFrameworkAgent(agent=stub, use_service_session=True)
+
+    payload = {
+        "thread_id": "service-thread-789",
+        "run_id": "run-3",
+        "messages": [{"role": "user", "content": "Hello"}],
+    }
+
+    _ = [event async for event in agent.run(payload)]
+
+    assert stub.last_session is not None
+    assert stub.last_session.session_id == "service-thread-789"
+    assert stub.last_session.service_session_id == "service-thread-789"
+
+
+async def test_session_id_generated_when_no_thread_id():
+    """Session gets a generated UUID as session_id when no thread_id is provided."""
+    import uuid
+
+    from conftest import StubAgent
+
+    from agent_framework_ag_ui import AgentFrameworkAgent
+
+    stub = StubAgent()
+    agent = AgentFrameworkAgent(agent=stub)
+
+    payload = {
+        "run_id": "run-4",
+        "messages": [{"role": "user", "content": "Hello"}],
+    }
+
+    _ = [event async for event in agent.run(payload)]
+
+    assert stub.last_session is not None
+    # Should be a valid UUID (auto-generated)
+    uuid.UUID(stub.last_session.session_id)
+
+
+async def test_service_session_no_thread_id_generates_uuid():
+    """With use_service_session=True and no thread_id, session_id is a UUID and service_session_id is None."""
+    import uuid
+
+    from conftest import StubAgent
+
+    from agent_framework_ag_ui import AgentFrameworkAgent
+
+    stub = StubAgent()
+    agent = AgentFrameworkAgent(agent=stub, use_service_session=True)
+
+    payload = {
+        "run_id": "run-5",
+        "messages": [{"role": "user", "content": "Hello"}],
+    }
+
+    _ = [event async for event in agent.run(payload)]
+
+    assert stub.last_session is not None
+    # session_id should be a valid auto-generated UUID
+    uuid.UUID(stub.last_session.session_id)
+    # service_session_id should be None since no thread_id was supplied
+    assert stub.last_session.service_session_id is None
@@ -455,8 +455,18 @@ class RawFoundryChatClient(  # type: ignore[misc]

        Returns:
            An MCPTool configuration ready to pass to an Agent.
+
+        Raises:
+            ValueError: If neither ``url`` nor ``project_connection_id`` is supplied
+                — one is required by the Foundry Responses API.
        """
-        mcp = FoundryMCPTool(server_label=name.replace(" ", "_"), server_url=url or "", **kwargs)
+        if not url and not project_connection_id:
+            raise ValueError("MCP tool requires either 'url' or 'project_connection_id' to be specified.")
+
+        mcp_kwargs: dict[str, Any] = {"server_label": name.replace(" ", "_"), **kwargs}
+        if url:
+            mcp_kwargs["server_url"] = url
+        mcp = FoundryMCPTool(**mcp_kwargs)

        if description:
            mcp["server_description"] = description
@@ -133,26 +133,55 @@ def select_toolbox_tools(
    return selected


+def _validate_hosted_tool_payload(sanitized: Mapping[str, Any]) -> None:
+    """Fail fast on hosted tool payloads that would always be rejected by the Responses API.
+
+    These mismatches are not injectable defaults — the caller must supply the
+    missing information — so surfacing a clear error here points at the toolbox
+    definition instead of letting the API return a generic 400.
+    """
+    tool_type = sanitized.get("type")
+    if tool_type == "file_search" and not sanitized.get("vector_store_ids"):
+        raise ValueError(
+            "'file_search' tool is missing required 'vector_store_ids'. "
+            "If this came from a Foundry toolbox, update the toolbox definition "
+            "to include at least one vector store ID."
+        )
+    if tool_type == "mcp" and not sanitized.get("server_url") and not sanitized.get("project_connection_id"):
+        raise ValueError(
+            "'mcp' tool is missing both 'server_url' and 'project_connection_id'. "
+            "If this came from a Foundry toolbox, update the toolbox definition "
+            "to include one of these."
+        )
+
+
@experimental(feature_id=ExperimentalFeature.TOOLBOXES)
 def sanitize_foundry_response_tool(tool_item: Any) -> Any:
    """Return a Responses-API-safe tool payload for Foundry hosted tools.

-    Azure AI Projects toolbox reads can currently return hosted tool objects with
-    extra read-model decoration fields such as top-level ``name`` and
-    ``description``. Azure AI Foundry rejects at least ``name`` on Responses API
-    requests with:
+    Reconciles known mismatches between toolbox reads and the Responses API:

-    ``Unknown parameter: 'tools[0].name'``.
+    1. Toolbox reads can return hosted tool objects decorated with read-model
+       fields such as top-level ``name`` and ``description``. The Responses API
+       rejects at least ``name`` with ``Unknown parameter: 'tools[0].name'``.
+       These fields are stripped from non-function hosted tool payloads.
+    2. ``code_interpreter`` tools stored in a toolbox without a ``container``
+       field (the Azure SDK treats it as optional) are rejected by the Responses
+       API with ``Missing required parameter: 'tools[N].container'``. A default
+       ``{"type": "auto"}`` container is injected when absent.
+    3. Hosted tools that are structurally incomplete in ways that cannot be
+       defaulted (``file_search`` without ``vector_store_ids``, ``mcp`` without
+       either ``server_url`` or ``project_connection_id``) raise ``ValueError``
+       with a message that points at the toolbox definition.

-    We defensively strip these decoration fields for non-function hosted tools so
-    the round-trip
-    ``toolbox.tools -> Agent(..., tools=...) -> run()`` works, while the Azure
-    SDK/service behavior is corrected upstream.
+    These are workarounds until the toolbox/Responses proxy normalizes payloads
+    server-side.
    """
    if isinstance(tool_item, FoundryMCPTool):
        sanitized: dict[str, Any] = dict(cast("Mapping[str, Any]", tool_item))
        sanitized.pop("name", None)
        sanitized.pop("description", None)
+        _validate_hosted_tool_payload(sanitized)
        return sanitized

    if isinstance(tool_item, Mapping):
@@ -161,6 +190,9 @@ def sanitize_foundry_response_tool(tool_item: Any) -> Any:
            sanitized = dict(mapping)
            sanitized.pop("name", None)
            sanitized.pop("description", None)
+            if sanitized.get("type") == "code_interpreter" and "container" not in sanitized:
+                sanitized["container"] = {"type": "auto"}
+            _validate_hosted_tool_payload(sanitized)
            return sanitized

    return cast(Any, tool_item)
@@ -607,6 +607,14 @@ def test_get_mcp_tool_with_project_connection_id() -> None:
    assert tool_config["project_connection_id"] == "conn-123"
    assert tool_config["allowed_tools"] == ["search_docs"]
    assert tool_config["server_label"] == "Docs_MCP"
+    # ``server_url`` should not be fabricated when only a project connection is supplied.
+    assert "server_url" not in tool_config
+
+
+def test_get_mcp_tool_requires_url_or_project_connection_id() -> None:
+    """Missing both ``url`` and ``project_connection_id`` is always invalid."""
+    with pytest.raises(ValueError, match="url.*project_connection_id"):
+        FoundryChatClient.get_mcp_tool(name="x")


 def test_prepare_tools_for_openai_strips_extraneous_name_from_foundry_mcp_tool() -> None:
@@ -655,6 +663,103 @@ def test_prepare_tools_for_openai_strips_read_model_fields_from_toolbox_code_int
    assert "description" not in prepared


+def test_prepare_tools_for_openai_injects_default_container_for_code_interpreter_dict() -> None:
+    """Toolbox-returned code_interpreter without a container must get a default injected.
+
+    The Azure SDK treats ``container`` as optional, but the Responses API rejects
+    ``code_interpreter`` entries without one. The sanitizer backfills ``{"type": "auto"}``.
+    """
+    project_client = MagicMock()
+    project_client.get_openai_client.return_value = _make_mock_openai_client()
+    client = FoundryChatClient(project_client=project_client, model="test-model")
+
+    tool = {
+        "type": "code_interpreter",
+        "name": "code_interpreter_t6bbtm",
+    }
+
+    response_tools = client._prepare_tools_for_openai([tool])
+
+    assert len(response_tools) == 1
+    prepared = response_tools[0]
+    assert prepared["type"] == "code_interpreter"
+    assert prepared["container"] == {"type": "auto"}
+    assert "name" not in prepared
+
+
+def test_prepare_tools_for_openai_injects_default_container_for_code_interpreter_sdk_instance() -> None:
+    """SDK ``CodeInterpreterTool`` instances without a container must also be backfilled.
+
+    Reproduces the toolbox creation path that calls
+    ``CodeInterpreterTool(name="code_interpreter")`` without a container.
+    """
+    from azure.ai.projects.models import CodeInterpreterTool
+
+    project_client = MagicMock()
+    project_client.get_openai_client.return_value = _make_mock_openai_client()
+    client = FoundryChatClient(project_client=project_client, model="test-model")
+
+    response_tools = client._prepare_tools_for_openai([CodeInterpreterTool(name="code_interpreter")])
+
+    assert len(response_tools) == 1
+    prepared = response_tools[0]
+    assert prepared["type"] == "code_interpreter"
+    assert prepared["container"] == {"type": "auto"}
+    assert "name" not in prepared
+
+
+def test_prepare_tools_for_openai_preserves_existing_code_interpreter_container() -> None:
+    """An already-populated container must not be overwritten by the sanitizer."""
+    project_client = MagicMock()
+    project_client.get_openai_client.return_value = _make_mock_openai_client()
+    client = FoundryChatClient(project_client=project_client, model="test-model")
+
+    explicit_container = {"file_ids": ["file_123"], "type": "auto"}
+    tool = {"type": "code_interpreter", "container": explicit_container}
+
+    response_tools = client._prepare_tools_for_openai([tool])
+
+    assert response_tools[0]["container"] == explicit_container
+
+
+def test_prepare_tools_for_openai_rejects_file_search_without_vector_store_ids() -> None:
+    """``file_search`` without ``vector_store_ids`` is always invalid — surface a clear error."""
+    project_client = MagicMock()
+    project_client.get_openai_client.return_value = _make_mock_openai_client()
+    client = FoundryChatClient(project_client=project_client, model="test-model")
+
+    with pytest.raises(ValueError, match="vector_store_ids"):
+        client._prepare_tools_for_openai([{"type": "file_search", "name": "fs"}])
+
+
+def test_prepare_tools_for_openai_rejects_mcp_without_server_destination() -> None:
+    """``mcp`` with neither ``server_url`` nor ``project_connection_id`` is always invalid."""
+    project_client = MagicMock()
+    project_client.get_openai_client.return_value = _make_mock_openai_client()
+    client = FoundryChatClient(project_client=project_client, model="test-model")
+
+    tool = FoundryMCPTool(server_label="orphan")
+
+    with pytest.raises(ValueError, match="server_url.*project_connection_id"):
+        client._prepare_tools_for_openai([tool])
+
+
+def test_prepare_tools_for_openai_accepts_mcp_with_only_project_connection_id() -> None:
+    """MCP tools backed by a Foundry connection (no ``server_url``) must still pass validation."""
+    project_client = MagicMock()
+    project_client.get_openai_client.return_value = _make_mock_openai_client()
+    client = FoundryChatClient(project_client=project_client, model="test-model")
+
+    tool = FoundryMCPTool(server_label="githubmcp")
+    tool["project_connection_id"] = "githubmcp"
+
+    response_tools = client._prepare_tools_for_openai([tool])
+
+    assert len(response_tools) == 1
+    assert response_tools[0]["project_connection_id"] == "githubmcp"
+    assert "server_url" not in response_tools[0]
+
+
 def test_prepare_tools_for_openai_strips_name_from_non_function_hosted_tool_dicts() -> None:
    """All non-function hosted tool payloads should drop top-level read-model names."""
    project_client = MagicMock()
@@ -12,7 +12,7 @@ urls.release_notes = "https://github.com/microsoft/agent-framework/releases?q=ta
 urls.issues = "https://github.com/microsoft/agent-framework/issues"
 classifiers = [
  "License :: OSI Approved :: MIT License",
-  "Development Status :: 4 - Alpha",
+  "Development Status :: 3 - Alpha",
  "Intended Audience :: Developers",
  "Programming Language :: Python :: 3",
  "Programming Language :: Python :: 3.10",
@@ -1405,7 +1405,7 @@ class RawOpenAIChatClient(  # type: ignore[misc]
                        else "auto",
                    }
                    file_id = content.additional_properties.get("file_id") if content.additional_properties else None
-                    if file_id:
+                    if file_id is not None:
                        result["file_id"] = file_id
                    return result
                if content.has_top_level_media_type("audio"):
@@ -2028,6 +2028,7 @@ class RawOpenAIChatClient(  # type: ignore[misc]
        local_shell_tool_name = self._get_local_shell_tool_name(options.get("tools"))
        conversation_id: str | None = None
        response_id: str | None = None
+        created_at: str | None = None
        continuation_token: OpenAIContinuationToken | None = None
        model = self.model
        match event.type:
@@ -2209,6 +2210,9 @@ class RawOpenAIChatClient(  # type: ignore[misc]
                response_id = event.response.id
                conversation_id = self._get_conversation_id(event.response, options.get("store"))
                model = event.response.model
+                created_at = datetime.fromtimestamp(event.response.created_at, tz=timezone.utc).strftime(
+                    "%Y-%m-%dT%H:%M:%S.%fZ"
+                )
                if event.response.usage:
                    usage = self._parse_usage_from_openai(event.response.usage)
                    if usage:
@@ -2589,6 +2593,7 @@ class RawOpenAIChatClient(  # type: ignore[misc]
            response_id=response_id,
            role="assistant",
            model=model,
+            created_at=created_at,
            continuation_token=continuation_token,
            additional_properties=metadata,
            raw_representation=event,
@@ -2192,6 +2192,7 @@ def test_streaming_chunk_with_usage_only() -> None:
    mock_event.response.id = "resp_usage"
    mock_event.response.model = "test-model"
    mock_event.response.conversation = None
+    mock_event.response.created_at = 1000000000.0
    mock_event.response.usage = MagicMock()
    mock_event.response.usage.input_tokens = 50
    mock_event.response.usage.output_tokens = 25
@@ -2975,6 +2976,17 @@ def test_prepare_content_for_openai_image_content() -> None:
    assert result["detail"] == "auto"
    assert "file_id" not in result

+    # Test image content with additional_properties present but no file_id key
+    image_content_detail_only = Content.from_uri(
+        uri="https://example.com/basic.png",
+        media_type="image/png",
+        additional_properties={"detail": "high"},
+    )
+    result = client._prepare_content_for_openai("user", image_content_detail_only)
+    assert result["type"] == "input_image"
+    assert result["detail"] == "high"
+    assert "file_id" not in result
+

 def test_prepare_content_for_openai_audio_content() -> None:
    """Test _prepare_content_for_openai with audio content variations."""
@@ -4438,6 +4450,7 @@ def test_streaming_response_completed_no_continuation_token() -> None:
    mock_event.response.conversation = MagicMock()
    mock_event.response.conversation.id = "conv_done"
    mock_event.response.model = "test-model"
+    mock_event.response.created_at = 1000000000.0
    mock_event.response.usage = None

    update = client._parse_chunk_from_openai(mock_event, chat_options, function_call_ids)
@@ -4445,6 +4458,28 @@ def test_streaming_response_completed_no_continuation_token() -> None:
    assert update.continuation_token is None


+def test_streaming_response_completed_sets_created_at() -> None:
+    """Test that response.completed sets created_at on the ChatResponseUpdate."""
+    client = OpenAIChatClient(model="test-model", api_key="test-key")
+    chat_options: dict[str, Any] = {}
+    function_call_ids: dict[int, tuple[str, str]] = {}
+
+    mock_event = MagicMock()
+    mock_event.type = "response.completed"
+    mock_event.response = MagicMock()
+    mock_event.response.id = "resp_created"
+    mock_event.response.conversation = MagicMock()
+    mock_event.response.conversation.id = "conv_created"
+    mock_event.response.model = "test-model"
+    mock_event.response.created_at = 1000000000.0
+    mock_event.response.usage = None
+
+    update = client._parse_chunk_from_openai(mock_event, chat_options, function_call_ids)
+
+    assert update.created_at is not None
+    assert update.created_at == "2001-09-09T01:46:40.000000Z"
+
+
 def test_map_chat_to_agent_update_preserves_continuation_token() -> None:
    """Test that map_chat_to_agent_update propagates continuation_token."""
    from agent_framework._types import map_chat_to_agent_update
@@ -21,7 +21,7 @@ from dotenv import load_dotenv
 from pydantic import Field

 try:
-    import orjson
+    import orjson  # pyright: ignore[reportMissingImports]
 except ImportError:
    orjson = None

@@ -22,7 +22,7 @@ from dotenv import load_dotenv
 from pydantic import Field

 try:
-    import orjson
+    import orjson  # pyright: ignore[reportMissingImports]
 except ImportError:
    orjson = None

@@ -42,11 +42,12 @@ def create_sample_toolbox(name: str) -> str:
    Toolboxes are normally configured in the Foundry portal or a deployment
    script, not the application itself. This helper exists so the samples can
    be run end-to-end without first setting a toolbox up by hand — delete any
-    existing toolbox under ``name``, then create a fresh version containing a
-    single MCP tool. Returns the created version identifier.
+    existing toolbox under ``name``, then create a fresh version containing an
+    MCP tool, a web search tool, and a code interpreter tool. Returns the
+    created version identifier.
    """
    from azure.ai.projects import AIProjectClient
-    from azure.ai.projects.models import MCPTool, Tool
+    from azure.ai.projects.models import CodeInterpreterTool, MCPTool, Tool, WebSearchTool
    from azure.core.exceptions import ResourceNotFoundError

    with (
@@ -67,6 +68,9 @@ def create_sample_toolbox(name: str) -> str:
            )
        ]

+        tools.append(WebSearchTool(name="web_search"))
+        tools.append(CodeInterpreterTool(name="code_interpreter"))
+
        created = project_client.beta.toolboxes.create_version(
            name=name,
            description="Toolbox version with MCP require_approval set to 'never'.",
@@ -3,6 +3,7 @@
 import os
 import subprocess
 from random import randint
+from typing import Annotated

 from agent_framework import Agent, tool
 from agent_framework.foundry import FoundryChatClient
@@ -10,7 +11,6 @@ from agent_framework_foundry_hosting import ResponsesHostServer
 from azure.identity import AzureCliCredential
 from dotenv import load_dotenv
 from pydantic import Field
-from typing import Annotated

 # Load environment variables from .env file
 load_dotenv()
@@ -0,0 +1,11 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""Flaky test report aggregation and trend generation.
+
+Parses JUnit XML (``pytest.xml``) files produced by each CI job, merges
+them with historical data, and generates a markdown trend report showing
+per-test status across the last N runs.
+
+Usage:
+    uv run python -m scripts.flaky_report <reports-dir> <history-file> <output-file>
+"""
@@ -0,0 +1,20 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""CLI entry point for the flaky test report tool.
+
+Usage:
+    uv run python -m scripts.flaky_report <reports-dir> <history-file> <output-file>
+
+Example (from python/ directory):
+    uv run python -m scripts.flaky_report \\
+        ../flaky-reports/ \\
+        flaky-report-history.json \\
+        flaky-test-report.md
+"""
+
+import sys
+
+from scripts.flaky_report.aggregate import main
+
+if __name__ == "__main__":
+    sys.exit(main())
@@ -0,0 +1,396 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""Aggregate per-provider JUnit XML test results and generate a trend report.
+
+Parses ``pytest.xml`` (JUnit XML) files produced by each CI job, merges them
+into a single run, combines with historical data, and generates a markdown
+trend table — the same pattern used by ``scripts/sample_validation/aggregate.py``.
+
+Usage (from CI):
+    python aggregate.py <reports-dir> <history-file> <output-file>
+
+The reports directory is expected to contain subdirectories named
+``test-results-<provider>/`` each containing a ``pytest.xml`` file
+(created by ``actions/download-artifact``).
+"""
+
+from __future__ import annotations
+
+import json
+import sys
+import xml.etree.ElementTree as ET
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any
+
+MAX_HISTORY = 5
+
+STATUS_EMOJI = {
+    "passed": "✅",
+    "failed": "❌",
+    "skipped": "⏭️",
+    "xfailed": "⚠️",
+    "error": "❌",
+}
+
+
+def _format_run_label(timestamp: str) -> str:
+    """Format a timestamp as a compact column label (e.g. '04-16 00:57')."""
+    try:
+        dt = datetime.fromisoformat(timestamp)
+        return dt.strftime("%m-%d %H:%M")
+    except (ValueError, TypeError):
+        return timestamp[:16]
+
+
+def _derive_provider(directory_name: str) -> str:
+    """Derive a provider label from a report directory name.
+
+    ``test-results-openai`` → ``OpenAI``
+    ``test-results-azure-openai`` → ``Azure OpenAI``
+    """
+    raw = directory_name.replace("test-results-", "")
+    known = {
+        "openai": "OpenAI",
+        "azure-openai": "Azure OpenAI",
+        "misc": "Misc (Anthropic, Ollama, MCP)",
+        "functions": "Functions",
+        "foundry": "Foundry",
+        "cosmos": "Cosmos",
+        "unit": "Unit",
+    }
+    if raw in known:
+        return known[raw]
+    parts = raw.split("-")
+    return " ".join(p.capitalize() for p in parts)
+
+
+def _parse_junit_xml(xml_path: Path) -> list[dict[str, str]]:
+    """Parse a JUnit XML file and return a list of test result dicts.
+
+    Each dict has keys: ``nodeid``, ``status``, ``duration``, ``message``.
+    """
+    results: list[dict[str, str]] = []
+    try:
+        tree = ET.parse(xml_path)  # noqa: S314
+    except ET.ParseError as exc:
+        print(f"Warning: failed to parse JUnit XML report '{xml_path}': {exc}", file=sys.stderr)
+        return results
+    root = tree.getroot()
+
+    # Handle both <testsuites><testsuite>... and <testsuite>... layouts
+    testcases: list[ET.Element] = []
+    if root.tag == "testsuites":
+        for suite in root.findall("testsuite"):
+            testcases.extend(suite.findall("testcase"))
+    elif root.tag == "testsuite":
+        testcases = list(root.findall("testcase"))
+
+    for tc in testcases:
+        classname = tc.get("classname", "")
+        name = tc.get("name", "")
+        duration = tc.get("time", "0")
+
+        # Use classname::name as a stable identifier.
+        # pytest writes classname as the dotted module path (possibly including
+        # a test class), e.g. "packages.openai.tests.openai.test_chat_client"
+        # or "packages.openai.tests.openai.test_chat_client.TestClass".
+        nodeid = f"{classname}::{name}" if classname else name
+
+        # Extract module/file name from classname for display context.
+        # pytest writes classname as a dotted path. For tests inside a class
+        # it appends the class name, e.g.:
+        #   "packages.foundry.tests.foundry.test_foundry_embedding_client.TestFoundryEmbeddingIntegration"
+        # We want the file-level module: "test_foundry_embedding_client"
+        if classname:
+            parts = classname.rsplit(".", 2)
+            # If the last segment starts with uppercase it's a class name — take the one before it
+            if len(parts) >= 2 and parts[-1][0:1].isupper():
+                module = parts[-2]
+            else:
+                module = parts[-1]
+        else:
+            module = ""
+
+        # Determine status from child elements
+        failure = tc.find("failure")
+        error = tc.find("error")
+        skipped = tc.find("skipped")
+
+        if failure is not None:
+            status = "failed"
+            message = failure.get("message", "")
+        elif error is not None:
+            status = "error"
+            message = error.get("message", "")
+        elif skipped is not None:
+            # pytest marks xfail as <skipped type="pytest.xfail">
+            skip_type = skipped.get("type", "")
+            status = "xfailed" if "xfail" in skip_type else "skipped"
+            message = skipped.get("message", "")
+        else:
+            status = "passed"
+            message = ""
+
+        results.append({
+            "nodeid": nodeid,
+            "status": status,
+            "duration": duration,
+            "message": message,
+            "module": module,
+        })
+
+    return results
+
+
+# ---------------------------------------------------------------------------
+# Loading
+# ---------------------------------------------------------------------------
+
+
+def load_current_run(reports_dir: Path) -> dict[str, Any]:
+    """Load per-provider JUnit XML reports from the current CI run and merge.
+
+    Args:
+        reports_dir: Directory containing ``test-results-<provider>/`` subdirs.
+
+    Returns:
+        Merged run dict with ``timestamp``, ``summary``, ``results``.
+    """
+    combined_results: dict[str, dict[str, str]] = {}  # nodeid → {status, provider}
+
+    # actions/download-artifact creates: reports_dir/test-results-openai/pytest.xml
+    xml_files: list[tuple[str, Path]] = []
+    if reports_dir.is_dir():
+        for subdir in sorted(reports_dir.iterdir()):
+            if subdir.is_dir():
+                xml_file = subdir / "pytest.xml"
+                if xml_file.exists():
+                    xml_files.append((subdir.name, xml_file))
+
+    if not xml_files:
+        print(f"Warning: No pytest.xml files found in {reports_dir}")
+        return {
+            "timestamp": datetime.now(timezone.utc).isoformat(),
+            "summary": {
+                "total": 0,
+                "passed": 0,
+                "failed": 0,
+                "skipped": 0,
+            },
+            "results": {},
+        }
+
+    for dir_name, xml_file in xml_files:
+        print(f"  Loading: {xml_file}")
+        provider = _derive_provider(dir_name)
+        tests = _parse_junit_xml(xml_file)
+        for test in tests:
+            combined_results[test["nodeid"]] = {
+                "status": test["status"],
+                "provider": provider,
+                "module": test.get("module", ""),
+            }
+
+    # Build summary counts using mutually exclusive status buckets.
+    # Errors are folded into the failed count for display purposes.
+    statuses = [r["status"] for r in combined_results.values()]
+    summary = {
+        "total": len(statuses),
+        "passed": statuses.count("passed"),
+        "failed": statuses.count("failed") + statuses.count("error"),
+        "skipped": statuses.count("skipped"),
+    }
+
+    return {
+        "timestamp": datetime.now(timezone.utc).isoformat(),
+        "summary": summary,
+        "results": combined_results,
+    }
+
+
+def load_history(history_path: Path) -> list[dict[str, Any]]:
+    """Load previous run history from a cache file."""
+    if history_path.exists():
+        with open(history_path, encoding="utf-8") as f:
+            data = json.load(f)
+        runs = data.get("runs", [])
+        print(f"  Loaded {len(runs)} previous run(s) from history")
+        return runs
+    print("  No previous history found")
+    return []
+
+
+def save_history(history_path: Path, runs: list[dict[str, Any]]) -> None:
+    """Save run history, keeping only the last ``MAX_HISTORY`` entries."""
+    history_path.parent.mkdir(parents=True, exist_ok=True)
+    trimmed = runs[-MAX_HISTORY:]
+    with open(history_path, "w", encoding="utf-8") as f:
+        json.dump({"runs": trimmed}, f, indent=2)
+    print(f"  Saved {len(trimmed)} run(s) to history")
+
+
+# ---------------------------------------------------------------------------
+# Report generation
+# ---------------------------------------------------------------------------
+
+
+def _short_name(nodeid: str) -> str:
+    """Extract a short test name from a full nodeid.
+
+    ``packages.openai.tests.openai.test_openai_chat_client::test_integration_options``
+    → ``test_integration_options``
+    """
+    return nodeid.split("::")[-1] if "::" in nodeid else nodeid
+
+
+def generate_trend_report(runs: list[dict[str, Any]]) -> str:
+    """Generate a markdown trend report from run history."""
+    lines = [
+        "# 🔬 Flaky Test Report",
+        "",
+        f"*Generated: {datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M UTC')}*",
+        "",
+    ]
+
+    # --- Overall status table (most recent first) ---
+    lines.append("## Overall Status (Last 5 Runs)")
+    lines.append("")
+    lines.append("| Run | Total | ✅ Passed | ❌ Failed | ⏭️ Skipped |")
+    lines.append("|-----|-------|-----------|-----------|------------|")
+
+    for run in reversed(runs):
+        s = run.get("summary", {})
+        total = s.get("total", 0)
+        label = _format_run_label(run["timestamp"])
+        lines.append(
+            f"| {label} "
+            f"| {total} "
+            f"| {s.get('passed', 0)}/{total} "
+            f"| {s.get('failed', 0)}/{total} "
+            f"| {s.get('skipped', 0)}/{total} |"
+        )
+
+    for _ in range(MAX_HISTORY - len(runs)):
+        lines.append("| N/A | N/A | N/A | N/A | N/A |")
+
+    lines.append("")
+
+    # --- Per-test results table ---
+    lines.append("## Per-Test Results")
+    lines.append("")
+
+    # Collect all test nodeids, providers, and modules across all runs
+    all_tests: dict[str, str] = {}  # nodeid → provider (from most recent run)
+    all_modules: dict[str, str] = {}  # nodeid → module (from most recent run)
+    for run in runs:
+        for nodeid, info in run.get("results", {}).items():
+            provider = info.get("provider", "Unknown") if isinstance(info, dict) else "Unknown"
+            module = info.get("module", "") if isinstance(info, dict) else ""
+            all_tests[nodeid] = provider
+            all_modules[nodeid] = module
+
+    if not all_tests:
+        lines.append("*No test results available.*")
+        return "\n".join(lines)
+
+    # Build header (most recent run first)
+    header = "| Test | File | Provider |"
+    separator = "|------|------|----------|"
+    for run in reversed(runs):
+        label = _format_run_label(run["timestamp"])
+        header += f" {label} |"
+        separator += "------------|"
+    for _ in range(MAX_HISTORY - len(runs)):
+        header += " N/A |"
+        separator += "-----|"
+
+    lines.append(header)
+    lines.append(separator)
+
+    # Sort by provider then test name
+    for nodeid in sorted(all_tests, key=lambda n: (all_tests[n], n)):
+        provider = all_tests[nodeid]
+        module = all_modules.get(nodeid, "")
+        short = _short_name(nodeid)
+        row = f"| `{short}` | `{module}` | {provider} |"
+
+        for run in reversed(runs):
+            result = run.get("results", {}).get(nodeid)
+            if result is None:
+                emoji = "N/A"
+            else:
+                status = result.get("status", "N/A") if isinstance(result, dict) else result
+                emoji = STATUS_EMOJI.get(status, "❓")
+            row += f" {emoji} |"
+
+        for _ in range(MAX_HISTORY - len(runs)):
+            row += " N/A |"
+
+        lines.append(row)
+
+    lines.append("")
+    lines.append("**Legend:** ✅ Passed · ❌ Failed · ⏭️ Skipped · ⚠️ Expected Failure (xfail) · N/A Not available")
+    lines.append("")
+
+    return "\n".join(lines)
+
+
+# ---------------------------------------------------------------------------
+# CLI entry point
+# ---------------------------------------------------------------------------
+
+
+def main() -> int:
+    if len(sys.argv) != 4:
+        print("Usage: python aggregate.py <reports-dir> <history-file> <output-file>")
+        return 1
+
+    reports_dir = Path(sys.argv[1])
+    history_path = Path(sys.argv[2])
+    output_path = Path(sys.argv[3])
+
+    print("Aggregating test results from JUnit XML...")
+
+    # Load current run's per-provider XML reports
+    print(f"\nLoading reports from {reports_dir}:")
+    current_run = load_current_run(reports_dir)
+    s = current_run.get("summary", {})
+    total = s.get("total", 0)
+    print(
+        f"  Current run: {s.get('passed', 0)} passed, "
+        f"{s.get('failed', 0)} failed, "
+        f"{s.get('skipped', 0)} skipped "
+        f"(total: {total})"
+    )
+
+    # Load history and append current run (skip empty runs to avoid polluting trend)
+    print(f"\nLoading history from {history_path}:")
+    runs = load_history(history_path)
+    if total > 0:
+        runs.append(current_run)
+        runs = runs[-MAX_HISTORY:]
+    else:
+        print("  Skipping history append (no test results in current run)")
+
+    # Save updated history
+    print(f"\nSaving history to {history_path}:")
+    save_history(history_path, runs)
+
+    # Generate trend report
+    print("\nGenerating trend report...")
+    report = generate_trend_report(runs)
+
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+    output_path.write_text(report, encoding="utf-8")
+    print(f"Trend report written to {output_path}")
+
+    # Print the report to stdout for CI visibility
+    print("\n" + "=" * 80)
+    print(report)
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())
Author	SHA1	Message	Date
dependabot[bot]andGitHub	35d17cbdc9	Bump actions/cache from 4 to 5 Bumps [actions/cache](https://github.com/actions/cache) from 4 to 5. - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](https://github.com/actions/cache/compare/v4...v5) --- updated-dependencies: - dependency-name: actions/cache dependency-version: '5' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>	2026-04-22 20:20:51 +00:00
Giles Odigwe GitHub Copilot	3f23e1dfbf	Python: Flaky test report (#5342 ) * Add flaky test trend reporting to CI workflows Parse JUnit XML (pytest.xml) from each integration test job and aggregate results into a markdown trend report showing per-test pass/fail/skip status across the last 5 runs. Changes: - Add python/scripts/flaky_report/ package (JUnit XML parser + trend report generator following the sample_validation pattern) - Add upload-artifact steps to all 6 integration test jobs in both python-merge-tests.yml and python-integration-tests.yml - Add python-flaky-test-report aggregation job with history caching - Add --junitxml=pytest.xml to integration-tests.yml jobs (already present in merge-tests.yml) - Fix Cosmos job --junitxml path (use absolute path since uv run --directory changes cwd) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix flaky report: handle missing test results gracefully - Guard against missing reports directory in load_current_run() - Only run report job when at least one integration test job completed (skip when all jobs are skipped, e.g. on pull_request events) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review: fix provider names and if-expression precedence - Use explicit provider name mapping in _derive_provider() so OpenAI renders correctly instead of 'Openai' - Fix operator precedence in workflow if-expressions by wrapping success/failure checks in parentheses Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add File column and xfail detection to flaky test report - Add File column showing module name (e.g., test_openai_chat_client) to disambiguate tests with the same function name across files - Detect pytest xfail tests in JUnit XML (type=pytest.xfail) and show them with a distinct warning emoji instead of skip emoji - Update legend to include xfail explanation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add Foundry embedding env vars to merge-tests workflow Sync the Foundry integration job in python-merge-tests.yml with python-integration-tests.yml by adding FOUNDRY_MODELS_ENDPOINT, FOUNDRY_MODELS_API_KEY, FOUNDRY_EMBEDDING_MODEL, and FOUNDRY_IMAGE_EMBEDDING_MODEL. Once the repo variables/secrets are configured, the embedding integration test will run in CI. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix File column showing class name instead of module name When a test is inside a class, pytest writes the classname as e.g. 'pkg.test_file.TestClass'. The previous rsplit logic extracted 'TestClass' instead of 'test_file'. Now detect uppercase-starting segments as class names and use the preceding segment instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review: UTC timestamps, XML error handling, summary fix, docstring - Use datetime.now(timezone.utc) for accurate UTC timestamps - Catch ET.ParseError per-file so corrupt XML doesn't crash the report - Remove separate 'error' key from summary (errors folded into 'failed') - Fix _short_name docstring to show actual dotted classname::name format Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>	2026-04-22 20:16:50 +00:00
Evan MattsonandGitHub	d75f874d78	Adjust dev status for alpha from 4 to 3 in foundry hosting pyproject.toml (#5387 )	2026-04-22 17:48:02 +00:00
Evan Mattson GitHub Copilot Copilot	0b50455e75	Python: Pass client thread_id as session_id when constructing AgentSession in AG-UI (#5384 ) * Pass thread_id as session_id when constructing AgentSession in AG-UI run_agent_stream() was constructing AgentSession without passing the client's thread_id as session_id, causing every request to receive a random UUID. This broke session continuity for HistoryProvider implementations that rely on session_id matching the client's thread_id. Pass session_id=thread_id in both the service-session and non-service code paths so the session identity is consistent with the AG-UI client. Fixes #5357 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add test for service_session with no thread_id edge case (#5357) When use_service_session=True but no thread_id/threadId is in the payload, verify session_id is a generated UUID and service_session_id is None. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <copilot@github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>	2026-04-22 17:45:25 +00:00
Evan Mattson GitHub Copilot Copilot	3ae86f098e	Python: Propagate thread_id and forwarded_props through AG-UI to A2A context_id (#5383 ) * Propagate session.service_session_id as A2A context_id When A2AAgent is used behind the AG-UI protocol, the client thread_id is stored in session.service_session_id but was never forwarded as the A2A context_id. This broke session continuity across the AG-UI → A2A boundary. Add an optional context_id keyword argument to _prepare_message_for_a2a() and pass session.service_session_id from run(). The explicit message.additional_properties["context_id"] still takes precedence. Fixes #5345 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add integration tests for session context_id wiring in run() (#5345) - Enhance MockA2AClient.send_message to capture last_message for assertions - Add test_run_passes_session_service_session_id_as_context_id: verifies run() passes session.service_session_id through to A2A message context_id - Add test_run_message_context_id_takes_precedence_over_session: verifies explicit message context_id wins over session fallback - Update _prepare_message_for_a2a docstring to document context_id param and its precedence rules Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback for #5345: Python: [Bug]: Inconvenient passing of context_id / thread_id in A2A/AG-UI implementations --------- Co-authored-by: Copilot <copilot@github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>	2026-04-22 17:44:41 +00:00
Evan MattsonandGitHub	fffd0acb3e	Python: fix(foundry): reconcile toolbox hosted-tool payloads with Responses API (#5414 ) * fix(foundry): reconcile toolbox hosted-tool payloads with Responses API * docs(foundry): update create_sample_toolbox docstring to reflect all tools created	2026-04-22 17:43:26 +00:00
Evan Mattson GitHub Copilot Copilot	ea3320d39f	Python: Fix OpenAI Responses streaming to propagate `created_at` from final `response.completed` event (#5382 ) * Fix streaming response losing created_at from response.completed event (#5347) The streaming path in _parse_chunk_from_openai did not extract created_at from the response.completed event, unlike the non-streaming path in _parse_responses_response. This caused durabletask persistence warnings when created_at was None. Extract created_at in the response.completed case and pass it to the returned ChatResponseUpdate. Also fix pre-existing pyright errors for optional orjson import in sample files. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix orjson import suppression to use pyright instead of mypy (#5347) Replace `# type: ignore[import-not-found]` with `# pyright: ignore[reportMissingImports]` on optional orjson imports in conversation sample files, matching the repo's Pyright strict configuration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <copilot@github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>	2026-04-22 06:19:31 +00:00
Evan Mattson GitHub Copilot	9e915b36b6	Add pr review GH workflow (#5418 ) * Add workflow PR review * Allow reviews on draft PRs * Update .github/workflows/devflow-pr-review.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update .github/workflows/devflow-pr-review.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Bump actions/checkout to v6 and uv to 0.11.x --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2026-04-22 13:52:42 +09:00
S3rj GitHub Sergey Borisov	bca40a7e90	Python: fix: exclude null file_id from input_image payload to prevent 400 sch… (#5125 ) * fix: exclude null file_id from input_image payload to prevent 400 schema error (#5120) * test: add case for additional_properties present without file_id key --------- Co-authored-by: Sergey Borisov <sergey.borisov@dataimpact.io>	2026-04-21 22:29:00 +00:00