Compare commits

...
Author SHA1 Message Date
Tao ChenandGitHub 4c0f0ec99a Merge branch 'main' into taochen/python-update-sample-validation-scripts 2026-03-24 18:16:22 -07:00
Tao Chen 6185ba2125 force node22 2026-03-24 17:10:25 -07:00
Tao Chen dcc1eeac36 force node24 2026-03-24 17:06:41 -07:00
Tao Chen 3f2096595f force node24 2026-03-24 16:58:18 -07:00
Tao Chen 45a9da5523 Comments 2026-03-24 16:51:44 -07:00
Tao Chen 6364c05efc Add more env vars 2026-03-24 16:19:36 -07:00
Tao Chen bbb871e4cd Add timestamp 2026-03-24 12:47:55 -07:00
Tao Chen 7d7b8dd1a4 Create trend report 2026-03-24 11:33:20 -07:00
westeyandGitHub 2c000b032d .NET: Update AIContextProviders to use Microsoft.Extensions.Compliance.Redaction (#4854)
* Update providers to use Microsoft.Extensions.Compliance.Redaction

* Fix formatting.

* Fix readme
2026-03-24 18:12:55 +00:00
Tao Chen 2f51a5ca78 Add .env 2026-03-24 08:58:33 -07:00
cc85bbc2dc .NET: Re-enable AzureAI.Persistent packaging and integration tests (#4769) (#4865)
Azure.AI.Agents.Persistent 1.2.0-beta.10 now targets ME.AI 10.4.0+,
resolving the compatibility issue that required disabling this package.

- Remove IsPackable=false from the csproj
- Re-enable all 6 integration test classes (IntegrationDisabled → Integration)
- Remove outdated compatibility warning from README.md

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-24 11:06:32 +00:00
Tao Chen ad5749c92a Split jobs 2026-03-23 17:16:19 -07:00
Tao Chen ed6b290457 Add fix suggestion 2026-03-23 16:15:47 -07:00
Tao Chen 63039cb748 Update autogen-migration samples 2026-03-23 15:51:17 -07:00
Tao Chen 3e7c94699f Adjust prompt 2026-03-23 15:17:45 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
01aaf2baea Bump actions/download-artifact from 7 to 8 (#4372)
Bumps [actions/download-artifact](https://github.com/actions/download-artifact) from 7 to 8.
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](https://github.com/actions/download-artifact/compare/v7...v8)

---
updated-dependencies:
- dependency-name: actions/download-artifact
  dependency-version: '8'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-23 21:55:19 +00:00
cb96347c95 Python: Fix PydanticSchemaGenerationError when using from __future__ import annotations with @tool (#4822)
* Fix PydanticSchemaGenerationError with PEP 563 annotations in @tool

_resolve_input_model used raw param.annotation from inspect.signature(),
which returns string annotations when 'from __future__ import annotations'
is active (PEP 563). This caused Pydantic's create_model to fail for
complex types like Optional[int] or FunctionInvocationContext.

Use typing.get_type_hints() to resolve annotations to actual types before
passing them to create_model, matching the approach already used by
_discover_injected_parameters.

Fixes #4809

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Apply pre-commit auto-fixes

* Remove reproduction report and unused test imports

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(tests): strengthen PEP 563 regression tests per review feedback (#4809)

- Verify type correctness in schema assertions (not just key presence)
- Fix ctx annotation to FunctionInvocationContext | None for type consistency
- Add test for Optional[CustomType] pattern (original bug trigger)
- Add test for get_type_hints() fallback with unresolvable forward refs

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Apply pre-commit auto-fixes

* Address review feedback for #4809: Python: [Bug]: PydanticSchemaGenerationError in FunctionInvocationContext

---------

Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-23 21:47:12 +00:00
Tao Chen 6320443969 Update sample validation scripts 2026-03-23 14:40:37 -07:00
westeyandGitHub 5070c67d0e Mark constructors of InvokingContext and InvokedContext as experimental (#4851) 2026-03-23 19:06:07 +00:00
9a47620f64 .NET: Update Hosted Samples References to latest beta.11 (#4853)
* Bump HostedAgents samples to AgentFramework beta.11 and pass credential to UseFoundryTools

Update all 8 HostedAgents samples:
- Azure.AI.AgentServer.AgentFramework -> 1.0.0-beta.11
- Microsoft.Agents.AI.OpenAI -> 1.0.0-rc4
- Microsoft.Agents.AI/AzureAI/Workflows -> 1.0.0-rc4
- Azure.AI.Projects -> 2.0.0-beta.1
- Fix Workflow.AsAgent() -> AsAIAgent() in FoundryMultiAgent
- Pass credential to UseFoundryTools in AgentWithTools (resolves #56802)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove AgentWithTools sample (UseFoundryTools no longer supported)

Remove the AgentWithTools hosted agent sample as the UseFoundryTools
backend is no longer supported. Updated HostedAgents README and solution
file to remove all references.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix AgentWithHostedMCP: downgrade Azure.AI.OpenAI to 2.8.0-beta.1 for rc4 compatibility

Azure.AI.OpenAI 2.9.0-beta.1 has breaking changes (GetResponsesClient no
longer accepts deployment name, ResponsesClient.Model removed) that are
incompatible with Microsoft.Agents.AI.OpenAI rc4. Pin to 2.8.0-beta.1 and
use GetResponsesClient(deploymentName).AsAIAgent() pattern.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-23 17:41:49 +00:00
68 changed files with 1404 additions and 585 deletions
@@ -24,7 +24,9 @@ runs:
using: "composite"
steps:
- name: Set up Node.js environment
uses: actions/setup-node@v4
uses: actions/setup-node@v6
with:
node-version: 22
- name: Install Copilot CLI
shell: bash
+520 -8
View File
@@ -41,6 +41,13 @@ jobs:
azure-subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
os: ${{ runner.os }}
- name: Create .env for samples
run: |
echo "AZURE_AI_PROJECT_ENDPOINT=$AZURE_AI_PROJECT_ENDPOINT" >> .env
echo "AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME=$AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME" >> .env
echo "AZURE_OPENAI_ENDPOINT=$AZURE_OPENAI_ENDPOINT" >> .env
echo "AZURE_OPENAI_CHAT_DEPLOYMENT_NAME=$AZURE_OPENAI_CHAT_DEPLOYMENT_NAME" >> .env
- name: Run sample validation
run: |
cd scripts && uv run python -m sample_validation --subdir 01-get-started --save-report --report-name 01-get-started
@@ -50,7 +57,7 @@ jobs:
if: always()
with:
name: validation-report-01-get-started
path: python/scripts/sample_validation/reports/
path: python/samples/sample_validation/reports/
validate-02-agents:
name: Validate 02-agents
@@ -64,10 +71,13 @@ jobs:
AZURE_OPENAI_ENDPOINT: ${{ vars.AZUREOPENAI__ENDPOINT }}
AZURE_OPENAI_CHAT_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__CHATDEPLOYMENTNAME }}
AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__RESPONSESDEPLOYMENTNAME }}
AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__EMBEDDINGDEPLOYMENTNAME }}
# OpenAI configuration
OPENAI_API_KEY: ${{ secrets.OPENAI__APIKEY }}
OPENAI_CHAT_MODEL_ID: ${{ vars.OPENAI__CHATMODELID }}
OPENAI_RESPONSES_MODEL_ID: ${{ vars.OPENAI__RESPONSESMODELID }}
# GitHub MCP
GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}
# Observability
ENABLE_INSTRUMENTATION: "true"
defaults:
@@ -84,16 +94,420 @@ jobs:
azure-subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
os: ${{ runner.os }}
- name: Create .env for samples
run: |
echo "AZURE_AI_PROJECT_ENDPOINT=$AZURE_AI_PROJECT_ENDPOINT" >> .env
echo "AZURE_AI_MODEL_DEPLOYMENT_NAME=$AZURE_AI_MODEL_DEPLOYMENT_NAME" >> .env
echo "AZURE_OPENAI_ENDPOINT=$AZURE_OPENAI_ENDPOINT" >> .env
echo "AZURE_OPENAI_CHAT_DEPLOYMENT_NAME=$AZURE_OPENAI_CHAT_DEPLOYMENT_NAME" >> .env
echo "AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME=$AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME" >> .env
echo "AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME=$AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME" >> .env
echo "OPENAI_API_KEY=$OPENAI_API_KEY" >> .env
echo "OPENAI_CHAT_MODEL_ID=$OPENAI_CHAT_MODEL_ID" >> .env
echo "OPENAI_RESPONSES_MODEL_ID=$OPENAI_RESPONSES_MODEL_ID" >> .env
echo "GITHUB_PAT=$GITHUB_PAT" >> .env
- name: Run sample validation
run: |
cd scripts && uv run python -m sample_validation --subdir 02-agents --save-report --report-name 02-agents
cd scripts && uv run python -m sample_validation --subdir 02-agents --exclude providers --save-report --report-name 02-agents
- name: Upload validation report
uses: actions/upload-artifact@v7
if: always()
with:
name: validation-report-02-agents
path: python/scripts/sample_validation/reports/
path: python/samples/sample_validation/reports/
validate-02-agents-openai:
name: Validate 02-agents/providers/openai
runs-on: ubuntu-latest
environment: integration
env:
OPENAI_API_KEY: ${{ secrets.OPENAI__APIKEY }}
OPENAI_CHAT_MODEL_ID: ${{ vars.OPENAI__CHATMODELID }}
OPENAI_RESPONSES_MODEL_ID: ${{ vars.OPENAI__RESPONSESMODELID }}
defaults:
run:
working-directory: python
steps:
- uses: actions/checkout@v6
- name: Setup environment
uses: ./.github/actions/sample-validation-setup
with:
azure-client-id: ${{ secrets.AZURE_CLIENT_ID }}
azure-tenant-id: ${{ secrets.AZURE_TENANT_ID }}
azure-subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
os: ${{ runner.os }}
- name: Create .env for samples
run: |
echo "OPENAI_API_KEY=$OPENAI_API_KEY" >> .env
echo "OPENAI_CHAT_MODEL_ID=$OPENAI_CHAT_MODEL_ID" >> .env
echo "OPENAI_RESPONSES_MODEL_ID=$OPENAI_RESPONSES_MODEL_ID" >> .env
- name: Run sample validation
run: |
cd scripts && uv run python -m sample_validation --subdir 02-agents/providers/openai --save-report --report-name 02-agents-openai
- name: Upload validation report
uses: actions/upload-artifact@v7
if: always()
with:
name: validation-report-02-agents-openai
path: python/samples/sample_validation/reports/
validate-02-agents-azure-openai:
name: Validate 02-agents/providers/azure_openai
runs-on: ubuntu-latest
environment: integration
env:
AZURE_AI_PROJECT_ENDPOINT: ${{ vars.AZURE_AI_PROJECT_ENDPOINT }}
AZURE_OPENAI_ENDPOINT: ${{ vars.AZUREOPENAI__ENDPOINT }}
AZURE_OPENAI_CHAT_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__CHATDEPLOYMENTNAME }}
AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__RESPONSESDEPLOYMENTNAME }}
defaults:
run:
working-directory: python
steps:
- uses: actions/checkout@v6
- name: Setup environment
uses: ./.github/actions/sample-validation-setup
with:
azure-client-id: ${{ secrets.AZURE_CLIENT_ID }}
azure-tenant-id: ${{ secrets.AZURE_TENANT_ID }}
azure-subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
os: ${{ runner.os }}
- name: Create .env for samples
run: |
echo "AZURE_AI_PROJECT_ENDPOINT=$AZURE_AI_PROJECT_ENDPOINT" >> .env
echo "AZURE_OPENAI_ENDPOINT=$AZURE_OPENAI_ENDPOINT" >> .env
echo "AZURE_OPENAI_CHAT_DEPLOYMENT_NAME=$AZURE_OPENAI_CHAT_DEPLOYMENT_NAME" >> .env
echo "AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME=$AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME" >> .env
- name: Run sample validation
run: |
cd scripts && uv run python -m sample_validation --subdir 02-agents/providers/azure_openai --save-report --report-name 02-agents-azure-openai
- name: Upload validation report
uses: actions/upload-artifact@v7
if: always()
with:
name: validation-report-02-agents-azure-openai
path: python/samples/sample_validation/reports/
validate-02-agents-azure-ai:
name: Validate 02-agents/providers/azure_ai
runs-on: ubuntu-latest
environment: integration
env:
AZURE_AI_PROJECT_ENDPOINT: ${{ vars.AZURE_AI_PROJECT_ENDPOINT }}
AZURE_AI_MODEL_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__RESPONSESDEPLOYMENTNAME }}
AZURE_AI_CHAT_MODEL_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__CHATDEPLOYMENTNAME }}
AZURE_AI_EMBEDDING_MODEL_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__EMBEDDINGDEPLOYMENTNAME }}
BING_CONNECTION_ID: ${{ secrets.BING_CONNECTION_ID }}
defaults:
run:
working-directory: python
steps:
- uses: actions/checkout@v6
- name: Setup environment
uses: ./.github/actions/sample-validation-setup
with:
azure-client-id: ${{ secrets.AZURE_CLIENT_ID }}
azure-tenant-id: ${{ secrets.AZURE_TENANT_ID }}
azure-subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
os: ${{ runner.os }}
- name: Create .env for samples
run: |
echo "AZURE_AI_PROJECT_ENDPOINT=$AZURE_AI_PROJECT_ENDPOINT" >> .env
echo "AZURE_AI_MODEL_DEPLOYMENT_NAME=$AZURE_AI_MODEL_DEPLOYMENT_NAME" >> .env
echo "AZURE_AI_CHAT_MODEL_DEPLOYMENT_NAME=$AZURE_AI_CHAT_MODEL_DEPLOYMENT_NAME" >> .env
echo "AZURE_AI_EMBEDDING_MODEL_DEPLOYMENT_NAME=$AZURE_AI_EMBEDDING_MODEL_DEPLOYMENT_NAME" >> .env
echo "BING_CONNECTION_ID=$BING_CONNECTION_ID" >> .env
- name: Run sample validation
run: |
cd scripts && uv run python -m sample_validation --subdir 02-agents/providers/azure_ai --save-report --report-name 02-agents-azure-ai
- name: Upload validation report
uses: actions/upload-artifact@v7
if: always()
with:
name: validation-report-02-agents-azure-ai
path: python/samples/sample_validation/reports/
validate-02-agents-azure-ai-agent:
name: Validate 02-agents/providers/azure_ai_agent
runs-on: ubuntu-latest
environment: integration
env:
AZURE_AI_PROJECT_ENDPOINT: ${{ vars.AZURE_AI_PROJECT_ENDPOINT }}
AZURE_AI_MODEL_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__RESPONSESDEPLOYMENTNAME }}
defaults:
run:
working-directory: python
steps:
- uses: actions/checkout@v6
- name: Setup environment
uses: ./.github/actions/sample-validation-setup
with:
azure-client-id: ${{ secrets.AZURE_CLIENT_ID }}
azure-tenant-id: ${{ secrets.AZURE_TENANT_ID }}
azure-subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
os: ${{ runner.os }}
- name: Create .env for samples
run: |
echo "AZURE_AI_PROJECT_ENDPOINT=$AZURE_AI_PROJECT_ENDPOINT" >> .env
echo "AZURE_AI_MODEL_DEPLOYMENT_NAME=$AZURE_AI_MODEL_DEPLOYMENT_NAME" >> .env
- name: Run sample validation
run: |
cd scripts && uv run python -m sample_validation --subdir 02-agents/providers/azure_ai_agent --save-report --report-name 02-agents-azure-ai-agent
- name: Upload validation report
uses: actions/upload-artifact@v7
if: always()
with:
name: validation-report-02-agents-azure-ai-agent
path: python/samples/sample_validation/reports/
validate-02-agents-anthropic:
name: Validate 02-agents/providers/anthropic
runs-on: ubuntu-latest
environment: integration
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
ANTHROPIC_CHAT_MODEL_ID: ${{ vars.ANTHROPIC_CHAT_MODEL_ID }}
defaults:
run:
working-directory: python
steps:
- uses: actions/checkout@v6
- name: Setup environment
uses: ./.github/actions/sample-validation-setup
with:
azure-client-id: ${{ secrets.AZURE_CLIENT_ID }}
azure-tenant-id: ${{ secrets.AZURE_TENANT_ID }}
azure-subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
os: ${{ runner.os }}
- name: Create .env for samples
run: |
echo "ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY" >> .env
echo "ANTHROPIC_CHAT_MODEL_ID=$ANTHROPIC_CHAT_MODEL_ID" >> .env
- name: Run sample validation
run: |
cd scripts && uv run python -m sample_validation --subdir 02-agents/providers/anthropic --save-report --report-name 02-agents-anthropic
- name: Upload validation report
uses: actions/upload-artifact@v7
if: always()
with:
name: validation-report-02-agents-anthropic
path: python/samples/sample_validation/reports/
validate-02-agents-github-copilot:
name: Validate 02-agents/providers/github_copilot
runs-on: ubuntu-latest
environment: integration
defaults:
run:
working-directory: python
steps:
- uses: actions/checkout@v6
- name: Setup environment
uses: ./.github/actions/sample-validation-setup
with:
azure-client-id: ${{ secrets.AZURE_CLIENT_ID }}
azure-tenant-id: ${{ secrets.AZURE_TENANT_ID }}
azure-subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
os: ${{ runner.os }}
- name: Run sample validation
run: |
cd scripts && uv run python -m sample_validation --subdir 02-agents/providers/github_copilot --save-report --report-name 02-agents-github-copilot
- name: Upload validation report
uses: actions/upload-artifact@v7
if: always()
with:
name: validation-report-02-agents-github-copilot
path: python/samples/sample_validation/reports/
validate-02-agents-amazon:
name: Validate 02-agents/providers/amazon
if: false # Temporarily disabled - requires AWS credentials
runs-on: ubuntu-latest
environment: integration
env:
BEDROCK_CHAT_MODEL_ID: ${{ vars.BEDROCK__CHATMODELID }}
defaults:
run:
working-directory: python
steps:
- uses: actions/checkout@v6
- name: Setup environment
uses: ./.github/actions/sample-validation-setup
with:
azure-client-id: ${{ secrets.AZURE_CLIENT_ID }}
azure-tenant-id: ${{ secrets.AZURE_TENANT_ID }}
azure-subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
os: ${{ runner.os }}
- name: Run sample validation
run: |
cd scripts && uv run python -m sample_validation --subdir 02-agents/providers/amazon --save-report --report-name 02-agents-amazon
- name: Upload validation report
uses: actions/upload-artifact@v7
if: always()
with:
name: validation-report-02-agents-amazon
path: python/samples/sample_validation/reports/
validate-02-agents-ollama:
name: Validate 02-agents/providers/ollama
if: false # Temporarily disabled - requires local Ollama server
runs-on: ubuntu-latest
environment: integration
env:
OLLAMA_MODEL: ${{ vars.OLLAMA__MODEL }}
defaults:
run:
working-directory: python
steps:
- uses: actions/checkout@v6
- name: Setup environment
uses: ./.github/actions/sample-validation-setup
with:
azure-client-id: ${{ secrets.AZURE_CLIENT_ID }}
azure-tenant-id: ${{ secrets.AZURE_TENANT_ID }}
azure-subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
os: ${{ runner.os }}
- name: Run sample validation
run: |
cd scripts && uv run python -m sample_validation --subdir 02-agents/providers/ollama --save-report --report-name 02-agents-ollama
- name: Upload validation report
uses: actions/upload-artifact@v7
if: always()
with:
name: validation-report-02-agents-ollama
path: python/samples/sample_validation/reports/
validate-02-agents-foundry-local:
name: Validate 02-agents/providers/foundry_local
if: false # Temporarily disabled - requires local Foundry setup
runs-on: ubuntu-latest
environment: integration
defaults:
run:
working-directory: python
steps:
- uses: actions/checkout@v6
- name: Setup environment
uses: ./.github/actions/sample-validation-setup
with:
azure-client-id: ${{ secrets.AZURE_CLIENT_ID }}
azure-tenant-id: ${{ secrets.AZURE_TENANT_ID }}
azure-subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
os: ${{ runner.os }}
- name: Run sample validation
run: |
cd scripts && uv run python -m sample_validation --subdir 02-agents/providers/foundry_local --save-report --report-name 02-agents-foundry-local
- name: Upload validation report
uses: actions/upload-artifact@v7
if: always()
with:
name: validation-report-02-agents-foundry-local
path: python/samples/sample_validation/reports/
validate-02-agents-copilotstudio:
name: Validate 02-agents/providers/copilotstudio
if: false # Temporarily disabled - requires Copilot Studio setup
runs-on: ubuntu-latest
environment: integration
env:
COPILOTSTUDIOAGENT__ENVIRONMENTID: ${{ secrets.COPILOTSTUDIOAGENT__ENVIRONMENTID }}
COPILOTSTUDIOAGENT__SCHEMANAME: ${{ secrets.COPILOTSTUDIOAGENT__SCHEMANAME }}
COPILOTSTUDIOAGENT__TENANTID: ${{ secrets.COPILOTSTUDIOAGENT__TENANTID }}
COPILOTSTUDIOAGENT__AGENTAPPID: ${{ secrets.COPILOTSTUDIOAGENT__AGENTAPPID }}
defaults:
run:
working-directory: python
steps:
- uses: actions/checkout@v6
- name: Setup environment
uses: ./.github/actions/sample-validation-setup
with:
azure-client-id: ${{ secrets.AZURE_CLIENT_ID }}
azure-tenant-id: ${{ secrets.AZURE_TENANT_ID }}
azure-subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
os: ${{ runner.os }}
- name: Create .env for samples
run: |
echo "COPILOTSTUDIOAGENT__ENVIRONMENTID=$COPILOTSTUDIOAGENT__ENVIRONMENTID" >> .env
echo "COPILOTSTUDIOAGENT__SCHEMANAME=$COPILOTSTUDIOAGENT__SCHEMANAME" >> .env
echo "COPILOTSTUDIOAGENT__TENANTID=$COPILOTSTUDIOAGENT__TENANTID" >> .env
echo "COPILOTSTUDIOAGENT__AGENTAPPID=$COPILOTSTUDIOAGENT__AGENTAPPID" >> .env
- name: Run sample validation
run: |
cd scripts && uv run python -m sample_validation --subdir 02-agents/providers/copilotstudio --save-report --report-name 02-agents-copilotstudio
- name: Upload validation report
uses: actions/upload-artifact@v7
if: always()
with:
name: validation-report-02-agents-copilotstudio
path: python/samples/sample_validation/reports/
validate-02-agents-custom:
name: Validate 02-agents/providers/custom
runs-on: ubuntu-latest
environment: integration
defaults:
run:
working-directory: python
steps:
- uses: actions/checkout@v6
- name: Setup environment
uses: ./.github/actions/sample-validation-setup
with:
azure-client-id: ${{ secrets.AZURE_CLIENT_ID }}
azure-tenant-id: ${{ secrets.AZURE_TENANT_ID }}
azure-subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
os: ${{ runner.os }}
- name: Run sample validation
run: |
cd scripts && uv run python -m sample_validation --subdir 02-agents/providers/custom --save-report --report-name 02-agents-custom
- name: Upload validation report
uses: actions/upload-artifact@v7
if: always()
with:
name: validation-report-02-agents-custom
path: python/samples/sample_validation/reports/
validate-03-workflows:
name: Validate 03-workflows
@@ -121,6 +535,14 @@ jobs:
azure-subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
os: ${{ runner.os }}
- name: Create .env for samples
run: |
echo "AZURE_AI_PROJECT_ENDPOINT=$AZURE_AI_PROJECT_ENDPOINT" >> .env
echo "AZURE_AI_MODEL_DEPLOYMENT_NAME=$AZURE_AI_MODEL_DEPLOYMENT_NAME" >> .env
echo "AZURE_OPENAI_ENDPOINT=$AZURE_OPENAI_ENDPOINT" >> .env
echo "AZURE_OPENAI_CHAT_DEPLOYMENT_NAME=$AZURE_OPENAI_CHAT_DEPLOYMENT_NAME" >> .env
echo "AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME=$AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME" >> .env
- name: Run sample validation
run: |
cd scripts && uv run python -m sample_validation --subdir 03-workflows --save-report --report-name 03-workflows
@@ -130,7 +552,7 @@ jobs:
if: always()
with:
name: validation-report-03-workflows
path: python/scripts/sample_validation/reports/
path: python/samples/sample_validation/reports/
validate-04-hosting:
name: Validate 04-hosting
@@ -169,7 +591,7 @@ jobs:
if: always()
with:
name: validation-report-04-hosting
path: python/scripts/sample_validation/reports/
path: python/samples/sample_validation/reports/
validate-05-end-to-end:
name: Validate 05-end-to-end
@@ -213,7 +635,7 @@ jobs:
if: always()
with:
name: validation-report-05-end-to-end
path: python/scripts/sample_validation/reports/
path: python/samples/sample_validation/reports/
validate-autogen-migration:
name: Validate autogen-migration
@@ -244,6 +666,16 @@ jobs:
azure-subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
os: ${{ runner.os }}
- name: Create .env for samples
run: |
echo "AZURE_AI_PROJECT_ENDPOINT=$AZURE_AI_PROJECT_ENDPOINT" >> .env
echo "AZURE_AI_MODEL_DEPLOYMENT_NAME=$AZURE_AI_MODEL_DEPLOYMENT_NAME" >> .env
echo "AZURE_OPENAI_ENDPOINT=$AZURE_OPENAI_ENDPOINT" >> .env
echo "AZURE_OPENAI_CHAT_DEPLOYMENT_NAME=$AZURE_OPENAI_CHAT_DEPLOYMENT_NAME" >> .env
echo "OPENAI_API_KEY=$OPENAI_API_KEY" >> .env
echo "OPENAI_CHAT_MODEL_ID=$OPENAI_CHAT_MODEL_ID" >> .env
echo "OPENAI_RESPONSES_MODEL_ID=$OPENAI_RESPONSES_MODEL_ID" >> .env
- name: Run sample validation
run: |
cd scripts && uv run python -m sample_validation --subdir autogen-migration --save-report --report-name autogen-migration
@@ -253,7 +685,7 @@ jobs:
if: always()
with:
name: validation-report-autogen-migration
path: python/scripts/sample_validation/reports/
path: python/samples/sample_validation/reports/
validate-semantic-kernel-migration:
name: Validate semantic-kernel-migration
@@ -290,6 +722,21 @@ jobs:
azure-subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
os: ${{ runner.os }}
- name: Create .env for samples
run: |
echo "AZURE_AI_PROJECT_ENDPOINT=$AZURE_AI_PROJECT_ENDPOINT" >> .env
echo "AZURE_AI_MODEL_DEPLOYMENT_NAME=$AZURE_AI_MODEL_DEPLOYMENT_NAME" >> .env
echo "AZURE_OPENAI_ENDPOINT=$AZURE_OPENAI_ENDPOINT" >> .env
echo "AZURE_OPENAI_CHAT_DEPLOYMENT_NAME=$AZURE_OPENAI_CHAT_DEPLOYMENT_NAME" >> .env
echo "AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME=$AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME" >> .env
echo "OPENAI_API_KEY=$OPENAI_API_KEY" >> .env
echo "OPENAI_CHAT_MODEL_ID=$OPENAI_CHAT_MODEL_ID" >> .env
echo "OPENAI_RESPONSES_MODEL_ID=$OPENAI_RESPONSES_MODEL_ID" >> .env
echo "COPILOTSTUDIOAGENT__ENVIRONMENTID=$COPILOTSTUDIOAGENT__ENVIRONMENTID" >> .env
echo "COPILOTSTUDIOAGENT__SCHEMANAME=$COPILOTSTUDIOAGENT__SCHEMANAME" >> .env
echo "COPILOTSTUDIOAGENT__TENANTID=$COPILOTSTUDIOAGENT__TENANTID" >> .env
echo "COPILOTSTUDIOAGENT__AGENTAPPID=$COPILOTSTUDIOAGENT__AGENTAPPID" >> .env
- name: Run sample validation
run: |
cd scripts && uv run python -m sample_validation --subdir semantic-kernel-migration --save-report --report-name semantic-kernel-migration
@@ -299,4 +746,69 @@ jobs:
if: always()
with:
name: validation-report-semantic-kernel-migration
path: python/scripts/sample_validation/reports/
path: python/samples/sample_validation/reports/
aggregate-results:
name: Aggregate Results
runs-on: ubuntu-latest
if: always()
needs:
- validate-01-get-started
- validate-02-agents
- validate-02-agents-openai
- validate-02-agents-azure-openai
- validate-02-agents-azure-ai
- validate-02-agents-azure-ai-agent
- validate-02-agents-anthropic
- validate-02-agents-github-copilot
- validate-02-agents-amazon
- validate-02-agents-ollama
- validate-02-agents-foundry-local
- validate-02-agents-copilotstudio
- validate-02-agents-custom
- validate-03-workflows
- validate-04-hosting
- validate-05-end-to-end
- validate-autogen-migration
- validate-semantic-kernel-migration
steps:
- uses: actions/checkout@v6
- name: Download all validation reports
uses: actions/download-artifact@v7
with:
pattern: validation-report-*
path: reports/
merge-multiple: true
- name: Restore validation history
id: cache-restore
uses: actions/cache/restore@v4
with:
path: validation-history/
key: validation-history-${{ github.run_id }}
restore-keys: |
validation-history-
- name: Aggregate results and generate trend report
run: |
python3 python/scripts/sample_validation/aggregate.py \
reports/ \
validation-history/history.json \
trend-report.md
- name: Write trend report to job summary
run: cat trend-report.md >> "$GITHUB_STEP_SUMMARY"
- name: Save validation history
uses: actions/cache/save@v4
with:
path: validation-history/
key: validation-history-${{ github.run_id }}
- name: Upload trend report
uses: actions/upload-artifact@v7
if: always()
with:
name: validation-trend-report
path: trend-report.md
@@ -21,7 +21,7 @@ jobs:
steps:
- uses: actions/checkout@v6
- name: Download coverage report
uses: actions/download-artifact@v7
uses: actions/download-artifact@v8
with:
github-token: ${{ secrets.GH_ACTIONS_PR_WRITE }}
run-id: ${{ github.event.workflow_run.id }}
+1
View File
@@ -70,6 +70,7 @@
<PackageVersion Include="Microsoft.Extensions.AI.Evaluation.Safety" Version="10.3.0-preview.1.26109.11" />
<PackageVersion Include="Microsoft.Extensions.AI.OpenAI" Version="10.4.0" />
<PackageVersion Include="Microsoft.Extensions.Caching.Memory" Version="10.0.0" />
<PackageVersion Include="Microsoft.Extensions.Compliance.Abstractions" Version="10.4.0" />
<PackageVersion Include="Microsoft.Extensions.Configuration" Version="10.0.0" />
<PackageVersion Include="Microsoft.Extensions.Configuration.Binder" Version="10.0.0" />
<PackageVersion Include="Microsoft.Extensions.Configuration.EnvironmentVariables" Version="10.0.0" />
+4 -1
View File
@@ -309,7 +309,6 @@
<Project Path="samples/05-end-to-end/HostedAgents/AgentWithHostedMCP/AgentWithHostedMCP.csproj" />
<Project Path="samples/05-end-to-end/HostedAgents/AgentWithLocalTools/AgentWithLocalTools.csproj" />
<Project Path="samples/05-end-to-end/HostedAgents/AgentWithTextSearchRag/AgentWithTextSearchRag.csproj" />
<Project Path="samples/05-end-to-end/HostedAgents/AgentWithTools/AgentWithTools.csproj" />
<Project Path="samples/05-end-to-end/HostedAgents/FoundryMultiAgent/FoundryMultiAgent.csproj" />
<Project Path="samples/05-end-to-end/HostedAgents/FoundrySingleAgent/FoundrySingleAgent.csproj" />
</Folder>
@@ -453,6 +452,10 @@
<File Path="src/Shared/Samples/TextOutputHelperExtensions.cs" />
<File Path="src/Shared/Samples/XunitLogger.cs" />
</Folder>
<Folder Name="/Solution Items/src/Shared/Redaction/">
<File Path="src/Shared/Redaction/README.md" />
<File Path="src/Shared/Redaction/ReplacingRedactor.cs" />
</Folder>
<Folder Name="/Solution Items/src/Shared/Throw/">
<File Path="src/Shared/Throw/README.md" />
<File Path="src/Shared/Throw/Throw.cs" />
+3
View File
@@ -29,4 +29,7 @@
<ItemGroup Condition="'$(InjectSharedDiagnosticIds)' == 'true'">
<Compile Include="$(MSBuildThisFileDirectory)\..\..\src\Shared\DiagnosticIds\*.cs" LinkBase="Shared\DiagnosticIds" />
</ItemGroup>
<ItemGroup Condition="'$(InjectSharedRedaction)' == 'true'">
<Compile Include="$(MSBuildThisFileDirectory)\..\..\src\Shared\Redaction\*.cs" LinkBase="Shared\Redaction" />
</ItemGroup>
</Project>
@@ -36,10 +36,10 @@
</ItemGroup>
<ItemGroup>
<PackageReference Include="Azure.AI.AgentServer.AgentFramework" Version="1.0.0-beta.9" />
<PackageReference Include="Azure.AI.AgentServer.AgentFramework" Version="1.0.0-beta.11" />
<PackageReference Include="Azure.AI.OpenAI" Version="2.9.0-beta.1" />
<PackageReference Include="Azure.Identity" Version="1.17.1" />
<PackageReference Include="Microsoft.Agents.AI.OpenAI" Version="1.0.0-rc1" />
<PackageReference Include="Microsoft.Agents.AI.OpenAI" Version="1.0.0-rc4" />
</ItemGroup>
<!-- Add analyzers with compatible versions -->
@@ -1,4 +1,4 @@
<Project Sdk="Microsoft.NET.Sdk">
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
@@ -35,11 +35,10 @@
</ItemGroup>
<ItemGroup>
<PackageReference Include="Azure.AI.AgentServer.AgentFramework" Version="1.0.0-beta.9" />
<PackageReference Include="Azure.AI.OpenAI" Version="2.9.0-beta.1" />
<PackageReference Include="Azure.AI.AgentServer.AgentFramework" Version="1.0.0-beta.11" />
<PackageReference Include="Azure.AI.OpenAI" Version="2.8.0-beta.1" />
<PackageReference Include="Azure.Identity" Version="1.17.1" />
<PackageReference Include="Microsoft.Agents.AI.OpenAI" Version="1.0.0-rc1" />
<PackageReference Include="Microsoft.Extensions.AI.OpenAI" Version="10.4.0" />
<PackageReference Include="Microsoft.Agents.AI.OpenAI" Version="1.0.0-rc4" />
</ItemGroup>
<!-- Add analyzers with compatible versions -->
@@ -31,8 +31,7 @@ AITool mcpTool = new HostedMcpServerTool(serverName: "microsoft_learn", serverAd
AIAgent agent = new AzureOpenAIClient(
new Uri(endpoint),
new DefaultAzureCredential())
.GetResponsesClient()
.AsIChatClient(deploymentName)
.GetResponsesClient(deploymentName)
.AsAIAgent(
instructions: "You answer questions by searching the Microsoft Learn content only.",
name: "MicrosoftLearnAgent",
@@ -36,11 +36,11 @@
</ItemGroup>
<ItemGroup>
<PackageReference Include="Azure.AI.AgentServer.AgentFramework" Version="1.0.0-beta.9" />
<PackageReference Include="Azure.AI.Projects" Version="1.2.0-beta.5" />
<PackageReference Include="Azure.AI.AgentServer.AgentFramework" Version="1.0.0-beta.11" />
<PackageReference Include="Azure.AI.Projects" Version="2.0.0-beta.1" />
<PackageReference Include="Azure.AI.OpenAI" Version="2.9.0-beta.1" />
<PackageReference Include="Azure.Identity" Version="1.17.1" />
<PackageReference Include="Microsoft.Agents.AI.OpenAI" Version="1.0.0-rc1" />
<PackageReference Include="Microsoft.Agents.AI.OpenAI" Version="1.0.0-rc4" />
</ItemGroup>
<!-- Add analyzers with compatible versions -->
@@ -1,4 +1,4 @@
<Project Sdk="Microsoft.NET.Sdk">
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
@@ -35,10 +35,10 @@
</ItemGroup>
<ItemGroup>
<PackageReference Include="Azure.AI.AgentServer.AgentFramework" Version="1.0.0-beta.9" />
<PackageReference Include="Azure.AI.AgentServer.AgentFramework" Version="1.0.0-beta.11" />
<PackageReference Include="Azure.AI.OpenAI" Version="2.9.0-beta.1" />
<PackageReference Include="Azure.Identity" Version="1.17.1" />
<PackageReference Include="Microsoft.Agents.AI.OpenAI" Version="1.0.0-rc1" />
<PackageReference Include="Microsoft.Agents.AI.OpenAI" Version="1.0.0-rc4" />
</ItemGroup>
<!-- Add analyzers with compatible versions -->
@@ -1,68 +0,0 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFrameworks>net10.0</TargetFrameworks>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<!--
Disable central package management for this project.
This project requires explicit package references with versions specified inline rather than
inheriting them from Directory.Packages.props. This is necessary because a Docker image will
be created from this project, and the Docker build process only has access to this folder
and cannot access parent folders where Directory.Packages.props resides.
-->
<ManagePackageVersionsCentrally>false</ManagePackageVersionsCentrally>
</PropertyGroup>
<!--
Remove analyzer PackageReference items inherited from Directory.Packages.props.
Note: ManagePackageVersionsCentrally only controls PackageVersion items, not PackageReference items.
Directory.Packages.props contains both PackageVersion and PackageReference entries for analyzers,
and the PackageReference items are always inherited through MSBuild imports regardless of the
ManagePackageVersionsCentrally setting. We must explicitly remove them before adding our own versions.
-->
<ItemGroup>
<PackageReference Remove="Microsoft.CodeAnalysis.NetAnalyzers" />
<PackageReference Remove="Microsoft.VisualStudio.Threading.Analyzers" />
<PackageReference Remove="xunit.analyzers" />
<PackageReference Remove="Moq.Analyzers" />
<PackageReference Remove="Roslynator.Analyzers" />
<PackageReference Remove="Roslynator.CodeAnalysis.Analyzers" />
<PackageReference Remove="Roslynator.Formatting.Analyzers" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="Azure.AI.AgentServer.AgentFramework" Version="1.0.0-beta.9" />
<PackageReference Include="Azure.AI.OpenAI" Version="2.9.0-beta.1" />
<PackageReference Include="Azure.Identity" Version="1.17.1" />
<PackageReference Include="Microsoft.Agents.AI.OpenAI" Version="1.0.0-rc1" />
</ItemGroup>
<!-- Add analyzers with compatible versions -->
<ItemGroup>
<PackageReference Include="Microsoft.CodeAnalysis.NetAnalyzers" Version="10.0.100">
<PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference>
<PackageReference Include="Microsoft.VisualStudio.Threading.Analyzers" Version="17.14.15">
<PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference>
<PackageReference Include="Roslynator.Analyzers" Version="4.14.1">
<PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference>
<PackageReference Include="Roslynator.CodeAnalysis.Analyzers" Version="4.14.1">
<PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference>
<PackageReference Include="Roslynator.Formatting.Analyzers" Version="4.14.1">
<PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference>
</ItemGroup>
</Project>
@@ -1,20 +0,0 @@
# Build the application
FROM mcr.microsoft.com/dotnet/sdk:10.0-alpine AS build
WORKDIR /src
# Copy files from the current directory on the host to the working directory in the container
COPY . .
RUN dotnet restore
RUN dotnet build -c Release --no-restore
RUN dotnet publish -c Release --no-build -o /app -f net10.0
# Run the application
FROM mcr.microsoft.com/dotnet/aspnet:10.0-alpine AS final
WORKDIR /app
# Copy everything needed to run the app from the "build" stage.
COPY --from=build /app .
EXPOSE 8088
ENTRYPOINT ["dotnet", "AgentWithTools.dll"]
@@ -1,46 +0,0 @@
// Copyright (c) Microsoft. All rights reserved.
// This sample demonstrates how to use Foundry tools (MCP and code interpreter)
// with an AI agent hosted using the Azure AI AgentServer SDK.
using Azure.AI.AgentServer.AgentFramework.Extensions;
using Azure.AI.OpenAI;
using Azure.Identity;
using Microsoft.Agents.AI;
using Microsoft.Extensions.AI;
string openAiEndpoint = Environment.GetEnvironmentVariable("AZURE_OPENAI_ENDPOINT") ?? throw new InvalidOperationException("AZURE_OPENAI_ENDPOINT is not set.");
string deploymentName = Environment.GetEnvironmentVariable("AZURE_OPENAI_DEPLOYMENT_NAME") ?? "gpt-4o-mini";
string toolConnectionId = Environment.GetEnvironmentVariable("MCP_TOOL_CONNECTION_ID") ?? throw new InvalidOperationException("MCP_TOOL_CONNECTION_ID is not set.");
// WARNING: DefaultAzureCredential is convenient for development but requires careful consideration in production.
// In production, consider using a specific credential (e.g., ManagedIdentityCredential) to avoid
// latency issues, unintended credential probing, and potential security risks from fallback mechanisms.
DefaultAzureCredential credential = new();
IChatClient chatClient = new AzureOpenAIClient(new Uri(openAiEndpoint), credential)
.GetChatClient(deploymentName)
.AsIChatClient()
.AsBuilder()
.UseFoundryTools(new { type = "mcp", project_connection_id = toolConnectionId }, new { type = "code_interpreter" })
.UseOpenTelemetry(sourceName: "Agents", configure: (cfg) => cfg.EnableSensitiveData = true)
.Build();
AIAgent agent = chatClient.AsAIAgent(
name: "AgentWithTools",
instructions: @"You are a helpful assistant with access to tools for fetching Microsoft documentation.
IMPORTANT: When the user asks about Microsoft Learn articles or documentation:
1. You MUST use the microsoft_docs_fetch tool to retrieve the actual content
2. Do NOT rely on your training data
3. Always fetch the latest information from the provided URL
Available tools:
- microsoft_docs_fetch: Fetches and converts Microsoft Learn documentation
- microsoft_docs_search: Searches Microsoft/Azure documentation
- microsoft_code_sample_search: Searches for code examples")
.AsBuilder()
.UseOpenTelemetry(sourceName: "Agents", configure: (cfg) => cfg.EnableSensitiveData = true)
.Build();
await agent.RunAIAgentAsync(telemetrySourceName: "Agents");
@@ -1,45 +0,0 @@
# What this sample demonstrates
This sample demonstrates how to use Foundry tools with an AI agent via the `UseFoundryTools` extension. The agent is configured with two tool types: an MCP (Model Context Protocol) connection for fetching Microsoft Learn documentation and a code interpreter for running code when needed.
Key features:
- Configuring Foundry tools using `UseFoundryTools` with MCP and code interpreter
- Connecting to an external MCP tool via a Foundry project connection
- Using `DefaultAzureCredential` for Azure authentication
- OpenTelemetry instrumentation for both the chat client and agent
> For common prerequisites and setup instructions, see the [Hosted Agent Samples README](../README.md).
## Prerequisites
In addition to the common prerequisites:
1. An **Azure AI Foundry project** with a chat model deployed (e.g., `gpt-5.2`, `gpt-4o-mini`)
2. The **Azure AI Developer** role assigned on the Foundry resource (includes the `agents/write` data action required by `UseFoundryTools`)
3. An **MCP tool connection** configured in your Foundry project pointing to `https://learn.microsoft.com/api/mcp`
## Environment Variables
In addition to the common environment variables in the root README:
```powershell
# Your Azure AI Foundry project endpoint (required by UseFoundryTools)
$env:AZURE_AI_PROJECT_ENDPOINT="https://your-resource.services.ai.azure.com/api/projects/your-project"
# Chat model deployment name (defaults to gpt-4o-mini if not set)
$env:AZURE_OPENAI_DEPLOYMENT_NAME="gpt-4o-mini"
# The MCP tool connection name (just the name, not the full ARM resource ID)
$env:MCP_TOOL_CONNECTION_ID="SampleMCPTool"
```
## How It Works
1. An `AzureOpenAIClient` is created with `DefaultAzureCredential` and used to get a chat client
2. The chat client is wrapped with `UseFoundryTools` which registers two Foundry tool types:
- **MCP connection**: Connects to an external MCP server (Microsoft Learn) via the project connection name, providing documentation fetch and search capabilities
- **Code interpreter**: Allows the agent to execute code snippets when needed
3. `UseFoundryTools` resolves the connection using `AZURE_AI_PROJECT_ENDPOINT` internally
4. A `ChatClientAgent` is created with instructions guiding it to use the MCP tools for documentation queries
5. The agent is hosted using `RunAIAgentAsync` which exposes the OpenAI Responses-compatible API endpoint
@@ -1,31 +0,0 @@
name: AgentWithTools
displayName: "Agent with Tools"
description: >
An AI agent that uses Foundry tools (MCP and code interpreter) with Azure OpenAI.
The agent can fetch Microsoft Learn documentation and run code when needed.
metadata:
authors:
- Microsoft Agent Framework Team
tags:
- Azure AI AgentServer
- Microsoft Agent Framework
- Tools
- MCP
- Code Interpreter
template:
kind: hosted
name: AgentWithTools
protocols:
- protocol: responses
version: v1
environment_variables:
- name: AZURE_OPENAI_ENDPOINT
value: ${AZURE_OPENAI_ENDPOINT}
- name: AZURE_OPENAI_DEPLOYMENT_NAME
value: gpt-4o-mini
- name: MCP_TOOL_CONNECTION_ID
value: ${MCP_TOOL_CONNECTION_ID}
resources:
- name: "gpt-4o-mini"
kind: model
id: gpt-4o-mini
@@ -1,30 +0,0 @@
@host = http://localhost:8088
@endpoint = {{host}}/responses
### Health Check
GET {{host}}/readiness
### Simple string input
POST {{endpoint}}
Content-Type: application/json
{
"input": "Please use the microsoft_docs_fetch tool to fetch and summarize the Microsoft Learn article at https://learn.microsoft.com/azure/ai-services/openai/overview"
}
### Explicit input
POST {{endpoint}}
Content-Type: application/json
{
"input": [
{
"type": "message",
"role": "user",
"content": [
{
"type": "input_text",
"text": "Please use the microsoft_docs_fetch tool to fetch and summarize the Microsoft Learn article at https://learn.microsoft.com/azure/ai-services/openai/overview"
}
]
}
]
}
@@ -35,10 +35,10 @@
</ItemGroup>
<ItemGroup>
<PackageReference Include="Azure.AI.AgentServer.AgentFramework" Version="1.0.0-beta.9" />
<PackageReference Include="Azure.AI.AgentServer.AgentFramework" Version="1.0.0-beta.11" />
<PackageReference Include="Azure.AI.OpenAI" Version="2.9.0-beta.1" />
<PackageReference Include="Azure.Identity" Version="1.17.1" />
<PackageReference Include="Microsoft.Agents.AI.OpenAI" Version="1.0.0-rc1" />
<PackageReference Include="Microsoft.Agents.AI.OpenAI" Version="1.0.0-rc4" />
</ItemGroup>
<!-- Add analyzers with compatible versions -->
@@ -1,4 +1,4 @@
<Project Sdk="Microsoft.NET.Sdk">
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFrameworks>net10.0</TargetFrameworks>
@@ -33,12 +33,12 @@
</ItemGroup>
<ItemGroup>
<PackageReference Include="Azure.AI.AgentServer.AgentFramework" Version="1.0.0-beta.8" />
<PackageReference Include="Azure.AI.Projects" Version="1.2.0-beta.5" />
<PackageReference Include="Azure.AI.AgentServer.AgentFramework" Version="1.0.0-beta.11" />
<PackageReference Include="Azure.AI.Projects" Version="2.0.0-beta.1" />
<PackageReference Include="Azure.Identity" Version="1.17.1" />
<PackageReference Include="Microsoft.Agents.AI" Version="1.0.0-preview.251219.1" />
<PackageReference Include="Microsoft.Agents.AI.AzureAI" Version="1.0.0-preview.251219.1" />
<PackageReference Include="Microsoft.Agents.AI.Workflows" Version="1.0.0-preview.251219.1" />
<PackageReference Include="Microsoft.Agents.AI" Version="1.0.0-rc4" />
<PackageReference Include="Microsoft.Agents.AI.AzureAI" Version="1.0.0-rc4" />
<PackageReference Include="Microsoft.Agents.AI.Workflows" Version="1.0.0-rc4" />
<PackageReference Include="OpenTelemetry" Version="1.12.0" />
<PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.12.0" />
</ItemGroup>
@@ -41,7 +41,7 @@ try
.Build();
Console.WriteLine("Starting Writer-Reviewer Workflow Agent Server on http://localhost:8088");
await workflow.AsAgent().RunAIAgentAsync();
await workflow.AsAIAgent().RunAIAgentAsync();
}
finally
{
@@ -33,11 +33,11 @@
</ItemGroup>
<ItemGroup>
<PackageReference Include="Azure.AI.AgentServer.AgentFramework" Version="1.0.0-beta.8" />
<PackageReference Include="Azure.AI.Projects" Version="1.2.0-beta.5" />
<PackageReference Include="Azure.AI.AgentServer.AgentFramework" Version="1.0.0-beta.11" />
<PackageReference Include="Azure.AI.Projects" Version="2.0.0-beta.1" />
<PackageReference Include="Azure.Identity" Version="1.17.1" />
<PackageReference Include="Microsoft.Agents.AI" Version="1.0.0-preview.251219.1" />
<PackageReference Include="Microsoft.Agents.AI.AzureAI" Version="1.0.0-preview.251219.1" />
<PackageReference Include="Microsoft.Agents.AI" Version="1.0.0-rc4" />
<PackageReference Include="Microsoft.Agents.AI.AzureAI" Version="1.0.0-rc4" />
</ItemGroup>
<!-- Add analyzers with compatible versions -->
@@ -6,7 +6,6 @@ These samples demonstrate how to build and host AI agents using the [Azure AI Ag
| Sample | Description |
|--------|-------------|
| [`AgentWithTools`](./AgentWithTools/) | Foundry tools (MCP + code interpreter) via `UseFoundryTools` |
| [`AgentWithLocalTools`](./AgentWithLocalTools/) | Local C# function tool execution (Seattle hotel search) |
| [`AgentThreadAndHITL`](./AgentThreadAndHITL/) | Human-in-the-loop with `ApprovalRequiredAIFunction` and thread persistence |
| [`AgentWithHostedMCP`](./AgentWithHostedMCP/) | Hosted MCP server tool (Microsoft Learn search) |
@@ -40,19 +39,18 @@ Most samples require one or more of these environment variables:
|----------|---------|-------------|
| `AZURE_OPENAI_ENDPOINT` | Most samples | Your Azure OpenAI resource endpoint URL |
| `AZURE_OPENAI_DEPLOYMENT_NAME` | Most samples | Chat model deployment name (defaults to `gpt-4o-mini`) |
| `AZURE_AI_PROJECT_ENDPOINT` | AgentWithTools, AgentWithLocalTools, FoundryMultiAgent, FoundrySingleAgent | Azure AI Foundry project endpoint |
| `MCP_TOOL_CONNECTION_ID` | AgentWithTools | Foundry MCP tool connection name |
| `AZURE_AI_PROJECT_ENDPOINT` | AgentWithLocalTools, FoundryMultiAgent, FoundrySingleAgent | Azure AI Foundry project endpoint |
| `MODEL_DEPLOYMENT_NAME` | AgentWithLocalTools, FoundryMultiAgent, FoundrySingleAgent | Chat model deployment name (defaults to `gpt-4o-mini`) |
See each sample's README for the specific variables required.
## Azure AI Foundry Setup (for samples that use Foundry)
Some samples (`AgentWithTools`, `AgentWithLocalTools`) connect to an Azure AI Foundry project. If you're using these samples, you'll need additional setup.
Some samples (`AgentWithLocalTools`, `FoundrySingleAgent`, `FoundryMultiAgent`) connect to an Azure AI Foundry project. If you're using these samples, you'll need additional setup.
### Azure AI Developer Role
The `UseFoundryTools` extension requires the **Azure AI Developer** role on the Cognitive Services resource. Even if you created the project, you may not have this role by default.
Some Foundry operations require the **Azure AI Developer** role on the Cognitive Services resource. Even if you created the project, you may not have this role by default.
```powershell
az role assignment create `
@@ -65,23 +63,6 @@ az role assignment create `
For more details on permissions, see [Azure AI Foundry Permissions](https://aka.ms/FoundryPermissions).
### Creating an MCP Tool Connection
The `AgentWithTools` sample requires an MCP tool connection configured in your Foundry project:
1. Go to the [Azure AI Foundry portal](https://ai.azure.com)
2. Navigate to your project
3. Go to **Connected resources** → **+ New connection** → **Model Context Protocol tool**
4. Fill in:
- **Name**: `SampleMCPTool` (or any name you prefer)
- **Remote MCP Server endpoint**: `https://learn.microsoft.com/api/mcp`
- **Authentication**: `Unauthenticated`
5. Click **Connect**
The connection **name** (e.g., `SampleMCPTool`) is used as the `MCP_TOOL_CONNECTION_ID` environment variable.
> **Important**: Use only the connection **name**, not the full ARM resource ID.
## Running a Sample
Each sample runs as a standalone hosted agent on `http://localhost:8088/`:
@@ -110,14 +91,6 @@ Each sample includes a `Dockerfile` and `agent.yaml` for deployment. To deploy y
Assign the **Azure AI Developer** role to your user. See [Azure AI Developer Role](#azure-ai-developer-role) above.
### `Project connection ... was not found`
Make sure `MCP_TOOL_CONNECTION_ID` contains only the connection **name** (e.g., `SampleMCPTool`), not the full ARM resource ID path.
### `AZURE_AI_PROJECT_ENDPOINT must be set`
The `UseFoundryTools` extension requires `AZURE_AI_PROJECT_ENDPOINT`. Set it to your Foundry project endpoint (e.g., `https://your-resource.services.ai.azure.com/api/projects/your-project`).
### Multi-framework error when running `dotnet run`
If you see "Your project targets multiple frameworks", specify the framework:
@@ -2,10 +2,12 @@
using System;
using System.Collections.Generic;
using System.Diagnostics.CodeAnalysis;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.AI;
using Microsoft.Shared.DiagnosticIds;
using Microsoft.Shared.Diagnostics;
namespace Microsoft.Agents.AI;
@@ -147,6 +149,7 @@ public abstract class AIContextProvider
// Create a filtered context for ProvideAIContextAsync, filtering input messages
// to exclude non-external messages (e.g. chat history, other AI context provider messages).
#pragma warning disable MAAI001 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
var filteredContext = new InvokingContext(
context.Agent,
context.Session,
@@ -156,6 +159,7 @@ public abstract class AIContextProvider
Messages = inputContext.Messages is not null ? this.ProvideInputMessageFilter(inputContext.Messages) : null,
Tools = inputContext.Tools
});
#pragma warning restore MAAI001 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
var provided = await this.ProvideAIContextAsync(filteredContext, cancellationToken).ConfigureAwait(false);
@@ -294,7 +298,9 @@ public abstract class AIContextProvider
return default;
}
#pragma warning disable MAAI001 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
var subContext = new InvokedContext(context.Agent, context.Session, this.StoreInputRequestMessageFilter(context.RequestMessages), this.StoreInputResponseMessageFilter(context.ResponseMessages!));
#pragma warning restore MAAI001 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
return this.StoreAIContextAsync(subContext, cancellationToken);
}
@@ -372,6 +378,7 @@ public abstract class AIContextProvider
/// <param name="session">The session associated with the agent invocation.</param>
/// <param name="aiContext">The AI context to be used by the agent for this invocation.</param>
/// <exception cref="ArgumentNullException"><paramref name="agent"/> or <paramref name="aiContext"/> is <see langword="null"/>.</exception>
[Experimental(DiagnosticIds.Experiments.AgentsAIExperiments)]
public InvokingContext(
AIAgent agent,
AgentSession? session,
@@ -431,6 +438,7 @@ public abstract class AIContextProvider
/// that were used by the agent for this invocation.</param>
/// <param name="responseMessages">The response messages generated during this invocation.</param>
/// <exception cref="ArgumentNullException"><paramref name="agent"/>, <paramref name="requestMessages"/>, or <paramref name="responseMessages"/> is <see langword="null"/>.</exception>
[Experimental(DiagnosticIds.Experiments.AgentsAIExperiments)]
public InvokedContext(
AIAgent agent,
AgentSession? session,
@@ -2,10 +2,12 @@
using System;
using System.Collections.Generic;
using System.Diagnostics.CodeAnalysis;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.AI;
using Microsoft.Shared.DiagnosticIds;
using Microsoft.Shared.Diagnostics;
namespace Microsoft.Agents.AI;
@@ -250,7 +252,9 @@ public abstract class ChatHistoryProvider
return default;
}
#pragma warning disable MAAI001 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
var subContext = new InvokedContext(context.Agent, context.Session, this._storeInputRequestMessageFilter(context.RequestMessages), this._storeInputResponseMessageFilter(context.ResponseMessages!));
#pragma warning restore MAAI001 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
return this.StoreChatHistoryAsync(subContext, cancellationToken);
}
@@ -340,6 +344,7 @@ public abstract class ChatHistoryProvider
/// <param name="session">The session associated with the agent invocation.</param>
/// <param name="requestMessages">The messages to be used by the agent for this invocation.</param>
/// <exception cref="ArgumentNullException"><paramref name="requestMessages"/> is <see langword="null"/>.</exception>
[Experimental(DiagnosticIds.Experiments.AgentsAIExperiments)]
public InvokingContext(
AIAgent agent,
AgentSession? session,
@@ -399,6 +404,7 @@ public abstract class ChatHistoryProvider
/// that were used by the agent for this invocation.</param>
/// <param name="responseMessages">The response messages generated during this invocation.</param>
/// <exception cref="ArgumentNullException"><paramref name="agent"/>, <paramref name="requestMessages"/>, or <paramref name="responseMessages"/> is <see langword="null"/>.</exception>
[Experimental(DiagnosticIds.Experiments.AgentsAIExperiments)]
public InvokedContext(
AIAgent agent,
AgentSession? session,
@@ -2,10 +2,12 @@
using System;
using System.Collections.Generic;
using System.Diagnostics.CodeAnalysis;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.AI;
using Microsoft.Shared.DiagnosticIds;
using Microsoft.Shared.Diagnostics;
namespace Microsoft.Agents.AI;
@@ -49,12 +51,14 @@ public abstract class MessageAIContextProvider : AIContextProvider
{
// Call ProvideMessagesAsync directly to return only additional messages.
// The base AIContextProvider.InvokingCoreAsync handles merging with the original input and stamping.
#pragma warning disable MAAI001 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
return new AIContext
{
Messages = await this.ProvideMessagesAsync(
new InvokingContext(context.Agent, context.Session, context.AIContext.Messages ?? []),
cancellationToken).ConfigureAwait(false)
};
#pragma warning restore MAAI001 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
}
/// <summary>
@@ -109,10 +113,12 @@ public abstract class MessageAIContextProvider : AIContextProvider
// Create a filtered context for ProvideMessagesAsync, filtering input messages
// to exclude non-external messages (e.g. chat history, other AI context provider messages).
#pragma warning disable MAAI001 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
var filteredContext = new InvokingContext(
context.Agent,
context.Session,
this.ProvideInputMessageFilter(inputMessages));
#pragma warning restore MAAI001 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
var providedMessages = await this.ProvideMessagesAsync(filteredContext, cancellationToken).ConfigureAwait(false);
@@ -163,6 +169,7 @@ public abstract class MessageAIContextProvider : AIContextProvider
/// <param name="session">The session associated with the agent invocation.</param>
/// <param name="requestMessages">The messages to be used by the agent for this invocation.</param>
/// <exception cref="ArgumentNullException"><paramref name="agent"/> or <paramref name="requestMessages"/> is <see langword="null"/>.</exception>
[Experimental(DiagnosticIds.Experiments.AgentsAIExperiments)]
public InvokingContext(
AIAgent agent,
AgentSession? session,
@@ -20,8 +20,7 @@
<!-- NuGet Package Settings -->
<Title>Microsoft Agent Framework AzureAI Persistent Agents</Title>
<Description>Provides Microsoft Agent Framework support for Azure AI Persistent Agents.</Description>
<!-- Disabled until Azure.AI.Agents.Persistent targets ME.AI 10.4.0+ (https://github.com/microsoft/agent-framework/issues/4769) -->
<IsPackable>false</IsPackable>
</PropertyGroup>
</Project>
@@ -1,19 +1,3 @@
# Microsoft.Agents.AI.AzureAI.Persistent
Provides integration between the Microsoft Agent Framework and Azure AI Agents Persistent (`Azure.AI.Agents.Persistent`).
## ⚠️ Known Compatibility Limitation
The underlying `Azure.AI.Agents.Persistent` package (currently 1.2.0-beta.9) targets `Microsoft.Extensions.AI.Abstractions` 10.1.x and references types that were renamed in 10.4.0 (e.g., `McpServerToolApprovalResponseContent` → `ToolApprovalResponseContent`). This causes `TypeLoadException` at runtime when used with ME.AI 10.4.0+.
**Compatible versions:**
| Package | Compatible Version |
|---|---|
| `Azure.AI.Agents.Persistent` | 1.2.0-beta.9 (targets ME.AI 10.1.x) |
| `Microsoft.Extensions.AI.Abstractions` | ≤ 10.3.0 |
| `OpenAI` | ≤ 2.8.0 |
**Resolution:** An updated version of `Azure.AI.Agents.Persistent` targeting ME.AI 10.4.0+ is expected in 1.2.0-beta.10. The upstream fix is tracked in [Azure/azure-sdk-for-net#56929](https://github.com/Azure/azure-sdk-for-net/pull/56929).
**Tracking issue:** [microsoft/agent-framework#4769](https://github.com/microsoft/agent-framework/issues/4769)
@@ -10,6 +10,7 @@ using System.Threading;
using System.Threading.Tasks;
using Azure.AI.Projects;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.Compliance.Redaction;
using Microsoft.Extensions.Logging;
using Microsoft.Shared.DiagnosticIds;
using Microsoft.Shared.Diagnostics;
@@ -37,7 +38,7 @@ public sealed class FoundryMemoryProvider : AIContextProvider
private readonly string _memoryStoreName;
private readonly int _maxMemories;
private readonly int _updateDelay;
private readonly bool _enableSensitiveTelemetryData;
private readonly Redactor _redactor;
private readonly AIProjectClient _client;
private readonly ILogger<FoundryMemoryProvider>? _logger;
@@ -79,7 +80,7 @@ public sealed class FoundryMemoryProvider : AIContextProvider
this._memoryStoreName = memoryStoreName;
this._maxMemories = effectiveOptions.MaxMemories;
this._updateDelay = effectiveOptions.UpdateDelay;
this._enableSensitiveTelemetryData = effectiveOptions.EnableSensitiveTelemetryData;
this._redactor = effectiveOptions.EnableSensitiveTelemetryData ? NullRedactor.Instance : (effectiveOptions.Redactor ?? new ReplacingRedactor("<redacted>"));
}
/// <inheritdoc />
@@ -416,7 +417,7 @@ public sealed class FoundryMemoryProvider : AIContextProvider
private static bool IsAllowedRole(ChatRole role) =>
role == ChatRole.User || role == ChatRole.Assistant || role == ChatRole.System;
private string? SanitizeLogData(string? data) => this._enableSensitiveTelemetryData ? data : "<redacted>";
private string SanitizeLogData(string? data) => this._redactor.Redact(data);
/// <summary>
/// Represents the state of a <see cref="FoundryMemoryProvider"/> stored in the <see cref="AgentSession.StateBag"/>.
@@ -3,6 +3,7 @@
using System;
using System.Collections.Generic;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.Compliance.Redaction;
namespace Microsoft.Agents.AI.FoundryMemory;
@@ -37,8 +38,22 @@ public sealed class FoundryMemoryProviderOptions
/// Gets or sets a value indicating whether sensitive data such as user ids and user messages may appear in logs.
/// </summary>
/// <value>Defaults to <see langword="false"/>.</value>
/// <remarks>
/// When set to <see langword="true"/>, sensitive data is passed through to logs unchanged and any
/// configured <see cref="Redactor"/> is ignored. This property takes precedence over <see cref="Redactor"/>.
/// </remarks>
public bool EnableSensitiveTelemetryData { get; set; }
/// <summary>
/// Gets or sets a custom <see cref="Redactor"/> used to redact sensitive data in log output.
/// </summary>
/// <value>
/// When <see langword="null"/> (the default), sensitive data is replaced with a placeholder.
/// When set, this redactor is used to transform sensitive values before they are logged.
/// Ignored when <see cref="EnableSensitiveTelemetryData"/> is <see langword="true"/>.
/// </value>
public Redactor? Redactor { get; set; }
/// <summary>
/// Gets or sets the key used to store the provider state in the session's <see cref="AgentSessionStateBag"/>.
/// </summary>
@@ -8,6 +8,7 @@
<PropertyGroup>
<InjectSharedThrow>true</InjectSharedThrow>
<InjectSharedDiagnosticIds>true</InjectSharedDiagnosticIds>
<InjectSharedRedaction>true</InjectSharedRedaction>
<InjectExperimentalAttributeOnLegacy>true</InjectExperimentalAttributeOnLegacy>
<InjectTrimAttributesOnLegacy>true</InjectTrimAttributesOnLegacy>
</PropertyGroup>
@@ -20,6 +21,7 @@
<ItemGroup>
<PackageReference Include="Azure.AI.Projects" />
<PackageReference Include="Microsoft.Extensions.Compliance.Abstractions" />
<PackageReference Include="OpenAI" />
</ItemGroup>
@@ -8,6 +8,7 @@ using System.Text.Json.Serialization;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.Compliance.Redaction;
using Microsoft.Extensions.Logging;
using Microsoft.Shared.Diagnostics;
@@ -51,7 +52,7 @@ public sealed class Mem0Provider : MessageAIContextProvider
private readonly ProviderSessionState<State> _sessionState;
private IReadOnlyList<string>? _stateKeys;
private readonly string _contextPrompt;
private readonly bool _enableSensitiveTelemetryData;
private readonly Redactor _redactor;
private readonly Mem0Client _client;
private readonly ILogger<Mem0Provider>? _logger;
@@ -91,7 +92,7 @@ public sealed class Mem0Provider : MessageAIContextProvider
this._client = new Mem0Client(httpClient);
this._contextPrompt = options?.ContextPrompt ?? DefaultContextPrompt;
this._enableSensitiveTelemetryData = options?.EnableSensitiveTelemetryData ?? false;
this._redactor = options?.EnableSensitiveTelemetryData == true ? NullRedactor.Instance : (options?.Redactor ?? new ReplacingRedactor("<redacted>"));
}
/// <inheritdoc />
@@ -297,5 +298,5 @@ public sealed class Mem0Provider : MessageAIContextProvider
public Mem0ProviderScope SearchScope { get; }
}
private string? SanitizeLogData(string? data) => this._enableSensitiveTelemetryData ? data : "<redacted>";
private string SanitizeLogData(string? data) => this._redactor.Redact(data);
}
@@ -3,6 +3,7 @@
using System;
using System.Collections.Generic;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.Compliance.Redaction;
namespace Microsoft.Agents.AI.Mem0;
@@ -21,8 +22,22 @@ public sealed class Mem0ProviderOptions
/// Gets or sets a value indicating whether sensitive data such as user ids and user messages may appear in logs.
/// </summary>
/// <value>Defaults to <see langword="false"/>.</value>
/// <remarks>
/// When set to <see langword="true"/>, sensitive data is passed through to logs unchanged and any
/// configured <see cref="Redactor"/> is ignored. This property takes precedence over <see cref="Redactor"/>.
/// </remarks>
public bool EnableSensitiveTelemetryData { get; set; }
/// <summary>
/// Gets or sets a custom <see cref="Redactor"/> used to redact sensitive data in log output.
/// </summary>
/// <value>
/// When <see langword="null"/> (the default), sensitive data is replaced with a placeholder.
/// When set, this redactor is used to transform sensitive values before they are logged.
/// Ignored when <see cref="EnableSensitiveTelemetryData"/> is <see langword="true"/>.
/// </value>
public Redactor? Redactor { get; set; }
/// <summary>
/// Gets or sets the key used to store the provider state in the session's <see cref="AgentSessionStateBag"/>.
/// </summary>
@@ -6,6 +6,7 @@
<PropertyGroup>
<InjectSharedThrow>true</InjectSharedThrow>
<InjectSharedRedaction>true</InjectSharedRedaction>
<InjectTrimAttributesOnLegacy>true</InjectTrimAttributesOnLegacy>
</PropertyGroup>
@@ -23,6 +24,10 @@
<ProjectReference Include="..\Microsoft.Agents.AI.Abstractions\Microsoft.Agents.AI.Abstractions.csproj" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Extensions.Compliance.Abstractions" />
</ItemGroup>
<PropertyGroup>
<!-- NuGet Package Settings -->
<Title>Microsoft Agent Framework - Mem0 integration</Title>
@@ -7,6 +7,7 @@ using System.Linq.Expressions;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.Compliance.Redaction;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.VectorData;
using Microsoft.Shared.Diagnostics;
@@ -80,7 +81,7 @@ public sealed class ChatHistoryMemoryProvider : MessageAIContextProvider, IDispo
private readonly VectorStoreCollection<object, Dictionary<string, object?>> _collection;
private readonly int _maxResults;
private readonly string _contextPrompt;
private readonly bool _enableSensitiveTelemetryData;
private readonly Redactor _redactor;
private readonly ChatHistoryMemoryProviderOptions.SearchBehavior _searchTime;
private readonly string _toolName;
private readonly string _toolDescription;
@@ -118,7 +119,7 @@ public sealed class ChatHistoryMemoryProvider : MessageAIContextProvider, IDispo
options ??= new ChatHistoryMemoryProviderOptions();
this._maxResults = options.MaxResults.HasValue ? Throw.IfLessThanOrEqual(options.MaxResults.Value, 0) : DefaultMaxResults;
this._contextPrompt = options.ContextPrompt ?? DefaultContextPrompt;
this._enableSensitiveTelemetryData = options.EnableSensitiveTelemetryData;
this._redactor = options.EnableSensitiveTelemetryData ? NullRedactor.Instance : (options.Redactor ?? new ReplacingRedactor("<redacted>"));
this._searchTime = options.SearchTime;
this._logger = loggerFactory?.CreateLogger<ChatHistoryMemoryProvider>();
this._toolName = options.FunctionToolName ?? DefaultFunctionToolName;
@@ -485,7 +486,7 @@ public sealed class ChatHistoryMemoryProvider : MessageAIContextProvider, IDispo
GC.SuppressFinalize(this);
}
private string? SanitizeLogData(string? data) => this._enableSensitiveTelemetryData ? data : "<redacted>";
private string SanitizeLogData(string? data) => this._redactor.Redact(data);
/// <summary>
/// Rebinds a filter expression's body to use the specified shared parameter,
@@ -3,6 +3,7 @@
using System;
using System.Collections.Generic;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.Compliance.Redaction;
namespace Microsoft.Agents.AI;
@@ -46,8 +47,22 @@ public sealed class ChatHistoryMemoryProviderOptions
/// Gets or sets a value indicating whether sensitive data such as user ids and user messages may appear in logs.
/// </summary>
/// <value>Defaults to <see langword="false"/>.</value>
/// <remarks>
/// When set to <see langword="true"/>, sensitive data is passed through to logs unchanged and any
/// configured <see cref="Redactor"/> is ignored. This property takes precedence over <see cref="Redactor"/>.
/// </remarks>
public bool EnableSensitiveTelemetryData { get; set; }
/// <summary>
/// Gets or sets a custom <see cref="Redactor"/> used to redact sensitive data in log output.
/// </summary>
/// <value>
/// When <see langword="null"/> (the default), sensitive data is replaced with a placeholder.
/// When set, this redactor is used to transform sensitive values before they are logged.
/// Ignored when <see cref="EnableSensitiveTelemetryData"/> is <see langword="true"/>.
/// </value>
public Redactor? Redactor { get; set; }
/// <summary>
/// Gets or sets the key used to store provider state in the <see cref="AgentSession.StateBag"/>.
/// </summary>
@@ -8,6 +8,7 @@
<PropertyGroup>
<InjectSharedThrow>true</InjectSharedThrow>
<InjectSharedDiagnosticIds>true</InjectSharedDiagnosticIds>
<InjectSharedRedaction>true</InjectSharedRedaction>
<InjectDiagnosticClassesOnLegacy>true</InjectDiagnosticClassesOnLegacy>
<InjectExperimentalAttributeOnLegacy>true</InjectExperimentalAttributeOnLegacy>
<InjectTrimAttributesOnLegacy>true</InjectTrimAttributesOnLegacy>
@@ -22,6 +23,7 @@
<ItemGroup>
<PackageReference Include="Microsoft.Extensions.AI" />
<PackageReference Include="Microsoft.Extensions.Compliance.Abstractions" />
<PackageReference Include="Microsoft.Extensions.VectorData.Abstractions" />
<PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" />
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" />
@@ -7,6 +7,7 @@ using System.Text;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.Compliance.Redaction;
using Microsoft.Extensions.Logging;
using Microsoft.Shared.Diagnostics;
@@ -62,6 +63,7 @@ public sealed class TextSearchProvider : MessageAIContextProvider
private readonly string _contextPrompt;
private readonly string _citationsPrompt;
private readonly Func<IList<TextSearchResult>, string>? _contextFormatter;
private readonly Redactor _redactor;
/// <summary>
/// Initializes a new instance of the <see cref="TextSearchProvider"/> class.
@@ -89,6 +91,7 @@ public sealed class TextSearchProvider : MessageAIContextProvider
this._contextPrompt = options?.ContextPrompt ?? DefaultContextPrompt;
this._citationsPrompt = options?.CitationsPrompt ?? DefaultCitationsPrompt;
this._contextFormatter = options?.ContextFormatter;
this._redactor = options?.EnableSensitiveTelemetryData == true ? NullRedactor.Instance : (options?.Redactor ?? new ReplacingRedactor("<redacted>"));
// Create the on-demand search tool (only used if behavior is OnDemandFunctionCalling)
this._tools =
@@ -180,7 +183,7 @@ public sealed class TextSearchProvider : MessageAIContextProvider
if (this._logger?.IsEnabled(LogLevel.Trace) is true)
{
this._logger.LogTrace("TextSearchProvider: Search Results\nInput:{Input}\nOutput:{MessageText}", input, formatted);
this._logger.LogTrace("TextSearchProvider: Search Results\nInput:{Input}\nOutput:{MessageText}", this.SanitizeLogData(input), this.SanitizeLogData(formatted));
}
return [new ChatMessage(ChatRole.User, formatted)];
@@ -249,7 +252,7 @@ public sealed class TextSearchProvider : MessageAIContextProvider
if (this._logger.IsEnabled(LogLevel.Trace))
{
this._logger.LogTrace("TextSearchProvider Input:{UserQuestion}\nOutput:{MessageText}", userQuestion, outputText);
this._logger.LogTrace("TextSearchProvider Input:{UserQuestion}\nOutput:{MessageText}", this.SanitizeLogData(userQuestion), this.SanitizeLogData(outputText));
}
}
@@ -325,6 +328,8 @@ public sealed class TextSearchProvider : MessageAIContextProvider
public object? RawRepresentation { get; set; }
}
private string SanitizeLogData(string? data) => this._redactor.Redact(data);
/// <summary>
/// Represents the per-session state of a <see cref="TextSearchProvider"/> stored in the <see cref="AgentSession.StateBag"/>.
/// </summary>
@@ -3,6 +3,7 @@
using System;
using System.Collections.Generic;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.Compliance.Redaction;
namespace Microsoft.Agents.AI;
@@ -117,6 +118,26 @@ public sealed class TextSearchProviderOptions
/// </value>
public List<ChatRole>? RecentMessageRolesIncluded { get; set; }
/// <summary>
/// Gets or sets a value indicating whether sensitive data such as user queries and search results may appear in logs.
/// </summary>
/// <value>Defaults to <see langword="false"/>.</value>
/// <remarks>
/// When set to <see langword="true"/>, sensitive data is passed through to logs unchanged and any
/// configured <see cref="Redactor"/> is ignored. This property takes precedence over <see cref="Redactor"/>.
/// </remarks>
public bool EnableSensitiveTelemetryData { get; set; }
/// <summary>
/// Gets or sets a custom <see cref="Redactor"/> used to redact sensitive data in log output.
/// </summary>
/// <value>
/// When <see langword="null"/> (the default), sensitive data is replaced with a placeholder.
/// When set, this redactor is used to transform sensitive values before they are logged.
/// Ignored when <see cref="EnableSensitiveTelemetryData"/> is <see langword="true"/>.
/// </value>
public Redactor? Redactor { get; set; }
/// <summary>
/// Behavior choices for the provider.
/// </summary>
+30
View File
@@ -0,0 +1,30 @@
# Redaction
Log data redaction utilities built on `Microsoft.Extensions.Compliance.Redaction.Redactor`.
Provides `ReplacingRedactor`, an internal `Redactor` implementation that replaces
any input with a fixed replacement string (e.g. `"<redacted>"`).
To use this in your project, add the following to your `.csproj` file:
```xml
<PropertyGroup>
<InjectSharedRedaction>true</InjectSharedRedaction>
</PropertyGroup>
```
You will also need to add a package reference to `Microsoft.Extensions.Compliance.Abstractions`:
```xml
<ItemGroup>
<PackageReference Include="Microsoft.Extensions.Compliance.Abstractions" />
</ItemGroup>
```
And finally, this also depends on the shared Throw class, so when using redaction, InjectSharedThrow should also be enabled:
```xml
<PropertyGroup>
<InjectSharedThrow>true</InjectSharedThrow>
</PropertyGroup>
```
@@ -0,0 +1,35 @@
// Copyright (c) Microsoft. All rights reserved.
using System;
using Microsoft.Extensions.Compliance.Redaction;
using Microsoft.Shared.Diagnostics;
namespace Microsoft.Agents.AI;
/// <summary>
/// A <see cref="Redactor"/> that replaces the entire input with a fixed replacement string.
/// </summary>
internal sealed class ReplacingRedactor : Redactor
{
private readonly string _replacementText;
/// <summary>
/// Initializes a new instance of the <see cref="ReplacingRedactor"/> class.
/// </summary>
/// <param name="replacementText">The text to substitute for any input value.</param>
/// <exception cref="ArgumentNullException">Thrown when <paramref name="replacementText"/> is <see langword="null"/>.</exception>
public ReplacingRedactor(string replacementText)
{
this._replacementText = Throw.IfNull(replacementText);
}
/// <inheritdoc />
public override int GetRedactedLength(ReadOnlySpan<char> input) => this._replacementText.Length;
/// <inheritdoc />
public override int Redact(ReadOnlySpan<char> source, Span<char> destination)
{
this._replacementText.AsSpan().CopyTo(destination);
return this._replacementText.Length;
}
}
@@ -4,10 +4,7 @@ using AgentConformance.IntegrationTests;
namespace AzureAIAgentsPersistent.IntegrationTests;
// Disabled: Azure.AI.Agents.Persistent 1.2.0-beta.9 references McpServerToolApprovalResponseContent
// which was removed in ME.AI 10.4.0. Re-enable once Persistent targets ME.AI 10.4.0+ (expected in 1.2.0-beta.10).
// Tracking: https://github.com/microsoft/agent-framework/issues/4769
[Trait("Category", "IntegrationDisabled")]
[Trait("Category", "Integration")]
public class AzureAIAgentsChatClientAgentRunStreamingTests() : ChatClientAgentRunStreamingTests<AzureAIAgentsPersistentFixture>(() => new())
{
}
@@ -4,10 +4,7 @@ using AgentConformance.IntegrationTests;
namespace AzureAIAgentsPersistent.IntegrationTests;
// Disabled: Azure.AI.Agents.Persistent 1.2.0-beta.9 references McpServerToolApprovalResponseContent
// which was removed in ME.AI 10.4.0. Re-enable once Persistent targets ME.AI 10.4.0+ (expected in 1.2.0-beta.10).
// Tracking: https://github.com/microsoft/agent-framework/issues/4769
[Trait("Category", "IntegrationDisabled")]
[Trait("Category", "Integration")]
public class AzureAIAgentsChatClientAgentRunTests() : ChatClientAgentRunTests<AzureAIAgentsPersistentFixture>(() => new())
{
}
@@ -14,10 +14,7 @@ using Shared.IntegrationTests;
namespace AzureAIAgentsPersistent.IntegrationTests;
// Disabled: Azure.AI.Agents.Persistent 1.2.0-beta.9 references McpServerToolApprovalResponseContent
// which was removed in ME.AI 10.4.0. Re-enable once Persistent targets ME.AI 10.4.0+ (expected in 1.2.0-beta.10).
// Tracking: https://github.com/microsoft/agent-framework/issues/4769
[Trait("Category", "IntegrationDisabled")]
[Trait("Category", "Integration")]
public class AzureAIAgentsPersistentCreateTests
{
private const string SkipCodeInterpreterReason = "Azure AI Code Interpreter intermittently fails to execute uploaded files in CI";
@@ -4,10 +4,7 @@ using AgentConformance.IntegrationTests;
namespace AzureAIAgentsPersistent.IntegrationTests;
// Disabled: Azure.AI.Agents.Persistent 1.2.0-beta.9 references McpServerToolApprovalResponseContent
// which was removed in ME.AI 10.4.0. Re-enable once Persistent targets ME.AI 10.4.0+ (expected in 1.2.0-beta.10).
// Tracking: https://github.com/microsoft/agent-framework/issues/4769
[Trait("Category", "IntegrationDisabled")]
[Trait("Category", "Integration")]
public class AzureAIAgentsPersistentRunStreamingTests() : RunStreamingTests<AzureAIAgentsPersistentFixture>(() => new())
{
}
@@ -4,10 +4,7 @@ using AgentConformance.IntegrationTests;
namespace AzureAIAgentsPersistent.IntegrationTests;
// Disabled: Azure.AI.Agents.Persistent 1.2.0-beta.9 references McpServerToolApprovalResponseContent
// which was removed in ME.AI 10.4.0. Re-enable once Persistent targets ME.AI 10.4.0+ (expected in 1.2.0-beta.10).
// Tracking: https://github.com/microsoft/agent-framework/issues/4769
[Trait("Category", "IntegrationDisabled")]
[Trait("Category", "Integration")]
public class AzureAIAgentsPersistentRunTests() : RunTests<AzureAIAgentsPersistentFixture>(() => new())
{
}
@@ -5,10 +5,7 @@ using AgentConformance.IntegrationTests;
namespace AzureAIAgentsPersistent.IntegrationTests;
// Disabled: Azure.AI.Agents.Persistent 1.2.0-beta.9 references McpServerToolApprovalResponseContent
// which was removed in ME.AI 10.4.0. Re-enable once Persistent targets ME.AI 10.4.0+ (expected in 1.2.0-beta.10).
// Tracking: https://github.com/microsoft/agent-framework/issues/4769
[Trait("Category", "IntegrationDisabled")]
[Trait("Category", "Integration")]
public class AzureAIAgentsPersistentStructuredOutputRunTests() : StructuredOutputRunTests<AzureAIAgentsPersistentFixture>(() => new())
{
private const string SkipReason = "Fails intermittently on the build agent/CI";
@@ -148,11 +148,15 @@ public sealed class Mem0ProviderTests : IDisposable
}
[Theory]
[InlineData(false, false, 4)]
[InlineData(true, false, 4)]
[InlineData(false, true, 2)]
[InlineData(true, true, 2)]
public async Task InvokingAsync_LogsUserIdBasedOnEnableSensitiveTelemetryDataAsync(bool enableSensitiveTelemetryData, bool requestThrows, int expectedLogInvocations)
[InlineData(false, false, false, 4)]
[InlineData(false, false, true, 4)]
[InlineData(true, false, false, 4)]
[InlineData(true, false, true, 4)]
[InlineData(false, true, false, 2)]
[InlineData(false, true, true, 2)]
[InlineData(true, true, false, 2)]
[InlineData(true, true, true, 2)]
public async Task InvokingAsync_RedactsLogDataBasedOnOptionsAsync(bool enableSensitiveTelemetryData, bool requestThrows, bool useCustomRedactor, int expectedLogInvocations)
{
// Arrange
if (requestThrows)
@@ -171,7 +175,11 @@ public sealed class Mem0ProviderTests : IDisposable
ThreadId = "session",
UserId = "user"
};
var options = new Mem0ProviderOptions { EnableSensitiveTelemetryData = enableSensitiveTelemetryData };
var options = new Mem0ProviderOptions
{
EnableSensitiveTelemetryData = enableSensitiveTelemetryData,
Redactor = useCustomRedactor ? new ReplacingRedactor("***") : null
};
var mockSession = new TestAgentSession();
var sut = new Mem0Provider(this._httpClient, _ => new Mem0Provider.State(storageScope), options: options, loggerFactory: this._loggerFactoryMock.Object);
@@ -180,7 +188,8 @@ public sealed class Mem0ProviderTests : IDisposable
// Act
await sut.InvokingAsync(invokingContext, CancellationToken.None);
// Assert
// Assert — EnableSensitiveTelemetryData takes precedence over Redactor
string expectedRedaction = enableSensitiveTelemetryData ? "user" : (useCustomRedactor ? "***" : "<redacted>");
Assert.Equal(expectedLogInvocations, this._loggerMock.Invocations.Count);
foreach (var logInvocation in this._loggerMock.Invocations)
{
@@ -191,18 +200,18 @@ public sealed class Mem0ProviderTests : IDisposable
var state = Assert.IsType<IReadOnlyList<KeyValuePair<string, object?>>>(logInvocation.Arguments[2], exactMatch: false);
var userIdValue = state.First(kvp => kvp.Key == "UserId").Value;
Assert.Equal(enableSensitiveTelemetryData ? "user" : "<redacted>", userIdValue);
Assert.Equal(expectedRedaction, userIdValue);
var inputValue = state.FirstOrDefault(kvp => kvp.Key == "Input").Value;
if (inputValue != null)
{
Assert.Equal(enableSensitiveTelemetryData ? "Who am I?" : "<redacted>", inputValue);
Assert.Equal(enableSensitiveTelemetryData ? "Who am I?" : expectedRedaction, inputValue);
}
var messageTextValue = state.FirstOrDefault(kvp => kvp.Key == "MessageText").Value;
if (messageTextValue != null)
{
Assert.Equal(enableSensitiveTelemetryData ? "## Memories\nConsider the following memories when answering user questions:\nName is Caoimhe" : "<redacted>", messageTextValue);
Assert.Equal(enableSensitiveTelemetryData ? "## Memories\nConsider the following memories when answering user questions:\nName is Caoimhe" : expectedRedaction, messageTextValue);
}
}
}
@@ -85,7 +85,8 @@ public sealed class TextSearchProviderTests
{
SearchTime = TextSearchProviderOptions.TextSearchBehavior.BeforeAIInvoke,
ContextPrompt = overrideContextPrompt,
CitationsPrompt = overrideCitationsPrompt
CitationsPrompt = overrideCitationsPrompt,
EnableSensitiveTelemetryData = true
};
var provider = new TextSearchProvider(SearchDelegateAsync, options, withLogging ? this._loggerFactoryMock.Object : null);
@@ -164,6 +165,65 @@ public sealed class TextSearchProviderTests
}
}
[Theory]
[InlineData(false, false)]
[InlineData(false, true)]
[InlineData(true, false)]
[InlineData(true, true)]
public async Task InvokingAsync_RedactsLogDataBasedOnOptionsAsync(bool enableSensitiveTelemetryData, bool useCustomRedactor)
{
// Arrange
List<TextSearchProvider.TextSearchResult> results =
[
new() { SourceName = "Doc1", SourceLink = "http://example.com/doc1", Text = "Content of Doc1" }
];
Task<IEnumerable<TextSearchProvider.TextSearchResult>> SearchDelegateAsync(string input, CancellationToken ct)
{
return Task.FromResult<IEnumerable<TextSearchProvider.TextSearchResult>>(results);
}
var options = new TextSearchProviderOptions
{
SearchTime = TextSearchProviderOptions.TextSearchBehavior.BeforeAIInvoke,
EnableSensitiveTelemetryData = enableSensitiveTelemetryData,
Redactor = useCustomRedactor ? new ReplacingRedactor("***") : null
};
var provider = new TextSearchProvider(SearchDelegateAsync, options, this._loggerFactoryMock.Object);
var invokingContext = new AIContextProvider.InvokingContext(
s_mockAgent,
new TestAgentSession(),
new AIContext { Messages = new List<ChatMessage> { new(ChatRole.User, "Sample user question?") } });
// Act
await provider.InvokingAsync(invokingContext, CancellationToken.None);
// Assert — EnableSensitiveTelemetryData takes precedence over Redactor
var traceInvocation = this._loggerMock.Invocations
.Where(i => i.Method.Name == nameof(ILogger.Log))
.FirstOrDefault(i => (LogLevel)i.Arguments[0]! == LogLevel.Trace);
Assert.NotNull(traceInvocation);
var state = Assert.IsType<IReadOnlyList<KeyValuePair<string, object?>>>(traceInvocation.Arguments[2], exactMatch: false);
var inputValue = state.First(kvp => kvp.Key == "Input").Value;
var messageTextValue = state.First(kvp => kvp.Key == "MessageText").Value;
if (enableSensitiveTelemetryData)
{
// EnableSensitiveTelemetryData=true: raw data passes through regardless of Redactor
Assert.Equal("Sample user question?", inputValue);
Assert.Contains("Content of Doc1", messageTextValue?.ToString()!);
}
else
{
// EnableSensitiveTelemetryData=false: custom redactor or default placeholder
string expectedRedaction = useCustomRedactor ? "***" : "<redacted>";
Assert.Equal(expectedRedaction, inputValue);
Assert.Equal(expectedRedaction, messageTextValue);
}
}
[Theory]
[InlineData(null, null, "Search", "Allows searching for additional information to help answer the user question.")]
[InlineData("CustomSearch", "CustomDescription", "CustomSearch", "CustomDescription")]
@@ -270,16 +270,21 @@ public class ChatHistoryMemoryProviderTests
}
[Theory]
[InlineData(false, false, 0)]
[InlineData(true, false, 0)]
[InlineData(false, true, 2)]
[InlineData(true, true, 2)]
public async Task InvokedAsync_LogsUserIdBasedOnEnableSensitiveTelemetryDataAsync(bool enableSensitiveTelemetryData, bool requestThrows, int expectedLogInvocations)
[InlineData(false, false, false, 0)]
[InlineData(false, false, true, 0)]
[InlineData(true, false, false, 0)]
[InlineData(true, false, true, 0)]
[InlineData(false, true, false, 2)]
[InlineData(false, true, true, 2)]
[InlineData(true, true, false, 2)]
[InlineData(true, true, true, 2)]
public async Task InvokedAsync_RedactsLogDataBasedOnOptionsAsync(bool enableSensitiveTelemetryData, bool requestThrows, bool useCustomRedactor, int expectedLogInvocations)
{
// Arrange
var options = new ChatHistoryMemoryProviderOptions
{
EnableSensitiveTelemetryData = enableSensitiveTelemetryData
EnableSensitiveTelemetryData = enableSensitiveTelemetryData,
Redactor = useCustomRedactor ? new ReplacingRedactor("***") : null
};
if (requestThrows)
@@ -309,7 +314,7 @@ public class ChatHistoryMemoryProviderTests
// Act
await provider.InvokedAsync(invokedContext, CancellationToken.None);
// Assert
// Assert — EnableSensitiveTelemetryData takes precedence over Redactor
Assert.Equal(expectedLogInvocations, this._loggerMock.Invocations.Count);
foreach (var logInvocation in this._loggerMock.Invocations)
{
@@ -320,7 +325,8 @@ public class ChatHistoryMemoryProviderTests
var state = Assert.IsType<IReadOnlyList<KeyValuePair<string, object?>>>(logInvocation.Arguments[2], exactMatch: false);
var userIdValue = state.First(kvp => kvp.Key == "UserId").Value;
Assert.Equal(enableSensitiveTelemetryData ? "user1" : "<redacted>", userIdValue);
string expectedRedaction = enableSensitiveTelemetryData ? "user1" : (useCustomRedactor ? "***" : "<redacted>");
Assert.Equal(expectedRedaction, userIdValue);
}
}
@@ -526,17 +532,22 @@ public class ChatHistoryMemoryProviderTests
}
[Theory]
[InlineData(false, false, 2)]
[InlineData(true, false, 2)]
[InlineData(false, true, 2)]
[InlineData(true, true, 2)]
public async Task InvokingAsync_LogsUserIdBasedOnEnableSensitiveTelemetryDataAsync(bool enableSensitiveTelemetryData, bool requestThrows, int expectedLogInvocations)
[InlineData(false, false, false, 2)]
[InlineData(false, false, true, 2)]
[InlineData(true, false, false, 2)]
[InlineData(true, false, true, 2)]
[InlineData(false, true, false, 2)]
[InlineData(false, true, true, 2)]
[InlineData(true, true, false, 2)]
[InlineData(true, true, true, 2)]
public async Task InvokingAsync_RedactsLogDataBasedOnOptionsAsync(bool enableSensitiveTelemetryData, bool requestThrows, bool useCustomRedactor, int expectedLogInvocations)
{
// Arrange
var options = new ChatHistoryMemoryProviderOptions
{
SearchTime = ChatHistoryMemoryProviderOptions.SearchBehavior.BeforeAIInvoke,
EnableSensitiveTelemetryData = enableSensitiveTelemetryData
EnableSensitiveTelemetryData = enableSensitiveTelemetryData,
Redactor = useCustomRedactor ? new ReplacingRedactor("***") : null
};
var scope = new ChatHistoryMemoryProviderScope
@@ -578,7 +589,8 @@ public class ChatHistoryMemoryProviderTests
// Act
await provider.InvokingAsync(invokingContext, CancellationToken.None);
// Assert
// Assert — EnableSensitiveTelemetryData takes precedence over Redactor
string expectedRedaction = enableSensitiveTelemetryData ? "user1" : (useCustomRedactor ? "***" : "<redacted>");
Assert.Equal(expectedLogInvocations, this._loggerMock.Invocations.Count);
foreach (var logInvocation in this._loggerMock.Invocations)
{
@@ -589,18 +601,18 @@ public class ChatHistoryMemoryProviderTests
var state = Assert.IsType<IReadOnlyList<KeyValuePair<string, object?>>>(logInvocation.Arguments[2], exactMatch: false);
var userIdValue = state.First(kvp => kvp.Key == "UserId").Value;
Assert.Equal(enableSensitiveTelemetryData ? "user1" : "<redacted>", userIdValue);
Assert.Equal(expectedRedaction, userIdValue);
var inputValue = state.FirstOrDefault(kvp => kvp.Key == "Input").Value;
if (inputValue != null)
{
Assert.Equal(enableSensitiveTelemetryData ? "Who am I?" : "<redacted>", inputValue);
Assert.Equal(enableSensitiveTelemetryData ? "Who am I?" : expectedRedaction, inputValue);
}
var messageTextValue = state.FirstOrDefault(kvp => kvp.Key == "MessageText").Value;
if (messageTextValue != null)
{
Assert.Equal(enableSensitiveTelemetryData ? "## Memories\nConsider the following memories when answering user questions:\nName is Caoimhe" : "<redacted>", messageTextValue);
Assert.Equal(enableSensitiveTelemetryData ? "## Memories\nConsider the following memories when answering user questions:\nName is Caoimhe" : expectedRedaction, messageTextValue);
}
}
}
@@ -466,9 +466,15 @@ class FunctionTool(SerializationMixin):
if func is None:
return create_model(f"{self.name}_input")
sig = inspect.signature(func)
try:
type_hints = typing.get_type_hints(func, include_extras=True)
except Exception:
type_hints = {}
fields: dict[str, Any] = {
pname: (
_parse_annotation(param.annotation) if param.annotation is not inspect.Parameter.empty else str,
_parse_annotation(type_hints.get(pname, param.annotation))
if type_hints.get(pname, param.annotation) is not inspect.Parameter.empty
else str,
param.default if param.default is not inspect.Parameter.empty else ...,
)
for pname, param in sig.parameters.items()
@@ -0,0 +1,134 @@
# Copyright (c) Microsoft. All rights reserved.
"""Tests for @tool with PEP 563 (from __future__ import annotations).
When ``from __future__ import annotations`` is active, all annotations
become strings. _resolve_input_model must resolve them via
typing.get_type_hints() before passing them to Pydantic's create_model.
"""
from __future__ import annotations
from pydantic import BaseModel
from agent_framework import tool
from agent_framework._middleware import FunctionInvocationContext
class SearchConfig(BaseModel):
max_results: int = 10
def test_tool_with_context_parameter():
"""FunctionInvocationContext parameter is excluded from schema under PEP 563."""
@tool
def get_weather(location: str, ctx: FunctionInvocationContext) -> str:
"""Get the weather for a given location."""
return f"Weather in {location}"
params = get_weather.parameters()
assert "ctx" not in params.get("properties", {})
assert "location" in params["properties"]
def test_tool_with_context_parameter_first():
"""FunctionInvocationContext as the first parameter is excluded under PEP 563."""
@tool
def get_weather(ctx: FunctionInvocationContext, location: str) -> str:
"""Get the weather for a given location."""
return f"Weather in {location}"
params = get_weather.parameters()
assert "ctx" not in params.get("properties", {})
assert "location" in params["properties"]
def test_tool_with_optional_param():
"""Optional[int] is resolved to the actual type, not left as a string."""
@tool
def search(query: str, limit: int | None = None) -> str:
"""Search for something."""
return query
params = search.parameters()
assert params["properties"]["query"]["type"] == "string"
limit_schema = params["properties"]["limit"]
limit_types = {t["type"] for t in limit_schema["anyOf"]}
assert limit_types == {"integer", "null"}
def test_tool_with_optional_param_and_context():
"""Optional param + FunctionInvocationContext both work under PEP 563."""
@tool
def search(query: str, limit: int | None = None, ctx: FunctionInvocationContext | None = None) -> str:
"""Search for something."""
return query
params = search.parameters()
assert params["properties"]["query"]["type"] == "string"
limit_schema = params["properties"]["limit"]
limit_types = {t["type"] for t in limit_schema["anyOf"]}
assert limit_types == {"integer", "null"}
assert "ctx" not in params.get("properties", {})
def test_tool_with_optional_custom_type():
"""Optional[CustomType] is resolved under PEP 563 (original bug pattern)."""
@tool
def search(query: str, config: SearchConfig | None = None) -> str:
"""Search for something."""
return query
params = search.parameters()
assert params["properties"]["query"]["type"] == "string"
config_schema = params["properties"]["config"]
config_types = [t.get("type") for t in config_schema["anyOf"]]
assert "null" in config_types
def test_tool_with_unresolvable_forward_ref():
"""Fallback to raw annotations when get_type_hints() fails."""
import types
# Build a function in an isolated namespace so get_type_hints() cannot resolve
# the forward reference, exercising the except-branch fallback.
ns: dict = {}
exec(
"def greet(name: str = 'world') -> str:\n '''Greet someone.'''\n return f'Hello {name}'\n",
ns,
)
func = ns["greet"]
# Place the function in a throwaway module so get_type_hints() will fail on
# any non-builtin forward ref while still having a valid __module__.
mod = types.ModuleType("_phantom")
func.__module__ = mod.__name__
t = tool(func)
params = t.parameters()
assert params["properties"]["name"]["type"] == "string"
async def test_tool_invoke_with_context():
"""Full invocation with FunctionInvocationContext under PEP 563."""
@tool
def get_weather(location: str, ctx: FunctionInvocationContext) -> str:
"""Get the weather for a given location."""
user = ctx.kwargs.get("user", "anon")
return f"Weather in {location} for {user}"
params = get_weather.parameters()
assert "ctx" not in params.get("properties", {})
context = FunctionInvocationContext(
function=get_weather,
arguments=get_weather.input_model(location="Seattle"),
kwargs={"user": "test_user"},
)
result = await get_weather.invoke(context=context)
assert result[0].text == "Weather in Seattle for test_user"
@@ -5,7 +5,7 @@ import os
from random import randint
from typing import Annotated, Any, Literal
from agent_framework import SupportsChatGetResponse, tool
from agent_framework import Message, SupportsChatGetResponse, tool
from agent_framework.azure import (
AzureAIAgentClient,
AzureOpenAIAssistantsClient,
@@ -117,35 +117,37 @@ async def main(client_name: ClientName = "openai_chat") -> None:
client = get_client(client_name)
# 1. Configure prompt and streaming mode.
message = "What's the weather in Amsterdam and in Paris?"
message = Message("user", text="What's the weather in Amsterdam and in Paris?")
stream = os.getenv("STREAM", "false").lower() == "true"
print(f"Client: {client_name}")
print(f"User: {message}")
print(f"User: {message.text}")
# 2. Run with context-managed clients.
if isinstance(client, OpenAIAssistantsClient | AzureOpenAIAssistantsClient | AzureAIAgentClient):
async with client:
if stream:
response_stream = client.get_response(message, stream=True, options={"tools": get_weather})
response_stream = client.get_response([message], stream=True, options={"tools": get_weather})
print("Assistant: ", end="")
async for chunk in response_stream:
if chunk.text:
print(chunk.text, end="")
print("")
else:
print(f"Assistant: {await client.get_response(message, stream=False, options={'tools': get_weather})}")
print(
f"Assistant: {await client.get_response([message], stream=False, options={'tools': get_weather})}"
)
return
# 3. Run with non-context-managed clients.
if stream:
response_stream = client.get_response(message, stream=True, options={"tools": get_weather})
response_stream = client.get_response([message], stream=True, options={"tools": get_weather})
print("Assistant: ", end="")
async for chunk in response_stream:
if chunk.text:
print(chunk.text, end="")
print("")
else:
print(f"Assistant: {await client.get_response(message, stream=False, options={'tools': get_weather})}")
print(f"Assistant: {await client.get_response([message], stream=False, options={'tools': get_weather})}")
if __name__ == "__main__":
@@ -1,25 +1,17 @@
# /// script
# requires-python = ">=3.10"
# dependencies = [
# "autogen-agentchat",
# "autogen-ext[openai]",
# ]
# ///
# Run with any PEP 723 compatible runner, e.g.:
# uv run samples/autogen-migration/orchestrations/01_round_robin_group_chat.py
# Copyright (c) Microsoft. All rights reserved.
"""AutoGen RoundRobinGroupChat vs Agent Framework GroupChatBuilder/SequentialBuilder.
Demonstrates sequential agent orchestration where agents take turns processing
the task in a round-robin fashion.
"""
import asyncio
from agent_framework import Message
from dotenv import load_dotenv
"""AutoGen RoundRobinGroupChat vs Agent Framework GroupChatBuilder/SequentialBuilder.
Demonstrates sequential agent orchestration where agents take turns processing
the task in a round-robin fashion.
"""
# Load environment variables from .env file
load_dotenv()
@@ -98,7 +90,7 @@ async def run_agent_framework() -> None:
print("[Agent Framework] Sequential conversation:")
async for event in workflow.run("Create a brief summary about electric vehicles", stream=True):
if event.type == "output" and isinstance(event.data, list):
for message in event.data:
for message in event.data: # type: ignore
if isinstance(message, Message) and message.role == "assistant" and message.text:
print(f"---------- {message.author_name} ----------")
print(message.text)
@@ -144,9 +136,7 @@ async def run_agent_framework_with_cycle() -> None:
if last_message and "APPROVED" in last_message.text:
await context.yield_output("Content approved.")
else:
await context.send_message(
AgentExecutorRequest(messages=response.full_conversation, should_respond=True)
)
await context.send_message(AgentExecutorRequest(messages=response.full_conversation, should_respond=True))
workflow = (
WorkflowBuilder(start_executor=researcher)
@@ -1,25 +1,17 @@
# /// script
# requires-python = ">=3.10"
# dependencies = [
# "autogen-agentchat",
# "autogen-ext[openai]",
# ]
# ///
# Run with any PEP 723 compatible runner, e.g.:
# uv run samples/autogen-migration/orchestrations/02_selector_group_chat.py
# Copyright (c) Microsoft. All rights reserved.
"""AutoGen SelectorGroupChat vs Agent Framework GroupChatBuilder.
Demonstrates LLM-based speaker selection where an orchestrator decides
which agent should speak next based on the conversation context.
"""
import asyncio
from agent_framework import Message
from dotenv import load_dotenv
"""AutoGen SelectorGroupChat vs Agent Framework GroupChatBuilder.
Demonstrates LLM-based speaker selection where an orchestrator decides
which agent should speak next based on the conversation context.
"""
# Load environment variables from .env file
load_dotenv()
@@ -113,7 +105,7 @@ async def run_agent_framework() -> None:
print("[Agent Framework] Group chat conversation:")
async for event in workflow.run("How do I connect to a PostgreSQL database using Python?", stream=True):
if event.type == "output" and isinstance(event.data, list):
for message in event.data:
for message in event.data: # type: ignore
if isinstance(message, Message) and message.role == "assistant" and message.text:
print(f"---------- {message.author_name} ----------")
print(message.text)
@@ -1,19 +1,4 @@
# /// script
# requires-python = ">=3.10"
# dependencies = [
# "autogen-agentchat",
# "autogen-ext[openai]",
# ]
# ///
# Run with any PEP 723 compatible runner, e.g.:
# uv run samples/autogen-migration/orchestrations/03_swarm.py
# Copyright (c) Microsoft. All rights reserved.
"""AutoGen Swarm pattern vs Agent Framework HandoffBuilder.
Demonstrates agent handoff coordination where agents can transfer control
to other specialized agents based on the task requirements.
"""
import asyncio
from typing import Any
@@ -21,6 +6,12 @@ from typing import Any
from agent_framework import AgentResponseUpdate, WorkflowEvent
from dotenv import load_dotenv
"""AutoGen Swarm pattern vs Agent Framework HandoffBuilder.
Demonstrates agent handoff coordination where agents can transfer control
to other specialized agents based on the task requirements.
"""
# Load environment variables from .env file
load_dotenv()
@@ -1,19 +1,4 @@
# /// script
# requires-python = ">=3.10"
# dependencies = [
# "autogen-agentchat",
# "autogen-ext[openai]",
# ]
# ///
# Run with any PEP 723 compatible runner, e.g.:
# uv run samples/autogen-migration/orchestrations/04_magentic_one.py
# Copyright (c) Microsoft. All rights reserved.
"""AutoGen MagenticOneGroupChat vs Agent Framework MagenticBuilder.
Demonstrates orchestrated multi-agent workflows with a central coordinator
managing specialized agents for complex tasks.
"""
import asyncio
import json
@@ -27,6 +12,12 @@ from agent_framework import (
from agent_framework.orchestrations import MagenticProgressLedger
from dotenv import load_dotenv
"""AutoGen MagenticOneGroupChat vs Agent Framework MagenticBuilder.
Demonstrates orchestrated multi-agent workflows with a central coordinator
managing specialized agents for complex tasks.
"""
# Load environment variables from .env file
load_dotenv()
@@ -1,14 +1,9 @@
# /// script
# requires-python = ">=3.10"
# dependencies = [
# "autogen-agentchat",
# "autogen-ext[openai]",
# ]
# ///
# Run with any PEP 723 compatible runner, e.g.:
# uv run samples/autogen-migration/single_agent/01_basic_assistant_agent.py
# Copyright (c) Microsoft. All rights reserved.
import asyncio
from dotenv import load_dotenv
"""Basic AutoGen AssistantAgent vs Agent Framework Agent.
Both samples expect OpenAI-compatible environment variables (OPENAI_API_KEY or
@@ -16,10 +11,6 @@ Azure OpenAI configuration). Update the prompts or client wiring to match your
model of choice before running.
"""
import asyncio
from dotenv import load_dotenv
# Load environment variables from .env file
load_dotenv()
@@ -1,24 +1,14 @@
# /// script
# requires-python = ">=3.10"
# dependencies = [
# "autogen-agentchat",
# "autogen-core",
# "autogen-ext[openai]",
# ]
# ///
# Run with any PEP 723 compatible runner, e.g.:
# uv run samples/autogen-migration/single_agent/02_assistant_agent_with_tool.py
# Copyright (c) Microsoft. All rights reserved.
"""AutoGen AssistantAgent vs Agent Framework Agent with function tools.
Demonstrates how to create and attach tools to agents in both frameworks.
"""
import asyncio
from dotenv import load_dotenv
"""AutoGen AssistantAgent vs Agent Framework Agent with function tools.
Demonstrates how to create and attach tools to agents in both frameworks.
"""
# Load environment variables from .env file
load_dotenv()
@@ -1,23 +1,14 @@
# /// script
# requires-python = ">=3.10"
# dependencies = [
# "autogen-agentchat",
# "autogen-ext[openai]",
# ]
# ///
# Run with any PEP 723 compatible runner, e.g.:
# uv run samples/autogen-migration/single_agent/03_assistant_agent_thread_and_stream.py
# Copyright (c) Microsoft. All rights reserved.
"""AutoGen vs Agent Framework: Thread management and streaming responses.
Demonstrates conversation state management and streaming in both frameworks.
"""
import asyncio
from dotenv import load_dotenv
"""AutoGen vs Agent Framework: Thread management and streaming responses.
Demonstrates conversation state management and streaming in both frameworks.
"""
# Load environment variables from .env file
load_dotenv()
@@ -1,24 +1,15 @@
# /// script
# requires-python = ">=3.10"
# dependencies = [
# "autogen-agentchat",
# "autogen-ext[openai]",
# ]
# ///
# Run with any PEP 723 compatible runner, e.g.:
# uv run samples/autogen-migration/single_agent/04_agent_as_tool.py
# Copyright (c) Microsoft. All rights reserved.
import asyncio
from dotenv import load_dotenv
"""AutoGen vs Agent Framework: Agent-as-a-Tool pattern.
Demonstrates hierarchical agent architectures where one agent delegates
work to specialized sub-agents wrapped as tools.
"""
import asyncio
from dotenv import load_dotenv
# Load environment variables from .env file
load_dotenv()
@@ -107,6 +98,7 @@ async def run_agent_framework() -> None:
if content.type == "function_call":
# Accumulate function call content as it streams in
call_id = content.call_id
assert call_id is not None, "Function call content must have a call_id"
if call_id in accumulated_calls:
# Add to existing call (arguments stream in gradually)
accumulated_calls[call_id] = accumulated_calls[call_id] + content
+6 -7
View File
@@ -165,18 +165,17 @@ Produces:
## Report Status Codes
| Status | Label | Description |
| ------- | --------- | ----------------------------------------- |
| SUCCESS | [PASS] | Sample ran to completion with exit code 0 |
| FAILURE | [FAIL] | Sample exited with non-zero code |
| TIMEOUT | [TIMEOUT] | Sample exceeded timeout limit |
| ERROR | [ERROR] | Exception during execution |
| Status | Label | Description |
| ------------- | --------------- | ----------------------------------------- |
| SUCCESS | [PASS] | Sample ran to completion with exit code 0 |
| FAILURE | [FAIL] | Sample did not complete successfully (non-zero exit code) |
| MISSING_SETUP | [MISSING_SETUP] | Sample skipped due to missing setup |
## Troubleshooting
### Agent output parsing errors
If an agent returns non-JSON content, that sample is marked as `ERROR` with parser details in the report.
If an agent returns non-JSON content, that sample is marked as `FAILURE` with parser details in the report.
### GitHub Copilot authentication or CLI issues
+9 -1
View File
@@ -75,6 +75,13 @@ Examples:
help="Custom name for the report files (without extension). If not provided, uses timestamp.",
)
parser.add_argument(
"--exclude",
nargs="+",
type=str,
help="Subdirectory paths to exclude (relative to the search directory set by --subdir)",
)
return parser.parse_args()
@@ -104,6 +111,7 @@ async def main() -> int:
samples_dir=samples_dir,
python_root=python_root,
subdir=args.subdir,
exclude=args.exclude,
max_parallel_workers=max(1, args.max_parallel_workers),
)
@@ -138,7 +146,7 @@ async def main() -> int:
print(f" JSON: {json_path}")
# Return appropriate exit code
failed = report.failure_count + report.timeout_count + report.error_count
failed = report.failure_count + report.missing_setup_count
return 1 if failed > 0 else 0
@@ -0,0 +1,224 @@
# Copyright (c) Microsoft. All rights reserved.
"""Aggregate validation reports across runs and produce a trend report.
Reads JSON reports from individual validation jobs, combines them with
cached history from previous runs, and produces a markdown trend report
showing per-sample status over the last 5 runs.
Usage:
python aggregate.py <reports-dir> <history-file> <output-file>
"""
import json
import sys
from datetime import datetime
from pathlib import Path
from typing import Any
MAX_HISTORY = 5
STATUS_EMOJI = {
"success": "âś…",
"failure": "❌",
"missing_setup": "⚠️",
}
def _format_run_label(timestamp: str) -> str:
"""Format a run timestamp as a compact column label (e.g. '03-24 18:05')."""
try:
dt = datetime.fromisoformat(timestamp)
return dt.strftime("%m-%d %H:%M")
except (ValueError, TypeError):
return timestamp[:16]
def load_current_run(reports_dir: Path) -> dict[str, Any]:
"""Load all JSON report files from the current run and merge them."""
combined_results: dict[str, str] = {}
total = success = failure = missing = 0
json_files = sorted(reports_dir.glob("*.json"))
if not json_files:
print(f"Warning: No JSON report files found in {reports_dir}")
return {
"timestamp": datetime.now().isoformat(),
"summary": {
"total_samples": 0,
"success_count": 0,
"failure_count": 0,
"missing_setup_count": 0,
},
"results": {},
}
for json_file in json_files:
print(f" Loading report: {json_file.name}")
with open(json_file, encoding="utf-8") as f:
report = json.load(f)
for result in report["results"]:
combined_results[result["path"]] = result["status"]
summary = report["summary"]
total += summary["total_samples"]
success += summary["success_count"]
failure += summary["failure_count"]
missing += summary["missing_setup_count"]
return {
"timestamp": datetime.now().isoformat(),
"summary": {
"total_samples": total,
"success_count": success,
"failure_count": failure,
"missing_setup_count": missing,
},
"results": combined_results,
}
def load_history(history_path: Path) -> list[dict[str, Any]]:
"""Load previous run history from cache."""
if history_path.exists():
with open(history_path, encoding="utf-8") as f:
data = json.load(f)
runs = data.get("runs", [])
print(f" Loaded {len(runs)} previous run(s) from history")
return runs
print(" No previous history found")
return []
def save_history(history_path: Path, runs: list[dict[str, Any]]) -> None:
"""Save run history, keeping only the last MAX_HISTORY entries."""
history_path.parent.mkdir(parents=True, exist_ok=True)
trimmed = runs[-MAX_HISTORY:]
with open(history_path, "w", encoding="utf-8") as f:
json.dump({"runs": trimmed}, f, indent=2)
print(f" Saved {len(trimmed)} run(s) to history")
def generate_trend_report(runs: list[dict[str, Any]]) -> str:
"""Generate a markdown trend report from run history."""
lines = [
"# Sample Validation Trend Report",
"",
f"*Generated: {datetime.now().strftime('%Y-%m-%d %H:%M UTC')}*",
"",
]
# --- Overall status table (most recent first) ---
lines.append("## Overall Status (Last 5 Runs)")
lines.append("")
lines.append("| Run | Success | Failure | Missing Setup | Total |")
lines.append("|-----|---------|---------|---------------|-------|")
for run in reversed(runs):
s = run["summary"]
label = _format_run_label(run["timestamp"])
lines.append(
f"| {label} | {s['success_count']}/{s['total_samples']} "
f"| {s['failure_count']}/{s['total_samples']} "
f"| {s['missing_setup_count']}/{s['total_samples']} "
f"| {s['total_samples']} |"
)
# Pad with N/A rows if fewer than 5 runs
for _ in range(MAX_HISTORY - len(runs)):
lines.append("| N/A | N/A | N/A | N/A | N/A |")
lines.append("")
# --- Per-sample results table ---
lines.append("## Per-Sample Results")
lines.append("")
# Collect all sample paths across all runs
all_paths: set[str] = set()
for run in runs:
all_paths.update(run["results"].keys())
if not all_paths:
lines.append("*No sample results available.*")
return "\n".join(lines)
# Build header (most recent run first)
header = "| Sample |"
separator = "|--------|"
for run in reversed(runs):
label = _format_run_label(run["timestamp"])
header += f" {label} |"
separator += "------------|"
for _ in range(MAX_HISTORY - len(runs)):
header += " N/A |"
separator += "-----|"
lines.append(header)
lines.append(separator)
for path in sorted(all_paths):
row = f"| `{path}` |"
for run in reversed(runs):
status = run["results"].get(path, "N/A")
emoji = STATUS_EMOJI.get(status, "N/A")
row += f" {emoji} |"
for _ in range(MAX_HISTORY - len(runs)):
row += " N/A |"
lines.append(row)
lines.append("")
lines.append("**Legend:** ✅ Success · ❌ Failure · ⚠️ Missing Setup · N/A Not available")
lines.append("")
return "\n".join(lines)
def main() -> int:
if len(sys.argv) != 4:
print("Usage: python aggregate.py <reports-dir> <history-file> <output-file>")
return 1
reports_dir = Path(sys.argv[1])
history_path = Path(sys.argv[2])
output_path = Path(sys.argv[3])
print("Aggregating validation results...")
# Load current run's reports
print(f"\nLoading reports from {reports_dir}:")
current_run = load_current_run(reports_dir)
s = current_run["summary"]
print(
f" Current run: {s['success_count']} success, "
f"{s['failure_count']} failure, "
f"{s['missing_setup_count']} missing setup "
f"(total: {s['total_samples']})"
)
# Load history and append current run
print(f"\nLoading history from {history_path}:")
runs = load_history(history_path)
runs.append(current_run)
runs = runs[-MAX_HISTORY:]
# Save updated history
print(f"\nSaving history to {history_path}:")
save_history(history_path, runs)
# Generate trend report
print("\nGenerating trend report...")
report = generate_trend_report(runs)
output_path.parent.mkdir(parents=True, exist_ok=True)
output_path.write_text(report, encoding="utf-8")
print(f"Trend report written to {output_path}")
# Also print the report to stdout
print("\n" + "=" * 80)
print(report)
return 0
if __name__ == "__main__":
sys.exit(main())
@@ -14,7 +14,8 @@ from agent_framework import (
handler,
)
from agent_framework.github import GitHubCopilotAgent
from copilot.types import PermissionRequest, PermissionRequestResult
from copilot.generated.session_events import PermissionRequest
from copilot.types import PermissionRequestResult
from pydantic import BaseModel
from typing_extensions import Never
@@ -36,6 +37,7 @@ class AgentResponseFormat(BaseModel):
status: str
output: str
error: str
fix: str
@dataclass
@@ -54,15 +56,20 @@ class BatchCompletion:
AgentInstruction = (
"You are validating exactly one Python sample.\n"
"Analyze the sample code and execute it. Based on the execution result, determine if it "
"runs successfully, fails, or times out. Feel free to install any required dependencies.\n"
"Analyze the sample code and execute it as it is. Based on the execution result, determine "
"if it runs successfully, fails, or is missing_setup. Use `missing_setup` if the sample reports "
"missing required environment variables. The environment you're given should contain the necessary "
"variables. Don't create new environment variables nor modify the sample code.\n"
"Feel free to install any required dependencies if needed.\n"
"The sample can be interactive. If it is interactive, respond to the sample when prompted "
"based on your analysis of the code. You do not need to consult human on what to respond.\n"
"If the sample fails, investigate the error and suggest a fix.\n"
"Return ONLY valid JSON with this schema:\n"
"{\n"
' "status": "success|failure|timeout|error",\n'
' "status": "success|failure|missing_setup",\n'
' "output": "short summary of the result and what you did if the sample was interactive",\n'
' "error": "error details or empty string"\n'
' "error": "error details or empty string",\n'
' "fix": "suggested code fix if the sample failed, otherwise empty string"\n'
"}\n\n"
)
@@ -87,16 +94,15 @@ def status_from_text(value: str) -> RunStatus:
for status in RunStatus:
if status.value == normalized:
return status
return RunStatus.ERROR
return RunStatus.FAILURE
def prompt_permission(
request: PermissionRequest, context: dict[str, str]
) -> PermissionRequestResult:
"""Permission handler that always approves."""
kind = request.get("kind", "unknown")
logger.debug(
f"[Permission Request: {kind}] ({context})Automatically approved for sample validation."
f"[Permission Request: {request.kind}] ({context})Automatically approved for sample validation."
)
return PermissionRequestResult(kind="approved")
@@ -108,39 +114,73 @@ class CustomAgentExecutor(Executor):
returned as error responses, otherwise an exception in one agent could crash the entire workflow.
"""
# Retry in case GitHub Copilot agent encounters transient errors unrelated to the sample execution.
RETRY_COUNT = 1
def __init__(self, agent: GitHubCopilotAgent):
super().__init__(id=agent.id)
self.agent = agent
self._session = agent.create_session()
@handler
async def handle_task(
self, sample: SampleInfo, ctx: WorkflowContext[WorkerFreed | RunResult]
) -> None:
"""Execute one sample task and notify collector + coordinator."""
try:
response = await self.agent.run(
[
Message(
role="user",
text=f"Validate the following sample:\n\n{sample.relative_path}",
current_retry = 0
while True:
try:
response = await self.agent.run(
[
Message(
role="user",
text=f"Validate the following sample:\n\n{sample.relative_path}",
)
],
session=self._session,
)
result_payload = parse_agent_json(response.text)
result = RunResult(
sample=sample,
status=status_from_text(result_payload.status),
output=result_payload.output,
error=result_payload.error,
fix=result_payload.fix,
)
break
except Exception as ex:
if current_retry < self.RETRY_COUNT:
logger.warning(
f"Error executing agent {self.agent.id} (attempt {current_retry + 1}/{self.RETRY_COUNT}): {ex}. Retrying..."
)
]
)
result_payload = parse_agent_json(response.text)
result = RunResult(
sample=sample,
status=status_from_text(result_payload.status),
output=result_payload.output,
error=result_payload.error,
)
except Exception as ex:
logger.error(f"Error executing agent {self.agent.id}: {ex}")
result = RunResult(
sample=sample,
status=RunStatus.ERROR,
output="",
error=str(ex),
)
try:
current_retry += 1
await self.agent.stop()
await self.agent.start()
self._session = self.agent.create_session() # Reset session for retry
continue
except Exception as restart_ex:
logger.error(
f"Error restarting agent {self.agent.id}: {restart_ex}. No more retries."
)
result = RunResult(
sample=sample,
status=RunStatus.FAILURE,
output="",
error=f"Original error: {ex}. Restart error: {restart_ex}",
fix="",
)
break
logger.error(f"Error executing agent {self.agent.id}: {ex}")
result = RunResult(
sample=sample,
status=RunStatus.FAILURE,
output="",
error=str(ex),
fix="",
)
break
await ctx.send_message(result, target_id="collector")
await ctx.send_message(WorkerFreed(worker_id=self.id), target_id="coordinator")
@@ -252,7 +292,7 @@ class CreateConcurrentValidationWorkflowExecutor(Executor):
instructions=AgentInstruction,
default_options={
"on_permission_request": prompt_permission,
"timeout": 180,
"timeout": 60,
}, # type: ignore
)
agents.append(agent)
+20 -4
View File
@@ -52,13 +52,18 @@ def _has_main_entrypoint_guard(path: Path) -> bool:
)
def discover_samples(samples_dir: Path, subdir: str | None = None) -> list[SampleInfo]:
def discover_samples(
samples_dir: Path,
subdir: str | None = None,
exclude: list[str] | None = None,
) -> list[SampleInfo]:
"""
Find all Python sample files in the samples directory.
Args:
samples_dir: Root samples directory
subdir: Optional subdirectory to filter to
exclude: Optional list of subdirectory paths (relative to the search directory) to exclude
Returns:
List of SampleInfo objects for each discovered sample
@@ -72,12 +77,21 @@ def discover_samples(samples_dir: Path, subdir: str | None = None) -> list[Sampl
else:
search_dir = samples_dir
# Resolve excluded paths to absolute for reliable comparison
exclude_paths = {(search_dir / exc).resolve() for exc in (exclude or [])}
python_files: list[Path] = []
# Walk through all subdirectories and find .py files
for root, dirs, files in os.walk(search_dir):
# Skip directories that start with _ (like _sample_validation)
dirs[:] = [d for d in dirs if not d.startswith("_") and d != "__pycache__"]
# Skip directories that start with _, __pycache__, or excluded paths
dirs[:] = [
d
for d in dirs
if not d.startswith("_")
and d != "__pycache__"
and (Path(root) / d).resolve() not in exclude_paths
]
for file in files:
# Skip files that start with _ and include only scripts with a main entrypoint guard
@@ -113,8 +127,10 @@ class DiscoverSamplesExecutor(Executor):
print(f"🔍 Discovering samples in {self.config.samples_dir}")
if self.config.subdir:
print(f" Filtering to subdirectory: {self.config.subdir}")
if self.config.exclude:
print(f" Excluding: {', '.join(self.config.exclude)}")
samples = discover_samples(self.config.samples_dir, self.config.subdir)
samples = discover_samples(self.config.samples_dir, self.config.subdir, self.config.exclude)
print(f" Found {len(samples)} samples")
await ctx.send_message(DiscoveryResult(samples=samples))
+9 -11
View File
@@ -18,6 +18,7 @@ class ValidationConfig:
samples_dir: Path
python_root: Path
subdir: str | None = None
exclude: list[str] | None = None
max_parallel_workers: int = 10
@@ -60,8 +61,7 @@ class RunStatus(Enum):
SUCCESS = "success"
FAILURE = "failure"
TIMEOUT = "timeout"
ERROR = "error"
MISSING_SETUP = "missing_setup"
@dataclass
@@ -72,6 +72,7 @@ class RunResult:
status: RunStatus
output: str
error: str
fix: str
@dataclass
@@ -89,8 +90,7 @@ class Report:
total_samples: int
success_count: int
failure_count: int
timeout_count: int
error_count: int
missing_setup_count: int
results: list[RunResult] = field(default_factory=list) # type: ignore
def to_markdown(self) -> str:
@@ -107,15 +107,14 @@ class Report:
f"| Total Samples | {self.total_samples} |",
f"| [PASS] Success | {self.success_count} |",
f"| [FAIL] Failure | {self.failure_count} |",
f"| [TIMEOUT] Timeout | {self.timeout_count} |",
f"| [ERROR] Error | {self.error_count} |",
f"| [MISSING_SETUP] Missing Setup | {self.missing_setup_count} |",
"",
"## Detailed Results",
"",
]
# Group by status
for status in [RunStatus.FAILURE, RunStatus.TIMEOUT, RunStatus.ERROR, RunStatus.SUCCESS]:
for status in [RunStatus.FAILURE, RunStatus.MISSING_SETUP, RunStatus.SUCCESS]:
status_results = [r for r in self.results if r.status == status]
if not status_results:
continue
@@ -123,8 +122,7 @@ class Report:
status_label = {
RunStatus.SUCCESS: "[PASS]",
RunStatus.FAILURE: "[FAIL]",
RunStatus.TIMEOUT: "[TIMEOUT]",
RunStatus.ERROR: "[ERROR]",
RunStatus.MISSING_SETUP: "[MISSING_SETUP]",
}
lines.append(f"### {status_label[status]} {status.value.title()} ({len(status_results)})")
@@ -148,8 +146,7 @@ class Report:
"total_samples": self.total_samples,
"success_count": self.success_count,
"failure_count": self.failure_count,
"timeout_count": self.timeout_count,
"error_count": self.error_count,
"missing_setup_count": self.missing_setup_count,
},
"results": [
{
@@ -157,6 +154,7 @@ class Report:
"status": r.status.value,
"output": r.output,
"error": r.error,
"fix": r.fix,
}
for r in self.results
],
+6 -10
View File
@@ -22,12 +22,11 @@ def generate_report(results: list[RunResult]) -> Report:
Returns:
Report object with aggregated statistics
"""
# Sort results: failures, timeouts, errors first, then successes
# Sort results: failures, missing setup first, then successes
status_priority = {
RunStatus.FAILURE: 0,
RunStatus.TIMEOUT: 1,
RunStatus.ERROR: 2,
RunStatus.SUCCESS: 3,
RunStatus.MISSING_SETUP: 1,
RunStatus.SUCCESS: 2,
}
sorted_results = sorted(results, key=lambda r: status_priority[r.status])
@@ -36,8 +35,7 @@ def generate_report(results: list[RunResult]) -> Report:
total_samples=len(results),
success_count=sum(1 for r in results if r.status == RunStatus.SUCCESS),
failure_count=sum(1 for r in results if r.status == RunStatus.FAILURE),
timeout_count=sum(1 for r in results if r.status == RunStatus.TIMEOUT),
error_count=sum(1 for r in results if r.status == RunStatus.ERROR),
missing_setup_count=sum(1 for r in results if r.status == RunStatus.MISSING_SETUP),
results=sorted_results,
)
@@ -86,8 +84,7 @@ def print_summary(report: Report) -> None:
if (
report.failure_count == 0
and report.timeout_count == 0
and report.error_count == 0
and report.missing_setup_count == 0
):
print("[PASS] ALL SAMPLES PASSED!")
else:
@@ -98,8 +95,7 @@ def print_summary(report: Report) -> None:
print("Results:")
print(f" [PASS] Success: {report.success_count}")
print(f" [FAIL] Failure: {report.failure_count}")
print(f" [TIMEOUT] Timeout: {report.timeout_count}")
print(f" [ERR] Errors: {report.error_count}")
print(f" [MISSING_SETUP] Missing Setup: {report.missing_setup_count}")
print("=" * 80)
# Print JSON output for GitHub Actions visibility
@@ -66,9 +66,10 @@ class RunDynamicValidationWorkflowExecutor(Executor):
fallback_results = [
RunResult(
sample=sample,
status=RunStatus.ERROR,
status=RunStatus.FAILURE,
output="",
error="Nested workflow did not return an ExecutionResult.",
fix="",
)
for sample in creation.samples
]