agent/agent-framework
2
0
Fork 0
mirror of https://github.com/microsoft/agent-framework.git synced 2026-06-16 21:04:09 +08:00
Code Issues Packages Projects Releases Wiki Activity

Python: consolidate lab packages into a single one; update contribution guidelines (#940)

Browse Source 
* consolidate lab packages into a single one; update contribution guidelines

* update dep list

* add poe tasks; fix tests and lint erros

* add lab tests for CI

* fix test

* update root pyproject.toml
This commit is contained in:
Eric Zhu
2025-09-26 20:28:05 -07:00
committed by GitHub
Unverified
parent 9fec0f5ef4
commit 514d0209a8
55 changed files with 615 additions and 1087 deletions
Download Patch File Download Diff File Expand all files Collapse all files
python/packages/lab/.gitignore
+1
View File
@@ -0,0 +1 @@
test-results.xml
python/packages/lab/cookiecutter-agent-framework-lab/{{cookiecutter.package_name}}/LICENSE → python/packages/lab/LICENSE
View File
python/packages/lab/README.md
+70 -116
View File
@@ -1,145 +1,99 @@
# Agent Framework Lab
This directory contains experimental packages for Microsoft Agent Framework that are distributed as separate installable packages under the `agent_framework.lab` namespace.
Lab packages are not part of the core framework and may experience breaking changes or be deprecated in the future.
This is the experimental package for Microsoft Agent Framework, `agent-framework-lab`, which contains
various lab modules built on top of the core framework.
Lab modules are not part of the core framework and may experience breaking changes or be deprecated in the future.
## What are Lab Packages?
## What are Lab Modules?
Lab packages are extensions to the core Agent Framework that falls into
Lab modules are extensions to the core Agent Framework that fall into
one of the following categories:
1. Incubation of new features that may get incorporated by the core framework.
1. Incubation of new features that may get incorporated by the core framework.
2. Research prototypes built on the core framework.
3. Benchmarks and experimentation tools.
## Lab Packages
## Lab Modules
- [**gaia**](./gaia/): GAIA benchmark implementation (`agent-framework-lab-gaia`)
- [**lightning**](./lightning/): Reinforcement learning for agents (`agent-framework-lab-lightning`)
- [**tau2**](./tau2/): Customer service agent simulation framework (`agent-framework-lab-tau2`)
- [**gaia**](./gaia/): Evaluate your agents using the GAIA benchmark for general assistant tasks
- [**tau2**](./tau2/): Evaluate your agents using the TAU2 benchmark for customer support tasks
- [**lightning**](./lightning/): RL training for agents (in development)
## How do I contribute?
This repo only contains lab packages maintained by Microsoft.
If you want to contribute, please take the following steps:
1. Follow the [Create a New Lab Package](#create-new-lab-package) guide
below to create your own lab package.
2. Create a new repo on GitHub and check in your package there.
3. Tag your repo with `agent-framework-lab` for better discovery.
4. Submit a PR to this repo (github.com/microsoft/agent-framework)
to add a link to your repo in the [list](#lab-packages) above.
**The PR title must contain "[New Lab Package]"**.
5. We will review your repo and decide whether to approve it.
Follow the [guidelines](#guidelines) when you create your package, our decision
to accept your PR will be based on your idea as well as the quality of your
code.
We may decide to maintain your package in this repo. In that case, we will
contact you directly.
## Package Structure
Each lab package follows this structure:
## Repository Structure
```
packages/lab/{lab_name}/
├── agent_framework/
│ └── lab/
│ └── {lab_name}/
│ └── __init__.py # Imports from agent_framework_lab_{lab_name}
├── agent_framework_lab_{lab_name}/ # Actual implementation package
│ ├── __init__.py # Main exports and __version__
│ ├── {module_files}.py # Implementation modules
│ └── py.typed # Type hints marker
├── tests/
│ ├── __init__.py
│ └── test_{lab_name}.py # Package tests
├── pyproject.toml # Package configuration
├── README.md # Package-specific documentation
└── LICENSE # MIT License
agent-framework-lab/
├── pyproject.toml # Single package configuration for agent-framework-lab
├── README.md # This file
├── LICENSE # License file
├── namespace/ # Centralized namespace package files
│ └── agent_framework/
│ └── lab/
│ ├── gaia/ # Re-exports from agent_framework_lab_gaia
│ ├── lightning/ # Re-exports from agent_framework_lab_lightning
│ └── tau2/ # Re-exports from agent_framework_lab_tau2
├── gaia/ # GAIA module implementation
│ └── agent_framework_lab_gaia/
├── lightning/ # Lightning module implementation
│ └── agent_framework_lab_lightning/
└── tau2/ # TAU2 module implementation
└── agent_framework_lab_tau2/
```
## Creating a New Lab Package
This structure maintains a single PyPI package `agent-framework-lab` while supporting modular imports through the namespace package mechanism.
### Create The Package
## Installation
First ensure `cookiecutter` is installed.
Install the base lab package:
```bash
pip install cookiecutter
pip install agent-framework-lab
```
Then go to the directory where you want to create the package:
For details on installing individual modules, see their respective README files listed above.
```bash
cookiecutter /path/to/agent-framework/python/packages/lab/cookiecutter-agent-framework-lab
```
## Usage
You will be prompted for:
- **package_name**: The name of your lab package (e.g., "lightning", "vision")
- **package_display_name**: Human-readable name (e.g., "Lighting Tools", "Computer Vision")
- **package_description**: Brief description (auto-generated from display name)
- **include_cli_script**: Whether to include a CLI script (y/n)
### After Package Creation
1. **Implement your functionality** in `agent_framework_lab_your_package_name/`
2. **Update exports** in `__init__.py` `__all__` list
3. **Add dependencies** to `pyproject.toml`
4. **Write tests** in the `tests/` directory
5. **Update README** with usage examples and API documentation
### Add to Workspace (only for packages maintained in this repo)
After creating your package, add it to the workspace configuration:
```
# Edit python/pyproject.toml
# Add to dependencies section:
dependencies = [
# ... existing packages ...
"agent-framework-lab-your-package-name",
]
# Add to [tool.uv.sources] section:
agent-framework-lab-your-package-name = { workspace = true }
```
### Usage
Once created, users can install your lab package
1. directly from your repo:
```bash
pip install git+https://github.com/your-username/your-lab-package-repo.git
```
2. or from PyPI if you have uploaded your lab package there:
```bash
pip install "agent-framework-lab-your-package-name"
```
Then, they can use your lab package:
Import and use lab modules from the `agent_framework.lab` namespace.
For example, to use the GAIA module:
```python
from agent_framework.lab.your_package_name import YourClass, your_function
# Use the functionality
instance = YourClass()
result = your_function()
# Using GAIA module
from agent_framework.lab.gaia import GAIA
```
## Guidelines
## Should I consume Lab Modules?
1. **Naming**: Use lowercase with hyphens for package names (`agent-framework-lab-your-package-name`)
2. **Namespace**: Always use `agent_framework.lab.your_package_name` for imports
3. **Versioning**: Start with `0.1.0b1` for beta releases
4. **Dependencies**: Minimize external dependencies, always include `agent-framework`
5. **Documentation**: Include comprehensive README with usage examples
6. **Tests**: Write comprehensive tests with good coverage
7. **Type hints**: Always include type hints and `py.typed` file
If you are looking for stable and production-ready features, you should not use lab modules. Stick to the core framework.
If you are looking for experimentation, research, or want to
benchmark different approaches -- most importantly, if you don't mind breaking changes and potential deprecations --
then lab modules are for you.
## Contributing to Lab Modules
### Microsoft-maintained modules
For Microsoft-maintained modules in this repository, please follow standard contribution guidelines and submit pull requests directly to this repository.
### Community modules
If you want to contribute a community-maintained lab module:
1. Create a new repository on GitHub for your module
2. Tag your repository with `agent-framework-lab` for discoverability
3. Submit a PR to add a link to your repository in the [Lab Modules](#lab-modules) section above
4. Use the PR title format: `[New Lab Module] Your Module Name`
We will review your submission based on the guidelines below.
### Guidelines
1. **Purpose**: Community modules should fit into one of the three categories of lab modules (incubation, research, benchmarks)
2. **Namespace**: Community modules should avoid the `agent_framework.lab` namespace (reserved for modules maintained in this repository)
3. **Dependencies**: Minimize external dependencies, always include `agent-framework` as a base dependency
4. **Documentation**: Include comprehensive README with installation instructions and usage examples
5. **Tests**: Write comprehensive tests with good coverage
6. **Type hints**: Always include type hints and a `py.typed` file
7. **Versioning**: Use semantic versioning, start with `0.1.0` for initial releases
python/packages/lab/cookiecutter-agent-framework-lab/README.md
-58
View File
@@ -1,58 +0,0 @@
# Cookiecutter Template for Agent Framework Lab Packages
This is a cookiecutter template for creating new lab packages in the Microsoft Agent Framework.
## Usage
```bash
cd /path/to/agent-framework/python/packages/lab
cookiecutter ./cookiecutter-agent-framework-lab
```
You will be prompted for the following information:
- **package_name**: The name of your lab package (e.g., "lightning", "vision")
- **package_display_name**: Human-readable name (e.g., "Lighting Tools", "Computer Vision")
- **package_description**: Brief description of the package (auto-generated from display name)
- **version**: Starting version (default: 0.1.0b1)
- **author_name**: Author name (default: Microsoft)
- **author_email**: Author email (default: af-support@microsoft.com)
- **include_cli_script**: Whether to include a CLI script (y/n)
- **cli_script_name**: Name of CLI script if included
## What Gets Generated
The template creates a complete lab package structure:
```
{package_name}/
├── agent_framework/
│ └── lab/
│ └── {package_name}/
│ └── __init__.py
├── agent_framework_lab_{package_name}/
│ ├── __init__.py
│ └── py.typed
├── tests/
│ ├── __init__.py
│ └── test_{package_name}.py
├── pyproject.toml
├── README.md
└── LICENSE
```
## After Generation
1. Implement your functionality in `agent_framework_lab_{package_name}/`
2. Update the `__all__` exports in `__init__.py`
3. Add your dependencies to `pyproject.toml`
4. Write comprehensive tests
5. Update the README with usage examples
## Integration
Don't forget to add your new package to the workspace:
1. Add to `python/pyproject.toml` dependencies
2. Add to `[tool.uv.sources]` section
3. Test installation with `uv run python -c "from agent_framework.lab.{name} import *"`
python/packages/lab/cookiecutter-agent-framework-lab/cookiecutter.json
-20
View File
@@ -1,20 +0,0 @@
{
"package_name": "",
"package_display_name": "",
"package_description": "{{ cookiecutter.package_display_name }} module for Microsoft Agent Framework.",
"version": "0.1.0b1",
"author_name": "Microsoft",
"author_email": "af-support@microsoft.com",
"license": "MIT",
"include_cli_script": ["y", "n"],
"cli_script_name": "{{ cookiecutter.package_name }}_cli",
"python_requires": ">=3.10",
"within_microsoft_agent_framework_repo": ["y", "n"],
"__prompts__": {
"within_microsoft_agent_framework_repo": "Are you creating this package within the github.com/microsoft/agent-framework repo or a fork of it? (If yes, ensure you create it in python/packages/lab/ directory)"
},
"_templates_suffix": "",
"_copy_without_render": [
"*.py.typed"
]
}
python/packages/lab/cookiecutter-agent-framework-lab/{{cookiecutter.package_name}}/README.md
-56
View File
@@ -1,56 +0,0 @@
# Agent Framework Lab - {{ cookiecutter.package_display_name }}
{{ cookiecutter.package_description }}
## Installation
```bash
pip install agent-framework-lab-{{ cookiecutter.package_name }}
```
## Usage
```python
from agent_framework.lab.{{ cookiecutter.package_name }} import YourClass
# Your usage example here
instance = YourClass()
```
## Overview
Brief description of what this lab package provides and its main features.
## Features
- Feature 1: Description
- Feature 2: Description
- Feature 3: Description
## Examples
### Basic Usage
```python
from agent_framework.lab.{{ cookiecutter.package_name }} import YourClass
# Example usage
```
### Advanced Usage
```python
# More advanced examples
```
## API Reference
Document your main classes and functions here.
## Contributing
This package is part of the Microsoft Agent Framework Lab. Please see the main repository for contribution guidelines.
## License
This project is licensed under the {{ cookiecutter.license }} License - see the LICENSE file for details.
python/packages/lab/cookiecutter-agent-framework-lab/{{cookiecutter.package_name}}/pyproject.toml
-95
View File
@@ -1,95 +0,0 @@
[project]
name = "agent-framework-lab-{{ cookiecutter.package_name }}"
description = "{{ cookiecutter.package_description }}"
authors = [{ name = "{{ cookiecutter.author_name }}", email = "{{ cookiecutter.author_email }}"}]
readme = "README.md"
requires-python = "{{ cookiecutter.python_requires }}"
version = "{{ cookiecutter.version }}"
license-files = ["LICENSE"]
urls.homepage = "https://learn.microsoft.com/en-us/semantic-kernel/overview/"
urls.source = "https://github.com/microsoft/agent-framework/tree/main/python"
urls.release_notes = "https://github.com/microsoft/agent-framework/releases?q=tag%3Apython-1&expanded=true"
urls.issues = "https://github.com/microsoft/agent-framework/issues"
classifiers = [
"License :: OSI Approved :: {{ cookiecutter.license }} License",
"Development Status :: 2 - Pre-Alpha",
"Intended Audience :: Developers",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
]
dependencies = [
"agent-framework",
"pydantic>=2.0.0",
# Add your specific dependencies here
]
{% if cookiecutter.include_cli_script == "y" %}
[project.scripts]
{{ cookiecutter.cli_script_name }} = "agent_framework_lab_{{ cookiecutter.package_name }}:main"
{% endif %}
[build-system]
requires = ["setuptools>=64", "wheel"]
build-backend = "setuptools.build_meta"
[tool.setuptools]
package-dir = {"" = "src"}
packages = ["agent_framework_lab_{{ cookiecutter.package_name }}", "agent_framework.lab.{{ cookiecutter.package_name }}"]
[tool.setuptools.package-data]
agent_framework_lab_{{ cookiecutter.package_name }} = ["py.typed"]
[tool.ruff]
line-length = 120
target-version = "py310"
extend-exclude = ["tests", "__pycache__"]
[tool.ruff.lint]
select = ["E", "F", "I", "W", "UP", "C4", "N"]
ignore = ["N803", "N806", "N999", "UP007"]
[tool.ruff.format]
quote-style = "double"
[tool.mypy]
python_version = "3.10"
strict = true
check_untyped_defs = true
disallow_untyped_defs = true
disallow_incomplete_defs = true
disallow_untyped_decorators = true
warn_redundant_casts = true
warn_unused_ignores = true
warn_return_any = true
warn_unreachable = true
show_error_codes = true
implicit_reexport = true
packages = ["src.agent_framework_lab_{{ cookiecutter.package_name }}"]
{% if cookiecutter.within_microsoft_agent_framework_repo == "y" %}
[tool.poe]
executor.type = "uv"
include = "../../../shared_tasks.toml"
[tool.poe.tasks]
test = "pytest --cov=agent_framework_lab_{{ cookiecutter.package_name }} --cov-report=term-missing:skip-covered tests"
mypy = "mypy agent_framework_lab_{{ cookiecutter.package_name }}"
{% else %}
[tool.poe.tasks]
fmt = "ruff format"
format = "ruff format"
lint = "ruff check"
test = "pytest --cov=agent_framework_lab_{{ cookiecutter.package_name }} --cov-report=term-missing:skip-covered tests"
mypy = "mypy agent_framework_lab_{{ cookiecutter.package_name }}"
{% endif %}
[tool.pytest.ini_options]
testpaths = ["tests"]
pythonpath = ["src"]
addopts = "--strict-markers --strict-config"
markers = [
"unit: marks tests as unit tests",
"integration: marks tests as integration tests",
]
python/packages/lab/cookiecutter-agent-framework-lab/{{cookiecutter.package_name}}/src/agent_framework/lab/{{cookiecutter.package_name}}/__init__.py
-4
View File
@@ -1,4 +0,0 @@
# Copyright (c) Microsoft. All rights reserved.
# Import and re-export from the actual implementation
from agent_framework_lab_{{cookiecutter.package_name}} import * # noqa: F403, F401
python/packages/lab/cookiecutter-agent-framework-lab/{{cookiecutter.package_name}}/src/agent_framework_lab_{{cookiecutter.package_name}}/__init__.py
-21
View File
@@ -1,21 +0,0 @@
# Copyright (c) Microsoft. All rights reserved.
"""
{{ cookiecutter.package_description }}
"""
import importlib.metadata
# Import your main exports here
# from .main_module import MainClass, main_function
try:
__version__ = importlib.metadata.version(__name__)
except importlib.metadata.PackageNotFoundError:
__version__ = "0.0.0" # Fallback for development mode
__all__ = [
# List your exports here
# "MainClass",
# "main_function",
]
python/packages/lab/cookiecutter-agent-framework-lab/{{cookiecutter.package_name}}/src/agent_framework_lab_{{cookiecutter.package_name}}/py.typed
View File
python/packages/lab/cookiecutter-agent-framework-lab/{{cookiecutter.package_name}}/tests/__init__.py
-1
View File
@@ -1 +0,0 @@
# Copyright (c) Microsoft. All rights reserved.
python/packages/lab/cookiecutter-agent-framework-lab/{{cookiecutter.package_name}}/tests/test_{{cookiecutter.package_name}}.py
-15
View File
@@ -1,15 +0,0 @@
# Copyright (c) Microsoft. All rights reserved.
"""Tests for {{ cookiecutter.package_name }} module."""
import pytest
from agent_framework_lab_{{cookiecutter.package_name}} import __version__
class Test{{cookiecutter.package_name | title}}:
"""Test the {{ cookiecutter.package_name }} module."""
def test_version(self):
"""Test package version is defined."""
assert __version__ is not None
assert __version__ == "{{ cookiecutter.version }}"
python/packages/lab/gaia/LICENSE
-21
View File
@@ -1,21 +0,0 @@
MIT License
Copyright (c) Microsoft Corporation.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
python/packages/lab/gaia/README.md
+4 -2
View File
@@ -3,12 +3,14 @@
The GAIA benchmark can be used for evaluating agents and workflows built using the Agent Framework.
It includes built-in benchmarks as well as utilities for running custom evaluations.
> **Note**: This module is part of the consolidated `agent-framework-lab` package. Install the package with the `gaia` extra to use this module.
## Setup
Use `uv` to install the package with GAIA dependencies:
Install the agent-framework-lab package with GAIA dependencies:
```bash
uv pip install "agent-framework-lab-gaia"
pip install "agent-framework-lab[gaia]"
```
Set up Hugging Face token:
python/packages/lab/gaia/agent_framework/__init__.py
-4
View File
@@ -1,4 +0,0 @@
# Copyright (c) Microsoft. All rights reserved.
# This makes agent_framework a namespace package
__path__ = __import__("pkgutil").extend_path(__path__, __name__)
python/packages/lab/gaia/agent_framework/lab/__init__.py
-4
View File
@@ -1,4 +0,0 @@
# Copyright (c) Microsoft. All rights reserved.
# This makes agent_framework.lab a namespace package
__path__ = __import__("pkgutil").extend_path(__path__, __name__)
python/packages/lab/gaia/agent_framework_lab_gaia/__init__.py
+7 -9
View File
@@ -1,8 +1,6 @@
# Copyright (c) Microsoft. All rights reserved.
"""
GAIA benchmark module for Agent Framework.
"""
"""GAIA benchmark module for Agent Framework."""
import importlib.metadata
@@ -16,13 +14,13 @@ except importlib.metadata.PackageNotFoundError:
__all__ = [
"GAIA",
"GAIATelemetryConfig",
"gaia_scorer",
"viewer_main",
"Task",
"Prediction",
"Evaluation",
"Evaluator",
"GAIATelemetryConfig",
"Prediction",
"Task",
"TaskResult",
"TaskRunner",
"Evaluator",
"gaia_scorer",
"viewer_main",
]
python/packages/lab/gaia/agent_framework_lab_gaia/_types.py
+3 -3
View File
@@ -6,12 +6,12 @@ from dataclasses import dataclass
from typing import Any, Protocol, runtime_checkable
__all__ = [
"Task",
"Prediction",
"Evaluation",
"Evaluator",
"Prediction",
"Task",
"TaskResult",
"TaskRunner",
"Evaluator",
]
python/packages/lab/gaia/agent_framework_lab_gaia/gaia.py
+42 -61
View File
@@ -1,8 +1,6 @@
# Copyright (c) Microsoft. All rights reserved.
"""
GAIA benchmark implementation for Agent Framework.
"""
"""GAIA benchmark implementation for Agent Framework."""
import asyncio
import json
@@ -22,7 +20,7 @@ from tqdm import tqdm
from ._types import Evaluation, Evaluator, Prediction, Task, TaskResult, TaskRunner
__all__ = ["GAIA", "gaia_scorer", "GAIATelemetryConfig"]
__all__ = ["GAIA", "GAIATelemetryConfig", "gaia_scorer"]
class GAIATelemetryConfig:
@@ -36,8 +34,7 @@ class GAIATelemetryConfig:
trace_to_file: bool = False,
file_path: str | None = None,
):
"""
Initialize telemetry configuration.
"""Initialize telemetry configuration.
Args:
enable_tracing: Whether to enable OpenTelemetry tracing
@@ -74,8 +71,9 @@ class GAIATelemetryConfig:
try:
import json
import os
from collections.abc import Sequence
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace import ReadableSpan, TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, SpanExporter, SpanExportResult
from opentelemetry.trace import get_tracer_provider
@@ -85,7 +83,7 @@ class GAIATelemetryConfig:
# Ensure directory exists
os.makedirs(os.path.dirname(os.path.abspath(file_path)), exist_ok=True)
def export(self, spans) -> SpanExportResult:
def export(self, spans: Sequence[ReadableSpan]) -> SpanExportResult:
try:
with open(self.file_path, "a", encoding="utf-8") as f:
for span in spans:
@@ -131,8 +129,10 @@ def _normalize_number_str(number_str: str) -> float:
return float("inf")
def _split_string(s: str, chars: list[str] = [",", ";"]) -> list[str]:
def _split_string(s: str, chars: list[str] | None = None) -> list[str]:
"""Split string by multiple delimiters."""
if chars is None:
chars = [",", ";"]
return re.split(f"[{''.join(chars)}]", s)
@@ -146,8 +146,7 @@ def _normalize_str(s: str, remove_punct: bool = True) -> str:
def gaia_scorer(model_answer: str, ground_truth: str) -> bool:
"""
Official GAIA scoring function.
"""Official GAIA scoring function.
Args:
model_answer: The model's answer
@@ -170,22 +169,21 @@ def gaia_scorer(model_answer: str, ground_truth: str) -> bool:
if is_float(ground_truth):
# numeric exact match after normalization
return _normalize_number_str(model_answer) == float(ground_truth)
elif any(ch in ground_truth for ch in [",", ";"]):
if any(ch in ground_truth for ch in [",", ";"]):
# list with per-element compare (number or string)
gt_elems = _split_string(ground_truth)
ma_elems = _split_string(model_answer)
if len(gt_elems) != len(ma_elems):
return False
comparisons = []
for ma, gt in zip(ma_elems, gt_elems):
for ma, gt in zip(ma_elems, gt_elems, strict=False):
if is_float(gt):
comparisons.append(_normalize_number_str(ma) == float(gt))
else:
comparisons.append(_normalize_str(ma, remove_punct=False) == _normalize_str(gt, remove_punct=False))
return all(comparisons)
else:
# string normalize + exact
return _normalize_str(model_answer) == _normalize_str(ground_truth)
# string normalize + exact
return _normalize_str(model_answer) == _normalize_str(ground_truth)
def _read_jsonl(path: Path) -> Iterable[dict[str, Any]]:
@@ -238,8 +236,7 @@ def _load_gaia_local(repo_dir: Path, wanted_levels: list[int] | None = None, max
class GAIA:
"""
GAIA benchmark runner for Agent Framework.
"""GAIA benchmark runner for Agent Framework.
GAIA (General AI Assistant) is a benchmark for general-purpose AI assistants.
This class provides utilities to run the benchmark with custom agents.
@@ -252,8 +249,7 @@ class GAIA:
hf_token: str | None = None,
telemetry_config: GAIATelemetryConfig | None = None,
):
"""
Initialize GAIA benchmark runner.
"""Initialize GAIA benchmark runner.
Args:
evaluator: Custom evaluator function. If None, uses default GAIA scorer.
@@ -282,7 +278,6 @@ class GAIA:
def _ensure_data(self) -> Path:
"""Ensure GAIA data is available locally."""
if self.data_dir.exists() and any(self.data_dir.rglob("metadata.jsonl")):
return self.data_dir
@@ -347,14 +342,12 @@ class GAIA:
# Add results to span
if span:
span.set_attributes(
{
"gaia.task.runtime_seconds": runtime_seconds,
"gaia.task.is_correct": evaluation.is_correct,
"gaia.task.score": evaluation.score,
"gaia.task.prediction_length": len(prediction.prediction or ""),
}
)
span.set_attributes({
"gaia.task.runtime_seconds": runtime_seconds,
"gaia.task.is_correct": evaluation.is_correct,
"gaia.task.score": evaluation.score,
"gaia.task.prediction_length": len(prediction.prediction or ""),
})
return TaskResult(
task_id=task.task_id,
@@ -368,14 +361,12 @@ class GAIA:
# Record error in span
if span:
span.set_attributes(
{
"gaia.task.runtime_seconds": runtime_seconds,
"gaia.task.error": str(e),
"gaia.task.is_correct": False,
"gaia.task.score": 0.0,
}
)
span.set_attributes({
"gaia.task.runtime_seconds": runtime_seconds,
"gaia.task.error": str(e),
"gaia.task.is_correct": False,
"gaia.task.score": 0.0,
})
span.record_exception(e)
return TaskResult(
@@ -396,8 +387,7 @@ class GAIA:
timeout: int | None = None,
out: str | None = None,
) -> list[TaskResult]:
"""
Run the GAIA benchmark.
"""Run the GAIA benchmark.
Args:
task_runner: Function that takes a Task and returns a Prediction
@@ -425,10 +415,7 @@ class GAIA:
data_path = self._ensure_data()
# Parse level parameter
if isinstance(level, int):
levels = [level]
else:
levels = level
levels = [level] if isinstance(level, int) else level
# Load tasks
with self.tracer.start_as_current_span(
@@ -442,11 +429,9 @@ class GAIA:
tasks = _load_gaia_local(data_path, wanted_levels=levels, max_n=max_n)
if load_span:
load_span.set_attributes(
{
"gaia.tasks.loaded_count": len(tasks),
}
)
load_span.set_attributes({
"gaia.tasks.loaded_count": len(tasks),
})
if not tasks:
raise RuntimeError(
@@ -458,11 +443,9 @@ class GAIA:
# Update benchmark span with task info
if benchmark_span:
benchmark_span.set_attributes(
{
"gaia.benchmark.total_tasks": len(tasks),
}
)
benchmark_span.set_attributes({
"gaia.benchmark.total_tasks": len(tasks),
})
# Run tasks
semaphore = asyncio.Semaphore(parallel)
@@ -484,14 +467,12 @@ class GAIA:
# Update benchmark span with final results
if benchmark_span:
benchmark_span.set_attributes(
{
"gaia.benchmark.accuracy": accuracy,
"gaia.benchmark.correct_count": correct,
"gaia.benchmark.total_count": len(results),
"gaia.benchmark.avg_runtime_seconds": avg_runtime,
}
)
benchmark_span.set_attributes({
"gaia.benchmark.accuracy": accuracy,
"gaia.benchmark.correct_count": correct,
"gaia.benchmark.total_count": len(results),
"gaia.benchmark.avg_runtime_seconds": avg_runtime,
})
print("\nGAIA Benchmark Results:")
print(f"Accuracy: {accuracy:.3f} ({correct}/{len(results)})")
python/packages/lab/gaia/pyproject.toml
-86
View File
@@ -1,86 +0,0 @@
[project]
name = "agent-framework-lab-gaia"
description = "GAIA benchmark module for Microsoft Agent Framework."
authors = [{ name = "Microsoft", email = "af-support@microsoft.com"}]
readme = "README.md"
requires-python = ">=3.10"
version = "0.1.0b1"
license-files = ["LICENSE"]
urls.homepage = "https://learn.microsoft.com/en-us/semantic-kernel/overview/"
urls.source = "https://github.com/microsoft/agent-framework/tree/main/python"
urls.release_notes = "https://github.com/microsoft/agent-framework/releases?q=tag%3Apython-1&expanded=true"
urls.issues = "https://github.com/microsoft/agent-framework/issues"
classifiers = [
"License :: OSI Approved :: MIT License",
"Development Status :: 2 - Pre-Alpha",
"Intended Audience :: Developers",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
]
dependencies = [
"agent-framework",
"pydantic>=2.0.0",
"opentelemetry-api>=1.24.0",
"tqdm>=4.60.0",
"huggingface-hub>=0.20.0",
"orjson>=3.8.0",
]
[project.scripts]
gaia_viewer = "agent_framework_lab_gaia:viewer_main"
[build-system]
requires = ["setuptools>=64", "wheel"]
build-backend = "setuptools.build_meta"
[tool.setuptools]
packages = ["agent_framework_lab_gaia", "agent_framework.lab.gaia"]
[tool.setuptools.package-data]
agent_framework_lab_gaia = ["py.typed"]
[tool.ruff]
line-length = 120
target-version = "py310"
extend-exclude = ["tests", "__pycache__"]
[tool.ruff.lint]
select = ["E", "F", "I", "W", "UP", "C4", "N"]
ignore = ["N803", "N806", "N999", "UP007"]
[tool.ruff.format]
quote-style = "double"
[tool.mypy]
python_version = "3.10"
strict = true
check_untyped_defs = true
disallow_untyped_defs = true
disallow_incomplete_defs = true
disallow_untyped_decorators = true
warn_redundant_casts = true
warn_unused_ignores = true
warn_return_any = true
warn_unreachable = true
show_error_codes = true
implicit_reexport = true
packages = ["agent_framework_lab_gaia"]
[tool.poe]
executor.type = "uv"
include = "../../../shared_tasks.toml"
[tool.poe.tasks]
test = "pytest --cov=agent_framework_lab_gaia --cov-report=term-missing:skip-covered tests"
[tool.pytest.ini_options]
testpaths = ["tests"]
pythonpath = ["."]
addopts = "--strict-markers --strict-config"
markers = [
"unit: marks tests as unit tests",
"integration: marks tests as integration tests",
]
python/packages/lab/gaia/samples/gaia_sample.py
+4 -8
View File
@@ -1,7 +1,6 @@
# Copyright (c) Microsoft. All rights reserved.
"""
GAIA Benchmark Sample
"""GAIA Benchmark Sample.
To run this sample, execute it from the root directory of the agent-framework repository:
cd /path/to/agent-framework
@@ -11,12 +10,11 @@ This avoids namespace package conflicts that occur when running from within the
"""
from agent_framework.azure import AzureAIAgentClient
from agent_framework.lab.gaia import GAIA, Evaluation, GAIATelemetryConfig, Prediction, Task
from azure.identity.aio import AzureCliCredential
from agent_framework.lab.gaia import GAIA, Evaluation, GAIATelemetryConfig, Prediction, Task
async def evaluate_task(task: Task, prediction: Prediction) -> Evaluation:
def evaluate_task(task: Task, prediction: Prediction) -> Evaluation:
"""Evaluate the prediction for a given task."""
# Simple evaluation: check if the prediction contains the answer
is_correct = (task.answer or "").lower() in prediction.prediction.lower()
@@ -24,12 +22,10 @@ async def evaluate_task(task: Task, prediction: Prediction) -> Evaluation:
async def main() -> None:
"""Run GAIA benchmark with telemetry configuration."""
# Configure telemetry for tracing
telemetry_config = GAIATelemetryConfig(
enable_tracing=True, # Enable OpenTelemetry tracing
# Optional: Configure external endpoints
# otlp_endpoint="http://localhost:4317", # For Aspire Dashboard or other OTLP endpoints
# applicationinsights_connection_string="your_connection_string", # For Azure Monitor
# Configure local file tracing
trace_to_file=True, # Export traces to local file
file_path="gaia_benchmark_traces.jsonl", # Custom file path for traces
python/packages/lab/gaia/tests/__init__.py
-1
View File
@@ -1 +0,0 @@
# Copyright (c) Microsoft. All rights reserved.
python/packages/lab/gaia/tests/test_gaia.py
+4 -5
View File
@@ -2,27 +2,26 @@
"""Tests for GAIA benchmark implementation."""
import pytest
from agent_framework_lab_gaia import gaia_scorer
class TestGAIAScorer:
"""Test the GAIA scoring function."""
def test_numeric_exact_match(self):
"""Test numeric exact matching."""
assert gaia_scorer("42", "42") is True
assert gaia_scorer("42.0", "42") is True
assert gaia_scorer("42", "42.0") is True
assert gaia_scorer("42", "43") is False
def test_string_normalization(self):
"""Test string normalization and matching."""
assert gaia_scorer("Hello World", "hello world") is True
assert gaia_scorer("Hello, World!", "helloworld") is True
assert gaia_scorer("test", "TEST") is True
assert gaia_scorer("test", "different") is False
def test_list_matching(self):
"""Test list matching with comma/semicolon separation."""
assert gaia_scorer("1,2,3", "1,2,3") is True
@@ -30,7 +29,7 @@ class TestGAIAScorer:
assert gaia_scorer("apple,banana", "apple,banana") is True
assert gaia_scorer("1,2,3", "1,2,4") is False
assert gaia_scorer("1,2", "1,2,3") is False
def test_none_handling(self):
"""Test handling of None values."""
assert gaia_scorer("None", "test") is False
python/packages/lab/lightning/LICENSE
-21
View File
@@ -1,21 +0,0 @@
MIT License
Copyright (c) Microsoft Corporation.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
python/packages/lab/lightning/README.md
+6 -2
View File
@@ -1,11 +1,15 @@
# Agent Framework Lab - Agent Framework x Agent Lightning
# Agent Framework Lab - Lightning
RL Module for Microsoft Agent Framework
> **Note**: This module is part of the consolidated `agent-framework-lab` package. Install the package with the `lightning` extra to use this module.
## Installation
Install the agent-framework-lab package with Lightning dependencies:
```bash
pip install agent-framework-lab-lightning
pip install "agent-framework-lab[lightning]"
```
## Usage
python/packages/lab/lightning/agent_framework/__init__.py
-4
View File
@@ -1,4 +0,0 @@
# Copyright (c) Microsoft. All rights reserved.
# This makes agent_framework a namespace package
__path__ = __import__("pkgutil").extend_path(__path__, __name__)
python/packages/lab/lightning/agent_framework/lab/__init__.py
-4
View File
@@ -1,4 +0,0 @@
# Copyright (c) Microsoft. All rights reserved.
# This makes agent_framework.lab a namespace package
__path__ = __import__("pkgutil").extend_path(__path__, __name__)
python/packages/lab/lightning/agent_framework_lab_lightning/__init__.py
+2 -11
View File
@@ -1,8 +1,6 @@
# Copyright (c) Microsoft. All rights reserved.
"""
RL Module for Microsoft Agent Framework
"""
"""RL Module for Microsoft Agent Framework."""
import importlib.metadata
@@ -11,11 +9,4 @@ try:
except importlib.metadata.PackageNotFoundError:
__version__ = "0.0.0" # Fallback for development mode
# Import your main exports here
# from .main_module import MainClass, main_function
__all__ = [
# List your exports here
# "MainClass",
# "main_function",
]
__all__: list[str] = []
python/packages/lab/lightning/pyproject.toml
-87
View File
@@ -1,87 +0,0 @@
[project]
name = "agent-framework-lab-lightning"
description = "RL Module for Microsoft Agent Framework"
authors = [{ name = "Microsoft", email = "af-support@microsoft.com"}]
readme = "README.md"
requires-python = ">=3.10"
version = "0.1.0b1"
license-files = ["LICENSE"]
urls.homepage = "https://learn.microsoft.com/en-us/semantic-kernel/overview/"
urls.source = "https://github.com/microsoft/agent-framework/tree/main/python"
urls.release_notes = "https://github.com/microsoft/agent-framework/releases?q=tag%3Apython-1&expanded=true"
urls.issues = "https://github.com/microsoft/agent-framework/issues"
classifiers = [
"License :: OSI Approved :: MIT License",
"Development Status :: 2 - Pre-Alpha",
"Intended Audience :: Developers",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
]
dependencies = [
"agent-framework",
"pydantic>=2.0.0",
# Add your specific dependencies here
]
[project.scripts]
lightning = "agent_framework_lab_lightning:main"
[build-system]
requires = ["setuptools>=64", "wheel"]
build-backend = "setuptools.build_meta"
[tool.setuptools]
packages = ["agent_framework_lab_lightning", "agent_framework.lab.lightning"]
[tool.setuptools.package-data]
agent_framework_lab_lightning = ["py.typed"]
[tool.ruff]
line-length = 120
target-version = "py310"
extend-exclude = ["tests", "__pycache__"]
[tool.ruff.lint]
select = ["E", "F", "I", "W", "UP", "C4", "N"]
ignore = ["N803", "N806", "N999", "UP007"]
[tool.ruff.format]
quote-style = "double"
[tool.mypy]
python_version = "3.10"
strict = true
check_untyped_defs = true
disallow_untyped_defs = true
disallow_incomplete_defs = true
disallow_untyped_decorators = true
warn_redundant_casts = true
warn_unused_ignores = true
warn_return_any = true
warn_unreachable = true
show_error_codes = true
implicit_reexport = true
packages = ["agent_framework_lab_lightning"]
[tool.poe]
executor.type = "uv"
include = "../../../shared_tasks.toml"
[tool.poe.tasks]
test = "pytest --cov=agent_framework_lab_lightning --cov-report=term-missing:skip-covered tests"
mypy = "mypy agent_framework_lab_lightning"
[tool.pytest.ini_options]
testpaths = ["tests"]
pythonpath = ["."]
addopts = "--strict-markers --strict-config"
markers = [
"unit: marks tests as unit tests",
"integration: marks tests as integration tests",
]
python/packages/lab/lightning/tests/__init__.py
-1
View File
@@ -1 +0,0 @@
# Copyright (c) Microsoft. All rights reserved.
python/packages/lab/lightning/tests/test_lightning.py
+5 -3
View File
@@ -2,14 +2,16 @@
"""Tests for lightning module."""
import pytest
from agent_framework_lab_lightning import __version__
class TestLightning:
"""Test the lightning module."""
def test_version(self):
"""Test package version is defined."""
assert __version__ is not None
assert __version__ == "0.1.0b1"
# In development mode, version falls back to "0.0.0"
# In installed mode, it would be the actual package version
assert isinstance(__version__, str)
assert len(__version__) > 0
python/packages/lab/cookiecutter-agent-framework-lab/{{cookiecutter.package_name}}/src/agent_framework/__init__.py → python/packages/lab/namespace/agent_framework/__init__.py
View File
python/packages/lab/cookiecutter-agent-framework-lab/{{cookiecutter.package_name}}/src/agent_framework/lab/__init__.py → python/packages/lab/namespace/agent_framework/lab/__init__.py
View File
python/packages/lab/gaia/agent_framework/lab/gaia/__init__.py → python/packages/lab/namespace/agent_framework/lab/gaia/__init__.py
+1 -1
View File
@@ -1,4 +1,4 @@
# Copyright (c) Microsoft. All rights reserved.
# Import and re-export from the actual implementation
from agent_framework_lab_gaia import * # noqa: F403, F401
from agent_framework_lab_gaia import * # noqa: F403
python/packages/lab/tau2/namespace/agent_framework/lab/tau2/__init__.py → python/packages/lab/namespace/agent_framework/lab/lightning/__init__.py
+1 -1
View File
@@ -1,4 +1,4 @@
# Copyright (c) Microsoft. All rights reserved.
# Import and re-export from the actual implementation
from agent_framework_lab_tau2 import * # noqa: F403, F401
from agent_framework_lab_lightning import * # noqa: F403
python/packages/lab/lightning/agent_framework/lab/lightning/__init__.py → python/packages/lab/namespace/agent_framework/lab/tau2/__init__.py
+1 -1
View File
@@ -1,4 +1,4 @@
# Copyright (c) Microsoft. All rights reserved.
# Import and re-export from the actual implementation
from agent_framework_lab_lightning import * # noqa: F403, F401
from agent_framework_lab_tau2 import * # noqa: F403
python/packages/lab/pyproject.toml
+136
View File
@@ -0,0 +1,136 @@
[project]
name = "agent-framework-lab"
description = "Experimental modules for Microsoft Agent Framework"
authors = [{ name = "Microsoft", email = "af-support@microsoft.com"}]
readme = "README.md"
requires-python = ">=3.10"
version = "0.1.0b1"
license-files = ["LICENSE"]
urls.homepage = "https://learn.microsoft.com/en-us/semantic-kernel/overview/"
urls.source = "https://github.com/microsoft/agent-framework/tree/main/python"
urls.release_notes = "https://github.com/microsoft/agent-framework/releases?q=tag%3Apython-1&expanded=true"
urls.issues = "https://github.com/microsoft/agent-framework/issues"
classifiers = [
"License :: OSI Approved :: MIT License",
"Development Status :: 2 - Pre-Alpha",
"Intended Audience :: Developers",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
]
dependencies = [
"agent-framework",
]
[project.optional-dependencies]
# GAIA benchmark module dependencies
gaia = [
"pydantic>=2.0.0",
"opentelemetry-api>=1.24.0",
"tqdm>=4.60.0",
"huggingface-hub>=0.20.0",
"orjson>=3.8.0",
]
# Lightning RL training module dependencies
lightning = [
"pydantic>=2.0.0",
]
# TAU2 benchmark module dependencies
tau2 = [
"pydantic>=2.0.0",
"tiktoken>=0.11.0",
"loguru>=0.7.3",
"numpy",
"tau2@ git+https://github.com/sierra-research/tau2-bench@5ba9e3e56db57c5e4114bf7f901291f09b2c5619",
]
[project.scripts]
gaia_viewer = "agent_framework_lab_gaia:viewer_main"
lightning = "agent_framework_lab_lightning:main"
[build-system]
requires = ["setuptools>=64", "wheel"]
build-backend = "setuptools.build_meta"
[tool.setuptools]
packages = [
"agent_framework_lab_gaia",
"agent_framework_lab_lightning",
"agent_framework_lab_tau2",
"agent_framework.lab.gaia",
"agent_framework.lab.lightning",
"agent_framework.lab.tau2",
]
[tool.setuptools.package-dir]
"agent_framework_lab_gaia" = "gaia/agent_framework_lab_gaia"
"agent_framework_lab_lightning" = "lightning/agent_framework_lab_lightning"
"agent_framework_lab_tau2" = "tau2/agent_framework_lab_tau2"
"agent_framework.lab.gaia" = "namespace/agent_framework/lab/gaia"
"agent_framework.lab.lightning" = "namespace/agent_framework/lab/lightning"
"agent_framework.lab.tau2" = "namespace/agent_framework/lab/tau2"
[tool.setuptools.package-data]
agent_framework_lab_gaia = ["py.typed"]
agent_framework_lab_lightning = ["py.typed"]
agent_framework_lab_tau2 = ["py.typed"]
[tool.ruff]
extend = "../../pyproject.toml"
[tool.ruff.lint]
ignore = ["T201", "ASYNC230", "INP001"] # Allow print statements, blocking file operations, and implicit namespace packages in lab modules
[tool.coverage.run]
omit = [
"**/__init__.py"
]
[tool.pyright]
extend = "../../pyproject.toml"
exclude = ['gaia/tests', 'lightning/tests', 'tau2/tests', 'namespace', '**/samples']
[tool.mypy]
plugins = ['pydantic.mypy']
strict = true
python_version = "3.10"
ignore_missing_imports = true
disallow_untyped_defs = true
no_implicit_optional = true
check_untyped_defs = true
warn_return_any = true
show_error_codes = true
warn_unused_ignores = false
disallow_incomplete_defs = true
disallow_untyped_decorators = true
[tool.bandit]
targets = ["agent_framework_lab_gaia", "agent_framework_lab_lightning", "agent_framework_lab_tau2"]
exclude_dirs = ["gaia/tests", "lightning/tests", "tau2/tests"]
[tool.poe]
executor.type = "uv"
include = "../../shared_tasks.toml"
[tool.poe.tasks]
mypy-gaia = "mypy --config-file $POE_ROOT/pyproject.toml gaia/agent_framework_lab_gaia"
mypy-lightning = "mypy --config-file $POE_ROOT/pyproject.toml lightning/agent_framework_lab_lightning"
mypy-tau2 = "mypy --config-file $POE_ROOT/pyproject.toml tau2/agent_framework_lab_tau2"
mypy = ["mypy-gaia", "mypy-lightning", "mypy-tau2"]
test = "pytest --cov-report=term-missing:skip-covered --junitxml=test-results.xml"
test-gaia = "pytest gaia/tests --cov=agent_framework_lab_gaia --cov-report=term-missing:skip-covered"
test-lightning = "pytest lightning/tests --cov=agent_framework_lab_lightning --cov-report=term-missing:skip-covered"
test-tau2 = "pytest tau2/tests --cov=agent_framework_lab_tau2 --cov-report=term-missing:skip-covered"
[tool.pytest.ini_options]
pythonpath = ["."]
addopts = "--strict-markers --strict-config"
markers = [
"unit: marks tests as unit tests",
"integration: marks tests as integration tests",
]
python/packages/lab/tau2/LICENSE
-21
View File
@@ -1,21 +0,0 @@
MIT License
Copyright (c) Microsoft Corporation.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
python/packages/lab/tau2/README.md
+12 -7
View File
@@ -2,7 +2,10 @@
τ²-bench implements a simulation framework for evaluating customer service agents across various domains.
> **Note**: This module is part of the consolidated `agent-framework-lab` package. Install the package with the `tau2` extra to use this module.
The framework orchestrates conversations between two AI agents:
- **Customer Service Agent**: Follows domain-specific policies and has access to tools (e.g., booking systems, databases)
- **User Simulator**: Simulates realistic customer behavior with specific goals and scenarios
@@ -20,8 +23,10 @@ Each evaluation runs a multi-turn conversation where the user simulator presents
## Installation
Install the agent-framework-lab package with TAU2 dependencies:
```bash
pip install agent-framework-lab-tau2
pip install "agent-framework-lab[tau2]"
```
Download data from [Tau2-Bench](https://github.com/sierra-research/tau2-bench):
@@ -45,7 +50,7 @@ export TAU2_DATA_DIR="data"
```python
import asyncio
from agent_framework.openai import OpenAIChatClient
from agent_framework_lab_tau2 import TaskRunner
from agent_framework.lab.tau2 import TaskRunner
from tau2.domains.airline.environment import get_tasks
async def run_single_task():
@@ -126,7 +131,7 @@ export OPENAI_BASE_URL="https://your-custom-endpoint.com/v1"
### Custom Agent Implementation
```python
from agent_framework_lab_tau2 import TaskRunner
from agent_framework.lab.tau2 import TaskRunner
from agent_framework import ChatAgent
class CustomTaskRunner(TaskRunner):
@@ -149,8 +154,8 @@ class CustomTaskRunner(TaskRunner):
### Custom Workflow Integration
```python
from agent_framework._workflow import WorkflowBuilder, AgentExecutor
from agent_framework_lab_tau2 import TaskRunner
from agent_framework import WorkflowBuilder, AgentExecutor
from agent_framework.lab.tau2 import TaskRunner
class WorkflowTaskRunner(TaskRunner):
def build_conversation_workflow(self, assistant_agent, user_simulator_agent):
@@ -172,7 +177,7 @@ class WorkflowTaskRunner(TaskRunner):
### Utility Functions
```python
from agent_framework_lab_tau2 import patch_env_set_state, unpatch_env_set_state
from agent_framework.lab.tau2 import patch_env_set_state, unpatch_env_set_state
# Enable compatibility patches for τ²-bench integration
patch_env_set_state()
@@ -187,4 +192,4 @@ This package is part of the Microsoft Agent Framework Lab. Please see the main r
## License
This project is licensed under the MIT License - see the LICENSE file for details.
This project is licensed under the MIT License - see the LICENSE file for details.
python/packages/lab/tau2/agent_framework_lab_tau2/__init__.py
+1 -3
View File
@@ -1,8 +1,6 @@
# Copyright (c) Microsoft. All rights reserved.
"""
Tau2 Benchmark for Agent Framework.
"""
"""Tau2 Benchmark for Agent Framework."""
import importlib.metadata
python/packages/lab/tau2/agent_framework_lab_tau2/_message_utils.py
+20 -20
View File
@@ -51,7 +51,7 @@ def log_messages(messages: list[ChatMessage]) -> None:
Provides visual debugging by color-coding different message roles and
content types. Escapes HTML-like characters to prevent log formatting issues.
"""
_logger = logger.opt(colors=True)
logger_ = logger.opt(colors=True)
for msg in messages:
# Handle different content types
if hasattr(msg, "contents") and msg.contents:
@@ -60,53 +60,53 @@ def log_messages(messages: list[ChatMessage]) -> None:
if content.type == "text":
escape_text = content.text.replace("<", r"\<")
if msg.role == Role.SYSTEM:
_logger.info(f"<cyan>[SYSTEM]</cyan> {escape_text}")
logger_.info(f"<cyan>[SYSTEM]</cyan> {escape_text}")
elif msg.role == Role.USER:
_logger.info(f"<green>[USER]</green> {escape_text}")
logger_.info(f"<green>[USER]</green> {escape_text}")
elif msg.role == Role.ASSISTANT:
_logger.info(f"<blue>[ASSISTANT]</blue> {escape_text}")
logger_.info(f"<blue>[ASSISTANT]</blue> {escape_text}")
elif msg.role == Role.TOOL:
_logger.info(f"<yellow>[TOOL]</yellow> {escape_text}")
logger_.info(f"<yellow>[TOOL]</yellow> {escape_text}")
else:
_logger.info(f"<magenta>[{msg.role.value.upper()}]</magenta> {escape_text}")
logger_.info(f"<magenta>[{msg.role.value.upper()}]</magenta> {escape_text}")
elif content.type == "function_call":
function_call_text = f"{content.name}({content.arguments})"
function_call_text = function_call_text.replace("<", r"\<")
_logger.info(f"<yellow>[TOOL_CALL]</yellow> 🔧 {function_call_text}")
logger_.info(f"<yellow>[TOOL_CALL]</yellow> 🔧 {function_call_text}")
elif content.type == "function_result":
function_result_text = f"ID:{content.call_id} -> {content.result}"
function_result_text = function_result_text.replace("<", r"\<")
_logger.info(f"<yellow>[TOOL_RESULT]</yellow> 🔨 {function_result_text}")
logger_.info(f"<yellow>[TOOL_RESULT]</yellow> 🔨 {function_result_text}")
else:
content_text = str(content).replace("<", r"\<")
_logger.info(f"<magenta>[{msg.role.value.upper()}] ({content.type})</magenta> {content_text}")
logger_.info(f"<magenta>[{msg.role.value.upper()}] ({content.type})</magenta> {content_text}")
else:
# Fallback for content without type
text_content = str(content).replace("<", r"\<")
if msg.role == Role.SYSTEM:
_logger.info(f"<cyan>[SYSTEM]</cyan> {text_content}")
logger_.info(f"<cyan>[SYSTEM]</cyan> {text_content}")
elif msg.role == Role.USER:
_logger.info(f"<green>[USER]</green> {text_content}")
logger_.info(f"<green>[USER]</green> {text_content}")
elif msg.role == Role.ASSISTANT:
_logger.info(f"<blue>[ASSISTANT]</blue> {text_content}")
logger_.info(f"<blue>[ASSISTANT]</blue> {text_content}")
elif msg.role == Role.TOOL:
_logger.info(f"<yellow>[TOOL]</yellow> {text_content}")
logger_.info(f"<yellow>[TOOL]</yellow> {text_content}")
else:
_logger.info(f"<magenta>[{msg.role.value.upper()}]</magenta> {text_content}")
logger_.info(f"<magenta>[{msg.role.value.upper()}]</magenta> {text_content}")
elif hasattr(msg, "text") and msg.text:
# Handle simple text messages
text_content = msg.text.replace("<", r"\<")
if msg.role == Role.SYSTEM:
_logger.info(f"<cyan>[SYSTEM]</cyan> {text_content}")
logger_.info(f"<cyan>[SYSTEM]</cyan> {text_content}")
elif msg.role == Role.USER:
_logger.info(f"<green>[USER]</green> {text_content}")
logger_.info(f"<green>[USER]</green> {text_content}")
elif msg.role == Role.ASSISTANT:
_logger.info(f"<blue>[ASSISTANT]</blue> {text_content}")
logger_.info(f"<blue>[ASSISTANT]</blue> {text_content}")
elif msg.role == Role.TOOL:
_logger.info(f"<yellow>[TOOL]</yellow> {text_content}")
logger_.info(f"<yellow>[TOOL]</yellow> {text_content}")
else:
_logger.info(f"<magenta>[{msg.role.value.upper()}]</magenta> {text_content}")
logger_.info(f"<magenta>[{msg.role.value.upper()}]</magenta> {text_content}")
else:
# Fallback for other message formats
text_content = str(msg).replace("<", r"\<")
_logger.info(f"<magenta>[{msg.role.value.upper()}]</magenta> {text_content}")
logger_.info(f"<magenta>[{msg.role.value.upper()}]</magenta> {text_content}")
python/packages/lab/tau2/agent_framework_lab_tau2/_sliding_window.py
+1
View File
@@ -58,6 +58,7 @@ class SlidingWindowChatMessageList(ChatMessageList):
def get_token_count(self) -> int:
"""Estimate token count for a list of messages using tiktoken.
Args:
messages: List of ChatMessage objects
system_message: Optional system message to include in count
python/packages/lab/tau2/agent_framework_lab_tau2/_tau2_utils.py
+9 -20
View File
@@ -35,11 +35,7 @@ def convert_tau2_tool_to_ai_function(tau2_tool: Tool) -> AIFunction[Any, Any]:
def wrapped_func(**kwargs: Any) -> Any:
result = tau2_tool(**kwargs)
# Deep copy to prevent mutations of returned data
if isinstance(result, BaseModel):
result = result.model_copy(deep=True)
else:
result = deepcopy(result)
return result
return result.model_copy(deep=True) if isinstance(result, BaseModel) else deepcopy(result)
return AIFunction(
name=tau2_tool.name,
@@ -55,7 +51,6 @@ def convert_agent_framework_messages_to_tau2_messages(messages: list[ChatMessage
Handles role mapping, text extraction, function calls, and function results.
Function results are converted to separate ToolMessage instances.
"""
tau2_messages = []
for msg in messages:
@@ -126,16 +121,13 @@ def patch_env_set_state() -> None:
initialization_actions: list[EnvFunctionCall] | None,
message_history: list[Message],
) -> None:
if self.solo_mode:
if any(isinstance(message, UserMessage) for message in message_history):
raise ValueError("User messages are not allowed in solo mode")
if self.solo_mode and any(isinstance(message, UserMessage) for message in message_history):
raise ValueError("User messages are not allowed in solo mode")
def get_actions_from_messages(
messages: list[Message],
) -> list[tuple[ToolCall, ToolMessage]]:
"""
Get the actions from the messages.
"""
"""Get the actions from the messages."""
messages = deepcopy(messages)[::-1]
actions = []
while messages:
@@ -194,14 +186,13 @@ def unpatch_env_set_state() -> None:
def _dump_function_result(result: Any) -> Any:
if isinstance(result, BaseModel):
return result.model_dump_json()
elif isinstance(result, list):
if isinstance(result, list):
return [_dump_function_result(item) for item in result]
elif isinstance(result, dict):
if isinstance(result, dict):
return {k: _dump_function_result(v) for k, v in result.items()}
elif result is None:
if result is None:
return None
else:
return result
return result
def _to_native(obj: Any) -> Any:
@@ -227,9 +218,7 @@ def _to_native(obj: Any) -> Any:
def _recursive_json_deserialize(obj: Any) -> Any:
"""
Recursively deserialize a JSON object.
"""
"""Recursively deserialize a JSON object."""
if isinstance(obj, str):
try:
deserialized = json.loads(obj)
python/packages/lab/tau2/agent_framework_lab_tau2/runner.py
+2 -9
View File
@@ -100,6 +100,7 @@ class TaskRunner:
return self
def __repr__(self) -> str:
"""Return string representation of TaskRunner."""
return (
f"TaskRunner(max_steps={self.max_steps}, step_count={self.step_count}, "
f"full_conversation_length={len(self.full_conversation)}, "
@@ -108,7 +109,6 @@ class TaskRunner:
def should_not_stop(self, response: AgentExecutorResponse) -> bool:
"""Based on the response, check whether we should or not stop the conversation."""
# Determine who sent this based on executor_id
is_from_agent = response.executor_id == ASSISTANT_AGENT_ID
is_from_user = response.executor_id == USER_SIMULATOR_ID
@@ -165,7 +165,6 @@ class TaskRunner:
Returns:
The assistant agent.
"""
# Initialize tau2 environment and extract tools/policy
# This provides the domain-specific context (airline customer service in this case)
env = get_environment()
@@ -216,7 +215,6 @@ class TaskRunner:
Returns:
The user simulator agent.
"""
# User simulator follows tau2's guidelines for realistic customer behavior
# No tools available - users typically don't have direct system access
user_sim_guidelines = get_global_user_sim_guidelines(use_tools=False)
@@ -277,7 +275,6 @@ class TaskRunner:
Returns:
The conversation workflow.
"""
# STEP 1: Create workflow executors
# Each executor wraps an agent or function for workflow orchestration
self._assistant_executor = AgentExecutor(assistant_agent, id=ASSISTANT_AGENT_ID)
@@ -287,7 +284,7 @@ class TaskRunner:
# STEP 2: Build the conversation workflow
# Creates a cyclic workflow: Orchestrator -> Assistant -> Orchestrator -> User -> Orchestrator...
# The orchestrator acts as a message router that flips roles and routes to appropriate agent
workflow = (
return (
WorkflowBuilder(max_iterations=10000) # Unlimited - we control termination via should_not_stop
.set_start_executor(orchestrator) # Orchestrator manages the conversation flow
.add_edge(orchestrator, self._assistant_executor) # Route messages to assistant
@@ -299,8 +296,6 @@ class TaskRunner:
.build()
)
return workflow
async def run(
self,
task: Task,
@@ -325,7 +320,6 @@ class TaskRunner:
Returns:
Complete conversation history as ChatMessage list for evaluation
"""
logger.info(f"Starting workflow agent for task {task.id}: {task.description.purpose}") # type: ignore[unused-ignore]
logger.info(f"Assistant chat client: {assistant_chat_client}")
logger.info(f"User simulator chat client: {user_simuator_chat_client}")
@@ -390,7 +384,6 @@ class TaskRunner:
Side Effects:
Stores detailed evaluation results in self.full_reward_info
"""
# Handle missing termination reason (can happen with unexpected workflow endings)
if termination_reason is None:
termination_reason = TerminationReason.TOO_MANY_ERRORS
python/packages/lab/tau2/namespace/agent_framework/__init__.py
-4
View File
@@ -1,4 +0,0 @@
# Copyright (c) Microsoft. All rights reserved.
# This makes agent_framework a namespace package
__path__ = __import__("pkgutil").extend_path(__path__, __name__)
python/packages/lab/tau2/namespace/agent_framework/lab/__init__.py
-4
View File
@@ -1,4 +0,0 @@
# Copyright (c) Microsoft. All rights reserved.
# This makes agent_framework.lab a namespace package
__path__ = __import__("pkgutil").extend_path(__path__, __name__)
python/packages/lab/tau2/pyproject.toml
-99
View File
@@ -1,99 +0,0 @@
[project]
name = "agent-framework-lab-tau2"
description = "Tau2 Benchmark for Agent Framework."
authors = [{ name = "Microsoft", email = "af-support@microsoft.com"}]
readme = "README.md"
requires-python = ">=3.10"
version = "0.1.0b1"
license-files = ["LICENSE"]
urls.homepage = "https://learn.microsoft.com/en-us/semantic-kernel/overview/"
urls.source = "https://github.com/microsoft/agent-framework/tree/main/python"
urls.release_notes = "https://github.com/microsoft/agent-framework/releases?q=tag%3Apython-1&expanded=true"
urls.issues = "https://github.com/microsoft/agent-framework/issues"
classifiers = [
"License :: OSI Approved :: MIT License",
"Development Status :: 2 - Pre-Alpha",
"Intended Audience :: Developers",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
]
dependencies = [
"agent-framework",
"pydantic>=2.0.0",
"tiktoken>=0.11.0",
"loguru>=0.7.3",
"tau2@git+https://github.com/sierra-research/tau2-bench@5ba9e3e56db57c5e4114bf7f901291f09b2c5619",
]
[build-system]
requires = ["setuptools>=64", "wheel"]
build-backend = "setuptools.build_meta"
[tool.setuptools]
packages = ["agent_framework_lab_tau2", "agent_framework.lab.tau2"]
[tool.setuptools.package-dir]
"agent_framework.lab.tau2" = "namespace/agent_framework/lab/tau2"
"agent_framework_lab_tau2" = "agent_framework_lab_tau2"
[tool.setuptools.package-data]
agent_framework_lab_tau2 = ["py.typed"]
[tool.ruff]
line-length = 120
target-version = "py310"
extend-exclude = ["tests", "__pycache__"]
[tool.ruff.lint]
select = ["E", "F", "I", "W", "UP", "C4", "N"]
ignore = ["N803", "N806", "N999", "UP007"]
[tool.ruff.format]
quote-style = "double"
[tool.mypy]
python_version = "3.10"
strict = true
check_untyped_defs = true
disallow_untyped_defs = true
disallow_incomplete_defs = true
disallow_untyped_decorators = true
warn_redundant_casts = true
warn_unused_ignores = true
warn_return_any = true
warn_unreachable = true
show_error_codes = true
implicit_reexport = true
packages = ["agent_framework_lab_tau2"]
exclude = [
"data",
]
[tool.pyright]
exclude = ["**/data"]
[tool.poe]
executor.type = "uv"
include = "../../../shared_tasks.toml"
[tool.poe.tasks]
test = "pytest --cov=agent_framework_lab_tau2 --cov-report=term-missing:skip-covered tests"
mypy = "mypy agent_framework_lab_tau2"
setup-data = "python tests/setup_data.py"
purge-data = "python tests/purge_data.py"
[tool.pytest.ini_options]
testpaths = ["tests"]
pythonpath = ["."]
addopts = "--strict-markers --strict-config"
markers = [
"unit: marks tests as unit tests",
"integration: marks tests as integration tests",
]
env = [
"TAU2_DATA_DIR=data",
]
python/packages/lab/tau2/samples/run_benchmark.py
+107 -68
View File
@@ -8,12 +8,11 @@ import traceback
from datetime import datetime
from typing import Any
from agent_framework.lab.tau2 import TaskRunner, patch_env_set_state
from agent_framework.openai import OpenAIChatClient
from loguru import logger
from tau2.domains.airline.environment import get_tasks
from agent_framework_lab_tau2 import TaskRunner, patch_env_set_state
def to_dumpable(result: dict[str, Any]) -> dict[str, Any]:
"""Convert benchmark result to JSONL-serializable format.
@@ -33,16 +32,15 @@ def to_dumpable(result: dict[str, Any]) -> dict[str, Any]:
"config": result["config"],
"task": result["task"].model_dump(),
}
else:
# Success case: full result structure
return {
"id": result["task"].id,
"evaluation": result["evaluation"].model_dump(), # Detailed evaluation metrics
"config": result["config"], # Model configuration used
"termination_reason": result["termination_reason"].value, # Enum to string
"messages": [m.model_dump() for m in result["messages"]], # Full conversation
"task": result["task"].model_dump(), # Task specification
}
# Success case: full result structure
return {
"id": result["task"].id,
"evaluation": result["evaluation"].model_dump(), # Detailed evaluation metrics
"config": result["config"], # Model configuration used
"termination_reason": result["termination_reason"].value, # Enum to string
"messages": [m.model_dump() for m in result["messages"]], # Full conversation
"task": result["task"].model_dump(), # Task specification
}
async def run_benchmark(assistant_model: str, user_model: str, debug_task_id: str | None, max_steps: int):
@@ -66,15 +64,13 @@ async def run_benchmark(assistant_model: str, user_model: str, debug_task_id: st
Creates timestamped JSONL file with detailed results for analysis
Prints summary statistics to console with colored logging
"""
# STEP 1: Configure output handling based on execution mode
result_fp = None
result_filename = None
if debug_task_id is None:
# Full benchmark mode: create timestamped results file
timestamp = datetime.now().strftime("%m%d%H%M") # Format: MMDDHHMM
result_filename = f"results/{assistant_model}_user-{user_model}_{timestamp}.jsonl"
os.makedirs("results", exist_ok=True)
result_fp = open(result_filename, "a") # Append mode for resumability
logger.info(f"Results will be saved to: {result_filename}")
else:
# Debug mode: single task, no file output, verbose logging
@@ -84,7 +80,7 @@ async def run_benchmark(assistant_model: str, user_model: str, debug_task_id: st
tasks = get_tasks() # Loads all tau2 airline customer service tasks
logger.info(f"Found {len(tasks)} tasks in the dataset")
_logger = logger.opt(colors=True) # Enable colored console output
logger_ = logger.opt(colors=True) # Enable colored console output
# Validate required OpenAI configuration
# Both models use the same endpoint but can be different model types
@@ -121,67 +117,110 @@ async def run_benchmark(assistant_model: str, user_model: str, debug_task_id: st
all_rewards: list[float] = [] # Stores reward scores for final statistics
task_runner = TaskRunner(max_steps=max_steps) # Reusable workflow orchestrator
# STEP 6: Execute benchmark across all tasks
for task in tasks:
_logger.info(f"<red>Testing task #{task.id}</red>")
_logger.info(f"<cyan>Purpose:</cyan> {task.description.purpose}") # type: ignore
# Initialize result structure for this task
result: dict[str, Any] = {
"config": {
"assistant": assistant_chat_client.ai_model_id,
"user": user_simulator_chat_client.ai_model_id,
},
"task": task,
}
# Log user scenario context for transparency
if task.user_scenario and task.user_scenario.instructions:
_logger.info(f"<cyan>User scenario:</cyan> {task.user_scenario.instructions.reason_for_call}") # type: ignore
try:
# Execute the workflow: agent + user simulator conversation
conversation = await task_runner.run(task, assistant_chat_client, user_simulator_chat_client)
# Evaluate performance using tau2's comprehensive metrics
reward_value = task_runner.evaluate(task, conversation, task_runner.termination_reason)
# Store detailed results for analysis
result["evaluation"] = task_runner.full_reward_info # Full evaluation breakdown
result["messages"] = conversation # Complete conversation history
result["termination_reason"] = task_runner.termination_reason # How conversation ended
# Log evaluation results (escape HTML for colored output)
reward_str = str(task_runner.full_reward_info).replace("<", r"\<")
_logger.info(f"<cyan>Final evaluation:</cyan> {reward_str}")
except Exception as e:
# Robust error handling: capture all failures for analysis
_logger.error(f"<red>Error testing task #{task.id}:</red> {e}")
result["error"] = traceback.format_exc() # Full stack trace for debugging
traceback.print_exc() # Console output for immediate debugging
reward_value = 0.0 # Zero score for failed runs
# STEP 7: Persist results incrementally (enables partial analysis)
# STEP 6: Execute benchmark across all tasks with proper file handling
def write_result(result_fp, result):
"""Write result to file if file pointer is provided."""
if result_fp is not None:
result_fp.write(json.dumps(to_dumpable(result), default=str) + "\n")
all_rewards.append(reward_value) # Track for final statistics
# Use context manager for file handling
if result_filename:
with open(result_filename, "a") as result_fp:
for task in tasks:
logger_.info(f"<red>Testing task #{task.id}</red>")
logger_.info(f"<cyan>Purpose:</cyan> {task.description.purpose}") # type: ignore
# Reset runner state for next task
task_runner.reinit()
# Initialize result structure for this task
result: dict[str, Any] = {
"config": {
"assistant": assistant_chat_client.ai_model_id,
"user": user_simulator_chat_client.ai_model_id,
},
"task": task,
}
# STEP 8: Finalize and report aggregate results
if result_fp is not None:
result_fp.close()
# Log user scenario context for transparency
if task.user_scenario and task.user_scenario.instructions:
logger_.info(f"<cyan>User scenario:</cyan> {task.user_scenario.instructions.reason_for_call}") # type: ignore
# Calculate overall benchmark performance
try:
# Execute the workflow: agent + user simulator conversation
conversation = await task_runner.run(task, assistant_chat_client, user_simulator_chat_client)
# Evaluate performance using tau2's comprehensive metrics
reward_value = task_runner.evaluate(task, conversation, task_runner.termination_reason)
# Store detailed results for analysis
result["evaluation"] = task_runner.full_reward_info # Full evaluation breakdown
result["messages"] = conversation # Complete conversation history
result["termination_reason"] = task_runner.termination_reason # How conversation ended
# Log evaluation results (escape HTML for colored output)
reward_str = str(task_runner.full_reward_info).replace("<", r"\<")
logger_.info(f"<cyan>Final evaluation:</cyan> {reward_str}")
except Exception as e:
# Robust error handling: capture all failures for analysis
logger_.error(f"<red>Error testing task #{task.id}:</red> {e}")
result["error"] = traceback.format_exc() # Full stack trace for debugging
traceback.print_exc() # Console output for immediate debugging
reward_value = 0.0 # Zero score for failed runs
# STEP 7: Persist results incrementally (enables partial analysis)
write_result(result_fp, result)
all_rewards.append(reward_value) # Track for final statistics
# Reset runner state for next task
task_runner.reinit()
else:
# Debug mode without file output
for task in tasks:
logger_.info(f"<red>Testing task #{task.id}</red>")
logger_.info(f"<cyan>Purpose:</cyan> {task.description.purpose}") # type: ignore
# Initialize result structure for this task
result: dict[str, Any] = {
"config": {
"assistant": assistant_chat_client.ai_model_id,
"user": user_simulator_chat_client.ai_model_id,
},
"task": task,
}
# Log user scenario context for transparency
if task.user_scenario and task.user_scenario.instructions:
logger_.info(f"<cyan>User scenario:</cyan> {task.user_scenario.instructions.reason_for_call}") # type: ignore
try:
# Execute the workflow: agent + user simulator conversation
conversation = await task_runner.run(task, assistant_chat_client, user_simulator_chat_client)
# Evaluate performance using tau2's comprehensive metrics
reward_value = task_runner.evaluate(task, conversation, task_runner.termination_reason)
# Log evaluation results (escape HTML for colored output)
reward_str = str(task_runner.full_reward_info).replace("<", r"\<")
logger_.info(f"<cyan>Final evaluation:</cyan> {reward_str}")
except Exception as e:
# Robust error handling: capture all failures for analysis
logger_.error(f"<red>Error testing task #{task.id}:</red> {e}")
traceback.print_exc() # Console output for immediate debugging
reward_value = 0.0 # Zero score for failed runs
all_rewards.append(reward_value) # Track for final statistics
# Reset runner state for next task
task_runner.reinit()
# STEP 8: Calculate overall benchmark performance and report final statistics
all_accuracy = sum(all_rewards) / len(all_rewards) if all_rewards else 0.0
# Report final statistics with colored formatting
_logger.info("<green>Final Results:</green>")
_logger.info(f"<cyan>All tasks accuracy:</cyan> {all_accuracy:.2f} ({int(sum(all_rewards))}/{len(tasks)})")
logger_.info("<green>Final Results:</green>")
logger_.info(f"<cyan>All tasks accuracy:</cyan> {all_accuracy:.2f} ({int(sum(all_rewards))}/{len(tasks)})")
if __name__ == "__main__":
python/packages/lab/tau2/tests/__init__.py
-1
View File
@@ -1 +0,0 @@
# Copyright (c) Microsoft. All rights reserved.
python/packages/lab/tau2/tests/test_message_utils.py
+3 -2
View File
@@ -2,7 +2,7 @@
from unittest.mock import patch
from agent_framework._types import ChatMessage, Role, TextContent, FunctionCallContent, FunctionResultContent
from agent_framework._types import ChatMessage, FunctionCallContent, FunctionResultContent, Role, TextContent
from agent_framework_lab_tau2._message_utils import flip_messages, log_messages
@@ -120,7 +120,8 @@ def test_flip_messages_mixed_conversation():
flipped = flip_messages(messages)
# Should have: system (unchanged), assistant (from user), user (from assistant, filtered), assistant (from final assistant)
# Should have: system (unchanged), assistant (from user), user (from assistant, filtered),
# assistant (from final assistant)
assert len(flipped) == 4
# Check each flipped message
python/packages/lab/tau2/tests/test_sliding_window.py
+6 -4
View File
@@ -2,10 +2,10 @@
"""Tests for sliding window message list."""
import pytest
from unittest.mock import patch
from agent_framework._types import ChatMessage, Role, TextContent, FunctionCallContent, FunctionResultContent
import pytest
from agent_framework._types import ChatMessage, FunctionCallContent, FunctionResultContent, Role, TextContent
from agent_framework_lab_tau2._sliding_window import SlidingWindowChatMessageList
@@ -225,7 +225,8 @@ def test_estimate_any_object_token_count_non_serializable():
async def test_real_world_scenario():
"""Test a realistic conversation scenario."""
sliding_window = SlidingWindowChatMessageList(
max_tokens=30, system_message="You are a helpful assistant" # Moderate limit
max_tokens=30,
system_message="You are a helpful assistant", # Moderate limit
)
# Simulate a conversation
@@ -239,7 +240,8 @@ async def test_real_world_scenario():
role=Role.ASSISTANT,
contents=[
TextContent(
text="I'd be happy to help with weather information, but I don't have access to current weather data."
text="I'd be happy to help with weather information, "
"but I don't have access to current weather data."
)
],
),
python/packages/lab/tau2/tests/test_tau2_utils.py
+20 -8
View File
@@ -2,19 +2,30 @@
"""Tests for tau2 utils module."""
from typing import Any, cast
from pydantic import BaseModel
import pytest
from agent_framework._tools import AIFunction
from agent_framework._types import ChatMessage, Role, TextContent, FunctionCallContent, FunctionResultContent
from agent_framework._types import ChatMessage, FunctionCallContent, FunctionResultContent, Role, TextContent
from agent_framework_lab_tau2._tau2_utils import (
convert_tau2_tool_to_ai_function,
convert_agent_framework_messages_to_tau2_messages,
convert_tau2_tool_to_ai_function,
)
from tau2.data_model.message import SystemMessage, UserMessage, AssistantMessage, ToolMessage, ToolCall
from tau2.domains.airline.environment import get_environment
from tau2.data_model.message import AssistantMessage, SystemMessage, ToolCall, ToolMessage, UserMessage
# Try to import get_environment and handle missing data files
try:
from tau2.domains.airline.environment import get_environment
# Try to initialize the environment to check if data files are available
try:
get_environment()
TAU2_DATA_AVAILABLE = True
except FileNotFoundError:
TAU2_DATA_AVAILABLE = False
except ImportError:
TAU2_DATA_AVAILABLE = False
@pytest.mark.skipif(not TAU2_DATA_AVAILABLE, reason="tau2 data files not available")
def test_convert_tau2_tool_to_ai_function_basic():
"""Test basic conversion from tau2 tool to AIFunction."""
# Get real tools from tau2 environment
@@ -38,6 +49,7 @@ def test_convert_tau2_tool_to_ai_function_basic():
assert callable(ai_function.func)
@pytest.mark.skipif(not TAU2_DATA_AVAILABLE, reason="tau2 data files not available")
def test_convert_tau2_tool_to_ai_function_multiple_tools():
"""Test conversion with multiple tau2 tools."""
# Get real tools from tau2 environment
@@ -48,7 +60,7 @@ def test_convert_tau2_tool_to_ai_function_multiple_tools():
ai_functions = [convert_tau2_tool_to_ai_function(tool) for tool in tools[:3]] # Test first 3 tools
# Verify all conversions
for ai_function, tau2_tool in zip(ai_functions, tools[:3]):
for ai_function, tau2_tool in zip(ai_functions, tools[:3], strict=False):
assert isinstance(ai_function, AIFunction)
assert ai_function.name == tau2_tool.name
assert ai_function.description == tau2_tool._get_description()