Python: Introduce Agent Framework Lab with GAIA Benchmark and Lighting project for RL (#719)

* prepare eval package * add gaia benchmark to eval package * update telemetry * renaming * organization * organize into namespace packages; rename to labs * update cookie cutter instruction * update gaia runner * use temp directory * Rename "labs" --> "lab" * update * update gaia sample * update status * Add lighting project * Add listing for lighting
2026-06-16 21:04:09 +08:00 · 2025-09-16 22:59:21 -07:00
parent a7e35a4fea
commit a1b0a28f9c
42 changed files with 1989 additions and 110 deletions
@@ -0,0 +1,144 @@
+# Agent Framework Lab
+
+This directory contains experimental packages for Microsoft Agent Framework that are distributed as separate installable packages under the `agent_framework.lab` namespace.
+Lab packages are not part of the core framework and may experience breaking changes or be deprecated in the future.
+
+## What are Lab Packages?
+
+Lab packages are extensions to the core Agent Framework that falls into
+one of the following categories:
+
+1. Incubation of new features that may get incorprated by the core framework.
+2. Research prototypes built on the core framework.
+3. Benchmarks and experimentation tools.
+
+## Lab Packages
+
+- [**gaia**](./gaia/): GAIA benchmark implementation (`agent-framework-lab-gaia`)
+- [**lighting**](./lighting/): Reinforcement learning for agents (`agent-framework-lab-lighting`)
+
+## How do I contribute?
+
+This repo only contains lab packages maintained by Microsoft.
+If you want to contribute, please take the following steps:
+
+1. Follow the [Create a New Lab Package](#create-new-lab-package) guide
+   below to create your own lab package.
+2. Create a new repo on GitHub and check in your package there.
+3. Tag your repo with `agent-framework-lab` for bettter discovery.
+4. Submit a PR to this repo (github.com/microsoft/agent-framework)
+   to add a link to your repo in the [list](#lab-packages) above.
+   **The PR title must contain "[New Lab Package]"**.
+5. We will review your repo and decide whether to approve it.
+
+Follow the [guidelines](#guidelines) when you create your package, our decision
+to accept your PR will be based on your idea as well as the quality of your
+code.
+
+We may decide to maintain your package in this repo. In that case, we will
+contact you directly.
+
+## Package Structure
+
+Each lab package follows this structure:
+
+```
+packages/lab/{lab_name}/
+├── agent_framework/
+│   └── lab/
+│       └── {lab_name}/
+│           └── __init__.py          # Imports from agent_framework_lab_{lab_name}
+├── agent_framework_lab_{lab_name}/ # Actual implementation package
+│   ├── __init__.py                  # Main exports and __version__
+│   ├── {module_files}.py           # Implementation modules
+│   └── py.typed                     # Type hints marker
+├── tests/
+│   ├── __init__.py
+│   └── test_{lab_name}.py          # Package tests
+├── pyproject.toml                   # Package configuration
+├── README.md                        # Package-specific documentation
+└── LICENSE                          # MIT License
+```
+
+## Creating a New Lab Package
+
+### Create The Package
+
+First ensure `cookiecutter` is installed.
+
+```bash
+pip install cookiecutter
+```
+
+Then go to the directory where you want to create the package:
+
+```bash
+cookiecutter /path/to/agent-framework/python/packages/lab/cookiecutter-agent-framework-lab
+```
+
+You will be prompted for:
+
+- **package_name**: The name of your lab package (e.g., "lighting", "vision")
+- **package_display_name**: Human-readable name (e.g., "Lighting Tools", "Computer Vision")
+- **package_description**: Brief description (auto-generated from display name)
+- **include_cli_script**: Whether to include a CLI script (y/n)
+
+### After Package Creation
+
+1. **Implement your functionality** in `agent_framework_lab_your_package_name/`
+2. **Update exports** in `__init__.py` `__all__` list
+3. **Add dependencies** to `pyproject.toml`
+4. **Write tests** in the `tests/` directory
+5. **Update README** with usage examples and API documentation
+
+### Add to Workspace (only for packages maintained in this repo)
+
+After creating your package, add it to the workspace configuration:
+
+```
+# Edit python/pyproject.toml
+# Add to dependencies section:
+dependencies = [
+    # ... existing packages ...
+    "agent-framework-lab-your-package-name",
+]
+
+# Add to [tool.uv.sources] section:
+agent-framework-lab-your-package-name = { workspace = true }
+```
+
+### Usage
+
+Once created, users can install your lab package
+
+1. directly from your repo:
+
+```bash
+pip install git+https://github.com/your-username/your-lab-package-repo.git
+```
+
+2. or from PyPI if you have uploaded your lab package there:
+
+```bash
+pip install "agent-framework-lab-your-package-name"
+```
+
+Then, they can use your lab package:
+
+```python
+from agent_framework.lab.your_package_name import YourClass, your_function
+
+# Use the functionality
+instance = YourClass()
+result = your_function()
+```
+
+## Guidelines
+
+1. **Naming**: Use lowercase with hyphens for package names (`agent-framework-lab-your-package-name`)
+2. **Namespace**: Always use `agent_framework.lab.your_package_name` for imports
+3. **Versioning**: Start with `0.1.0b1` for beta releases
+4. **Dependencies**: Minimize external dependencies, always include `agent-framework`
+5. **Documentation**: Include comprehensive README with usage examples
+6. **Tests**: Write comprehensive tests with good coverage
+7. **Type hints**: Always include type hints and `py.typed` file
@@ -0,0 +1,58 @@
+# Cookiecutter Template for Agent Framework Lab Packages
+
+This is a cookiecutter template for creating new lab packages in the Microsoft Agent Framework.
+
+## Usage
+
+```bash
+cd /path/to/agent-framework/python/packages/lab
+cookiecutter ./cookiecutter-agent-framework-lab
+```
+
+You will be prompted for the following information:
+
+- **package_name**: The name of your lab package (e.g., "lighting", "vision")
+- **package_display_name**: Human-readable name (e.g., "Lighting Tools", "Computer Vision")
+- **package_description**: Brief description of the package (auto-generated from display name)
+- **version**: Starting version (default: 0.1.0b1)
+- **author_name**: Author name (default: Microsoft)
+- **author_email**: Author email (default: SK-Support@microsoft.com)
+- **include_cli_script**: Whether to include a CLI script (y/n)
+- **cli_script_name**: Name of CLI script if included
+
+## What Gets Generated
+
+The template creates a complete lab package structure:
+
+```
+{package_name}/
+├── agent_framework/
+│   └── lab/
+│       └── {package_name}/
+│           └── __init__.py
+├── agent_framework_lab_{package_name}/
+│   ├── __init__.py
+│   └── py.typed
+├── tests/
+│   ├── __init__.py
+│   └── test_{package_name}.py
+├── pyproject.toml
+├── README.md
+└── LICENSE
+```
+
+## After Generation
+
+1. Implement your functionality in `agent_framework_lab_{package_name}/`
+2. Update the `__all__` exports in `__init__.py`
+3. Add your dependencies to `pyproject.toml`
+4. Write comprehensive tests
+5. Update the README with usage examples
+
+## Integration
+
+Don't forget to add your new package to the workspace:
+
+1. Add to `python/pyproject.toml` dependencies
+2. Add to `[tool.uv.sources]` section
+3. Test installation with `uv run python -c "from agent_framework.lab.{name} import *"`
@@ -0,0 +1,20 @@
+{
+    "package_name": "",
+    "package_display_name": "",
+    "package_description": "{{ cookiecutter.package_display_name }} module for Microsoft Agent Framework.",
+    "version": "0.1.0b1",
+    "author_name": "Microsoft",
+    "author_email": "SK-Support@microsoft.com",
+    "license": "MIT",
+    "include_cli_script": ["y", "n"],
+    "cli_script_name": "{{ cookiecutter.package_name }}_cli",
+    "python_requires": ">=3.10",
+    "within_microsoft_agent_framework_repo": ["y", "n"],
+    "__prompts__": {
+        "within_microsoft_agent_framework_repo": "Are you creating this package within the github.com/microsoft/agent-framework repo or a fork of it? (If yes, ensure you create it in python/packages/lab/ directory)"
+    },
+    "_templates_suffix": "",
+    "_copy_without_render": [
+        "*.py.typed"
+    ]
+}
@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) Microsoft Corporation.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
@@ -0,0 +1,56 @@
+# Agent Framework Lab - {{ cookiecutter.package_display_name }}
+
+{{ cookiecutter.package_description }}
+
+## Installation
+
+```bash
+pip install agent-framework-lab-{{ cookiecutter.package_name }}
+```
+
+## Usage
+
+```python
+from agent_framework.lab.{{ cookiecutter.package_name }} import YourClass
+
+# Your usage example here
+instance = YourClass()
+```
+
+## Overview
+
+Brief description of what this lab package provides and its main features.
+
+## Features
+
+- Feature 1: Description
+- Feature 2: Description
+- Feature 3: Description
+
+## Examples
+
+### Basic Usage
+
+```python
+from agent_framework.lab.{{ cookiecutter.package_name }} import YourClass
+
+# Example usage
+```
+
+### Advanced Usage
+
+```python
+# More advanced examples
+```
+
+## API Reference
+
+Document your main classes and functions here.
+
+## Contributing
+
+This package is part of the Microsoft Agent Framework Lab. Please see the main repository for contribution guidelines.
+
+## License
+
+This project is licensed under the {{ cookiecutter.license }} License - see the LICENSE file for details.
@@ -0,0 +1,4 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+# This makes agent_framework a namespace package
+__path__ = __import__("pkgutil").extend_path(__path__, __name__)
@@ -0,0 +1,4 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+# This makes agent_framework.lab a namespace package
+__path__ = __import__("pkgutil").extend_path(__path__, __name__)
@@ -0,0 +1,4 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+# Import and re-export from the actual implementation
+from agent_framework_lab_{{cookiecutter.package_name}} import *  # noqa: F403, F401
@@ -0,0 +1,16 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""
+{{ cookiecutter.package_description }}
+"""
+
+# Import your main exports here
+# from .main_module import MainClass, main_function
+
+__all__ = [
+    # List your exports here
+    # "MainClass",
+    # "main_function",
+]
+
+__version__ = "{{ cookiecutter.version }}"
@@ -0,0 +1,94 @@
+[project]
+name = "agent-framework-lab-{{ cookiecutter.package_name }}"
+description = "{{ cookiecutter.package_description }}"
+authors = [{ name = "{{ cookiecutter.author_name }}", email = "{{ cookiecutter.author_email }}"}]
+readme = "README.md"
+requires-python = "{{ cookiecutter.python_requires }}"
+version = "{{ cookiecutter.version }}"
+license-files = ["LICENSE"]
+urls.homepage = "https://learn.microsoft.com/en-us/semantic-kernel/overview/"
+urls.source = "https://github.com/microsoft/agent-framework/tree/main/python"
+urls.release_notes = "https://github.com/microsoft/agent-framework/releases?q=tag%3Apython-1&expanded=true"
+urls.issues = "https://github.com/microsoft/agent-framework/issues"
+classifiers = [
+  "License :: OSI Approved :: {{ cookiecutter.license }} License",
+  "Development Status :: 2 - Pre-Alpha",
+  "Intended Audience :: Developers",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Programming Language :: Python :: 3.13",
+]
+dependencies = [
+  "agent-framework",
+  "pydantic>=2.0.0",
+  # Add your specific dependencies here
+]
+
+{% if cookiecutter.include_cli_script == "y" %}
+[project.scripts]
+{{ cookiecutter.cli_script_name }} = "agent_framework_lab_{{ cookiecutter.package_name }}:main"
+{% endif %}
+
+[build-system]
+requires = ["setuptools>=64", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools]
+packages = ["agent_framework_lab_{{ cookiecutter.package_name }}", "agent_framework.lab.{{ cookiecutter.package_name }}"]
+
+[tool.setuptools.package-data]
+agent_framework_lab_{{ cookiecutter.package_name }} = ["py.typed"]
+
+[tool.ruff]
+line-length = 120
+target-version = "py310"
+extend-exclude = ["tests", "__pycache__"]
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "W", "UP", "C4", "N"]
+ignore = ["N803", "N806", "N999", "UP007"]
+
+[tool.ruff.format]
+quote-style = "double"
+
+[tool.mypy]
+python_version = "3.10"
+strict = true
+check_untyped_defs = true
+disallow_untyped_defs = true
+disallow_incomplete_defs = true
+disallow_untyped_decorators = true
+warn_redundant_casts = true
+warn_unused_ignores = true
+warn_return_any = true
+warn_unreachable = true
+show_error_codes = true
+implicit_reexport = true
+packages = ["agent_framework_lab_{{ cookiecutter.package_name }}"]
+
+{% if cookiecutter.within_microsoft_agent_framework_repo == "y" %}
+[tool.poe]
+executor.type = "uv"
+include = "../../../shared_tasks.toml"
+[tool.poe.tasks]
+test = "pytest --cov=agent_framework_lab_{{ cookiecutter.package_name }} --cov-report=term-missing:skip-covered tests"
+mypy = "mypy agent_framework_lab_{{ cookiecutter.package_name }}"
+{% else %}
+[tool.poe.tasks]
+fmt = "ruff format"
+format = "ruff format"
+lint = "ruff check"
+test = "pytest --cov=agent_framework_lab_{{ cookiecutter.package_name }} --cov-report=term-missing:skip-covered tests"
+mypy = "mypy agent_framework_lab_{{ cookiecutter.package_name }}"
+{% endif %}
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+pythonpath = ["."]
+addopts = "--strict-markers --strict-config"
+markers = [
+  "unit: marks tests as unit tests",
+  "integration: marks tests as integration tests",
+]
@@ -0,0 +1 @@
+# Copyright (c) Microsoft. All rights reserved.
@@ -0,0 +1,15 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""Tests for {{ cookiecutter.package_name }} module."""
+
+import pytest
+from agent_framework_lab_{{cookiecutter.package_name}} import __version__
+
+
+class Test{{cookiecutter.package_name | title}}:
+    """Test the {{ cookiecutter.package_name }} module."""
+    
+    def test_version(self):
+        """Test package version is defined."""
+        assert __version__ is not None
+        assert __version__ == "{{ cookiecutter.version }}"
@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) Microsoft Corporation.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
@@ -0,0 +1,65 @@
+# Agent Framework Lab - GAIA
+
+The GAIA benchmark can be used for evaluating agents and workflows built using the Agent Framework.
+It includes built-in benchmarks as well as utilities for running custom evaluations.
+
+## Setup
+
+Use `uv` to install the package with GAIA dependencies:
+
+```bash
+uv pip install "agent-framework-lab-gaia"
+```
+
+Set up Hugging Face token:
+
+```bash
+export HF_TOKEN="hf\*..." # must have access to gaia-benchmark/GAIA
+```
+
+## Create an evaluation script
+
+Create a Python script (e.g., `run_gaia.py`) with the following content:
+
+```python
+from agent_framework.lab.gaia import GAIA, Task, Prediction, GAIATelemetryConfig
+
+async def run_task(task: Task) -> Prediction:
+    return Prediction(prediction="answer here", messages=[])
+
+async def main() -> None:
+    # Optional: Enable telemetry for detailed tracing
+    telemetry_config = GAIATelemetryConfig(
+        enable_tracing=True,
+        trace_to_file=True,
+        file_path="gaia_traces.jsonl"
+    )
+
+    runner = GAIA(telemetry_config=telemetry_config)
+    await runner.run(run_task, level=1, max_n=5, parallel=2)
+```
+
+See the [gaia_sample.py](./samples/gaia_sample.py) for more detail.
+
+### Run the evaluation
+
+Run the evaluation script using `uv`:
+
+```bash
+uv run python run_gaia.py
+```
+
+By default, the script will first look for cached GAIA data in the `data_gaia_hub` directory,
+and download it if not found.
+The result will be saved to `gaia_results_<timestamp>.jsonl`.
+
+**Don't run the script inside this directory because it will confuse the local `agent_framework` namespace
+package with the real one.**
+
+## View results
+
+We provide a console viewer for reading GAIA results:
+
+```bash
+uv run gaia_viewer "gaia_results_<timestamp>.jsonl" --detailed
+```
@@ -0,0 +1,4 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+# This makes agent_framework a namespace package
+__path__ = __import__("pkgutil").extend_path(__path__, __name__)
@@ -0,0 +1,4 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+# This makes agent_framework.lab a namespace package
+__path__ = __import__("pkgutil").extend_path(__path__, __name__)
@@ -0,0 +1,4 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+# Import and re-export from the actual implementation
+from agent_framework_lab_gaia import *  # noqa: F403, F401
@@ -0,0 +1,23 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""
+GAIA benchmark module for Agent Framework.
+"""
+
+from ._types import Evaluation, Evaluator, Prediction, Task, TaskResult, TaskRunner
+from .gaia import GAIA, GAIATelemetryConfig, gaia_scorer, viewer_main
+
+__all__ = [
+    "GAIA",
+    "GAIATelemetryConfig",
+    "gaia_scorer",
+    "viewer_main",
+    "Task",
+    "Prediction",
+    "Evaluation",
+    "TaskResult",
+    "TaskRunner",
+    "Evaluator",
+]
+
+__version__ = "0.1.0b1"
@@ -0,0 +1,79 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""Common types for agent evaluation."""
+
+from dataclasses import dataclass
+from typing import Any, Protocol, runtime_checkable
+
+__all__ = [
+    "Task",
+    "Prediction",
+    "Evaluation",
+    "TaskResult",
+    "TaskRunner",
+    "Evaluator",
+]
+
+
+@dataclass
+class Task:
+    """Represents a task to be evaluated."""
+
+    task_id: str
+    question: str
+    answer: str | None = None
+    level: int | None = None
+    file_name: str | None = None
+    metadata: dict[str, Any] | None = None
+
+
+@dataclass
+class Prediction:
+    """Represents a prediction made by an agent for a task."""
+
+    prediction: str
+    messages: list[Any] | None = None
+    metadata: dict[str, Any] | None = None
+
+    def __post_init__(self) -> None:
+        if self.messages is None:
+            self.messages = []
+
+
+@dataclass
+class Evaluation:
+    """Represents the evaluation result of a prediction."""
+
+    is_correct: bool
+    score: float
+    details: dict[str, Any] | None = None
+
+
+@dataclass
+class TaskResult:
+    """Complete result for a single task evaluation."""
+
+    task_id: str
+    task: Task
+    prediction: Prediction
+    evaluation: Evaluation
+    runtime_seconds: float | None = None
+    error: str | None = None
+
+
+@runtime_checkable
+class TaskRunner(Protocol):
+    """Protocol for running tasks."""
+
+    async def __call__(self, task: Task) -> Prediction:
+        """Run a single task and return the prediction."""
+        ...
+
+
+@runtime_checkable
+class Evaluator(Protocol):
+    """Protocol for evaluating predictions."""
+
+    async def __call__(self, task: Task, prediction: Prediction) -> Evaluation:
+        """Evaluate a prediction for a given task."""
+        ...
@@ -0,0 +1,625 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""
+GAIA benchmark implementation for Agent Framework.
+"""
+
+import asyncio
+import json
+import os
+import random
+import re
+import string
+import tempfile
+import time
+from collections.abc import Iterable
+from datetime import datetime
+from pathlib import Path
+from typing import Any
+
+from opentelemetry.trace import NoOpTracer, SpanKind, get_tracer
+from tqdm import tqdm
+
+from ._types import Evaluation, Evaluator, Prediction, Task, TaskResult, TaskRunner
+
+__all__ = ["GAIA", "gaia_scorer", "GAIATelemetryConfig"]
+
+
+class GAIATelemetryConfig:
+    """Configuration for GAIA telemetry and tracing."""
+
+    def __init__(
+        self,
+        enable_tracing: bool = False,
+        otlp_endpoint: str | None = None,
+        application_insights_connection_string: str | None = None,
+        enable_live_metrics: bool = False,
+        trace_to_file: bool = False,
+        file_path: str | None = None,
+    ):
+        """
+        Initialize telemetry configuration.
+
+        Args:
+            enable_tracing: Whether to enable OpenTelemetry tracing
+            otlp_endpoint: OTLP endpoint for trace export
+            application_insights_connection_string: Azure Monitor connection string
+            enable_live_metrics: Enable Azure Monitor live metrics
+            trace_to_file: Whether to export traces to local file
+            file_path: Path for local file export (defaults to gaia_traces.json)
+        """
+        self.enable_tracing = enable_tracing
+        self.otlp_endpoint = otlp_endpoint
+        self.application_insights_connection_string = application_insights_connection_string
+        self.enable_live_metrics = enable_live_metrics
+        self.trace_to_file = trace_to_file
+        self.file_path = file_path or "gaia_traces.json"
+
+    def setup_telemetry(self) -> None:
+        """Set up OpenTelemetry based on configuration."""
+        if not self.enable_tracing:
+            return
+
+        from agent_framework.telemetry import setup_telemetry
+
+        setup_telemetry(
+            enable_otel=True,
+            enable_sensitive_data=True,  # Enable for detailed task traces
+            otlp_endpoint=self.otlp_endpoint,
+            application_insights_connection_string=self.application_insights_connection_string,
+            enable_live_metrics=self.enable_live_metrics,
+        )
+
+        # Set up local file export if requested
+        if self.trace_to_file:
+            self._setup_file_export()
+
+    def _setup_file_export(self) -> None:
+        """Set up local file export for traces."""
+        try:
+            import json
+            import os
+
+            from opentelemetry.sdk.trace import TracerProvider
+            from opentelemetry.sdk.trace.export import BatchSpanProcessor, SpanExporter, SpanExportResult
+            from opentelemetry.trace import get_tracer_provider
+
+            class FileSpanExporter(SpanExporter):
+                def __init__(self, file_path: str):
+                    self.file_path = file_path
+                    # Ensure directory exists
+                    os.makedirs(os.path.dirname(os.path.abspath(file_path)), exist_ok=True)
+
+                def export(self, spans) -> SpanExportResult:
+                    try:
+                        with open(self.file_path, "a", encoding="utf-8") as f:
+                            for span in spans:
+                                span_data = {
+                                    "trace_id": format(span.context.trace_id, "032x") if span.context else "unknown",
+                                    "span_id": format(span.context.span_id, "016x") if span.context else "unknown",
+                                    "name": span.name,
+                                    "start_time": span.start_time,
+                                    "end_time": span.end_time,
+                                    "duration_ns": (span.end_time - span.start_time)
+                                    if (span.end_time and span.start_time)
+                                    else None,
+                                    "attributes": dict(span.attributes) if span.attributes else {},
+                                    "status": {
+                                        "status_code": span.status.status_code.name if span.status else "UNSET",
+                                        "description": span.status.description if span.status else None,
+                                    },
+                                }
+                                f.write(json.dumps(span_data, default=str) + "\n")
+                        return SpanExportResult.SUCCESS
+                    except Exception:
+                        return SpanExportResult.FAILURE
+
+                def shutdown(self) -> None:
+                    pass
+
+            tracer_provider = get_tracer_provider()
+            if isinstance(tracer_provider, TracerProvider):
+                file_exporter = FileSpanExporter(self.file_path)
+                tracer_provider.add_span_processor(BatchSpanProcessor(file_exporter))
+
+        except ImportError:
+            print("Warning: Could not set up file export for traces. Missing dependencies.")
+
+
+def _normalize_number_str(number_str: str) -> float:
+    """Normalize a number string for comparison."""
+    for ch in ["$", "%", ","]:
+        number_str = number_str.replace(ch, "")
+    try:
+        return float(number_str)
+    except ValueError:
+        return float("inf")
+
+
+def _split_string(s: str, chars: list[str] = [",", ";"]) -> list[str]:
+    """Split string by multiple delimiters."""
+    return re.split(f"[{''.join(chars)}]", s)
+
+
+def _normalize_str(s: str, remove_punct: bool = True) -> str:
+    """Normalize string for comparison."""
+    no_spaces = re.sub(r"\s", "", s or "")
+    if remove_punct:
+        table = str.maketrans("", "", string.punctuation)
+        return no_spaces.lower().translate(table)
+    return no_spaces.lower()
+
+
+def gaia_scorer(model_answer: str, ground_truth: str) -> bool:
+    """
+    Official GAIA scoring function.
+
+    Args:
+        model_answer: The model's answer
+        ground_truth: The ground truth answer
+
+    Returns:
+        True if the answer is correct, False otherwise
+    """
+
+    def is_float(x: Any) -> bool:
+        try:
+            float(x)
+            return True
+        except Exception:
+            return False
+
+    if model_answer is None:
+        model_answer = "None"
+
+    if is_float(ground_truth):
+        # numeric exact match after normalization
+        return _normalize_number_str(model_answer) == float(ground_truth)
+    elif any(ch in ground_truth for ch in [",", ";"]):
+        # list with per-element compare (number or string)
+        gt_elems = _split_string(ground_truth)
+        ma_elems = _split_string(model_answer)
+        if len(gt_elems) != len(ma_elems):
+            return False
+        comparisons = []
+        for ma, gt in zip(ma_elems, gt_elems):
+            if is_float(gt):
+                comparisons.append(_normalize_number_str(ma) == float(gt))
+            else:
+                comparisons.append(_normalize_str(ma, remove_punct=False) == _normalize_str(gt, remove_punct=False))
+        return all(comparisons)
+    else:
+        # string normalize + exact
+        return _normalize_str(model_answer) == _normalize_str(ground_truth)
+
+
+def _read_jsonl(path: Path) -> Iterable[dict[str, Any]]:
+    """Read JSONL file and yield parsed records."""
+    with path.open("rb") as f:
+        for line in f:
+            if not line.strip():
+                continue
+            try:
+                import orjson
+
+                yield orjson.loads(line)
+            except Exception:
+                yield json.loads(line)
+
+
+def _load_gaia_local(repo_dir: Path, wanted_levels: list[int] | None = None, max_n: int | None = None) -> list[Task]:
+    """Load GAIA tasks from local repository directory."""
+    tasks: list[Task] = []
+
+    for p in repo_dir.rglob("metadata.jsonl"):
+        for rec in _read_jsonl(p):
+            # Robustly extract fields used across variants
+            q = rec.get("Question") or rec.get("question") or rec.get("query") or rec.get("prompt")
+            ans = rec.get("Final answer") or rec.get("answer") or rec.get("final_answer")
+            qid = str(
+                rec.get("task_id")
+                or rec.get("question_id")
+                or rec.get("id")
+                or rec.get("uuid")
+                or f"{p.stem}:{len(tasks)}"
+            )
+            lvl = rec.get("Level") or rec.get("level")
+            fname = rec.get("file_name") or rec.get("filename") or None
+
+            # Only evaluate examples with public answers (dev/validation split)
+            if not q or ans is None:
+                continue
+
+            if wanted_levels and (lvl not in wanted_levels):
+                continue
+
+            tasks.append(Task(task_id=qid, question=q, answer=str(ans), level=lvl, file_name=fname, metadata=rec))
+
+    # Shuffle to help with rate-limits and fairness if max_n is provided
+    random.shuffle(tasks)
+    if max_n:
+        tasks = tasks[:max_n]
+    return tasks
+
+
+class GAIA:
+    """
+    GAIA benchmark runner for Agent Framework.
+
+    GAIA (General AI Assistant) is a benchmark for general-purpose AI assistants.
+    This class provides utilities to run the benchmark with custom agents.
+    """
+
+    def __init__(
+        self,
+        evaluator: Evaluator | None = None,
+        data_dir: str | None = None,
+        hf_token: str | None = None,
+        telemetry_config: GAIATelemetryConfig | None = None,
+    ):
+        """
+        Initialize GAIA benchmark runner.
+
+        Args:
+            evaluator: Custom evaluator function. If None, uses default GAIA scorer.
+            data_dir: Directory to cache GAIA data. Defaults to a temporary directory.
+            hf_token: Hugging Face token for accessing the GAIA dataset.
+            telemetry_config: Configuration for telemetry and tracing. If None, no tracing is performed.
+        """
+        self.evaluator = evaluator or self._default_evaluator
+        self.data_dir = Path(data_dir or Path(tempfile.gettempdir()) / "data_gaia_hub")
+        self.hf_token = hf_token
+        self.telemetry_config = telemetry_config or GAIATelemetryConfig()
+
+        # Set up telemetry
+        self.telemetry_config.setup_telemetry()
+
+        # Initialize tracer
+        if self.telemetry_config.enable_tracing:
+            self.tracer = get_tracer("gaia_benchmark", "1.0.0")
+        else:
+            self.tracer = NoOpTracer()
+
+    async def _default_evaluator(self, task: Task, prediction: Prediction) -> Evaluation:
+        """Default evaluator using GAIA official scoring."""
+        is_correct = gaia_scorer(prediction.prediction, task.answer or "")
+        return Evaluation(is_correct=is_correct, score=1.0 if is_correct else 0.0)
+
+    def _ensure_data(self) -> Path:
+        """Ensure GAIA data is available locally."""
+
+        if self.data_dir.exists() and any(self.data_dir.rglob("metadata.jsonl")):
+            return self.data_dir
+
+        # Download data if not available
+        token = self.hf_token or os.environ.get("HF_TOKEN")
+        if not token:
+            raise RuntimeError(
+                "HF_TOKEN environment variable or hf_token parameter is required "
+                "to access the GAIA dataset. Please set your Hugging Face token "
+                "with access to gaia-benchmark/GAIA."
+            )
+
+        print(f"Downloading GAIA dataset to {self.data_dir}...")
+        from huggingface_hub import snapshot_download
+
+        local_dir = snapshot_download(
+            repo_id="gaia-benchmark/GAIA",
+            repo_type="dataset",
+            token=token,
+            local_dir=str(self.data_dir),
+            local_dir_use_symlinks=False,
+            force_download=False,
+        )
+        return Path(local_dir)
+
+    async def _run_single_task(
+        self, task: Task, task_runner: TaskRunner, semaphore: asyncio.Semaphore, timeout: int | None = None
+    ) -> TaskResult:
+        """Run a single task with error handling and timing."""
+        async with semaphore:
+            with self.tracer.start_as_current_span(
+                "gaia.task.run",
+                kind=SpanKind.INTERNAL,
+                attributes={
+                    "gaia.task.id": task.task_id,
+                    "gaia.task.level": task.level or 0,
+                    "gaia.task.has_file": task.file_name is not None,
+                    "gaia.task.timeout": timeout or 0,
+                },
+            ) as span:
+                start_time = time.time()
+                try:
+                    # Add task execution span
+                    with self.tracer.start_as_current_span(
+                        "gaia.task.execute",
+                        kind=SpanKind.INTERNAL,
+                        attributes={
+                            "gaia.task.question_length": len(task.question or ""),
+                            "gaia.task.file_name": task.file_name or "",
+                        },
+                    ):
+                        if timeout:
+                            prediction = await asyncio.wait_for(task_runner(task), timeout=timeout)
+                        else:
+                            prediction = await task_runner(task)
+
+                    # Add evaluation span
+                    with self.tracer.start_as_current_span("gaia.task.evaluate", kind=SpanKind.INTERNAL):
+                        evaluation = await self.evaluator(task, prediction)
+
+                    runtime_seconds = time.time() - start_time
+
+                    # Add results to span
+                    if span:
+                        span.set_attributes(
+                            {
+                                "gaia.task.runtime_seconds": runtime_seconds,
+                                "gaia.task.is_correct": evaluation.is_correct,
+                                "gaia.task.score": evaluation.score,
+                                "gaia.task.prediction_length": len(prediction.prediction or ""),
+                            }
+                        )
+
+                    return TaskResult(
+                        task_id=task.task_id,
+                        task=task,
+                        prediction=prediction,
+                        evaluation=evaluation,
+                        runtime_seconds=runtime_seconds,
+                    )
+                except Exception as e:
+                    runtime_seconds = time.time() - start_time
+
+                    # Record error in span
+                    if span:
+                        span.set_attributes(
+                            {
+                                "gaia.task.runtime_seconds": runtime_seconds,
+                                "gaia.task.error": str(e),
+                                "gaia.task.is_correct": False,
+                                "gaia.task.score": 0.0,
+                            }
+                        )
+                        span.record_exception(e)
+
+                    return TaskResult(
+                        task_id=task.task_id,
+                        task=task,
+                        prediction=Prediction(prediction="", messages=[]),
+                        evaluation=Evaluation(is_correct=False, score=0.0),
+                        runtime_seconds=runtime_seconds,
+                        error=str(e),
+                    )
+
+    async def run(
+        self,
+        task_runner: TaskRunner,
+        level: int | list[int] = 1,
+        max_n: int | None = None,
+        parallel: int = 1,
+        timeout: int | None = None,
+        out: str | None = None,
+    ) -> list[TaskResult]:
+        """
+        Run the GAIA benchmark.
+
+        Args:
+            task_runner: Function that takes a Task and returns a Prediction
+            level: GAIA level(s) to run (1, 2, 3, or list of levels)
+            max_n: Maximum number of tasks to run per level
+            parallel: Number of parallel tasks to run
+            timeout: Timeout per task in seconds
+            out: Output file to save results including detailed traces (optional)
+
+        Returns:
+            List of TaskResult objects
+        """
+        with self.tracer.start_as_current_span(
+            "gaia.benchmark.run",
+            kind=SpanKind.INTERNAL,
+            attributes={
+                "gaia.benchmark.levels": str(level),
+                "gaia.benchmark.max_n": max_n or 0,
+                "gaia.benchmark.parallel": parallel,
+                "gaia.benchmark.timeout": timeout or 0,
+            },
+        ) as benchmark_span:
+            # Ensure data is available
+            with self.tracer.start_as_current_span("gaia.data.ensure", kind=SpanKind.INTERNAL):
+                data_path = self._ensure_data()
+
+            # Parse level parameter
+            if isinstance(level, int):
+                levels = [level]
+            else:
+                levels = level
+
+            # Load tasks
+            with self.tracer.start_as_current_span(
+                "gaia.tasks.load",
+                kind=SpanKind.INTERNAL,
+                attributes={
+                    "gaia.tasks.levels": str(levels),
+                    "gaia.tasks.max_n": max_n or 0,
+                },
+            ) as load_span:
+                tasks = _load_gaia_local(data_path, wanted_levels=levels, max_n=max_n)
+
+                if load_span:
+                    load_span.set_attributes(
+                        {
+                            "gaia.tasks.loaded_count": len(tasks),
+                        }
+                    )
+
+            if not tasks:
+                raise RuntimeError(
+                    f"No GAIA tasks found for levels {levels}. "
+                    "Make sure you have dataset access and selected valid levels."
+                )
+
+            print(f"Running {len(tasks)} GAIA tasks (levels={levels}) with {parallel} parallel workers...")
+
+            # Update benchmark span with task info
+            if benchmark_span:
+                benchmark_span.set_attributes(
+                    {
+                        "gaia.benchmark.total_tasks": len(tasks),
+                    }
+                )
+
+            # Run tasks
+            semaphore = asyncio.Semaphore(parallel)
+            results = []
+
+            tasks_coroutines = [self._run_single_task(task, task_runner, semaphore, timeout) for task in tasks]
+
+            with self.tracer.start_as_current_span("gaia.tasks.execute_all", kind=SpanKind.INTERNAL):
+                for coro in tqdm(
+                    asyncio.as_completed(tasks_coroutines), total=len(tasks_coroutines), desc="Evaluating tasks"
+                ):
+                    result = await coro
+                    results.append(result)
+
+            # Calculate summary statistics
+            correct = sum(1 for r in results if r.evaluation.is_correct)
+            accuracy = correct / len(results) if results else 0.0
+            avg_runtime = sum(r.runtime_seconds or 0 for r in results) / len(results) if results else 0.0
+
+            # Update benchmark span with final results
+            if benchmark_span:
+                benchmark_span.set_attributes(
+                    {
+                        "gaia.benchmark.accuracy": accuracy,
+                        "gaia.benchmark.correct_count": correct,
+                        "gaia.benchmark.total_count": len(results),
+                        "gaia.benchmark.avg_runtime_seconds": avg_runtime,
+                    }
+                )
+
+            print("\nGAIA Benchmark Results:")
+            print(f"Accuracy: {accuracy:.3f} ({correct}/{len(results)})")
+            print(f"Average runtime: {avg_runtime:.2f}s")
+
+            # Save results if requested
+            if out:
+                with self.tracer.start_as_current_span(
+                    "gaia.results.save", kind=SpanKind.INTERNAL, attributes={"gaia.results.output_file": out}
+                ):
+                    self._save_results(results, out)
+                    print(f"Results saved to {out}")
+
+            return results
+
+    def _save_results(self, results: list[TaskResult], output_path: str) -> None:
+        """Save results with detailed trace information to JSONL file."""
+        with open(output_path, "w", encoding="utf-8") as f:
+            for result in results:
+                # Convert messages to serializable format
+                serializable_messages = []
+                if result.prediction.messages:
+                    for msg in result.prediction.messages:
+                        if hasattr(msg, "model_dump"):
+                            # Pydantic model
+                            serializable_messages.append(msg.model_dump())
+                        elif hasattr(msg, "__dict__"):
+                            # Regular object with attributes
+                            serializable_messages.append(vars(msg))
+                        else:
+                            # Fallback to string representation
+                            serializable_messages.append(str(msg))
+
+                record = {
+                    "task_id": result.task_id,
+                    "level": result.task.level,
+                    "question": result.task.question,
+                    "answer": result.task.answer,
+                    "prediction": result.prediction.prediction,
+                    "is_correct": result.evaluation.is_correct,
+                    "score": result.evaluation.score,
+                    "runtime_seconds": result.runtime_seconds,
+                    "error": result.error,
+                    "timestamp": datetime.now().isoformat(),
+                    # Include detailed trace information
+                    "task_metadata": result.task.metadata,
+                    "file_name": result.task.file_name,
+                    "messages": serializable_messages,
+                    "prediction_metadata": result.prediction.metadata,
+                    "evaluation_details": result.evaluation.details,
+                }
+                try:
+                    import orjson
+
+                    f.write(orjson.dumps(record, default=str).decode("utf-8") + "\n")
+                except ImportError:
+                    f.write(json.dumps(record, default=str) + "\n")
+
+
+def viewer_main() -> None:
+    """Main function for the gaia_viewer script."""
+    import argparse
+
+    parser = argparse.ArgumentParser(description="View GAIA benchmark results")
+    parser.add_argument("results_file", help="Path to results JSONL file")
+    parser.add_argument("--detailed", action="store_true", help="Show detailed view")
+    parser.add_argument("--level", type=int, help="Filter by level")
+    parser.add_argument("--correct-only", action="store_true", help="Show only correct answers")
+    parser.add_argument("--incorrect-only", action="store_true", help="Show only incorrect answers")
+
+    args = parser.parse_args()
+
+    # Load results
+    results = []
+    with open(args.results_file, encoding="utf-8") as f:
+        for line in f:
+            if line.strip():
+                try:
+                    import orjson
+
+                    results.append(orjson.loads(line))
+                except ImportError:
+                    results.append(json.loads(line))
+
+    # Apply filters
+    if args.level is not None:
+        results = [r for r in results if r.get("level") == args.level]
+
+    if args.correct_only:
+        results = [r for r in results if r.get("is_correct")]
+    elif args.incorrect_only:
+        results = [r for r in results if not r.get("is_correct")]
+
+    # Display results
+    if not results:
+        print("No results match the filters.")
+        return
+
+    total = len(results)
+    correct = sum(1 for r in results if r.get("is_correct"))
+    accuracy = correct / total if total > 0 else 0.0
+
+    print("GAIA Results Summary:")
+    print(f"Total: {total}, Correct: {correct}, Accuracy: {accuracy:.3f}")
+    print("-" * 80)
+
+    for i, result in enumerate(results, 1):
+        status = "✓" if result.get("is_correct") else "✗"
+        level = result.get("level", "?")
+        task_id = result.get("task_id", "unknown")
+
+        print(f"[{i}/{total}] {status} Level {level} - {task_id}")
+
+        if args.detailed:
+            print(f"Question: {result.get('question', 'N/A')[:100]}...")
+            print(f"Answer: {result.get('answer', 'N/A')}")
+            print(f"Prediction: {result.get('prediction', 'N/A')}")
+            if result.get("error"):
+                print(f"Error: {result.get('error')}")
+            if result.get("runtime_seconds"):
+                print(f"Runtime: {result.get('runtime_seconds'):.2f}s")
+            print("-" * 40)
+
+
+if __name__ == "__main__":
+    viewer_main()
@@ -0,0 +1 @@
+py.typed
@@ -0,0 +1,84 @@
+[project]
+name = "agent-framework-lab-gaia"
+description = "GAIA benchmark module for Microsoft Agent Framework."
+authors = [{ name = "Microsoft", email = "SK-Support@microsoft.com"}]
+readme = "README.md"
+requires-python = ">=3.10"
+version = "0.1.0b1"
+license-files = ["LICENSE"]
+urls.homepage = "https://learn.microsoft.com/en-us/semantic-kernel/overview/"
+urls.source = "https://github.com/microsoft/agent-framework/tree/main/python"
+urls.release_notes = "https://github.com/microsoft/agent-framework/releases?q=tag%3Apython-1&expanded=true"
+urls.issues = "https://github.com/microsoft/agent-framework/issues"
+classifiers = [
+  "License :: OSI Approved :: MIT License",
+  "Development Status :: 2 - Pre-Alpha",
+  "Intended Audience :: Developers",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Programming Language :: Python :: 3.13",
+]
+dependencies = [
+  "agent-framework",
+  "pydantic>=2.0.0",
+  "opentelemetry-api>=1.24.0",
+  "tqdm>=4.60.0",
+  "huggingface-hub>=0.20.0",
+  "orjson>=3.8.0",
+]
+
+[project.scripts]
+gaia_viewer = "agent_framework_lab_gaia:viewer_main"
+
+[build-system]
+requires = ["setuptools>=64", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools]
+packages = ["agent_framework_lab_gaia", "agent_framework.lab.gaia"]
+
+[tool.setuptools.package-data]
+agent_framework_lab_gaia = ["py.typed"]
+
+[tool.ruff]
+line-length = 120
+target-version = "py310"
+extend-exclude = ["tests", "__pycache__"]
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "W", "UP", "C4", "N"]
+ignore = ["N803", "N806", "N999", "UP007"]
+
+[tool.ruff.format]
+quote-style = "double"
+
+[tool.mypy]
+python_version = "3.10"
+strict = true
+check_untyped_defs = true
+disallow_untyped_defs = true
+disallow_incomplete_defs = true
+disallow_untyped_decorators = true
+warn_redundant_casts = true
+warn_unused_ignores = true
+warn_return_any = true
+warn_unreachable = true
+show_error_codes = true
+implicit_reexport = true
+packages = ["agent_framework_lab_gaia"]
+
+[tool.poe]
+executor.type = "uv"
+include = "../../../shared_tasks.toml"
+[tool.poe.tasks]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+pythonpath = ["."]
+addopts = "--strict-markers --strict-config"
+markers = [
+  "unit: marks tests as unit tests",
+  "integration: marks tests as integration tests",
+]
@@ -0,0 +1,83 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""
+GAIA Benchmark Sample
+
+To run this sample, execute it from the root directory of the agent-framework repository:
+    cd /path/to/agent-framework
+    uv run python python/packages/lab/gaia/gaia_sample.py
+
+This avoids namespace package conflicts that occur when running from within the gaia package directory.
+"""
+
+from agent_framework.foundry import FoundryChatClient
+from azure.identity.aio import AzureCliCredential
+
+from agent_framework.lab.gaia import GAIA, Evaluation, GAIATelemetryConfig, Prediction, Task
+
+
+async def evaluate_task(task: Task, prediction: Prediction) -> Evaluation:
+    """Evaluate the prediction for a given task."""
+    # Simple evaluation: check if the prediction contains the answer
+    is_correct = (task.answer or "").lower() in prediction.prediction.lower()
+    return Evaluation(is_correct=is_correct, score=1 if is_correct else 0)
+
+
+async def main() -> None:
+    # Configure telemetry for tracing
+    telemetry_config = GAIATelemetryConfig(
+        enable_tracing=True,  # Enable OpenTelemetry tracing
+        # Optional: Configure external endpoints
+        # otlp_endpoint="http://localhost:4317",  # For Aspire Dashboard or other OTLP endpoints
+        # application_insights_connection_string="your_connection_string",  # For Azure Monitor
+        # enable_live_metrics=True,  # Enable Azure Monitor live metrics
+        # Configure local file tracing
+        trace_to_file=True,  # Export traces to local file
+        file_path="gaia_benchmark_traces.jsonl",  # Custom file path for traces
+    )
+
+    # Create a single agent once and reuse it for all tasks
+    async with (
+        AzureCliCredential() as credential,
+        FoundryChatClient(async_credential=credential).create_agent(
+            name="GaiaAgent",
+            instructions="Solve tasks to your best ability.",
+        ) as agent,
+    ):
+
+        async def run_task(task: Task) -> Prediction:
+            """Run a single GAIA task and return the prediction using the shared agent."""
+            input_message = f"Task: {task.question}"
+            if task.file_name:
+                input_message += f"\nFile: {task.file_name}"
+            result = await agent.run(input_message)
+            return Prediction(prediction=result.text, messages=result.messages)
+
+        # Create the GAIA benchmark runner with telemetry configuration
+        runner = GAIA(evaluator=evaluate_task, telemetry_config=telemetry_config)
+
+        # Run the benchmark with the task runner.
+        # By default, this will check for locally cached benchmark data and checkout
+        # the latest version from HuggingFace if not found.
+        results = await runner.run(
+            run_task,
+            level=1,  # Level 1, 2, or 3 or multiple levels like [1, 2]
+            max_n=5,  # Maximum number of tasks to run per level
+            parallel=2,  # Number of parallel tasks to run
+            timeout=60,  # Timeout per task in seconds
+            out="gaia_results_level1.jsonl",  # Output file to save results including detailed traces (optional)
+        )
+
+    # Print the results.
+    print("\n=== GAIA Benchmark Results ===")
+    for result in results:
+        print(f"\n--- Task ID: {result.task_id} ---")
+        print(f"Task: {result.task.question[:100]}...")
+        print(f"Prediction: {result.prediction.prediction}")
+        print(f"Evaluation: Correct={result.evaluation.is_correct}, Score={result.evaluation.score}")
+
+
+if __name__ == "__main__":
+    import asyncio
+
+    asyncio.run(main())
@@ -0,0 +1 @@
+# Copyright (c) Microsoft. All rights reserved.
@@ -0,0 +1,37 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""Tests for GAIA benchmark implementation."""
+
+import pytest
+from agent_framework_lab_gaia import gaia_scorer
+
+
+class TestGAIAScorer:
+    """Test the GAIA scoring function."""
+    
+    def test_numeric_exact_match(self):
+        """Test numeric exact matching."""
+        assert gaia_scorer("42", "42") is True
+        assert gaia_scorer("42.0", "42") is True
+        assert gaia_scorer("42", "42.0") is True
+        assert gaia_scorer("42", "43") is False
+        
+    def test_string_normalization(self):
+        """Test string normalization and matching."""
+        assert gaia_scorer("Hello World", "hello world") is True
+        assert gaia_scorer("Hello, World!", "helloworld") is True
+        assert gaia_scorer("test", "TEST") is True
+        assert gaia_scorer("test", "different") is False
+        
+    def test_list_matching(self):
+        """Test list matching with comma/semicolon separation."""
+        assert gaia_scorer("1,2,3", "1,2,3") is True
+        assert gaia_scorer("1; 2; 3", "1,2,3") is True
+        assert gaia_scorer("apple,banana", "apple,banana") is True
+        assert gaia_scorer("1,2,3", "1,2,4") is False
+        assert gaia_scorer("1,2", "1,2,3") is False
+        
+    def test_none_handling(self):
+        """Test handling of None values."""
+        assert gaia_scorer("None", "test") is False
+        assert gaia_scorer("", "test") is False
@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) Microsoft Corporation.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
@@ -0,0 +1,56 @@
+# Agent Framework Lab - Agent Framework x Agent Lighting
+
+RL Module for Microsoft Agent Framework
+
+## Installation
+
+```bash
+pip install agent-framework-lab-lighting
+```
+
+## Usage
+
+```python
+from agent_framework.lab.lighting import YourClass
+
+# Your usage example here
+instance = YourClass()
+```
+
+## Overview
+
+Brief description of what this lab package provides and its main features.
+
+## Features
+
+- Feature 1: Description
+- Feature 2: Description
+- Feature 3: Description
+
+## Examples
+
+### Basic Usage
+
+```python
+from agent_framework.lab.lighting import YourClass
+
+# Example usage
+```
+
+### Advanced Usage
+
+```python
+# More advanced examples
+```
+
+## API Reference
+
+Document your main classes and functions here.
+
+## Contributing
+
+This package is part of the Microsoft Agent Framework Lab. Please see the main repository for contribution guidelines.
+
+## License
+
+This project is licensed under the MIT License - see the LICENSE file for details.
@@ -0,0 +1,4 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+# This makes agent_framework a namespace package
+__path__ = __import__("pkgutil").extend_path(__path__, __name__)
@@ -0,0 +1,4 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+# This makes agent_framework.lab a namespace package
+__path__ = __import__("pkgutil").extend_path(__path__, __name__)
@@ -0,0 +1,4 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+# Import and re-export from the actual implementation
+from agent_framework_lab_lighting import *  # noqa: F403, F401
@@ -0,0 +1,16 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""
+RL Module for Microsoft Agent Framework
+"""
+
+# Import your main exports here
+# from .main_module import MainClass, main_function
+
+__all__ = [
+    # List your exports here
+    # "MainClass",
+    # "main_function",
+]
+
+__version__ = "0.1.0b1"
@@ -0,0 +1,87 @@
+[project]
+name = "agent-framework-lab-lighting"
+description = "RL Module for Microsoft Agent Framework"
+authors = [{ name = "Microsoft", email = "SK-Support@microsoft.com"}]
+readme = "README.md"
+requires-python = ">=3.10"
+version = "0.1.0b1"
+license-files = ["LICENSE"]
+urls.homepage = "https://learn.microsoft.com/en-us/semantic-kernel/overview/"
+urls.source = "https://github.com/microsoft/agent-framework/tree/main/python"
+urls.release_notes = "https://github.com/microsoft/agent-framework/releases?q=tag%3Apython-1&expanded=true"
+urls.issues = "https://github.com/microsoft/agent-framework/issues"
+classifiers = [
+  "License :: OSI Approved :: MIT License",
+  "Development Status :: 2 - Pre-Alpha",
+  "Intended Audience :: Developers",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Programming Language :: Python :: 3.13",
+]
+dependencies = [
+  "agent-framework",
+  "pydantic>=2.0.0",
+  # Add your specific dependencies here
+]
+
+
+[project.scripts]
+lighting = "agent_framework_lab_lighting:main"
+
+
+[build-system]
+requires = ["setuptools>=64", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools]
+packages = ["agent_framework_lab_lighting", "agent_framework.lab.lighting"]
+
+[tool.setuptools.package-data]
+agent_framework_lab_lighting = ["py.typed"]
+
+[tool.ruff]
+line-length = 120
+target-version = "py310"
+extend-exclude = ["tests", "__pycache__"]
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "W", "UP", "C4", "N"]
+ignore = ["N803", "N806", "N999", "UP007"]
+
+[tool.ruff.format]
+quote-style = "double"
+
+[tool.mypy]
+python_version = "3.10"
+strict = true
+check_untyped_defs = true
+disallow_untyped_defs = true
+disallow_incomplete_defs = true
+disallow_untyped_decorators = true
+warn_redundant_casts = true
+warn_unused_ignores = true
+warn_return_any = true
+warn_unreachable = true
+show_error_codes = true
+implicit_reexport = true
+packages = ["agent_framework_lab_lighting"]
+
+
+[tool.poe]
+executor.type = "uv"
+include = "../../../shared_tasks.toml"
+[tool.poe.tasks]
+test = "pytest --cov=agent_framework_lab_lighting --cov-report=term-missing:skip-covered tests"
+mypy = "mypy agent_framework_lab_lighting"
+
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+pythonpath = ["."]
+addopts = "--strict-markers --strict-config"
+markers = [
+  "unit: marks tests as unit tests",
+  "integration: marks tests as integration tests",
+]
@@ -0,0 +1 @@
+# Copyright (c) Microsoft. All rights reserved.
@@ -0,0 +1,15 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""Tests for lighting module."""
+
+import pytest
+from agent_framework_lab_lighting import __version__
+
+
+class TestLighting:
+    """Test the lighting module."""
+    
+    def test_version(self):
+        """Test package version is defined."""
+        assert __version__ is not None
+        assert __version__ == "0.1.0b1"
@@ -0,0 +1,4 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+# This makes agent_framework.lab a namespace package
+__path__ = __import__("pkgutil").extend_path(__path__, __name__)
				`@@ -0,0 +1 @@`
				`# Copyright (c) Microsoft. All rights reserved.`