Python: Foundry Evals integration for Python (#4750)

* Foundry Evals integration for Python Merged and refactored eval module per Eduard's PR review: - Merge _eval.py + _local_eval.py into single _evaluation.py - Convert EvalItem from dataclass to regular class - Rename to_dict() to to_eval_data() - Convert _AgentEvalData to TypedDict - Simplify check system: unified async pattern with isawaitable - Parallelize checks and evaluators with asyncio.gather - Add all/any mode to tool_called_check - Fix bool(passed) truthy bug in _coerce_result - Remove deprecated function_evaluator/async_function_evaluator aliases - Remove _MinimalAgent, tighten evaluate_agent signature - Set self.name in __init__ (LocalEvaluator, FoundryEvals) - Limit FoundryEvals to AsyncOpenAI only - Type project_client as AIProjectClient - Remove NotImplementedError continuous eval code - Add evaluation samples in 02-agents/ and 03-workflows/ - Update all imports and tests (167 passing) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: resolve mypy redundant-cast errors while keeping pyright happy Use cast(list[Any], x) with type: ignore[redundant-cast] comments to satisfy both mypy (which considers casting Any redundant) and pyright strict mode (which needs explicit casts to narrow Unknown types). Also fix evaluator decorator check_name type annotation to be explicitly str, resolving mypy str|Any|None mismatch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: CI failures — pyupgrade, evaluator overloads, sample API, reset attr - Apply pyupgrade: Sequence from collections.abc, remove forward-ref quotes - Add @overload signatures to evaluator() for proper @evaluator usage - Fix evaluate_workflow sample to use WorkflowBuilder(start_executor=) API - Fix _workflow.py executor.reset() to use getattr pattern for pyright - Remove unused EvalResults forward-ref string in default_factory lambda Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: skip gRPC-dependent observability test The test_configure_otel_providers_with_env_file_and_vs_code_port test triggers gRPC OTLP exporter creation, but the grpc dependency is optional and not installed by default. Add skipif decorator matching the pattern used by all other gRPC exporter tests in the same file. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: add nosec B101 for bandit assert check Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * style: align eval samples with repo conventions - Move module docstrings before imports (after copyright header) - Add -> None return type to all main() and helper functions - Fix line-too-long in multiturn sample conversation data - Add Workflow import for typed return in all_patterns_sample Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review feedback: async fixes, sample bugs, deprecation warnings - Simplify _ensure_async_result to direct await (async-only clients) - Replace get_event_loop() with get_running_loop() - Narrow _fetch_output_items exception handling to specific types - Add warning log when _filter_tool_evaluators falls back to defaults - Add DeprecationWarning to options alias in Agent.__init__ - Add DeprecationWarning to evaluate_response() - Rename raw key to _raw_arguments in convert_message fallback - Fix evaluate_agent_sample.py: replace evals.select() with FoundryEvals() - Fix evaluate_multiturn_sample.py: use Message/Content/FunctionTool types - Fix evaluate_workflow_sample.py: replace evals.select() with FoundryEvals() - Update test mocks to use AsyncMock for awaited API calls Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add test coverage for review feedback items - Add num_repetitions=2 positive test verifying 2×items and 4 agent calls - Add _poll_eval_run tests: timeout, failed, and canceled paths - Add evaluate_traces tests: validation error, response_ids path, trace_ids path - Add evaluate_foundry_target happy-path test with target/query verification Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix ruff ISC004 lint error and apply formatter - Wrap implicit string concatenation in parens in evaluate_multiturn_sample.py - Apply ruff formatter to 6 other files with minor formatting drift Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove core type changes (extracted to fix/workflow-stale-session branch) Reverts changes to _agents.py, _agent_executor.py, and _workflow.py back to upstream/main. These fixes are now in a separate PR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review round 2: bugs, tests, and architecture Code fixes: - Fix _normalize_queries inverted condition (single query now replicates to match expected_count) - Fix substring match bug: 'end' in 'backend' matched; use exact set lookup for executor ID filtering - Fix used_available_tools sample: tool_definitions→tools param, use FunctionTool attribute access instead of dict .get() - Add None-check in _resolve_openai_client for misconfigured project - Add Returns section to evaluate_workflow docstring - Cache inspect.signature in @evaluator wrapper (avoid per-item reflection) Architecture: - Extract _evaluate_via_responses as module-level helper; evaluate_traces now calls it directly instead of creating a FoundryEvals instance - Move Foundry-specific typed-content conversion out of core to_eval_data; core now returns plain role/content dicts, FoundryEvals applies AgentEvalConverter in _evaluate_via_dataset Tests: - evaluate_response() deprecation warning emission and delegation - num_repetitions > 1 with expected_output and expected_tool_calls - Mock output_items.list in test_evaluate_calls_evals_api - Update to_eval_data assertions for plain-dict format - Unknown param error now raised at @evaluator decoration time Skipped (separate PR): executor reset loop, xfail removal, options alias Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix CI: revert test_full_conversation, fix pyright errors - Revert test_full_conversation.py to upstream/main (the session preservation test was incorrectly changed to assert clearing) - Fix pyright reportUnnecessaryComparison on get_openai_client() None check by adding ignore comment - Fix pyright reportPrivateUsage: add public EvalItem.split_messages() method and use it in FoundryEvals._evaluate_via_dataset instead of accessing private _split_conversation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review round 3: reliability, test gaps, cleanup - Add try/except guard for non-numeric score in _coerce_result - Add poll_interval minimum bound (0.1s) to prevent tight loops - Add runtime async client check in _resolve_openai_client - Remove _ensure_async_result wrapper (10 call sites → direct await) - Better error message when queries provided without agent - Import-time asserts for evaluator set consistency - Remove 28 redundant @pytest.mark.asyncio decorators - Add doc note about _raw_arguments sensitive data - Tests: tool_called_check mode=any, _normalize_queries branches, _extract_result_counts paths, _extract_per_evaluator, bare check via evaluate_agent, output_items assertion, modulo wrapping, async client check, queries-without-agent error Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix CI: ruff S101 assert, pyright and mypy arg-type errors - Replace module-level assert with if/raise for evaluator set consistency checks (ruff S101 disallows bare assert) - Add type: ignore[arg-type] and pyright: ignore[reportArgumentType] on OpenAI SDK evals API calls that pass dicts where typed params are expected (SDK accepts dicts at runtime) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review round 4: bugs, reliability, test fixes - Fix all_passed ignoring parent result_counts when sub_results present - Fix _extract_tool_calls: parse string arguments via json.loads before falling back to None (real LLM responses use string arguments) - Sanitize _raw_arguments to '[unparseable]' to avoid leaking sensitive tool-call data to external evaluation services - Add NOTE comment on to_eval_data message serialization dropping non-text content (tool calls, results) - Eliminate double conversation split in _evaluate_via_dataset: build JSONL dicts directly from split_messages + AgentEvalConverter - Raise poll_interval floor from 0.1s to 1.0s to prevent rate-limit exhaustion - Fix MagicMock(name=...) bug in test: sets display name not .name attr - Fix mock_output_item.sample: use MagicMock object instead of dict so _fetch_output_items exercises error/usage/input/output extraction Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review round 5: reliability, docs, test coverage Code fixes: - Move import-time RuntimeError checks to unit tests (avoids breaking imports for all users on developer set-drift mistake) - _filter_tool_evaluators now raises ValueError when all evaluators require tools but no items have tools (was silently substituting) - Add poll_interval upper bound (60s) to prevent single-iteration sleep - Log exc_info=True in _fetch_output_items for debugging API changes - Fix evaluate() docstring: remove claim about Responses API optimization - Validate target dict has 'type' key in evaluate_foundry_target - Document to_eval_data() limitation: non-text content is omitted Tests: - TestEvaluatorSetConsistency: verify _AGENT/_TOOL subsets of _BUILTIN - TestEvaluateTracesAgentId: agent_id-only path with lookback_hours - TestFilterToolEvaluatorsRaises: ValueError on all-tool no-items - TestEvaluateFoundryTargetValidation: target without 'type' key - Assert items==[] on failed/canceled poll results - Mock output_items.list in response_ids test for full flow - TestAllPassedSubResults: result_counts=None + sub_results delegation and parent failures override sub_results - TestBuildOverallItemEmpty: empty workflow outputs returns None Skipped r5-07 (_raw_arguments length hint): marginal debugging value, could leak content size information. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix error message: evaluate_responses() → evaluate_traces(response_ids=...) The referenced function doesn't exist; the correct API is evaluate_traces(response_ids=...) from the azure-ai package. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove dead to_eval_data() method, fix docstring claims - Remove to_eval_data() from EvalItem (dead code after r4-05 JSONL refactor) - Migrate 15 tests from to_eval_data() to split_messages() - Update sample to use split_messages() + Message properties - Remove unimplemented Responses API optimization docstring claim - Update split_messages() docstring to not reference removed method Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Reduce default eval timeout from 600s to 180s (3 minutes) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove dead _evaluate_via_responses method from FoundryEvals The method was never called — evaluate() uses _evaluate_via_dataset, and evaluate_traces() calls _evaluate_via_responses_impl directly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert unrelated formatting changes to get-started samples Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix pyright: remove phantom FoundryMemoryProvider import, apply ruff format - Remove import of non-existent _foundry_memory_provider module (incorrectly kept during rebase conflict resolution) - Apply ruff formatter to test_local_eval.py and get-started samples Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix eval samples: use FoundryChatClient for Agent() The upstream provider-leading client refactor (#4818) made client= a required parameter on Agent(). Update the three getting-started eval samples to use FoundryChatClient with FOUNDRY_PROJECT_ENDPOINT, matching the standard pattern from 01-get-started samples. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Simplify self-reflection sample using FoundryEvals Replace ~80 lines of manual OpenAI evals API code (create_eval, run_eval, manual polling, raw JSONL params) with FoundryEvals: - evaluate_groundedness() uses FoundryEvals.evaluate() with EvalItem - Remove create_openai_client(), create_eval(), run_eval() functions - Remove openai SDK type imports (DataSourceConfigCustom, etc.) - run_self_reflection_batch creates FoundryEvals instance once, reuses it for all iterations across all prompts Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update eval samples to FoundryChatClient and FOUNDRY_PROJECT_ENDPOINT - Migrate all foundry_evals samples from AzureOpenAIResponsesClient to FoundryChatClient - Update env var from AZURE_AI_PROJECT_ENDPOINT to FOUNDRY_PROJECT_ENDPOINT - Use AzureCliCredential consistently across all samples - Fix README.md: correct function names (evaluate_dataset -> FoundryEvals.evaluate, evaluate_responses -> evaluate_traces) - Update self_reflection .env.example and README.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix lint errors in eval samples (E501, ASYNC240, formatting) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove evaluate_all_patterns_sample.py (redundant with focused samples) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix async credential mismatch: use azure.identity.aio for async AIProjectClient AIProjectClient from azure.ai.projects.aio requires an async credential. Switch all foundry_evals samples from azure.identity.AzureCliCredential to azure.identity.aio.AzureCliCredential. Also pass project_client to FoundryChatClient instead of duplicating endpoint+credential. Close credential in self_reflection sample to avoid resource leak. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert test_observability.py to upstream/main (not our test) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address moonbox3 review: sphinx docstrings, pagination, isinstance check - Convert all Example:: / Typical usage:: code blocks to .. code-block:: python format matching codebase convention (both _evaluation.py and _foundry_evals.py) - Add async pagination in _fetch_output_items via async for (handles large result sets) - Replace hasattr(__aenter__) with isinstance(client, AsyncOpenAI) in _resolve_openai_client - Move AsyncOpenAI import from TYPE_CHECKING to runtime (needed for isinstance) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix test failures and address remaining moonbox3 review comments - Fix tests: use MagicMock(spec=AsyncOpenAI) for project_client mocks (isinstance check now requires proper type, not duck-typing) - Fix tests: replace mock_page.__iter__ with _AsyncPage helper for async for - Fix evaluate_response: auto-extract queries from response messages when query is not provided (previously always raised ValueError) - Add debug logging when skipping internal _-prefixed executor IDs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address Tao's PR review comments on Foundry Evals - T1: Add comment explaining builtin.* pass-through in _resolve_evaluator - T2: Add comment referencing OpenAI evals API for testing_criteria dict - T3: Document Mustache-style {{item.*}} template placeholders - T4: Document poll loop 60s sleep upper bound rationale - T5: Narrow run type to RunRetrieveResponse, use typed field access instead of vars()/getattr dance in _extract_result_counts and _extract_per_evaluator; use run.error and run.report_url directly - T6: Clarify openai_client docstring re: Azure Foundry endpoint - T8: Remove misleading empty expected_tool_calls from sample - Update tests to match real SDK PerTestingCriteriaResult shape Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove unnecessary Any union from run type annotations RunRetrieveResponse is the correct type — no backward compat needed for a brand new feature. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Accept FoundryChatClient instead of raw AsyncOpenAI FoundryEvals now takes client: FoundryChatClient as its primary parameter instead of openai_client: AsyncOpenAI. The builtin.* evaluators require a Foundry endpoint, so the type should reflect that. - FoundryEvals.__init__: client: FoundryChatClient replaces openai_client - evaluate_traces / evaluate_foundry_target: same change - _resolve_openai_client: extracts .client from FoundryChatClient - project_client fallback retained for standalone functions - All samples updated to construct FoundryChatClient and pass as client= - Tests updated (openai_client= → client=) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove implicit 60s upper bound on poll interval If a developer sets a higher poll_interval, respect it. Only clamp to remaining time and enforce a 1s minimum for rate-limit protection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove 1s floor on poll interval — let the developer control it Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update python/samples/05-end-to-end/evaluation/foundry_evals/.env.example Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com> * Update python/samples/02-agents/evaluation/evaluate_agent.py Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com> * Address eavanvalkenburg review (round 2) on Python eval PR - Rename model_deployment -> model across FoundryEvals and all samples - Make model param optional, resolves from client.model - Convert EvalResults from dataclass to regular class - Remove deprecated evaluate_response() function - Refactor splitters: BUILT_IN_SPLITTERS dict + standalone functions - Change per_turn_items from classmethod to staticmethod - Simplify EvalCheck type alias to use Awaitable[CheckResult] - Remove errored property from EvalResults - Remove default value from Evaluator protocol eval_name - Rename assert_passed -> raise_for_status, add EvalNotPassedError - Type agent param as SupportsAgentRun | None - Fix Arguments docstring - Update __init__.py exports - Update all tests and samples Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Move FoundryEvals to foundry package, split tool eval sample - Move _foundry_evals.py from azure-ai to foundry package - Move test_foundry_evals.py to foundry/tests/ - Update lazy re-exports in agent_framework.foundry namespace - Update .pyi type stubs - All samples now import from agent_framework.foundry - Split tool-call evaluation into evaluate_tool_calls_sample.py - Fix all_passed to check errored count from result_counts - Fix raise_for_status to include errored item details Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Auto-create FoundryChatClient from env vars when no client provided FoundryEvals() now works zero-config when FOUNDRY_PROJECT_ENDPOINT and FOUNDRY_MODEL environment variables are set. Auto-creates a FoundryChatClient under the hood, matching the established env var pattern. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix pyright errors: remove dead _normalize_queries, suppress EvalAPIError check - Remove unused _normalize_queries function and its tests - Add pyright ignore for EvalAPIError None check (defensive guard) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Support multimodal image content in eval pipeline Add image (data/uri) content handling to AgentEvalConverter.convert_message() so that Content.from_data() and Content.from_uri() image payloads are preserved as input_image parts in the Foundry evaluator format. - Handle Content type='data' and type='uri' → emit input_image parts - Add 6 unit tests for image content through convert_message/convert_messages - Add integration test verifying images flow through EvalItem → JSONL path - Add evaluate_multimodal.py sample demonstrating local image eval Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address remaining review comments - Fix project_client docstring to say async-only (not sync/async) - Add builtin evaluator name validation warning in _resolve_evaluator - Replace getattr with typed attribute access in _poll_eval_run, _extract_result_counts, _extract_per_evaluator, _fetch_output_items - Remove cast import from _foundry_evals (no longer needed) - Tighten _coerce_result: honour explicit 'passed' when both 'score' and 'passed' are present; remove performative cast - Fix self_reflection sample: add env file existence check - Fix traces sample: correct Pattern 2 section label - Update all Foundry eval samples to FoundryChatClient + FOUNDRY_MODEL (remove AIProjectClient + AZURE_AI_MODEL_DEPLOYMENT_NAME pattern) - Add eval_name and OpenAI client docs to FoundryEvals docstring - Update test mocks to match typed SDK objects (_MockResultCounts) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix ruff lint errors (E501, SIM108, SIM102) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix pyright errors: type-narrow dict to dict[str, Any], add ignore comments Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Replace ConversationSplitter type alias with Protocol ConversationSplitter is now a runtime-checkable Protocol with a named 'conversation' parameter, making the expected signature self-documenting. ConversationSplit enum members gain a __call__ method so they satisfy the protocol directly -- ConversationSplit.LAST_TURN(conversation) works. This simplifies _split_conversation from an isinstance dispatch to a single split(conversation) call. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Standardize on AZURE_AI_MODEL_DEPLOYMENT_NAME and fix Unicode in samples - Replace FOUNDRY_MODEL with AZURE_AI_MODEL_DEPLOYMENT_NAME in all eval samples to match repo convention - Replace Unicode symbols with ASCII equivalents in all eval sample print statements to avoid cp1252 encoding errors on Windows Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update python/samples/03-workflows/evaluation/evaluate_workflow.py Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com> * Rename ADR 0020 to 0023 (foundry evals integration) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: alliscode <bentho@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com>
2026-06-16 21:04:09 +08:00 · 2026-03-31 08:53:06 -07:00
parent 3f964c4cdb
commit 35adfdb318
24 changed files with 7830 additions and 166 deletions
@@ -0,0 +1,81 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""Evaluate an agent with local checks — no API keys needed.
+
+Demonstrates the simplest evaluation workflow:
+1. Define checks using the @evaluator decorator
+2. Run evaluate_agent() which calls agent.run() under the covers
+3. Assert results in CI or inspect interactively
+
+Usage:
+    uv run python samples/02-agents/evaluation/evaluate_agent.py
+"""
+
+import asyncio
+import os
+
+from agent_framework import (
+    Agent,
+    LocalEvaluator,
+    evaluate_agent,
+    evaluator,
+    keyword_check,
+)
+from agent_framework.foundry import FoundryChatClient
+from azure.identity import AzureCliCredential
+from dotenv import load_dotenv
+
+load_dotenv()
+
+
+# A custom check — parameter names determine what data you receive
+@evaluator
+def is_helpful(response: str) -> bool:
+    """Check the response isn't empty or a refusal."""
+    refusals = ["i can't", "i'm not able", "i don't know"]
+    return len(response) > 10 and not any(r in response.lower() for r in refusals)
+
+
+async def main() -> None:
+    client = FoundryChatClient(
+        project_endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
+        model=os.environ.get("FOUNDRY_MODEL", "gpt-4o"),
+        credential=AzureCliCredential(),
+    )
+
+    agent = Agent(
+        client=client,
+        name="weather-assistant",
+        instructions="You are a helpful weather assistant.",
+    )
+
+    # Combine built-in and custom checks
+    local = LocalEvaluator(
+        keyword_check("weather"),  # response must mention "weather"
+        is_helpful,  # custom check
+    )
+
+    # evaluate_agent() calls agent.run() for each query, then evaluates
+    results = await evaluate_agent(
+        agent=agent,
+        queries=[
+            "What's the weather like in Seattle?",
+            "Will it rain in London tomorrow?",
+            "What should I wear for 30°C weather?",
+        ],
+        evaluators=local,
+    )
+
+    for r in results:
+        print(f"{r.provider}: {r.passed}/{r.total} passed")
+        for item in r.items:
+            print(f"  [{item.status}] Q: {item.input_text[:50]}  A: {item.output_text[:50]}...")
+            for score in item.scores:
+                print(f"    {'PASS' if score.passed else 'FAIL'} {score.name}")
+
+    # Use in CI: will raise EvalNotPassedError if any check fails
+    # results[0].raise_for_status()
+
+
+if __name__ == "__main__":
+    asyncio.run(main())
@@ -0,0 +1,122 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""Evaluate multimodal (image) conversations locally.
+
+Demonstrates that the evaluation pipeline preserves image content:
+1. Build EvalItems with image content in conversations
+2. Use @evaluator checks that inspect multimodal content
+3. Verify images flow through the eval pipeline intact
+
+Usage:
+    uv run python samples/02-agents/evaluation/evaluate_multimodal.py
+"""
+
+import asyncio
+import base64
+
+from agent_framework import (
+    Content,
+    EvalItem,
+    LocalEvaluator,
+    Message,
+    evaluator,
+)
+
+
+# -- Custom evaluators that inspect multimodal content --
+
+
+@evaluator
+def has_image_content(conversation: list) -> bool:
+    """Check that the conversation contains at least one image."""
+    return any(
+        c.type in ("data", "uri") and c.media_type and c.media_type.startswith("image/")
+        for m in conversation
+        for c in (m.contents or [])
+    )
+
+
+@evaluator
+def response_describes_image(response: str) -> bool:
+    """Check that the assistant response acknowledges the image."""
+    image_words = {"image", "picture", "photo", "shows", "depicts", "see"}
+    return any(word in response.lower() for word in image_words)
+
+
+@evaluator
+def image_count(conversation: list) -> float:
+    """Return the number of images in the conversation as a score."""
+    count = sum(
+        1
+        for m in conversation
+        for c in (m.contents or [])
+        if c.type in ("data", "uri") and c.media_type and c.media_type.startswith("image/")
+    )
+    return float(count)
+
+
+# A tiny 1x1 red PNG for demonstration (no external dependencies needed)
+_TINY_PNG = base64.b64decode(
+    "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8/5+hHgAHggJ/PchI7wAAAABJRU5ErkJggg=="
+)
+
+
+async def main() -> None:
+    # Build eval items with multimodal content (no agent run needed)
+    items = [
+        # Item 1: User sends an image URL with a question
+        EvalItem(
+            conversation=[
+                Message(
+                    "user",
+                    [
+                        Content.from_text("What do you see in this image?"),
+                        Content.from_uri(
+                            "https://upload.wikimedia.org/wikipedia/commons/thumb/4/47/PNG_transparency_demonstration_1.png/300px-PNG_transparency_demonstration_1.png",
+                            media_type="image/png",
+                        ),
+                    ],
+                ),
+                Message("assistant", ["The image shows two dice on a transparent background."]),
+            ]
+        ),
+        # Item 2: User sends inline image bytes
+        EvalItem(
+            conversation=[
+                Message(
+                    "user",
+                    [
+                        Content.from_text("Describe this picture"),
+                        Content.from_data(data=_TINY_PNG, media_type="image/png"),
+                    ],
+                ),
+                Message("assistant", ["I see a small red image — it appears to be a single pixel."]),
+            ]
+        ),
+        # Item 3: Text-only conversation (should fail has_image_content)
+        EvalItem(
+            conversation=[
+                Message("user", ["Tell me about cats"]),
+                Message("assistant", ["Cats are wonderful pets."]),
+            ]
+        ),
+    ]
+
+    local = LocalEvaluator(
+        has_image_content,
+        response_describes_image,
+        image_count,
+    )
+
+    results = await local.evaluate(items)
+
+    print(f"\n{results.provider}: {results.passed}/{results.total} passed")
+    for item in results.items:
+        print(f"\n  [{item.status}] Q: {item.input_text[:60]}...")
+        for score in item.scores:
+            symbol = "PASS" if score.passed else "FAIL"
+            print(f"    {symbol} {score.name}: {score.score}")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())
@@ -0,0 +1,73 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""Evaluate an agent with expected outputs and tool call checks.
+
+Demonstrates ground-truth comparison and tool usage evaluation:
+1. Provide expected outputs alongside queries
+2. Use built-in tool_calls_present for tool verification
+3. Combine multiple evaluation criteria
+
+Usage:
+    uv run python samples/02-agents/evaluation/evaluate_with_expected.py
+"""
+
+import asyncio
+import os
+
+from agent_framework import (
+    Agent,
+    LocalEvaluator,
+    evaluate_agent,
+    evaluator,
+    tool_calls_present,
+)
+from agent_framework.foundry import FoundryChatClient
+from azure.identity import AzureCliCredential
+from dotenv import load_dotenv
+
+load_dotenv()
+
+
+@evaluator
+def response_matches_expected(response: str, expected_output: str) -> float:
+    """Score based on word overlap with expected output."""
+    if not expected_output:
+        return 1.0
+    response_words = set(response.lower().split())
+    expected_words = set(expected_output.lower().split())
+    return len(response_words & expected_words) / max(len(expected_words), 1)
+
+
+async def main() -> None:
+    client = FoundryChatClient(
+        project_endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
+        model=os.environ.get("AZURE_AI_MODEL_DEPLOYMENT_NAME", "gpt-4o"),
+        credential=AzureCliCredential(),
+    )
+
+    agent = Agent(
+        client=client,
+        name="math-tutor",
+        instructions="You are a math tutor. Answer concisely.",
+    )
+
+    local = LocalEvaluator(
+        response_matches_expected,
+        tool_calls_present,  # verifies expected tools were called
+    )
+
+    results = await evaluate_agent(
+        agent=agent,
+        queries=["What is 2 + 2?", "What is the square root of 144?"],
+        expected_output=["4", "12"],
+        evaluators=local,
+    )
+
+    for r in results:
+        print(f"{r.provider}: {r.passed}/{r.total} passed")
+        for item in r.items:
+            print(f"  [{item.status}] {item.input_text} -> {item.output_text[:80]}")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())
@@ -0,0 +1,69 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""Evaluate a multi-agent workflow with per-agent breakdown.
+
+Demonstrates workflow evaluation:
+1. Build a simple two-agent workflow
+2. Run evaluate_workflow() which runs the workflow and evaluates each agent
+3. Inspect per-agent results in sub_results
+
+Usage:
+    uv run python samples/03-workflows/evaluation/evaluate_workflow.py
+"""
+
+import asyncio
+import os
+
+from agent_framework import (
+    Agent,
+    LocalEvaluator,
+    WorkflowBuilder,
+    evaluate_workflow,
+    evaluator,
+    keyword_check,
+)
+from agent_framework.foundry import FoundryChatClient
+from azure.identity import AzureCliCredential
+from dotenv import load_dotenv
+
+load_dotenv()
+
+
+@evaluator
+def is_nonempty(response: str) -> bool:
+    """Check the agent produced a non-trivial response."""
+    return len(response.strip()) > 5
+
+
+async def main() -> None:
+    # Build a simple planner -> executor workflow
+    client = FoundryChatClient(
+        project_endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
+        model=os.environ.get("FOUNDRY_MODEL", "gpt-4o"),
+        credential=AzureCliCredential(),
+    )
+    planner = Agent(client=client, name="planner", instructions="You plan trips. Output a bullet-point plan.")
+    executor_agent = Agent(
+        client=client, name="executor", instructions="You execute travel plans. Book the items listed."
+    )
+
+    workflow = WorkflowBuilder(start_executor=planner).add_edge(planner, executor_agent).build()
+
+    # Evaluate with per-agent breakdown
+    local = LocalEvaluator(is_nonempty, keyword_check("plan", "trip"))
+
+    results = await evaluate_workflow(
+        workflow=workflow,
+        queries=["Plan a weekend trip to Paris"],
+        evaluators=local,
+    )
+
+    for r in results:
+        print(f"{r.provider}: {r.passed}/{r.total} passed (overall)")
+        for agent_name, sub in r.sub_results.items():
+            error = f" (error: {sub.error})" if sub.error else ""
+            print(f"  {agent_name}: {sub.passed}/{sub.total} {error}")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())
@@ -0,0 +1,3 @@
+FOUNDRY_PROJECT_ENDPOINT="<your-project-endpoint>"
+FOUNDRY_MODEL="<your-model-deployment>"
+
@@ -0,0 +1,46 @@
+# Foundry Evals Integration Samples
+
+These samples demonstrate evaluating agent-framework agents using Azure AI Foundry's built-in evaluators.
+
+## Available Evaluators
+
+| Category | Evaluators |
+|----------|-----------|
+| **Agent behavior** | `intent_resolution`, `task_adherence`, `task_completion`, `task_navigation_efficiency` |
+| **Tool usage** | `tool_call_accuracy`, `tool_selection`, `tool_input_accuracy`, `tool_output_utilization`, `tool_call_success` |
+| **Quality** | `coherence`, `fluency`, `relevance`, `groundedness`, `response_completeness`, `similarity` |
+| **Safety** | `violence`, `sexual`, `self_harm`, `hate_unfairness` |
+
+## Samples
+
+### `evaluate_agent_sample.py` — Dataset Evaluation (Path 3)
+
+The dev inner loop. Two patterns from simplest to most control:
+
+1. **`evaluate_agent()`** — One call: runs agent → converts → evaluates
+2. **`FoundryEvals.evaluate()`** — Run agent yourself, convert with `AgentEvalConverter`, inspect/modify, then evaluate
+
+```bash
+uv run samples/05-end-to-end/evaluation/foundry_evals/evaluate_agent_sample.py
+```
+
+### `evaluate_traces_sample.py` — Trace & Response Evaluation (Path 1)
+
+Evaluate what already happened — zero changes to agent code:
+
+1. **`evaluate_traces(response_ids=...)`** — Evaluate Responses API responses by ID
+2. **`evaluate_traces(agent_id=...)`** — Evaluate agent behavior from OTel traces in App Insights
+
+```bash
+uv run samples/05-end-to-end/evaluation/foundry_evals/evaluate_traces_sample.py
+```
+
+## Setup
+
+Create a `.env` file with configuration as in the `.env.example` file in this folder.
+
+## Which sample should I start with?
+
+- **"I want to test my agent during development"** → `evaluate_agent_sample.py`, Pattern 1
+- **"I want to evaluate past agent runs"** → `evaluate_traces_sample.py`
+- **"I want to inspect/modify eval data before submitting"** → `evaluate_agent_sample.py`, Pattern 2
@@ -0,0 +1,154 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""Evaluate an agent using Azure AI Foundry's built-in evaluators.
+
+This sample demonstrates two patterns:
+1. evaluate_agent(responses=...) — Evaluate a response you already have.
+2. evaluate_agent(queries=...) — Run the agent against test queries and evaluate in one call.
+
+See ``evaluate_tool_calls_sample.py`` for tool-call accuracy evaluation.
+
+Prerequisites:
+- An Azure AI Foundry project with a deployed model
+- Set FOUNDRY_PROJECT_ENDPOINT and AZURE_AI_MODEL_DEPLOYMENT_NAME in .env
+"""
+
+import asyncio
+import os
+
+from agent_framework import Agent, ConversationSplit, evaluate_agent
+from agent_framework.foundry import FoundryChatClient, FoundryEvals
+from azure.identity import AzureCliCredential
+from dotenv import load_dotenv
+
+load_dotenv()
+
+
+# Define a simple tool for the agent
+def get_weather(location: str) -> str:
+    """Get the current weather for a location."""
+    weather_data = {
+        "seattle": "62°F, cloudy with a chance of rain",
+        "london": "55°F, overcast",
+        "paris": "68°F, partly sunny",
+    }
+    return weather_data.get(location.lower(), f"Weather data not available for {location}")
+
+
+def get_flight_price(origin: str, destination: str) -> str:
+    """Get the price of a flight between two cities."""
+    return f"Flights from {origin} to {destination}: $450 round-trip"
+
+
+async def main() -> None:
+    # 1. Set up the FoundryChatClient
+    chat_client = FoundryChatClient(
+        project_endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
+        model=os.environ.get("FOUNDRY_MODEL", "gpt-4o"),
+        credential=AzureCliCredential(),
+    )
+
+    # 2. Create an agent with tools
+    agent = Agent(
+        client=chat_client,
+        name="travel-assistant",
+        instructions=(
+            "You are a helpful travel assistant. Use your tools to answer questions about weather and flights."
+        ),
+        tools=[get_weather, get_flight_price],
+    )
+
+    # 3. Create the evaluator — provider config goes here, once
+    evals = FoundryEvals(client=chat_client)
+
+    # =========================================================================
+    # Pattern 1: evaluate_agent(responses=...) — evaluate a response you already have
+    # =========================================================================
+    print("=" * 60)
+    print("Pattern 1: evaluate_agent(responses=...) — evaluate existing response")
+    print("=" * 60)
+
+    query = "How much does a flight from Seattle to Paris cost?"
+    response = await agent.run(query)
+    print(f"Agent said: {response.text[:100]}...")
+
+    # Pass agent= so tool definitions are extracted, queries= for the eval item context
+    results = await evaluate_agent(
+        agent=agent,
+        responses=response,
+        queries=[query],
+        evaluators=FoundryEvals(
+            client=chat_client,
+            evaluators=[FoundryEvals.RELEVANCE, FoundryEvals.TOOL_CALL_ACCURACY],
+        ),
+    )
+
+    for r in results:
+        print(f"Status: {r.status}")
+        print(f"Results: {r.passed}/{r.total} passed")
+        print(f"Portal: {r.report_url}")
+        if r.all_passed:
+            print("[PASS] All passed")
+        else:
+            print(f"[FAIL] {r.failed} failed")
+
+    # =========================================================================
+    # Pattern 2a: evaluate_agent() — batch test queries
+    # =========================================================================
+    print()
+    print("=" * 60)
+    print("Pattern 2a: evaluate_agent()")
+    print("=" * 60)
+
+    # Calls agent.run() under the covers for each query, then evaluates
+    results = await evaluate_agent(
+        agent=agent,
+        queries=[
+            "What's the weather like in Seattle?",
+            "How much does a flight from Seattle to Paris cost?",
+            "What should I pack for London?",
+        ],
+        evaluators=evals,  # uses smart defaults (auto-adds tool_call_accuracy)
+    )
+
+    for r in results:
+        print(f"Status: {r.status}")
+        print(f"Results: {r.passed}/{r.total} passed")
+        print(f"Portal: {r.report_url}")
+        if r.all_passed:
+            print("[PASS] All passed")
+        else:
+            print(f"[FAIL] {r.failed} failed")
+
+    # =========================================================================
+    # Pattern 2b: evaluate_agent() — with conversation split override
+    # =========================================================================
+    print()
+    print("=" * 60)
+    print("Pattern 2b: evaluate_agent() with conversation_split")
+    print("=" * 60)
+
+    # conversation_split forces all evaluators to use the same split strategy.
+    # FULL evaluates the entire conversation trajectory against the original query.
+    results = await evaluate_agent(
+        agent=agent,
+        queries=[
+            "What's the weather like in Seattle?",
+            "What should I pack for London?",
+        ],
+        evaluators=evals,
+        conversation_split=ConversationSplit.FULL,  # overrides evaluator defaults
+    )
+
+    for r in results:
+        print(f"Status: {r.status}")
+        print(f"Results: {r.passed}/{r.total} passed")
+        print(f"Portal: {r.report_url}")
+        if r.all_passed:
+            print("[PASS] All passed")
+        else:
+            print(f"[FAIL] {r.failed} failed")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())
@@ -0,0 +1,159 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""Mix local and cloud evaluation providers in a single evaluate_agent() call.
+
+This sample demonstrates three patterns:
+1. Local-only: Fast, API-free checks for inner-loop development.
+2. Cloud-only: Full Foundry evaluators for comprehensive quality assessment.
+3. Mixed: Local + Foundry evaluators in a single evaluate_agent() call.
+
+Mixing lets you get instant local feedback (keyword presence, tool usage)
+alongside deeper cloud-based quality evaluation (relevance, coherence)
+in one call.
+
+Prerequisites:
+- An Azure AI Foundry project with a deployed model
+- Set FOUNDRY_PROJECT_ENDPOINT and FOUNDRY_MODEL in .env
+"""
+
+import asyncio
+import os
+
+from agent_framework import (
+    Agent,
+    LocalEvaluator,
+    evaluate_agent,
+    keyword_check,
+    tool_called_check,
+)
+from agent_framework.foundry import FoundryChatClient, FoundryEvals
+from azure.identity import AzureCliCredential
+from dotenv import load_dotenv
+
+load_dotenv()
+
+
+# Define a simple tool for the agent
+def get_weather(location: str) -> str:
+    """Get the current weather for a location."""
+    weather_data = {
+        "seattle": "62°F, cloudy with a chance of rain",
+        "london": "55°F, overcast",
+        "paris": "68°F, partly sunny",
+    }
+    return weather_data.get(location.lower(), f"Weather data not available for {location}")
+
+
+async def main() -> None:
+    # 1. Set up the chat client
+    chat_client = FoundryChatClient(
+        project_endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
+        model=os.environ.get("AZURE_AI_MODEL_DEPLOYMENT_NAME", "gpt-4o"),
+        credential=AzureCliCredential(),
+    )
+
+    # 2. Create an agent with a tool
+    agent = Agent(
+        client=chat_client,
+        name="weather-assistant",
+        instructions="You are a helpful weather assistant. Use the get_weather tool to answer questions.",
+        tools=[get_weather],
+    )
+
+    # =========================================================================
+    # Pattern 1: Local evaluation only (no API calls, instant results)
+    # =========================================================================
+    print("=" * 60)
+    print("Pattern 1: Local evaluation only")
+    print("=" * 60)
+
+    local = LocalEvaluator(
+        keyword_check("weather", "seattle"),
+        tool_called_check("get_weather"),
+    )
+
+    results = await evaluate_agent(
+        agent=agent,
+        queries=["What's the weather in Seattle?"],
+        evaluators=local,
+    )
+
+    for r in results:
+        print(f"Status: {r.status}")
+        print(f"Results: {r.passed}/{r.total} passed")
+        for check_name, counts in r.per_evaluator.items():
+            print(f"  {check_name}: {counts['passed']} passed, {counts['failed']} failed")
+        if r.all_passed:
+            print("[PASS] All local checks passed!")
+        else:
+            print(f"[FAIL] Failures: {r.error}")
+
+    # =========================================================================
+    # Pattern 2: Foundry evaluation only (cloud-based quality assessment)
+    # =========================================================================
+    print()
+    print("=" * 60)
+    print("Pattern 2: Foundry evaluation only")
+    print("=" * 60)
+
+    foundry = FoundryEvals(client=chat_client)
+
+    results = await evaluate_agent(
+        agent=agent,
+        queries=["What's the weather in Seattle?"],
+        evaluators=foundry,
+    )
+
+    for r in results:
+        print(f"Status: {r.status}")
+        print(f"Results: {r.passed}/{r.total} passed")
+        print(f"Portal: {r.report_url}")
+        if r.all_passed:
+            print("[PASS] All passed")
+        else:
+            print(f"[FAIL] {r.failed} failed")
+
+    # =========================================================================
+    # Pattern 3: Mixed — local + Foundry in one call
+    # =========================================================================
+    print()
+    print("=" * 60)
+    print("Pattern 3: Mixed local + Foundry evaluation")
+    print("=" * 60)
+
+    # Local checks: fast smoke tests
+    local = LocalEvaluator(
+        keyword_check("weather"),
+        tool_called_check("get_weather"),
+    )
+
+    # Foundry: deep quality assessment
+    foundry = FoundryEvals(client=chat_client)
+
+    # Pass both as a list — returns one EvalResults per provider
+    results = await evaluate_agent(
+        agent=agent,
+        queries=[
+            "What's the weather in Seattle?",
+            "Tell me the weather in London",
+        ],
+        evaluators=[local, foundry],
+    )
+
+    for r in results:
+        status = "PASS" if r.all_passed else "FAIL"
+        print(f"  {status} {r.provider}: {r.passed}/{r.total} passed")
+        for check_name, counts in r.per_evaluator.items():
+            print(f"      {check_name}: {counts['passed']}/{counts['passed'] + counts['failed']}")
+        if r.report_url:
+            print(f"      Portal: {r.report_url}")
+
+    if all(r.all_passed for r in results):
+        print("[PASS] All checks passed (local + Foundry)!")
+    else:
+        failed = [r.provider for r in results if not r.all_passed]
+        print(f"[FAIL] Failed providers: {', '.join(failed)}")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())
@@ -0,0 +1,182 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""Evaluate multi-turn conversations with different split strategies.
+
+The same multi-turn conversation can be split different ways, each evaluating
+a different aspect of agent behavior:
+
+1. LAST_TURN (default) — "Was the last response good given context?"
+2. FULL — "Did the whole conversation serve the original request?"
+3. per_turn_items — "Was each individual response appropriate?"
+
+Prerequisites:
+- An Azure AI Foundry project with a deployed model
+- Set FOUNDRY_PROJECT_ENDPOINT and AZURE_AI_MODEL_DEPLOYMENT_NAME in .env
+"""
+
+import asyncio
+import os
+
+from agent_framework import Content, ConversationSplit, EvalItem, FunctionTool, Message
+from agent_framework.foundry import FoundryChatClient, FoundryEvals
+from azure.identity import AzureCliCredential
+from dotenv import load_dotenv
+
+load_dotenv()
+
+# A multi-turn conversation with tool calls that we'll evaluate three ways.
+# Uses framework Message/Content types for type-safe conversation construction.
+CONVERSATION: list[Message] = [
+    # Turn 1: user asks about weather -> agent calls tool -> responds
+    Message("user", ["What's the weather in Seattle?"]),
+    Message(
+        "assistant",
+        [
+            Content.from_function_call("c1", "get_weather", arguments={"location": "seattle"}),
+        ],
+    ),
+    Message(
+        "tool",
+        [
+            Content.from_function_result("c1", result="62°F, cloudy with a chance of rain"),
+        ],
+    ),
+    Message("assistant", ["Seattle is 62°F, cloudy with a chance of rain."]),
+    # Turn 2: user asks about Paris -> agent calls tool -> responds
+    Message("user", ["And Paris?"]),
+    Message(
+        "assistant",
+        [
+            Content.from_function_call("c2", "get_weather", arguments={"location": "paris"}),
+        ],
+    ),
+    Message(
+        "tool",
+        [
+            Content.from_function_result("c2", result="68°F, partly sunny"),
+        ],
+    ),
+    Message("assistant", ["Paris is 68°F, partly sunny."]),
+    # Turn 3: user asks for comparison -> agent synthesizes without tool
+    Message("user", ["Can you compare them?"]),
+    Message(
+        "assistant",
+        [
+            (
+                "Seattle is cooler at 62°F with rain likely, while Paris is warmer "
+                "at 68°F and partly sunny. Paris is the better choice for outdoor activities."
+            ),
+        ],
+    ),
+]
+
+TOOLS = [
+    FunctionTool(
+        name="get_weather",
+        description="Get the current weather for a location.",
+    ),
+]
+
+
+def print_split(item: EvalItem, split: ConversationSplit = ConversationSplit.LAST_TURN) -> None:
+    """Print the query/response split for an EvalItem."""
+    query_msgs, response_msgs = item.split_messages(split)
+    print(f"  query_messages ({len(query_msgs)}):")
+    for m in query_msgs:
+        text = m.text or ""
+        print(f"    {m.role}: {text[:70]}")
+    print(f"  response_messages ({len(response_msgs)}):")
+    for m in response_msgs:
+        text = m.text or ""
+        print(f"    {m.role}: {text[:70]}")
+
+
+async def main() -> None:
+    chat_client = FoundryChatClient(
+        project_endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
+        model=os.environ.get("AZURE_AI_MODEL_DEPLOYMENT_NAME", "gpt-4o"),
+        credential=AzureCliCredential(),
+    )
+
+    # =========================================================================
+    # Strategy 1: LAST_TURN (default)
+    # "Given all context, was the last response good?"
+    # =========================================================================
+    print("=" * 70)
+    print("Strategy 1: LAST_TURN — evaluate the final response")
+    print("=" * 70)
+
+    # EvalItem takes conversation + tools; query/response are derived via split strategy
+    item = EvalItem(CONVERSATION, tools=TOOLS)
+
+    print_split(item, ConversationSplit.LAST_TURN)
+
+    results = await FoundryEvals(
+        client=chat_client,
+        evaluators=[FoundryEvals.RELEVANCE, FoundryEvals.COHERENCE],
+        # conversation_split defaults to LAST_TURN
+    ).evaluate([item], eval_name="Split Strategy: LAST_TURN")
+
+    print(f"\n  Result: {results.passed}/{results.total} passed")
+    print(f"  Portal: {results.report_url}")
+    for ir in results.items:
+        for s in ir.scores:
+            print(f"    {'PASS' if s.passed else 'FAIL'} {s.name}: {s.score}")
+    print()
+
+    # =========================================================================
+    # Strategy 2: FULL
+    # "Given the original request, did the whole conversation serve the user?"
+    # =========================================================================
+    print("=" * 70)
+    print("Strategy 2: FULL — evaluate the entire conversation trajectory")
+    print("=" * 70)
+
+    print_split(item, ConversationSplit.FULL)
+
+    results = await FoundryEvals(
+        client=chat_client,
+        evaluators=[FoundryEvals.RELEVANCE, FoundryEvals.COHERENCE],
+        conversation_split=ConversationSplit.FULL,
+    ).evaluate([item], eval_name="Split Strategy: FULL")
+
+    print(f"\n  Result: {results.passed}/{results.total} passed")
+    print(f"  Portal: {results.report_url}")
+    for ir in results.items:
+        for s in ir.scores:
+            print(f"    {'PASS' if s.passed else 'FAIL'} {s.name}: {s.score}")
+    print()
+
+    # =========================================================================
+    # Strategy 3: per_turn_items
+    # "Was each individual response appropriate at that point?"
+    # =========================================================================
+    print("=" * 70)
+    print("Strategy 3: per_turn_items — evaluate each turn independently")
+    print("=" * 70)
+
+    items = EvalItem.per_turn_items(CONVERSATION, tools=TOOLS)
+    print(f"  Split into {len(items)} items from {len(CONVERSATION)} messages:\n")
+    for i, it in enumerate(items):
+        print(f"  Turn {i + 1}: query={it.query!r}, response={it.response[:60]!r}...")
+    print()
+
+    results = await FoundryEvals(
+        client=chat_client,
+        evaluators=[FoundryEvals.RELEVANCE, FoundryEvals.COHERENCE],
+    ).evaluate(items, eval_name="Split Strategy: Per-Turn")
+
+    print(f"\n  Result: {results.passed}/{results.total} passed ({len(items)} items × 2 evaluators)")
+    print(f"  Portal: {results.report_url}")
+    for ir in results.items:
+        for s in ir.scores:
+            print(f"    {'PASS' if s.passed else 'FAIL'} {s.name}: {s.score}")
+    print()
+
+    print("=" * 70)
+    print("All strategies complete. Compare results in the Foundry portal.")
+    print("=" * 70)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())
@@ -0,0 +1,89 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""Evaluate tool-calling accuracy using Azure AI Foundry's TOOL_CALL_ACCURACY evaluator.
+
+This sample demonstrates evaluating how well an agent selects and invokes tools
+by using ``FoundryEvals.evaluate()`` with ``TOOL_CALL_ACCURACY``.
+
+Prerequisites:
+- An Azure AI Foundry project with a deployed model
+- Set FOUNDRY_PROJECT_ENDPOINT and AZURE_AI_MODEL_DEPLOYMENT_NAME in .env
+"""
+
+import asyncio
+import os
+
+from agent_framework import Agent, AgentEvalConverter
+from agent_framework.foundry import FoundryChatClient, FoundryEvals
+from azure.identity import AzureCliCredential
+from dotenv import load_dotenv
+
+load_dotenv()
+
+
+def get_weather(location: str) -> str:
+    """Get the current weather for a location."""
+    weather_data = {
+        "seattle": "62°F, cloudy with a chance of rain",
+        "london": "55°F, overcast",
+        "paris": "68°F, partly sunny",
+    }
+    return weather_data.get(location.lower(), f"Weather data not available for {location}")
+
+
+def get_flight_price(origin: str, destination: str) -> str:
+    """Get the price of a flight between two cities."""
+    return f"Flights from {origin} to {destination}: $450 round-trip"
+
+
+async def main() -> None:
+    chat_client = FoundryChatClient(
+        project_endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
+        model=os.environ.get("AZURE_AI_MODEL_DEPLOYMENT_NAME", "gpt-4o"),
+        credential=AzureCliCredential(),
+    )
+
+    # Create an agent with tools
+    agent = Agent(
+        client=chat_client,
+        name="travel-assistant",
+        instructions=(
+            "You are a helpful travel assistant. "
+            "Use your tools to answer questions about weather and flights."
+        ),
+        tools=[get_weather, get_flight_price],
+    )
+
+    # Run the agent and convert responses to eval items
+    queries = [
+        "What's the weather in Paris?",
+        "Find me a flight from London to Seattle",
+    ]
+
+    items = []
+    for q in queries:
+        response = await agent.run(q)
+        print(f"Query: {q}")
+        print(f"Response: {response.text[:100]}...")
+
+        item = AgentEvalConverter.to_eval_item(query=q, response=response, agent=agent)
+        items.append(item)
+
+        print(f"  Has tools: {item.tools is not None}")
+        if item.tools:
+            print(f"  Tools: {[t.name for t in item.tools]}")
+
+    # Submit to Foundry with tool_call_accuracy evaluator
+    evals = FoundryEvals(
+        client=chat_client,
+        evaluators=[FoundryEvals.RELEVANCE, FoundryEvals.TOOL_CALL_ACCURACY],
+    )
+    results = await evals.evaluate(items, eval_name="Tool Call Accuracy Eval")
+
+    print(f"\nStatus: {results.status}")
+    print(f"Results: {results.passed}/{results.total} passed")
+    print(f"Portal: {results.report_url}")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())
@@ -0,0 +1,114 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""Evaluate agent responses that already exist in Foundry (zero-code-change).
+
+This sample demonstrates two patterns:
+1. evaluate_traces(response_ids=...) — Evaluate specific Responses API responses by ID.
+2. evaluate_traces(agent_id=...) — Evaluate agent behavior from OTel traces in App Insights.
+
+These are the "zero-code-change" evaluation paths — the agent has already run,
+and you're evaluating what happened after the fact.
+
+Prerequisites:
+- An Azure AI Foundry project with a deployed model
+- Response IDs from prior agent runs (for Pattern 1)
+- OTel traces exported to App Insights (for Pattern 2)
+- Set FOUNDRY_PROJECT_ENDPOINT and AZURE_AI_MODEL_DEPLOYMENT_NAME in .env
+"""
+
+import asyncio
+import os
+
+from agent_framework.foundry import FoundryChatClient, FoundryEvals, evaluate_traces
+from azure.identity import AzureCliCredential
+from dotenv import load_dotenv
+
+load_dotenv()
+
+
+async def main() -> None:
+    # 1. Set up the chat client
+    chat_client = FoundryChatClient(
+        project_endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
+        model=os.environ.get("AZURE_AI_MODEL_DEPLOYMENT_NAME", "gpt-4o"),
+        credential=AzureCliCredential(),
+    )
+
+    # =========================================================================
+    # Pattern 1: evaluate_traces(response_ids=...) — By response ID
+    # =========================================================================
+    # If your agent uses the Responses API (e.g., FoundryChatClient),
+    # each run produces a response_id. Pass those IDs to evaluate_traces()
+    # and Foundry retrieves the full conversation for evaluation.
+    print("=" * 60)
+    print("Pattern 1: evaluate_traces(response_ids=...)")
+    print("=" * 60)
+
+    # Replace these with actual response IDs from your agent runs
+    response_ids = [
+        "resp_abc123",
+        "resp_def456",
+    ]
+
+    results = await evaluate_traces(
+        response_ids=response_ids,
+        evaluators=[FoundryEvals.RELEVANCE, FoundryEvals.GROUNDEDNESS, FoundryEvals.TOOL_CALL_ACCURACY],
+        client=chat_client,
+    )
+
+    print(f"Status: {results.status}")
+    print(f"Results: {results.result_counts}")
+    print(f"Portal: {results.report_url}")
+
+    # =========================================================================
+    # Pattern 2: evaluate_traces(response_ids=...) — Batch response evaluation
+    # =========================================================================
+    # Evaluate multiple prior responses by their IDs.  This uses the same
+    # response-based data source under the covers but lets you batch them.
+    #
+    # A future trace-based pattern (agent_id + lookback_hours) is shown
+    # commented out below — it requires OTel traces exported to App Insights.
+    print()
+    print("=" * 60)
+    print("Pattern 2: evaluate_traces(response_ids=...)")
+    print("=" * 60)
+
+    # Evaluate by response IDs (uses response-based data source internally)
+    results = await evaluate_traces(
+        response_ids=response_ids,
+        evaluators=[FoundryEvals.RELEVANCE, FoundryEvals.COHERENCE],
+        client=chat_client,
+    )
+
+    print(f"Status: {results.status}")
+    print(f"Portal: {results.report_url}")
+
+    # Evaluate by agent ID + time window (when trace-based API is available)
+    # results = await evaluate_traces(
+    #     agent_id="travel-bot",
+    #     evaluators=[FoundryEvals.INTENT_RESOLUTION, FoundryEvals.TASK_ADHERENCE],
+    #     client=chat_client,
+    #     lookback_hours=24,
+    # )
+
+
+if __name__ == "__main__":
+    asyncio.run(main())
+
+
+"""
+Sample output (with actual Azure AI Foundry project and valid response IDs):
+
+============================================================
+Pattern 1: evaluate_traces(response_ids=...)
+============================================================
+Status: completed
+Results: {'passed': 2, 'failed': 0, 'errored': 0}
+Portal: https://ai.azure.com/...
+
+============================================================
+Pattern 2: evaluate_traces(response_ids=...)
+============================================================
+Status: completed
+Portal: https://ai.azure.com/...
+"""
@@ -0,0 +1,176 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""Evaluate a multi-agent workflow using Azure AI Foundry evaluators.
+
+This sample demonstrates two patterns:
+1. Post-hoc: Run the workflow, then evaluate the result you already have.
+2. Run + evaluate: Pass queries and let evaluate_workflow() run the workflow for you.
+
+Both patterns return a list of results (one per provider), each with a per-agent
+breakdown in sub_results so you can identify which agent is underperforming.
+
+Prerequisites:
+- An Azure AI Foundry project with a deployed model
+- Set FOUNDRY_PROJECT_ENDPOINT and AZURE_AI_MODEL_DEPLOYMENT_NAME in .env
+"""
+
+import asyncio
+import os
+
+from agent_framework import Agent, evaluate_workflow
+from agent_framework.foundry import FoundryChatClient, FoundryEvals
+from agent_framework_orchestrations import SequentialBuilder
+from azure.identity import AzureCliCredential
+from dotenv import load_dotenv
+
+load_dotenv()
+
+
+# Simple tools for the agents
+def get_weather(location: str) -> str:
+    """Get the current weather for a location."""
+    weather_data = {
+        "seattle": "62°F, cloudy with a chance of rain",
+        "london": "55°F, overcast",
+        "paris": "68°F, partly sunny",
+    }
+    return weather_data.get(location.lower(), f"Weather data not available for {location}")
+
+
+def get_flight_price(origin: str, destination: str) -> str:
+    """Get the price of a flight between two cities."""
+    return f"Flights from {origin} to {destination}: $450 round-trip"
+
+
+async def main() -> None:
+    # 1. Set up the chat client
+    client = FoundryChatClient(
+        project_endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
+        model=os.environ.get("AZURE_AI_MODEL_DEPLOYMENT_NAME", "gpt-4o"),
+        credential=AzureCliCredential(),
+    )
+
+    # 2. Create agents for a sequential workflow
+    # Use store=False so agents don't chain conversation state via previous_response_id.
+    # This allows the workflow to be run multiple times without stale state issues.
+    researcher = Agent(
+        client=client,
+        name="researcher",
+        instructions=(
+            "You are a travel researcher. Use your tools to gather weather "
+            "and flight information for the destination the user asks about."
+        ),
+        tools=[get_weather, get_flight_price],
+        default_options={"store": False},
+    )
+
+    planner = Agent(
+        client=client,
+        name="planner",
+        instructions=(
+            "You are a travel planner. Based on the research provided, "
+            "create a concise travel recommendation with packing tips."
+        ),
+        default_options={"store": False},
+    )
+
+    # 3. Build a sequential workflow: researcher -> planner
+    workflow = SequentialBuilder(participants=[researcher, planner]).build()
+
+    # 4. Create the evaluator — provider config goes here, once
+    evals = FoundryEvals(client=client)
+
+    # =========================================================================
+    # Pattern 1: Post-hoc — evaluate a workflow run you already did
+    # =========================================================================
+    print("=" * 60)
+    print("Pattern 1: Post-hoc workflow evaluation")
+    print("=" * 60)
+
+    result = await workflow.run("Plan a trip from Seattle to Paris")
+
+    eval_results = await evaluate_workflow(
+        workflow=workflow,
+        workflow_result=result,
+        evaluators=evals,
+    )
+
+    for r in eval_results:
+        print(f"\nOverall: {r.status}")
+        print(f"  Passed: {r.passed}/{r.total}")
+        print(f"  Portal: {r.report_url}")
+
+        print("\nPer-agent breakdown:")
+        for agent_name, agent_eval in r.sub_results.items():
+            print(f"  {agent_name}: {agent_eval.passed}/{agent_eval.total} passed")
+            if agent_eval.report_url:
+                print(f"    Portal: {agent_eval.report_url}")
+
+    # =========================================================================
+    # Pattern 2: Run + evaluate with multiple queries
+    # =========================================================================
+    # Build a fresh workflow to avoid stale session state from Pattern 1.
+    # The Responses API tracks previous_response_id per session, so reusing
+    # a workflow after a run would reference stale tool calls.
+    workflow2 = SequentialBuilder(participants=[researcher, planner]).build()
+
+    print()
+    print("=" * 60)
+    print("Pattern 2: Run + evaluate with multiple queries")
+    print("=" * 60)
+
+    eval_results = await evaluate_workflow(
+        workflow=workflow2,
+        queries=[
+            "Plan a trip from London to Tokyo",
+            "Plan a trip from New York to Rome",
+        ],
+        evaluators=FoundryEvals(
+            client=client,
+            evaluators=[FoundryEvals.RELEVANCE, FoundryEvals.TASK_ADHERENCE],
+        ),
+    )
+
+    for r in eval_results:
+        print(f"\nOverall: {r.status}")
+        print(f"  Passed: {r.passed}/{r.total}")
+        if r.report_url:
+            print(f"  Portal: {r.report_url}")
+
+        print("\nPer-agent breakdown:")
+        for agent_name, agent_eval in r.sub_results.items():
+            print(f"  {agent_name}: {agent_eval.passed}/{agent_eval.total} passed")
+            if agent_eval.report_url:
+                print(f"    Portal: {agent_eval.report_url}")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())
+
+
+"""
+Sample output (with actual Azure AI Foundry project):
+
+============================================================
+Pattern 1: Post-hoc workflow evaluation
+============================================================
+
+Overall: completed
+  Passed: 2/2
+  Portal: https://ai.azure.com/...
+
+Per-agent breakdown:
+  researcher: 1/1 passed
+  planner: 1/1 passed
+
+============================================================
+Pattern 2: Run + evaluate with multiple queries
+============================================================
+
+Overall: completed
+  Passed: 4/4
+
+Per-agent breakdown:
+  researcher: 2/2 passed
+  planner: 2/2 passed
+"""
@@ -1,3 +1 @@
-AZURE_OPENAI_ENDPOINT="..."
-AZURE_OPENAI_API_KEY="..."
-AZURE_AI_PROJECT_ENDPOINT="https://<your-ai-resource>.services.ai.azure.com/api/projects/<your-ai-project>/"
+FOUNDRY_PROJECT_ENDPOINT=https://<your-project>.services.ai.azure.com
@@ -6,31 +6,27 @@ This sample demonstrates the self-reflection pattern using Agent Framework and A

 **What it demonstrates:**
 - Iterative self-reflection loop that automatically improves responses based on groundedness evaluation
+- Using `FoundryEvals` to score each iteration via the Foundry Groundedness evaluator
 - Batch processing of prompts from JSONL files with progress tracking
- Using `AzureOpenAIResponsesClient` with a Project Endpoint and Azure CLI authentication
+- Using `FoundryChatClient` with a Project Endpoint and Azure CLI authentication
 - Comprehensive summary statistics and detailed result tracking

 ## Prerequisites

 ### Azure Resources
- **Azure OpenAI Responses in Foundry**: Deploy models (default: gpt-5.2 for both agent and judge)
+- **Azure AI Foundry project**: Deploy models (default: gpt-5.2 for both agent and judge)
 - **Azure CLI**: Run `az login` to authenticate

-### Python Environment
-```bash
-pip install agent-framework-core pandas --pre
-```
-
 ### Environment Variables
 ```bash
-AZURE_AI_PROJECT_ENDPOINT=https://<your-ai-resource>.services.ai.azure.com/api/projects/<your-ai-project>/
+FOUNDRY_PROJECT_ENDPOINT=https://<your-project>.services.ai.azure.com
 ```

 ## Running the Sample

 ```bash
 # Basic usage
-python self_reflection.py
+uv run python samples/05-end-to-end/evaluation/self_reflection/self_reflection.py

 # With options
 python self_reflection.py --input my_prompts.jsonl \
@@ -42,8 +38,8 @@ python self_reflection.py --input my_prompts.jsonl \
 **CLI Options:**
 - `--input`, `-i`: Input JSONL file
 - `--output`, `-o`: Output JSONL file
- `--agent-model`, `-m`: Agent model name (default: gpt-4.1)
- `--judge-model`, `-e`: Evaluator model name (default: gpt-4.1)
+- `--agent-model`, `-m`: Agent model name (default: gpt-5.2)
+- `--judge-model`, `-e`: Evaluator model name (default: gpt-5.2)
 - `--max-reflections`: Max iterations (default: 3)
 - `--limit`, `-n`: Process only first N prompts

@@ -51,7 +47,7 @@ python self_reflection.py --input my_prompts.jsonl \

 The agent iteratively improves responses:
 1. Generate initial response
-2. Evaluate groundedness (1-5 scale)
+2. Evaluate groundedness via `FoundryEvals` (1-5 scale)
 3. If score < 5, provide feedback and retry
 4. Stop at max iterations or perfect score (5/5)

@@ -70,7 +66,7 @@ In the Foundry UI, under `Build`/`Evaluations` you can view detailed results for
 - Context
 - Query
 - Response
- Groundedness scores and reasoning for each interation of each prompt
+- Groundedness scores and reasoning for each iteration of each prompt

 ## Related Resources

@@ -17,26 +17,20 @@ import time
 from pathlib import Path
 from typing import Any

-import openai
 import pandas as pd
-from agent_framework import Agent, Message
-from agent_framework.foundry import FoundryChatClient
-from azure.ai.projects import AIProjectClient
-from azure.identity import AzureCliCredential
+from agent_framework import Agent, EvalItem, Message
+from agent_framework.foundry import FoundryChatClient, FoundryEvals
+from azure.identity.aio import AzureCliCredential as AsyncAzureCliCredential
 from dotenv import load_dotenv
-from openai.types.eval_create_params import DataSourceConfigCustom
-from openai.types.evals.create_eval_jsonl_run_data_source_param import (
-    CreateEvalJSONLRunDataSourceParam,
-    SourceFileContent,
-    SourceFileContentContent,
-)

 """
 Self-Reflection LLM Runner

 Reflexion: language agents with verbal reinforcement learning.
 Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023.
-In Proceedings of the 37th International Conference on Neural Information Processing Systems (NIPS '23). Curran Associates Inc., Red Hook, NY, USA, Article 377, 8634–8652.
+In Proceedings of the 37th International Conference on Neural Information
+Processing Systems (NIPS '23). Curran Associates Inc., Red Hook, NY, USA,
+Article 377, 8634–8652.
 https://arxiv.org/abs/2303.11366

 This module implements a self-reflection loop for LLM responses using groundedness evaluation.
@@ -59,8 +53,8 @@ Usage as CLI with extra options:
 SUMMARY
 ============================================================
 Total prompts processed: 31
-  ✓ Successful: 30
-  ✗ Failed: 1
+  [PASS] Successful: 30
+  [FAIL] Failed: 1

 Groundedness Scores:
  Average best score: 4.77/5
@@ -77,7 +71,7 @@ Iteration Statistics:
  Best on first try: 25/30 (83.3%)
 ============================================================

-✓ Processing complete!
+[PASS] Processing complete!

 """

@@ -86,104 +80,37 @@ DEFAULT_AGENT_MODEL = "gpt-5.2"
 DEFAULT_JUDGE_MODEL = "gpt-5.2"


-def create_openai_client():
-    endpoint = os.environ["FOUNDRY_PROJECT_ENDPOINT"]
-    credential = AzureCliCredential()
-    project_client = AIProjectClient(endpoint=endpoint, credential=credential)
-    return project_client.get_openai_client()
-
-
-def create_async_project_client():
-    from azure.ai.projects.aio import AIProjectClient as AsyncAIProjectClient
-    from azure.identity.aio import AzureCliCredential as AsyncAzureCliCredential
-
-    return AsyncAIProjectClient(endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"], credential=AsyncAzureCliCredential())
-
-
-def create_eval(client: openai.OpenAI, judge_model: str) -> openai.types.EvalCreateResponse:
-    print("Creating Eval")
-    data_source_config = DataSourceConfigCustom({
-        "type": "custom",
-        "item_schema": {
-            "type": "object",
-            "properties": {
-                "query": {"type": "string"},
-                "response": {"type": "string"},
-                "context": {"type": "string"},
-            },
-            "required": [],
-        },
-        "include_sample_schema": True,
-    })
-
-    testing_criteria = [
-        {
-            "type": "azure_ai_evaluator",
-            "name": "groundedness",
-            "evaluator_name": "builtin.groundedness",
-            "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}", "context": "{{item.context}}"},
-            "initialization_parameters": {"deployment_name": f"{judge_model}"},
-        }
-    ]
-
-    return client.evals.create(
-        name="Eval",
-        data_source_config=data_source_config,
-        testing_criteria=testing_criteria,  # type: ignore
-    )
-
-
-def run_eval(
-    client: openai.OpenAI,
-    eval_object: openai.types.EvalCreateResponse,
+async def evaluate_groundedness(
+    evals: FoundryEvals,
    query: str,
    response: str,
    context: str,
-):
-    eval_run_object = client.evals.runs.create(
-        eval_id=eval_object.id,
-        name="inline_data_run",
-        metadata={"team": "eval-exp", "scenario": "inline-data-v1"},
-        data_source=CreateEvalJSONLRunDataSourceParam(
-            type="jsonl",
-            source=SourceFileContent(
-                type="file_content",
-                content=[
-                    SourceFileContentContent(
-                        item={
-                            "query": query,
-                            "context": context,
-                            "response": response,
-                        }
-                    ),
-                ],
-            ),
-        ),
+) -> float | None:
+    """Run a single groundedness evaluation and return the score."""
+    item = EvalItem(
+        conversation=[
+            Message("user", [query]),
+            Message("assistant", [response]),
+        ],
+        context=context,
    )
-
-    eval_run_response = client.evals.runs.retrieve(run_id=eval_run_object.id, eval_id=eval_object.id)
-
-    MAX_RETRY = 10
-    for _ in range(0, MAX_RETRY):
-        run = client.evals.runs.retrieve(run_id=eval_run_response.id, eval_id=eval_object.id)
-        if run.status == "failed":
-            print(
-                f"Eval run failed. Run ID: {run.id}, Status: {run.status}, Error: {getattr(run, 'error', 'Unknown error')}"
-            )
-            continue
-        if run.status == "completed":
-            return list(client.evals.runs.output_items.list(run_id=run.id, eval_id=eval_object.id))
-        time.sleep(5)
-
-    print("Eval result retrieval timeout.")
+    results = await evals.evaluate(
+        [item],
+        eval_name="Self-Reflection Groundedness",
+    )
+    if results.status != "completed" or not results.items:
+        return None
+    # Return the first evaluator score
+    for score in results.items[0].scores:
+        if score.score is not None:
+            return float(score.score)
    return None


 async def execute_query_with_self_reflection(
    *,
-    client: openai.OpenAI,
+    evals: FoundryEvals,
    agent: Agent,
-    eval_object: openai.types.EvalCreateResponse,
    full_user_query: str,
    context: str,
    max_self_reflections: int = 3,
@@ -192,10 +119,10 @@ async def execute_query_with_self_reflection(
    Execute a query with self-reflection loop.

    Args:
+        evals: FoundryEvals instance for groundedness scoring
        agent: Agent instance to use for generating responses
        full_user_query: Complete prompt including system prompt, user request, and context
        context: Context document for groundedness evaluation
-        evaluator: Groundedness evaluator function
        max_self_reflections: Maximum number of self-reflection iterations

    Returns:
@@ -205,7 +132,6 @@ async def execute_query_with_self_reflection(
            - best_iteration: Iteration number where best score was achieved
            - iteration_scores: List of groundedness scores for each iteration
            - messages: Full conversation history
-            - usage_metadata: Token usage information
            - num_retries: Number of iterations performed
            - total_groundedness_eval_time: Time spent on evaluations (seconds)
            - total_end_to_end_time: Total execution time (seconds)
@@ -219,7 +145,7 @@ async def execute_query_with_self_reflection(
    raw_response = None
    total_groundedness_eval_time = 0.0
    start_time = time.time()
-    iteration_scores = []  # Store all iteration scores in structured format
+    iteration_scores = []

    for i in range(max_self_reflections):
        print(f"  Self-reflection iteration {i + 1}/{max_self_reflections}...")
@@ -227,22 +153,16 @@ async def execute_query_with_self_reflection(
        raw_response = await agent.run(messages=messages)
        agent_response = raw_response.text

-        # Evaluate groundedness
+        # Evaluate groundedness using FoundryEvals
        start_time_eval = time.time()
-        eval_run_output_items = run_eval(
-            client=client,
-            eval_object=eval_object,
-            query=full_user_query,
-            response=agent_response,
-            context=context,
-        )
-        if eval_run_output_items is None:
-            print(f"  ⚠️ Groundedness evaluation failed (timeout or error) for iteration {i + 1}.")
-            continue
-        score = eval_run_output_items[0].results[0].score
+        score = await evaluate_groundedness(evals, full_user_query, agent_response, context)
        end_time_eval = time.time()
        total_groundedness_eval_time += end_time_eval - start_time_eval

+        if score is None:
+            print(f"  ⚠️ Groundedness evaluation failed for iteration {i + 1}.")
+            continue
+
        # Store score in structured format
        iteration_scores.append(score)

@@ -252,15 +172,15 @@ async def execute_query_with_self_reflection(
        # Update best response if improved
        if score > best_score:
            if best_score > 0:
-                print(f"  ✓ Score improved from {best_score} to {score}/{max_score}")
+                print(f"  [PASS] Score improved from {best_score} to {score}/{max_score}")
            best_score = score
            best_response = agent_response
            best_iteration = i + 1
            if score == max_score:
-                print("  ✓ Perfect groundedness score achieved!")
+                print("  [PASS] Perfect groundedness score achieved!")
                break
        else:
-            print(f"  → No improvement (score: {score}/{max_score}). Trying again...")
+            print(f"  -> No improvement (score: {score}/{max_score}). Trying again...")

        # Add to conversation history
        messages.append(Message("assistant", [agent_response]))
@@ -293,7 +213,6 @@ async def execute_query_with_self_reflection(


 async def run_self_reflection_batch(
-    project_client: AIProjectClient,
    input_file: str,
    output_file: str,
    agent_model: str = DEFAULT_AGENT_MODEL,
@@ -301,7 +220,7 @@ async def run_self_reflection_batch(
    max_self_reflections: int = 3,
    env_file: str | None = None,
    limit: int | None = None,
-):
+) -> None:
    """
    Run self-reflection on a batch of prompts.

@@ -315,17 +234,36 @@ async def run_self_reflection_batch(
        limit: Optional limit to process only the first N prompts
    """
    # Load environment variables
-    if env_file and os.path.exists(env_file):
+    if env_file:
+        if not os.path.isfile(env_file):
+            raise FileNotFoundError(f"Env file not found: {env_file}")
        load_dotenv(env_file, override=True)
    else:
        load_dotenv(override=True)

-    # Create agent, it loads environment variables AZURE_OPENAI_API_KEY and AZURE_OPENAI_ENDPOINT automatically
-    responses_client = FoundryChatClient(
+    from azure.ai.projects.aio import AIProjectClient as AsyncAIProjectClient
+
+    endpoint = os.environ["FOUNDRY_PROJECT_ENDPOINT"]
+    credential = AsyncAzureCliCredential()
+    project_client = AsyncAIProjectClient(endpoint=endpoint, credential=credential)
+
+    # Create agent client
+    agent_client = FoundryChatClient(
        project_client=project_client,
        model=agent_model,
    )

+    # Create FoundryEvals for groundedness scoring
+    judge_client = FoundryChatClient(
+        project_client=project_client,
+        model=judge_model,
+    )
+    evals = FoundryEvals(
+        client=judge_client,
+        model=judge_model,
+        evaluators=[FoundryEvals.GROUNDEDNESS],
+    )
+
    # Load input data
    input_path = (Path(__file__).parent / input_file).resolve()
    print(f"Loading prompts from: {input_path}")
@@ -351,13 +289,6 @@ async def run_self_reflection_batch(
    if missing_columns:
        raise ValueError(f"Input file missing required columns: {missing_columns}")

-    # Configure clients
-    print("Configuring Azure OpenAI client...")
-    client = create_openai_client()
-
-    # Create Eval
-    eval_object = create_eval(client=client, judge_model=judge_model)
-
    # Process each prompt
    print(f"Max self-reflections: {max_self_reflections}\n")

@@ -367,9 +298,8 @@ async def run_self_reflection_batch(

        try:
            result = await execute_query_with_self_reflection(
-                client=client,
-                agent=Agent(client=responses_client, instructions=row["system_instruction"]),
-                eval_object=eval_object,
+                evals=evals,
+                agent=Agent(client=agent_client, instructions=row["system_instruction"]),
                full_user_query=row["full_prompt"],
                context=row["context_document"],
                max_self_reflections=max_self_reflections,
@@ -393,13 +323,13 @@ async def run_self_reflection_batch(
            results.append(result_data)

            print(
-                f"  ✓ Completed with score: {result['best_response_score']}/5 "
+                f"  [PASS] Completed with score: {result['best_response_score']}/5 "
                f"(best at iteration {result['best_iteration']}/{result['num_retries']}, "
                f"time: {result['total_end_to_end_time']:.1f}s)\n"
            )

        except Exception as e:
-            print(f"  ✗ Error: {str(e)}\n")
+            print(f"  [FAIL] Error: {str(e)}\n")

            # Save error information
            error_data = {
@@ -434,8 +364,8 @@ async def run_self_reflection_batch(
    print("SUMMARY")
    print("=" * 60)
    print(f"Total prompts processed: {len(results_df)}")
-    print(f"  ✓ Successful: {len(successful_runs)}")
-    print(f"  ✗ Failed: {len(failed_runs)}")
+    print(f"  [PASS] Successful: {len(successful_runs)}")
+    print(f"  [FAIL] Failed: {len(failed_runs)}")

    if len(successful_runs) > 0:
        # Extract scores and iteration data from nested agent_response dict
@@ -452,9 +382,8 @@ async def run_self_reflection_batch(
            perfect_scores = sum(1 for s in best_scores if s == 5)
            print("\nGroundedness Scores:")
            print(f"  Average best score: {avg_score:.2f}/5")
-            print(
-                f"  Perfect scores (5/5): {perfect_scores}/{len(best_scores)} ({100 * perfect_scores / len(best_scores):.1f}%)"
-            )
+            pct = 100 * perfect_scores / len(best_scores)
+            print(f"  Perfect scores (5/5): {perfect_scores}/{len(best_scores)} ({pct:.1f}%)")

            # Calculate improvement metrics
            if iteration_scores_list:
@@ -472,9 +401,8 @@ async def run_self_reflection_batch(
                    print(f"  Average first score: {avg_first_score:.2f}/5")
                    print(f"  Average final score: {avg_last_score:.2f}/5")
                    print(f"  Average improvement: +{avg_improvement:.2f}")
-                    print(
-                        f"  Responses that improved: {improved_count}/{len(improvements)} ({100 * improved_count / len(improvements):.1f}%)"
-                    )
+                    pct = 100 * improved_count / len(improvements)
+                    print(f"  Responses that improved: {improved_count}/{len(improvements)} ({pct:.1f}%)")

            # Show iteration statistics
            if iterations:
@@ -486,6 +414,8 @@ async def run_self_reflection_batch(

    print("=" * 60)

+    await credential.close()
+

 async def main():
    """CLI entry point."""
@@ -519,7 +449,6 @@ async def main():
    # Run the batch processing
    try:
        await run_self_reflection_batch(
-            project_client=create_async_project_client(),
            input_file=args.input,
            output_file=args.output,
            agent_model=args.agent_model,
@@ -528,10 +457,10 @@ async def main():
            env_file=args.env_file,
            limit=args.limit,
        )
-        print("\n✓ Processing complete!")
+        print("\n[PASS] Processing complete!")

    except Exception as e:
-        print(f"\n✗ Error: {str(e)}")
+        print(f"\n[FAIL] Error: {str(e)}")
        return 1
    return 0