mirror of
https://github.com/microsoft/agent-framework.git
synced 2026-06-16 21:04:09 +08:00
Python: Foundry Evals integration for Python (#4750)
* Foundry Evals integration for Python Merged and refactored eval module per Eduard's PR review: - Merge _eval.py + _local_eval.py into single _evaluation.py - Convert EvalItem from dataclass to regular class - Rename to_dict() to to_eval_data() - Convert _AgentEvalData to TypedDict - Simplify check system: unified async pattern with isawaitable - Parallelize checks and evaluators with asyncio.gather - Add all/any mode to tool_called_check - Fix bool(passed) truthy bug in _coerce_result - Remove deprecated function_evaluator/async_function_evaluator aliases - Remove _MinimalAgent, tighten evaluate_agent signature - Set self.name in __init__ (LocalEvaluator, FoundryEvals) - Limit FoundryEvals to AsyncOpenAI only - Type project_client as AIProjectClient - Remove NotImplementedError continuous eval code - Add evaluation samples in 02-agents/ and 03-workflows/ - Update all imports and tests (167 passing) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: resolve mypy redundant-cast errors while keeping pyright happy Use cast(list[Any], x) with type: ignore[redundant-cast] comments to satisfy both mypy (which considers casting Any redundant) and pyright strict mode (which needs explicit casts to narrow Unknown types). Also fix evaluator decorator check_name type annotation to be explicitly str, resolving mypy str|Any|None mismatch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: CI failures — pyupgrade, evaluator overloads, sample API, reset attr - Apply pyupgrade: Sequence from collections.abc, remove forward-ref quotes - Add @overload signatures to evaluator() for proper @evaluator usage - Fix evaluate_workflow sample to use WorkflowBuilder(start_executor=) API - Fix _workflow.py executor.reset() to use getattr pattern for pyright - Remove unused EvalResults forward-ref string in default_factory lambda Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: skip gRPC-dependent observability test The test_configure_otel_providers_with_env_file_and_vs_code_port test triggers gRPC OTLP exporter creation, but the grpc dependency is optional and not installed by default. Add skipif decorator matching the pattern used by all other gRPC exporter tests in the same file. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: add nosec B101 for bandit assert check Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * style: align eval samples with repo conventions - Move module docstrings before imports (after copyright header) - Add -> None return type to all main() and helper functions - Fix line-too-long in multiturn sample conversation data - Add Workflow import for typed return in all_patterns_sample Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review feedback: async fixes, sample bugs, deprecation warnings - Simplify _ensure_async_result to direct await (async-only clients) - Replace get_event_loop() with get_running_loop() - Narrow _fetch_output_items exception handling to specific types - Add warning log when _filter_tool_evaluators falls back to defaults - Add DeprecationWarning to options alias in Agent.__init__ - Add DeprecationWarning to evaluate_response() - Rename raw key to _raw_arguments in convert_message fallback - Fix evaluate_agent_sample.py: replace evals.select() with FoundryEvals() - Fix evaluate_multiturn_sample.py: use Message/Content/FunctionTool types - Fix evaluate_workflow_sample.py: replace evals.select() with FoundryEvals() - Update test mocks to use AsyncMock for awaited API calls Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add test coverage for review feedback items - Add num_repetitions=2 positive test verifying 2×items and 4 agent calls - Add _poll_eval_run tests: timeout, failed, and canceled paths - Add evaluate_traces tests: validation error, response_ids path, trace_ids path - Add evaluate_foundry_target happy-path test with target/query verification Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix ruff ISC004 lint error and apply formatter - Wrap implicit string concatenation in parens in evaluate_multiturn_sample.py - Apply ruff formatter to 6 other files with minor formatting drift Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove core type changes (extracted to fix/workflow-stale-session branch) Reverts changes to _agents.py, _agent_executor.py, and _workflow.py back to upstream/main. These fixes are now in a separate PR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review round 2: bugs, tests, and architecture Code fixes: - Fix _normalize_queries inverted condition (single query now replicates to match expected_count) - Fix substring match bug: 'end' in 'backend' matched; use exact set lookup for executor ID filtering - Fix used_available_tools sample: tool_definitions→tools param, use FunctionTool attribute access instead of dict .get() - Add None-check in _resolve_openai_client for misconfigured project - Add Returns section to evaluate_workflow docstring - Cache inspect.signature in @evaluator wrapper (avoid per-item reflection) Architecture: - Extract _evaluate_via_responses as module-level helper; evaluate_traces now calls it directly instead of creating a FoundryEvals instance - Move Foundry-specific typed-content conversion out of core to_eval_data; core now returns plain role/content dicts, FoundryEvals applies AgentEvalConverter in _evaluate_via_dataset Tests: - evaluate_response() deprecation warning emission and delegation - num_repetitions > 1 with expected_output and expected_tool_calls - Mock output_items.list in test_evaluate_calls_evals_api - Update to_eval_data assertions for plain-dict format - Unknown param error now raised at @evaluator decoration time Skipped (separate PR): executor reset loop, xfail removal, options alias Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix CI: revert test_full_conversation, fix pyright errors - Revert test_full_conversation.py to upstream/main (the session preservation test was incorrectly changed to assert clearing) - Fix pyright reportUnnecessaryComparison on get_openai_client() None check by adding ignore comment - Fix pyright reportPrivateUsage: add public EvalItem.split_messages() method and use it in FoundryEvals._evaluate_via_dataset instead of accessing private _split_conversation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review round 3: reliability, test gaps, cleanup - Add try/except guard for non-numeric score in _coerce_result - Add poll_interval minimum bound (0.1s) to prevent tight loops - Add runtime async client check in _resolve_openai_client - Remove _ensure_async_result wrapper (10 call sites → direct await) - Better error message when queries provided without agent - Import-time asserts for evaluator set consistency - Remove 28 redundant @pytest.mark.asyncio decorators - Add doc note about _raw_arguments sensitive data - Tests: tool_called_check mode=any, _normalize_queries branches, _extract_result_counts paths, _extract_per_evaluator, bare check via evaluate_agent, output_items assertion, modulo wrapping, async client check, queries-without-agent error Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix CI: ruff S101 assert, pyright and mypy arg-type errors - Replace module-level assert with if/raise for evaluator set consistency checks (ruff S101 disallows bare assert) - Add type: ignore[arg-type] and pyright: ignore[reportArgumentType] on OpenAI SDK evals API calls that pass dicts where typed params are expected (SDK accepts dicts at runtime) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review round 4: bugs, reliability, test fixes - Fix all_passed ignoring parent result_counts when sub_results present - Fix _extract_tool_calls: parse string arguments via json.loads before falling back to None (real LLM responses use string arguments) - Sanitize _raw_arguments to '[unparseable]' to avoid leaking sensitive tool-call data to external evaluation services - Add NOTE comment on to_eval_data message serialization dropping non-text content (tool calls, results) - Eliminate double conversation split in _evaluate_via_dataset: build JSONL dicts directly from split_messages + AgentEvalConverter - Raise poll_interval floor from 0.1s to 1.0s to prevent rate-limit exhaustion - Fix MagicMock(name=...) bug in test: sets display name not .name attr - Fix mock_output_item.sample: use MagicMock object instead of dict so _fetch_output_items exercises error/usage/input/output extraction Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review round 5: reliability, docs, test coverage Code fixes: - Move import-time RuntimeError checks to unit tests (avoids breaking imports for all users on developer set-drift mistake) - _filter_tool_evaluators now raises ValueError when all evaluators require tools but no items have tools (was silently substituting) - Add poll_interval upper bound (60s) to prevent single-iteration sleep - Log exc_info=True in _fetch_output_items for debugging API changes - Fix evaluate() docstring: remove claim about Responses API optimization - Validate target dict has 'type' key in evaluate_foundry_target - Document to_eval_data() limitation: non-text content is omitted Tests: - TestEvaluatorSetConsistency: verify _AGENT/_TOOL subsets of _BUILTIN - TestEvaluateTracesAgentId: agent_id-only path with lookback_hours - TestFilterToolEvaluatorsRaises: ValueError on all-tool no-items - TestEvaluateFoundryTargetValidation: target without 'type' key - Assert items==[] on failed/canceled poll results - Mock output_items.list in response_ids test for full flow - TestAllPassedSubResults: result_counts=None + sub_results delegation and parent failures override sub_results - TestBuildOverallItemEmpty: empty workflow outputs returns None Skipped r5-07 (_raw_arguments length hint): marginal debugging value, could leak content size information. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix error message: evaluate_responses() → evaluate_traces(response_ids=...) The referenced function doesn't exist; the correct API is evaluate_traces(response_ids=...) from the azure-ai package. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove dead to_eval_data() method, fix docstring claims - Remove to_eval_data() from EvalItem (dead code after r4-05 JSONL refactor) - Migrate 15 tests from to_eval_data() to split_messages() - Update sample to use split_messages() + Message properties - Remove unimplemented Responses API optimization docstring claim - Update split_messages() docstring to not reference removed method Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Reduce default eval timeout from 600s to 180s (3 minutes) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove dead _evaluate_via_responses method from FoundryEvals The method was never called — evaluate() uses _evaluate_via_dataset, and evaluate_traces() calls _evaluate_via_responses_impl directly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert unrelated formatting changes to get-started samples Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix pyright: remove phantom FoundryMemoryProvider import, apply ruff format - Remove import of non-existent _foundry_memory_provider module (incorrectly kept during rebase conflict resolution) - Apply ruff formatter to test_local_eval.py and get-started samples Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix eval samples: use FoundryChatClient for Agent() The upstream provider-leading client refactor (#4818) made client= a required parameter on Agent(). Update the three getting-started eval samples to use FoundryChatClient with FOUNDRY_PROJECT_ENDPOINT, matching the standard pattern from 01-get-started samples. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Simplify self-reflection sample using FoundryEvals Replace ~80 lines of manual OpenAI evals API code (create_eval, run_eval, manual polling, raw JSONL params) with FoundryEvals: - evaluate_groundedness() uses FoundryEvals.evaluate() with EvalItem - Remove create_openai_client(), create_eval(), run_eval() functions - Remove openai SDK type imports (DataSourceConfigCustom, etc.) - run_self_reflection_batch creates FoundryEvals instance once, reuses it for all iterations across all prompts Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update eval samples to FoundryChatClient and FOUNDRY_PROJECT_ENDPOINT - Migrate all foundry_evals samples from AzureOpenAIResponsesClient to FoundryChatClient - Update env var from AZURE_AI_PROJECT_ENDPOINT to FOUNDRY_PROJECT_ENDPOINT - Use AzureCliCredential consistently across all samples - Fix README.md: correct function names (evaluate_dataset -> FoundryEvals.evaluate, evaluate_responses -> evaluate_traces) - Update self_reflection .env.example and README.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix lint errors in eval samples (E501, ASYNC240, formatting) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove evaluate_all_patterns_sample.py (redundant with focused samples) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix async credential mismatch: use azure.identity.aio for async AIProjectClient AIProjectClient from azure.ai.projects.aio requires an async credential. Switch all foundry_evals samples from azure.identity.AzureCliCredential to azure.identity.aio.AzureCliCredential. Also pass project_client to FoundryChatClient instead of duplicating endpoint+credential. Close credential in self_reflection sample to avoid resource leak. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert test_observability.py to upstream/main (not our test) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address moonbox3 review: sphinx docstrings, pagination, isinstance check - Convert all Example:: / Typical usage:: code blocks to .. code-block:: python format matching codebase convention (both _evaluation.py and _foundry_evals.py) - Add async pagination in _fetch_output_items via async for (handles large result sets) - Replace hasattr(__aenter__) with isinstance(client, AsyncOpenAI) in _resolve_openai_client - Move AsyncOpenAI import from TYPE_CHECKING to runtime (needed for isinstance) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix test failures and address remaining moonbox3 review comments - Fix tests: use MagicMock(spec=AsyncOpenAI) for project_client mocks (isinstance check now requires proper type, not duck-typing) - Fix tests: replace mock_page.__iter__ with _AsyncPage helper for async for - Fix evaluate_response: auto-extract queries from response messages when query is not provided (previously always raised ValueError) - Add debug logging when skipping internal _-prefixed executor IDs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address Tao's PR review comments on Foundry Evals - T1: Add comment explaining builtin.* pass-through in _resolve_evaluator - T2: Add comment referencing OpenAI evals API for testing_criteria dict - T3: Document Mustache-style {{item.*}} template placeholders - T4: Document poll loop 60s sleep upper bound rationale - T5: Narrow run type to RunRetrieveResponse, use typed field access instead of vars()/getattr dance in _extract_result_counts and _extract_per_evaluator; use run.error and run.report_url directly - T6: Clarify openai_client docstring re: Azure Foundry endpoint - T8: Remove misleading empty expected_tool_calls from sample - Update tests to match real SDK PerTestingCriteriaResult shape Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove unnecessary Any union from run type annotations RunRetrieveResponse is the correct type — no backward compat needed for a brand new feature. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Accept FoundryChatClient instead of raw AsyncOpenAI FoundryEvals now takes client: FoundryChatClient as its primary parameter instead of openai_client: AsyncOpenAI. The builtin.* evaluators require a Foundry endpoint, so the type should reflect that. - FoundryEvals.__init__: client: FoundryChatClient replaces openai_client - evaluate_traces / evaluate_foundry_target: same change - _resolve_openai_client: extracts .client from FoundryChatClient - project_client fallback retained for standalone functions - All samples updated to construct FoundryChatClient and pass as client= - Tests updated (openai_client= → client=) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove implicit 60s upper bound on poll interval If a developer sets a higher poll_interval, respect it. Only clamp to remaining time and enforce a 1s minimum for rate-limit protection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove 1s floor on poll interval — let the developer control it Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update python/samples/05-end-to-end/evaluation/foundry_evals/.env.example Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com> * Update python/samples/02-agents/evaluation/evaluate_agent.py Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com> * Address eavanvalkenburg review (round 2) on Python eval PR - Rename model_deployment -> model across FoundryEvals and all samples - Make model param optional, resolves from client.model - Convert EvalResults from dataclass to regular class - Remove deprecated evaluate_response() function - Refactor splitters: BUILT_IN_SPLITTERS dict + standalone functions - Change per_turn_items from classmethod to staticmethod - Simplify EvalCheck type alias to use Awaitable[CheckResult] - Remove errored property from EvalResults - Remove default value from Evaluator protocol eval_name - Rename assert_passed -> raise_for_status, add EvalNotPassedError - Type agent param as SupportsAgentRun | None - Fix Arguments docstring - Update __init__.py exports - Update all tests and samples Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Move FoundryEvals to foundry package, split tool eval sample - Move _foundry_evals.py from azure-ai to foundry package - Move test_foundry_evals.py to foundry/tests/ - Update lazy re-exports in agent_framework.foundry namespace - Update .pyi type stubs - All samples now import from agent_framework.foundry - Split tool-call evaluation into evaluate_tool_calls_sample.py - Fix all_passed to check errored count from result_counts - Fix raise_for_status to include errored item details Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Auto-create FoundryChatClient from env vars when no client provided FoundryEvals() now works zero-config when FOUNDRY_PROJECT_ENDPOINT and FOUNDRY_MODEL environment variables are set. Auto-creates a FoundryChatClient under the hood, matching the established env var pattern. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix pyright errors: remove dead _normalize_queries, suppress EvalAPIError check - Remove unused _normalize_queries function and its tests - Add pyright ignore for EvalAPIError None check (defensive guard) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Support multimodal image content in eval pipeline Add image (data/uri) content handling to AgentEvalConverter.convert_message() so that Content.from_data() and Content.from_uri() image payloads are preserved as input_image parts in the Foundry evaluator format. - Handle Content type='data' and type='uri' → emit input_image parts - Add 6 unit tests for image content through convert_message/convert_messages - Add integration test verifying images flow through EvalItem → JSONL path - Add evaluate_multimodal.py sample demonstrating local image eval Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address remaining review comments - Fix project_client docstring to say async-only (not sync/async) - Add builtin evaluator name validation warning in _resolve_evaluator - Replace getattr with typed attribute access in _poll_eval_run, _extract_result_counts, _extract_per_evaluator, _fetch_output_items - Remove cast import from _foundry_evals (no longer needed) - Tighten _coerce_result: honour explicit 'passed' when both 'score' and 'passed' are present; remove performative cast - Fix self_reflection sample: add env file existence check - Fix traces sample: correct Pattern 2 section label - Update all Foundry eval samples to FoundryChatClient + FOUNDRY_MODEL (remove AIProjectClient + AZURE_AI_MODEL_DEPLOYMENT_NAME pattern) - Add eval_name and OpenAI client docs to FoundryEvals docstring - Update test mocks to match typed SDK objects (_MockResultCounts) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix ruff lint errors (E501, SIM108, SIM102) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix pyright errors: type-narrow dict to dict[str, Any], add ignore comments Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Replace ConversationSplitter type alias with Protocol ConversationSplitter is now a runtime-checkable Protocol with a named 'conversation' parameter, making the expected signature self-documenting. ConversationSplit enum members gain a __call__ method so they satisfy the protocol directly -- ConversationSplit.LAST_TURN(conversation) works. This simplifies _split_conversation from an isinstance dispatch to a single split(conversation) call. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Standardize on AZURE_AI_MODEL_DEPLOYMENT_NAME and fix Unicode in samples - Replace FOUNDRY_MODEL with AZURE_AI_MODEL_DEPLOYMENT_NAME in all eval samples to match repo convention - Replace Unicode symbols with ASCII equivalents in all eval sample print statements to avoid cp1252 encoding errors on Windows Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update python/samples/03-workflows/evaluation/evaluate_workflow.py Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com> * Rename ADR 0020 to 0023 (foundry evals integration) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: alliscode <bentho@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com>
This commit is contained in:
Ben Thomas
committed by
GitHub
Unverified
GitHub
parent
3f964c4cdb
commit
35adfdb318
@@ -57,6 +57,27 @@ from ._compaction import (
|
||||
included_messages,
|
||||
included_token_count,
|
||||
)
|
||||
from ._evaluation import (
|
||||
AgentEvalConverter,
|
||||
CheckResult,
|
||||
ConversationSplit,
|
||||
ConversationSplitter,
|
||||
EvalItem,
|
||||
EvalItemResult,
|
||||
EvalNotPassedError,
|
||||
EvalResults,
|
||||
EvalScoreResult,
|
||||
Evaluator,
|
||||
ExpectedToolCall,
|
||||
LocalEvaluator,
|
||||
evaluate_agent,
|
||||
evaluate_workflow,
|
||||
evaluator,
|
||||
keyword_check,
|
||||
tool_call_args_match,
|
||||
tool_called_check,
|
||||
tool_calls_present,
|
||||
)
|
||||
from ._mcp import MCPStdioTool, MCPStreamableHTTPTool, MCPWebsocketTool
|
||||
from ._middleware import (
|
||||
AgentContext,
|
||||
@@ -242,6 +263,7 @@ __all__ = [
|
||||
"USER_AGENT_TELEMETRY_DISABLED_ENV_VAR",
|
||||
"Agent",
|
||||
"AgentContext",
|
||||
"AgentEvalConverter",
|
||||
"AgentExecutor",
|
||||
"AgentExecutorRequest",
|
||||
"AgentExecutorResponse",
|
||||
@@ -268,11 +290,14 @@ __all__ = [
|
||||
"ChatOptions",
|
||||
"ChatResponse",
|
||||
"ChatResponseUpdate",
|
||||
"CheckResult",
|
||||
"CheckpointStorage",
|
||||
"CompactionProvider",
|
||||
"CompactionStrategy",
|
||||
"Content",
|
||||
"ContinuationToken",
|
||||
"ConversationSplit",
|
||||
"ConversationSplitter",
|
||||
"Default",
|
||||
"Edge",
|
||||
"EdgeCondition",
|
||||
@@ -281,7 +306,14 @@ __all__ = [
|
||||
"EmbeddingGenerationOptions",
|
||||
"EmbeddingInputT",
|
||||
"EmbeddingT",
|
||||
"EvalItem",
|
||||
"EvalItemResult",
|
||||
"EvalNotPassedError",
|
||||
"EvalResults",
|
||||
"EvalScoreResult",
|
||||
"Evaluator",
|
||||
"Executor",
|
||||
"ExpectedToolCall",
|
||||
"FanInEdgeGroup",
|
||||
"FanOutEdgeGroup",
|
||||
"FileCheckpointStorage",
|
||||
@@ -300,6 +332,7 @@ __all__ = [
|
||||
"InMemoryCheckpointStorage",
|
||||
"InMemoryHistoryProvider",
|
||||
"InProcRunnerContext",
|
||||
"LocalEvaluator",
|
||||
"MCPStdioTool",
|
||||
"MCPStreamableHTTPTool",
|
||||
"MCPWebsocketTool",
|
||||
@@ -379,11 +412,15 @@ __all__ = [
|
||||
"chat_middleware",
|
||||
"create_edge_runner",
|
||||
"detect_media_type_from_base64",
|
||||
"evaluate_agent",
|
||||
"evaluate_workflow",
|
||||
"evaluator",
|
||||
"executor",
|
||||
"function_middleware",
|
||||
"handler",
|
||||
"included_messages",
|
||||
"included_token_count",
|
||||
"keyword_check",
|
||||
"load_settings",
|
||||
"map_chat_to_agent_update",
|
||||
"merge_chat_options",
|
||||
@@ -396,6 +433,9 @@ __all__ = [
|
||||
"resolve_agent_id",
|
||||
"response_handler",
|
||||
"tool",
|
||||
"tool_call_args_match",
|
||||
"tool_called_check",
|
||||
"tool_calls_present",
|
||||
"validate_chat_options",
|
||||
"validate_tool_mode",
|
||||
"validate_tools",
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -12,6 +12,7 @@ _IMPORTS: dict[str, tuple[str, str]] = {
|
||||
"FoundryAgent": ("agent_framework_foundry", "agent-framework-foundry"),
|
||||
"FoundryChatClient": ("agent_framework_foundry", "agent-framework-foundry"),
|
||||
"FoundryChatOptions": ("agent_framework_foundry", "agent-framework-foundry"),
|
||||
"FoundryEvals": ("agent_framework_foundry", "agent-framework-foundry"),
|
||||
"FoundryMemoryProvider": ("agent_framework_foundry", "agent-framework-foundry"),
|
||||
"FoundryLocalChatOptions": ("agent_framework_foundry_local", "agent-framework-foundry-local"),
|
||||
"FoundryLocalClient": ("agent_framework_foundry_local", "agent-framework-foundry-local"),
|
||||
@@ -19,6 +20,8 @@ _IMPORTS: dict[str, tuple[str, str]] = {
|
||||
"RawFoundryAgent": ("agent_framework_foundry", "agent-framework-foundry"),
|
||||
"RawFoundryAgentChatClient": ("agent_framework_foundry", "agent-framework-foundry"),
|
||||
"RawFoundryChatClient": ("agent_framework_foundry", "agent-framework-foundry"),
|
||||
"evaluate_foundry_target": ("agent_framework_foundry", "agent-framework-foundry"),
|
||||
"evaluate_traces": ("agent_framework_foundry", "agent-framework-foundry"),
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -7,10 +7,13 @@ from agent_framework_foundry import (
|
||||
FoundryAgent,
|
||||
FoundryChatClient,
|
||||
FoundryChatOptions,
|
||||
FoundryEvals,
|
||||
FoundryMemoryProvider,
|
||||
RawFoundryAgent,
|
||||
RawFoundryAgentChatClient,
|
||||
RawFoundryChatClient,
|
||||
evaluate_foundry_target,
|
||||
evaluate_traces,
|
||||
)
|
||||
from agent_framework_foundry_local import (
|
||||
FoundryLocalChatOptions,
|
||||
@@ -22,6 +25,7 @@ __all__ = [
|
||||
"FoundryAgent",
|
||||
"FoundryChatClient",
|
||||
"FoundryChatOptions",
|
||||
"FoundryEvals",
|
||||
"FoundryLocalChatOptions",
|
||||
"FoundryLocalClient",
|
||||
"FoundryLocalSettings",
|
||||
@@ -29,4 +33,6 @@ __all__ = [
|
||||
"RawFoundryAgent",
|
||||
"RawFoundryAgentChatClient",
|
||||
"RawFoundryChatClient",
|
||||
"evaluate_foundry_target",
|
||||
"evaluate_traces",
|
||||
]
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -4,6 +4,11 @@ import importlib.metadata
|
||||
|
||||
from ._agent import FoundryAgent, RawFoundryAgent, RawFoundryAgentChatClient
|
||||
from ._chat_client import FoundryChatClient, FoundryChatOptions, RawFoundryChatClient
|
||||
from ._foundry_evals import (
|
||||
FoundryEvals,
|
||||
evaluate_foundry_target,
|
||||
evaluate_traces,
|
||||
)
|
||||
from ._memory_provider import FoundryMemoryProvider
|
||||
|
||||
try:
|
||||
@@ -15,9 +20,12 @@ __all__ = [
|
||||
"FoundryAgent",
|
||||
"FoundryChatClient",
|
||||
"FoundryChatOptions",
|
||||
"FoundryEvals",
|
||||
"FoundryMemoryProvider",
|
||||
"RawFoundryAgent",
|
||||
"RawFoundryAgentChatClient",
|
||||
"RawFoundryChatClient",
|
||||
"__version__",
|
||||
"evaluate_foundry_target",
|
||||
"evaluate_traces",
|
||||
]
|
||||
|
||||
@@ -0,0 +1,891 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
"""Microsoft Foundry Evals integration for Microsoft Agent Framework.
|
||||
|
||||
Provides ``FoundryEvals``, an ``Evaluator`` implementation backed by Azure AI
|
||||
Foundry's built-in evaluators. See docs/decisions/0018-foundry-evals-integration.md
|
||||
for the design rationale.
|
||||
|
||||
Example:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from agent_framework import evaluate_agent
|
||||
from agent_framework.foundry import FoundryEvals
|
||||
|
||||
# Zero-config: reads FOUNDRY_PROJECT_ENDPOINT and FOUNDRY_MODEL from env
|
||||
evals = FoundryEvals()
|
||||
results = await evaluate_agent(
|
||||
agent=my_agent,
|
||||
queries=["What's the weather in Seattle?"],
|
||||
evaluators=evals,
|
||||
)
|
||||
results[0].raise_for_status()
|
||||
print(results[0].report_url)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from collections.abc import Sequence
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from agent_framework._evaluation import (
|
||||
AgentEvalConverter,
|
||||
ConversationSplit,
|
||||
ConversationSplitter,
|
||||
EvalItem,
|
||||
EvalItemResult,
|
||||
EvalResults,
|
||||
EvalScoreResult,
|
||||
)
|
||||
from openai import AsyncOpenAI
|
||||
|
||||
from ._chat_client import FoundryChatClient
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from azure.ai.projects.aio import AIProjectClient
|
||||
from openai.types.evals import RunRetrieveResponse
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Agent evaluators that accept query/response as conversation arrays.
|
||||
# Maintained manually — check https://learn.microsoft.com/en-us/azure/ai-studio/how-to/develop/evaluate-sdk
|
||||
# for the latest evaluator list. These are the evaluators that need conversation-format input.
|
||||
_AGENT_EVALUATORS: set[str] = {
|
||||
"builtin.intent_resolution",
|
||||
"builtin.task_adherence",
|
||||
"builtin.task_completion",
|
||||
"builtin.task_navigation_efficiency",
|
||||
"builtin.tool_call_accuracy",
|
||||
"builtin.tool_selection",
|
||||
"builtin.tool_input_accuracy",
|
||||
"builtin.tool_output_utilization",
|
||||
"builtin.tool_call_success",
|
||||
}
|
||||
|
||||
# Evaluators that additionally require tool_definitions.
|
||||
_TOOL_EVALUATORS: set[str] = {
|
||||
"builtin.tool_call_accuracy",
|
||||
"builtin.tool_selection",
|
||||
"builtin.tool_input_accuracy",
|
||||
"builtin.tool_output_utilization",
|
||||
"builtin.tool_call_success",
|
||||
}
|
||||
|
||||
_BUILTIN_EVALUATORS: dict[str, str] = {
|
||||
# Agent behavior
|
||||
"intent_resolution": "builtin.intent_resolution",
|
||||
"task_adherence": "builtin.task_adherence",
|
||||
"task_completion": "builtin.task_completion",
|
||||
"task_navigation_efficiency": "builtin.task_navigation_efficiency",
|
||||
# Tool usage
|
||||
"tool_call_accuracy": "builtin.tool_call_accuracy",
|
||||
"tool_selection": "builtin.tool_selection",
|
||||
"tool_input_accuracy": "builtin.tool_input_accuracy",
|
||||
"tool_output_utilization": "builtin.tool_output_utilization",
|
||||
"tool_call_success": "builtin.tool_call_success",
|
||||
# Quality
|
||||
"coherence": "builtin.coherence",
|
||||
"fluency": "builtin.fluency",
|
||||
"relevance": "builtin.relevance",
|
||||
"groundedness": "builtin.groundedness",
|
||||
"response_completeness": "builtin.response_completeness",
|
||||
"similarity": "builtin.similarity",
|
||||
# Safety
|
||||
"violence": "builtin.violence",
|
||||
"sexual": "builtin.sexual",
|
||||
"self_harm": "builtin.self_harm",
|
||||
"hate_unfairness": "builtin.hate_unfairness",
|
||||
}
|
||||
|
||||
# Default evaluator sets used when evaluators=None
|
||||
_DEFAULT_EVALUATORS: list[str] = [
|
||||
"relevance",
|
||||
"coherence",
|
||||
"task_adherence",
|
||||
]
|
||||
|
||||
_DEFAULT_TOOL_EVALUATORS: list[str] = [
|
||||
"tool_call_accuracy",
|
||||
]
|
||||
|
||||
# Consistency between evaluator sets is enforced by tests in
|
||||
# test_foundry_evals.py — see TestEvaluatorSetConsistency.
|
||||
|
||||
|
||||
def _resolve_evaluator(name: str) -> str:
|
||||
"""Resolve a short evaluator name to its fully-qualified ``builtin.*`` form.
|
||||
|
||||
Args:
|
||||
name: Short name (e.g. ``"relevance"``) or fully-qualified name
|
||||
(e.g. ``"builtin.relevance"``).
|
||||
|
||||
Returns:
|
||||
The fully-qualified evaluator name.
|
||||
|
||||
Raises:
|
||||
ValueError: If the name is not recognized.
|
||||
"""
|
||||
if name.startswith("builtin."):
|
||||
# Already fully-qualified — pass through, but warn if not in our
|
||||
# known list (may indicate a typo or a newly-added evaluator).
|
||||
short = name.removeprefix("builtin.")
|
||||
if short not in _BUILTIN_EVALUATORS:
|
||||
logger.warning(
|
||||
"Evaluator '%s' is not in the known built-in list. "
|
||||
"If this is a new evaluator, consider updating _BUILTIN_EVALUATORS.",
|
||||
name,
|
||||
)
|
||||
return name
|
||||
resolved = _BUILTIN_EVALUATORS.get(name)
|
||||
if resolved is None:
|
||||
raise ValueError(f"Unknown evaluator '{name}'. Available: {sorted(_BUILTIN_EVALUATORS)}")
|
||||
return resolved
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Internal helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _build_testing_criteria(
|
||||
evaluators: Sequence[str],
|
||||
model: str,
|
||||
*,
|
||||
include_data_mapping: bool = False,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Build ``testing_criteria`` for ``evals.create()``.
|
||||
|
||||
Args:
|
||||
evaluators: Evaluator names.
|
||||
model: Model deployment for the LLM judge.
|
||||
include_data_mapping: Whether to include field-level data mapping
|
||||
(required for the JSONL data source, not needed for response-based).
|
||||
"""
|
||||
criteria: list[dict[str, Any]] = []
|
||||
for name in evaluators:
|
||||
qualified = _resolve_evaluator(name)
|
||||
short = name if not name.startswith("builtin.") else name.split(".")[-1]
|
||||
|
||||
# Structure dictated by the OpenAI evals API — see
|
||||
# https://platform.openai.com/docs/api-reference/evals/create
|
||||
entry: dict[str, Any] = {
|
||||
"type": "azure_ai_evaluator",
|
||||
"name": short,
|
||||
"evaluator_name": qualified,
|
||||
"initialization_parameters": {"deployment_name": model},
|
||||
}
|
||||
|
||||
if include_data_mapping:
|
||||
if qualified in _AGENT_EVALUATORS:
|
||||
# Agent evaluators: query/response as conversation arrays.
|
||||
# {{item.*}} are Mustache-style placeholders resolved by the
|
||||
# evals API against fields in the JSONL data items.
|
||||
mapping: dict[str, str] = {
|
||||
"query": "{{item.query_messages}}",
|
||||
"response": "{{item.response_messages}}",
|
||||
}
|
||||
else:
|
||||
# Quality evaluators: query/response as strings
|
||||
mapping = {
|
||||
"query": "{{item.query}}",
|
||||
"response": "{{item.response}}",
|
||||
}
|
||||
if qualified == "builtin.groundedness":
|
||||
mapping["context"] = "{{item.context}}"
|
||||
if qualified in _TOOL_EVALUATORS:
|
||||
mapping["tool_definitions"] = "{{item.tool_definitions}}"
|
||||
entry["data_mapping"] = mapping
|
||||
|
||||
criteria.append(entry)
|
||||
return criteria
|
||||
|
||||
|
||||
def _build_item_schema(*, has_context: bool = False, has_tools: bool = False) -> dict[str, Any]:
|
||||
"""Build the ``item_schema`` for custom JSONL eval definitions."""
|
||||
properties: dict[str, Any] = {
|
||||
"query": {"type": "string"},
|
||||
"response": {"type": "string"},
|
||||
"query_messages": {"type": "array"},
|
||||
"response_messages": {"type": "array"},
|
||||
}
|
||||
if has_context:
|
||||
properties["context"] = {"type": "string"}
|
||||
if has_tools:
|
||||
properties["tool_definitions"] = {"type": "array"}
|
||||
return {
|
||||
"type": "object",
|
||||
"properties": properties,
|
||||
"required": ["query", "response"],
|
||||
}
|
||||
|
||||
|
||||
def _resolve_default_evaluators(
|
||||
evaluators: Sequence[str] | None,
|
||||
items: Sequence[EvalItem | dict[str, Any]] | None = None,
|
||||
) -> list[str]:
|
||||
"""Resolve evaluators, applying defaults when ``None``.
|
||||
|
||||
Defaults to relevance + coherence + task_adherence. Automatically adds
|
||||
tool_call_accuracy when items contain tools.
|
||||
"""
|
||||
if evaluators is not None:
|
||||
return list(evaluators)
|
||||
|
||||
result = list(_DEFAULT_EVALUATORS)
|
||||
if items is not None:
|
||||
has_tools = any((item.tools if isinstance(item, EvalItem) else item.get("tool_definitions")) for item in items)
|
||||
if has_tools:
|
||||
result.extend(_DEFAULT_TOOL_EVALUATORS)
|
||||
return result
|
||||
|
||||
|
||||
def _filter_tool_evaluators(
|
||||
evaluators: list[str],
|
||||
items: Sequence[EvalItem | dict[str, Any]],
|
||||
) -> list[str]:
|
||||
"""Remove tool evaluators if no items have tool definitions."""
|
||||
has_tools = any((item.tools if isinstance(item, EvalItem) else item.get("tool_definitions")) for item in items)
|
||||
if has_tools:
|
||||
return evaluators
|
||||
filtered = [e for e in evaluators if _resolve_evaluator(e) not in _TOOL_EVALUATORS]
|
||||
if not filtered:
|
||||
raise ValueError(
|
||||
f"All requested evaluators {evaluators} require tool definitions, "
|
||||
"but no items have tools. Either add tool definitions to your items "
|
||||
"or choose evaluators that do not require tools."
|
||||
)
|
||||
if len(filtered) < len(evaluators):
|
||||
removed = [e for e in evaluators if _resolve_evaluator(e) in _TOOL_EVALUATORS]
|
||||
logger.info("Removed tool evaluators %s (no items have tools)", removed)
|
||||
return filtered
|
||||
|
||||
|
||||
async def _poll_eval_run(
|
||||
client: AsyncOpenAI,
|
||||
eval_id: str,
|
||||
run_id: str,
|
||||
poll_interval: float = 5.0,
|
||||
timeout: float = 180.0,
|
||||
provider: str = "Microsoft Foundry",
|
||||
*,
|
||||
fetch_output_items: bool = True,
|
||||
) -> EvalResults:
|
||||
"""Poll an eval run until completion or timeout."""
|
||||
loop = asyncio.get_running_loop()
|
||||
deadline = loop.time() + timeout
|
||||
while True:
|
||||
run = await client.evals.runs.retrieve(run_id=run_id, eval_id=eval_id)
|
||||
if run.status in ("completed", "failed", "canceled"):
|
||||
error_msg = None
|
||||
if run.status == "failed":
|
||||
err = run.error
|
||||
if err is not None: # pyright: ignore[reportUnnecessaryComparison]
|
||||
error_msg = err if isinstance(err, str) else err.message or str(err)
|
||||
|
||||
items: list[EvalItemResult] = []
|
||||
if fetch_output_items and run.status == "completed":
|
||||
items = await _fetch_output_items(client, eval_id, run_id)
|
||||
|
||||
return EvalResults(
|
||||
provider=provider,
|
||||
eval_id=eval_id,
|
||||
run_id=run_id,
|
||||
status=run.status,
|
||||
result_counts=_extract_result_counts(run),
|
||||
report_url=run.report_url,
|
||||
error=error_msg,
|
||||
per_evaluator=_extract_per_evaluator(run),
|
||||
items=items,
|
||||
)
|
||||
remaining = deadline - loop.time()
|
||||
if remaining <= 0:
|
||||
return EvalResults(provider=provider, eval_id=eval_id, run_id=run_id, status="timeout")
|
||||
logger.debug("Eval run %s status: %s (%.0fs remaining)", run_id, run.status, remaining)
|
||||
await asyncio.sleep(min(poll_interval, remaining))
|
||||
|
||||
|
||||
def _extract_result_counts(run: RunRetrieveResponse) -> dict[str, int] | None:
|
||||
"""Extract result_counts from an eval run as a plain dict."""
|
||||
counts = run.result_counts
|
||||
if counts is None: # pyright: ignore[reportUnnecessaryComparison]
|
||||
return None
|
||||
return {
|
||||
"errored": counts.errored,
|
||||
"failed": counts.failed,
|
||||
"passed": counts.passed,
|
||||
"total": counts.total,
|
||||
}
|
||||
|
||||
|
||||
def _extract_per_evaluator(run: RunRetrieveResponse) -> dict[str, dict[str, int]]:
|
||||
"""Extract per-evaluator result breakdowns from an eval run."""
|
||||
per_eval: dict[str, dict[str, int]] = {}
|
||||
for item in run.per_testing_criteria_results or []:
|
||||
name = item.testing_criteria
|
||||
if name:
|
||||
per_eval[name] = {"passed": item.passed, "failed": item.failed}
|
||||
return per_eval
|
||||
|
||||
|
||||
async def _fetch_output_items(
|
||||
client: AsyncOpenAI,
|
||||
eval_id: str,
|
||||
run_id: str,
|
||||
) -> list[EvalItemResult]:
|
||||
"""Fetch per-item results from the output_items API.
|
||||
|
||||
Converts the provider-specific ``OutputItemListResponse`` objects into
|
||||
provider-agnostic ``EvalItemResult`` instances with per-evaluator scores,
|
||||
error categorization, and token usage. Uses async pagination to handle
|
||||
eval runs with more items than a single page.
|
||||
"""
|
||||
items: list[EvalItemResult] = []
|
||||
try:
|
||||
output_items_page = await client.evals.runs.output_items.list(
|
||||
run_id=run_id,
|
||||
eval_id=eval_id,
|
||||
)
|
||||
|
||||
async for oi in output_items_page:
|
||||
# Extract per-evaluator scores
|
||||
scores: list[EvalScoreResult] = []
|
||||
for r in oi.results or []:
|
||||
scores.append(
|
||||
EvalScoreResult(
|
||||
name=r.name,
|
||||
score=r.score,
|
||||
passed=r.passed,
|
||||
sample=r.sample,
|
||||
)
|
||||
)
|
||||
|
||||
# Extract error info from sample
|
||||
error_code: str | None = None
|
||||
error_message: str | None = None
|
||||
token_usage: dict[str, int] | None = None
|
||||
input_text: str | None = None
|
||||
output_text: str | None = None
|
||||
response_id: str | None = None
|
||||
|
||||
sample = oi.sample
|
||||
if sample is not None: # pyright: ignore[reportUnnecessaryComparison]
|
||||
err = sample.error
|
||||
if err is not None and (err.code or err.message): # pyright: ignore[reportUnnecessaryComparison]
|
||||
error_code = err.code or None
|
||||
error_message = err.message or None
|
||||
|
||||
usage = sample.usage
|
||||
if usage is not None and usage.total_tokens: # pyright: ignore[reportUnnecessaryComparison]
|
||||
token_usage = {
|
||||
"prompt_tokens": usage.prompt_tokens,
|
||||
"completion_tokens": usage.completion_tokens,
|
||||
"total_tokens": usage.total_tokens,
|
||||
"cached_tokens": usage.cached_tokens,
|
||||
}
|
||||
|
||||
# Extract input/output text
|
||||
if sample.input:
|
||||
parts = [si.content for si in sample.input if si.role == "user"]
|
||||
if parts:
|
||||
input_text = " ".join(parts)
|
||||
|
||||
if sample.output:
|
||||
parts = [so.content or "" for so in sample.output if so.role == "assistant"]
|
||||
if parts:
|
||||
output_text = " ".join(parts)
|
||||
|
||||
# Extract response_id from datasource_item
|
||||
ds_item = oi.datasource_item
|
||||
if ds_item:
|
||||
resp_id_val = ds_item.get("resp_id") or ds_item.get("response_id")
|
||||
response_id = str(resp_id_val) if resp_id_val else None
|
||||
|
||||
items.append(
|
||||
EvalItemResult(
|
||||
item_id=oi.id,
|
||||
status=oi.status,
|
||||
scores=scores,
|
||||
error_code=error_code,
|
||||
error_message=error_message,
|
||||
response_id=response_id,
|
||||
input_text=input_text,
|
||||
output_text=output_text,
|
||||
token_usage=token_usage,
|
||||
)
|
||||
)
|
||||
except (AttributeError, KeyError, TypeError):
|
||||
logger.warning("Could not fetch output_items for run %s", run_id, exc_info=True)
|
||||
|
||||
return items
|
||||
|
||||
|
||||
def _resolve_openai_client(
|
||||
client: FoundryChatClient | AsyncOpenAI | None = None,
|
||||
project_client: AIProjectClient | None = None,
|
||||
) -> AsyncOpenAI:
|
||||
"""Resolve an AsyncOpenAI client from a FoundryChatClient, raw client, or project_client."""
|
||||
if client is not None:
|
||||
if isinstance(client, FoundryChatClient):
|
||||
return client.client
|
||||
return client
|
||||
if project_client is not None:
|
||||
oai = project_client.get_openai_client()
|
||||
if oai is None: # pyright: ignore[reportUnnecessaryComparison]
|
||||
raise ValueError("project_client.get_openai_client() returned None. Check project configuration.")
|
||||
if not isinstance(oai, AsyncOpenAI):
|
||||
raise TypeError(
|
||||
"project_client.get_openai_client() returned a sync client. "
|
||||
"FoundryEvals requires an async AIProjectClient (from azure.ai.projects.aio)."
|
||||
)
|
||||
return oai
|
||||
raise ValueError("Provide either 'client' or 'project_client'.")
|
||||
|
||||
|
||||
async def _evaluate_via_responses_impl(
|
||||
*,
|
||||
client: AsyncOpenAI,
|
||||
response_ids: Sequence[str],
|
||||
evaluators: list[str],
|
||||
model: str,
|
||||
eval_name: str,
|
||||
poll_interval: float,
|
||||
timeout: float,
|
||||
provider: str = "foundry",
|
||||
) -> EvalResults:
|
||||
"""Evaluate using Foundry's Responses API retrieval path.
|
||||
|
||||
Module-level helper used by both ``FoundryEvals`` and ``evaluate_traces``.
|
||||
"""
|
||||
eval_obj = await client.evals.create(
|
||||
name=eval_name,
|
||||
data_source_config={"type": "azure_ai_source", "scenario": "responses"}, # type: ignore[arg-type] # pyright: ignore[reportArgumentType]
|
||||
testing_criteria=_build_testing_criteria(evaluators, model), # type: ignore[arg-type] # pyright: ignore[reportArgumentType]
|
||||
)
|
||||
|
||||
data_source = {
|
||||
"type": "azure_ai_responses",
|
||||
"item_generation_params": {
|
||||
"type": "response_retrieval",
|
||||
"data_mapping": {"response_id": "{{item.resp_id}}"},
|
||||
"source": {
|
||||
"type": "file_content",
|
||||
"content": [{"item": {"resp_id": rid}} for rid in response_ids],
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
run = await client.evals.runs.create(
|
||||
eval_id=eval_obj.id,
|
||||
name=f"{eval_name} Run",
|
||||
data_source=data_source, # type: ignore[arg-type] # pyright: ignore[reportArgumentType]
|
||||
)
|
||||
|
||||
return await _poll_eval_run(client, eval_obj.id, run.id, poll_interval, timeout, provider=provider)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# FoundryEvals — Evaluator implementation for Microsoft Foundry
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class FoundryEvals:
|
||||
"""Evaluation provider backed by Microsoft Foundry.
|
||||
|
||||
Implements the ``Evaluator`` protocol so it can be passed to the
|
||||
provider-agnostic ``evaluate_agent()`` and
|
||||
``evaluate_workflow()`` functions from ``agent_framework``.
|
||||
|
||||
Also provides constants for built-in evaluator names for IDE
|
||||
autocomplete and typo prevention:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from agent_framework.foundry import FoundryEvals
|
||||
|
||||
evaluators = [FoundryEvals.RELEVANCE, FoundryEvals.TOOL_CALL_ACCURACY]
|
||||
|
||||
Examples:
|
||||
Basic usage:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from agent_framework import evaluate_agent
|
||||
from agent_framework.foundry import FoundryEvals, FoundryChatClient
|
||||
|
||||
chat_client = FoundryChatClient(model="gpt-4o")
|
||||
evals = FoundryEvals(client=chat_client)
|
||||
results = await evaluate_agent(agent=agent, queries=queries, evaluators=evals)
|
||||
|
||||
Zero-config with environment variables (``FOUNDRY_PROJECT_ENDPOINT``
|
||||
and ``FOUNDRY_MODEL``):
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
evals = FoundryEvals() # reads env vars via FoundryChatClient
|
||||
|
||||
**Evaluator selection:**
|
||||
|
||||
By default, runs ``relevance``, ``coherence``, and ``task_adherence``.
|
||||
Automatically adds ``tool_call_accuracy`` when items contain tool
|
||||
definitions. Override with ``evaluators=``.
|
||||
|
||||
.. note::
|
||||
|
||||
The ``builtin.*`` evaluators are accessed through the OpenAI Evals
|
||||
API (``client.evals.create`` / ``client.evals.runs.create``). Any
|
||||
``AsyncOpenAI`` client pointing at a Foundry endpoint can run them.
|
||||
|
||||
Args:
|
||||
client: A ``FoundryChatClient`` instance. The ``builtin.*``
|
||||
evaluators are a Foundry feature and require a Foundry endpoint.
|
||||
When omitted (and *project_client* is also omitted), a
|
||||
``FoundryChatClient`` is auto-created from ``FOUNDRY_PROJECT_ENDPOINT``
|
||||
and ``FOUNDRY_MODEL`` environment variables.
|
||||
project_client: An async ``AIProjectClient`` instance
|
||||
(from ``azure.ai.projects.aio``). Provide this or *client*.
|
||||
model: Model deployment name for the evaluator LLM judge.
|
||||
Resolved from ``client.model`` when omitted.
|
||||
evaluators: Evaluator names (e.g. ``["relevance", "tool_call_accuracy"]``).
|
||||
When ``None`` (default), uses smart defaults based on item data.
|
||||
conversation_split: How to split multi-turn conversations into
|
||||
query/response halves. Defaults to ``LAST_TURN``. Pass a
|
||||
``ConversationSplit`` enum value or a custom callable — see
|
||||
``ConversationSplitter``.
|
||||
poll_interval: Seconds between status polls (default 5.0).
|
||||
timeout: Maximum seconds to wait for completion (default 180.0).
|
||||
eval_name: Display name for the eval definition created in Foundry.
|
||||
Defaults to ``"agent-framework-eval"``. The name is visible in
|
||||
the Foundry portal; it does not affect evaluation behavior.
|
||||
"""
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Built-in evaluator name constants
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Agent behavior
|
||||
INTENT_RESOLUTION: str = "intent_resolution"
|
||||
TASK_ADHERENCE: str = "task_adherence"
|
||||
TASK_COMPLETION: str = "task_completion"
|
||||
TASK_NAVIGATION_EFFICIENCY: str = "task_navigation_efficiency"
|
||||
|
||||
# Tool usage
|
||||
TOOL_CALL_ACCURACY: str = "tool_call_accuracy"
|
||||
TOOL_SELECTION: str = "tool_selection"
|
||||
TOOL_INPUT_ACCURACY: str = "tool_input_accuracy"
|
||||
TOOL_OUTPUT_UTILIZATION: str = "tool_output_utilization"
|
||||
TOOL_CALL_SUCCESS: str = "tool_call_success"
|
||||
|
||||
# Quality
|
||||
COHERENCE: str = "coherence"
|
||||
FLUENCY: str = "fluency"
|
||||
RELEVANCE: str = "relevance"
|
||||
GROUNDEDNESS: str = "groundedness"
|
||||
RESPONSE_COMPLETENESS: str = "response_completeness"
|
||||
SIMILARITY: str = "similarity"
|
||||
|
||||
# Safety
|
||||
VIOLENCE: str = "violence"
|
||||
SEXUAL: str = "sexual"
|
||||
SELF_HARM: str = "self_harm"
|
||||
HATE_UNFAIRNESS: str = "hate_unfairness"
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
client: FoundryChatClient | None = None,
|
||||
project_client: AIProjectClient | None = None,
|
||||
model: str | None = None,
|
||||
evaluators: Sequence[str] | None = None,
|
||||
conversation_split: ConversationSplitter = ConversationSplit.LAST_TURN,
|
||||
poll_interval: float = 5.0,
|
||||
timeout: float = 180.0,
|
||||
):
|
||||
self.name = "Microsoft Foundry"
|
||||
|
||||
# Auto-create a FoundryChatClient from env vars when no client is provided
|
||||
if client is None and project_client is None:
|
||||
client = FoundryChatClient(model=model or "gpt-4o")
|
||||
|
||||
self._client = _resolve_openai_client(client, project_client)
|
||||
# Resolve model: explicit param > client.model > error
|
||||
resolved_model = model or (client.model if client is not None else None)
|
||||
if not resolved_model:
|
||||
raise ValueError(
|
||||
"Model is required. Pass model= explicitly or use a FoundryChatClient that has a model configured."
|
||||
)
|
||||
self._model = resolved_model
|
||||
self._evaluators = list(evaluators) if evaluators is not None else None
|
||||
self._conversation_split = conversation_split
|
||||
self._poll_interval = poll_interval
|
||||
self._timeout = timeout
|
||||
|
||||
async def evaluate(
|
||||
self,
|
||||
items: Sequence[EvalItem],
|
||||
*,
|
||||
eval_name: str = "Agent Framework Eval",
|
||||
) -> EvalResults:
|
||||
"""Evaluate items using Foundry evaluators.
|
||||
|
||||
Implements the ``Evaluator`` protocol. Automatically resolves default
|
||||
evaluators and filters tool evaluators for items without tool definitions.
|
||||
|
||||
Args:
|
||||
items: Eval data items from ``AgentEvalConverter.to_eval_item()``.
|
||||
eval_name: Display name for the evaluation run.
|
||||
|
||||
Returns:
|
||||
``EvalResults`` with status, counts, and portal link.
|
||||
"""
|
||||
# Resolve evaluators with auto-detection
|
||||
resolved = _resolve_default_evaluators(self._evaluators, items=items)
|
||||
# Filter tool evaluators if items don't have tools
|
||||
resolved = _filter_tool_evaluators(resolved, items)
|
||||
|
||||
# Standard JSONL dataset path
|
||||
return await self._evaluate_via_dataset(items, resolved, eval_name)
|
||||
|
||||
# -- Internal evaluation paths --
|
||||
|
||||
async def _evaluate_via_dataset(
|
||||
self,
|
||||
items: Sequence[EvalItem],
|
||||
evaluators: list[str],
|
||||
eval_name: str,
|
||||
) -> EvalResults:
|
||||
"""Evaluate using JSONL dataset upload path."""
|
||||
dicts: list[dict[str, Any]] = []
|
||||
for item in items:
|
||||
# Build JSONL dict directly from split_messages + converter
|
||||
# to avoid splitting the conversation twice.
|
||||
effective_split = item.split_strategy or self._conversation_split
|
||||
query_msgs, response_msgs = item.split_messages(effective_split)
|
||||
|
||||
query_text = " ".join(m.text for m in query_msgs if m.role == "user" and m.text).strip()
|
||||
response_text = " ".join(m.text for m in response_msgs if m.role == "assistant" and m.text).strip()
|
||||
|
||||
d: dict[str, Any] = {
|
||||
"query": query_text,
|
||||
"response": response_text,
|
||||
"query_messages": AgentEvalConverter.convert_messages(query_msgs),
|
||||
"response_messages": AgentEvalConverter.convert_messages(response_msgs),
|
||||
}
|
||||
if item.tools:
|
||||
d["tool_definitions"] = [
|
||||
{"name": t.name, "description": t.description, "parameters": t.parameters()} for t in item.tools
|
||||
]
|
||||
if item.context:
|
||||
d["context"] = item.context
|
||||
dicts.append(d)
|
||||
|
||||
has_context = any("context" in d for d in dicts)
|
||||
has_tools = any("tool_definitions" in d for d in dicts)
|
||||
|
||||
eval_obj = await self._client.evals.create(
|
||||
name=eval_name,
|
||||
data_source_config={ # type: ignore[arg-type] # pyright: ignore[reportArgumentType]
|
||||
"type": "custom",
|
||||
"item_schema": _build_item_schema(has_context=has_context, has_tools=has_tools),
|
||||
"include_sample_schema": True,
|
||||
},
|
||||
testing_criteria=_build_testing_criteria( # type: ignore[arg-type] # pyright: ignore[reportArgumentType]
|
||||
evaluators,
|
||||
self._model,
|
||||
include_data_mapping=True,
|
||||
),
|
||||
)
|
||||
|
||||
data_source = {
|
||||
"type": "jsonl",
|
||||
"source": {
|
||||
"type": "file_content",
|
||||
"content": [{"item": d} for d in dicts],
|
||||
},
|
||||
}
|
||||
|
||||
run = await self._client.evals.runs.create(
|
||||
eval_id=eval_obj.id,
|
||||
name=f"{eval_name} Run",
|
||||
data_source=data_source, # type: ignore[arg-type] # pyright: ignore[reportArgumentType]
|
||||
)
|
||||
|
||||
return await _poll_eval_run(
|
||||
self._client,
|
||||
eval_obj.id,
|
||||
run.id,
|
||||
self._poll_interval,
|
||||
self._timeout,
|
||||
provider=self.name,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Foundry-specific functions (not part of the Evaluator protocol)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def evaluate_traces(
|
||||
*,
|
||||
evaluators: Sequence[str] | None = None,
|
||||
client: FoundryChatClient | None = None,
|
||||
project_client: AIProjectClient | None = None,
|
||||
model: str,
|
||||
response_ids: Sequence[str] | None = None,
|
||||
trace_ids: Sequence[str] | None = None,
|
||||
agent_id: str | None = None,
|
||||
lookback_hours: int = 24,
|
||||
eval_name: str = "Agent Framework Trace Eval",
|
||||
poll_interval: float = 5.0,
|
||||
timeout: float = 180.0,
|
||||
) -> EvalResults:
|
||||
"""Evaluate agent behavior from OTel traces or response IDs.
|
||||
|
||||
Foundry-specific function — works with any agent that emits OTel traces
|
||||
to App Insights. Provide *response_ids* for specific responses,
|
||||
*trace_ids* for specific traces, or *agent_id* with *lookback_hours*
|
||||
to evaluate recent activity.
|
||||
|
||||
Args:
|
||||
evaluators: Evaluator names (e.g. ``[FoundryEvals.RELEVANCE]``).
|
||||
Defaults to relevance, coherence, and task_adherence.
|
||||
client: A ``FoundryChatClient`` instance. Provide this or *project_client*.
|
||||
project_client: An ``AIProjectClient`` instance.
|
||||
model: Model deployment name for the evaluator LLM judge.
|
||||
response_ids: Evaluate specific Responses API responses.
|
||||
trace_ids: Evaluate specific OTel trace IDs from App Insights.
|
||||
agent_id: Filter traces by agent ID (used with *lookback_hours*).
|
||||
lookback_hours: Hours of trace history to evaluate (default 24).
|
||||
eval_name: Display name for the evaluation.
|
||||
poll_interval: Seconds between status polls.
|
||||
timeout: Maximum seconds to wait for completion.
|
||||
|
||||
Returns:
|
||||
``EvalResults`` with status, result counts, and portal link.
|
||||
|
||||
Example:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
results = await evaluate_traces(
|
||||
response_ids=[response.response_id],
|
||||
evaluators=[FoundryEvals.RELEVANCE],
|
||||
client=chat_client,
|
||||
model="gpt-4o",
|
||||
)
|
||||
"""
|
||||
oai_client = _resolve_openai_client(client, project_client)
|
||||
resolved_evaluators = _resolve_default_evaluators(evaluators)
|
||||
|
||||
if response_ids:
|
||||
return await _evaluate_via_responses_impl(
|
||||
client=oai_client,
|
||||
response_ids=response_ids,
|
||||
evaluators=resolved_evaluators,
|
||||
model=model,
|
||||
eval_name=eval_name,
|
||||
poll_interval=poll_interval,
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
if not trace_ids and not agent_id:
|
||||
raise ValueError("Provide at least one of: response_ids, trace_ids, or agent_id")
|
||||
|
||||
trace_source: dict[str, Any] = {
|
||||
"type": "azure_ai_traces",
|
||||
"lookback_hours": lookback_hours,
|
||||
}
|
||||
if trace_ids:
|
||||
trace_source["trace_ids"] = list(trace_ids)
|
||||
if agent_id:
|
||||
trace_source["agent_id"] = agent_id
|
||||
|
||||
eval_obj = await oai_client.evals.create(
|
||||
name=eval_name,
|
||||
data_source_config={"type": "azure_ai_source", "scenario": "traces"}, # type: ignore[arg-type] # pyright: ignore[reportArgumentType]
|
||||
testing_criteria=_build_testing_criteria(resolved_evaluators, model), # type: ignore[arg-type] # pyright: ignore[reportArgumentType]
|
||||
)
|
||||
|
||||
run = await oai_client.evals.runs.create(
|
||||
eval_id=eval_obj.id,
|
||||
name=f"{eval_name} Run",
|
||||
data_source=trace_source, # type: ignore[arg-type] # pyright: ignore[reportArgumentType]
|
||||
)
|
||||
|
||||
return await _poll_eval_run(oai_client, eval_obj.id, run.id, poll_interval, timeout)
|
||||
|
||||
|
||||
async def evaluate_foundry_target(
|
||||
*,
|
||||
target: dict[str, Any],
|
||||
test_queries: Sequence[str],
|
||||
evaluators: Sequence[str] | None = None,
|
||||
client: FoundryChatClient | None = None,
|
||||
project_client: AIProjectClient | None = None,
|
||||
model: str,
|
||||
eval_name: str = "Agent Framework Target Eval",
|
||||
poll_interval: float = 5.0,
|
||||
timeout: float = 180.0,
|
||||
) -> EvalResults:
|
||||
"""Evaluate a Foundry-registered agent or model deployment.
|
||||
|
||||
Foundry invokes the target, captures the output, and evaluates it. Use
|
||||
this for scheduled evals, red teaming, and CI/CD quality gates.
|
||||
|
||||
Args:
|
||||
target: Target configuration dict.
|
||||
test_queries: Queries for Foundry to send to the target.
|
||||
evaluators: Evaluator names.
|
||||
client: A ``FoundryChatClient`` instance. Provide this or *project_client*.
|
||||
project_client: An ``AIProjectClient`` instance.
|
||||
model: Model deployment name for the evaluator LLM judge.
|
||||
eval_name: Display name for the evaluation.
|
||||
poll_interval: Seconds between status polls.
|
||||
timeout: Maximum seconds to wait for completion.
|
||||
|
||||
Returns:
|
||||
``EvalResults`` with status, result counts, and portal link.
|
||||
|
||||
Example:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
results = await evaluate_foundry_target(
|
||||
target={"type": "azure_ai_agent", "name": "my-agent"},
|
||||
test_queries=["Book a flight to Paris"],
|
||||
client=chat_client,
|
||||
model="gpt-4o",
|
||||
)
|
||||
"""
|
||||
if "type" not in target:
|
||||
raise ValueError("target dict must include a 'type' key (e.g., 'azure_ai_agent').")
|
||||
oai_client = _resolve_openai_client(client, project_client)
|
||||
resolved_evaluators = _resolve_default_evaluators(evaluators)
|
||||
|
||||
eval_obj = await oai_client.evals.create(
|
||||
name=eval_name,
|
||||
data_source_config={ # type: ignore[arg-type] # pyright: ignore[reportArgumentType]
|
||||
"type": "azure_ai_source",
|
||||
"scenario": "target_completions",
|
||||
},
|
||||
testing_criteria=_build_testing_criteria(resolved_evaluators, model), # type: ignore[arg-type] # pyright: ignore[reportArgumentType]
|
||||
)
|
||||
|
||||
data_source: dict[str, Any] = {
|
||||
"type": "azure_ai_target_completions",
|
||||
"target": target,
|
||||
"source": {
|
||||
"type": "file_content",
|
||||
"content": [{"item": {"query": q}} for q in test_queries],
|
||||
},
|
||||
}
|
||||
|
||||
run = await oai_client.evals.runs.create(
|
||||
eval_id=eval_obj.id,
|
||||
name=f"{eval_name} Run",
|
||||
data_source=data_source, # type: ignore[arg-type] # pyright: ignore[reportArgumentType]
|
||||
)
|
||||
|
||||
return await _poll_eval_run(oai_client, eval_obj.id, run.id, poll_interval, timeout)
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user