mirror of
https://github.com/microsoft/agent-framework.git
synced 2026-06-16 21:04:09 +08:00
chetantoshniwal
aa582d021d
* feat(evals): add ground_truth support for similarity evaluator - Include expected_output as ground_truth in Foundry JSONL dataset rows - Add ground_truth to item schema and data mapping for similarity evaluator - Add expected_output parameter to evaluate_workflow - Add similarity Pattern 3 to evaluate_agent and evaluate_workflow samples - Add tests for ground_truth in dataset, schema, and evaluate_workflow * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix: wrap long line to satisfy ruff E501 --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
aa582d021d
·
2026-04-21 19:40:53 +00:00
History
Foundry Evals Integration Samples
These samples demonstrate evaluating agent-framework agents using Azure AI Foundry's built-in evaluators.
Available Evaluators
| Category | Evaluators |
|---|---|
| Agent behavior | intent_resolution, task_adherence, task_completion, task_navigation_efficiency |
| Tool usage | tool_call_accuracy, tool_selection, tool_input_accuracy, tool_output_utilization, tool_call_success |
| Quality | coherence, fluency, relevance, groundedness, response_completeness, similarity |
| Safety | violence, sexual, self_harm, hate_unfairness |
Samples
evaluate_agent_sample.py — Dataset Evaluation (Path 3)
The dev inner loop. Two patterns from simplest to most control:
evaluate_agent()— One call: runs agent → converts → evaluatesFoundryEvals.evaluate()— Run agent yourself, convert withAgentEvalConverter, inspect/modify, then evaluate
uv run samples/05-end-to-end/evaluation/foundry_evals/evaluate_agent_sample.py
evaluate_traces_sample.py — Trace & Response Evaluation (Path 1)
Evaluate what already happened — zero changes to agent code:
evaluate_traces(response_ids=...)— Evaluate Responses API responses by IDevaluate_traces(agent_id=...)— Evaluate agent behavior from OTel traces in App Insights
uv run samples/05-end-to-end/evaluation/foundry_evals/evaluate_traces_sample.py
Setup
Create a .env file with configuration as in the .env.example file in this folder.
Which sample should I start with?
- "I want to test my agent during development" →
evaluate_agent_sample.py, Pattern 1 - "I want to evaluate past agent runs" →
evaluate_traces_sample.py - "I want to inspect/modify eval data before submitting" →
evaluate_agent_sample.py, Pattern 2