mirror of
https://github.com/microsoft/agent-framework.git
synced 2026-06-16 21:04:09 +08:00
copilot/python-release-bump-181
5 Commits
-
Python: feat(evals): Foundry Adaptive Evals integration (rubric-generation) (#6101)
* Python: feat(evals): RubricScore type + EvalScoreResult.dimensions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Python: feat(foundry-evals): RubricDimension + GeneratedEvaluatorRef + accept in evaluators= Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Python: feat(evals): parse rubric_scores from output items + assertion helpers Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Python: feat(evals): BaseAgent.as_eval_source / Workflow.as_eval_source Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Python: feat(foundry-evals): EvalGenerationSource + generate_rubric helper Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Python: feat(foundry-evals): YAML config loader + sample Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Python: fix(evals): address PR review feedback Addresses 4 Copilot review comments on PR #6101: 1. assert_dimension_score_at_least: drop the (not evaluator or found_any) guard so require_applicable=True correctly raises when the named evaluator produces no entries for the dimension. Adds TestRubricAssertions covering the regression. 2. GeneratedEvaluatorRef docstring: reword to describe actual behaviour (pinning recommended, not required) so it matches the dataclass default and FoundryEvals warning path. 3. _poll_generation_job: switch from asyncio.get_event_loop() to get_running_loop() and bound the per-iteration sleep by remaining time, matching _poll_eval_run. 4. generate_rubric: type category as Literal['quality','safety'] and validate at the entry point with a ValueError; drop the silent 'invalid -> quality' rewrite in _generation_job_to_ref. Adds a regression test. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Python: feat(foundry-evals): hosted-agent-aware rubric generation * Auto-detect hosted Foundry agents in agent_as_eval_source: when the agent's chat_client exposes a string agent_name (the convention used by RawFoundryAgentChatClient for PromptAgents/HostedAgents), emit a type='agent' EvalGenerationSource so the service fetches instructions and tools from the agent registry instead of relying on the local wrapper (which holds neither for hosted agents). * Add hosted_agent_version kwarg and a new agent_version field on EvalGenerationSource so PromptAgent runs can pin to a specific hosted version for reproducible rubric generation. * Add force_prompt_source escape hatch to bypass auto-detection and always emit a rendered prompt dossier - useful when the local wrapper carries overrides the service-side agent doesnt see. * Fix _to_sdk_source for dataset sources: SDK ctor takes name=/version=, not dataset_name=/dataset_version=. The mismatch would raise TypeError against the real azure-ai-projects 2.3.0a* SDK; only unmocked integration paths were affected. Tests cover: auto-detection happy path, versionless hosted agent, explicit hosted_agent_version forwarding, force_prompt_source override, non-string chat_client attrs (MagicMock test doubles) not mis-detected, agent_version forwarded through _to_sdk_source, and the corrected dataset SDK kwarg names. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(foundry-evals): accept canonical dimension_scores key per docs The published Foundry rubric-evaluator output (Microsoft Learn 'Rubric evaluators' reference) places per-dimension breakdowns under properties.dimension_scores, not properties.rubric_scores. The parser now tries dimension_scores first and falls back to rubric_scores for preview-build compatibility, and tolerates non-list payloads (e.g. MagicMock auto-attrs) by trying the next candidate when parsing yields zero entries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(foundry-evals): add manual create_rubric_evaluator Adds FoundryEvals.create_rubric_evaluator as the agent-framework surface over project_client.beta.evaluators.create_version. This is the manual counterpart to generate_rubric: callers supply RubricDimension instances (authored locally, ported from another framework, or hand-tuned) and we POST a RubricBasedEvaluatorDefinition. The service auto-attaches the non-editable residual dimension (general_quality for quality, general_policy_compliance for safety). Per the Microsoft Learn 'Rubric evaluators' reference, the auto-generation path (create_generation_job) is primarily a portal/UI feature; external SDK clients with rich local agent context are better served by manual create_version. This keeps generate_rubric for users who want to round-trip through a Foundry-registered agent. Validation up front: weight must be in [1,10], ids unique, descriptions non-empty, pass_threshold in [0,1]. The returned GeneratedEvaluatorRef is identical in shape to one obtained from generate_rubric, so downstream evaluators= lists work unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * samples(foundry-evals): manual rubric sample + namespace re-exports Adds evaluate_with_manual_rubric_sample.py demonstrating the end-to-end dev scenario for FoundryEvals.create_rubric_evaluator: hand-author a list of RubricDimension, register via create_rubric_evaluator, then use the pinned GeneratedEvaluatorRef alongside built-in evaluators in an agent regression run. Also re-exports RubricDimension, GeneratedEvaluatorRef, build_sources, and load_evals_config from agent_framework.foundry (both the lazy runtime shim and the type stub) so the rubric samples can import everything from a single namespace; the auto-generate sample was previously broken because the shim was missing build_sources / load_evals_config. Updates the foundry-evals README with a chooser entry for the two rubric paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(foundry-evals): remove rubric creation flows; keep consumption only Reframes agent-framework as a pure consumer of Foundry rubric evaluators: scoring against rubrics that already exist (authored in the Foundry portal or via the dedicated SDK / REST surface) instead of creating them from the SDK. Removed creation surface area: - FoundryEvals.generate_rubric (auto-generate path) and create_rubric_evaluator (manual path), plus all _GenerationSdkTypes / _ManualRubricSdkTypes / _to_sdk_dimensions / _coalesce_generation_sources / _to_sdk_source / _poll_generation_job / _generation_job_to_ref / _evaluator_version_to_ref / _get_beta_evaluators / _import_*_sdk_types helpers. - EvalGenerationSource (the input source discriminator), RubricDimension (the input dimension type), agent_as_eval_source / workflow_as_eval_source / _detect_hosted_foundry_agent helpers, and the YAML-config loader (_evals_config.py with RubricGenerationSpec / RubricSourceSpec / parse_evals_config / load_evals_config / build_sources). - BaseAgent.as_eval_source / Workflow.as_eval_source plus the _render_agent_dossier / _render_workflow_dossier helpers in core. These existed only to feed the now-removed generation pipeline. - Samples evaluate_with_generated_rubric_sample.py, evaluate_with_manual_rubric_sample.py, and evaluators.yaml. Replaced with a short README section showing how to reference an existing rubric evaluator via GeneratedEvaluatorRef. Kept (consumption surface): - GeneratedEvaluatorRef, slimmed to (name, version, display_name). Still accepted alongside built-in evaluator strings in FoundryEvals(evaluators=[...]). Versionless refs still warn. - RubricScore on EvalScoreResult.dimensions plus EvalResults.assert_dimension_score_at_least for per-dimension CI gates. - _parse_dimension_entries / _extract_rubric_scores output parsing (both canonical dimension_scores and the legacy rubric_scores key). Tests: 160/160 foundry unit tests and 71/71 core local-eval tests pass; pyright is clean across changed files. The pre-existing tests/core/test_telemetry.py::test_detect_hosted_fallback_import_error failure is unrelated and reproduces on the prior commit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * samples(foundry-evals): add evaluate_with_rubric_sample Adds a runnable end-to-end sample showing how to consume a pre-existing rubric evaluator created in Foundry: reference it with GeneratedEvaluatorRef(name, version), mix it with built-in evaluators in FoundryEvals, and gate CI with assert_dimension_score_at_least on a specific dimension. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(foundry-evals): satisfy mypy on _fetch_output_items mypy infers OutputItemListResponse.sample as dict[str, object] | None while pyright correctly infers the typed Sample model. Cast to Any so both type checkers accept the attribute access pattern, rename the local to avoid shadowing the inner-loop sample binding, and drop the now-stale pyright suppressions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs(foundry-evals): drop unpublished rubric-evaluators learn.microsoft.com link The Adaptive Evals authoring docs are not yet published on Microsoft Learn, so the link 404s. Keep the descriptive text without the broken hyperlink; we can re-add it once the docs ship. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(foundry-evals): hoist repeated local imports to module top Per code review feedback (eavanvalkenburg): the test file repeated 'from agent_framework_foundry._foundry_evals import ...' inside 22 test bodies and 'from agent_framework_foundry import GeneratedEvaluatorRef' inside 8 more. Move all of them to the existing top-level imports; the symbols are the same across tests and the local imports were redundant. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Ben Thomas <25218250+alliscode@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Ben Thomas ·
2026-06-01 23:01:56 +00:00 -
Yufeng He ·
2026-05-21 16:23:36 +00:00 -
Python: Improve the handling of intermediate outputs for workflows and orchestrations (#5623)
* Improve the handling of intermediate outputs for workflows and orchestrations * Address PR review feedback on intermediate output forwarding - Switch workflow.as_agent() forwarding to an explicit allowlist of {output, intermediate, data, request_info} so orchestration-internal events (group_chat, handoff_sent, magentic_orchestrator) stay inside the workflow instead of leaking into agent responses via str(data) coercion. - Stop raising on intermediate AgentResponseUpdate in non-streaming run(); surface the partial as a Message with text_reasoning content. The defensive raise still applies to terminal output events, where Update payloads would corrupt message ordering. - Extend the DevUI workflow-event mapper so intermediate yields wrapping plain strings, Messages, and list[Message] render as visible output items instead of generic completed-trace events. - Add orchestration coverage for GroupChat, Handoff, and Magentic builders (default vs intermediate_outputs=True; structural where end-to-end is heavy). * Lift output-designation policy into a value type Replace the ``Workflow._output_executors`` list and the ``RunnerContext.should_label_as_intermediate`` Protocol method with a single immutable ``OutputDesignation`` value type owned by ``Workflow``. Thread the designation as a parameter through the existing call chain (Runner -> EdgeRunner -> Executor -> WorkflowContext) so ``yield_output`` consults the threaded snapshot directly rather than calling back into the runner context. Removes the ``InProcRunnerContext._workflow`` back-reference and the ``WorkflowBuilder.build()`` assignment that wired it up. Adds the public predicate ``Workflow.is_terminal_executor(executor_id)`` for external observers; ``OutputDesignation`` itself stays package-internal. Key decisions - ``OutputDesignation.designated`` is ``frozenset[str] | None`` -- ``None`` preserves legacy "every yield is type='output'" behavior, any frozenset (including empty) opts into strict mode. The ``DeprecationWarning`` for legacy mode at build time is unchanged. - ``output_designation`` is an optional parameter on ``Runner``, ``EdgeRunner.send_message``, ``EdgeRunner._execute_on_target``, ``Executor.execute``, ``Executor._create_context_for_handler``, and ``WorkflowContext.__init__``. Each defaults to legacy ``OutputDesignation()`` so direct callers (Azure Functions ``CapturingRunnerContext``, ``test_runner`` recording fixtures) keep working without ceremony. - The workflow-level filter in ``_run_core`` reads ``self._output_designation`` live, preserving today's semantics where mutating the designation after build still affects subsequent runs (used by two existing tests). - ``Workflow.to_dict()`` continues to emit ``"output_executors": list[str] | None`` (sorted from the frozenset). Checkpoint format unchanged. Files changed - _workflow.py: add ``OutputDesignation`` dataclass; replace ``_output_executors`` with ``_output_designation``; add ``is_terminal_executor``; delete ``_should_yield_output_event``. - _runner_context.py: drop ``should_label_as_intermediate`` Protocol method and ``InProcRunnerContext`` impl; drop ``_workflow`` back-reference. - _workflow_builder.py: remove ``context._workflow = workflow`` assignment. - _runner.py, _edge_runner.py, _executor.py, _workflow_context.py: thread ``output_designation`` parameter through the call chain. - tests/workflow/test_output_designation.py (new): three-state coverage of the value type plus the public predicate delegation. - tests/workflow/test_workflow_builder.py, test_validation.py, test_workflow.py, test_runner.py and orchestrations/tests/test_orchestration_intermediate_vs_terminal.py: switch probes from ``_output_executors`` set checks to ``get_output_executors`` / ``is_terminal_executor``; update two post-build mutation tests to set ``_output_designation`` instead. Verification - core/tests/workflow/, orchestrations/tests/, azurefunctions/tests/: 1119 passed, 42 skipped, 2 xfailed. - ``uv run poe lint``: clean. - ``uv run poe typing``: only the pre-existing ``_AGENT_FORWARDED_EVENT_TYPES`` pyright warning from394bcd607remains. Notes for next iteration - The builder's own ``_output_executors`` attribute (``list[Executor | SupportsAgentRun]``) is intentionally untouched; the issue scoped the rename to the workflow attribute. - Adjacent review candidates (twin ``WorkflowAgent`` translators, ``_AGENT_FORWARDED_EVENT_TYPES`` kind classifier, ``_event_origin_context`` ContextVar removal, ``WorkflowEvent`` ADT split, legacy-mode removal) remain out of scope. * Add explicit workflow output designation Key decisions - Extend the internal OutputDesignation value type from terminal-only membership to output/intermediate/hidden classification. Legacy mode remains outputs=None, so workflows built without output_executors or intermediate_executors still label every yield_output as type='output'. - WorkflowBuilder now accepts intermediate_executors. Providing either designation enters explicit mode; output executors emit output, intermediate executors emit intermediate, and unlisted yield_output payloads are hidden from caller-facing events while remaining in executor_completed data. - Empty explicit designation, duplicate entries, overlaps, unknown executors, and designated executors without workflow output annotations fail build validation. Existing orchestration builders pass intermediate-capable participants through intermediate_executors to preserve current intermediate_outputs behavior until participant-oriented designation lands. Files changed - packages/core/agent_framework/_workflows/_workflow.py, _workflow_builder.py, _workflow_context.py, _validation.py, _events.py - packages/core/tests/workflow/test_output_designation.py, test_output_executors_contract.py, test_strict_mode_event_labeling.py, test_validation.py, test_workflow.py, test_workflow_agent_intermediate.py - packages/orchestrations/agent_framework_orchestrations/_sequential.py, _concurrent.py, _group_chat.py, _magentic.py - packages/core/AGENTS.md Verification - uv run pytest packages/core/tests/workflow packages/orchestrations/tests packages/devui/tests/devui/test_mapper.py -q - uv run pytest packages/azurefunctions/tests -q - uv run poe lint - uv run poe typing fails only on pre-existing packages/core/agent_framework/_workflows/_agent.py _AGENT_FORWARDED_EVENT_TYPES private-use pyright error. Notes for next iteration - issues/03-core-workflow-explicit-designation.md was moved to issues/done but issues/ remains untracked and intentionally excluded from this commit. - Slice 4 should tighten workflow.as_agent() mapping for hidden emissions and streaming-only update payloads; Slice 5 should replace orchestration intermediate_outputs with participant-oriented designation. * Tighten workflow-as-agent output mapping Key decisions - Treat AgentResponseUpdate as a streaming-only payload across the workflow.as_agent() adapter, so non-streaming agent runs now reject both terminal output and intermediate workflow events carrying updates. - Keep streaming classification behavior explicit: terminal update payloads remain normal text content, while intermediate update payloads are rewritten to text_reasoning content. - Add explicit-mode coverage proving hidden yield_output emissions do not appear in non-streaming AgentResponse messages or streaming AgentResponseUpdate chunks. Files changed - packages/core/agent_framework/_workflows/_agent.py - packages/core/tests/workflow/test_workflow_agent_intermediate.py Verification - uv run pytest packages/core/tests/workflow/test_workflow_agent_intermediate.py -q - uv run pytest packages/core/tests/workflow/test_workflow_agent.py packages/core/tests/workflow/test_workflow_agent_intermediate.py -q - uv run pytest packages/core/tests/workflow packages/orchestrations/tests packages/devui/tests/devui/test_mapper.py -q - uv run poe lint - uv run poe typing fails only on the pre-existing packages/core/agent_framework/_workflows/_agent.py _AGENT_FORWARDED_EVENT_TYPES private-use pyright error. Blockers or notes for next iteration - issues/04-workflow-as-agent-output-mapping.md was moved to issues/done/ but issues/ remains untracked and intentionally excluded from this commit. - Slice 5 should replace orchestration intermediate_outputs with participant-oriented designation. * Add orchestration participant output designation Key decisions - Replace orchestration intermediate_outputs with participant-oriented output_participants and intermediate_participants across Sequential, Concurrent, GroupChat, Magentic, and Handoff builders. - Keep synthetic final executors terminal by default for Concurrent, GroupChat, and Magentic; keep Sequential's final participant terminal by default; keep Handoff participants terminal by default. - Centralize participant designation validation for empty explicit designation, duplicates, overlaps, and unknown participants, then map validated participants to workflow output/intermediate executors. Files changed - packages/orchestrations/agent_framework_orchestrations/_participant_designation.py - packages/orchestrations/agent_framework_orchestrations/_sequential.py - packages/orchestrations/agent_framework_orchestrations/_concurrent.py - packages/orchestrations/agent_framework_orchestrations/_group_chat.py - packages/orchestrations/agent_framework_orchestrations/_magentic.py - packages/orchestrations/agent_framework_orchestrations/_handoff.py - packages/orchestrations/tests/test_orchestration_intermediate_vs_terminal.py - packages/orchestrations/tests/test_magentic.py Blockers or notes for next iteration - issues/05-orchestration-participant-designation.md was moved to issues/done/ but issues/ remains untracked and intentionally excluded from this commit. - Slice 7 should migrate samples and docs away from intermediate_outputs to the new participant designation API. - uv run poe typing still fails only on the pre-existing packages/core/agent_framework/_workflows/_agent.py _AGENT_FORWARDED_EVENT_TYPES private-use pyright error. * Migrate samples to explicit output designation Key decisions - Replace sample usage of the removed orchestration intermediate_outputs boolean with participant-oriented intermediate_participants designation. - Update raw workflow guidance to show output_executors together with intermediate_executors, and document that unlisted yields are hidden in explicit designation mode. - Keep orchestration final outputs terminal while streaming designated participant responses as intermediate progress, including workflow.as_agent() samples where intermediates map to text_reasoning content. - Refresh workflow and orchestration README guidance plus the changelog reference so public docs no longer point users at intermediate_outputs. Files changed - CHANGELOG.md - packages/orchestrations/README.md - samples/README.md - samples/03-workflows/README.md - samples/03-workflows/control-flow/intermediate_vs_terminal_outputs.py - samples/03-workflows/orchestrations/README.md - samples/03-workflows/orchestrations/group_chat_agent_manager.py - samples/03-workflows/orchestrations/group_chat_philosophical_debate.py - samples/03-workflows/orchestrations/group_chat_simple_selector.py - samples/03-workflows/orchestrations/magentic.py - samples/03-workflows/orchestrations/magentic_human_plan_review.py - samples/03-workflows/orchestrations/sequential_chain_only_agent_responses.py - samples/03-workflows/agents/group_chat_workflow_as_agent.py - samples/03-workflows/agents/magentic_workflow_as_agent.py - samples/03-workflows/agents/sequential_workflow_as_agent.py - samples/semantic-kernel-migration/orchestrations/group_chat.py - samples/semantic-kernel-migration/orchestrations/magentic.py Blockers or notes for next iteration - issues/07-samples-and-docs-explicit-output-designation.md was moved to issues/done/ but issues/ remains untracked and intentionally excluded from this commit. - issues/06-devui-intermediate-event-rendering.md remains present and appears already satisfied by existing DevUI mapper/tests from the prior implementation slice. - PRD-explicit-workflow-output-designation.md remains untracked and intentionally excluded from this commit. * Render DevUI intermediate workflow outputs Key decisions - Preserve workflow output designation metadata on visible DevUI output messages and text deltas so intermediate/data emissions remain distinguishable from terminal output. - Render intermediate workflow message items in the execution timeline using executor metadata, while excluding them from the final workflow result aggregation. - Keep terminal output message rendering unchanged and retain legacy data events on the intermediate compatibility path. Files changed - packages/devui/agent_framework_devui/_mapper.py - packages/devui/frontend/src/components/features/workflow/execution-timeline.tsx - packages/devui/frontend/src/components/features/workflow/workflow-view.tsx - packages/devui/frontend/src/types/openai.ts - packages/devui/tests/devui/test_mapper.py Blockers or notes for next iteration - issues/06-devui-intermediate-event-rendering.md was moved to issues/done/ but issues/ remains untracked and intentionally excluded from this commit. - PRD-explicit-workflow-output-designation.md remains untracked and intentionally excluded from this commit. - uv run poe typing still fails only on the pre-existing packages/core/agent_framework/_workflows/_agent.py _AGENT_FORWARDED_EVENT_TYPES private-use pyright error. * Fix mypy * Clarify orchestration participant output config * Rename participant output kwargs for clarity output_participants -> final_output_from, intermediate_participants -> intermediate_output_from. The old names read like categories of participant; the new names make it clear the kwarg designates which participants' outputs surface as final vs. intermediate events. * Rename core workflow output kwargs with deprecation shim Adds final_output_from / intermediate_output_from as canonical kwargs on Workflow and WorkflowBuilder. Old output_executors / intermediate_executors kwargs continue to work but emit DeprecationWarning via a shared coalesce helper that also rejects supplying both. Wire-format keys in to_dict() stay as output_executors / intermediate_executors so checkpoint compatibility is preserved. Internal call sites in orchestrations and samples updated to the new names so users following sample code learn the canonical vocabulary; legacy callers still work with a one-shot warning. * Suppress pyright reportPrivateUsage on cross-module sentinel import * Update docstrings * Propagate sub-workflow intermediate outputs, fix handoff/sequential intermediate-only designation, and shore up tests, sample, and docstrings around the intermediate output contract. * Add canonical workflow output_from selection Key decisions:\n- Make output_from the canonical workflow-output allow-list and keep output_executors/final_output_from as deprecated compatibility aliases.\n- Treat empty output_from/intermediate_output_from lists as explicit selections and keep validation responsible for empty, duplicate, overlap, and unknown selections.\n- Remove the branch-only public intermediate_executors WorkflowBuilder kwarg while preserving legacy wire keys in to_dict().\n\nFiles changed:\n- packages/core/agent_framework/_workflows/_workflow.py\n- packages/core/agent_framework/_workflows/_workflow_builder.py\n- packages/core/agent_framework/_workflows/_workflow_context.py\n- packages/core/agent_framework/_workflows/_agent.py\n- packages/core/agent_framework/_workflows/_agent_executor.py\n- packages/core/tests/workflow/* output-selection coverage updates\n- packages/core/AGENTS.md\n- issues/done/001-canonical-list-based-output-selection.md\n\nBlockers/notes:\n- Orchestration builders still pass final_output_from internally; follow-up issue 004 should migrate them to output_from.\n- Legacy omitted-selection behavior and explicit all/all_other literals are left for issues 002 and 003. * Add explicit all workflow output selection Key decisions: - Treat output_from='all' as an explicit workflow-output selection sentinel and expand it at build time to executors with declared workflow output types. - Keep omitted output selections in legacy all-output mode with a deprecation warning that names output_from and intermediate_output_from and points to output_from='all'. - Reject intermediate_output_from='all' at construction because the all-output literal is output-only for this issue. Files changed: - packages/core/agent_framework/_workflows/_workflow_builder.py - packages/core/tests/workflow/test_output_executors_contract.py - issues/done/002-explicit-all-output-and-legacy-migration.md Blockers/notes: - all_other intermediate-output selection remains for issue 003. - Workflow-as-agent/orchestration parity remains for issue 004. * Add all-other intermediate output selection Key decisions: - Treat intermediate_output_from='all_other' as an explicit intermediate-output selection sentinel and expand it at build time after the workflow graph is complete. - Expand all_other to output-capable executors not selected by output_from; omitted or empty output_from selects no workflow outputs, while output_from='all' leaves an empty intermediate selection. - Keep output_from='all_other' invalid so all_other remains intermediate-output-only and runtime classification still receives concrete executor-id sets. Files changed: - packages/core/agent_framework/_workflows/_workflow_builder.py - packages/core/tests/workflow/test_output_executors_contract.py - issues/done/003-all-other-intermediate-output-selection.md Blockers/notes: - Workflow-as-agent and orchestration parity remains for issue 004. - Full documentation updates remain for issue 005. * Add orchestration output selection parity Key decisions: - Expose output_from on sequential, concurrent, group chat, handoff, and magentic builders while keeping final_output_from as a deprecated compatibility alias. - Resolve orchestration participant selections through the same explicit rules as workflows: output_from='all', intermediate_output_from='all_other', hidden unselected participant payloads, and overlap/duplicate/unknown/invalid-literal validation. - Continue preserving documented orchestration defaults by always designating each pattern's terminal internal executor where applicable. Files changed: - packages/orchestrations/agent_framework_orchestrations/_participant_output_config.py - packages/orchestrations/agent_framework_orchestrations/_sequential.py - packages/orchestrations/agent_framework_orchestrations/_concurrent.py - packages/orchestrations/agent_framework_orchestrations/_group_chat.py - packages/orchestrations/agent_framework_orchestrations/_handoff.py - packages/orchestrations/agent_framework_orchestrations/_magentic.py - packages/orchestrations/agent_framework_orchestrations/_orchestration_request_info.py - packages/orchestrations/tests/test_orchestration_intermediate_vs_terminal.py - issues/done/004-workflow-as-agent-and-orchestration-parity.md Blockers/notes: - Full documentation and sample migration wording remains for issue 005. - Existing tests that intentionally use final_output_from now emit the new deprecation warning. * Document workflow output selection contract Key decisions: - Use Workflow Output and Intermediate Output as the developer-facing terms for selected caller-facing emissions. - Document output_from and intermediate_output_from as the canonical API, with output_from as an allow-list and unselected payloads hidden unless explicitly selected as intermediate. - Add scenario and invalid-selection tables for workflow and orchestration docs, including legacy omission warnings, output_from='all', intermediate_output_from='all_other', list selections, invalid literals, overlap, duplicates, unknown selections, and empty explicit selections. - Migrate samples away from final_output_from and output_executors except where compatibility aliases are explicitly documented. Files changed: - packages/core/AGENTS.md - packages/orchestrations/README.md - packages/orchestrations/agent_framework_orchestrations/_handoff.py - packages/orchestrations/agent_framework_orchestrations/_sequential.py - samples/03-workflows/README.md - samples/03-workflows/control-flow/intermediate_vs_terminal_outputs.py - samples/03-workflows/human-in-the-loop/agents_with_approval_requests.py - samples/03-workflows/orchestrations/README.md - samples/04-hosting/foundry-hosted-agents/responses/05_workflows/main.py - scripts/sample_validation/create_dynamic_workflow_executor.py - issues/done/005-document-output-selection-contract.md Blockers/notes: - Direct full Ruff on scripts/sample_validation/create_dynamic_workflow_executor.py still reports pre-existing docstring/print/line-length issues outside this docs migration; syntax-focused checks for changed files pass. - No remaining AFK issue files are present under issues/. * Latest updates * Typing fixes * CleanupEvan Mattson ·
2026-05-19 00:15:25 +00:00 -
Python: feat(evals): add ground_truth support for similarity evaluator (#5234)
* feat(evals): add ground_truth support for similarity evaluator - Include expected_output as ground_truth in Foundry JSONL dataset rows - Add ground_truth to item schema and data mapping for similarity evaluator - Add expected_output parameter to evaluate_workflow - Add similarity Pattern 3 to evaluate_agent and evaluate_workflow samples - Add tests for ground_truth in dataset, schema, and evaluate_workflow * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix: wrap long line to satisfy ruff E501 --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
chetantoshniwal ·
2026-04-21 19:40:53 +00:00 -
Python: Foundry Evals integration for Python (#4750)
* Foundry Evals integration for Python Merged and refactored eval module per Eduard's PR review: - Merge _eval.py + _local_eval.py into single _evaluation.py - Convert EvalItem from dataclass to regular class - Rename to_dict() to to_eval_data() - Convert _AgentEvalData to TypedDict - Simplify check system: unified async pattern with isawaitable - Parallelize checks and evaluators with asyncio.gather - Add all/any mode to tool_called_check - Fix bool(passed) truthy bug in _coerce_result - Remove deprecated function_evaluator/async_function_evaluator aliases - Remove _MinimalAgent, tighten evaluate_agent signature - Set self.name in __init__ (LocalEvaluator, FoundryEvals) - Limit FoundryEvals to AsyncOpenAI only - Type project_client as AIProjectClient - Remove NotImplementedError continuous eval code - Add evaluation samples in 02-agents/ and 03-workflows/ - Update all imports and tests (167 passing) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: resolve mypy redundant-cast errors while keeping pyright happy Use cast(list[Any], x) with type: ignore[redundant-cast] comments to satisfy both mypy (which considers casting Any redundant) and pyright strict mode (which needs explicit casts to narrow Unknown types). Also fix evaluator decorator check_name type annotation to be explicitly str, resolving mypy str|Any|None mismatch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: CI failures — pyupgrade, evaluator overloads, sample API, reset attr - Apply pyupgrade: Sequence from collections.abc, remove forward-ref quotes - Add @overload signatures to evaluator() for proper @evaluator usage - Fix evaluate_workflow sample to use WorkflowBuilder(start_executor=) API - Fix _workflow.py executor.reset() to use getattr pattern for pyright - Remove unused EvalResults forward-ref string in default_factory lambda Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: skip gRPC-dependent observability test The test_configure_otel_providers_with_env_file_and_vs_code_port test triggers gRPC OTLP exporter creation, but the grpc dependency is optional and not installed by default. Add skipif decorator matching the pattern used by all other gRPC exporter tests in the same file. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: add nosec B101 for bandit assert check Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * style: align eval samples with repo conventions - Move module docstrings before imports (after copyright header) - Add -> None return type to all main() and helper functions - Fix line-too-long in multiturn sample conversation data - Add Workflow import for typed return in all_patterns_sample Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review feedback: async fixes, sample bugs, deprecation warnings - Simplify _ensure_async_result to direct await (async-only clients) - Replace get_event_loop() with get_running_loop() - Narrow _fetch_output_items exception handling to specific types - Add warning log when _filter_tool_evaluators falls back to defaults - Add DeprecationWarning to options alias in Agent.__init__ - Add DeprecationWarning to evaluate_response() - Rename raw key to _raw_arguments in convert_message fallback - Fix evaluate_agent_sample.py: replace evals.select() with FoundryEvals() - Fix evaluate_multiturn_sample.py: use Message/Content/FunctionTool types - Fix evaluate_workflow_sample.py: replace evals.select() with FoundryEvals() - Update test mocks to use AsyncMock for awaited API calls Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add test coverage for review feedback items - Add num_repetitions=2 positive test verifying 2×items and 4 agent calls - Add _poll_eval_run tests: timeout, failed, and canceled paths - Add evaluate_traces tests: validation error, response_ids path, trace_ids path - Add evaluate_foundry_target happy-path test with target/query verification Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix ruff ISC004 lint error and apply formatter - Wrap implicit string concatenation in parens in evaluate_multiturn_sample.py - Apply ruff formatter to 6 other files with minor formatting drift Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove core type changes (extracted to fix/workflow-stale-session branch) Reverts changes to _agents.py, _agent_executor.py, and _workflow.py back to upstream/main. These fixes are now in a separate PR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review round 2: bugs, tests, and architecture Code fixes: - Fix _normalize_queries inverted condition (single query now replicates to match expected_count) - Fix substring match bug: 'end' in 'backend' matched; use exact set lookup for executor ID filtering - Fix used_available_tools sample: tool_definitions→tools param, use FunctionTool attribute access instead of dict .get() - Add None-check in _resolve_openai_client for misconfigured project - Add Returns section to evaluate_workflow docstring - Cache inspect.signature in @evaluator wrapper (avoid per-item reflection) Architecture: - Extract _evaluate_via_responses as module-level helper; evaluate_traces now calls it directly instead of creating a FoundryEvals instance - Move Foundry-specific typed-content conversion out of core to_eval_data; core now returns plain role/content dicts, FoundryEvals applies AgentEvalConverter in _evaluate_via_dataset Tests: - evaluate_response() deprecation warning emission and delegation - num_repetitions > 1 with expected_output and expected_tool_calls - Mock output_items.list in test_evaluate_calls_evals_api - Update to_eval_data assertions for plain-dict format - Unknown param error now raised at @evaluator decoration time Skipped (separate PR): executor reset loop, xfail removal, options alias Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix CI: revert test_full_conversation, fix pyright errors - Revert test_full_conversation.py to upstream/main (the session preservation test was incorrectly changed to assert clearing) - Fix pyright reportUnnecessaryComparison on get_openai_client() None check by adding ignore comment - Fix pyright reportPrivateUsage: add public EvalItem.split_messages() method and use it in FoundryEvals._evaluate_via_dataset instead of accessing private _split_conversation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review round 3: reliability, test gaps, cleanup - Add try/except guard for non-numeric score in _coerce_result - Add poll_interval minimum bound (0.1s) to prevent tight loops - Add runtime async client check in _resolve_openai_client - Remove _ensure_async_result wrapper (10 call sites → direct await) - Better error message when queries provided without agent - Import-time asserts for evaluator set consistency - Remove 28 redundant @pytest.mark.asyncio decorators - Add doc note about _raw_arguments sensitive data - Tests: tool_called_check mode=any, _normalize_queries branches, _extract_result_counts paths, _extract_per_evaluator, bare check via evaluate_agent, output_items assertion, modulo wrapping, async client check, queries-without-agent error Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix CI: ruff S101 assert, pyright and mypy arg-type errors - Replace module-level assert with if/raise for evaluator set consistency checks (ruff S101 disallows bare assert) - Add type: ignore[arg-type] and pyright: ignore[reportArgumentType] on OpenAI SDK evals API calls that pass dicts where typed params are expected (SDK accepts dicts at runtime) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review round 4: bugs, reliability, test fixes - Fix all_passed ignoring parent result_counts when sub_results present - Fix _extract_tool_calls: parse string arguments via json.loads before falling back to None (real LLM responses use string arguments) - Sanitize _raw_arguments to '[unparseable]' to avoid leaking sensitive tool-call data to external evaluation services - Add NOTE comment on to_eval_data message serialization dropping non-text content (tool calls, results) - Eliminate double conversation split in _evaluate_via_dataset: build JSONL dicts directly from split_messages + AgentEvalConverter - Raise poll_interval floor from 0.1s to 1.0s to prevent rate-limit exhaustion - Fix MagicMock(name=...) bug in test: sets display name not .name attr - Fix mock_output_item.sample: use MagicMock object instead of dict so _fetch_output_items exercises error/usage/input/output extraction Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review round 5: reliability, docs, test coverage Code fixes: - Move import-time RuntimeError checks to unit tests (avoids breaking imports for all users on developer set-drift mistake) - _filter_tool_evaluators now raises ValueError when all evaluators require tools but no items have tools (was silently substituting) - Add poll_interval upper bound (60s) to prevent single-iteration sleep - Log exc_info=True in _fetch_output_items for debugging API changes - Fix evaluate() docstring: remove claim about Responses API optimization - Validate target dict has 'type' key in evaluate_foundry_target - Document to_eval_data() limitation: non-text content is omitted Tests: - TestEvaluatorSetConsistency: verify _AGENT/_TOOL subsets of _BUILTIN - TestEvaluateTracesAgentId: agent_id-only path with lookback_hours - TestFilterToolEvaluatorsRaises: ValueError on all-tool no-items - TestEvaluateFoundryTargetValidation: target without 'type' key - Assert items==[] on failed/canceled poll results - Mock output_items.list in response_ids test for full flow - TestAllPassedSubResults: result_counts=None + sub_results delegation and parent failures override sub_results - TestBuildOverallItemEmpty: empty workflow outputs returns None Skipped r5-07 (_raw_arguments length hint): marginal debugging value, could leak content size information. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix error message: evaluate_responses() → evaluate_traces(response_ids=...) The referenced function doesn't exist; the correct API is evaluate_traces(response_ids=...) from the azure-ai package. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove dead to_eval_data() method, fix docstring claims - Remove to_eval_data() from EvalItem (dead code after r4-05 JSONL refactor) - Migrate 15 tests from to_eval_data() to split_messages() - Update sample to use split_messages() + Message properties - Remove unimplemented Responses API optimization docstring claim - Update split_messages() docstring to not reference removed method Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Reduce default eval timeout from 600s to 180s (3 minutes) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove dead _evaluate_via_responses method from FoundryEvals The method was never called — evaluate() uses _evaluate_via_dataset, and evaluate_traces() calls _evaluate_via_responses_impl directly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert unrelated formatting changes to get-started samples Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix pyright: remove phantom FoundryMemoryProvider import, apply ruff format - Remove import of non-existent _foundry_memory_provider module (incorrectly kept during rebase conflict resolution) - Apply ruff formatter to test_local_eval.py and get-started samples Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix eval samples: use FoundryChatClient for Agent() The upstream provider-leading client refactor (#4818) made client= a required parameter on Agent(). Update the three getting-started eval samples to use FoundryChatClient with FOUNDRY_PROJECT_ENDPOINT, matching the standard pattern from 01-get-started samples. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Simplify self-reflection sample using FoundryEvals Replace ~80 lines of manual OpenAI evals API code (create_eval, run_eval, manual polling, raw JSONL params) with FoundryEvals: - evaluate_groundedness() uses FoundryEvals.evaluate() with EvalItem - Remove create_openai_client(), create_eval(), run_eval() functions - Remove openai SDK type imports (DataSourceConfigCustom, etc.) - run_self_reflection_batch creates FoundryEvals instance once, reuses it for all iterations across all prompts Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update eval samples to FoundryChatClient and FOUNDRY_PROJECT_ENDPOINT - Migrate all foundry_evals samples from AzureOpenAIResponsesClient to FoundryChatClient - Update env var from AZURE_AI_PROJECT_ENDPOINT to FOUNDRY_PROJECT_ENDPOINT - Use AzureCliCredential consistently across all samples - Fix README.md: correct function names (evaluate_dataset -> FoundryEvals.evaluate, evaluate_responses -> evaluate_traces) - Update self_reflection .env.example and README.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix lint errors in eval samples (E501, ASYNC240, formatting) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove evaluate_all_patterns_sample.py (redundant with focused samples) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix async credential mismatch: use azure.identity.aio for async AIProjectClient AIProjectClient from azure.ai.projects.aio requires an async credential. Switch all foundry_evals samples from azure.identity.AzureCliCredential to azure.identity.aio.AzureCliCredential. Also pass project_client to FoundryChatClient instead of duplicating endpoint+credential. Close credential in self_reflection sample to avoid resource leak. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert test_observability.py to upstream/main (not our test) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address moonbox3 review: sphinx docstrings, pagination, isinstance check - Convert all Example:: / Typical usage:: code blocks to .. code-block:: python format matching codebase convention (both _evaluation.py and _foundry_evals.py) - Add async pagination in _fetch_output_items via async for (handles large result sets) - Replace hasattr(__aenter__) with isinstance(client, AsyncOpenAI) in _resolve_openai_client - Move AsyncOpenAI import from TYPE_CHECKING to runtime (needed for isinstance) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix test failures and address remaining moonbox3 review comments - Fix tests: use MagicMock(spec=AsyncOpenAI) for project_client mocks (isinstance check now requires proper type, not duck-typing) - Fix tests: replace mock_page.__iter__ with _AsyncPage helper for async for - Fix evaluate_response: auto-extract queries from response messages when query is not provided (previously always raised ValueError) - Add debug logging when skipping internal _-prefixed executor IDs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address Tao's PR review comments on Foundry Evals - T1: Add comment explaining builtin.* pass-through in _resolve_evaluator - T2: Add comment referencing OpenAI evals API for testing_criteria dict - T3: Document Mustache-style {{item.*}} template placeholders - T4: Document poll loop 60s sleep upper bound rationale - T5: Narrow run type to RunRetrieveResponse, use typed field access instead of vars()/getattr dance in _extract_result_counts and _extract_per_evaluator; use run.error and run.report_url directly - T6: Clarify openai_client docstring re: Azure Foundry endpoint - T8: Remove misleading empty expected_tool_calls from sample - Update tests to match real SDK PerTestingCriteriaResult shape Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove unnecessary Any union from run type annotations RunRetrieveResponse is the correct type — no backward compat needed for a brand new feature. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Accept FoundryChatClient instead of raw AsyncOpenAI FoundryEvals now takes client: FoundryChatClient as its primary parameter instead of openai_client: AsyncOpenAI. The builtin.* evaluators require a Foundry endpoint, so the type should reflect that. - FoundryEvals.__init__: client: FoundryChatClient replaces openai_client - evaluate_traces / evaluate_foundry_target: same change - _resolve_openai_client: extracts .client from FoundryChatClient - project_client fallback retained for standalone functions - All samples updated to construct FoundryChatClient and pass as client= - Tests updated (openai_client= → client=) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove implicit 60s upper bound on poll interval If a developer sets a higher poll_interval, respect it. Only clamp to remaining time and enforce a 1s minimum for rate-limit protection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove 1s floor on poll interval — let the developer control it Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update python/samples/05-end-to-end/evaluation/foundry_evals/.env.example Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com> * Update python/samples/02-agents/evaluation/evaluate_agent.py Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com> * Address eavanvalkenburg review (round 2) on Python eval PR - Rename model_deployment -> model across FoundryEvals and all samples - Make model param optional, resolves from client.model - Convert EvalResults from dataclass to regular class - Remove deprecated evaluate_response() function - Refactor splitters: BUILT_IN_SPLITTERS dict + standalone functions - Change per_turn_items from classmethod to staticmethod - Simplify EvalCheck type alias to use Awaitable[CheckResult] - Remove errored property from EvalResults - Remove default value from Evaluator protocol eval_name - Rename assert_passed -> raise_for_status, add EvalNotPassedError - Type agent param as SupportsAgentRun | None - Fix Arguments docstring - Update __init__.py exports - Update all tests and samples Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Move FoundryEvals to foundry package, split tool eval sample - Move _foundry_evals.py from azure-ai to foundry package - Move test_foundry_evals.py to foundry/tests/ - Update lazy re-exports in agent_framework.foundry namespace - Update .pyi type stubs - All samples now import from agent_framework.foundry - Split tool-call evaluation into evaluate_tool_calls_sample.py - Fix all_passed to check errored count from result_counts - Fix raise_for_status to include errored item details Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Auto-create FoundryChatClient from env vars when no client provided FoundryEvals() now works zero-config when FOUNDRY_PROJECT_ENDPOINT and FOUNDRY_MODEL environment variables are set. Auto-creates a FoundryChatClient under the hood, matching the established env var pattern. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix pyright errors: remove dead _normalize_queries, suppress EvalAPIError check - Remove unused _normalize_queries function and its tests - Add pyright ignore for EvalAPIError None check (defensive guard) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Support multimodal image content in eval pipeline Add image (data/uri) content handling to AgentEvalConverter.convert_message() so that Content.from_data() and Content.from_uri() image payloads are preserved as input_image parts in the Foundry evaluator format. - Handle Content type='data' and type='uri' → emit input_image parts - Add 6 unit tests for image content through convert_message/convert_messages - Add integration test verifying images flow through EvalItem → JSONL path - Add evaluate_multimodal.py sample demonstrating local image eval Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address remaining review comments - Fix project_client docstring to say async-only (not sync/async) - Add builtin evaluator name validation warning in _resolve_evaluator - Replace getattr with typed attribute access in _poll_eval_run, _extract_result_counts, _extract_per_evaluator, _fetch_output_items - Remove cast import from _foundry_evals (no longer needed) - Tighten _coerce_result: honour explicit 'passed' when both 'score' and 'passed' are present; remove performative cast - Fix self_reflection sample: add env file existence check - Fix traces sample: correct Pattern 2 section label - Update all Foundry eval samples to FoundryChatClient + FOUNDRY_MODEL (remove AIProjectClient + AZURE_AI_MODEL_DEPLOYMENT_NAME pattern) - Add eval_name and OpenAI client docs to FoundryEvals docstring - Update test mocks to match typed SDK objects (_MockResultCounts) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix ruff lint errors (E501, SIM108, SIM102) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix pyright errors: type-narrow dict to dict[str, Any], add ignore comments Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Replace ConversationSplitter type alias with Protocol ConversationSplitter is now a runtime-checkable Protocol with a named 'conversation' parameter, making the expected signature self-documenting. ConversationSplit enum members gain a __call__ method so they satisfy the protocol directly -- ConversationSplit.LAST_TURN(conversation) works. This simplifies _split_conversation from an isinstance dispatch to a single split(conversation) call. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Standardize on AZURE_AI_MODEL_DEPLOYMENT_NAME and fix Unicode in samples - Replace FOUNDRY_MODEL with AZURE_AI_MODEL_DEPLOYMENT_NAME in all eval samples to match repo convention - Replace Unicode symbols with ASCII equivalents in all eval sample print statements to avoid cp1252 encoding errors on Windows Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update python/samples/03-workflows/evaluation/evaluate_workflow.py Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com> * Rename ADR 0020 to 0023 (foundry evals integration) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: alliscode <bentho@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com>
2026-03-31 15:53:06 +00:00