* Python: bump package versions for 1.2.1 release
PATCH bump (1.2.0 -> 1.2.1) for the released cohort. The release window
covers two PRs, no new public APIs:
- agent-framework-core: prevent inner_exception from being lost in
AgentFrameworkException (#5167)
- samples: add requirements.txt and .env.example to the a2a/ hosting
sample for pip-based setup (#5510)
Per lockstep convention, all 21 beta packages stamp 1.0.0b260428 and all
3 alpha packages stamp 1.0.0a260428, regardless of per-package code
churn. Every non-core package floor on agent-framework-core is raised to
>=1.2.1 to keep cohort signaling consistent. Date stamp reflects the
local (Asia) cut date 2026-04-28.
* Python: silence pyright unknown-type warnings in hosted-env detection
`azure.ai.agentserver.core` is probed at runtime via `importlib.util.find_spec`
and is not a declared dependency. The existing `# pyright: ignore[reportMissingImports]`
suppresses the missing-import warning, but at `lowest-direct` resolution pyright
still reports the imported symbol (`AgentConfig`) and its members (`from_env`,
`is_hosted`) as unknown, breaking `validate-dependency-bounds-test` for
`packages/core`.
Extend the existing ignore to cover `reportUnknownVariableType` on the import
and `reportUnknownMemberType` on the call site so the bounds check returns to
green. Behavior is unchanged.
Latent since #5455 (shipped in 1.2.0).
* Python: raise agent-framework-gemini lower bound to google-genai>=1.65.0
The Gemini chat client references several `google.genai.types` symbols
(`FileSearch`, `ThinkingLevel`, `SearchTypes`, `McpServer`,
`StreamableHttpTransport`, plus call-site keyword args `mcp_servers` and
`search_types`) that are not present at the lower bound of `google-genai>=1.0.0`.
At `lowest-direct` resolution this caused `validate-dependency-bounds-test` to
fail for `packages/gemini` with eleven `reportAttributeAccessIssue` /
`reportUnknownVariableType` errors.
Walking the upstream `google.genai.types` API:
- `GoogleMaps`, `AuthConfig`: present from 1.40.0
- `FileSearch`: introduced in 1.49.0
- `ThinkingLevel`: introduced in 1.55.0
- `SearchTypes`, `McpServer`, `StreamableHttpTransport`: introduced in 1.65.0
Bump the lower bound to 1.65.0 — the minimum version that exposes every symbol
the package actually uses. Keep the `<2.0.0` upper cap unchanged. With this
bump `validate-dependency-bounds-test` passes for both lower and upper
resolution scenarios across all 27 workspace packages.
Latent since #4847 (Gemini package introduction in 1.1.0); aggravated by
subsequent feature additions that pulled in newer `types.*` symbols.
* Python: add dependabot bumps to 1.2.1 CHANGELOG
Catalog the 15 dependabot dependency updates that merged on `upstream/main`
between python-1.2.0 and the 1.2.1 cut window under a new Changed section:
- Workspace dev/runtime deps: `rich`, `prek`, `python-multipart`, `pyasn1`,
`pytest` (ag-ui, devui, lab), `uv` (lab)
- Frontend deps: `vite` (devui, chatkit), `postcss` (devui, chatkit, handoff),
`picomatch` (devui, handoff)
CHANGELOG-only — no source or pyproject.toml changes. PRs themselves merged
upstream independently of this release branch and will be brought in via the
PR merge.
* Bump Python package versions for 1.2.0 release
Released tier bumps 1.1.1 -> 1.2.0 (core, openai, foundry, root) to
reflect additive public APIs landed since 1.1.0: functional workflow API
(#4238) and FunctionTool SKIP_PARSING sentinel (#5424). All beta packages
stamped 1.0.0b260424, alpha packages 1.0.0a260424. All 26 non-core
agent-framework-core floors raised to >=1.2.0,<2. CHANGELOG consolidates
the never-tagged 1.1.1 entries with the post-merge additions into [1.2.0].
* Update CHANGELOG footer links for 1.2.0
Advance [Unreleased] comparison base from python-1.1.0 to python-1.2.0
and add a [1.2.0] reference link comparing python-1.1.0...python-1.2.0
so the heading links resolve correctly.
* Fix CHANGELOG: restore [1.1.1] section and add proper [1.2.0]
Previous commit incorrectly renamed the [1.1.1] header to [1.2.0], which
wiped the historical 1.1.1 entries and wrongly attributed them to 1.2.0.
This restores [1.1.1] to its origin/main content and adds a new [1.2.0]
section above containing only the commits in python-1.1.1..HEAD:
- #4238 functional workflow API
- #5142 GitHub Copilot OpenTelemetry
- #2403 A2A bridge support
- #5070 oauth_consent_request events in Foundry clients
- #5447 FoundryAgent hosted agent sessions
- #5459 hosting server dependency upgrade + types
- #5389 AG-UI reasoning/multimodal parsing fix
- #5440 stop [TOOLBOXES] warning spam
- #5455 user agent prefix fix
Also corrects the [1.2.0] compare base to python-1.1.1 (not 1.1.0) and
adds the missing [1.1.1] reference link.
* Fix Foundry clients not surfacing oauth_consent_request events (#5054)
Override _parse_chunk_from_openai in both RawFoundryChatClient and
RawFoundryAgentChatClient to intercept response.output_item.added
events with item.type == 'oauth_consent_request'. The consent link
is validated (HTTPS required) and converted to
Content.from_oauth_consent_request, which the AG-UI layer already
knows how to emit as a CUSTOM event.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR review feedback for #5054 OAuth consent parsing
- Extract shared helper (try_parse_oauth_consent_event) to avoid
duplicated logic between RawFoundryChatClient and
RawFoundryAgentChatClient
- Use urllib.parse.urlparse() for HTTPS validation instead of
case-sensitive startswith check
- Sanitize log messages to avoid leaking consent_link tokens;
log only item id
- Add model=self.model to ChatResponseUpdate to match parent behavior
- Add assertions on role, raw_representation, and model in happy-path
tests
- Add test for empty-string consent_link
- Add test verifying non-oauth events delegate to super()
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Handle response.oauth_consent_requested top-level event (#5054)
Add support for the top-level response.oauth_consent_requested stream
event in addition to the response.output_item.added variant. The
service may emit either form; handle both so the consent link is
reliably surfaced.
Extract _validate_consent_link helper within _oauth_helpers.py to
reduce nesting.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback for #5054: Python: [Bug]: `FoundryAgent` (Responses API) Does Not Surface `oauth_consent_request` as a CUSTOM AG-UI Event
* Address review feedback: defensive getattr and dedicated helper tests (#5054)
- Use getattr(event, 'type', None) in try_parse_oauth_consent_event
for defensive access against malformed events without a type attribute
- Add test_oauth_helpers.py with unit tests for _validate_consent_link
and try_parse_oauth_consent_event covering edge cases:
- HTTPS URL with empty netloc (https:///path)
- Warning log messages for rejected consent links
- Event objects missing 'type' attribute
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback for #5054: Python: [Bug]: `FoundryAgent` (Responses API) Does Not Surface `oauth_consent_request` as a CUSTOM AG-UI Event
* Fix mypy: match _parse_chunk_from_openai signature with superclass
Add seen_reasoning_delta_item_ids parameter to _parse_chunk_from_openai
overrides in both RawFoundryChatClient and RawFoundryAgentChatClient to
match the updated superclass signature on main. Update super() calls and
test assertions accordingly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Evan Mattson <evan.mattson@microsoft.com>
* fixes to FoundryAgent to connect to new hosted agents
Co-authored-by: Copilot <copilot@github.com>
* fix mypy
Co-authored-by: Copilot <copilot@github.com>
* Python: remove Foundry service session helpers
Remove the public hosted-agent service session CRUD helpers from FoundryAgent and drop the related feature-stage inventory entry.
Update the hosted-agent sample to create and delete service sessions directly through the preview AIProjectClient APIs, and tighten a few test harnesses surfaced by full workspace validation.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix from merge
* fix hosted env detection
Co-authored-by: Copilot <copilot@github.com>
* reverted sample update
* fix tests and code
Co-authored-by: Copilot <copilot@github.com>
* remove aenter
* skipping some tests
Co-authored-by: Copilot <copilot@github.com>
---------
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Eduard van Valkenburg
·
2026-04-24 09:25:03 +00:00
* Python: Foundry: make response tool sanitizer internal, drop TOOLBOXES warning
sanitize_foundry_response_tool runs on every tool passed to the Foundry
Responses API, so its @experimental(TOOLBOXES) decorator was emitting a
[TOOLBOXES] ExperimentalWarning for any FoundryChatClient call, even when
no toolbox was involved. The function isn't in __all__ and has no external
callers. Rename to _sanitize_foundry_response_tool and drop the decorator;
the actual toolbox-facing public helpers remain gated.
* Python: Foundry: silence pyright on intentional cross-module private import
* Bump Python version for a release.
* Revert lockstep bumps on unchanged connectors
Per PR review: only connectors that changed (or whose published metadata
changed) should get new versions. Keeps released tier at 1.1.1, a2a/ag-ui
at 1.0.0b260422, foundry-hosting at 1.0.0a260422; reverts the 19 unchanged
betas and 2 unchanged alphas to 1.0.0b260421/1.0.0a260421. Reverts all 26
non-core agent-framework-core floors to >=1.1.0,<2 since no connector
actually depends on a 1.1.1 API or bug fix.
* Restore lockstep prerelease bumps and raise core floors to >=1.1.1
Reverses the lean-revert: all beta packages stamped 1.0.0b260423 and alpha
packages stamped 1.0.0a260423 (Asia date, matching release cut time). All
26 non-core packages raise agent-framework-core lower bound from >=1.1.0,<2
to >=1.1.1,<2 to signal the validated cohort for this release. CHANGELOG
date updated to 2026-04-23.
* fix(foundry): reconcile toolbox hosted-tool payloads with Responses API
* docs(foundry): update create_sample_toolbox docstring to reflect all tools created
* feat(evals): add ground_truth support for similarity evaluator
- Include expected_output as ground_truth in Foundry JSONL dataset rows
- Add ground_truth to item schema and data mapping for similarity evaluator
- Add expected_output parameter to evaluate_workflow
- Add similarity Pattern 3 to evaluate_agent and evaluate_workflow samples
- Add tests for ground_truth in dataset, schema, and evaluate_workflow
* Apply suggestions from code review
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* fix: wrap long line to satisfy ruff E501
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Add support for the Foundry Toolbox in MAF
Introduces a Foundry Toolbox integration: FoundryChatClient gains a
get_toolbox() helper plus select_toolbox_tools(), normalize_tools in
the core package flattens tool-collection wrappers (ToolboxVersionObject
and generic iterables, while leaving Pydantic BaseModel instances
alone), and the new agent_framework.foundry namespace re-exports the
toolbox helpers. Ships with unit tests, a sample, and a design doc.
azure-ai-projects is pinned to the public >=2.0.0,<3.0 range and the
lockfile resolves from public PyPI. The toolbox test module skips when
Toolbox* types are unavailable so CI stays green until the public 2.1.0
SDK lands. OMC tooling directories (.omc/, .omx/) are gitignored.
* Update to latest azure ai projects package
* Improve sample
* Rename ADR to 0025
* Update ADR
* Apply suggestion from @alliscode
Co-authored-by: Ben Thomas <ben.thomas@microsoft.com>
* Improve samples
* Update test
---------
Co-authored-by: Ben Thomas <ben.thomas@microsoft.com>
* Bump Python version to 1.1.0 for a release
* Fix changelog
* 1.0.1 instead of 1.1.0
* Update CHANGELOG.md
* update version and changelog
* Bump lower bounds
_prepare_options() now removes tools, tool_choice, and parallel_tool_calls
from run_options after injecting agent_reference. The Foundry API rejects
requests containing both fields. FunctionTools are still invoked client-side
by the function invocation layer.
Fixes#5087
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Evan Mattson <35585003+moonbox3@users.noreply.github.com>
* Raise clear handler registration error for unresolved TypeVar (#4943)
Detect unresolved TypeVar in message parameter annotations during handler
registration in both _validate_handler_signature (Executor) and
_validate_function_signature (FunctionExecutor). Raises a ValueError with
an actionable message recommending @handler(input=..., output=...) or
@executor(input=..., output=...) instead of letting TypeVar leak through
to a confusing TypeCompatibilityError during workflow edge validation.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback for #4943: reorder checks and harden function executor
- Move TypeVar check before validate_workflow_context_annotation in
_executor.py so users see the more actionable error first
- Wrap get_type_hints in try/except in _function_executor.py matching
the defensive pattern in _executor.py
- Repurpose duplicate test to cover bounded TypeVar rejection
- Add test_function_executor_allows_concrete_types for test symmetry
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Narrow get_type_hints except clause and add missing tests (#4943)
- Narrow `except Exception` to `except (NameError, AttributeError, RecursionError)`
in both _executor.py and _function_executor.py so unexpected failures in
get_type_hints are not silently swallowed.
- Add test_handler_unresolvable_annotation_raises to test_function_executor_future.py
exercising the except branch of get_type_hints in the function executor path.
- Add test_function_executor_rejects_bounded_typevar_in_message_annotation to
test_function_executor.py for parity with the Executor bounded TypeVar test.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add error ordering test for TypeVar vs WorkflowContext priority (#4943)
Add test_handler_typevar_error_takes_priority_over_context_error to verify
that when a handler has both a TypeVar message and an unannotated ctx, the
TypeVar error is raised first (the more actionable issue).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Python: Fix image content serialization sending null file_id to Foundry API
Omit file_id from input_image dict when not present instead of including
it as null, which Azure AI Foundry's stricter schema validation rejects.
* Python: Fix Foundry API rejecting rich content in function_call_output
Azure AI Foundry does not support list-format output in function_call_output
items. Add SUPPORTS_RICH_FUNCTION_OUTPUT flag (default True) to
RawOpenAIChatClient, set to False in RawFoundryChatClient so Foundry
falls back to string output for tool results with images/files.
Also omit file_id from input_image dicts when not set, since Foundry
rejects explicit nulls.
* Python: Surface rich tool content as user message when Foundry lacks support
When SUPPORTS_RICH_FUNCTION_OUTPUT is False, image/file items from tool
results are injected as a follow-up user message so the model can still
process the visual content via Foundry's supported user message format.
* Xfail Foundry image integration test for the meantime
---------
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Refactor Anthropic model option and provider clients
Rename the Anthropic client model option from model_id to model, add provider-specific Anthropic wrappers for Foundry, Bedrock, and Vertex, and expose them through the Anthropic, Foundry, Amazon, and Google namespaces. Update core option handling, docs, samples, and tests accordingly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix Anthropic skills sample typing
Cast the Anthropic beta client to Any in the skills sample so the pre-commit sample pyright check no longer fails on beta skills and files endpoints that are not exposed by the current SDK stubs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* undo sample mypy
* Retry CI after transient external failures
Retrigger PR validation after an unrelated Copilot review workflow SAML failure and a transient external tau2 git fetch failure in the Windows Python test setup.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback on model option merging
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address Anthropic compatibility review feedback
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* moved all to `model`
* fixes for azure ai search
* Python: standardize remaining sample env var names
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Python: fix foundry-local pyright compatibility
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* updated env vars in cicd
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Eduard van Valkenburg
·
2026-04-01 19:00:18 +00:00
* Foundry Evals integration for Python
Merged and refactored eval module per Eduard's PR review:
- Merge _eval.py + _local_eval.py into single _evaluation.py
- Convert EvalItem from dataclass to regular class
- Rename to_dict() to to_eval_data()
- Convert _AgentEvalData to TypedDict
- Simplify check system: unified async pattern with isawaitable
- Parallelize checks and evaluators with asyncio.gather
- Add all/any mode to tool_called_check
- Fix bool(passed) truthy bug in _coerce_result
- Remove deprecated function_evaluator/async_function_evaluator aliases
- Remove _MinimalAgent, tighten evaluate_agent signature
- Set self.name in __init__ (LocalEvaluator, FoundryEvals)
- Limit FoundryEvals to AsyncOpenAI only
- Type project_client as AIProjectClient
- Remove NotImplementedError continuous eval code
- Add evaluation samples in 02-agents/ and 03-workflows/
- Update all imports and tests (167 passing)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: resolve mypy redundant-cast errors while keeping pyright happy
Use cast(list[Any], x) with type: ignore[redundant-cast] comments to
satisfy both mypy (which considers casting Any redundant) and pyright
strict mode (which needs explicit casts to narrow Unknown types).
Also fix evaluator decorator check_name type annotation to be
explicitly str, resolving mypy str|Any|None mismatch.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: CI failures — pyupgrade, evaluator overloads, sample API, reset attr
- Apply pyupgrade: Sequence from collections.abc, remove forward-ref quotes
- Add @overload signatures to evaluator() for proper @evaluator usage
- Fix evaluate_workflow sample to use WorkflowBuilder(start_executor=) API
- Fix _workflow.py executor.reset() to use getattr pattern for pyright
- Remove unused EvalResults forward-ref string in default_factory lambda
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: skip gRPC-dependent observability test
The test_configure_otel_providers_with_env_file_and_vs_code_port test
triggers gRPC OTLP exporter creation, but the grpc dependency is
optional and not installed by default. Add skipif decorator matching
the pattern used by all other gRPC exporter tests in the same file.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: add nosec B101 for bandit assert check
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* style: align eval samples with repo conventions
- Move module docstrings before imports (after copyright header)
- Add -> None return type to all main() and helper functions
- Fix line-too-long in multiturn sample conversation data
- Add Workflow import for typed return in all_patterns_sample
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR review feedback: async fixes, sample bugs, deprecation warnings
- Simplify _ensure_async_result to direct await (async-only clients)
- Replace get_event_loop() with get_running_loop()
- Narrow _fetch_output_items exception handling to specific types
- Add warning log when _filter_tool_evaluators falls back to defaults
- Add DeprecationWarning to options alias in Agent.__init__
- Add DeprecationWarning to evaluate_response()
- Rename raw key to _raw_arguments in convert_message fallback
- Fix evaluate_agent_sample.py: replace evals.select() with FoundryEvals()
- Fix evaluate_multiturn_sample.py: use Message/Content/FunctionTool types
- Fix evaluate_workflow_sample.py: replace evals.select() with FoundryEvals()
- Update test mocks to use AsyncMock for awaited API calls
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add test coverage for review feedback items
- Add num_repetitions=2 positive test verifying 2×items and 4 agent calls
- Add _poll_eval_run tests: timeout, failed, and canceled paths
- Add evaluate_traces tests: validation error, response_ids path, trace_ids path
- Add evaluate_foundry_target happy-path test with target/query verification
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix ruff ISC004 lint error and apply formatter
- Wrap implicit string concatenation in parens in evaluate_multiturn_sample.py
- Apply ruff formatter to 6 other files with minor formatting drift
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove core type changes (extracted to fix/workflow-stale-session branch)
Reverts changes to _agents.py, _agent_executor.py, and _workflow.py
back to upstream/main. These fixes are now in a separate PR.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR review round 2: bugs, tests, and architecture
Code fixes:
- Fix _normalize_queries inverted condition (single query now replicates
to match expected_count)
- Fix substring match bug: 'end' in 'backend' matched; use exact set
lookup for executor ID filtering
- Fix used_available_tools sample: tool_definitions→tools param, use
FunctionTool attribute access instead of dict .get()
- Add None-check in _resolve_openai_client for misconfigured project
- Add Returns section to evaluate_workflow docstring
- Cache inspect.signature in @evaluator wrapper (avoid per-item reflection)
Architecture:
- Extract _evaluate_via_responses as module-level helper; evaluate_traces
now calls it directly instead of creating a FoundryEvals instance
- Move Foundry-specific typed-content conversion out of core to_eval_data;
core now returns plain role/content dicts, FoundryEvals applies
AgentEvalConverter in _evaluate_via_dataset
Tests:
- evaluate_response() deprecation warning emission and delegation
- num_repetitions > 1 with expected_output and expected_tool_calls
- Mock output_items.list in test_evaluate_calls_evals_api
- Update to_eval_data assertions for plain-dict format
- Unknown param error now raised at @evaluator decoration time
Skipped (separate PR): executor reset loop, xfail removal, options alias
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix CI: revert test_full_conversation, fix pyright errors
- Revert test_full_conversation.py to upstream/main (the session
preservation test was incorrectly changed to assert clearing)
- Fix pyright reportUnnecessaryComparison on get_openai_client() None
check by adding ignore comment
- Fix pyright reportPrivateUsage: add public EvalItem.split_messages()
method and use it in FoundryEvals._evaluate_via_dataset instead of
accessing private _split_conversation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR review round 3: reliability, test gaps, cleanup
- Add try/except guard for non-numeric score in _coerce_result
- Add poll_interval minimum bound (0.1s) to prevent tight loops
- Add runtime async client check in _resolve_openai_client
- Remove _ensure_async_result wrapper (10 call sites → direct await)
- Better error message when queries provided without agent
- Import-time asserts for evaluator set consistency
- Remove 28 redundant @pytest.mark.asyncio decorators
- Add doc note about _raw_arguments sensitive data
- Tests: tool_called_check mode=any, _normalize_queries branches,
_extract_result_counts paths, _extract_per_evaluator, bare check
via evaluate_agent, output_items assertion, modulo wrapping,
async client check, queries-without-agent error
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix CI: ruff S101 assert, pyright and mypy arg-type errors
- Replace module-level assert with if/raise for evaluator set
consistency checks (ruff S101 disallows bare assert)
- Add type: ignore[arg-type] and pyright: ignore[reportArgumentType]
on OpenAI SDK evals API calls that pass dicts where typed params
are expected (SDK accepts dicts at runtime)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR review round 4: bugs, reliability, test fixes
- Fix all_passed ignoring parent result_counts when sub_results present
- Fix _extract_tool_calls: parse string arguments via json.loads before
falling back to None (real LLM responses use string arguments)
- Sanitize _raw_arguments to '[unparseable]' to avoid leaking sensitive
tool-call data to external evaluation services
- Add NOTE comment on to_eval_data message serialization dropping
non-text content (tool calls, results)
- Eliminate double conversation split in _evaluate_via_dataset: build
JSONL dicts directly from split_messages + AgentEvalConverter
- Raise poll_interval floor from 0.1s to 1.0s to prevent rate-limit
exhaustion
- Fix MagicMock(name=...) bug in test: sets display name not .name attr
- Fix mock_output_item.sample: use MagicMock object instead of dict so
_fetch_output_items exercises error/usage/input/output extraction
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR review round 5: reliability, docs, test coverage
Code fixes:
- Move import-time RuntimeError checks to unit tests (avoids breaking
imports for all users on developer set-drift mistake)
- _filter_tool_evaluators now raises ValueError when all evaluators
require tools but no items have tools (was silently substituting)
- Add poll_interval upper bound (60s) to prevent single-iteration sleep
- Log exc_info=True in _fetch_output_items for debugging API changes
- Fix evaluate() docstring: remove claim about Responses API optimization
- Validate target dict has 'type' key in evaluate_foundry_target
- Document to_eval_data() limitation: non-text content is omitted
Tests:
- TestEvaluatorSetConsistency: verify _AGENT/_TOOL subsets of _BUILTIN
- TestEvaluateTracesAgentId: agent_id-only path with lookback_hours
- TestFilterToolEvaluatorsRaises: ValueError on all-tool no-items
- TestEvaluateFoundryTargetValidation: target without 'type' key
- Assert items==[] on failed/canceled poll results
- Mock output_items.list in response_ids test for full flow
- TestAllPassedSubResults: result_counts=None + sub_results delegation
and parent failures override sub_results
- TestBuildOverallItemEmpty: empty workflow outputs returns None
Skipped r5-07 (_raw_arguments length hint): marginal debugging value,
could leak content size information.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix error message: evaluate_responses() → evaluate_traces(response_ids=...)
The referenced function doesn't exist; the correct API is
evaluate_traces(response_ids=...) from the azure-ai package.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove dead to_eval_data() method, fix docstring claims
- Remove to_eval_data() from EvalItem (dead code after r4-05 JSONL refactor)
- Migrate 15 tests from to_eval_data() to split_messages()
- Update sample to use split_messages() + Message properties
- Remove unimplemented Responses API optimization docstring claim
- Update split_messages() docstring to not reference removed method
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Reduce default eval timeout from 600s to 180s (3 minutes)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove dead _evaluate_via_responses method from FoundryEvals
The method was never called — evaluate() uses _evaluate_via_dataset,
and evaluate_traces() calls _evaluate_via_responses_impl directly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Revert unrelated formatting changes to get-started samples
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix pyright: remove phantom FoundryMemoryProvider import, apply ruff format
- Remove import of non-existent _foundry_memory_provider module
(incorrectly kept during rebase conflict resolution)
- Apply ruff formatter to test_local_eval.py and get-started samples
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix eval samples: use FoundryChatClient for Agent()
The upstream provider-leading client refactor (#4818) made client=
a required parameter on Agent(). Update the three getting-started
eval samples to use FoundryChatClient with FOUNDRY_PROJECT_ENDPOINT,
matching the standard pattern from 01-get-started samples.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Simplify self-reflection sample using FoundryEvals
Replace ~80 lines of manual OpenAI evals API code (create_eval,
run_eval, manual polling, raw JSONL params) with FoundryEvals:
- evaluate_groundedness() uses FoundryEvals.evaluate() with EvalItem
- Remove create_openai_client(), create_eval(), run_eval() functions
- Remove openai SDK type imports (DataSourceConfigCustom, etc.)
- run_self_reflection_batch creates FoundryEvals instance once,
reuses it for all iterations across all prompts
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update eval samples to FoundryChatClient and FOUNDRY_PROJECT_ENDPOINT
- Migrate all foundry_evals samples from AzureOpenAIResponsesClient to FoundryChatClient
- Update env var from AZURE_AI_PROJECT_ENDPOINT to FOUNDRY_PROJECT_ENDPOINT
- Use AzureCliCredential consistently across all samples
- Fix README.md: correct function names (evaluate_dataset -> FoundryEvals.evaluate, evaluate_responses -> evaluate_traces)
- Update self_reflection .env.example and README.md
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix lint errors in eval samples (E501, ASYNC240, formatting)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove evaluate_all_patterns_sample.py (redundant with focused samples)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix async credential mismatch: use azure.identity.aio for async AIProjectClient
AIProjectClient from azure.ai.projects.aio requires an async credential.
Switch all foundry_evals samples from azure.identity.AzureCliCredential
to azure.identity.aio.AzureCliCredential. Also pass project_client to
FoundryChatClient instead of duplicating endpoint+credential.
Close credential in self_reflection sample to avoid resource leak.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Revert test_observability.py to upstream/main (not our test)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address moonbox3 review: sphinx docstrings, pagination, isinstance check
- Convert all Example:: / Typical usage:: code blocks to .. code-block:: python
format matching codebase convention (both _evaluation.py and _foundry_evals.py)
- Add async pagination in _fetch_output_items via async for (handles large result sets)
- Replace hasattr(__aenter__) with isinstance(client, AsyncOpenAI) in _resolve_openai_client
- Move AsyncOpenAI import from TYPE_CHECKING to runtime (needed for isinstance)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix test failures and address remaining moonbox3 review comments
- Fix tests: use MagicMock(spec=AsyncOpenAI) for project_client mocks
(isinstance check now requires proper type, not duck-typing)
- Fix tests: replace mock_page.__iter__ with _AsyncPage helper for async for
- Fix evaluate_response: auto-extract queries from response messages when
query is not provided (previously always raised ValueError)
- Add debug logging when skipping internal _-prefixed executor IDs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address Tao's PR review comments on Foundry Evals
- T1: Add comment explaining builtin.* pass-through in _resolve_evaluator
- T2: Add comment referencing OpenAI evals API for testing_criteria dict
- T3: Document Mustache-style {{item.*}} template placeholders
- T4: Document poll loop 60s sleep upper bound rationale
- T5: Narrow run type to RunRetrieveResponse, use typed field access
instead of vars()/getattr dance in _extract_result_counts and
_extract_per_evaluator; use run.error and run.report_url directly
- T6: Clarify openai_client docstring re: Azure Foundry endpoint
- T8: Remove misleading empty expected_tool_calls from sample
- Update tests to match real SDK PerTestingCriteriaResult shape
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove unnecessary Any union from run type annotations
RunRetrieveResponse is the correct type — no backward compat needed
for a brand new feature.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Accept FoundryChatClient instead of raw AsyncOpenAI
FoundryEvals now takes client: FoundryChatClient as its primary
parameter instead of openai_client: AsyncOpenAI. The builtin.*
evaluators require a Foundry endpoint, so the type should reflect that.
- FoundryEvals.__init__: client: FoundryChatClient replaces openai_client
- evaluate_traces / evaluate_foundry_target: same change
- _resolve_openai_client: extracts .client from FoundryChatClient
- project_client fallback retained for standalone functions
- All samples updated to construct FoundryChatClient and pass as client=
- Tests updated (openai_client= → client=)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove implicit 60s upper bound on poll interval
If a developer sets a higher poll_interval, respect it. Only clamp
to remaining time and enforce a 1s minimum for rate-limit protection.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove 1s floor on poll interval — let the developer control it
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update python/samples/05-end-to-end/evaluation/foundry_evals/.env.example
Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com>
* Update python/samples/02-agents/evaluation/evaluate_agent.py
Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com>
* Address eavanvalkenburg review (round 2) on Python eval PR
- Rename model_deployment -> model across FoundryEvals and all samples
- Make model param optional, resolves from client.model
- Convert EvalResults from dataclass to regular class
- Remove deprecated evaluate_response() function
- Refactor splitters: BUILT_IN_SPLITTERS dict + standalone functions
- Change per_turn_items from classmethod to staticmethod
- Simplify EvalCheck type alias to use Awaitable[CheckResult]
- Remove errored property from EvalResults
- Remove default value from Evaluator protocol eval_name
- Rename assert_passed -> raise_for_status, add EvalNotPassedError
- Type agent param as SupportsAgentRun | None
- Fix Arguments docstring
- Update __init__.py exports
- Update all tests and samples
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Move FoundryEvals to foundry package, split tool eval sample
- Move _foundry_evals.py from azure-ai to foundry package
- Move test_foundry_evals.py to foundry/tests/
- Update lazy re-exports in agent_framework.foundry namespace
- Update .pyi type stubs
- All samples now import from agent_framework.foundry
- Split tool-call evaluation into evaluate_tool_calls_sample.py
- Fix all_passed to check errored count from result_counts
- Fix raise_for_status to include errored item details
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Auto-create FoundryChatClient from env vars when no client provided
FoundryEvals() now works zero-config when FOUNDRY_PROJECT_ENDPOINT and
FOUNDRY_MODEL environment variables are set. Auto-creates a FoundryChatClient
under the hood, matching the established env var pattern.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix pyright errors: remove dead _normalize_queries, suppress EvalAPIError check
- Remove unused _normalize_queries function and its tests
- Add pyright ignore for EvalAPIError None check (defensive guard)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Support multimodal image content in eval pipeline
Add image (data/uri) content handling to AgentEvalConverter.convert_message()
so that Content.from_data() and Content.from_uri() image payloads are
preserved as input_image parts in the Foundry evaluator format.
- Handle Content type='data' and type='uri' → emit input_image parts
- Add 6 unit tests for image content through convert_message/convert_messages
- Add integration test verifying images flow through EvalItem → JSONL path
- Add evaluate_multimodal.py sample demonstrating local image eval
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address remaining review comments
- Fix project_client docstring to say async-only (not sync/async)
- Add builtin evaluator name validation warning in _resolve_evaluator
- Replace getattr with typed attribute access in _poll_eval_run,
_extract_result_counts, _extract_per_evaluator, _fetch_output_items
- Remove cast import from _foundry_evals (no longer needed)
- Tighten _coerce_result: honour explicit 'passed' when both 'score'
and 'passed' are present; remove performative cast
- Fix self_reflection sample: add env file existence check
- Fix traces sample: correct Pattern 2 section label
- Update all Foundry eval samples to FoundryChatClient + FOUNDRY_MODEL
(remove AIProjectClient + AZURE_AI_MODEL_DEPLOYMENT_NAME pattern)
- Add eval_name and OpenAI client docs to FoundryEvals docstring
- Update test mocks to match typed SDK objects (_MockResultCounts)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix ruff lint errors (E501, SIM108, SIM102)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix pyright errors: type-narrow dict to dict[str, Any], add ignore comments
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Replace ConversationSplitter type alias with Protocol
ConversationSplitter is now a runtime-checkable Protocol with a named
'conversation' parameter, making the expected signature self-documenting.
ConversationSplit enum members gain a __call__ method so they satisfy
the protocol directly -- ConversationSplit.LAST_TURN(conversation) works.
This simplifies _split_conversation from an isinstance dispatch to a
single split(conversation) call.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Standardize on AZURE_AI_MODEL_DEPLOYMENT_NAME and fix Unicode in samples
- Replace FOUNDRY_MODEL with AZURE_AI_MODEL_DEPLOYMENT_NAME in all
eval samples to match repo convention
- Replace Unicode symbols with ASCII equivalents in all eval sample
print statements to avoid cp1252 encoding errors on Windows
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update python/samples/03-workflows/evaluation/evaluate_workflow.py
Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com>
* Rename ADR 0020 to 0023 (foundry evals integration)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: alliscode <bentho@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com>
* Small fix in dotnet conformance tests
* Added CopilotStudioAgent implementation
* Added examples
* Updated package README
* Small fixes
* Small improvements
* Fixed dotnet tests
* Add unit tests
* Updated tests
* Small updates
* Small test fixes
* Revert "Small test fixes"
This reverts commit 983ac44a70.
* Small fixes in documentation
* Updated test configuration
* Revert "Updated test configuration"
This reverts commit 2a16fea815.
* Small fix
* Reverted TODO item
* Small suppressions
* More fixes
* Small fixes
* Fixed tests
* Removed disallow_any_unimported rule in all packages
* Fixes
* test with stack and simplified names
* quick demo of agent decorator
* moved builder to protocol to enhance functionality
* undid chatclientAgent -> agent rename
* one more
* reverted AIAgent rename
* final reverts
* fixed foundry import
* revert changes
* streamlined otel and fcc decorators
* cleanup of telemetry
* further refinement
* lots of updates
* fixed typing
* fix for mypy
* added input and output atttributes
* fix import
* initial work on baking in otel
* major update to telemetry
* final fixes after rename
* fix
* fix test
* updated tests
* fix for tests
* fixes for tests
* updated based on comments
* removed agent decorator
* fix for Python: ServiceResponseException when using multiple tools
Fixes#649
* addressed comments
* fix tests
* fix tests
* fix tools tests
* fix for conversation_id in assistants client
* fix responses test
* fix tests and mypy
* updated test
* foundry fix
---------
Co-authored-by: Chris <66376200+crickman@users.noreply.github.com>
Eduard van Valkenburg
·
2025-09-10 14:52:42 +00:00
* initial work on User Approval (and hosted mcp to validate)
* small update to the comments in the sample
* enable local MCP tools in chatClient get methods
* working streaming and improved setup
* fix for pyright
* updated create_approval -> create_response method
* added tests
* updated HostedMcpTool and addressed feedback
* update type name
* naming updates
* small docstring update
* mypy fix
* fixes and updates
* fixes for responses
* fix int tests
* removed broken tests
* updated test running
* removed specific content check on websearch
* increased timeout
* split slow foundry test
* don't parallel run samples
* add dist load to unit tests
---------
Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>
Eduard van Valkenburg
·
2025-09-10 13:37:34 +00:00
* Updates for message store support
* Added unit tests
* Added suspend-resume example
* Added example with custom chat message store
* Small fix
* Addressed PR feedback
* Renaming and documentation
* More renaming
* Addressed more PR feedback
* Small fixes in Foundry chat client and examples
* Small update
* Addressed PR feedback
* Increased timeout for Azure tests
* openai updates
* rebuild of openai structure
* updated responses structure
* renamed sample
* added file id support to code interpreter
* added hosted file ids to code interpretor
* mypy fixes
* removed default az cred from codebase
* updated agent name setup
* added kwargs to entra methods
* and further kwargs
* extra comment
* updated all samples
* readded custom get methods for responses
* updated int tests with ad credential
* missed one
Eduard van Valkenburg
·
2025-08-12 06:14:22 +00:00
* small fixes in foundry
* other samples updated
* make it optional
* added instructions and response format to create agent
* mypy fix
* shortened main readme and improved python readme
Eduard van Valkenburg
·
2025-08-04 08:13:44 +00:00
* Initial version of assistant client
* More updates to assistant client
* Finished assistant chat client implementation
* Small fixes and basic example
* Added code interpreter example
* More examples
* Added chat client example
* Small fixes
* Added tests
* Enabled telemetry
* Small fix
* Removed files temporarily
* Revert "Removed files temporarily"
This reverts commit 5cdfa0d299.
* Small fixes
* Addressed PR feedback
* Fixed tests
* Small update
Evan Mattson ·
Giles Odigwe ·
Eduard van Valkenburg ·
Tao Chen ·
chetantoshniwal ·
Matt Van Horn ·
Copilot ·
Ben Thomas ·
Dmytro Struk ·