* Python: Skip get_final_response in OTel _finalize_stream when stream errored
When a streaming error occurs, _finalize_stream (a cleanup hook registered by
AgentTelemetryLayer) was unconditionally calling get_final_response(), which
triggers all registered result hooks including after_run context providers.
This caused providers to fire incorrectly on error paths.
Guard against this by checking result_stream._consumed: True only after
StopAsyncIteration (normal completion), False when an exception was raised.
The fix applies to both the chat client and agent telemetry layers.
Closes#5231
* Python: Expose consumed/stream_error on ResponseStream and capture error in OTel span
Address Copilot review feedback on #5232:
- Add `_stream_error: Exception | None` to ResponseStream, set in __anext__'s
except branch so cleanup hooks can inspect the failure.
- Expose public `consumed` and `stream_error` properties to avoid coupling
observability.py to private stream internals.
- Update both _finalize_stream closures (chat and agent layers) to use the
public properties and call capture_exception() with the stream error before
returning early, ensuring the OTel span records the failure rather than
closing silently.
* Python: Address Copilot review feedback on stream error handling
- Use stream_error is not None as the guard in _finalize_stream instead of
not consumed, so the early-return path is keyed precisely to actual errors
rather than any non-normal completion state.
- Clear _stream_error after _run_cleanup_hooks() completes to avoid retaining
the exception traceback (and any large object graphs it references) on the
stream instance beyond the cleanup phase.
* Python: Remove consumed/stream_error properties, use private attrs directly
Per review feedback: since observability.py and _types.py are in the same
package, accessing _stream_error directly is fine and the public properties
are unnecessary.
* Python: Fix Pyright reportPrivateUsage via inline ignore comments
Keep _stream_error private (consistent with rest of ResponseStream), and
suppress reportPrivateUsage at the call sites in observability.py with
inline pyright: ignore comments — access is intentional within the package.
* AG-UI deterministic state updates from tool results
* fix(ag-ui): address PR #5201 review comments
1. Add missing AGUIEventConverter, AGUIHttpService, __version__ to
_IMPORTS in core ag_ui lazy-export list to match the .pyi stub.
2. Coalesce predictive and deterministic state snapshots into a single
StateSnapshotEvent when both mechanisms are active on the same tool
result, reducing redundant snapshot traffic.
3. Update state_update() docstring to clarify that a predictive snapshot
may be emitted before the deterministic one when predict_state_config
is active.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Bump Python version to 1.1.0 for a release
* Fix changelog
* 1.0.1 instead of 1.1.0
* Update CHANGELOG.md
* update version and changelog
* Bump lower bounds
* Harden Python checkpoint persistence defaults
Add RestrictedUnpickler to _checkpoint_encoding.py that limits which
types may be instantiated during pickle deserialization. By default
FileCheckpointStorage now uses the restricted unpickler, allowing only:
- Built-in Python value types (primitives, datetime, uuid, decimal,
collections, etc.)
- All agent_framework.* internal types
- Additional types specified via the new allowed_checkpoint_types
parameter on FileCheckpointStorage
This narrows the default type surface area for persisted checkpoints
while keeping framework-owned scenarios working without extra
configuration. Developers can extend the allowed set by passing
"module:qualname" strings to allowed_checkpoint_types.
The decode_checkpoint_value function retains backward-compatible
unrestricted behavior when called without the new allowed_types kwarg.
Fixes#4894
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: resolve mypy no-any-return error in checkpoint encoding
Add explicit type annotation for super().find_class() return value
to satisfy mypy's no-any-return check.
Fixes#4894
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Simplify find_class return in _RestrictedUnpickler (#4894)
Remove unnecessary intermediate variable and apply # noqa: S301 # nosec
directly on the super().find_class() call, matching the established
pattern used on the pickle.loads() call in the same file.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback for #4894: Python: Harden Python checkpoint persistence defaults
* Restore # noqa: S301 on line 102 of _checkpoint_encoding.py (#4894)
The review feedback correctly identified that removing the # noqa: S301
suppression from the find_class return statement would cause a ruff S301
lint failure, since the project enables bandit ("S") rules. This
restores consistency with lines 82 and 246 in the same file.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback for #4894: Python: Harden Python checkpoint persistence defaults
* Address PR review comments on checkpoint encoding (#4894)
- Move module docstring to proper position after __future__ import
- Fix find_class return type annotation to type[Any]
- Add missing # noqa: S301 pragma on find_class return
- Improve error message to reference both allowed_types param and
FileCheckpointStorage.allowed_checkpoint_types
- Add -> None return annotation to FileCheckpointStorage.__init__
- Replace tempfile.mktemp with TemporaryDirectory in test
- Replace contextlib.suppress with pytest.raises for precise assertion
- Remove unused contextlib import
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR #4941 review comments: fix docstring position and return type
- Move module docstring before 'from __future__' import so it populates
__doc__ (comment #4)
- Change find_class return annotation from type[Any] to type to avoid
misleading callers about non-type returns like copyreg._reconstructor
(comment #2)
Comments #1, #3, #5, #6, #7, #8 were already addressed in the current code.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback for #4894: review comment fixes
* fix: use pickle.UnpicklingError in RestrictedUnpickler and improve docstring (#4894)
- Change _RestrictedUnpickler.find_class to raise pickle.UnpicklingError
instead of WorkflowCheckpointException, since it is pickle-level concern
that gets wrapped by the caller in _base64_to_unpickle.
- Remove now-unnecessary WorkflowCheckpointException re-raise in
_base64_to_unpickle (pickle.UnpicklingError is caught by the generic
except Exception handler and wrapped).
- Expand decode_checkpoint_value docstring to show a concrete example of
the module:qualname format with a user-defined class.
- Add regression test verifying find_class raises pickle.UnpicklingError.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: address PR #4941 review comments for checkpoint encoding
- Comment 1 (line 103): Already resolved in prior commit — _RestrictedUnpickler
now raises pickle.UnpicklingError instead of WorkflowCheckpointException.
- Comment 2 (line 140): Add concrete usage examples to decode_checkpoint_value
docstring showing both direct allowed_types usage and FileCheckpointStorage
allowed_checkpoint_types usage. Rename 'SafeState' to 'MyState' across all
docstrings for consistency, making it clear this is a user-defined class name.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: replace deprecated 'builtin' repo with pre-commit-hooks in pre-commit config
pre-commit 4.x no longer supports 'repo: builtin'. Merge those hooks into
the existing pre-commit-hooks repo entry.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* style: apply pyupgrade formatting to docstring example
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: resolve pre-commit hook paths for monorepo git root
The poe-check and bandit hooks referenced paths relative to python/
but pre-commit runs hooks from the git root (monorepo root). Fix
poe-check entry to cd into python/ first, and update bandit config
path to python/pyproject.toml.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix pre-commit config paths for prek --cd python execution
Revert bandit config path from 'python/pyproject.toml' to 'pyproject.toml'
and poe-check entry from explicit 'cd python' wrapper to direct invocation,
since prek --cd python already sets the working directory to python/.
Also apply ruff formatting fixes to cosmos checkpoint storage files.
Fixes#4894
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: add builtins:getattr to checkpoint deserialization allowlist
Pickle uses builtins:getattr to reconstruct enum members (e.g.,
WorkflowMessage.type which is a MessageType enum). Without it in the
allowlist, checkpoint roundtrip tests fail with
WorkflowCheckpointException.
Fixes#4894
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback for #4894: review comment fixes
---------
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Guard against empty text in _parse_structured_response_value (#5145)
When using response_format with background=True (Responses API), polling
an in-progress response produces empty text. _parse_structured_response_value
unconditionally passed this to model_validate_json/json.loads, causing
ValidationError or JSONDecodeError.
Add an early return of None when text is empty, matching the existing
guard for response_format=None. This allows .value to safely return None
for in-progress background responses.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Python: Fix `response_format` crash on background polling with empty text
Fixes#5145
---------
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Eduard van Valkenburg
·
2026-04-07 22:26:03 +00:00
* Raise clear handler registration error for unresolved TypeVar (#4943)
Detect unresolved TypeVar in message parameter annotations during handler
registration in both _validate_handler_signature (Executor) and
_validate_function_signature (FunctionExecutor). Raises a ValueError with
an actionable message recommending @handler(input=..., output=...) or
@executor(input=..., output=...) instead of letting TypeVar leak through
to a confusing TypeCompatibilityError during workflow edge validation.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback for #4943: reorder checks and harden function executor
- Move TypeVar check before validate_workflow_context_annotation in
_executor.py so users see the more actionable error first
- Wrap get_type_hints in try/except in _function_executor.py matching
the defensive pattern in _executor.py
- Repurpose duplicate test to cover bounded TypeVar rejection
- Add test_function_executor_allows_concrete_types for test symmetry
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Narrow get_type_hints except clause and add missing tests (#4943)
- Narrow `except Exception` to `except (NameError, AttributeError, RecursionError)`
in both _executor.py and _function_executor.py so unexpected failures in
get_type_hints are not silently swallowed.
- Add test_handler_unresolvable_annotation_raises to test_function_executor_future.py
exercising the except branch of get_type_hints in the function executor path.
- Add test_function_executor_rejects_bounded_typevar_in_message_annotation to
test_function_executor.py for parity with the Executor bounded TypeVar test.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add error ordering test for TypeVar vs WorkflowContext priority (#4943)
Add test_handler_typevar_error_takes_priority_over_context_error to verify
that when a handler has both a TypeVar message and an unannotated ctx, the
TypeVar error is raised first (the more actionable issue).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Python: Fix image content serialization sending null file_id to Foundry API
Omit file_id from input_image dict when not present instead of including
it as null, which Azure AI Foundry's stricter schema validation rejects.
* Python: Fix Foundry API rejecting rich content in function_call_output
Azure AI Foundry does not support list-format output in function_call_output
items. Add SUPPORTS_RICH_FUNCTION_OUTPUT flag (default True) to
RawOpenAIChatClient, set to False in RawFoundryChatClient so Foundry
falls back to string output for tool results with images/files.
Also omit file_id from input_image dicts when not set, since Foundry
rejects explicit nulls.
* Python: Surface rich tool content as user message when Foundry lacks support
When SUPPORTS_RICH_FUNCTION_OUTPUT is False, image/file items from tool
results are injected as a follow-up user message so the model can still
process the visual content via Foundry's supported user message format.
* Xfail Foundry image integration test for the meantime
---------
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add deduplication to `prepend_instructions_to_messages()` to skip
instructions that are already present as leading messages with the
same role and text. This prevents duplicate system messages when
instructions are injected by multiple layers (e.g. Agent + chat client).
Fixes#5049
* Refactor Anthropic model option and provider clients
Rename the Anthropic client model option from model_id to model, add provider-specific Anthropic wrappers for Foundry, Bedrock, and Vertex, and expose them through the Anthropic, Foundry, Amazon, and Google namespaces. Update core option handling, docs, samples, and tests accordingly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix Anthropic skills sample typing
Cast the Anthropic beta client to Any in the skills sample so the pre-commit sample pyright check no longer fails on beta skills and files endpoints that are not exposed by the current SDK stubs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* undo sample mypy
* Retry CI after transient external failures
Retrigger PR validation after an unrelated Copilot review workflow SAML failure and a transient external tau2 git fetch failure in the Windows Python test setup.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback on model option merging
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address Anthropic compatibility review feedback
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* moved all to `model`
* fixes for azure ai search
* Python: standardize remaining sample env var names
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Python: fix foundry-local pyright compatibility
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* updated env vars in cicd
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Eduard van Valkenburg
·
2026-04-01 19:00:18 +00:00
* Fix _add_text_reasoning_content dropping id during coalescing (#4852)
Preserve the id field (rs_* identifier) when coalescing text_reasoning
Content objects by passing id=self.id or other.id to the Content
constructor. This fixes the encrypted reasoning round-trip where the
missing id prevented _prepare_content_for_openai from including it in
the serialized reasoning item.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Python: Fix `_add_text_reasoning_content` to preserve `id` during coalescing
Fixes#4852
* Raise AdditionItemMismatch on conflicting text_reasoning ids (#4852)
Detect when both operands have different non-empty ids during
text_reasoning Content coalescing and raise AdditionItemMismatch
instead of silently keeping one. This prevents mis-associating
encrypted_content during round-trips.
Also adds tests for conflicting ids and the neither-has-id edge case.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback for #4852: Python: [Bug]: Content._add_text_reasoning_content drops id during coalescing, breaking encrypted reasoning round-trip
* test fix
---------
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Python: Add header_provider to MCPStreamableHTTPTool (#4808)
Add a header_provider callback parameter to MCPStreamableHTTPTool that
enables injecting dynamic per-request HTTP headers from runtime kwargs
(originating from FunctionInvocationContext.kwargs set in agent middleware).
The implementation uses contextvars and httpx event hooks to ensure headers
are task-local and safe for concurrent tool calls:
- header_provider receives the runtime kwargs dict and returns headers
- call_tool sets a ContextVar before delegating to MCPTool.call_tool
- An httpx request event hook reads from the ContextVar and injects headers
Example usage:
mcp_tool = MCPStreamableHTTPTool(
name="web-api",
url="https://api.example.com/mcp",
header_provider=lambda kwargs: {
"X-Auth-Token": kwargs.get("auth_token", ""),
},
)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback for #4808: Python: [Bug]: Unable to pass AgentContext to MCPStreamableHTTPTool
* Add test for header_provider via FunctionTool.invoke with FunctionInvocationContext
Addresses PR review comment: exercises the full pipeline from
FunctionInvocationContext.kwargs through FunctionTool.invoke to
MCPStreamableHTTPTool.call_tool and header_provider, rather than
testing call_tool in isolation.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback for #4808: review comment fixes
* Fix streamable MCP transport defaults
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix Azure AI test client mocks
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix MCP runtime kwarg regressions
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Stabilize MCP tool runtime kwargs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Use context kwargs in MCP wrappers
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* updated mcp samples
* fix link
---------
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Eduard van Valkenburg
·
2026-03-31 17:23:49 +00:00
* Foundry Evals integration for Python
Merged and refactored eval module per Eduard's PR review:
- Merge _eval.py + _local_eval.py into single _evaluation.py
- Convert EvalItem from dataclass to regular class
- Rename to_dict() to to_eval_data()
- Convert _AgentEvalData to TypedDict
- Simplify check system: unified async pattern with isawaitable
- Parallelize checks and evaluators with asyncio.gather
- Add all/any mode to tool_called_check
- Fix bool(passed) truthy bug in _coerce_result
- Remove deprecated function_evaluator/async_function_evaluator aliases
- Remove _MinimalAgent, tighten evaluate_agent signature
- Set self.name in __init__ (LocalEvaluator, FoundryEvals)
- Limit FoundryEvals to AsyncOpenAI only
- Type project_client as AIProjectClient
- Remove NotImplementedError continuous eval code
- Add evaluation samples in 02-agents/ and 03-workflows/
- Update all imports and tests (167 passing)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: resolve mypy redundant-cast errors while keeping pyright happy
Use cast(list[Any], x) with type: ignore[redundant-cast] comments to
satisfy both mypy (which considers casting Any redundant) and pyright
strict mode (which needs explicit casts to narrow Unknown types).
Also fix evaluator decorator check_name type annotation to be
explicitly str, resolving mypy str|Any|None mismatch.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: CI failures — pyupgrade, evaluator overloads, sample API, reset attr
- Apply pyupgrade: Sequence from collections.abc, remove forward-ref quotes
- Add @overload signatures to evaluator() for proper @evaluator usage
- Fix evaluate_workflow sample to use WorkflowBuilder(start_executor=) API
- Fix _workflow.py executor.reset() to use getattr pattern for pyright
- Remove unused EvalResults forward-ref string in default_factory lambda
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: skip gRPC-dependent observability test
The test_configure_otel_providers_with_env_file_and_vs_code_port test
triggers gRPC OTLP exporter creation, but the grpc dependency is
optional and not installed by default. Add skipif decorator matching
the pattern used by all other gRPC exporter tests in the same file.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: add nosec B101 for bandit assert check
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* style: align eval samples with repo conventions
- Move module docstrings before imports (after copyright header)
- Add -> None return type to all main() and helper functions
- Fix line-too-long in multiturn sample conversation data
- Add Workflow import for typed return in all_patterns_sample
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR review feedback: async fixes, sample bugs, deprecation warnings
- Simplify _ensure_async_result to direct await (async-only clients)
- Replace get_event_loop() with get_running_loop()
- Narrow _fetch_output_items exception handling to specific types
- Add warning log when _filter_tool_evaluators falls back to defaults
- Add DeprecationWarning to options alias in Agent.__init__
- Add DeprecationWarning to evaluate_response()
- Rename raw key to _raw_arguments in convert_message fallback
- Fix evaluate_agent_sample.py: replace evals.select() with FoundryEvals()
- Fix evaluate_multiturn_sample.py: use Message/Content/FunctionTool types
- Fix evaluate_workflow_sample.py: replace evals.select() with FoundryEvals()
- Update test mocks to use AsyncMock for awaited API calls
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add test coverage for review feedback items
- Add num_repetitions=2 positive test verifying 2×items and 4 agent calls
- Add _poll_eval_run tests: timeout, failed, and canceled paths
- Add evaluate_traces tests: validation error, response_ids path, trace_ids path
- Add evaluate_foundry_target happy-path test with target/query verification
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix ruff ISC004 lint error and apply formatter
- Wrap implicit string concatenation in parens in evaluate_multiturn_sample.py
- Apply ruff formatter to 6 other files with minor formatting drift
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove core type changes (extracted to fix/workflow-stale-session branch)
Reverts changes to _agents.py, _agent_executor.py, and _workflow.py
back to upstream/main. These fixes are now in a separate PR.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR review round 2: bugs, tests, and architecture
Code fixes:
- Fix _normalize_queries inverted condition (single query now replicates
to match expected_count)
- Fix substring match bug: 'end' in 'backend' matched; use exact set
lookup for executor ID filtering
- Fix used_available_tools sample: tool_definitions→tools param, use
FunctionTool attribute access instead of dict .get()
- Add None-check in _resolve_openai_client for misconfigured project
- Add Returns section to evaluate_workflow docstring
- Cache inspect.signature in @evaluator wrapper (avoid per-item reflection)
Architecture:
- Extract _evaluate_via_responses as module-level helper; evaluate_traces
now calls it directly instead of creating a FoundryEvals instance
- Move Foundry-specific typed-content conversion out of core to_eval_data;
core now returns plain role/content dicts, FoundryEvals applies
AgentEvalConverter in _evaluate_via_dataset
Tests:
- evaluate_response() deprecation warning emission and delegation
- num_repetitions > 1 with expected_output and expected_tool_calls
- Mock output_items.list in test_evaluate_calls_evals_api
- Update to_eval_data assertions for plain-dict format
- Unknown param error now raised at @evaluator decoration time
Skipped (separate PR): executor reset loop, xfail removal, options alias
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix CI: revert test_full_conversation, fix pyright errors
- Revert test_full_conversation.py to upstream/main (the session
preservation test was incorrectly changed to assert clearing)
- Fix pyright reportUnnecessaryComparison on get_openai_client() None
check by adding ignore comment
- Fix pyright reportPrivateUsage: add public EvalItem.split_messages()
method and use it in FoundryEvals._evaluate_via_dataset instead of
accessing private _split_conversation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR review round 3: reliability, test gaps, cleanup
- Add try/except guard for non-numeric score in _coerce_result
- Add poll_interval minimum bound (0.1s) to prevent tight loops
- Add runtime async client check in _resolve_openai_client
- Remove _ensure_async_result wrapper (10 call sites → direct await)
- Better error message when queries provided without agent
- Import-time asserts for evaluator set consistency
- Remove 28 redundant @pytest.mark.asyncio decorators
- Add doc note about _raw_arguments sensitive data
- Tests: tool_called_check mode=any, _normalize_queries branches,
_extract_result_counts paths, _extract_per_evaluator, bare check
via evaluate_agent, output_items assertion, modulo wrapping,
async client check, queries-without-agent error
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix CI: ruff S101 assert, pyright and mypy arg-type errors
- Replace module-level assert with if/raise for evaluator set
consistency checks (ruff S101 disallows bare assert)
- Add type: ignore[arg-type] and pyright: ignore[reportArgumentType]
on OpenAI SDK evals API calls that pass dicts where typed params
are expected (SDK accepts dicts at runtime)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR review round 4: bugs, reliability, test fixes
- Fix all_passed ignoring parent result_counts when sub_results present
- Fix _extract_tool_calls: parse string arguments via json.loads before
falling back to None (real LLM responses use string arguments)
- Sanitize _raw_arguments to '[unparseable]' to avoid leaking sensitive
tool-call data to external evaluation services
- Add NOTE comment on to_eval_data message serialization dropping
non-text content (tool calls, results)
- Eliminate double conversation split in _evaluate_via_dataset: build
JSONL dicts directly from split_messages + AgentEvalConverter
- Raise poll_interval floor from 0.1s to 1.0s to prevent rate-limit
exhaustion
- Fix MagicMock(name=...) bug in test: sets display name not .name attr
- Fix mock_output_item.sample: use MagicMock object instead of dict so
_fetch_output_items exercises error/usage/input/output extraction
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR review round 5: reliability, docs, test coverage
Code fixes:
- Move import-time RuntimeError checks to unit tests (avoids breaking
imports for all users on developer set-drift mistake)
- _filter_tool_evaluators now raises ValueError when all evaluators
require tools but no items have tools (was silently substituting)
- Add poll_interval upper bound (60s) to prevent single-iteration sleep
- Log exc_info=True in _fetch_output_items for debugging API changes
- Fix evaluate() docstring: remove claim about Responses API optimization
- Validate target dict has 'type' key in evaluate_foundry_target
- Document to_eval_data() limitation: non-text content is omitted
Tests:
- TestEvaluatorSetConsistency: verify _AGENT/_TOOL subsets of _BUILTIN
- TestEvaluateTracesAgentId: agent_id-only path with lookback_hours
- TestFilterToolEvaluatorsRaises: ValueError on all-tool no-items
- TestEvaluateFoundryTargetValidation: target without 'type' key
- Assert items==[] on failed/canceled poll results
- Mock output_items.list in response_ids test for full flow
- TestAllPassedSubResults: result_counts=None + sub_results delegation
and parent failures override sub_results
- TestBuildOverallItemEmpty: empty workflow outputs returns None
Skipped r5-07 (_raw_arguments length hint): marginal debugging value,
could leak content size information.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix error message: evaluate_responses() → evaluate_traces(response_ids=...)
The referenced function doesn't exist; the correct API is
evaluate_traces(response_ids=...) from the azure-ai package.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove dead to_eval_data() method, fix docstring claims
- Remove to_eval_data() from EvalItem (dead code after r4-05 JSONL refactor)
- Migrate 15 tests from to_eval_data() to split_messages()
- Update sample to use split_messages() + Message properties
- Remove unimplemented Responses API optimization docstring claim
- Update split_messages() docstring to not reference removed method
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Reduce default eval timeout from 600s to 180s (3 minutes)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove dead _evaluate_via_responses method from FoundryEvals
The method was never called — evaluate() uses _evaluate_via_dataset,
and evaluate_traces() calls _evaluate_via_responses_impl directly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Revert unrelated formatting changes to get-started samples
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix pyright: remove phantom FoundryMemoryProvider import, apply ruff format
- Remove import of non-existent _foundry_memory_provider module
(incorrectly kept during rebase conflict resolution)
- Apply ruff formatter to test_local_eval.py and get-started samples
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix eval samples: use FoundryChatClient for Agent()
The upstream provider-leading client refactor (#4818) made client=
a required parameter on Agent(). Update the three getting-started
eval samples to use FoundryChatClient with FOUNDRY_PROJECT_ENDPOINT,
matching the standard pattern from 01-get-started samples.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Simplify self-reflection sample using FoundryEvals
Replace ~80 lines of manual OpenAI evals API code (create_eval,
run_eval, manual polling, raw JSONL params) with FoundryEvals:
- evaluate_groundedness() uses FoundryEvals.evaluate() with EvalItem
- Remove create_openai_client(), create_eval(), run_eval() functions
- Remove openai SDK type imports (DataSourceConfigCustom, etc.)
- run_self_reflection_batch creates FoundryEvals instance once,
reuses it for all iterations across all prompts
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update eval samples to FoundryChatClient and FOUNDRY_PROJECT_ENDPOINT
- Migrate all foundry_evals samples from AzureOpenAIResponsesClient to FoundryChatClient
- Update env var from AZURE_AI_PROJECT_ENDPOINT to FOUNDRY_PROJECT_ENDPOINT
- Use AzureCliCredential consistently across all samples
- Fix README.md: correct function names (evaluate_dataset -> FoundryEvals.evaluate, evaluate_responses -> evaluate_traces)
- Update self_reflection .env.example and README.md
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix lint errors in eval samples (E501, ASYNC240, formatting)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove evaluate_all_patterns_sample.py (redundant with focused samples)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix async credential mismatch: use azure.identity.aio for async AIProjectClient
AIProjectClient from azure.ai.projects.aio requires an async credential.
Switch all foundry_evals samples from azure.identity.AzureCliCredential
to azure.identity.aio.AzureCliCredential. Also pass project_client to
FoundryChatClient instead of duplicating endpoint+credential.
Close credential in self_reflection sample to avoid resource leak.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Revert test_observability.py to upstream/main (not our test)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address moonbox3 review: sphinx docstrings, pagination, isinstance check
- Convert all Example:: / Typical usage:: code blocks to .. code-block:: python
format matching codebase convention (both _evaluation.py and _foundry_evals.py)
- Add async pagination in _fetch_output_items via async for (handles large result sets)
- Replace hasattr(__aenter__) with isinstance(client, AsyncOpenAI) in _resolve_openai_client
- Move AsyncOpenAI import from TYPE_CHECKING to runtime (needed for isinstance)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix test failures and address remaining moonbox3 review comments
- Fix tests: use MagicMock(spec=AsyncOpenAI) for project_client mocks
(isinstance check now requires proper type, not duck-typing)
- Fix tests: replace mock_page.__iter__ with _AsyncPage helper for async for
- Fix evaluate_response: auto-extract queries from response messages when
query is not provided (previously always raised ValueError)
- Add debug logging when skipping internal _-prefixed executor IDs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address Tao's PR review comments on Foundry Evals
- T1: Add comment explaining builtin.* pass-through in _resolve_evaluator
- T2: Add comment referencing OpenAI evals API for testing_criteria dict
- T3: Document Mustache-style {{item.*}} template placeholders
- T4: Document poll loop 60s sleep upper bound rationale
- T5: Narrow run type to RunRetrieveResponse, use typed field access
instead of vars()/getattr dance in _extract_result_counts and
_extract_per_evaluator; use run.error and run.report_url directly
- T6: Clarify openai_client docstring re: Azure Foundry endpoint
- T8: Remove misleading empty expected_tool_calls from sample
- Update tests to match real SDK PerTestingCriteriaResult shape
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove unnecessary Any union from run type annotations
RunRetrieveResponse is the correct type — no backward compat needed
for a brand new feature.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Accept FoundryChatClient instead of raw AsyncOpenAI
FoundryEvals now takes client: FoundryChatClient as its primary
parameter instead of openai_client: AsyncOpenAI. The builtin.*
evaluators require a Foundry endpoint, so the type should reflect that.
- FoundryEvals.__init__: client: FoundryChatClient replaces openai_client
- evaluate_traces / evaluate_foundry_target: same change
- _resolve_openai_client: extracts .client from FoundryChatClient
- project_client fallback retained for standalone functions
- All samples updated to construct FoundryChatClient and pass as client=
- Tests updated (openai_client= → client=)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove implicit 60s upper bound on poll interval
If a developer sets a higher poll_interval, respect it. Only clamp
to remaining time and enforce a 1s minimum for rate-limit protection.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove 1s floor on poll interval — let the developer control it
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update python/samples/05-end-to-end/evaluation/foundry_evals/.env.example
Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com>
* Update python/samples/02-agents/evaluation/evaluate_agent.py
Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com>
* Address eavanvalkenburg review (round 2) on Python eval PR
- Rename model_deployment -> model across FoundryEvals and all samples
- Make model param optional, resolves from client.model
- Convert EvalResults from dataclass to regular class
- Remove deprecated evaluate_response() function
- Refactor splitters: BUILT_IN_SPLITTERS dict + standalone functions
- Change per_turn_items from classmethod to staticmethod
- Simplify EvalCheck type alias to use Awaitable[CheckResult]
- Remove errored property from EvalResults
- Remove default value from Evaluator protocol eval_name
- Rename assert_passed -> raise_for_status, add EvalNotPassedError
- Type agent param as SupportsAgentRun | None
- Fix Arguments docstring
- Update __init__.py exports
- Update all tests and samples
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Move FoundryEvals to foundry package, split tool eval sample
- Move _foundry_evals.py from azure-ai to foundry package
- Move test_foundry_evals.py to foundry/tests/
- Update lazy re-exports in agent_framework.foundry namespace
- Update .pyi type stubs
- All samples now import from agent_framework.foundry
- Split tool-call evaluation into evaluate_tool_calls_sample.py
- Fix all_passed to check errored count from result_counts
- Fix raise_for_status to include errored item details
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Auto-create FoundryChatClient from env vars when no client provided
FoundryEvals() now works zero-config when FOUNDRY_PROJECT_ENDPOINT and
FOUNDRY_MODEL environment variables are set. Auto-creates a FoundryChatClient
under the hood, matching the established env var pattern.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix pyright errors: remove dead _normalize_queries, suppress EvalAPIError check
- Remove unused _normalize_queries function and its tests
- Add pyright ignore for EvalAPIError None check (defensive guard)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Support multimodal image content in eval pipeline
Add image (data/uri) content handling to AgentEvalConverter.convert_message()
so that Content.from_data() and Content.from_uri() image payloads are
preserved as input_image parts in the Foundry evaluator format.
- Handle Content type='data' and type='uri' → emit input_image parts
- Add 6 unit tests for image content through convert_message/convert_messages
- Add integration test verifying images flow through EvalItem → JSONL path
- Add evaluate_multimodal.py sample demonstrating local image eval
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address remaining review comments
- Fix project_client docstring to say async-only (not sync/async)
- Add builtin evaluator name validation warning in _resolve_evaluator
- Replace getattr with typed attribute access in _poll_eval_run,
_extract_result_counts, _extract_per_evaluator, _fetch_output_items
- Remove cast import from _foundry_evals (no longer needed)
- Tighten _coerce_result: honour explicit 'passed' when both 'score'
and 'passed' are present; remove performative cast
- Fix self_reflection sample: add env file existence check
- Fix traces sample: correct Pattern 2 section label
- Update all Foundry eval samples to FoundryChatClient + FOUNDRY_MODEL
(remove AIProjectClient + AZURE_AI_MODEL_DEPLOYMENT_NAME pattern)
- Add eval_name and OpenAI client docs to FoundryEvals docstring
- Update test mocks to match typed SDK objects (_MockResultCounts)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix ruff lint errors (E501, SIM108, SIM102)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix pyright errors: type-narrow dict to dict[str, Any], add ignore comments
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Replace ConversationSplitter type alias with Protocol
ConversationSplitter is now a runtime-checkable Protocol with a named
'conversation' parameter, making the expected signature self-documenting.
ConversationSplit enum members gain a __call__ method so they satisfy
the protocol directly -- ConversationSplit.LAST_TURN(conversation) works.
This simplifies _split_conversation from an isinstance dispatch to a
single split(conversation) call.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Standardize on AZURE_AI_MODEL_DEPLOYMENT_NAME and fix Unicode in samples
- Replace FOUNDRY_MODEL with AZURE_AI_MODEL_DEPLOYMENT_NAME in all
eval samples to match repo convention
- Replace Unicode symbols with ASCII equivalents in all eval sample
print statements to avoid cp1252 encoding errors on Windows
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update python/samples/03-workflows/evaluation/evaluate_workflow.py
Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com>
* Rename ADR 0020 to 0023 (foundry evals integration)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: alliscode <bentho@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com>
* Support MCP sampling tools capability (#4625)
Forward systemPrompt, tools, and toolChoice from MCP sampling requests
to the chat client's get_response() call. Also advertise the
sampling.tools capability to MCP servers when a client is configured.
- Pass SamplingCapability with tools support to ClientSession
- Convert systemPrompt to instructions in options
- Convert MCP Tool objects to FunctionTool instances for options
- Map MCP ToolChoice.mode to tool_choice in options
- Add tests for all new behaviors and update existing sampling tests
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix#4625: Support MCP sampling tool with proper typing and structured content
- Fix mypy error by typing sampling callback options as ChatOptions[None]
instead of dict[str, Any], and importing ChatOptions from _types
- Handle structuredContent from CallToolResult in _parse_tool_result_from_mcp,
serializing it as JSON text Content when present
- Add tests for structuredContent parsing (with and without regular content)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix lint: add author to TODO comment
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback for #4625: remove default=str, add edge-case tests
- Remove default=str from json.dumps for structuredContent to fail fast
on non-JSON-serializable values instead of silently converting
- Add test for non-JSON-serializable structuredContent (TypeError)
- Add tests for empty systemPrompt ('') and empty tools list ([]) edge
cases in sampling callback
- Expand TODO comment noting list[Content] return type constraint for
future result_type support
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Sanitize sampling callback error to avoid leaking internals (#4625)
Log exception details at DEBUG level instead of including them in the
ErrorData message returned to the MCP server, which may be untrusted.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback for #4625: move params to options, restore error info
- Remove stale TODO comment about response_format (ChatOptions already has it)
- Restore {ex} in sampling callback error message for useful debugging info
- Set structuredContent as additional_property on Content for structured access
- Move temperature, max_tokens, stop into options dict (not top-level kwargs)
- Only set temperature when provided (not all models support it)
- Add tests for generation params in options and temperature omission
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix MCP sampling callback and structured content error handling (#4625)
- Guard max_tokens like temperature: only set when not None, so options
can properly evaluate to None when all params are absent
- Wrap json.dumps of structuredContent in try/except to fall back to
str() for non-serializable values instead of propagating TypeError
- Extract test_connect_sampling_capabilities_with_client into its own
test function so pytest can discover it independently
- Add test for max_tokens=None omission from options
- Update structured content non-serializable test to expect fallback
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback for #4625: review comment fixes
* Fix MCP and Azure validation regressions
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Eduard van Valkenburg
·
2026-03-26 07:33:19 +00:00
* Fix PydanticSchemaGenerationError with PEP 563 annotations in @tool
_resolve_input_model used raw param.annotation from inspect.signature(),
which returns string annotations when 'from __future__ import annotations'
is active (PEP 563). This caused Pydantic's create_model to fail for
complex types like Optional[int] or FunctionInvocationContext.
Use typing.get_type_hints() to resolve annotations to actual types before
passing them to create_model, matching the approach already used by
_discover_injected_parameters.
Fixes#4809
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Apply pre-commit auto-fixes
* Remove reproduction report and unused test imports
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix(tests): strengthen PEP 563 regression tests per review feedback (#4809)
- Verify type correctness in schema assertions (not just key presence)
- Fix ctx annotation to FunctionInvocationContext | None for type consistency
- Add test for Optional[CustomType] pattern (original bug trigger)
- Add test for get_type_hints() fallback with unresolvable forward refs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Apply pre-commit auto-fixes
* Address review feedback for #4809: Python: [Bug]: PydanticSchemaGenerationError in FunctionInvocationContext
---------
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Python: avoid duplicate agent response telemetry
* Python: conditionally suppress duplicate agent telemetry
* Simplify telemetry ownership tracking
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Aggregate token usage from inner chat spans on invoke_agent span
The invoke_agent span now carries the aggregated input/output token
counts from all inner chat completion spans that occur during an agent
run. Previously, when inner ChatTelemetryLayer spans captured usage,
the outer AgentTelemetryLayer skipped setting usage entirely to avoid
duplication. Now a new INNER_ACCUMULATED_USAGE context variable tracks
cumulative usage across all inner completions, and the agent span
always reports the total.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Eduard van Valkenburg
·
2026-03-20 10:09:46 +00:00
* [BREAKING] Refactor middleware layering and raw clients
Reorder chat client layers so function invocation wraps chat middleware, and chat middleware stays outside telemetry while still running for each inner model call. Add middleware pipeline caching, refresh docs and samples, and split Anthropic into raw and public clients to match the standard layering model.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Tighten typing ignores in ancillary modules
Add targeted typing ignores in workflow visualization and lab modules so pyright stays clean alongside the middleware refactor work.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix categorize_middleware to unpack tuple/Sequence and use relative MRO assertions
- Broaden isinstance check in categorize_middleware from list to Sequence
so tuples and other Sequence types are properly unpacked instead of
being appended as a single item.
- Replace fragile hardcoded MRO index assertions in anthropic test with
relative ordering via mro.index().
- Add regression tests for categorize_middleware with tuple, list, and
None inputs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix middleware string decomposition, add middleware param to FunctionInvocationLayer, and add tests (#4710)
- Guard categorize_middleware Sequence check against str/bytes to prevent
character-by-character decomposition of accidentally passed strings
- Add explicit middleware parameter to FunctionInvocationLayer.get_response
and merge it into client_kwargs before categorization, fixing the
inconsistency where only OpenAIChatClient supported this parameter
- Add assertions that RawAnthropicClient does not inherit convenience layers
- Add chat middleware cache test with non-empty base middleware
- Add tests for single unwrapped middleware item and string input
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Apply pre-commit auto-fixes
* Apply pre-commit auto-fixes
* Address review feedback for #4710: review comment fixes
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <copilot@github.com>
Eduard van Valkenburg
·
2026-03-20 00:43:37 +00:00
* Support detail field in OpenAI image_url payload (#4616)
Include the optional 'detail' field from Content.additional_properties
when building image_url payloads for the OpenAI Chat API, matching the
existing pattern used for 'filename' in document file payloads.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Apply pre-commit auto-fixes
* Remove reproduction report
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Simplify detail extraction from additional_properties (#4616)
- Remove unnecessary hasattr check; additional_properties is always
initialized as a dict on Content instances.
- Use 'is not None' instead of truthy check to be more precise.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Apply pre-commit auto-fixes
* Remove detail allowlist in chat client to align with responses client
Replace the strict allowlist check ('low', 'high', 'auto') with an
isinstance(detail, str) check so that any valid string detail value is
passed through to OpenAI. This aligns the chat client behavior with the
responses client, which passes detail through unconditionally.
Also add test coverage for:
- Future/unknown string detail values being passed through
- Data URI images (covering the 'data' branch of the match)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix zero-argument MCP tool schema missing 'properties' key (#4540)
MCP servers for zero-argument tools (e.g. matlab-mcp-core-server's
detect_matlab_toolboxes) declare inputSchema as {"type": "object"}
without a "properties" key. OpenAI's API requires "properties" to
be present on object schemas, causing a 400 invalid_request_error.
Normalize inputSchema at MCP ingestion in load_tools() to inject an
empty "properties": {} when it is missing from object-type schemas.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback for #4540: improve test robustness and add defensive guard
- Look up loaded functions by name instead of index to avoid brittle
ordering assumptions
- Add negative-path test cases: non-object schema (type: string) and
empty schema ({}) to verify guard clause skips them correctly
- Assert original inputSchema dicts are not mutated by load_tools()
- Add defensive guard for tool.inputSchema being None
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback for #4540: Python: [Bug]: Local stdio MCP works for calculator but fails for official matlab-mcp-core-server on LM Studio /v1/responses
---------
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix invoke_agent span to aggregate token usage across LLM calls (#4062)
The FunctionInvocationLayer._get_response() loop was overwriting the
response on each iteration, so usage_details only reflected the last
chat completion call. Now tracks aggregated_usage across all iterations
using add_usage_details() and sets it on the returned response.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Apply pre-commit auto-fixes
* Remove reproduction report artifact
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Apply pre-commit auto-fixes
* Apply pre-commit auto-fixes
---------
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Eduard van Valkenburg
·
2026-03-19 06:41:33 +00:00
* updated automation tasks and commands, with alias for the time being
* Restore aggregate test exclusions
Preserve the legacy all-tests scope for test --all by excluding lab and devui from the default aggregate sweep, while still allowing explicit package selection. Also ignore hidden/generated test directories such as .mypy_cache during aggregate discovery.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* updated versions in pre-commit
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Eduard van Valkenburg
·
2026-03-18 18:39:11 +00:00
* Python: Re-read env vars in configure_otel_providers and enable_instrumentation (#4119)
Fix ENABLE_SENSITIVE_DATA and VS_CODE_EXTENSION_PORT env vars being ignored
when load_dotenv() runs after module import. The module-level
OBSERVABILITY_SETTINGS singleton cached env state at import time, and
configure_otel_providers() / enable_instrumentation() never re-read from
os.environ when parameters were None.
Both functions now construct a fresh ObservabilitySettings() to pick up
current env vars when explicit parameters are not provided, matching the
existing behavior of the env_file_path branch.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR review feedback for #4119: avoid throwaway ObservabilitySettings
- Add _read_bool_env/_read_int_env helpers to read env vars without
constructing a full ObservabilitySettings (which calls create_resource())
- Replace ObservabilitySettings() in enable_instrumentation() and
configure_otel_providers() else-branch with direct env reads
- Add enable_console_exporters parameter to configure_otel_providers()
for override parity with enable_sensitive_data and vs_code_extension_port
- Propagate _resource and _executed_setup in the non-env_file_path branch
- Make existing tests hermetic (clear VS_CODE_EXTENSION_PORT and
ENABLE_CONSOLE_EXPORTERS env vars)
- Add tests: enable_console_exporters env refresh, explicit param overrides
for both enable_instrumentation() and configure_otel_providers()
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address remaining review feedback for #4119
- Refresh enable_console_exporters in enable_instrumentation() for
consistency with configure_otel_providers(), so env var changes
after import are picked up by both public API functions
- Make test_configure_otel_providers_reads_env_vs_code_port hermetic
by clearing ENABLE_CONSOLE_EXPORTERS from the environment
- Add test_enable_instrumentation_reads_env_console_exporters to
cover the new refresh behavior
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove unconditional enable_console_exporters overwrite from enable_instrumentation() (#4119)
enable_instrumentation() is documented as not configuring exporters, so
managing enable_console_exporters there was a leaky abstraction. The
unconditional _read_bool_env call silently reset the value to False when
ENABLE_CONSOLE_EXPORTERS was absent from env, clobbering any value
previously set by configure_otel_providers(enable_console_exporters=True).
- Remove the unconditional overwrite line from enable_instrumentation()
- Replace test_enable_instrumentation_reads_env_console_exporters with
test_enable_instrumentation_does_not_touch_console_exporters
- Add regression test: enable_instrumentation() does not clobber a
previously configured enable_console_exporters value
- Add test: explicit enable_sensitive_data param still leaves
enable_console_exporters untouched
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Eduard van Valkenburg
·
2026-03-18 15:58:22 +00:00
* Fix Content serialization in DurableAgentStateUnknownContent (#4719)
DurableAgentStateUnknownContent.from_unknown_content() stored raw Content
objects without converting them to dicts, causing json.dumps to fail in
Azure Durable Functions' entity state serialization. This affected content
types not explicitly handled (e.g., mcp_server_tool_call/result).
The fix converts Content objects to dicts via to_dict() when storing in
DurableAgentStateUnknownContent, and restores them via Content.from_dict()
in to_ai_content().
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add to_json and from_json methods to Content class (#4719)
Add to_json() and from_json() methods to the Content class to match the
serialization interface provided by SerializationMixin on other model classes.
Also fix pre-existing pyright type errors in durabletask's
DurableAgentStateUnknownContent.to_ai_content().
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR review: add type guard, remove to_json, add fallback, and tests
- Remove Content.to_json() per reviewer request (comment 3)
- Add type guard in Content.from_json() for non-dict JSON (comments 1, 4)
- Wrap json.JSONDecodeError as ValueError for consistent exception contract
- Add try/except fallback in to_ai_content() for invalid Content dicts (comment 5)
- Add test_content_to_dict_exclude_none and test_content_to_dict_exclude_fields (comment 2)
- Add test_unknown_content_to_ai_content_fallback_on_invalid_type_dict (comment 5)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Apply pre-commit auto-fixes
* Address review feedback for #4719: review comment fixes
* Remove Content.from_json, move logic to consuming code (#4719)
Remove the from_json convenience method from Content class per review
feedback. This is the same trivial json.loads + from_dict wrapper as
to_json which was already removed. Consumers should call json.loads
and Content.from_dict directly.
Update tests to use Content.from_dict(json.loads(...)) pattern and
remove from_json-specific error handling tests (those errors are
already covered by json.loads and Content.from_dict).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Eduard van Valkenburg
·
2026-03-18 08:08:44 +00:00
* Python: clean up kwargs across agents, chat clients, tools, and sessions (#3642)
Audit and refactor public **kwargs usage across core agents, chat clients,
tools, sessions, and provider packages per the migration strategy codified
in CODING_STANDARD.md.
Key changes:
- Add explicit runtime buckets: function_invocation_kwargs and client_kwargs
on RawAgent.run() and chat client get_response() layers.
- Refactor FunctionTool to prefer explicit ctx: FunctionInvocationContext
injection; legacy **kwargs tools still work via _forward_runtime_kwargs.
- Refactor Agent.as_tool() to use direct JSON schema, always-streaming
wrapper, approval_mode parameter, and UserInputRequiredException
propagation (integrates PR #4568 behavior).
- Remove implicit session bleeding into FunctionInvocationContext; tools
that need a session must receive it via function_invocation_kwargs.
- Lower chat-client layers after FunctionInvocationLayer accept only
compatibility **kwargs (client_kwargs flattened, function_invocation_kwargs
ignored).
- Add layered docstring composition from Raw... implementations via
_docstrings.py helper.
- Clean up provider constructors to use explicit additional_properties.
- Deprecation warnings on legacy direct kwargs paths.
- Update samples, tests, and typing across all 23 packages.
Resolves#3642
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* clarified docstring
* feedback fixes
* Add unit tests for _docstrings.py build/apply helpers
Tests cover: no docstring source, no extra kwargs, appending to existing
Keyword Args section, inserting after Args, inserting in plain docstrings,
multiline descriptions, ordering, and apply_layered_docstring.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add test for propagate_session TypeError on non-AgentSession values
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add tests for multi-content and empty UserInputRequiredException propagation
Cover the branching logic in _try_execute_function_calls for:
- Multiple user_input_request items in a single exception (extra_user_input_contents path)
- Empty contents list (fallback function_result path)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add tests for DurableAIAgent.get_session forwarding service_session_id
Verifies get_session correctly forwards service_session_id and session_id
to the executor's get_new_session, replacing the removed kwargs test.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Simplify ag-ui test stub to read session from client_kwargs only
Remove dual-mode detection (client_kwargs vs raw kwargs fallback) from
the test mock. Session is now read exclusively from client_kwargs,
matching the settled public calling convention.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* updated create and get sessions in durable
* fixed docstrings
* fix test
* updated session handling
* updated from main
* updated tests
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Eduard van Valkenburg
·
2026-03-13 08:58:32 +00:00
* feat(python): allow @tool functions to return rich content (images, audio)
Add support for tool functions to return Content objects that the model can perceive natively. Closes#4272
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Anthropic logging + mypy fix
* Address PR review: fix MCP ordering, fold helper into from_function_result, fix Chat client
- Preserve original content order in MCP tool results instead of text-first
- Move _build_function_result logic into Content.from_function_result()
- Chat Completions: inject user message for rich items (API only supports string tool content)
- Update tests for ordering and new from_function_result behavior
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Use native Responses API multi-part output, warn+omit for Chat client
- Responses client: put rich items directly in function_call_output's
output field as list (native API support) instead of user message injection
- Chat client: warn and omit rich items (API doesn't support multi-part
tool results), matching Ollama/Bedrock pattern
- Unify test image: use sample_image.jpg across all integration tests
- Add Azure OpenAI Responses integration test
- Assert model describes house image to verify perception
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix lint: remove print statement, wrap long line
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback: bug fixes, single-pass MCP, unit tests
- Add isinstance guard in from_function_result for non-Content lists
- Fix Anthropic empty tool_content fallback to string result
- Fix Content(type='text', text=None) edge case in parse_result
- Rewrite MCP _parse_tool_result_from_mcp as single-pass (no index counters)
- Add Anthropic unit tests: data image, uri image, unsupported media, all-unsupported
- Add OpenAI Chat unit test: rich items warning and omission
- Add OpenAI Responses unit tests: function_result with/without items
- Add test_types tests: only-rich-items list, non-Content list fallback
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix pyright errors: add type ignore comments for Any list iteration
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix mypy/pyright: ensure ToolExecutionException receives str
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix lint: remove duplicate test_prepare_options_excludes_conversation_id
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* refactor: unify all tool results into Content items
* addressed copilot comments
* pyright fix
* small fix
* comments
* fix: address Copilot review - warnings, blob safety, dedup
- Add warning logs when rich content is dropped in Claude agent and
MCP server handlers (matching Chat/Bedrock/Ollama pattern)
- Defensive blob URI construction: wrap plain base64 in data: prefix
- Simplify Chat client _prepare_content_for_openai to use content.result
- Simplify Responses client text-only path, remove redundant nesting
- Add test for plain base64 blob without data: prefix
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix token double-counting in compaction and address review comments
- Exclude items from _serialize_content() to prevent double-counting
tokens when items mirrors result in function_result content
- Add rich content warning in GitHub Copilot agent tool handler
- Replace raw Content debug log with concise item count/type summary
- Update stale test comments about FunctionTool.invoke return type
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Dineshsuriya D ·
Tao Chen ·
Evan Mattson ·
Copilot ·
Eduard van Valkenburg ·
hashwnath ·
Giles Odigwe ·
Ben Thomas ·
Chinedum Echeta ·