* Bump Python package versions to 1.5.0 for a release
* Promote orchestrations to 1.0.0rc1
* ci(python-setup): merge dynamic exclude into existing workspace exclude
The python-setup action injected exclude = [...] verbatim into
[tool.uv.workspace], producing a duplicate 'exclude' key when the
section already had a static exclude. Scope the rewrite to the
[tool.uv.workspace] section and append the package to the existing
array when present; idempotent if the package is already excluded.
* Address Copilot review feedback: raise inter-package floors to 1.5.0
- foundry, foundry-local: agent-framework-openai >=1.4.0 -> >=1.5.0
- azure-contentunderstanding: agent-framework-foundry >=1.4.0 -> >=1.5.0
- azurefunctions: pin agent-framework-durabletask to >=1.0.0b260519,<2
Keeps lockstep cohort consistent and avoids mixed 1.4.x / 1.5.0 installs.
* Re-include azurefunctions and durabletask in the uv workspace
The pinned durabletask>=1.4.0 floor is enough to make resolution succeed;
the workspace exclude was over-correction and broke CI samples and pyright
type-checking (re-exports in agent_framework/azure/__init__.pyi plus
samples/04-hosting/{azure_functions,durabletask}/ could not resolve their
imports). Dropping them from agent-framework-core[all] still stands so the
metapackage does not pull them.
* Restore azurefunctions and durabletask in agent-framework-core[all]
The durabletask floor pin keeps users on the safe 1.4.0, so they are once
again included in the metapackage. Update CHANGELOG to reflect the pin
rather than an [all] removal.
* Raise uvicorn ceiling in ag-ui and devui to allow 0.42+
The root override-dependencies pins uvicorn[standard]>=0.34.0 (no upper)
and the workspace lock resolves to 0.47.0. The package ceiling <0.42.0
meant the workspace was no longer testing the declared supported range.
Bump to <1 so the lock fits within the declared bounds.
Also picked up by validate-dependency-bounds: refresh stale orchestrations
RC pin in devui dev deps.
The shared composite action ran `uv sync --all-packages --all-extras
--dev -U` on every job, which upgrades every dependency to the latest
compatible version instead of using the pinned versions in `uv.lock`.
That is currently producing a hard resolver failure on every CI job:
No solution found when resolving dependencies for split
(markers: python_full_version >= '3.11' and sys_platform == 'darwin')
Because there are no versions of durabletask and
agent-framework-durabletask depends on durabletask>=1.3.0,<2,
we can conclude that agent-framework-durabletask's requirements
are unsatisfiable.
Dropping `-U` makes the install use the workspace lockfile, which is
what is reproducible locally and what we publish releases against.
Upgrades should be opt-in (via a scheduled job or a separate workflow)
rather than implicit on every CI run.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Eduard van Valkenburg
·
2026-05-19 19:33:11 +00:00
* Add sample that shows code execution and skills together
* Use nuget for python module path
* Update readme.
* Fix formatting.
* Reduce flashing in rendering.
* Improve screen clearing for Powershell
* Add a couple of small UX fixes
The second self._cache.pop(key, None) call is a guaranteed no-op: the first pop has already removed the key (or returned None), and there is no await between the two statements that could allow another coroutine to re-add it. Removing the dead line clarifies intent without changing behavior.
* Python: fix(hyperlight): skip symlinks when staging files into the sandbox
The helpers that populate the sandbox input tree (``_copy_path`` and the
``_path_tree_signature`` walker used for cache invalidation) relied on
``Path.is_file()``, ``Path.is_dir()`` and ``shutil.copy2`` - all of which
follow symlinks by default. When the source tree contains symlinks, that
let entries from outside the configured input source surface inside the
sandbox.
Harden both code paths to never follow symlinks:
- ``_copy_path`` now bails out via ``Path.is_symlink()`` before any
``is_dir()`` / ``is_file()`` check, skips non-regular files, and uses
``shutil.copy2(..., follow_symlinks=False)`` as defense in depth.
- New ``_iter_real_entries`` walker replaces the previous ``Path.rglob``
call inside ``_path_tree_signature`` (rglob follows directory symlinks).
- ``_path_tree_signature`` switches to ``Path.lstat()`` so size/mtime are
never read through a symlink target.
Added regression tests covering:
- A pre-placed file symlink in ``workspace_root`` (top level).
- A pre-placed directory symlink in ``workspace_root``.
- A nested file symlink inside a real subdirectory.
- ``_path_tree_signature`` ignoring symlinks so the cache key reflects only
what is actually staged.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Python: fix(hyperlight): address PR #5919 review feedback
- _iter_real_entries now yields directories and regular files only,
skipping non-regular entries (sockets/FIFOs/devices). Keeps the
cache-key signature consistent with what _copy_path actually stages.
- The four new symlink regression tests skip when the platform does not
support symlink creation (e.g. unprivileged Windows runners), via a
local _symlinks_supported helper modelled on the one in
packages/core/tests/core/test_skills.py. Prevents OSError /
NotImplementedError from failing CI jobs that have nothing to do with
the change under test.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Python: fix(hyperlight): address PR #5919 follow-up review feedback
- _copy_path docstring: narrow the scope to "symlink entries present in
the source tree at rest" and explicitly call out that the copy is NOT
atomic with respect to concurrent mutation of the source tree.
Callers who need that stronger guarantee should snapshot their
workspace before passing it in. Avoids overpromising on a TOCTOU
window that pathlib cannot express; closing it properly would need
fd-based traversal (O_NOFOLLOW | O_DIRECTORY + os.scandir(fd)) with
a separate Windows story, which is out of scope for this targeted
fix.
- _path_tree_signature: drop the `if path.is_symlink(): return ()`
short-circuit. Resolve a symlink root to its real target before
walking instead. The public construction flow already resolves
workspace_root / file_mounts[].host_path up front so this never
affected user-facing code, but the short-circuit was misleading and
would have produced an empty, stable signature for any direct
caller that builds a _RunConfig without going through the public
constructor. Defense in depth: even if a future call site forgets
to resolve the root, the cache key still reflects real contents.
- Added regression test
test_path_tree_signature_walks_through_symlinked_root: a symlinked
workspace root must produce a non-empty signature, AND the signature
must change when the real target's contents change so the cache key
actually invalidates.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Eduard van Valkenburg
·
2026-05-19 11:41:53 +00:00
* Record actual served model as response model for Azure OpenAI
* Formatting
* Fix tests
* Fix pipeline error
* Comments
* Address review: surface served model via ChatResponse.model
Apply blocking review feedback from PR #5910:
- Use ChatResponse.model / ChatResponseUpdate.model as the source of truth
for the Azure x-ms-served-model header value, instead of stashing it in
additional_properties and overriding it again in observability.
Observability already reads response.model; the chat client now overwrites
it post-parse when the served-model header is present. Empirically the
Azure Responses API returns the deployment alias in body.model and the
actual snapshot (e.g. gpt-5-nano-2025-08-07) in this header.
- Move the AZURE_OPENAI_SERVED_MODEL_HEADER constant out of observability.py
and into RawOpenAIChatClient (as the SERVED_MODEL_HEADER ClassVar). The
header is Azure-OpenAI-Responses-API-specific so observability does not
need to know about it.
- Revert the streaming text_format path to client.responses.stream(...) and
drop the _pydantic_model_to_text_format_param helper. That helper imported
from openai.lib._parsing._responses (a private SDK path) and the swap to
responses.create(stream=True) dropped client-side output_parsed for
structured-output streaming. The streaming-with-text_format path is the
only one that does not surface the served-model header - documented inline.
- Wrap the raw streaming responses in async with so the underlying socket
closes deterministically (continuation_token retrieve + create paths).
- Fix the empty-string / whitespace-only header at the source by stripping
in _extract_served_model and returning None when nothing remains.
- Revert unrelated formatting-only churn in _skills.py and test_mcp.py.
- Update unit tests to assert against chat_response.model / update.model
and add an aggregated streaming assertion plus a pin that the
streaming-with-text_format path does not get the header.
Verified end-to-end against Azure OpenAI Responses API: deployment alias
gpt-5-nano now reports gpt-5-nano-2025-08-07 as ChatResponse.model in both
the non-streaming and streaming paths.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: preserve streaming structured output finalization
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/f62076ef-558d-49e8-8fe2-f38d527c9639
Co-authored-by: eavanvalkenburg <13749212+eavanvalkenburg@users.noreply.github.com>
* refactor: name streaming response finalizer
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/f62076ef-558d-49e8-8fe2-f38d527c9639
Co-authored-by: eavanvalkenburg <13749212+eavanvalkenburg@users.noreply.github.com>
* fix: capture streaming response format after prepare
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/f62076ef-558d-49e8-8fe2-f38d527c9639
Co-authored-by: eavanvalkenburg <13749212+eavanvalkenburg@users.noreply.github.com>
* refactor: clarify streaming response format capture
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/f62076ef-558d-49e8-8fe2-f38d527c9639
Co-authored-by: eavanvalkenburg <13749212+eavanvalkenburg@users.noreply.github.com>
* test: use public API for streaming structured output
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/f62076ef-558d-49e8-8fe2-f38d527c9639
Co-authored-by: eavanvalkenburg <13749212+eavanvalkenburg@users.noreply.github.com>
* Inline the served-model header override at its two call sites
The `_apply_served_model_header` helper was a 1-line wrapper around
`_extract_served_model`. Inlining the `if served_model is not None: ...`
matches the pattern already used in the streaming paths and folds the
explanatory docstring onto `_extract_served_model` (which is now the
single place that knows about the header).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: eavanvalkenburg <13749212+eavanvalkenburg@users.noreply.github.com>
* Improve the handling of intermediate outputs for workflows and orchestrations
* Address PR review feedback on intermediate output forwarding
- Switch workflow.as_agent() forwarding to an explicit allowlist of {output,
intermediate, data, request_info} so orchestration-internal events
(group_chat, handoff_sent, magentic_orchestrator) stay inside the workflow
instead of leaking into agent responses via str(data) coercion.
- Stop raising on intermediate AgentResponseUpdate in non-streaming run();
surface the partial as a Message with text_reasoning content. The defensive
raise still applies to terminal output events, where Update payloads would
corrupt message ordering.
- Extend the DevUI workflow-event mapper so intermediate yields wrapping
plain strings, Messages, and list[Message] render as visible output items
instead of generic completed-trace events.
- Add orchestration coverage for GroupChat, Handoff, and Magentic builders
(default vs intermediate_outputs=True; structural where end-to-end is heavy).
* Lift output-designation policy into a value type
Replace the ``Workflow._output_executors`` list and the
``RunnerContext.should_label_as_intermediate`` Protocol method with a single
immutable ``OutputDesignation`` value type owned by ``Workflow``. Thread the
designation as a parameter through the existing call chain (Runner ->
EdgeRunner -> Executor -> WorkflowContext) so ``yield_output`` consults the
threaded snapshot directly rather than calling back into the runner context.
Removes the ``InProcRunnerContext._workflow`` back-reference and the
``WorkflowBuilder.build()`` assignment that wired it up. Adds the public
predicate ``Workflow.is_terminal_executor(executor_id)`` for external
observers; ``OutputDesignation`` itself stays package-internal.
Key decisions
- ``OutputDesignation.designated`` is ``frozenset[str] | None`` -- ``None``
preserves legacy "every yield is type='output'" behavior, any frozenset
(including empty) opts into strict mode. The ``DeprecationWarning`` for
legacy mode at build time is unchanged.
- ``output_designation`` is an optional parameter on ``Runner``,
``EdgeRunner.send_message``, ``EdgeRunner._execute_on_target``,
``Executor.execute``, ``Executor._create_context_for_handler``, and
``WorkflowContext.__init__``. Each defaults to legacy ``OutputDesignation()``
so direct callers (Azure Functions ``CapturingRunnerContext``,
``test_runner`` recording fixtures) keep working without ceremony.
- The workflow-level filter in ``_run_core`` reads ``self._output_designation``
live, preserving today's semantics where mutating the designation after
build still affects subsequent runs (used by two existing tests).
- ``Workflow.to_dict()`` continues to emit ``"output_executors":
list[str] | None`` (sorted from the frozenset). Checkpoint format unchanged.
Files changed
- _workflow.py: add ``OutputDesignation`` dataclass; replace
``_output_executors`` with ``_output_designation``; add
``is_terminal_executor``; delete ``_should_yield_output_event``.
- _runner_context.py: drop ``should_label_as_intermediate`` Protocol method
and ``InProcRunnerContext`` impl; drop ``_workflow`` back-reference.
- _workflow_builder.py: remove ``context._workflow = workflow`` assignment.
- _runner.py, _edge_runner.py, _executor.py, _workflow_context.py: thread
``output_designation`` parameter through the call chain.
- tests/workflow/test_output_designation.py (new): three-state coverage of
the value type plus the public predicate delegation.
- tests/workflow/test_workflow_builder.py, test_validation.py,
test_workflow.py, test_runner.py and
orchestrations/tests/test_orchestration_intermediate_vs_terminal.py:
switch probes from ``_output_executors`` set checks to
``get_output_executors`` / ``is_terminal_executor``; update two
post-build mutation tests to set ``_output_designation`` instead.
Verification
- core/tests/workflow/, orchestrations/tests/, azurefunctions/tests/:
1119 passed, 42 skipped, 2 xfailed.
- ``uv run poe lint``: clean.
- ``uv run poe typing``: only the pre-existing
``_AGENT_FORWARDED_EVENT_TYPES`` pyright warning from 394bcd607 remains.
Notes for next iteration
- The builder's own ``_output_executors`` attribute (``list[Executor |
SupportsAgentRun]``) is intentionally untouched; the issue scoped the
rename to the workflow attribute.
- Adjacent review candidates (twin ``WorkflowAgent`` translators,
``_AGENT_FORWARDED_EVENT_TYPES`` kind classifier,
``_event_origin_context`` ContextVar removal, ``WorkflowEvent`` ADT
split, legacy-mode removal) remain out of scope.
* Add explicit workflow output designation
Key decisions
- Extend the internal OutputDesignation value type from terminal-only membership to output/intermediate/hidden classification. Legacy mode remains outputs=None, so workflows built without output_executors or intermediate_executors still label every yield_output as type='output'.
- WorkflowBuilder now accepts intermediate_executors. Providing either designation enters explicit mode; output executors emit output, intermediate executors emit intermediate, and unlisted yield_output payloads are hidden from caller-facing events while remaining in executor_completed data.
- Empty explicit designation, duplicate entries, overlaps, unknown executors, and designated executors without workflow output annotations fail build validation. Existing orchestration builders pass intermediate-capable participants through intermediate_executors to preserve current intermediate_outputs behavior until participant-oriented designation lands.
Files changed
- packages/core/agent_framework/_workflows/_workflow.py, _workflow_builder.py, _workflow_context.py, _validation.py, _events.py
- packages/core/tests/workflow/test_output_designation.py, test_output_executors_contract.py, test_strict_mode_event_labeling.py, test_validation.py, test_workflow.py, test_workflow_agent_intermediate.py
- packages/orchestrations/agent_framework_orchestrations/_sequential.py, _concurrent.py, _group_chat.py, _magentic.py
- packages/core/AGENTS.md
Verification
- uv run pytest packages/core/tests/workflow packages/orchestrations/tests packages/devui/tests/devui/test_mapper.py -q
- uv run pytest packages/azurefunctions/tests -q
- uv run poe lint
- uv run poe typing fails only on pre-existing packages/core/agent_framework/_workflows/_agent.py _AGENT_FORWARDED_EVENT_TYPES private-use pyright error.
Notes for next iteration
- issues/03-core-workflow-explicit-designation.md was moved to issues/done but issues/ remains untracked and intentionally excluded from this commit.
- Slice 4 should tighten workflow.as_agent() mapping for hidden emissions and streaming-only update payloads; Slice 5 should replace orchestration intermediate_outputs with participant-oriented designation.
* Tighten workflow-as-agent output mapping
Key decisions
- Treat AgentResponseUpdate as a streaming-only payload across the workflow.as_agent() adapter, so non-streaming agent runs now reject both terminal output and intermediate workflow events carrying updates.
- Keep streaming classification behavior explicit: terminal update payloads remain normal text content, while intermediate update payloads are rewritten to text_reasoning content.
- Add explicit-mode coverage proving hidden yield_output emissions do not appear in non-streaming AgentResponse messages or streaming AgentResponseUpdate chunks.
Files changed
- packages/core/agent_framework/_workflows/_agent.py
- packages/core/tests/workflow/test_workflow_agent_intermediate.py
Verification
- uv run pytest packages/core/tests/workflow/test_workflow_agent_intermediate.py -q
- uv run pytest packages/core/tests/workflow/test_workflow_agent.py packages/core/tests/workflow/test_workflow_agent_intermediate.py -q
- uv run pytest packages/core/tests/workflow packages/orchestrations/tests packages/devui/tests/devui/test_mapper.py -q
- uv run poe lint
- uv run poe typing fails only on the pre-existing packages/core/agent_framework/_workflows/_agent.py _AGENT_FORWARDED_EVENT_TYPES private-use pyright error.
Blockers or notes for next iteration
- issues/04-workflow-as-agent-output-mapping.md was moved to issues/done/ but issues/ remains untracked and intentionally excluded from this commit.
- Slice 5 should replace orchestration intermediate_outputs with participant-oriented designation.
* Add orchestration participant output designation
Key decisions
- Replace orchestration intermediate_outputs with participant-oriented output_participants and intermediate_participants across Sequential, Concurrent, GroupChat, Magentic, and Handoff builders.
- Keep synthetic final executors terminal by default for Concurrent, GroupChat, and Magentic; keep Sequential's final participant terminal by default; keep Handoff participants terminal by default.
- Centralize participant designation validation for empty explicit designation, duplicates, overlaps, and unknown participants, then map validated participants to workflow output/intermediate executors.
Files changed
- packages/orchestrations/agent_framework_orchestrations/_participant_designation.py
- packages/orchestrations/agent_framework_orchestrations/_sequential.py
- packages/orchestrations/agent_framework_orchestrations/_concurrent.py
- packages/orchestrations/agent_framework_orchestrations/_group_chat.py
- packages/orchestrations/agent_framework_orchestrations/_magentic.py
- packages/orchestrations/agent_framework_orchestrations/_handoff.py
- packages/orchestrations/tests/test_orchestration_intermediate_vs_terminal.py
- packages/orchestrations/tests/test_magentic.py
Blockers or notes for next iteration
- issues/05-orchestration-participant-designation.md was moved to issues/done/ but issues/ remains untracked and intentionally excluded from this commit.
- Slice 7 should migrate samples and docs away from intermediate_outputs to the new participant designation API.
- uv run poe typing still fails only on the pre-existing packages/core/agent_framework/_workflows/_agent.py _AGENT_FORWARDED_EVENT_TYPES private-use pyright error.
* Migrate samples to explicit output designation
Key decisions
- Replace sample usage of the removed orchestration intermediate_outputs boolean with participant-oriented intermediate_participants designation.
- Update raw workflow guidance to show output_executors together with intermediate_executors, and document that unlisted yields are hidden in explicit designation mode.
- Keep orchestration final outputs terminal while streaming designated participant responses as intermediate progress, including workflow.as_agent() samples where intermediates map to text_reasoning content.
- Refresh workflow and orchestration README guidance plus the changelog reference so public docs no longer point users at intermediate_outputs.
Files changed
- CHANGELOG.md
- packages/orchestrations/README.md
- samples/README.md
- samples/03-workflows/README.md
- samples/03-workflows/control-flow/intermediate_vs_terminal_outputs.py
- samples/03-workflows/orchestrations/README.md
- samples/03-workflows/orchestrations/group_chat_agent_manager.py
- samples/03-workflows/orchestrations/group_chat_philosophical_debate.py
- samples/03-workflows/orchestrations/group_chat_simple_selector.py
- samples/03-workflows/orchestrations/magentic.py
- samples/03-workflows/orchestrations/magentic_human_plan_review.py
- samples/03-workflows/orchestrations/sequential_chain_only_agent_responses.py
- samples/03-workflows/agents/group_chat_workflow_as_agent.py
- samples/03-workflows/agents/magentic_workflow_as_agent.py
- samples/03-workflows/agents/sequential_workflow_as_agent.py
- samples/semantic-kernel-migration/orchestrations/group_chat.py
- samples/semantic-kernel-migration/orchestrations/magentic.py
Blockers or notes for next iteration
- issues/07-samples-and-docs-explicit-output-designation.md was moved to issues/done/ but issues/ remains untracked and intentionally excluded from this commit.
- issues/06-devui-intermediate-event-rendering.md remains present and appears already satisfied by existing DevUI mapper/tests from the prior implementation slice.
- PRD-explicit-workflow-output-designation.md remains untracked and intentionally excluded from this commit.
* Render DevUI intermediate workflow outputs
Key decisions
- Preserve workflow output designation metadata on visible DevUI output messages and text deltas so intermediate/data emissions remain distinguishable from terminal output.
- Render intermediate workflow message items in the execution timeline using executor metadata, while excluding them from the final workflow result aggregation.
- Keep terminal output message rendering unchanged and retain legacy data events on the intermediate compatibility path.
Files changed
- packages/devui/agent_framework_devui/_mapper.py
- packages/devui/frontend/src/components/features/workflow/execution-timeline.tsx
- packages/devui/frontend/src/components/features/workflow/workflow-view.tsx
- packages/devui/frontend/src/types/openai.ts
- packages/devui/tests/devui/test_mapper.py
Blockers or notes for next iteration
- issues/06-devui-intermediate-event-rendering.md was moved to issues/done/ but issues/ remains untracked and intentionally excluded from this commit.
- PRD-explicit-workflow-output-designation.md remains untracked and intentionally excluded from this commit.
- uv run poe typing still fails only on the pre-existing packages/core/agent_framework/_workflows/_agent.py _AGENT_FORWARDED_EVENT_TYPES private-use pyright error.
* Fix mypy
* Clarify orchestration participant output config
* Rename participant output kwargs for clarity
output_participants -> final_output_from, intermediate_participants ->
intermediate_output_from. The old names read like categories of
participant; the new names make it clear the kwarg designates which
participants' outputs surface as final vs. intermediate events.
* Rename core workflow output kwargs with deprecation shim
Adds final_output_from / intermediate_output_from as canonical kwargs on
Workflow and WorkflowBuilder. Old output_executors / intermediate_executors
kwargs continue to work but emit DeprecationWarning via a shared coalesce
helper that also rejects supplying both. Wire-format keys in to_dict()
stay as output_executors / intermediate_executors so checkpoint
compatibility is preserved.
Internal call sites in orchestrations and samples updated to the new
names so users following sample code learn the canonical vocabulary;
legacy callers still work with a one-shot warning.
* Suppress pyright reportPrivateUsage on cross-module sentinel import
* Update docstrings
* Propagate sub-workflow intermediate outputs, fix handoff/sequential intermediate-only designation, and shore up tests, sample, and docstrings around the intermediate output contract.
* Add canonical workflow output_from selection
Key decisions:\n- Make output_from the canonical workflow-output allow-list and keep output_executors/final_output_from as deprecated compatibility aliases.\n- Treat empty output_from/intermediate_output_from lists as explicit selections and keep validation responsible for empty, duplicate, overlap, and unknown selections.\n- Remove the branch-only public intermediate_executors WorkflowBuilder kwarg while preserving legacy wire keys in to_dict().\n\nFiles changed:\n- packages/core/agent_framework/_workflows/_workflow.py\n- packages/core/agent_framework/_workflows/_workflow_builder.py\n- packages/core/agent_framework/_workflows/_workflow_context.py\n- packages/core/agent_framework/_workflows/_agent.py\n- packages/core/agent_framework/_workflows/_agent_executor.py\n- packages/core/tests/workflow/* output-selection coverage updates\n- packages/core/AGENTS.md\n- issues/done/001-canonical-list-based-output-selection.md\n\nBlockers/notes:\n- Orchestration builders still pass final_output_from internally; follow-up issue 004 should migrate them to output_from.\n- Legacy omitted-selection behavior and explicit all/all_other literals are left for issues 002 and 003.
* Add explicit all workflow output selection
Key decisions:
- Treat output_from='all' as an explicit workflow-output selection sentinel and expand it at build time to executors with declared workflow output types.
- Keep omitted output selections in legacy all-output mode with a deprecation warning that names output_from and intermediate_output_from and points to output_from='all'.
- Reject intermediate_output_from='all' at construction because the all-output literal is output-only for this issue.
Files changed:
- packages/core/agent_framework/_workflows/_workflow_builder.py
- packages/core/tests/workflow/test_output_executors_contract.py
- issues/done/002-explicit-all-output-and-legacy-migration.md
Blockers/notes:
- all_other intermediate-output selection remains for issue 003.
- Workflow-as-agent/orchestration parity remains for issue 004.
* Add all-other intermediate output selection
Key decisions:
- Treat intermediate_output_from='all_other' as an explicit intermediate-output selection sentinel and expand it at build time after the workflow graph is complete.
- Expand all_other to output-capable executors not selected by output_from; omitted or empty output_from selects no workflow outputs, while output_from='all' leaves an empty intermediate selection.
- Keep output_from='all_other' invalid so all_other remains intermediate-output-only and runtime classification still receives concrete executor-id sets.
Files changed:
- packages/core/agent_framework/_workflows/_workflow_builder.py
- packages/core/tests/workflow/test_output_executors_contract.py
- issues/done/003-all-other-intermediate-output-selection.md
Blockers/notes:
- Workflow-as-agent and orchestration parity remains for issue 004.
- Full documentation updates remain for issue 005.
* Add orchestration output selection parity
Key decisions:
- Expose output_from on sequential, concurrent, group chat, handoff, and magentic builders while keeping final_output_from as a deprecated compatibility alias.
- Resolve orchestration participant selections through the same explicit rules as workflows: output_from='all', intermediate_output_from='all_other', hidden unselected participant payloads, and overlap/duplicate/unknown/invalid-literal validation.
- Continue preserving documented orchestration defaults by always designating each pattern's terminal internal executor where applicable.
Files changed:
- packages/orchestrations/agent_framework_orchestrations/_participant_output_config.py
- packages/orchestrations/agent_framework_orchestrations/_sequential.py
- packages/orchestrations/agent_framework_orchestrations/_concurrent.py
- packages/orchestrations/agent_framework_orchestrations/_group_chat.py
- packages/orchestrations/agent_framework_orchestrations/_handoff.py
- packages/orchestrations/agent_framework_orchestrations/_magentic.py
- packages/orchestrations/agent_framework_orchestrations/_orchestration_request_info.py
- packages/orchestrations/tests/test_orchestration_intermediate_vs_terminal.py
- issues/done/004-workflow-as-agent-and-orchestration-parity.md
Blockers/notes:
- Full documentation and sample migration wording remains for issue 005.
- Existing tests that intentionally use final_output_from now emit the new deprecation warning.
* Document workflow output selection contract
Key decisions:
- Use Workflow Output and Intermediate Output as the developer-facing terms for selected caller-facing emissions.
- Document output_from and intermediate_output_from as the canonical API, with output_from as an allow-list and unselected payloads hidden unless explicitly selected as intermediate.
- Add scenario and invalid-selection tables for workflow and orchestration docs, including legacy omission warnings, output_from='all', intermediate_output_from='all_other', list selections, invalid literals, overlap, duplicates, unknown selections, and empty explicit selections.
- Migrate samples away from final_output_from and output_executors except where compatibility aliases are explicitly documented.
Files changed:
- packages/core/AGENTS.md
- packages/orchestrations/README.md
- packages/orchestrations/agent_framework_orchestrations/_handoff.py
- packages/orchestrations/agent_framework_orchestrations/_sequential.py
- samples/03-workflows/README.md
- samples/03-workflows/control-flow/intermediate_vs_terminal_outputs.py
- samples/03-workflows/human-in-the-loop/agents_with_approval_requests.py
- samples/03-workflows/orchestrations/README.md
- samples/04-hosting/foundry-hosted-agents/responses/05_workflows/main.py
- scripts/sample_validation/create_dynamic_workflow_executor.py
- issues/done/005-document-output-selection-contract.md
Blockers/notes:
- Direct full Ruff on scripts/sample_validation/create_dynamic_workflow_executor.py still reports pre-existing docstring/print/line-length issues outside this docs migration; syntax-focused checks for changed files pass.
- No remaining AFK issue files are present under issues/.
* Latest updates
* Typing fixes
* Cleanup
* .NET: Bump Azure.AI.Projects to 2.1.0-beta.2 and add agent-endpoint AsAIAgent path
Bumps Azure.AI.Projects to 2.1.0-beta.2 with the matching transitive pins (Azure.Core 1.55.0, System.ClientModel 1.11.0).
Foundry agent endpoint plumbing:
* FoundryAgent now routes the agent-endpoint constructor through the new GetProjectResponsesClientForAgentEndpoint helper.
* Adds an internal FoundryAgent ctor that takes an existing AIProjectClient plus a parsed agent endpoint so the public extension does not need to construct a second project client.
* Adds public AIProjectClient.AsAIAgent(Uri agentEndpoint, ...) extension. This is the path consumer samples are expected to use for hosted agents because version selection happens server-side.
* Trims the dangling "If you want to construct a FoundryAgent against a project endpoint..." sentence from ParseAgentEndpoint.
Unit tests:
* Four new tests in AzureAIProjectChatClientExtensionsTests cover the AIProjectClient.AsAIAgent(Uri agentEndpoint, ...) overload. 263/263 pass.
Consumer samples (Using-Samples):
* SimpleAgent and SessionFilesClient now read AZURE_AI_PROJECT_ENDPOINT and AZURE_AI_AGENT_NAME (both required, throw on missing), derive the agent endpoint with new Uri($"{projectEndpoint}/agents/{agentName}/endpoint/protocols/openai"), then call aiProjectClient.AsAIAgent(agentEndpoint, ...).
* SessionFilesClient README updated.
Contributor samples (responses/*):
* New HostedContributorRouteExtensions.MapDevTemporaryLocalAgentEndpoint() wildcard route extension so localhost contributor servers accept the per-agent OpenAI endpoint shape the production Hosted runtime exposes.
* All 11 contributor Program.cs files call MapDevTemporaryLocalAgentEndpoint() with a contributor-only warning comment.
* Hosted-Files and Hosted-AzureSearchRag were importing Hosted_Shared_Contributor_Setup but never calling AddDevTemporaryLocalContributorSetup(). Both now call it so HostedSessionIsolationKeyProvider resolves correctly in dev.
* Hosted-AzureSearchRag, Hosted-Files, Hosted-MemoryAgent csprojs drop stale VersionOverride="2.1.0-beta.1" pins.
* Hosted-AzureSearchRag and Hosted-Files csprojs add ProjectReference to Hosted_Shared_Contributor_Setup.
* Hosted-Observability/.dockerignore removed the out/ exclusion that was blocking COPY out/ . in Dockerfile.contributor.
Verified:
* Full solution-scoped build of changed projects: green.
* Scoped CI-parity dotnet format via WSL2 + Docker (mcr.microsoft.com/dotnet/sdk:10.0) over every changed csproj: clean.
* Foundry unit tests: 263/263.
* Contributor docker smoke for 8 hosted samples (publish + docker build + docker run + curl POST to the wildcard route): HTTP 200 / 500 with route matched.
* End-to-end smoke against the real Azure Foundry project with a fresh bearer token: Hosted-Files contributor container served HTTP 200, the agent invoked ListBundledFiles, and returned the expected file name.
* Address PR review: forward pipeline settings; add UTs
- CreateProjectClientOptions also carries RetryPolicy, NetworkTimeout, ClientLoggingOptions, MessageLoggingPolicy (was Transport+UserAgentApplicationId only).
- Make CreateProjectClientOptions internal so tests can verify the copy directly.
- Add AsAIAgent(Uri) UTs covering tools forwarding to inner ChatOptions and null tools handling.
- Add CreateProjectClientOptions UTs covering null caller and full pipeline-settings copy.
* Fix GitHubCopilotAgent ignoring tools from context providers (#5736)
_create_session and _resume_session only forwarded self._tools (constructor
tools) to CopilotClient.create_session, dropping any tools contributed by
context providers via session_context.extend_tools() during before_run.
Merge provider-contributed tools into runtime_options in both _run_impl and
_stream_updates before session creation, mirroring how RawAgent handles the
merge at lines 1435-1440 in _agents.py. Update _create_session and
_resume_session to combine self._tools with the merged runtime tools.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Python: Fix GitHubCopilotAgent to include tools added by ContextProvider.before_run in session creation
Fixes#5736
* Fix provider tool merge to avoid mutating caller's list
- Replace in-place .extend() with fresh list creation in both
_run_impl and _stream_updates paths to prevent mutating the
caller-provided options['tools'] list (shallow copy issue)
- Also handles immutable Sequence types (e.g. tuple) correctly
- Add test for provider tools forwarded via _resume_session path
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback for #5736: review comment fixes
---------
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The frontmatter parser previously matched only single-line `key: value` pairs, so block scalar indicators (`|` literal, `>` folded, with chomping `-`/`+`) were silently truncated to the indicator character. Multi-line descriptions like `description: >\n ...` lost their content.
Add `_parse_yaml_scalar_value()` which detects block scalar indicators, collects indented continuation lines, strips the common leading indentation, joins per scalar style (newlines for `|`, spaces for `>`), and applies chomping per the YAML 1.2 spec. Update `_extract_frontmatter()` to use the helper for unquoted values.
Adds 15 unit tests covering literal/folded styles, all chomping variants, indentation handling, content containing colons, non-description fields, tab indentation, blank-line preservation, and a regression test for plain values.
Fixes#5713.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* .NET: Add Hosted-MemoryAgent sample with isolation key plumbing (#5692)
Adds HostedSessionContext + HostedSessionIsolationKeyProvider in Microsoft.Agents.AI.Foundry.Hosting so AIContextProviders (notably FoundryMemoryProvider) can scope per user via the platform's x-agent-user-isolation-key / x-agent-chat-isolation-key headers.
- New types: HostedSessionContext (sealed), HostedSessionContextExtensions (public Get, internal Set), abstract HostedSessionIsolationKeyProvider (async), internal PlatformHostedSessionIsolationKeyProvider mapping ResponseContext.Isolation.
- AgentFrameworkResponseHandler now resolves the provider, tags fresh sessions, and validates resumed sessions against the live request (strict 403 'Hosted session identity context mismatch' on any mismatch; 500 on null keys).
- New shared sample project Hosted_Shared_Contributor_Setup hosts DevTemporaryTokenCredential and DevTemporaryLocalSessionIsolationKeyProvider plus AddDevTemporaryLocalContributorSetup. All 9 existing responses samples migrated to consume it so local runs keep working under the strict isolation contract.
- New Hosted-MemoryAgent sample: travel assistant wired through FoundryMemoryProvider with stateInitializer reading session.GetHostedContext().UserId. Includes Dockerfile, smoke.ps1, agent.yaml/manifest.
- New IT scenario 'memory' in Foundry.Hosting.IntegrationTests + MemoryHostedAgentFixture + MemoryHostedAgentTests. Verified end to end against the tao Foundry project.
- ADR 0026 captures the design tree.
* Address PR review feedback
- Dockerfile: add header noting it targets NuGet builds; contributors must use Dockerfile.contributor for ProjectReference source builds.
- PlatformHostedSessionIsolationKeyProvider: doc said 'returns context with empty values'; corrected to 'returns null' which the handler treats as 500.
- FakeHostedSessionIsolationKeyProvider: doc clarifies that null configurations are allowed for testing the handler error path.
- HostedSessionContextExtensions.SetHostedContext: enforce write-once with InvalidOperationException; doc + xml exception updated.
- AgentFrameworkResponseHandler: cache PlatformHostedSessionIsolationKeyProvider as static readonly to avoid per-request allocation.
- MemoryHostedAgentTests: tighten waits from 20s to 5s (FoundryMemoryProvider defaults UpdateDelay=0; ingestion ~3s).
- Sample Program.cs imports reordered to satisfy IDE0005.
* Add HostedFoundryMemoryProviderScopes built-in helpers (#5692)
Addresses review feedback from @lokitoth on Hosted-MemoryAgent/Program.cs:54.
- New HostedFoundryMemoryProviderScopes static class with PerUser, PerChat, PerUserAndChat factories returning Func<AgentSession?, FoundryMemoryProvider.State>.
- All helpers throw InvalidOperationException when GetHostedContext() is null, with a message pointing at writing a custom stateInitializer for non-hosted scenarios.
- New HostedFoundryMemoryScope enum and AddHostedFoundryMemoryProvider DI extension (two overloads: explicit AIProjectClient and DI-resolved). Singleton lifetime. Default scope = PerUser.
- Hosted-MemoryAgent sample and the memory IT scenario container both swap their inline lambdas for HostedFoundryMemoryProviderScopes.PerUser().
- 14 new unit tests (241/241 hosting unit tests pass).
* Replace HostedFoundryMemoryScope enum with Func<...> parameter (#5692)
Address PR review feedback from @westey-m: enums are a breaking-change hazard when extended, and the enum was redundant with the existing HostedFoundryMemoryProviderScopes static class.
- Delete HostedFoundryMemoryScope.cs.
- AddHostedFoundryMemoryProvider DI extensions now take Func<AgentSession?, FoundryMemoryProvider.State>? stateInitializer = null. When null, default to HostedFoundryMemoryProviderScopes.PerUser().
- Callers pick a built-in helper (PerUser/PerChat/PerUserAndChat) or pass a custom delegate. New built-ins are a single static method addition with zero impact on existing callers.
- Tests updated; 244/244 hosting unit tests pass.
* Fix isolation context resume for externally-created conversations (#5692)
Branch on the session's existing hosted-context (not on conversation_id presence) so a conversation provisioned externally (e.g. via conversations.CreateProjectConversationAsync) is treated as fresh on first hosted-agent request and stamped, rather than rejected with 403 hosted_session_identity_mismatch. Strict equality is preserved on real resume of an already-stamped session.
Also tighten dotnet/global.json to version 10.0.204 + rollForward latestPatch so local builds match the CI Docker image SDK and avoid 10.0.300 dotnet format stripping required usings.
* Revert global.json SDK pin to upstream (#5692)
The 10.0.204 + latestPatch pin from the previous commit broke the dotnet-format CI job (hostfxr_resolve_sdk2 could not find a compatible SDK in the mcr.microsoft.com/dotnet/sdk:10.0 image). Restore upstream 10.0.200 + minor; local Release builds with SDK 10.0.300 should set GITHUB_ACTIONS=true to bypass the auto-format-on-build target.
- Fix non-streaming empty response by accumulating intermediate WORKING
status updates and flushing them when an empty terminal event arrives
- Fix sample agent_executor.py to enqueue Task before status events
(required by v1.0 ActiveTask validation)
- Fix create_jsonrpc_routes() calls to include required rpc_url param
- Fix TYPE_CHECKING imports in sample agent_definitions.py
- Add tests for non-streaming content accumulation behavior
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Restructure harness console so that reactive app is the entry point
* Further refactoring to split tool formatters, improve UX, make console configurable and fix bugs
* Address PR comments.
* UX tweak
* Fix streaming text bug
* Address PR comments.
TestServerWithDevUI_ResolvesMixedAgentsAndWorkflows_AllRegistrationsAsync fails intermittently in the merge_group with NRE on the discovery response, blocking PRs unrelated to DevUI from merging. Skip via Fact(Skip=...) referencing #5845 while the underlying race is investigated.
* Python: DevUI: tighten default access controls and CORS posture
Adjusts the default configuration of the DevUI server so the out-of-the-box
posture matches what most callers expect when running locally. Adds explicit
opt-outs for callers who need the previous behavior.
- DevServer gains auth_enabled and auth_token constructor params; auth is on by
default. Auto-generates and logs a token when none provided.
- CORS default is an empty allowlist on every host. Callers wanting cross-origin
pass cors_origins explicitly.
- Streaming /v1/responses no longer sets Access-Control-Allow-Origin directly;
CORSMiddleware owns all CORS decisions.
- Loopback binds enforce a Host-header allowlist.
- /meta moved out of the auth bypass list (was alongside /health and /).
- serve() default flipped to auth_enabled=True; passes auth args through to
DevServer instead of using env-var indirection.
- CLI: --auth opt-in replaced with --no-auth opt-out; --auth-token preserved.
- Tests cover the eight behaviors above in test_server.py.
* Python: DevUI: address PR review comments
- /meta now derives auth_required from self.auth_enabled instead of
reading DEVUI_AUTH_TOKEN, so the auto-generated and explicit
auth_token paths report correctly.
- Reorder middleware so the loopback Host-header allowlist is registered
last; Starlette wraps later-added middleware around earlier-added ones,
so the host check now runs outermost (before CORS/auth) as intended.
- Rework comments to describe the behavior rather than threat scenarios.
- Streaming-headers and CORS tests now construct the server with an
explicit auth_token and send a Bearer header, so the assertions
actually exercise the streaming/CORS path instead of short-circuiting
in the auth middleware.
Replaces two ILogger.LogWarning(string, params object?[]) calls in DevUIAuthFilter and DevUIExtensions with allocation-free [LoggerMessage] partial methods on a new internal DevUILog class. Preserves original message templates and structured property names ({RemoteIp}, {EnvVar}).
Co-authored-by: alliscode <25218250+alliscode@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fixes microsoft/agent-framework#3295. When the OpenAI Responses chat
client sends a request that carries previous_response_id / conversation_id
/ conversation, the server already has the prior turn's response items
and rejects duplicates with "Duplicate item found with id fc_xxx". The
chat client was re-sending them inline whenever the input messages still
carried the items in additional_properties (workflow replay, history
providers, etc.), which broke any tool-using agent with persistent
history.
Decisions:
- Single chokepoint: _prepare_message_for_openai. When the resulting
request uses service-side storage, drop function_call, reasoning,
approval-request/response, and local-shell-call items from the wire
input. Keep function_result with its call_id; the server pairs it to
the prior function_call via that key.
- function_result is preserved unconditionally except for the local-shell
variant, which carries its own server-issued item id.
- No public API change. Wire format change is subtractive and only on
requests that would otherwise 400.
- Re-pointed the strict-xfail in test_full_conversation.py from #4047 to
#3295. Kept xfail because the test asserts executor-level session-id
clearing, which is the defense-in-depth half tracked by 3295-03; this
slice closes the wire-level half.
Files:
- python/packages/openai/agent_framework_openai/_chat_client.py: strip
rule applied alongside the existing reasoning-item branch.
- python/packages/openai/tests/openai/test_openai_chat_client.py: four
new tests pin the contract (function_call, approval, local-shell-call
stripped under storage; everything kept without storage). Updated
pre-existing tests that exercised the storage-on path to either pass
request_uses_service_side_storage=False explicitly or assert the new
strip behavior.
- python/packages/foundry/tests/foundry/test_foundry_chat_client.py:
same explicit storage-off opt-in for the inherited test.
- python/packages/core/tests/workflow/test_full_conversation.py:
re-pointed xfail reason to #3295 and the executor-level follow-up.
Notes for next iteration:
- 3295-01 (HITL wire-format validation against live OpenAI/Foundry) was
not run; it requires the user's API credentials. The PRD design is
locked but the empirical confirmation is still pending. If script 3
fails on either provider, this slice may need to be revisited.
- 3295-03 (clear service_session_id in AgentExecutor on full-history
replay) remains open. After it lands the xfail in
test_full_conversation.py can be removed.
- pytest was not run in this iteration because uv-based pytest commands
required interactive approval. Validation rests on careful reading;
next iteration should run the openai + core test suites.
* Fix Skill docstring consistency and spelling
- Add ClassSkill to Skill class docstring concrete implementations list
- Normalize 'defence' to 'defense' for American English consistency
- Remove extra blank line in InlineSkill docstring example
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix E501 line-too-long lint error in test_skills.py
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix stale test section header to reflect SkillFrontmatter API
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix metadata children overriding top-level frontmatter fields
Scope YAML_KV_RE to column-0 keys only so indented children
under metadata: are not mistakenly parsed as top-level fields.
Add regression test and spec fields to sample SKILL.md files.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix(python): prevent MCP message_handler deadlock on notification reload
When an MCP server sends a notifications/tools/list_changed or
notifications/prompts/list_changed notification, the message_handler
previously awaited load_tools()/load_prompts() directly. Since the
handler runs on the MCP SDK's single-threaded receive loop, this
caused a deadlock: load_tools() sends a list_tools request and waits
for its response, but the receive loop cannot deliver that response
while blocked in the handler.
This manifested as a timeout in call_tool(), which then surfaced as
"Error: Function failed." to the model instead of the real tool
output. The MATLAB MCP server reliably triggers this because it sends
a tools/list_changed notification during tool execution.
Fix: schedule reloads as background asyncio.Tasks via a new
_schedule_reload() helper, freeing the receive loop immediately.
Fixes#4828
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR review feedback: fix exc_info, coalesce reloads, shutdown cleanup, tests
- Fix exc_info=exc -> exc_info=True in _schedule_reload and message_handler
- Tighten _schedule_reload param type from Any to Coroutine[Any, Any, None]
- Coalesce reloads: cancel-and-replace per reload kind to prevent unbounded growth
- Cancel pending reload tasks in _close_on_owner before tearing down session
- Re-raise CancelledError in _safe_reload to respect task cancellation
- Replace flaky asyncio.sleep(0) with asyncio.wait_for/gather in tests
- Add caplog assertions to verify reload failure is actually logged
- Assert _pending_reload_tasks cleanup on error path
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: address review comments on MCP reload handling
- Fix exc_info=True -> exc_info=message in message_handler error logging,
since the handler is not called from an except block
- Await cancelled reload tasks in _close_on_owner before tearing down
the session to avoid 'Task was destroyed but pending' warnings
- Add cancel-and-replace test verifying duplicate notifications cancel
the first reload task and only keep one in flight
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: remove Task.cancelling() call for Python 3.10 compat
Task.cancelling() was added in Python 3.11. Replace with awaiting
the task and checking cancelled() instead.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add debug log when cancelling superseded reload task
Log at DEBUG level when a new notification cancels an in-flight reload
task, improving observability of the cancel-and-replace behavior.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* .NET: feat(evals): add ground_truth/expected_output support for workflow eval
Brings .NET to parity with Python PR #5234 for issue #5135:
- Add expectedOutput parameter to Run.EvaluateAsync (workflow) and stamp on the overall EvalItem.ExpectedOutput.
- Map EvalItem.ExpectedOutput -> ground_truth in the Foundry JSONL payload, item_schema, and data_mapping for similarity.
- Add GroundTruthEvaluators set (currently builtin.similarity) and a FindMissingGroundTruthEvaluators helper.
- Fail fast with InvalidOperationException when a ground-truth evaluator is selected but no item provides an ExpectedOutput, instead of surfacing a remote provider error.
- Add tests in FoundryEvalConverterTests and WorkflowEvaluationTests.
- Add Evaluation_WorkflowExpectedOutputs sample (workflow + Foundry similarity).
Fixesmicrosoft/agent-framework#5135 (.NET side).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review: relax BuildOverallItem events to IReadOnlyList<WorkflowEvent>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Sample: disable per-agent breakdown when using reference-based evaluator
Per-agent EvalItems are intentionally left without ExpectedOutput, so the new fail-fast validation in FoundryEvals would throw when Similarity is invoked for per-agent items. Pass includePerAgent: false in the workflow + similarity sample, and document this gotcha in the EvaluateAsync XML doc.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix BuildOverallItem: fall back to last ExecutorCompletedEvent
AgentResponseEvent is only emitted when AIAgentHostOptions.EmitAgentResponseEvents is enabled, which is not the default for WorkflowBuilder(agent).AddEdge(...). When it is absent, fall back to the last non-internal ExecutorCompletedEvent whose Data is an AgentResponse / ChatMessage / string so the overall EvalItem (and any expectedOutput) is produced. Without this, samples wired up the standard way returned 0 evaluation items.
Update test to cover the fallback path.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Sample: enable EmitAgentResponseEvents; eval throws clear error when no overall response found
Root cause of '0 results': AIAgentHostExecutor only emits AgentResponseEvent when AIAgentHostOptions.EmitAgentResponseEvents is true (default false). For ordinary AIAgent executors the runtime's ExecutorCompletedEvent.Data is null, so the prior fallback couldn't find a final response either.
Sample now builds executors with EmitAgentResponseEvents=true via BindAsExecutor(hostOptions). EvaluateAsync now throws InvalidOperationException with a remediation hint when the user supplies expectedOutput but no overall final response can be located, instead of silently returning 0/0.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Guard against null sample/error/usage/datasource_item in ParseDetailedItem
Foundry eval responses can have these properties present with JSON null
or non-object values, which caused JsonElement.TryGetProperty to throw
'requires Object, has Null'. Check ValueKind == Object before drilling in.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR review: reorder expectedOutput, tighten ground-truth check, add fail-fast test
* WorkflowEvaluationExtensions.EvaluateAsync: move 'expectedOutput' to
after 'splitter' so the original positional contract of (splitter,
cancellationToken) is preserved for existing callers.
* FoundryEvals: require ALL items to carry ExpectedOutput when a
ground-truth evaluator is selected (e.g. similarity), not just any.
Reference-based evaluators score per-item, so a single missing GT
would still surface as a provider-side validation error. Updated
fail-fast message accordingly.
* WorkflowEvaluationTests: add EvaluateAsync_WithExpectedOutputButNoFinalResponse_ThrowsAsync
to verify the InvalidOperationException is thrown (and that the
message mentions EmitAgentResponseEvents).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fail-fast on missing overall item regardless of expectedOutput; harden BuildOverallItem default
* EvaluateAsync now throws InvalidOperationException whenever 'includeOverall'
is requested but BuildOverallItem cannot produce an item, instead of only
when 'expectedOutput' is supplied. Same misconfiguration (agents not bound
with EmitAgentResponseEvents) used to silently return empty results — now
it surfaces a clear, actionable error in both cases.
* BuildOverallItem switch default now throws instead of returning null. The
preceding for-loop already constrains Data to AgentResponse/ChatMessage/
string, so reaching default would indicate a contract drift; throw to make
the bug visible.
* Test renamed and broadened to verify the throw fires without expectedOutput.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: alliscode <25218250+alliscode@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Evan Mattson ·
Eduard van Valkenburg ·
westey ·
Taisir Hassan ·
Tao Chen ·
Peter Ibekwe ·
Roger Barreto ·
Yufeng He ·
Giles Odigwe ·
Challa Ravi ·
SergeyMenshykh ·
Copilot ·
Ben Thomas ·