Jacob Alber 8ed2159c4b .NET: Workflow Outputs Overhaul: Support Tagging, Filtering Agent Outputs (#6045) ...

* test: reshuffle .NET Workflow tests in preparation for Outputs overhaul

Phase 1 of the .NET Workflows outputs overhaul (see
working/implementation-plan.md). Pure moves/renames in
dotnet/tests/Microsoft.Agents.AI.Workflows.UnitTests; no production code
changes, no new test cases. The split keeps each orchestration mode in
its own source file so the upcoming tag-aware and orchestration-default
test additions land on clean diffs.

Renames:
* WorkflowBuilderSmokeTests.cs -> WorkflowBuilderTests.cs (with class
  rename to match). The scope is no longer "smoke"-only once subsequent
  phases add tag-aware builder tests.
* InputWaiterAndOutputFilterTests.cs -> InputWaiterTests.cs +
  OutputFilterTests.cs. The file already declared the two test classes
  separately; this split simply gives each its own file so the
  output-filter cases have a dedicated home for tag-aware additions.

Split of AgentWorkflowBuilderTests.cs:
* AgentWorkflowBuilderTests.cs is now the outer
  `public static partial class AgentWorkflowBuilderTests` holding the
  shared test helpers (DoubleEchoAgent + session + WithBarrier variant,
  WorkflowRunResult, RunWorkflow* methods) bumped from `private` to
  `internal` so the new top-level GroupChatWorkflowBuilderTests in the
  same assembly can reach them.
* AgentWorkflowBuilder.SequentialTests.cs (nested SequentialTests):
  BuildSequential_InvalidArguments_Throws,
  BuildSequential_AgentsRunInOrderAsync.
* AgentWorkflowBuilder.ConcurrentTests.cs (nested ConcurrentTests):
  BuildConcurrent_InvalidArguments_Throws,
  BuildConcurrent_AgentsRunInParallelAsync.

Sequential and Concurrent are kept as nested classes because they're
modes of the same `AgentWorkflowBuilder` static factory and do not
produce dedicated builder types.

New file:
* GroupChatWorkflowBuilderTests.cs (top-level): the existing
  BuildGroupChat_* and GroupChatManager_* cases moved out of the old
  AgentWorkflowBuilderTests file. They exercise the
  `GroupChatWorkflowBuilder` type (returned by
  `AgentWorkflowBuilder.CreateGroupChatBuilderWith`), so a dedicated
  top-level test class - matching the convention reserved by the plan
  for HandoffWorkflowBuilderTests / MagenticWorkflowBuilderTests - is
  the right home. Cross-class helper references qualify with
  `AgentWorkflowBuilderTests.DoubleEchoAgent` and
  `AgentWorkflowBuilderTests.RunWorkflowAsync`.

The outer partial class is `static` (and nested classes carry the
instance test methods) because the outer holds only static helpers;
this satisfies CA1052 without suppressions and is invisible to xUnit
discovery, which finds tests on the nested classes as
`AgentWorkflowBuilderTests.SequentialTests.*` etc.

Validation: `dotnet build` clean on both target frameworks; all 547
tests in Microsoft.Agents.AI.Workflows.UnitTests pass on net10.0.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat: introduce OutputTag, Futures, and tag-aware WorkflowBuilder API

Phase 2 of the .NET Workflows outputs overhaul. Additive code change
only - no observable runtime behavior change. The runner still uses the
legacy bypass for AgentResponse / AgentResponseUpdate payloads, and the
new `Futures.EnableAgentResponseOutputTaggingAndFiltering` flag defaults
to false. Phase 3 will wire the flag into the runner; this commit only
introduces the types and the builder API.

New public surface:
* `OutputTag` (readonly struct): wraps a string Value with ordinal
  equality (IEquatable, GetHashCode, == / !=) so it can participate as a
  HashSet element. Internal ctor closes the set. One public singleton:
  `OutputTag.Intermediate`. Terminal / regular outputs carry no tag
  (empty Tags set). JSON-serialized as a bare string via
  [JsonConverter(typeof(OutputTagJsonConverter))], with the converter
  rehydrating to the well-known singleton on read.
* `Futures` (static class): hosts opt-in pre-GA behavior switches.
  First flag is `EnableAgentResponseOutputTaggingAndFiltering`; XML doc
  captures the v2.0.0 obsoletion / v3.0.0 removal lifecycle.
* `WorkflowOutputEvent.Tags`: `HashSet<OutputTag>` exposed directly
  (concrete collection, matches the JSON-serialization convention used
  for `WorkflowInfo.OutputExecutorIds`). Never null; empty for legacy /
  terminal events. New ctors take a single `OutputTag` or
  `IEnumerable<OutputTag>?`; the existing (data, executorId) ctor
  remains and produces an untagged event. `HasTag(OutputTag)` helper.
  `AgentResponseEvent` and `AgentResponseUpdateEvent` gain matching
  tag-accepting ctors forwarding to the base.
* `WorkflowOutputEventExtensions.IsIntermediate(this WorkflowOutputEvent)`:
  extension method returning `evt.HasTag(OutputTag.Intermediate)`. The
  preferred way to ask "is this an intermediate output?" without
  reaching into the Tags set.
* `WorkflowBuilder.WithOutputFrom(IEnumerable<ExecutorBinding>, OutputTag)`
  and `WorkflowBuilder.WithOutputFrom(ExecutorBinding, OutputTag)`:
  forward-looking tagged overloads. The IEnumerable form is the primary
  tagged surface; the single-executor form is a convenience for the
  common one-executor case. Currently usable for the
  `OutputTag.Intermediate` singleton; will become the primary surface
  once the `OutputTag` constructor is opened to user-defined tags in
  a future release. Callers in this release should prefer the
  intent-specific `WithIntermediateOutputFrom` extension for the
  intermediate case. Tags accumulate across repeated calls; same tag
  repeated dedupes via the HashSet.
* `WorkflowBuilderExtensions.WithIntermediateOutputFrom(this WorkflowBuilder, IEnumerable<ExecutorBinding>)`:
  helper that forwards to `WithOutputFrom(executors, OutputTag.Intermediate)`.
  Takes an IEnumerable (matching the tagged WithOutputFrom shape) -
  callers pass collection literals: `builder.WithIntermediateOutputFrom([a, b])`.
  XML doc remarks call out the Futures-flag interaction and the
  AIAgent-payload forwarding contract.

Internal shape changes:
* `WorkflowBuilder._outputExecutors`: HashSet<string> -> Dictionary<
  string, HashSet<OutputTag>>. The value set is empty for executors
  designated only via the untagged WithOutputFrom; contains Intermediate
  (and possibly future tags) otherwise.
* `Workflow.OutputExecutors`: HashSet<string> -> Dictionary<string,
  HashSet<OutputTag>>.
* `OutputFilter.CanOutput`: `Contains(id)` -> `ContainsKey(id)`.
* `WorkflowInfo.OutputExecutorIds`: HashSet<string> -> Dictionary<
  string, HashSet<OutputTag>>, with a custom JsonConverter that reads
  both the new map shape (`{id: ["intermediate", ...]}`) and the legacy
  array shape (`[id1, id2]`, where each id is treated as an untagged
  output). Always writes the map shape. IsMatch updated to compare
  per-id tag sets.

Tests landing in this commit (per the test-with-feature principle):
* `OutputTagTests.cs` (6 tests): KnownValues, EqualityIsOrdinalOnValue,
  DefaultStructValueIsDistinct (default(OutputTag) does not collide
  with the Intermediate singleton in a HashSet),
  GetHashCodeMatchesEquals, JsonConverter_RoundtripsValueAsString,
  ConstructorIsInternal (reflection-based assertion that the (string)
  ctor is `internal`).
* `WorkflowBuilderTests.cs` adds 7 new tests pinning the builder
  API contract: RegistersWithEmptyTagSet, AddsIntermediateTag,
  MultipleExecutorsAllUntagged, ThenIntermediate_AccumulatesTags,
  RepeatedDedupes, OnlyRegistersWithoutPriorWithOutputFrom,
  TracksExecutorBinding.
* `BackwardsCompatibility/JsonCheckpointSerializationTests.cs`
  (new folder + file, 5 tests): event-level ctor contract tests
  (single-tag, no-tag, multi-tag — the last with a custom tag);
  IsIntermediate() asserted; load-bearing JSON BC tests for
  `WorkflowInfo.OutputExecutorIds` -
  `WorkflowOutputExecutorsReadsLegacyArrayShape` (legacy ids map to
  empty tag sets) and `WorkflowOutputExecutorsWritesMapShape`.

The plan's three JSON round-trip tests for `WorkflowOutputEvent.Tags`
were dropped: `WorkflowEvent` is not currently a serialized checkpoint
shape (see the comment in WorkflowsJsonUtilities.cs about events not
being persisted), so there is no real back-compat surface to pin
through JSON. They are substituted with in-process ctor/property
round-trip tests that exercise the `Tags` / `HasTag` / `IsIntermediate`
contract.

Validation: full `Microsoft.Agents.AI.Workflows.UnitTests` suite runs
green on net10.0 (565 passing, 0 failing). Core library builds clean
on net472, netstandard2.0, net8.0, net9.0, and net10.0. Test project
builds clean on net472 + net10.0.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat: route AgentResponse(Update) through the output filter under a Futures flag

`InProcessRunnerContext.YieldOutputAsync` historically special-cased AgentResponse and
AgentResponseUpdate payloads: it built the typed event subclass and emitted it directly,
bypassing the output filter. Rewrites the method so that:

- When `Futures.EnableAgentResponseOutputTaggingAndFiltering` is `false` (the current
  default), AgentResponse(Update) keep the legacy bypass — emitted as
  AgentResponseEvent / AgentResponseUpdateEvent with no tags. Existing callers see no
  behavior change.
- When the flag is `true`, AIAgent payloads flow through the output filter just like
  every other payload type: undesignated sources are dropped, and the emitted event
  carries the source's tag set (empty for terminal `WithOutputFrom`, `{Intermediate}`
  for `WithIntermediateOutputFrom`, the set union when both designations apply).

Non-AIAgent (POCO) outputs also now carry the source's tag set on the emitted
WorkflowOutputEvent unconditionally — additive, since no existing assertion inspected
Tags. Subclass events (`AgentResponseEvent` / `AgentResponseUpdateEvent`) continue to
be emitted under both modes so `switch (evt) { case AgentResponseEvent: ... }`
consumer code keeps matching.

Adds `OutputFilter.TryGetTags` as the tag-aware lookup used by the runner.
`OutputFilter.CanOutput` is kept (still used by the existing sync tests in
`OutputFilterTests.cs`).

Tests
-----
- `Futures/Futures.AgentResponseOutputFilteringAndTaggingTests.cs` (new): the F1–F13
  matrix from the plan, covering every combination of `(flag on/off) × (designation)
  × (payload shape)`. Uses a `FuturesScope` IDisposable + a `FuturesSerial` xUnit
  collection (DisableParallelization = true) to keep the process-global flag from
  leaking across parallel tests.
- `OutputFilterTests.cs`: four new `Test_OutputFilter_…` cases for the `TryGetTags`
  surface (empty-tag-set for terminal designation, `{Intermediate}` for intermediate
  designation, union for accumulated designation, `false` for unregistered).

582/582 unit tests pass on net10.0 (565 baseline + 17 new).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat: tag-aware defaults and designation API on orchestration builders

Aligns the .NET orchestration builders with Python's output / intermediate-output
distinction. Each builder either applies a Python-aligned default designation set or
replays the user's explicit `WithOutputFrom` / `WithIntermediateOutputFrom` calls,
never both.

Static `AgentWorkflowBuilder.BuildSequential` / `BuildConcurrent` apply defaults
unconditionally (no user-facing fluent surface to take control through):

- Sequential: terminal `end` + every agent designated intermediate.
- Concurrent: terminal `end` + every agent and per-agent accumulator designated
  intermediate.

The three fluent instance builders memoize agent-typed designation calls in a
`Dictionary<AIAgent, HashSet<OutputTag>>` (empty set = terminal-only, non-empty =
intermediate tag(s)) so repeated calls dedupe naturally. They replay the entries
at `Build()` time, suppressing defaults when any call has been made:

- `HandoffWorkflowBuilder` / `HandoffWorkflowBuilderCore<TBuilder>` (also picked up
  by the obsolete `HandoffsWorkflowBuilder` via inheritance).
  Default: terminal `HandoffEnd` + every handoff agent intermediate.
  (Bug fix: legacy code relied on `WithOutputFrom(end)` to bind `HandoffEnd`. The
  new explicit-designation path bypasses that, so `Build()` now calls
  `BindExecutor(end)` unconditionally to keep validation happy.)
- `GroupChatWorkflowBuilder` — default: terminal host + every participant intermediate.
- `MagenticWorkflowBuilder` — default: terminal orchestrator + every team member
  intermediate.

Designating a non-participant agent throws `InvalidOperationException`.

The bare `WorkflowBuilder` default is unchanged — only the orchestration-style
builders gain implicit defaults, matching the plan's non-goal.

Tests
-----
- `AgentWorkflowBuilder.SequentialTests` / `.ConcurrentTests`: one default-spec
  assertion each.
- `GroupChatWorkflowBuilderTests`: defaults-match-spec, explicit-replaces-defaults,
  non-participant throws.
- `HandoffWorkflowBuilderTests` (new file): same three.
- `MagenticWorkflowBuilderTests` (new file): same three.

593/593 unit tests pass on net10.0 (582 baseline + 11 new).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat: WorkflowHostAgent forwards AgentResponseEvent unconditionally under Futures-on

Aligns the .NET Workflow-as-Agent surface with Python `as_agent`. Under
`Futures.EnableAgentResponseOutputTaggingAndFiltering = true`,
`WorkflowSession.InvokeStageAsync` now forwards `AgentResponseEvent`
unconditionally — joining `AgentResponseUpdateEvent` in ignoring the host's
`includeWorkflowOutputsInResponse` switch. That switch keeps governing the
generic `WorkflowOutputEvent` path for non-AIAgent payloads, where it is
further short-circuited by an `IsIntermediate()` check (tagged intermediate
outputs always surface).

Under Futures-off the legacy asymmetry is preserved: `AgentResponseUpdateEvent`
always forwarded, `AgentResponseEvent` gated by `includeWorkflowOutputsInResponse`.

Back-compat: with `Futures.EnableAgentResponseOutputTaggingAndFiltering` left at
its default `false`, observable behavior is identical to before.

`Futures` documentation gains a remark explaining the `Workflow.AsAIAgent()`
interaction in both flag states.

Runner fix
----------
`InProcessRunnerContext.YieldOutputAsync` now skips `Executor.CanOutput` for
AgentResponse-shaped payloads under both Futures branches. `AIAgentHostExecutor`
doesn't declare AgentResponse(Update) in its `Yields` set, so the historical
legacy bypass had silently skipped the check; Phase 3's Futures-on path was
running it and would reject AIAgent payloads. AIAgent-shaped payloads are now
always a valid output shape, matching the legacy bypass semantics.

Phase 4 follow-on
-----------------
Switched the three orchestration-builder designation-replay loops to iterate
`Dictionary.Keys` with a value lookup instead of constructing/destructuring
`KeyValuePair<,>`. Cleaner shape and avoids the netstandard2.0 / net472
`KeyValuePair<,>.Deconstruct` unavailability that surfaced when this branch
multi-TFM-built.

Tests
-----
`WorkflowHostSmokeTests.IntermediateForwarding` (new nested class, 6 tests):
- intermediate AgentResponse forwarded past the include-outputs gate (Futures on)
- terminal AgentResponse forwarded unconditionally (Futures on)
- terminal AgentResponse gated by include flag (Futures off, legacy)
- undesignated AIAgent executor emits no AgentResponseEvent under Futures-on
- legacy bypass still emits AgentResponseEvent under Futures-off
- intermediate tag is observable via `update.RawRepresentation`

The class joins the `FuturesSerial` xUnit collection so the process-global flag
is serialized against other Futures-toggling tests.

599/599 unit tests pass on net10.0 (593 baseline + 6 new).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat: SequentialWorkflowBuilder and ConcurrentWorkflowBuilder, OrchestrationBuilderBase

Promotes the Sequential and Concurrent orchestration shapes to first-class fluent
builder classes, matching Handoff / GroupChat / Magentic. Users can call
`WithOutputFrom(agents)` / `WithIntermediateOutputFrom(agents)` to control which
agents are designated output / intermediate sources; when no designation call is
made, the Python-aligned defaults apply (terminal aggregator output + every agent
intermediate; Concurrent also tags per-agent accumulators).

`AgentWorkflowBuilder.BuildSequential(...)` and `BuildConcurrent(...)` are kept
and now delegate to the new builders; observable behavior unchanged. Five static
factories now mirror each other:

- `AgentWorkflowBuilder.CreateSequentialBuilderWith(params IEnumerable<AIAgent>)`
- `AgentWorkflowBuilder.CreateConcurrentBuilderWith(params IEnumerable<AIAgent>)`
- `AgentWorkflowBuilder.CreateHandoffBuilderWith(AIAgent)`        (already existed)
- `AgentWorkflowBuilder.CreateGroupChatBuilderWith(Func<...>)`    (already existed)
- `AgentWorkflowBuilder.CreateMagenticBuilderWith(AIAgent)`       (new)

OrchestrationBuilderBase
------------------------
New abstract `OrchestrationBuilderBase<TBuilder>` unifies the shared fluent
surface across all five orchestration builders: `WithName`, `WithDescription`,
`WithOutputFrom`, `WithIntermediateOutputFrom`, and the
`ApplyOutputDesignations(builder, agentMap, kind, applyDefaults)` helper that
either replays the user's designations or invokes the orchestration-specific
defaults.

Removes ~150 LOC of duplicated designation-management code from the four
non-Handoff builders, plus the equivalent from `HandoffWorkflowBuilderCore`.

Tests
-----
- New `SequentialWorkflowBuilderTests.cs` / `ConcurrentWorkflowBuilderTests.cs`
  (replace the old `AgentWorkflowBuilder.{Sequential,Concurrent}Tests.cs`
  nested-class files). Method names normalized to
  `Test_<BuilderType>_<Scenario>[Async]`.
- Shared helpers (`DoubleEchoAgent`, `DoubleEchoAgentWithBarrier`,
  `WorkflowRunResult`, `RunWorkflow*`) moved from the old
  `AgentWorkflowBuilderTests` partial class into a new
  `OrchestrationTestHelpers` static class in `OrchestrationTestHelpers.cs`.
  Downstream test files (Group Chat, Handoff, Sequential, Concurrent) updated
  to qualify with `OrchestrationTestHelpers.*`.
- A new `AgentWorkflowBuilderTests.cs` covers the static surface directly:
  `BuildSequential` / `BuildConcurrent` invariants and aggregator wiring, plus
  null-rejection + round-trip checks for every `Create*BuilderWith` factory.
- New AsAgent intermediate-suppression tests on a nested `AsAgentForwarding`
  class for each of Sequential and Concurrent: build with only the terminal
  agent designated via `WithOutputFrom`, run via `AsAIAgent(...)`, assert via
  `AgentResponseUpdate.AuthorName` that intermediate agents do not surface.
  Both join the `FuturesSerial` collection.
- New `Test_<Builder>_WithDescriptionPropagatesToWorkflow` smoke tests on
  Sequential and Concurrent (newly available via the base class).

625/625 unit tests pass on net10.0.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chore: dotnet format

* fixup: encoding

* fixup: charset

* fixup: Updates for PR feedback

* fixup: format

* fixup: merge issue

* Fix intermediate filtering on .AsAgent()

* fix filter logic

* fix: Revert logic change and add comments

---------

Co-authored-by: Jacob Alber <jalber@lokitoth.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

8ed2159c4b · 2026-05-28 21:26:31 +00:00

History

..

JsonCheckpointSerializationTests.cs

.NET: Workflow Outputs Overhaul: Support Tagging, Filtering Agent Outputs (#6045)

2026-05-28 21:26:31 +00:00