Compare commits

...
Author SHA1 Message Date
Eduard van ValkenburgandCopilot 9a653af73e docs: replace stale observability migration guide with current PR's only relevant migration
The old guide documented the move away from setup_observability(otlp_endpoint=...)
which was an earlier-release API change unrelated to this PR and stale enough that
it's more confusing than helpful at this point. Replace it with a short note on the
single migration this PR introduces: callers of
enable_instrumentation(enable_sensitive_data=True) should switch to
enable_sensitive_telemetry(). Cross-link to the Disabling instrumentation section
for the rare 'force on without enabling sensitive data' use case where
enable_instrumentation() still applies.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-19 11:38:17 +02:00
Eduard van ValkenburgandCopilot d1bc78108d docs: correct observability Dependencies section
- opentelemetry-sdk is no longer a hard dependency; it is lazily imported by
  create_resource(), create_metric_views(), and configure_otel_providers()
  with a clear ImportError when missing. Day-to-day instrumentation works
  with opentelemetry-api alone provided some other component configures the
  global OpenTelemetry providers (Azure Monitor, an APM agent, application
  bootstrap, etc.).
- opentelemetry-semantic-conventions-ai is no longer used anywhere in the
  source; remove it from the listed dependencies.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-19 11:34:37 +02:00
Eduard van ValkenburgandCopilot c2f9544763 docs: soften disable_instrumentation() overclaim about telemetry guarantees
Replace 'no telemetry will be emitted no matter what' (which is too strong,
since callers can still pass force=True or mutate private attributes) with
language framing the disable as a user-intent contract that library and
framework code is expected to honor: the framework actively short-circuits
the public enable paths, force=True and private-attribute writes are
acknowledged as out-of-contract escape hatches that integrations should
not use on the user's behalf.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-19 11:28:15 +02:00
Eduard van ValkenburgandCopilot 6312001ecb docs: document disable_instrumentation() and force=True paths
Add a "Disabling instrumentation" section to the observability sample README
that walks through:

- The distinction between the ENABLE_INSTRUMENTATION env var (initial,
  non-sticky) and disable_instrumentation() (process-wide, sticky).
- Why the sticky semantics matter: framework integrations like
  FoundryChatClient.configure_azure_monitor() can call
  enable_instrumentation() as part of their setup, and the user's opt-out
  needs to win.
- All five surfaces guarded by the sticky disable (property reads, public
  enable functions, configure_otel_providers, direct attribute writes,
  is_user_disabled-aware integrations).
- The force=True escape hatch on both enable_instrumentation() and
  enable_sensitive_telemetry().
- How third-party integrations should consult OBSERVABILITY_SETTINGS.is_user_disabled.
- The limits of the disable (does not tear down existing providers /
  in-flight spans / third-party instrumentation, does not persist across
  processes).

Cross-links the new section from the ENABLE_INSTRUMENTATION row in the env
vars table.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-19 11:24:04 +02:00
Eduard van ValkenburgandCopilot 6a23dcd555 Add disable_instrumentation() with sticky user-intent semantics
Add a public disable_instrumentation() entry point so users can explicitly opt
out of Agent Framework telemetry, with a sticky-disable flag that makes the
user's intent "leading" — no framework code path (foundry's
configure_azure_monitor, configure_otel_providers, enable_instrumentation,
enable_sensitive_telemetry, or direct OBSERVABILITY_SETTINGS.enable_*
writes) can re-enable instrumentation until the user explicitly clears the
disable with enable_instrumentation(force=True) /
enable_sensitive_telemetry(force=True).

Also addresses the two remaining unresolved review threads on the PR:
1. test_observability_settings_defaults_instrumentation_true pins the new
   "ENABLE_INSTRUMENTATION defaults to True when env unset" behavior.
2. test_enable_instrumentation_reads_env_sensitive_data restores coverage
   for the post-import load_dotenv() fallback path.

Implementation:
- ObservabilitySettings.enable_instrumentation / enable_sensitive_data become
  properties backed by _enable_*. While _user_disabled is True, the getters
  return False and the setters drop True writes (defense in depth so third-
  party writes can't subvert the disable).
- Public is_user_disabled read-only property lets integrations (e.g. foundry's
  configure_azure_monitor) cheaply check the disable state without poking at
  privates.
- enable_instrumentation() and enable_sensitive_telemetry() short-circuit with
  an info log when disabled; gain a force=True kwarg that clears the disable.
- configure_otel_providers() still creates providers / exporters / views so a
  later force-enable can use them, but logs an info message when called while
  disabled.
- Foundry's FoundryChatClient.configure_azure_monitor and
  FoundryAgent.configure_azure_monitor early-return when the user has
  disabled, so Azure Monitor's global providers aren't installed unnecessarily.

Tests: 11 new tests covering default-on, env re-read at call time, sticky
behavior against each re-enable surface (enable_instrumentation,
enable_sensitive_telemetry, configure_otel_providers, direct attribute
writes), force=True override, re-arming the disable, and the __all__ export.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-19 09:30:41 +02:00
Tao Chen 1d1e58e12a Fix mypy 2026-05-15 10:41:18 -07:00
Tao Chen 5e5c6976f1 Formatting 2026-05-14 17:13:33 -07:00
Tao Chen 94ce49dcf5 Merge branch 'main' into local-branch-python-enable-observability-by-default 2026-05-14 17:06:28 -07:00
Tao Chen 9007b3262a Add warning 2026-05-14 17:05:59 -07:00
Tao Chen f5f0d828ab Revert uv.lock 2026-05-14 17:01:58 -07:00
Tao Chen 13b8e68503 Address Copilot comments 2026-05-14 16:56:29 -07:00
Tao Chen 07ea764469 Optimization when span is not recording 2026-05-14 16:27:58 -07:00
68357b0250 Python: Fix A2A v1.0 non-streaming response and sample runtime issues (#5849)
- Fix non-streaming empty response by accumulating intermediate WORKING
  status updates and flushing them when an empty terminal event arrives
- Fix sample agent_executor.py to enqueue Task before status events
  (required by v1.0 ActiveTask validation)
- Fix create_jsonrpc_routes() calls to include required rpc_url param
- Fix TYPE_CHECKING imports in sample agent_definitions.py
- Add tests for non-streaming content accumulation behavior

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-14 22:28:02 +00:00
Tao Chen 4a83d92c96 Update samples 2026-05-14 15:02:15 -07:00
Yufeng HeandGitHub 410268b624 Python: forward MCP tool call metadata (#5815)
* Python: forward MCP tool call metadata

* fix: preserve MCP tool meta after prompt reload
2026-05-14 21:50:39 +00:00
CopilotGitHublokitothcopilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>Jacob Alber
67f3db6280 Python: Reject path-traversal context ids in Foundry Hosting Checkpoint Storage (#5851)
* Reject path-traversal context ids in foundry workflow checkpoint storage

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/fca3aae6-50eb-4726-8baf-2718217d4e79

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Address PR review feedback: clarify URL-decode comment, isolate test root, add e2e workflow rejection tests

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/832f45a6-c01e-4da9-bf85-1ba7b5f302e6

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Clarify MSRC repro padding length in regression test

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/832f45a6-c01e-4da9-bf85-1ba7b5f302e6

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* add E2E http test for checkpoint context id rejection

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/730258ef-2781-4a7d-b7cf-b5c40c11defc

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>
Co-authored-by: Jacob Alber <jaalber@microsoft.com>
2026-05-14 21:38:37 +00:00
Tao Chen 0219e17be2 Enable instrumentation by default 2026-05-14 13:42:00 -07:00
CopilotGitHublokitothcopilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>Jacob Alber
2ef20cd0aa .NET: Add Magentic E2E workflow coverage (#5833)
* Add E2E test plan for Magentic orchestrator

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/96d76349-1ffd-482b-a3ee-ed208778b1bb

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Add MagenticOrchestrationTests.cs scaffold for Magentic E2E tests

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/44a4fd8a-3828-40e5-9435-90381aeffdb8

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Fix MagenticOrchestrator output declaration and add first E2E test

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/322c9e2d-59bc-42ad-9a1e-f6fd4c866b26

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Add plan review test and event emission tests

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/322c9e2d-59bc-42ad-9a1e-f6fd4c866b26

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Add next speaker validation test

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/322c9e2d-59bc-42ad-9a1e-f6fd4c866b26

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Add Magentic E2E implementation review

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/b2c60ce7-4d05-4a0d-b05d-d4284f5b7bb3

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Add PlanSignoff_Disabled_Proceeds_Immediately E2E test

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/6e8bca46-448d-4f21-a7e9-240179571970

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Add NextSpeaker_Empty_Falls_Back_To_First E2E test

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/6e8bca46-448d-4f21-a7e9-240179571970

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Add Task_Completes_After_Multiple_Rounds E2E test

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/6e8bca46-448d-4f21-a7e9-240179571970

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Add PlanReview_Revised_Triggers_Replan E2E test

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/6e8bca46-448d-4f21-a7e9-240179571970

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Add MaxRoundLimit_Terminates_Workflow E2E test

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/6e8bca46-448d-4f21-a7e9-240179571970

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Add MaxStallCount_Triggers_Reset E2E test

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/6e8bca46-448d-4f21-a7e9-240179571970

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Update MagenticE2E_ImplementationReview.md with full coverage status

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/6e8bca46-448d-4f21-a7e9-240179571970

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Rewrite Magentic E2E implementation review

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/1f878ef4-61b0-410a-a8bc-ebf618b3e5de

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Add MaxResetLimit_Terminates_Workflow E2E test

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/aba19507-7c7e-40dd-850d-d1fabb5dfa65

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Add PlanReview_On_Stall_Replan E2E test

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/aba19507-7c7e-40dd-850d-d1fabb5dfa65

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Add Instruction_Message_Sent_When_Present E2E test

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/aba19507-7c7e-40dd-850d-d1fabb5dfa65

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Update ImplementationReview.md to reflect 14 tests

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/aba19507-7c7e-40dd-850d-d1fabb5dfa65

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Rewrite Magentic E2E implementation review

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/6fe88a80-2e05-40d5-9539-ca7c59b9022b

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Add ProgressLedger_Retry_On_Parse_Failure E2E test

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/125f6628-6b3b-4c51-9a51-ae84baece6bb

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Add ProgressLedger_Max_Retries_Triggers_Reset E2E test

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/125f6628-6b3b-4c51-9a51-ae84baece6bb

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Add Stall_NoProgress_Increments_StallCount E2E test

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/125f6628-6b3b-4c51-9a51-ae84baece6bb

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Add PlanReview_Multiple_Revisions E2E test

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/125f6628-6b3b-4c51-9a51-ae84baece6bb

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Update ImplementationReview.md to reflect 18 tests and new coverage

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/125f6628-6b3b-4c51-9a51-ae84baece6bb

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Rewrite Magentic E2E implementation review

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/21f3b1ae-183e-4fea-99ad-14efc19f084d

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Preserve IsStalled on stall-triggered plan review requests

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/1b9e74e8-69e1-43f2-8467-c5ba963c2622

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Rename isStalled parameter to replanAfterStall for clarity

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/1b9e74e8-69e1-43f2-8467-c5ba963c2622

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Add Task_Delegates_To_Correct_Agent E2E test with multi-participant routing assertion

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/9b34e409-61b8-4650-ae55-34efad034ed0

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Add Progress_Made_Decrements_StallCount E2E test verifying stall count decrement avoids reset

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/9b34e409-61b8-4650-ae55-34efad034ed0

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Add Consecutive_Stalls_Trigger_Reset E2E test for multi-stall threshold reset

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/9b34e409-61b8-4650-ae55-34efad034ed0

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Magentic E2E: preserve IsStalled on stall-triggered plan reviews, add routing/stall tests

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/9b34e409-61b8-4650-ae55-34efad034ed0

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Fix replan-on-every-turn: skip plan on agent return; align StallCount to > (match Python)

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/43e46b0d-4263-4353-856a-c3730abb1734

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Update implementation review doc for replan-fix and stall threshold alignment

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/43e46b0d-4263-4353-856a-c3730abb1734

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Rewrite Magentic E2E implementation review

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/3d15763b-3a68-488e-9412-3fa280e083c0

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Update stall docs to use > semantics, skip checkpoint-state tests, simplify NextSpeaker fallback test

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/cc9ea5a8-84d8-4b6d-bb60-ac9619824d81

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Rewrite Magentic implementation review

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/ed87670a-bf4d-4ba5-a2f3-395a2eead9de

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Add empty-team validation to MagenticWorkflowBuilder.Build() and E2E test

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/e490fdf7-f107-4fde-ba1f-efdfd9a729c6

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Add IsTerminated guard to TakeTurnAsync and post-termination rejection test

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/e490fdf7-f107-4fde-ba1f-efdfd9a729c6

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Rewrite ImplementationReview.md with final 23-test status

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/e490fdf7-f107-4fde-ba1f-efdfd9a729c6

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Add PR description markdown

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/df9b4579-10c3-4bfb-927e-da3a0e70009e

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Remove temporary markdown files

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/b3e67553-a3a3-4282-98f2-afd8ad7a6b5d

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Fix IDE1006: add Async suffix to async test methods in MagenticOrchestrationTests

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/629fcc07-865e-4832-9e59-ea13df561c5a

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Update error messages per review comments in MagenticOrchestrator and MagenticWorkflowBuilder

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/053e5ded-81e3-4e56-acf1-2a8a939a04b0

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Escape JSON string values in CreateProgressLedgerResponse test helper

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/ec610c61-0a14-44e2-82fd-1cf35e85d6cc

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>
Co-authored-by: Jacob Alber <jaalber@microsoft.com>
2026-05-14 19:53:07 +00:00
CopilotGitHubcopilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>lokitothJacob Alber
27671974c2 .NET: Re-enable ObservabilityTests and WorkflowRunActivityStopTests (#5837)
Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/220699b9-7f9e-4d5d-87d0-fb621d169d84

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>
Co-authored-by: Jacob Alber <jaalber@microsoft.com>
2026-05-14 19:17:26 +00:00
7432105ebe Python: Support list[str] arguments for file-based skill scripts (#5850)
Port of .NET PR #5475. Broadens the args type from dict[str, Any] | None
to dict[str, Any] | list[str] | None across the skill script API surface,
enabling CLI-style argv forwarding to subprocess scripts.

Changes:
- SkillScript.run(), InlineSkillScript.run(), FileSkillScript.run(): widen
  args type; InlineSkillScript rejects list with TypeError
- FileSkillScript.parameters_schema: returns array-of-strings schema
- FileSkill.content: appends <scripts> block with parameters_schema
- SkillScriptRunner protocol: widen args type
- SkillsProvider._run_skill_script: widen args type
- run_skill_script tool schema: accept object, array, or null
- subprocess_script_runner sample: accept list[str], reject dict
- class_based_skill sample: fix missing SkillFrontmatter wrapper
- Standardize 'folder' to 'directory' in docstrings (#5712)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-14 17:58:10 +00:00
3256550c55 .NET: fix: allow naming handoff workflows (#5799)
* fix: allow naming handoff workflows

* Only set name/description if not NullOrWhitespace

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Jacob Alber <jalber@fernir.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Jacob Alber <jaalber@microsoft.com>
2026-05-14 17:10:27 +00:00
CopilotGitHublokitothcopilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>Jacob Alber
190ca75b6a .NET: Add Workflow Builder Specialized Edge tests (#5826)
* Add workflow builder edge tests

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/3c3d5324-cdcd-4a38-8c67-94e4e78e29c5

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Strengthen workflow edge helper tests

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/af831ee2-0a99-4427-9ffd-a3b5022c1b3b

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Normalize edge helper bad input validation

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/af831ee2-0a99-4427-9ffd-a3b5022c1b3b

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Clarify edge helper target validation

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/af831ee2-0a99-4427-9ffd-a3b5022c1b3b

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Use explicit target parameter names

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/af831ee2-0a99-4427-9ffd-a3b5022c1b3b

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Document workflow edge test helpers

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/af831ee2-0a99-4427-9ffd-a3b5022c1b3b

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Clarify null element validation messages

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/af831ee2-0a99-4427-9ffd-a3b5022c1b3b

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Add repeated chain executor coverage

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/af831ee2-0a99-4427-9ffd-a3b5022c1b3b

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Preserve Throw helper validation style

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/af831ee2-0a99-4427-9ffd-a3b5022c1b3b

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Cover empty switch case targets

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/af831ee2-0a99-4427-9ffd-a3b5022c1b3b

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Relax builder null assertion parameter checks

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/af831ee2-0a99-4427-9ffd-a3b5022c1b3b

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Inline ValidateTargets into call sites

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/cb9a6a6a-02c7-41a8-a4b4-da16ad62ef86

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Refactor ForwardExcept with TFM-specialized TryGetNonEnumeratedCount

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/b081f61f-93ce-45dc-abbd-82c465395470

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Use TFM-specialized count check: TryGetNonEnumeratedCount for NET6+, ICollection pattern for NETFX

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/8ec28a43-e7b7-456e-8d8e-921511b4accc

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Apply TFM-specialized count check to ForwardMessage as well

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/9238ea32-a3e8-4b83-9683-484ad400071f

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Address review feedback: simplify Throw.IfNull in SwitchBuilder per westey-m suggestion

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/299950fd-4457-47f3-a373-f65d601b7ea5

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Use indexed parameter name in SwitchBuilder Throw.IfNull: executors[index]

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/c5655707-5b0b-44f3-98a9-5f3961e32cfe

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Revert #if NET6_0_OR_GREATER back to #if NET; inline executorIndex++

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/c5655707-5b0b-44f3-98a9-5f3961e32cfe

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Add comment explaining unusual Throw.IfNull use for null elements inside collection

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/c5655707-5b0b-44f3-98a9-5f3961e32cfe

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>
Co-authored-by: Jacob Alber <jaalber@microsoft.com>
2026-05-14 16:23:41 +00:00
CopilotGitHublokitothcopilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>Jacob Alber
8058fb1c5b .NET: Fix flaky InputWaiter_WaitForInputAsync_BlocksUntilSignaledAsync (#5835)
* test: remove finite timeout in BlocksUntilSignaledAsync to fix race

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/962b7404-4266-4a16-906c-ba3e607c2764

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* address review: clarify comment, add timeout test, cross-reference test names

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/e406a5f2-ad31-4d37-b090-69e10713f885

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>
Co-authored-by: Jacob Alber <jaalber@microsoft.com>
2026-05-14 15:36:42 +00:00
Peter IbekweandGitHub 189e64bfdd .NET: Add sample for invoking Foundry Toolbox tools from declarative workflows (#5829)
* Add sample for invoking Foundry Toolbox tools from declarative workflows

* Addressed initial PR comments.
2026-05-14 15:30:48 +00:00
westeyandGitHub 3047ad3066 .NET: Harness console refactoring (#5811)
* Restructure harness console so that reactive app is the entry point

* Further refactoring to split tool formatters, improve UX, make console configurable and fix bugs

* Address PR comments.

* UX tweak

* Fix streaming text bug

* Address PR comments.
2026-05-14 15:22:11 +00:00
Evan MattsonandGitHub 0e12640c70 Improvements for DevUI (#5840) 2026-05-14 15:05:27 +00:00
Evan MattsonandGitHub ae666a4887 Python: Bump agent-framework-ag-ui to release candidate stage (#5844)
* Bump agent-framework-ag-ui to release candidate stage

* Mark agent-framework-ag-ui as rc in PACKAGE_STATUS
2026-05-14 14:56:34 +00:00
CopilotGitHublokitothcopilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>Jacob Alber
eb40535436 .NET: Add Executor RouteBuilder Unit Tests (#5824)
* Add RouteBuilder unit tests

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/012f3b3b-acb9-4869-9084-b767cbe1885b

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Address RouteBuilder test review feedback

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/012f3b3b-acb9-4869-9084-b767cbe1885b

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Fix RouteBuilder test nullability warning

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/012f3b3b-acb9-4869-9084-b767cbe1885b

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Refine RouteBuilder test helpers

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/012f3b3b-acb9-4869-9084-b767cbe1885b

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Refactor overload int constants to HandlerOverload enum

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/19397f58-a88a-41cf-bd85-588f520e0d0f

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Fix ValueTask compatibility with .NET Framework 4.7.2

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/a8437809-0898-43a6-a950-09eb3417f58a

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

* Fix IDE0001 format errors - simplify generic type names

Agent-Logs-Url: https://github.com/microsoft/agent-framework/sessions/8573214e-ec42-4969-ba94-76bdc8ad3e59

Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lokitoth <6936551+lokitoth@users.noreply.github.com>
Co-authored-by: Jacob Alber <jaalber@microsoft.com>
2026-05-14 13:36:38 +00:00
107 changed files with 7199 additions and 2316 deletions
+1
View File
@@ -242,6 +242,7 @@
<Project Path="samples/03-workflows/Declarative/HostedWorkflow/HostedWorkflow.csproj" />
<Project Path="samples/03-workflows/Declarative/InputArguments/InputArguments.csproj" />
<Project Path="samples/03-workflows/Declarative/InvokeFunctionTool/InvokeFunctionTool.csproj" />
<Project Path="samples/03-workflows/Declarative/InvokeFoundryToolboxMcp/InvokeFoundryToolboxMcp.csproj" />
<Project Path="samples/03-workflows/Declarative/InvokeHttpRequest/InvokeHttpRequest.csproj" />
<Project Path="samples/03-workflows/Declarative/InvokeMcpTool/InvokeMcpTool.csproj" />
<Project Path="samples/03-workflows/Declarative/Marketing/Marketing.csproj" />
@@ -478,6 +478,17 @@ internal static class WorkflowSamples
ExpectedOutputDescription = ["The output should show a workflow invoking a function tool (e.g. a menu plugin) to answer a question about the soup of the day."],
},
new SampleDefinition
{
Name = "Workflow_Declarative_InvokeFoundryToolboxMcp",
ProjectPath = "samples/03-workflows/Declarative/InvokeFoundryToolboxMcp",
RequiredEnvironmentVariables = ["AZURE_AI_PROJECT_ENDPOINT"],
OptionalEnvironmentVariables = ["AZURE_AI_MODEL_DEPLOYMENT_NAME", "FOUNDRY_TOOLBOX_NAME", "FOUNDRY_AGENT_TOOLSET_API_VERSION"],
Inputs = ["How do I use Azure OpenAI with my data?"],
InputDelayMs = 3000,
ExpectedOutputDescription = ["The output should show a workflow using Foundry Toolbox MCP tools to search Microsoft Learn documentation and web search to provide a summary of results."],
},
new SampleDefinition
{
Name = "Workflow_Declarative_InvokeMcpTool",
@@ -9,42 +9,30 @@ namespace Harness.ConsoleReactiveComponents;
/// </summary>
public record TextPanelProps : ConsoleReactiveProps
{
/// <summary>Gets the items to render in the panel.</summary>
public IReadOnlyList<object> Items { get; init; } = [];
/// <summary>Gets the items to render in the panel. Each item is a pre-rendered
/// console string (may include ANSI escape sequences and newlines).</summary>
public IReadOnlyList<string> Items { get; init; } = [];
}
/// <summary>
/// A component that renders a list of items vertically using a custom render delegate.
/// A component that renders a list of pre-rendered string items vertically.
/// Designed for rendering dynamic items in a non-scroll region that may be
/// re-rendered on each update. If the component's <see cref="ConsoleReactiveComponent.Height"/>
/// exceeds the number of output lines, leftover lines are erased.
/// </summary>
public class TextPanel : ConsoleReactiveComponent<TextPanelProps, ConsoleReactiveState>
{
private readonly Func<object, string> _renderItem;
/// <summary>
/// Initializes a new instance of the <see cref="TextPanel"/> class.
/// </summary>
/// <param name="renderItem">A delegate that renders an item and returns the text to display (may contain newlines).</param>
public TextPanel(Func<object, string> renderItem)
{
this._renderItem = renderItem;
}
/// <summary>
/// Calculates the height (in lines) needed to render all items.
/// </summary>
/// <param name="items">The items to measure.</param>
/// <param name="renderItem">The render delegate to use for measuring.</param>
/// <returns>The total number of lines all items will occupy.</returns>
public static int CalculateHeight(IReadOnlyList<object> items, Func<object, string> renderItem)
public static int CalculateHeight(IReadOnlyList<string> items)
{
int total = 0;
for (int i = 0; i < items.Count; i++)
{
string text = renderItem(items[i]);
total += CountLines(text);
total += CountLines(items[i]);
}
return total;
@@ -57,7 +45,7 @@ public class TextPanel : ConsoleReactiveComponent<TextPanelProps, ConsoleReactiv
for (int i = 0; i < props.Items.Count; i++)
{
string text = this._renderItem(props.Items[i]);
string text = props.Items[i];
string[] lines = text.Split('\n');
int lineCount = CountLines(text);
@@ -9,8 +9,9 @@ namespace Harness.ConsoleReactiveComponents;
/// </summary>
public record TextScrollPanelProps : ConsoleReactiveProps
{
/// <summary>Gets the items to render in the scroll panel.</summary>
public IReadOnlyList<object> Items { get; init; } = [];
/// <summary>Gets the items to render in the scroll panel. Each item is a pre-rendered
/// console string (may include ANSI escape sequences and newlines).</summary>
public IReadOnlyList<string> Items { get; init; } = [];
}
/// <summary>
@@ -20,21 +21,17 @@ public record TextScrollPanelProps : ConsoleReactiveProps
public record TextScrollPanelState(int RenderedCount = 0) : ConsoleReactiveState;
/// <summary>
/// A component that renders items within a scroll area using a custom render delegate.
/// A component that renders pre-rendered string items within a scroll area.
/// All items are considered finalized — only new items since the last render are output.
/// Use <see cref="Reset"/> to force a full re-render.
/// </summary>
public class TextScrollPanel : ConsoleReactiveComponent<TextScrollPanelProps, TextScrollPanelState>
{
private readonly Func<object, string> _renderItem;
/// <summary>
/// Initializes a new instance of the <see cref="TextScrollPanel"/> class.
/// </summary>
/// <param name="renderItem">A delegate that renders a single item and returns the text to display (may contain newlines).</param>
public TextScrollPanel(Func<object, string> renderItem)
public TextScrollPanel()
{
this._renderItem = renderItem;
this.State = new TextScrollPanelState();
}
@@ -60,8 +57,7 @@ public class TextScrollPanel : ConsoleReactiveComponent<TextScrollPanelProps, Te
// Output only new items since last rendered
for (int i = state.RenderedCount; i < props.Items.Count; i++)
{
string text = this._renderItem(props.Items[i]);
Console.Write(text);
Console.Write(props.Items[i]);
}
// Update state to track what we've rendered
@@ -1,315 +0,0 @@
// Copyright (c) Microsoft. All rights reserved.
using Harness.ConsoleReactiveComponents;
using Harness.ConsoleReactiveFramework;
namespace Harness.ConsoleSandbox;
/// <summary>
/// Determines which component is shown in the bottom panel.
/// </summary>
public enum BottomPanelMode
{
/// <summary>Show the list selection component.</summary>
ListSelection,
/// <summary>Show the text input component.</summary>
TextInput
}
public record AppComponentProps : ConsoleReactiveProps
{
public IReadOnlyList<string> Items { get; init; } = Array.Empty<string>();
public IReadOnlyList<object> ScrollItems { get; init; } = [];
/// <summary>Gets the bottom panel mode.</summary>
public BottomPanelMode Mode { get; init; } = BottomPanelMode.ListSelection;
/// <summary>Gets the prompt string for text input mode.</summary>
public string Prompt { get; init; } = "> ";
/// <summary>Gets the placeholder text shown when the input is empty.</summary>
public string Placeholder { get; init; } = "";
/// <summary>Gets the highlight color for the active list item. Defaults to <see cref="ConsoleColor.Cyan"/>.</summary>
public ConsoleColor ListHighlightColor { get; init; } = ConsoleColor.Cyan;
/// <summary>Gets the placeholder text for the custom text input option in the list. If <c>null</c>, no custom option is shown.</summary>
public string? ListCustomTextPlaceholder { get; init; }
/// <summary>Gets the foreground color for the rule borders. If <c>null</c>, uses the default terminal color.</summary>
public ConsoleColor? RuleColor { get; init; }
}
/// <summary>
/// Internal state for the <see cref="AppComponent"/>.
/// </summary>
public record AppComponentState : ConsoleReactiveState
{
/// <summary>Gets the selected index in list selection mode.</summary>
public int SelectedIndex { get; init; }
/// <summary>Gets the current input text being typed in text input mode.</summary>
public string InputText { get; init; } = "";
/// <summary>Gets the current text being typed into the list's custom text option.</summary>
public string ListInputText { get; init; } = "";
}
public class AppComponent : ConsoleReactiveComponent<AppComponentProps, AppComponentState>
{
private readonly TopBottomRule _rule = new();
private readonly ListSelection _listSelection = new();
private readonly TextInput _textInput = new();
private readonly TextScrollPanel _textScrollPanel;
private readonly TextPanel _textPanel;
private readonly Func<object, string> _renderItem;
private readonly Action<string> _onTextInputSubmit;
private readonly Action<string> _onListInputSubmit;
private bool _resizedSinceLastRender;
private int _lastScrollBottom;
/// <summary>
/// Initializes a new instance of the <see cref="AppComponent"/> class.
/// </summary>
/// <param name="renderScrollItem">A delegate that renders a single scroll panel item and returns the text to display.</param>
/// <param name="onTextInputSubmit">A callback invoked with the input text when the user presses Enter in text input mode.</param>
/// <param name="onListInputSubmit">A callback invoked with the selected or typed text when the user presses Enter in list selection mode.</param>
public AppComponent(Func<object, string> renderScrollItem, Action<string> onTextInputSubmit, Action<string> onListInputSubmit)
{
this._renderItem = renderScrollItem;
this._onTextInputSubmit = onTextInputSubmit;
this._onListInputSubmit = onListInputSubmit;
this._textScrollPanel = new TextScrollPanel(renderScrollItem);
this._textPanel = new TextPanel(renderScrollItem);
this.State = new AppComponentState();
KeyEventListener.Instance.KeyPressed += this.OnKeyPressed;
ConsoleResizeListener.Instance.ConsoleResized += this.OnConsoleResized;
}
private void OnKeyPressed(object? sender, KeyPressEventArgs e)
{
if (this.Props!.Mode == BottomPanelMode.TextInput)
{
this.HandleTextInputKey(e);
}
else
{
this.HandleListSelectionKey(e);
}
}
private void HandleTextInputKey(KeyPressEventArgs e)
{
if (e.KeyInfo.Key == ConsoleKey.Enter)
{
string text = this.State!.InputText;
this.SetState(this.State with { InputText = "" });
this._onTextInputSubmit(text);
}
else if (e.KeyInfo.Key == ConsoleKey.Backspace)
{
if (this.State!.InputText.Length > 0)
{
this.SetState(this.State with { InputText = this.State.InputText[..^1] });
}
}
else if (e.KeyInfo.KeyChar != '\0' && !char.IsControl(e.KeyInfo.KeyChar))
{
this.SetState(this.State! with { InputText = this.State.InputText + e.KeyInfo.KeyChar });
}
}
private void HandleListSelectionKey(KeyPressEventArgs e)
{
int maxIndex = this.Props!.Items.Count - 1;
if (this.Props.ListCustomTextPlaceholder != null)
{
maxIndex = this.Props.Items.Count; // extra option at the end
}
bool isOnCustomTextOption = this.Props.ListCustomTextPlaceholder != null
&& this.State!.SelectedIndex == this.Props.Items.Count;
if (e.KeyInfo.Key == ConsoleKey.UpArrow)
{
this.SetState(this.State! with { SelectedIndex = Math.Max(0, this.State.SelectedIndex - 1) });
}
else if (e.KeyInfo.Key == ConsoleKey.DownArrow)
{
this.SetState(this.State! with { SelectedIndex = Math.Min(maxIndex, this.State.SelectedIndex + 1) });
}
else if (e.KeyInfo.Key == ConsoleKey.Enter)
{
if (isOnCustomTextOption)
{
string text = this.State!.ListInputText;
this.SetState(this.State with { ListInputText = "" });
this._onListInputSubmit(text);
}
else
{
this._onListInputSubmit(this.Props.Items[this.State!.SelectedIndex]);
}
}
else if (isOnCustomTextOption)
{
// Typing only works when on the custom text option
if (e.KeyInfo.Key == ConsoleKey.Backspace)
{
if (this.State!.ListInputText.Length > 0)
{
this.SetState(this.State with { ListInputText = this.State.ListInputText[..^1] });
}
}
else if (e.KeyInfo.KeyChar != '\0' && !char.IsControl(e.KeyInfo.KeyChar))
{
this.SetState(this.State! with { ListInputText = this.State.ListInputText + e.KeyInfo.KeyChar });
}
}
}
private void OnConsoleResized(object? sender, ConsoleResizeEventArgs e)
{
this._resizedSinceLastRender = true;
this.Render();
}
public override void RenderCore(AppComponentProps props, AppComponentState state)
{
// Determine the text panel height for the last scroll item
object? lastItem = props.ScrollItems.Count > 0 ? props.ScrollItems[^1] : null;
IReadOnlyList<object> lastItems = lastItem != null ? [lastItem] : [];
int textPanelHeight = TextPanel.CalculateHeight(lastItems, this._renderItem);
if (textPanelHeight > 0)
{
textPanelHeight++; // Extra line for spacing between text panel and rule
}
// Build the bottom panel child based on mode
ConsoleReactiveComponent bottomChild;
int bottomChildHeight;
if (props.Mode == BottomPanelMode.TextInput)
{
var textInputProps = new TextInputProps
{
Prompt = props.Prompt,
Text = state.InputText,
Placeholder = props.Placeholder
};
bottomChildHeight = TextInput.CalculateHeight(textInputProps, Console.WindowWidth);
this._textInput.Width = Console.WindowWidth;
this._textInput.Height = bottomChildHeight;
this._textInput.Props = textInputProps;
bottomChild = this._textInput;
}
else
{
var listProps = new ListSelectionProps
{
Items = props.Items,
SelectedIndex = state.SelectedIndex,
HighlightColor = props.ListHighlightColor,
CustomTextPlaceholder = props.ListCustomTextPlaceholder,
CustomText = state.ListInputText
};
bottomChildHeight = ListSelection.CalculateHeight(listProps);
this._listSelection.Height = bottomChildHeight;
this._listSelection.Props = listProps;
bottomChild = this._listSelection;
}
var ruleProps = new TopBottomRuleProps
{
Width = Console.WindowWidth,
Color = props.RuleColor,
Children = [bottomChild]
};
int ruleHeight = TopBottomRule.CalculateHeight(ruleProps);
int scrollBottom = Console.WindowHeight - ruleHeight - textPanelHeight;
// If scroll region changed or a clear is needed, reset everything
if (this._resizedSinceLastRender || (this._lastScrollBottom != 0 && scrollBottom != this._lastScrollBottom))
{
Console.Write(AnsiEscapes.EraseEntireScreen);
Console.Write(AnsiEscapes.EraseScrollbackBuffer);
this._textScrollPanel.Reset();
this._resizedSinceLastRender = false;
}
this._lastScrollBottom = scrollBottom;
Console.Write(AnsiEscapes.SetScrollRegion(scrollBottom));
// Render text scroll panel in the scroll area (all items except the last)
IReadOnlyList<object> scrollItems = props.ScrollItems.Count > 1
? props.ScrollItems.Take(props.ScrollItems.Count - 1).ToList()
: [];
this._textScrollPanel.X = 1;
this._textScrollPanel.Y = 1;
this._textScrollPanel.Width = Console.WindowWidth;
this._textScrollPanel.Height = scrollBottom;
this._textScrollPanel.Props = new TextScrollPanelProps
{
Items = scrollItems
};
this._textScrollPanel.Render();
// Render the text panel for the last (dynamic) item just below the scroll region
this._textPanel.X = 1;
this._textPanel.Y = scrollBottom + 1;
this._textPanel.Width = Console.WindowWidth;
this._textPanel.Height = textPanelHeight;
this._textPanel.Props = new TextPanelProps
{
Items = lastItems,
};
this._textPanel.Render();
// Render the bottom rule + child below the text panel
this._rule.X = 1;
this._rule.Y = scrollBottom + textPanelHeight + 1;
this._rule.Props = ruleProps;
this._rule.Render();
// Position cursor for natural typing appearance
if (props.Mode == BottomPanelMode.TextInput)
{
int promptLength = props.Prompt.Length;
int textWidth = Console.WindowWidth - promptLength;
int textLength = state.InputText.Length;
// The TextInput starts at rule.Y + 1 (first row inside the rule)
int textInputY = this._rule.Y + 1;
if (textWidth <= 0 || textLength == 0)
{
// Cursor right after the prompt
Console.Write(AnsiEscapes.MoveCursor(textInputY, promptLength + 1));
}
else
{
// Calculate which row and column the cursor lands on
int cursorRow = textLength < textWidth ? 0 : 1 + ((textLength - textWidth) / textWidth);
int cursorCol = textLength < textWidth ? textLength : (textLength - textWidth) % textWidth;
Console.Write(AnsiEscapes.MoveCursor(textInputY + cursorRow, promptLength + cursorCol + 1));
}
}
else if (props.Mode == BottomPanelMode.ListSelection
&& props.ListCustomTextPlaceholder != null
&& state.SelectedIndex == props.Items.Count)
{
// Cursor after the typed text in the custom text option
// The custom text option is at rule.Y + 1 + Items.Count (0-based row inside rule)
int customOptionY = this._rule.Y + 1 + props.Items.Count;
// "> " prefix is 2 chars, then the typed text
int cursorCol = 2 + state.ListInputText.Length + 1;
Console.Write(AnsiEscapes.MoveCursor(customOptionY, cursorCol));
}
}
}
@@ -23,7 +23,7 @@ public abstract class CommandHandler
/// </summary>
/// <param name="input">The raw user input string.</param>
/// <param name="session">The current agent session.</param>
/// <param name="ux">The UX container for rendering output.</param>
/// <param name="ux">The UX state driver for rendering output.</param>
/// <returns><see langword="true"/> if this handler handled the input; <see langword="false"/> otherwise.</returns>
public abstract ValueTask<bool> TryHandleAsync(string input, AgentSession session, HarnessUXContainer ux);
public abstract ValueTask<bool> TryHandleAsync(string input, AgentSession session, IUXStateDriver ux);
}
@@ -0,0 +1,26 @@
// Copyright (c) Microsoft. All rights reserved.
using Microsoft.Agents.AI;
namespace Harness.Shared.Console.Commands;
/// <summary>
/// Handles the <c>/exit</c> command to shut down the console application.
/// </summary>
public sealed class ExitCommandHandler : CommandHandler
{
/// <inheritdoc/>
public override string? GetHelpText() => "/exit (quit)";
/// <inheritdoc/>
public override ValueTask<bool> TryHandleAsync(string input, AgentSession session, IUXStateDriver ux)
{
if (!input.Equals("/exit", StringComparison.OrdinalIgnoreCase))
{
return new ValueTask<bool>(false);
}
ux.RequestShutdown();
return new ValueTask<bool>(true);
}
}
@@ -7,7 +7,7 @@ namespace Harness.Shared.Console.Commands;
/// <summary>
/// Handles the <c>/mode</c> command to display or switch the current agent mode.
/// </summary>
internal sealed class ModeCommandHandler : CommandHandler
public sealed class ModeCommandHandler : CommandHandler
{
private readonly AgentModeProvider? _modeProvider;
private readonly IReadOnlyDictionary<string, ConsoleColor>? _modeColors;
@@ -27,7 +27,7 @@ internal sealed class ModeCommandHandler : CommandHandler
public override string? GetHelpText() => this._modeProvider is not null ? "/mode [plan|execute] (show or switch mode)" : null;
/// <inheritdoc/>
public override async ValueTask<bool> TryHandleAsync(string input, AgentSession session, HarnessUXContainer ux)
public override async ValueTask<bool> TryHandleAsync(string input, AgentSession session, IUXStateDriver ux)
{
if (!input.StartsWith("/mode ", StringComparison.OrdinalIgnoreCase) && !input.Equals("/mode", StringComparison.OrdinalIgnoreCase))
{
@@ -7,7 +7,7 @@ namespace Harness.Shared.Console.Commands;
/// <summary>
/// Handles the <c>/todos</c> command to display the current todo list.
/// </summary>
internal sealed class TodoCommandHandler : CommandHandler
public sealed class TodoCommandHandler : CommandHandler
{
private readonly TodoProvider? _todoProvider;
@@ -24,7 +24,7 @@ internal sealed class TodoCommandHandler : CommandHandler
public override string? GetHelpText() => this._todoProvider is not null ? "/todos (show todo list)" : null;
/// <inheritdoc/>
public override async ValueTask<bool> TryHandleAsync(string input, AgentSession session, HarnessUXContainer ux)
public override async ValueTask<bool> TryHandleAsync(string input, AgentSession session, IUXStateDriver ux)
{
if (!input.Equals("/todos", StringComparison.OrdinalIgnoreCase))
{
@@ -0,0 +1,59 @@
// Copyright (c) Microsoft. All rights reserved.
using Microsoft.Extensions.AI;
namespace Harness.Shared.Console;
/// <summary>
/// Represents an action returned by an observer at the end of an agent turn.
/// Subtypes describe either a question to ask the user (<see cref="FollowUpQuestion"/>)
/// or a message to add directly to the next agent input (<see cref="FollowUpMessage"/>).
/// </summary>
public abstract record FollowUpAction;
/// <summary>
/// Represents a question that should be presented to the user. The
/// <see cref="Continuation"/> delegate is invoked with the user's answer and the
/// UX state driver, and returns an optional <see cref="ChatMessage"/> to add to the
/// next agent invocation.
/// </summary>
/// <param name="Prompt">The question text shown to the user.</param>
/// <param name="Continuation">
/// Invoked with the user's answer and the UX state driver. The driver lets the
/// continuation write output (e.g., an action label like "Approved") in addition
/// to producing an optional <see cref="ChatMessage"/> for the next agent invocation.
/// </param>
public abstract record FollowUpQuestion(
string Prompt,
Func<string, IUXStateDriver, Task<ChatMessage?>> Continuation) : FollowUpAction;
/// <summary>
/// A free-form text question. The user may type any response.
/// </summary>
/// <param name="Prompt">The question text shown to the user.</param>
/// <param name="Continuation">Continuation that builds the response message.</param>
public sealed record TextFollowUpQuestion(
string Prompt,
Func<string, IUXStateDriver, Task<ChatMessage?>> Continuation)
: FollowUpQuestion(Prompt, Continuation);
/// <summary>
/// A choice question. The user picks from <paramref name="Choices"/>, optionally with
/// the ability to enter custom text when <paramref name="AllowCustomText"/> is true.
/// </summary>
/// <param name="Prompt">The question text shown to the user.</param>
/// <param name="Choices">The list of pre-defined choices.</param>
/// <param name="AllowCustomText">If true, the user may type a custom response in addition to the listed choices.</param>
/// <param name="Continuation">Continuation that builds the response message.</param>
public sealed record ChoiceFollowUpQuestion(
string Prompt,
IReadOnlyList<string> Choices,
bool AllowCustomText,
Func<string, IUXStateDriver, Task<ChatMessage?>> Continuation)
: FollowUpQuestion(Prompt, Continuation);
/// <summary>
/// A message to add directly to the next agent invocation without prompting the user.
/// </summary>
/// <param name="Message">The chat message to add.</param>
public sealed record FollowUpMessage(ChatMessage Message) : FollowUpAction;
@@ -0,0 +1,279 @@
// Copyright (c) Microsoft. All rights reserved.
using Harness.Shared.Console.Commands;
using Harness.Shared.Console.Observers;
using Microsoft.Agents.AI;
using Microsoft.Extensions.AI;
namespace Harness.Shared.Console;
/// <summary>
/// Orchestrates agent invocations driven by user-input events from the UI.
/// The component invokes the runner's input handlers (<see cref="OnUserInputAsync"/>,
/// <see cref="OnStreamingInputAsync"/>, <see cref="StartAgentTurnAsync"/>) directly;
/// the runner mutates UI state through the supplied <see cref="IUXStateDriver"/>.
/// All per-turn follow-up state (pending questions and accumulated responses) lives
/// in the component's state record — the runner reads/writes it exclusively through
/// the driver and holds no per-turn fields itself.
/// </summary>
public sealed class HarnessAgentRunner : IDisposable
{
private readonly AIAgent _agent;
private readonly AgentSession _session;
private readonly AgentModeProvider? _modeProvider;
private readonly MessageInjectingChatClient? _messageInjector;
private readonly IReadOnlyList<CommandHandler> _commandHandlers;
private readonly IReadOnlyList<ConsoleObserver> _observers;
private readonly IUXStateDriver _ux;
private readonly SemaphoreSlim _inputGate = new(1, 1);
/// <summary>
/// Initializes a new instance of the <see cref="HarnessAgentRunner"/> class.
/// </summary>
public HarnessAgentRunner(
AIAgent agent,
AgentSession session,
AgentModeProvider? modeProvider,
MessageInjectingChatClient? messageInjector,
IReadOnlyList<CommandHandler> commandHandlers,
IReadOnlyList<ConsoleObserver> observers,
IUXStateDriver ux)
{
this._agent = agent;
this._session = session;
this._modeProvider = modeProvider;
this._messageInjector = messageInjector;
this._commandHandlers = commandHandlers;
this._observers = observers;
this._ux = ux;
this.HelpText = string.Join(
", ",
commandHandlers
.Select(h => h.GetHelpText())
.Where(t => t is not null)!);
}
/// <summary>
/// Gets the help text describing all available commands (joined by ", "), suitable
/// for display in the mode-and-help bar. Computed from the supplied
/// <c>commandHandlers</c>.
/// </summary>
public string HelpText { get; }
/// <inheritdoc/>
public void Dispose() => this._inputGate.Dispose();
/// <summary>
/// Handles a top-level user input submission (TextInput mode, no pending question).
/// Dispatches to command handlers, or starts an agent turn.
/// </summary>
internal async Task OnUserInputAsync(string text)
{
await this._inputGate.WaitAsync().ConfigureAwait(false);
try
{
this._ux.WriteUserInputEcho(text);
foreach (var handler in this._commandHandlers)
{
if (await handler.TryHandleAsync(text, this._session, this._ux).ConfigureAwait(false))
{
this._ux.CurrentMode = this._modeProvider?.GetMode(this._session);
return;
}
}
await this.RunAgentLoopAsync([new ChatMessage(ChatRole.User, text)]).ConfigureAwait(false);
}
finally
{
this._inputGate.Release();
}
}
/// <summary>
/// Handles a user input submission while an agent turn is streaming. The text is
/// enqueued via the <see cref="MessageInjectingChatClient"/> so it can be picked up
/// by the agent on its next opportunity.
/// </summary>
internal Task OnStreamingInputAsync(string text)
{
if (this._messageInjector is null)
{
return Task.CompletedTask;
}
this._messageInjector.EnqueueMessages(this._session, [new ChatMessage(ChatRole.User, text)]);
this._ux.SetQueuedMessages(this._messageInjector.GetPendingMessages(this._session));
return Task.CompletedTask;
}
/// <summary>
/// Resumes (or completes) a turn after the user has answered all pending follow-up
/// questions. The component invokes this with the messages drained from
/// <see cref="IUXStateDriver.TakeFollowUpResponses"/>; an empty list simply ends
/// the streaming display state without invoking the agent.
/// </summary>
internal async Task StartAgentTurnAsync(IList<ChatMessage> messages)
{
await this._inputGate.WaitAsync().ConfigureAwait(false);
try
{
if (messages.Count == 0)
{
this.CompleteTurn();
return;
}
await this.RunAgentLoopAsync(messages).ConfigureAwait(false);
}
finally
{
this._inputGate.Release();
}
}
private async Task RunAgentLoopAsync(IList<ChatMessage> messages)
{
IList<ChatMessage>? nextMessages = messages;
IReadOnlyList<ChatMessage> lastPendingMessages = this._messageInjector?.GetPendingMessages(this._session) ?? [];
while (nextMessages is not null)
{
var runOptions = new AgentRunOptions();
foreach (var observer in this._observers)
{
observer.ConfigureRunOptions(runOptions, this._agent, this._session);
}
this._ux.CurrentMode = this._modeProvider?.GetMode(this._session);
this._ux.BeginStreaming();
this._ux.BeginStreamingOutput();
try
{
await foreach (var update in this._agent.RunStreamingAsync(nextMessages, this._session, runOptions))
{
if (this._modeProvider is not null)
{
string currentMode = this._modeProvider.GetMode(this._session);
if (currentMode != this._ux.CurrentMode)
{
this._ux.CurrentMode = currentMode;
}
}
foreach (var content in update.Contents)
{
foreach (var observer in this._observers)
{
await observer.OnContentAsync(this._ux, content, this._agent, this._session).ConfigureAwait(false);
}
}
if (!string.IsNullOrEmpty(update.Text))
{
foreach (var observer in this._observers)
{
await observer.OnTextAsync(this._ux, update.Text, this._agent, this._session).ConfigureAwait(false);
}
}
this.SyncQueuedMessageDisplay(ref lastPendingMessages);
}
}
catch (Exception ex)
{
await this._ux.WriteInfoLineAsync($"❌ Stream error: {ex.GetType().Name}:\n{ex}", ConsoleColor.Red).ConfigureAwait(false);
}
// Final sync after streaming.
this.SyncQueuedMessageDisplay(ref lastPendingMessages);
this._ux.StopSpinner();
await this._ux.EndStreamingOutputAsync().ConfigureAwait(false);
// Collect FollowUpActions from each observer.
var directMessages = new List<ChatMessage>();
var questions = new List<FollowUpQuestion>();
foreach (var observer in this._observers)
{
var actions = await observer.OnStreamCompleteAsync(this._ux, this._agent, this._session).ConfigureAwait(false);
if (actions is null)
{
continue;
}
foreach (var action in actions)
{
switch (action)
{
case FollowUpMessage msg:
directMessages.Add(msg.Message);
break;
case FollowUpQuestion q:
questions.Add(q);
break;
}
}
}
bool hasFollowUpActions = directMessages.Count > 0 || questions.Count > 0;
await this._ux.WriteNoTextWarningAsync(hasFollowUpActions).ConfigureAwait(false);
// Add any direct messages to the accumulator regardless of whether questions follow —
// they're sent on the next agent invocation, either by us (if no questions) or by
// the component (after the user finishes answering, via StartAgentTurnAsync).
foreach (var msg in directMessages)
{
this._ux.AddFollowUpResponse(msg);
}
if (questions.Count > 0)
{
// Pause: hand control back to the UX to collect answers.
this._ux.QueueFollowUpQuestions(questions);
return;
}
// No questions to ask — drain anything we just accumulated and loop with it.
IReadOnlyList<ChatMessage> drained = this._ux.TakeFollowUpResponses();
nextMessages = drained.Count > 0 ? [.. drained] : null;
}
this.CompleteTurn();
}
private void CompleteTurn()
{
this._ux.EndStreaming();
this._ux.CurrentMode = this._modeProvider?.GetMode(this._session);
}
/// <summary>
/// Synchronizes the queued items display with the message injector's pending messages.
/// Messages that have been consumed (drained by the service) are echoed to the output
/// area as regular user-input entries.
/// </summary>
private void SyncQueuedMessageDisplay(ref IReadOnlyList<ChatMessage> lastPendingMessages)
{
if (this._messageInjector is null)
{
return;
}
var pending = this._messageInjector.GetPendingMessages(this._session);
int consumedCount = lastPendingMessages.Count - pending.Count;
for (int i = 0; i < consumedCount && i < lastPendingMessages.Count; i++)
{
string text = lastPendingMessages[i].Text ?? string.Empty;
this._ux.WriteUserInputEcho(text);
}
lastPendingMessages = pending;
this._ux.SetQueuedMessages(pending);
}
}
@@ -3,171 +3,89 @@
using Harness.ConsoleReactiveComponents;
using Harness.ConsoleReactiveFramework;
using Harness.Shared.Console.Components;
using Microsoft.Extensions.AI;
namespace Harness.Shared.Console;
/// <summary>
/// Determines which component is shown in the bottom panel.
/// </summary>
public enum BottomPanelMode
{
/// <summary>Show the text input component for user input.</summary>
TextInput,
/// <summary>Show the list selection component for interactive prompts.</summary>
ListSelection,
/// <summary>Show a disabled input indicator during agent streaming.</summary>
Streaming,
}
/// <summary>
/// Event arguments for the <see cref="HarnessAppComponent.InputSubmitted"/> event.
/// </summary>
public sealed class InputSubmittedEventArgs : EventArgs
{
/// <summary>
/// Initializes a new instance of the <see cref="InputSubmittedEventArgs"/> class.
/// </summary>
/// <param name="text">The submitted text.</param>
/// <param name="mode">The bottom panel mode in which the input was submitted.</param>
public InputSubmittedEventArgs(string text, BottomPanelMode mode)
{
this.Text = text;
this.Mode = mode;
}
/// <summary>Gets the submitted text.</summary>
public string Text { get; }
/// <summary>Gets the bottom panel mode in which the input was submitted.</summary>
public BottomPanelMode Mode { get; }
}
/// <summary>
/// Props for <see cref="HarnessAppComponent"/>.
/// </summary>
public record HarnessAppComponentProps : ConsoleReactiveProps
{
/// <summary>Gets or sets the list selection choices (for ListSelection mode).</summary>
public IReadOnlyList<string> Items { get; set; } = Array.Empty<string>();
/// <summary>Gets or sets the scroll items (output entries) to render in the scroll panel.</summary>
public IReadOnlyList<object> ScrollItems { get; set; } = [];
/// <summary>Gets or sets the bottom panel mode.</summary>
public BottomPanelMode Mode { get; set; } = BottomPanelMode.TextInput;
/// <summary>Gets or sets the prompt string for text input mode.</summary>
public string Prompt { get; set; } = "You: ";
/// <summary>Gets or sets the placeholder text shown when the input is empty.</summary>
public string Placeholder { get; set; } = "";
/// <summary>Gets or sets the highlight color for the active list item.</summary>
public ConsoleColor ListHighlightColor { get; set; } = ConsoleColor.Cyan;
/// <summary>Gets or sets the placeholder text for the custom text input option in the list.</summary>
public string? ListCustomTextPlaceholder { get; set; }
/// <summary>Gets or sets the foreground color for the rule borders and mode label.</summary>
public ConsoleColor? ModeColor { get; set; }
/// <summary>Gets or sets the current mode name displayed below the bottom rule (e.g. "plan").</summary>
public string? ModeText { get; set; }
/// <summary>Gets or sets the help text displayed below the bottom rule (available commands).</summary>
public string? HelpText { get; set; }
/// <summary>Gets or sets the title text displayed above the list selection (for interactive prompts).</summary>
public string? ListTitle { get; set; }
/// <summary>Gets or sets a value indicating whether input is enabled during streaming.</summary>
public bool InputEnabled { get; set; }
/// <summary>Gets or sets the prompt to show during streaming when input is disabled.</summary>
public string StreamingPrompt { get; set; } = "(agent is running...)";
/// <summary>Gets or sets a value indicating whether the agent status spinner is visible.</summary>
public bool ShowSpinner { get; set; }
/// <summary>Gets or sets the formatted token usage text to display in the status bar.</summary>
public string? UsageText { get; set; }
/// <summary>Gets or sets the queued input items to display above the rule.</summary>
public IReadOnlyList<object> QueuedItems { get; set; } = [];
}
/// <summary>
/// Internal state for <see cref="HarnessAppComponent"/>.
/// </summary>
public record HarnessAppComponentState : ConsoleReactiveState
{
/// <summary>Gets the selected index in list selection mode.</summary>
public int SelectedIndex { get; init; }
/// <summary>Gets the current input text being typed.</summary>
public string InputText { get; init; } = "";
/// <summary>Gets the current text being typed into the list's custom text option.</summary>
public string ListInputText { get; init; } = "";
/// <summary>Gets the current console width in columns.</summary>
public int ConsoleWidth { get; init; }
/// <summary>Gets the current console height in rows.</summary>
public int ConsoleHeight { get; init; }
}
/// <summary>
/// The main application component for the Harness console. Manages the scroll region
/// and bottom panel (text input, list selection, or streaming indicator), and emits
/// an <see cref="InputSubmitted"/> event when the user submits text in any mode.
/// and bottom panel (text input, list selection, or streaming indicator). Owns the
/// <see cref="HarnessConsoleUXStateDriver"/> and routes user input events to the
/// registered <see cref="HarnessAgentRunner"/>.
/// </summary>
public class HarnessAppComponent : ConsoleReactiveComponent<HarnessAppComponentProps, HarnessAppComponentState>, IDisposable
public class HarnessAppComponent : ConsoleReactiveComponent<ConsoleReactiveProps, HarnessAppComponentState>, IDisposable
{
private readonly TopBottomRule _rule = new();
private readonly ListSelection _listSelection = new();
private readonly TextInput _textInput = new();
private readonly TextScrollPanel _textScrollPanel;
private readonly TextPanel _textPanel;
private readonly TextPanel _queuedPanel;
private readonly TextScrollPanel _textScrollPanel = new();
private readonly TextPanel _textPanel = new();
private readonly TextPanel _queuedPanel = new();
private readonly AgentStatus _agentStatus = new();
private readonly AgentModeAndHelp _modeAndHelp = new();
private readonly Func<object, string> _renderItem;
private bool _resizedSinceLastRender;
private readonly HarnessConsoleUXStateDriver _uxDriver;
private readonly TaskCompletionSource<bool> _shutdownTcs = new(TaskCreationOptions.RunContinuationsAsynchronously);
private readonly SemaphoreSlim _followUpGate = new(1, 1);
private int _scrollRegionBottom;
private bool _resizedSinceLastRender = true;
private bool _deactivated;
/// <summary>
/// Initializes a new instance of the <see cref="HarnessAppComponent"/> class.
/// </summary>
/// <param name="renderScrollItem">A delegate that renders a single output entry and returns the text to display.</param>
public HarnessAppComponent(Func<object, string> renderScrollItem)
/// <param name="placeholder">Placeholder text shown when the input is empty.</param>
/// <param name="initialMode">The current agent mode, used to colour the rule and prompt.</param>
/// <param name="inputEnabled">Whether the bottom-panel input accepts keystrokes during streaming.</param>
/// <param name="runnerFactory">Factory invoked with the component's <see cref="IUXStateDriver"/>
/// to construct the <see cref="HarnessAgentRunner"/> that owns the agent loop.</param>
/// <param name="modeColors">Optional mapping of mode names to console colors.</param>
public HarnessAppComponent(
string placeholder,
string? initialMode,
bool inputEnabled,
Func<IUXStateDriver, HarnessAgentRunner> runnerFactory,
IReadOnlyDictionary<string, ConsoleColor>? modeColors = null)
{
this._renderItem = renderScrollItem;
this._textScrollPanel = new TextScrollPanel(renderScrollItem);
this._textPanel = new TextPanel(renderScrollItem);
this._queuedPanel = new TextPanel(renderScrollItem);
this.Props = new ConsoleReactiveProps();
this.State = new HarnessAppComponentState
{
Mode = BottomPanelMode.TextInput,
Prompt = "> ",
Placeholder = placeholder,
ModeColor = ModeColors.Get(initialMode, modeColors),
ModeText = initialMode,
InputEnabled = inputEnabled,
ConsoleWidth = System.Console.WindowWidth,
ConsoleHeight = System.Console.WindowHeight,
};
this._uxDriver = new HarnessConsoleUXStateDriver(
getState: () => this.State!,
setState: s => this.SetState(s),
requestShutdown: () => this._shutdownTcs.TrySetResult(true),
modeColors: modeColors);
this.Runner = runnerFactory(this._uxDriver);
// Seed help text now that the runner (which knows the registered command handlers)
// is available. Direct assignment — no Render is triggered until the caller invokes Render().
this.State = this.State with { HelpText = this.Runner.HelpText };
KeyEventListener.Instance.KeyPressed += this.OnKeyPressed;
ConsoleResizeListener.Instance.ConsoleResized += this.OnConsoleResized;
}
/// <summary>
/// Gets the 1-based row number of the last row in the output scroll region.
/// Gets the agent runner that owns the agent loop. Constructed by the factory
/// passed to the component's constructor.
/// </summary>
public int ScrollRegionBottom { get; private set; }
public HarnessAgentRunner Runner { get; }
/// <summary>
/// Occurs when the user submits input via Enter, in any mode (text input, list selection,
/// or streaming injection). Consumers inspect <see cref="InputSubmittedEventArgs.Mode"/>
/// to decide how to handle the submission.
/// Completes when a command handler requests application shutdown (e.g. the user types <c>/exit</c>).
/// Awaited by <see cref="HarnessConsole.RunAgentAsync"/>.
/// </summary>
public event EventHandler<InputSubmittedEventArgs>? InputSubmitted;
public Task ShutdownTask => this._shutdownTcs.Task;
/// <summary>
/// Deactivates the component, resetting the scroll region and unsubscribing from events.
@@ -184,9 +102,6 @@ public class HarnessAppComponent : ConsoleReactiveComponent<HarnessAppComponentP
this._agentStatus.Dispose();
KeyEventListener.Instance.KeyPressed -= this.OnKeyPressed;
ConsoleResizeListener.Instance.ConsoleResized -= this.OnConsoleResized;
System.Console.Write(AnsiEscapes.ResetScrollRegion);
System.Console.Write(AnsiEscapes.MoveCursor(System.Console.WindowHeight, 1));
System.Console.WriteLine();
}
/// <inheritdoc/>
@@ -205,20 +120,23 @@ public class HarnessAppComponent : ConsoleReactiveComponent<HarnessAppComponentP
if (disposing)
{
this.Deactivate();
this._followUpGate.Dispose();
this.Runner.Dispose();
}
}
private void OnKeyPressed(object? sender, KeyPressEventArgs e)
{
if (this.Props!.Mode == BottomPanelMode.TextInput)
BottomPanelMode mode = this.State!.Mode;
if (mode == BottomPanelMode.TextInput)
{
this.HandleTextInputKey(e);
}
else if (this.Props.Mode == BottomPanelMode.ListSelection)
else if (mode == BottomPanelMode.ListSelection)
{
this.HandleListSelectionKey(e);
}
else if (this.Props.Mode == BottomPanelMode.Streaming && this.Props.InputEnabled)
else if (mode == BottomPanelMode.Streaming && this.State.InputEnabled)
{
this.HandleStreamingInputKey(e);
}
@@ -235,7 +153,7 @@ public class HarnessAppComponent : ConsoleReactiveComponent<HarnessAppComponentP
}
this.SetState(this.State with { InputText = "" });
this.InputSubmitted?.Invoke(this, new InputSubmittedEventArgs(text, BottomPanelMode.TextInput));
this.DispatchTextInputSubmission(text);
}
else if (e.KeyInfo.Key == ConsoleKey.Backspace)
{
@@ -252,51 +170,50 @@ public class HarnessAppComponent : ConsoleReactiveComponent<HarnessAppComponentP
private void HandleListSelectionKey(KeyPressEventArgs e)
{
int maxIndex = this.Props!.Items.Count - 1;
if (this.Props.ListCustomTextPlaceholder != null)
int maxIndex = this.State!.ListSelectionOptions.Count - 1;
if (this.State.ListSelectionCustomTextPlaceholder != null)
{
maxIndex = this.Props.Items.Count;
maxIndex = this.State.ListSelectionOptions.Count;
}
bool isOnCustomTextOption = this.Props.ListCustomTextPlaceholder != null
&& this.State!.SelectedIndex == this.Props.Items.Count;
bool isOnCustomTextOption = this.State.ListSelectionCustomTextPlaceholder != null
&& this.State.ListSelectionIndex == this.State.ListSelectionOptions.Count;
if (e.KeyInfo.Key == ConsoleKey.UpArrow)
{
this.SetState(this.State! with { SelectedIndex = Math.Max(0, this.State.SelectedIndex - 1) });
this.SetState(this.State with { ListSelectionIndex = Math.Max(0, this.State.ListSelectionIndex - 1) });
}
else if (e.KeyInfo.Key == ConsoleKey.DownArrow)
{
this.SetState(this.State! with { SelectedIndex = Math.Min(maxIndex, this.State.SelectedIndex + 1) });
this.SetState(this.State with { ListSelectionIndex = Math.Min(maxIndex, this.State.ListSelectionIndex + 1) });
}
else if (e.KeyInfo.Key == ConsoleKey.Enter)
{
string result = isOnCustomTextOption
? this.State!.ListInputText
: this.Props.Items[this.State!.SelectedIndex];
? this.State.ListSelectionCustomInputText
: this.State.ListSelectionOptions[this.State.ListSelectionIndex];
this.SetState(this.State with { ListInputText = "", SelectedIndex = 0 });
this.InputSubmitted?.Invoke(this, new InputSubmittedEventArgs(result, BottomPanelMode.ListSelection));
this.SetState(this.State with { ListSelectionCustomInputText = "", ListSelectionIndex = 0 });
this.DispatchListSelectionSubmission(result);
}
else if (isOnCustomTextOption)
{
if (e.KeyInfo.Key == ConsoleKey.Backspace)
{
if (this.State!.ListInputText.Length > 0)
if (this.State.ListSelectionCustomInputText.Length > 0)
{
this.SetState(this.State with { ListInputText = this.State.ListInputText[..^1] });
this.SetState(this.State with { ListSelectionCustomInputText = this.State.ListSelectionCustomInputText[..^1] });
}
}
else if (e.KeyInfo.KeyChar != '\0' && !char.IsControl(e.KeyInfo.KeyChar))
{
this.SetState(this.State! with { ListInputText = this.State.ListInputText + e.KeyInfo.KeyChar });
this.SetState(this.State with { ListSelectionCustomInputText = this.State.ListSelectionCustomInputText + e.KeyInfo.KeyChar });
}
}
}
private void HandleStreamingInputKey(KeyPressEventArgs e)
{
// During streaming with input enabled, capture text for message injection
if (e.KeyInfo.Key == ConsoleKey.Enter)
{
string text = this.State!.InputText;
@@ -306,7 +223,7 @@ public class HarnessAppComponent : ConsoleReactiveComponent<HarnessAppComponentP
}
this.SetState(this.State with { InputText = "" });
this.InputSubmitted?.Invoke(this, new InputSubmittedEventArgs(text, BottomPanelMode.Streaming));
_ = this.Runner.OnStreamingInputAsync(text);
}
else if (e.KeyInfo.Key == ConsoleKey.Backspace)
{
@@ -321,6 +238,90 @@ public class HarnessAppComponent : ConsoleReactiveComponent<HarnessAppComponentP
}
}
private void DispatchTextInputSubmission(string text)
{
if (this.State!.PendingQuestions.Count > 0)
{
_ = this.HandleFollowUpAnswerAsync(text);
}
else
{
_ = this.Runner.OnUserInputAsync(text);
}
}
private void DispatchListSelectionSubmission(string text)
{
// List selection is only used to answer FollowUpQuestions.
_ = this.HandleFollowUpAnswerAsync(text);
}
/// <summary>
/// Handles a user answer to the head of the pending follow-up question queue:
/// awaits the question's continuation (which is responsible for echoing both the
/// question and answer to the scroll area as it sees fit), appends any returned
/// chat message to the response accumulator, advances the queue, and — when the
/// queue empties — drains the accumulator and resumes the runner.
/// </summary>
private async Task HandleFollowUpAnswerAsync(string text)
{
IReadOnlyList<ChatMessage>? messagesToSend = null;
await this._followUpGate.WaitAsync().ConfigureAwait(false);
try
{
HarnessConsoleUXStateDriver ux = this._uxDriver;
IReadOnlyList<FollowUpQuestion> queue = this.State!.PendingQuestions;
if (queue.Count == 0)
{
return;
}
FollowUpQuestion head = queue[0];
ChatMessage? response;
try
{
response = await head.Continuation(text, ux).ConfigureAwait(false);
}
catch (Exception ex)
{
await ux.WriteInfoLineAsync($"❌ Follow-up handler error: {ex.GetType().Name}: {ex.Message}", ConsoleColor.Red).ConfigureAwait(false);
response = null;
}
if (response is not null)
{
ux.AddFollowUpResponse(response);
}
ux.AdvanceFollowUpQuestion();
if (this.State!.PendingQuestions.Count == 0)
{
messagesToSend = ux.TakeFollowUpResponses();
}
}
finally
{
this._followUpGate.Release();
}
// Resume the agent outside the gate — StartAgentTurnAsync runs the full agent
// loop which may queue new follow-up questions (re-entering this method).
if (messagesToSend is not null)
{
try
{
await this.Runner.StartAgentTurnAsync([.. messagesToSend]).ConfigureAwait(false);
}
catch (Exception ex)
{
await this._uxDriver.WriteInfoLineAsync($"❌ Agent error: {ex.GetType().Name}: {ex.Message}", ConsoleColor.Red).ConfigureAwait(false);
}
}
}
private void OnConsoleResized(object? sender, ConsoleResizeEventArgs e)
{
this._resizedSinceLastRender = true;
@@ -332,35 +333,40 @@ public class HarnessAppComponent : ConsoleReactiveComponent<HarnessAppComponentP
}
/// <inheritdoc />
public override void RenderCore(HarnessAppComponentProps props, HarnessAppComponentState state)
public override void RenderCore(ConsoleReactiveProps props, HarnessAppComponentState state)
{
if (this._deactivated)
{
return;
}
// Determine the text panel height for the last scroll item
IReadOnlyList<object> lastItems = props.ScrollItems.Count > 0
? [props.ScrollItems[^1]]
IReadOnlyList<string> lastItems = state.ScrollAreaContentItems.Count > 0
? [state.ScrollAreaContentItems[^1]]
: [];
int textPanelHeight = TextPanel.CalculateHeight(lastItems, this._renderItem);
int textPanelHeight = TextPanel.CalculateHeight(lastItems);
if (textPanelHeight > 0)
{
textPanelHeight++; // Extra line for spacing between text panel and rule
}
// Calculate queued items panel height
int queuedPanelHeight = TextPanel.CalculateHeight(props.QueuedItems, this._renderItem);
int queuedPanelHeight = TextPanel.CalculateHeight(state.QueuedItems);
// Build the bottom panel child based on mode
ConsoleReactiveComponent bottomChild;
int bottomChildHeight;
if (props.Mode == BottomPanelMode.ListSelection)
if (state.Mode == BottomPanelMode.ListSelection)
{
var listProps = new ListSelectionProps
{
Title = props.ListTitle,
Items = props.Items,
SelectedIndex = state.SelectedIndex,
HighlightColor = props.ListHighlightColor,
CustomTextPlaceholder = props.ListCustomTextPlaceholder,
CustomText = state.ListInputText,
Title = state.ListSelectionTitle,
Items = state.ListSelectionOptions,
SelectedIndex = state.ListSelectionIndex,
HighlightColor = state.ListHighlightColor,
CustomTextPlaceholder = state.ListSelectionCustomTextPlaceholder,
CustomText = state.ListSelectionCustomInputText,
};
bottomChildHeight = ListSelection.CalculateHeight(listProps);
@@ -368,25 +374,25 @@ public class HarnessAppComponent : ConsoleReactiveComponent<HarnessAppComponentP
this._listSelection.Props = listProps;
bottomChild = this._listSelection;
}
else if (props.Mode == BottomPanelMode.Streaming)
else if (state.Mode == BottomPanelMode.Streaming)
{
TextInputProps textInputProps;
if (props.InputEnabled)
if (state.InputEnabled)
{
textInputProps = new TextInputProps
{
Prompt = props.Prompt,
Prompt = state.Prompt,
Text = state.InputText,
Placeholder = props.Placeholder,
Placeholder = state.Placeholder,
};
}
else
{
textInputProps = new TextInputProps
{
Prompt = props.Prompt,
Prompt = state.Prompt,
Text = "",
Placeholder = props.StreamingPrompt,
Placeholder = state.StreamingPrompt,
};
}
@@ -400,9 +406,9 @@ public class HarnessAppComponent : ConsoleReactiveComponent<HarnessAppComponentP
{
var textInputProps = new TextInputProps
{
Prompt = props.Prompt,
Prompt = state.Prompt,
Text = state.InputText,
Placeholder = props.Placeholder,
Placeholder = state.Placeholder,
};
bottomChildHeight = TextInput.CalculateHeight(textInputProps, state.ConsoleWidth);
@@ -415,46 +421,52 @@ public class HarnessAppComponent : ConsoleReactiveComponent<HarnessAppComponentP
var ruleProps = new TopBottomRuleProps
{
Width = state.ConsoleWidth,
Color = props.ModeColor,
Color = state.ModeColor,
Children = [bottomChild],
};
// Calculate the agent status height
var agentStatusProps = new AgentStatusProps
{
ShowSpinner = props.ShowSpinner,
UsageText = props.UsageText,
ShowSpinner = state.ShowSpinner,
UsageText = state.UsageText,
};
int agentStatusHeight = AgentStatus.CalculateHeight(agentStatusProps);
// Calculate the mode-and-help height
var modeAndHelpProps = new AgentModeAndHelpProps
{
Mode = props.ModeText,
ModeColor = props.ModeColor,
HelpText = props.HelpText,
Mode = state.ModeText,
ModeColor = state.ModeColor,
HelpText = state.HelpText,
};
int modeAndHelpHeight = AgentModeAndHelp.CalculateHeight(modeAndHelpProps);
// Hide agent status and mode/help during follow-up questions (ListSelection mode)
// as they clutter the UI and aren't relevant.
bool showStatusAndHelp = state.Mode != BottomPanelMode.ListSelection;
int agentStatusHeight = showStatusAndHelp ? AgentStatus.CalculateHeight(agentStatusProps) : 0;
int modeAndHelpHeight = showStatusAndHelp ? AgentModeAndHelp.CalculateHeight(modeAndHelpProps) : 0;
int ruleHeight = TopBottomRule.CalculateHeight(ruleProps);
int scrollBottom = Math.Max(1, state.ConsoleHeight - ruleHeight - textPanelHeight - agentStatusHeight - queuedPanelHeight - modeAndHelpHeight);
int nonScrollHeight = ruleHeight + textPanelHeight + agentStatusHeight + queuedPanelHeight + modeAndHelpHeight + 1; // +1 for bottom padding
int scrollBottom = Math.Max(1, state.ConsoleHeight - nonScrollHeight);
// If scroll region changed or a clear is needed, reset everything
if (this._resizedSinceLastRender || (this.ScrollRegionBottom != 0 && scrollBottom != this.ScrollRegionBottom))
if (this._resizedSinceLastRender || (this._scrollRegionBottom != 0 && scrollBottom != this._scrollRegionBottom))
{
// Reset scroll region to full screen before erasing so the erase covers all rows —
// some terminals only erase within the active DECSTBM region.
System.Console.Write(AnsiEscapes.ResetScrollRegion);
System.Console.Write(AnsiEscapes.EraseEntireScreen);
System.Console.Write(AnsiEscapes.EraseScrollbackBuffer);
this._textScrollPanel.Reset();
this._resizedSinceLastRender = false;
}
this.ScrollRegionBottom = scrollBottom;
this._scrollRegionBottom = scrollBottom;
System.Console.Write(AnsiEscapes.SetScrollRegion(scrollBottom));
// Render text scroll panel in the scroll area (all items except the last)
IReadOnlyList<object> scrollItems = props.ScrollItems.Count > 1
? props.ScrollItems.Take(props.ScrollItems.Count - 1).ToList()
IReadOnlyList<string> scrollItems = state.ScrollAreaContentItems.Count > 1
? state.ScrollAreaContentItems.Take(state.ScrollAreaContentItems.Count - 1).ToList()
: [];
this._textScrollPanel.X = 1;
@@ -486,18 +498,21 @@ public class HarnessAppComponent : ConsoleReactiveComponent<HarnessAppComponentP
this._queuedPanel.Height = queuedPanelHeight;
this._queuedPanel.Props = new TextPanelProps
{
Items = props.QueuedItems,
Items = state.QueuedItems,
};
this._queuedPanel.Render();
// Render the agent status line between queued items and rule
int agentStatusY = queuedPanelY + queuedPanelHeight;
this._agentStatus.X = 1;
this._agentStatus.Y = agentStatusY;
this._agentStatus.Width = state.ConsoleWidth;
this._agentStatus.Height = agentStatusHeight;
this._agentStatus.Props = agentStatusProps;
this._agentStatus.Render();
if (showStatusAndHelp)
{
this._agentStatus.X = 1;
this._agentStatus.Y = agentStatusY;
this._agentStatus.Width = state.ConsoleWidth;
this._agentStatus.Height = agentStatusHeight;
this._agentStatus.Props = agentStatusProps;
this._agentStatus.Render();
}
// Render the bottom rule + child below the agent status
this._rule.X = 1;
@@ -506,24 +521,27 @@ public class HarnessAppComponent : ConsoleReactiveComponent<HarnessAppComponentP
this._rule.Render();
// Render the mode-and-help line below the bottom rule
int modeAndHelpY = this._rule.Y + ruleHeight;
this._modeAndHelp.X = 1;
this._modeAndHelp.Y = modeAndHelpY;
this._modeAndHelp.Width = state.ConsoleWidth;
this._modeAndHelp.Height = modeAndHelpHeight;
this._modeAndHelp.Props = modeAndHelpProps;
this._modeAndHelp.Render();
if (showStatusAndHelp)
{
int modeAndHelpY = this._rule.Y + ruleHeight;
this._modeAndHelp.X = 1;
this._modeAndHelp.Y = modeAndHelpY;
this._modeAndHelp.Width = state.ConsoleWidth;
this._modeAndHelp.Height = modeAndHelpHeight;
this._modeAndHelp.Props = modeAndHelpProps;
this._modeAndHelp.Render();
}
// Position cursor for natural typing appearance
this.PositionCursor(props, state);
this.PositionCursor(state);
}
private void PositionCursor(HarnessAppComponentProps props, HarnessAppComponentState state)
private void PositionCursor(HarnessAppComponentState state)
{
if (props.Mode == BottomPanelMode.TextInput
|| (props.Mode == BottomPanelMode.Streaming && props.InputEnabled))
if (state.Mode == BottomPanelMode.TextInput
|| (state.Mode == BottomPanelMode.Streaming && state.InputEnabled))
{
int promptLength = props.Prompt.Length;
int promptLength = state.Prompt.Length;
int textWidth = state.ConsoleWidth - promptLength;
int textLength = state.InputText.Length;
@@ -540,13 +558,13 @@ public class HarnessAppComponent : ConsoleReactiveComponent<HarnessAppComponentP
System.Console.Write(AnsiEscapes.MoveCursor(textInputY + cursorRow, promptLength + cursorCol + 1));
}
}
else if (props.Mode == BottomPanelMode.ListSelection
&& props.ListCustomTextPlaceholder != null
&& state.SelectedIndex == props.Items.Count)
else if (state.Mode == BottomPanelMode.ListSelection
&& state.ListSelectionCustomTextPlaceholder != null
&& state.ListSelectionIndex == state.ListSelectionOptions.Count)
{
int titleLines = props.ListTitle?.Split('\n').Length ?? 0;
int customOptionY = this._rule.Y + 1 + titleLines + props.Items.Count;
int cursorCol = 2 + state.ListInputText.Length + 1;
int titleLines = state.ListSelectionTitle?.Split('\n').Length ?? 0;
int customOptionY = this._rule.Y + 1 + titleLines + state.ListSelectionOptions.Count;
int cursorCol = 2 + state.ListSelectionCustomInputText.Length + 1;
System.Console.Write(AnsiEscapes.MoveCursor(customOptionY, cursorCol));
}
}
@@ -0,0 +1,125 @@
// Copyright (c) Microsoft. All rights reserved.
using Harness.ConsoleReactiveFramework;
using Microsoft.Extensions.AI;
namespace Harness.Shared.Console;
/// <summary>
/// Determines which component is shown in the bottom panel.
/// </summary>
public enum BottomPanelMode
{
/// <summary>Show the text input component for user input.</summary>
TextInput,
/// <summary>Show the list selection component for interactive prompts.</summary>
ListSelection,
/// <summary>Show a disabled input indicator during agent streaming.</summary>
Streaming,
}
/// <summary>
/// Internal state for <see cref="HarnessAppComponent"/>. All UI fields that may
/// change after construction live here; they are mutated exclusively via
/// <see cref="ConsoleReactiveComponent{TProps,TState}.SetState"/> by the
/// owning <see cref="HarnessConsoleUXStateDriver"/>.
/// </summary>
public record HarnessAppComponentState : ConsoleReactiveState
{
// --- Console dimensions ---
/// <summary>Gets the current console width in columns.</summary>
public int ConsoleWidth { get; init; }
/// <summary>Gets the current console height in rows.</summary>
public int ConsoleHeight { get; init; }
// --- Bottom panel mode ---
/// <summary>Gets the bottom panel mode.</summary>
public BottomPanelMode Mode { get; init; } = BottomPanelMode.TextInput;
/// <summary>
/// Gets the queue of follow-up questions waiting for user answers. The head
/// (<c>[0]</c>) is the question currently being displayed; subsequent items
/// are dispatched in order as each is answered. While this queue is non-empty,
/// the next user submission is treated as the answer to the head question
/// instead of going to the agent runner's normal input handler.
/// </summary>
public IReadOnlyList<FollowUpQuestion> PendingQuestions { get; init; } = [];
/// <summary>
/// Gets the accumulated follow-up response messages collected during the
/// current agent turn — both direct <see cref="FollowUpMessage"/>s emitted
/// by observers and continuation results from answered questions. Consumed
/// by the runner via <see cref="IUXStateDriver.TakeFollowUpResponses"/>
/// before the next agent invocation.
/// </summary>
public IReadOnlyList<ChatMessage> AccumulatedFollowUpResponses { get; init; } = [];
// --- Text input (active in TextInput / Streaming modes) ---
/// <summary>Gets the prompt string for text input mode.</summary>
public string Prompt { get; init; } = "> ";
/// <summary>Gets the placeholder text shown when the input is empty.</summary>
public string Placeholder { get; init; } = "";
/// <summary>Gets the current input text being typed.</summary>
public string InputText { get; init; } = "";
/// <summary>Gets a value indicating whether input is enabled during streaming.</summary>
public bool InputEnabled { get; init; }
/// <summary>Gets the prompt to show during streaming when input is disabled.</summary>
public string StreamingPrompt { get; init; } = "(agent is running...)";
// --- List selection (active in ListSelection mode) ---
/// <summary>Gets the title text displayed above the list selection (for interactive prompts).</summary>
public string? ListSelectionTitle { get; init; }
/// <summary>Gets the list selection options.</summary>
public IReadOnlyList<string> ListSelectionOptions { get; init; } = [];
/// <summary>Gets the highlighted option index in list selection mode.</summary>
public int ListSelectionIndex { get; init; }
/// <summary>Gets the placeholder text for the custom text input option in the list.</summary>
public string? ListSelectionCustomTextPlaceholder { get; init; }
/// <summary>Gets the current text being typed into the list's custom text option.</summary>
public string ListSelectionCustomInputText { get; init; } = "";
/// <summary>Gets the highlight color for the active list item.</summary>
public ConsoleColor ListHighlightColor { get; init; } = ConsoleColor.Cyan;
// --- Scroll / output area ---
/// <summary>Gets the items rendered in the scroll-area. Each item is a pre-rendered
/// console string (may include ANSI escape sequences and newlines).</summary>
public IReadOnlyList<string> ScrollAreaContentItems { get; init; } = [];
/// <summary>Gets the queued input items to display above the rule. Each item is a
/// pre-rendered console string (may include ANSI escape sequences and newlines).</summary>
public IReadOnlyList<string> QueuedItems { get; init; } = [];
// --- Agent mode + status display ---
/// <summary>Gets the foreground color for the rule borders and mode label.</summary>
public ConsoleColor? ModeColor { get; init; }
/// <summary>Gets the current mode name displayed below the bottom rule (e.g. "plan").</summary>
public string? ModeText { get; init; }
/// <summary>Gets the help text displayed below the bottom rule (available commands).</summary>
public string? HelpText { get; init; }
/// <summary>Gets a value indicating whether the agent status spinner is visible.</summary>
public bool ShowSpinner { get; init; }
/// <summary>Gets the formatted token usage text to display in the status bar.</summary>
public string? UsageText { get; init; }
}
@@ -1,9 +1,7 @@
// Copyright (c) Microsoft. All rights reserved.
using Harness.Shared.Console.Commands;
using Harness.Shared.Console.Observers;
using Harness.ConsoleReactiveComponents;
using Microsoft.Agents.AI;
using Microsoft.Extensions.AI;
namespace Harness.Shared.Console;
@@ -15,244 +13,58 @@ public static class HarnessConsole
{
/// <summary>
/// Runs an interactive console session with the specified agent.
/// Supports streaming output, tool call display, spinner animation,
/// optional planning UX with structured output, and the <c>/todos</c> command.
/// Constructs the reactive UI component and the <see cref="HarnessAgentRunner"/>,
/// wires them together, and awaits the component's <see cref="HarnessAppComponent.ShutdownTask"/>
/// (which completes when the user types <c>/exit</c>).
/// </summary>
/// <param name="agent">The agent to interact with.</param>
/// <param name="title">The title displayed in the console header.</param>
/// <param name="userPrompt">A short prompt to the user, displayed below the title.</param>
/// <param name="userPrompt">A short prompt to the user, displayed as a placeholder in the input area.</param>
/// <param name="options">Optional configuration options for the console session.</param>
public static async Task RunAgentAsync(AIAgent agent, string title, string userPrompt, HarnessConsoleOptions? options = null)
public static async Task RunAgentAsync(AIAgent agent, string userPrompt, HarnessConsoleOptions? options = null)
{
options ??= new();
if (options.EnablePlanningUx
&& (string.IsNullOrWhiteSpace(options.PlanningModeName) || string.IsNullOrWhiteSpace(options.ExecutionModeName)))
{
throw new ArgumentException(
"When EnablePlanningUx is true, both PlanningModeName and ExecutionModeName must be configured.",
nameof(options));
}
// Null means use defaults; an explicit (possibly empty) list means use exactly what was provided.
var observers = options.Observers
?? HarnessConsoleOptions.BuildDefaultObservers();
var commandHandlers = options.CommandHandlers
?? HarnessConsoleOptions.BuildDefaultCommandHandlers(agent, options.ModeColors);
var todoProvider = agent.GetService<TodoProvider>();
var modeProvider = agent.GetService<AgentModeProvider>();
var messageInjector = agent.GetService<MessageInjectingChatClient>();
var commandHandlers = new List<CommandHandler>
{
new TodoCommandHandler(todoProvider),
new ModeCommandHandler(modeProvider, options.ModeColors),
};
AgentSession session = await agent.CreateSessionAsync();
using var ux = new HarnessUXContainer(
using var component = new HarnessAppComponent(
placeholder: userPrompt,
initialMode: modeProvider?.GetMode(session),
inputEnabled: messageInjector is not null,
runnerFactory: ux => new HarnessAgentRunner(
agent: agent,
session: session,
modeProvider: modeProvider,
messageInjector: messageInjector,
commandHandlers: commandHandlers,
observers: observers,
ux: ux),
modeColors: options.ModeColors);
// Streaming-mode submissions are enqueued for injection; the queued display
// is then refreshed from the injector's current pending list.
ux.StreamingInputReceived += (sender, e) =>
// Trigger the initial render of the component now that state is seeded.
component.Render();
try
{
if (messageInjector is null)
{
return;
}
messageInjector.EnqueueMessages(session, [new ChatMessage(ChatRole.User, e.Text)]);
ux.ShowQueuedMessages(messageInjector.GetPendingMessages(session));
};
var commandHelp = commandHandlers
.Select(h => h.GetHelpText())
.Where(t => t is not null)
.Append("exit (quit)")!;
ux.Initialize(title, commandHelp!, messageInjector is not null);
string userInput = await ux.WaitForInputAsync();
while (!string.IsNullOrWhiteSpace(userInput) && !userInput.Equals("exit", StringComparison.OrdinalIgnoreCase))
await component.ShutdownTask.ConfigureAwait(false);
}
finally
{
ux.WriteUserInputEcho(userInput);
// Check command handlers first — first one to handle wins.
bool handled = false;
foreach (var handler in commandHandlers)
{
if (await handler.TryHandleAsync(userInput, session, ux).ConfigureAwait(false))
{
handled = true;
break;
}
}
if (!handled)
{
await RunAgentTurnAsync(agent, session, modeProvider, messageInjector, options, ux, userInput);
}
ux.CurrentMode = modeProvider?.GetMode(session);
userInput = await ux.WaitForInputAsync();
component.Deactivate();
}
ux.Deactivate();
System.Console.ResetColor();
System.Console.Write(AnsiEscapes.ResetScrollRegion);
System.Console.Write(AnsiEscapes.EraseEntireScreen);
System.Console.Write(AnsiEscapes.MoveCursor(1, 1));
System.Console.WriteLine("Goodbye!");
}
/// <summary>
/// Runs one or more agent invocations for a single user turn, using the current
/// observers. Re-invokes automatically for tool approvals and mode-driven follow-ups
/// (e.g., planning clarification loops).
/// </summary>
private static async Task RunAgentTurnAsync(
AIAgent agent,
AgentSession session,
AgentModeProvider? modeProvider,
MessageInjectingChatClient? messageInjector,
HarnessConsoleOptions options,
HarnessUXContainer ux,
string userInput)
{
IList<ChatMessage>? nextMessages = [new ChatMessage(ChatRole.User, userInput)];
IReadOnlyList<ChatMessage> lastPendingMessages = messageInjector?.GetPendingMessages(session) ?? [];
while (nextMessages is not null)
{
var observers = CreateObservers(options, modeProvider, session);
var runOptions = new AgentRunOptions();
foreach (var observer in observers)
{
observer.ConfigureRunOptions(runOptions);
}
ux.CurrentMode = modeProvider?.GetMode(session);
ux.BeginStreaming();
ux.BeginStreamingOutput();
try
{
await foreach (var update in agent.RunStreamingAsync(nextMessages, session, runOptions))
{
// Update mode color if the mode changed during streaming.
if (modeProvider is not null)
{
string currentMode = modeProvider.GetMode(session);
if (currentMode != ux.CurrentMode)
{
ux.CurrentMode = currentMode;
}
}
foreach (var content in update.Contents)
{
foreach (var observer in observers)
{
await observer.OnContentAsync(ux, content);
}
}
if (!string.IsNullOrEmpty(update.Text))
{
foreach (var observer in observers)
{
await observer.OnTextAsync(ux, update.Text);
}
}
SyncQueuedMessageDisplay(messageInjector, session, ux, ref lastPendingMessages);
}
}
catch (Exception ex)
{
await ux.WriteInfoLineAsync($"❌ Stream error: {ex.GetType().Name}:\n{ex}", ConsoleColor.Red);
}
// Final sync after streaming — messages may have been consumed during the last iteration.
SyncQueuedMessageDisplay(messageInjector, session, ux, ref lastPendingMessages);
// Stop spinner before observer completions (which may prompt for input).
ux.StopSpinner();
// Close the streaming output to provide visual separation from observer output.
await ux.EndStreamingOutputAsync();
var combinedMessages = new List<ChatMessage>();
bool hasObserverMessages = false;
foreach (var observer in observers)
{
var messages = await observer.OnStreamCompleteAsync(ux, agent, session, options);
if (messages is { Count: > 0 })
{
combinedMessages.AddRange(messages);
hasObserverMessages = true;
}
}
await ux.WriteNoTextWarningAsync(hasFollowUpMessages: hasObserverMessages);
ux.EndStreaming();
nextMessages = combinedMessages.Count > 0 ? combinedMessages : null;
}
}
/// <summary>
/// Synchronizes the queued items display with the message injector's pending messages.
/// Messages that have been consumed (drained by the service) are echoed to the output
/// area as regular user-input entries.
/// </summary>
private static void SyncQueuedMessageDisplay(
MessageInjectingChatClient? messageInjector,
AgentSession session,
HarnessUXContainer ux,
ref IReadOnlyList<ChatMessage> lastPendingMessages)
{
if (messageInjector is null)
{
return;
}
var pending = messageInjector.GetPendingMessages(session);
// If previously pending messages exceed current pending count, some were consumed.
int consumedCount = lastPendingMessages.Count - pending.Count;
for (int i = 0; i < consumedCount && i < lastPendingMessages.Count; i++)
{
string text = lastPendingMessages[i].Text ?? string.Empty;
ux.WriteUserInputEcho(text);
}
lastPendingMessages = pending;
ux.ShowQueuedMessages(pending);
}
private static List<ConsoleObserver> CreateObservers(HarnessConsoleOptions options, AgentModeProvider? modeProvider, AgentSession session)
{
var observers = new List<ConsoleObserver>
{
new ToolCallDisplayObserver(),
new ToolApprovalObserver(),
new ErrorDisplayObserver(),
new ReasoningDisplayObserver(),
new UsageDisplayObserver(options.MaxContextWindowTokens, options.MaxOutputTokens),
};
if (options.EnablePlanningUx
&& modeProvider is not null
&& string.Equals(modeProvider.GetMode(session), options.PlanningModeName, StringComparison.OrdinalIgnoreCase))
{
observers.Add(new PlanningOutputObserver(modeProvider));
}
else
{
observers.Add(new TextOutputObserver());
}
return observers;
}
}
@@ -1,5 +1,11 @@
// Copyright (c) Microsoft. All rights reserved.
using System.Collections.ObjectModel;
using Harness.Shared.Console.Commands;
using Harness.Shared.Console.Observers;
using Harness.Shared.Console.ToolFormatters;
using Microsoft.Agents.AI;
namespace Harness.Shared.Console;
/// <summary>
@@ -8,45 +14,120 @@ namespace Harness.Shared.Console;
public class HarnessConsoleOptions
{
/// <summary>
/// Gets or sets the optional maximum context window size in tokens.
/// When set, token usage is displayed as a percentage of the budget.
/// Gets or sets the list of console observers that participate in the agent response
/// streaming lifecycle. Use the factory methods on this class to create common observer sets.
/// When <see langword="null"/> (the default), a default set of observers is used.
/// Set to an empty list to disable all observers.
/// </summary>
public int? MaxContextWindowTokens { get; set; }
public IReadOnlyList<ConsoleObserver>? Observers { get; set; }
/// <summary>
/// Gets or sets the optional maximum output tokens.
/// Used with <see cref="MaxContextWindowTokens"/> to show input/output budget breakdown.
/// Gets or sets the list of command handlers to check before sending user input to the agent.
/// Use <see cref="BuildDefaultCommandHandlers"/> to create the default set.
/// When <see langword="null"/> (the default), a default set of handlers is used.
/// Set to an empty list to disable all command handlers.
/// </summary>
public int? MaxOutputTokens { get; set; }
public IReadOnlyList<CommandHandler>? CommandHandlers { get; set; }
/// <summary>
/// Gets or sets a value indicating whether the planning UX is enabled.
/// When <see langword="true"/> and the agent is in the mode specified by <see cref="PlanningModeName"/>,
/// the console uses structured output to present clarification questions and approval requests
/// instead of streaming free-form text.
/// The default mode-to-color mapping used when no custom <see cref="ModeColors"/> are provided.
/// </summary>
/// <value>Defaults to <see langword="false"/>.</value>
public bool EnablePlanningUx { get; set; }
/// <summary>
/// Gets or sets the name of the agent mode that activates the planning UX.
/// Must be set when <see cref="EnablePlanningUx"/> is <see langword="true"/>.
/// </summary>
public string? PlanningModeName { get; set; }
/// <summary>
/// Gets or sets the name of the agent mode to switch to when the user approves a plan.
/// Must be set when <see cref="EnablePlanningUx"/> is <see langword="true"/>.
/// </summary>
public string? ExecutionModeName { get; set; }
public static readonly IReadOnlyDictionary<string, ConsoleColor> DefaultModeColors = new ReadOnlyDictionary<string, ConsoleColor>(
new Dictionary<string, ConsoleColor>(StringComparer.OrdinalIgnoreCase)
{
["plan"] = ConsoleColor.Cyan,
["execute"] = ConsoleColor.Green,
});
/// <summary>
/// Gets or sets a mapping of agent mode names to console colors.
/// When a mode is not found in this dictionary, the default color (<see cref="ConsoleColor.Gray"/>) is used.
/// </summary>
public Dictionary<string, ConsoleColor> ModeColors { get; set; } = new(StringComparer.OrdinalIgnoreCase)
public Dictionary<string, ConsoleColor> ModeColors { get; set; } = new(DefaultModeColors, StringComparer.OrdinalIgnoreCase);
/// <summary>
/// Creates the default set of observers without planning support.
/// Includes tool call display, tool approval, error display, reasoning display,
/// usage display, and text output.
/// </summary>
/// <param name="maxContextWindowTokens">Optional maximum context window size in tokens for usage display.</param>
/// <param name="maxOutputTokens">Optional maximum output tokens for usage display.</param>
/// <param name="toolFormatters">Optional tool call formatters. When <see langword="null"/>,
/// each observer uses the default formatters from <see cref="ToolCallFormatter.BuildDefaultToolFormatters"/>.</param>
/// <returns>A list of observers for a standard (non-planning) console session.</returns>
public static List<ConsoleObserver> BuildDefaultObservers(
int? maxContextWindowTokens = null,
int? maxOutputTokens = null,
IReadOnlyList<ToolCallFormatter>? toolFormatters = null)
{
["plan"] = ConsoleColor.Cyan,
["execute"] = ConsoleColor.Green,
};
return
[
new ToolCallDisplayObserver(toolFormatters),
new ToolApprovalObserver(toolFormatters),
new ErrorDisplayObserver(),
new ReasoningDisplayObserver(),
new UsageDisplayObserver(maxContextWindowTokens, maxOutputTokens),
new TextOutputObserver(),
];
}
/// <summary>
/// Creates the default set of observers with planning support.
/// Includes a <see cref="PlanningOutputObserver"/> instead of <see cref="TextOutputObserver"/>.
/// </summary>
/// <param name="agent">The agent, used to resolve <see cref="AgentModeProvider"/>.</param>
/// <param name="planModeName">The mode name that represents the planning mode.</param>
/// <param name="executionModeName">The mode name to switch to when the user approves a plan.</param>
/// <param name="modeColors">Optional mode-to-color mapping for display.
/// Defaults to <see cref="DefaultModeColors"/> when <see langword="null"/>.</param>
/// <param name="maxContextWindowTokens">Optional maximum context window size in tokens for usage display.</param>
/// <param name="maxOutputTokens">Optional maximum output tokens for usage display.</param>
/// <param name="toolFormatters">Optional tool call formatters. When <see langword="null"/>,
/// each observer uses the default formatters from <see cref="ToolCallFormatter.BuildDefaultToolFormatters"/>.</param>
/// <returns>A list of observers for a planning-enabled console session.</returns>
public static List<ConsoleObserver> BuildObserversWithPlanning(
AIAgent agent,
string planModeName,
string executionModeName,
IReadOnlyDictionary<string, ConsoleColor>? modeColors = null,
int? maxContextWindowTokens = null,
int? maxOutputTokens = null,
IReadOnlyList<ToolCallFormatter>? toolFormatters = null)
{
var modeProvider = agent.GetService<AgentModeProvider>()
?? throw new InvalidOperationException("Planning requires an AgentModeProvider service on the agent.");
return
[
new ToolCallDisplayObserver(toolFormatters),
new ToolApprovalObserver(toolFormatters),
new ErrorDisplayObserver(),
new ReasoningDisplayObserver(),
new UsageDisplayObserver(maxContextWindowTokens, maxOutputTokens),
new PlanningOutputObserver(modeProvider, planModeName, executionModeName, modeColors ?? DefaultModeColors),
];
}
/// <summary>
/// Creates the default set of command handlers.
/// Includes exit, todo, and mode command handlers.
/// </summary>
/// <param name="agent">The agent, used to resolve <see cref="TodoProvider"/> and <see cref="AgentModeProvider"/>.</param>
/// <param name="modeColors">Optional mode-to-color mapping for the mode command display.
/// Defaults to <see cref="DefaultModeColors"/> when <see langword="null"/>.</param>
/// <returns>A list of command handlers for a standard console session.</returns>
public static List<CommandHandler> BuildDefaultCommandHandlers(
AIAgent agent,
IReadOnlyDictionary<string, ConsoleColor>? modeColors = null)
{
var todoProvider = agent.GetService<TodoProvider>();
var modeProvider = agent.GetService<AgentModeProvider>();
return
[
new ExitCommandHandler(),
new TodoCommandHandler(todoProvider),
new ModeCommandHandler(modeProvider, modeColors ?? DefaultModeColors),
];
}
}
@@ -0,0 +1,408 @@
// Copyright (c) Microsoft. All rights reserved.
using Harness.ConsoleReactiveComponents;
using Microsoft.Extensions.AI;
namespace Harness.Shared.Console;
/// <summary>
/// Default <see cref="IUXStateDriver"/> implementation. Owned by
/// <see cref="HarnessAppComponent"/>; mutates the component's state via a
/// <c>SetState</c>-style callback. Each public operation updates state and lets
/// the component's render-skip optimization handle the actual draw.
/// </summary>
internal sealed class HarnessConsoleUXStateDriver : IUXStateDriver
{
private readonly Func<HarnessAppComponentState> _getState;
private readonly Action<HarnessAppComponentState> _setState;
private readonly Action _requestShutdown;
private readonly IReadOnlyDictionary<string, ConsoleColor>? _modeColors;
private readonly List<string> _outputItems = [];
private readonly object _stateLock = new();
private OutputEntryType? _lastEntryType;
private bool _hasReceivedAnyText;
private OutputEntry? _currentStreamingEntry;
private int _currentStreamingEntryIndex = -1;
private string? _currentMode;
/// <summary>
/// Initializes a new instance of the <see cref="HarnessConsoleUXStateDriver"/> class.
/// </summary>
/// <param name="getState">Returns the component's current state.</param>
/// <param name="setState">Replaces the component's state and triggers a re-render.</param>
/// <param name="requestShutdown">Callback invoked when a command handler requests application shutdown.</param>
/// <param name="modeColors">Optional mapping of mode names to console colors.</param>
public HarnessConsoleUXStateDriver(
Func<HarnessAppComponentState> getState,
Action<HarnessAppComponentState> setState,
Action requestShutdown,
IReadOnlyDictionary<string, ConsoleColor>? modeColors = null)
{
this._getState = getState;
this._setState = setState;
this._requestShutdown = requestShutdown;
this._modeColors = modeColors;
this._currentMode = getState().ModeText;
}
/// <inheritdoc/>
public string? CurrentMode
{
get => this._currentMode;
set
{
this.UpdateState(s =>
{
this._currentMode = value;
return s with
{
ModeColor = ModeColors.Get(value, this._modeColors),
ModeText = value,
};
});
}
}
/// <inheritdoc/>
public void BeginStreaming() =>
this.UpdateState(s => s with
{
Mode = BottomPanelMode.Streaming,
ShowSpinner = true,
});
/// <inheritdoc/>
public void StopSpinner() =>
this.UpdateState(s => s with { ShowSpinner = false });
/// <inheritdoc/>
public void EndStreaming() =>
this.UpdateState(s => s with
{
Mode = BottomPanelMode.TextInput,
ShowSpinner = false,
});
/// <inheritdoc/>
public void BeginStreamingOutput()
{
lock (this._stateLock)
{
this._hasReceivedAnyText = false;
this._currentStreamingEntry = null;
this._currentStreamingEntryIndex = -1;
}
}
/// <inheritdoc/>
public void SetUsageText(string usageText) =>
this.UpdateState(s => s with { UsageText = usageText });
/// <inheritdoc/>
public void SetQueuedMessages(IReadOnlyList<ChatMessage> pending)
{
var newQueued = new List<string>(pending.Count);
foreach (var msg in pending)
{
string text = msg.Text ?? string.Empty;
newQueued.Add(RenderEntry($" 💬 {text}\n", ConsoleColor.DarkGray));
}
this.UpdateState(s => s with { QueuedItems = newQueued });
}
/// <inheritdoc/>
public void QueueFollowUpQuestions(IReadOnlyList<FollowUpQuestion> questions)
{
if (questions.Count == 0)
{
return;
}
this.UpdateState(s =>
{
bool wasEmpty = s.PendingQuestions.Count == 0;
var combined = new List<FollowUpQuestion>(s.PendingQuestions.Count + questions.Count);
combined.AddRange(s.PendingQuestions);
combined.AddRange(questions);
HarnessAppComponentState next = s with { PendingQuestions = combined };
if (wasEmpty)
{
next = this.ConfigureForHeadQuestion(next, combined[0]);
}
return next;
});
}
/// <inheritdoc/>
public void AddFollowUpResponse(ChatMessage response)
{
this.UpdateState(s =>
{
var combined = new List<ChatMessage>(s.AccumulatedFollowUpResponses.Count + 1);
combined.AddRange(s.AccumulatedFollowUpResponses);
combined.Add(response);
return s with { AccumulatedFollowUpResponses = combined };
});
}
/// <inheritdoc/>
public void AdvanceFollowUpQuestion()
{
this.UpdateState(s =>
{
if (s.PendingQuestions.Count == 0)
{
return s;
}
var remaining = s.PendingQuestions.Skip(1).ToList();
HarnessAppComponentState next = s with { PendingQuestions = remaining };
if (remaining.Count > 0)
{
return this.ConfigureForHeadQuestion(next, remaining[0]);
}
return next with
{
Mode = BottomPanelMode.TextInput,
ListSelectionOptions = [],
ListSelectionTitle = null,
ListSelectionCustomTextPlaceholder = null,
ListSelectionIndex = 0,
ListSelectionCustomInputText = "",
};
});
}
/// <inheritdoc/>
public IReadOnlyList<ChatMessage> TakeFollowUpResponses()
{
return this.UpdateState(s =>
{
IReadOnlyList<ChatMessage> responses = s.AccumulatedFollowUpResponses;
if (responses.Count == 0)
{
return (s, responses);
}
return (s with { AccumulatedFollowUpResponses = [] }, responses);
});
}
/// <summary>
/// Configures the bottom-panel display fields on the supplied state for the
/// given head question. For text questions, also writes the prompt as an
/// info line above the input row as a side effect.
/// </summary>
private HarnessAppComponentState ConfigureForHeadQuestion(HarnessAppComponentState state, FollowUpQuestion question)
{
if (question is ChoiceFollowUpQuestion choice)
{
return state with
{
Mode = BottomPanelMode.ListSelection,
ListSelectionOptions = choice.Choices.ToList(),
ListSelectionTitle = choice.Prompt,
ListSelectionCustomTextPlaceholder = choice.AllowCustomText ? "✏️ Type a custom response..." : null,
ListSelectionIndex = 0,
ListSelectionCustomInputText = "",
};
}
// Text question — prompt is rendered as an info line above the input row.
// We append entries and capture the scroll snapshot inline so the caller's
// single _setState picks up both the new output and the UI mode change.
ConsoleColor ruleColor = ModeColors.Get(this._currentMode, this._modeColors);
List<string> scrollSnapshot = this.AppendOutputEntriesAndSnapshot(
new OutputEntry(OutputEntryType.InfoLine, "\n", ruleColor),
new OutputEntry(OutputEntryType.InfoLine, $" {question.Prompt}", ruleColor));
return state with
{
Mode = BottomPanelMode.TextInput,
ListSelectionOptions = [],
ListSelectionTitle = null,
ListSelectionCustomTextPlaceholder = null,
ListSelectionIndex = 0,
ListSelectionCustomInputText = "",
ScrollAreaContentItems = scrollSnapshot,
};
}
/// <inheritdoc/>
public void WriteUserInputEcho(string text)
{
this.UpdateState(s =>
{
List<string> snapshot = this.AppendOutputEntriesAndSnapshot(new OutputEntry(
OutputEntryType.UserInput,
$"\nYou: {text}\n\n",
ConsoleColor.Green));
return s with { ScrollAreaContentItems = snapshot };
});
}
/// <inheritdoc/>
public Task WriteInfoAsync(string text, ConsoleColor? color = null) =>
this.WriteInfoCoreAsync(text, color, newLine: false);
/// <inheritdoc/>
public Task WriteInfoLineAsync(string text, ConsoleColor? color = null) =>
this.WriteInfoCoreAsync(text, color, newLine: true);
private Task WriteInfoCoreAsync(string text, ConsoleColor? color, bool newLine)
{
this.UpdateState(s =>
{
// Add a blank line separator when transitioning from streaming text or user input.
string prefix = this._lastEntryType is OutputEntryType.StreamingText or OutputEntryType.StreamFooter
? "\n "
: " ";
string fullText = newLine ? prefix + text + "\n\n" : prefix + text;
List<string> snapshot = this.AppendOutputEntriesAndSnapshot(new OutputEntry(
OutputEntryType.InfoLine,
fullText,
color ?? ModeColors.Get(this._currentMode, this._modeColors)));
return s with { ScrollAreaContentItems = snapshot };
});
return Task.CompletedTask;
}
/// <inheritdoc/>
public Task WriteTextAsync(string text, ConsoleColor? color = null)
{
this.UpdateState(s =>
{
this._lastEntryType = OutputEntryType.StreamingText;
this._hasReceivedAnyText = true;
ConsoleColor effectiveColor = color ?? ModeColors.Get(this._currentMode, this._modeColors);
if (this._currentStreamingEntry is not null
&& this._currentStreamingEntryIndex == this._outputItems.Count - 1)
{
// The streaming entry is still the last item — safe to replace in place.
this._currentStreamingEntry = this._currentStreamingEntry with
{
Text = this._currentStreamingEntry.Text + text,
};
this._outputItems[^1] = RenderEntry(this._currentStreamingEntry.Text, this._currentStreamingEntry.Color);
}
else
{
// Either the first text delta or other entries (tool calls, info lines)
// were appended after the previous streaming entry — start a fresh one.
const string Prefix = "\n";
this._currentStreamingEntry = new OutputEntry(OutputEntryType.StreamingText, Prefix + text, effectiveColor);
this._outputItems.Add(RenderEntry(this._currentStreamingEntry.Text, this._currentStreamingEntry.Color));
this._currentStreamingEntryIndex = this._outputItems.Count - 1;
}
return s with { ScrollAreaContentItems = new List<string>(this._outputItems) };
});
return Task.CompletedTask;
}
/// <inheritdoc/>
public Task EndStreamingOutputAsync()
{
this.UpdateState(s =>
{
if (this._hasReceivedAnyText)
{
this._outputItems.Add(RenderEntry("\n", null));
this._currentStreamingEntry = null;
this._lastEntryType = OutputEntryType.StreamFooter;
return s with { ScrollAreaContentItems = new List<string>(this._outputItems) };
}
return s;
});
return Task.CompletedTask;
}
/// <inheritdoc/>
public Task WriteNoTextWarningAsync(bool hasFollowUpActions)
{
if (!this._hasReceivedAnyText && !hasFollowUpActions)
{
this.UpdateState(s =>
{
List<string> snapshot = this.AppendOutputEntriesAndSnapshot(new OutputEntry(
OutputEntryType.StreamFooter,
" (no text response from agent)\n",
ConsoleColor.DarkYellow));
return s with { ScrollAreaContentItems = snapshot };
});
}
return Task.CompletedTask;
}
/// <summary>
/// Wraps the supplied text with ANSI foreground color escape sequences (or returns
/// the text unchanged when no color is specified). Output is appended to
/// <see cref="_outputItems"/> and consumed verbatim by <see cref="TextScrollPanel"/>
/// and <see cref="TextPanel"/>.
/// </summary>
private static string RenderEntry(string text, ConsoleColor? color) =>
color.HasValue
? $"{AnsiEscapes.SetForegroundColor(color.Value)}{text}{AnsiEscapes.ResetAttributes}"
: text;
private void UpdateState(Func<HarnessAppComponentState, HarnessAppComponentState> update)
{
lock (this._stateLock)
{
this._setState(update(this._getState()));
}
}
private T UpdateState<T>(Func<HarnessAppComponentState, (HarnessAppComponentState State, T Result)> update)
{
lock (this._stateLock)
{
var (newState, result) = update(this._getState());
this._setState(newState);
return result;
}
}
/// <summary>
/// Appends one or more output entries to the output list, updates
/// <see cref="_lastEntryType"/> to the last entry's type, and returns a
/// snapshot of <see cref="_outputItems"/>. Must be called inside a locked
/// context (e.g. within an <see cref="UpdateState"/> callback).
/// </summary>
private List<string> AppendOutputEntriesAndSnapshot(params OutputEntry[] entries)
{
this.AppendOutputEntriesCore(entries);
return new List<string>(this._outputItems);
}
private void AppendOutputEntriesCore(OutputEntry[] entries)
{
foreach (OutputEntry entry in entries)
{
this._outputItems.Add(RenderEntry(entry.Text, entry.Color));
}
if (entries.Length > 0)
{
this._lastEntryType = entries[^1].Type;
}
}
/// <inheritdoc/>
public void RequestShutdown() => this._requestShutdown();
}
@@ -1,478 +0,0 @@
// Copyright (c) Microsoft. All rights reserved.
using Harness.ConsoleReactiveComponents;
using Microsoft.Extensions.AI;
namespace Harness.Shared.Console;
/// <summary>
/// Event arguments raised when the user submits text while the bottom panel is in
/// streaming mode (i.e. an agent turn is in progress).
/// </summary>
public sealed class StreamingInputReceivedEventArgs : EventArgs
{
/// <summary>
/// Initializes a new instance of the <see cref="StreamingInputReceivedEventArgs"/> class.
/// </summary>
/// <param name="text">The submitted text.</param>
public StreamingInputReceivedEventArgs(string text)
{
this.Text = text;
}
/// <summary>
/// Gets the submitted text.
/// </summary>
public string Text { get; }
}
/// <summary>
/// Façade over the harness UI: owns the <see cref="HarnessAppComponent"/>, manages
/// its props, dispatches input submissions, and provides the high-level read/write
/// operations used by observers, command handlers, and the harness loop.
/// </summary>
/// <remarks>
/// All callers interact with the UI exclusively through this class. The underlying
/// <see cref="HarnessAppComponent"/> and its props are an implementation detail and
/// must not be exposed.
/// </remarks>
public sealed class HarnessUXContainer : IDisposable
{
/// <summary>
/// The prompt displayed in the bottom-panel input area.
/// </summary>
private const string UserPrompt = "> ";
private readonly IReadOnlyDictionary<string, ConsoleColor>? _modeColors;
private readonly List<object> _outputItems = [];
private readonly HarnessAppComponent _appComponent;
private readonly object _outputLock = new();
private TaskCompletionSource<string>? _pendingInputTcs;
private OutputEntryType? _lastEntryType;
private bool _hasReceivedAnyText;
private OutputEntry? _currentStreamingEntry;
private string? _currentMode;
/// <summary>
/// Initializes a new instance of the <see cref="HarnessUXContainer"/> class.
/// </summary>
/// <param name="placeholder">Placeholder text shown when the input is empty.</param>
/// <param name="initialMode">The current agent mode, used to colour the rule and prompt.</param>
/// <param name="inputEnabled">Whether the bottom-panel input accepts keystrokes during streaming.</param>
/// <param name="modeColors">Optional mapping of mode names to console colors.</param>
public HarnessUXContainer(
string placeholder,
string? initialMode,
bool inputEnabled,
IReadOnlyDictionary<string, ConsoleColor>? modeColors = null)
{
this._modeColors = modeColors;
this._currentMode = initialMode;
this._appComponent = new HarnessAppComponent(RenderOutputEntry)
{
Props = new HarnessAppComponentProps
{
ScrollItems = this._outputItems,
Mode = BottomPanelMode.TextInput,
Prompt = UserPrompt,
Placeholder = placeholder,
ModeColor = ModeColors.Get(initialMode, modeColors),
ModeText = initialMode,
InputEnabled = inputEnabled,
},
};
this._appComponent.InputSubmitted += this.OnInputSubmitted;
}
/// <summary>
/// Raised when the user submits text while the bottom panel is in streaming mode.
/// Subscribers typically enqueue the text into a message-injecting chat client.
/// </summary>
public event EventHandler<StreamingInputReceivedEventArgs>? StreamingInputReceived;
/// <summary>
/// Gets or sets the current agent mode (e.g. "plan", "execute"). Updating this
/// also refreshes the rule colour and bottom-panel prompt to match the new mode.
/// </summary>
public string? CurrentMode
{
get => this._currentMode;
set
{
this._currentMode = value;
this._appComponent.Props = this._appComponent.Props! with
{
ModeColor = ModeColors.Get(value, this._modeColors),
ModeText = value,
};
this._appComponent.Render();
}
}
/// <summary>
/// Performs the initial screen clear, sets the help text in the mode-and-help bar,
/// and adds the title to the output area.
/// </summary>
/// <param name="title">The title displayed in the console header.</param>
/// <param name="commandHelpTexts">The command help strings displayed in the mode-and-help bar.</param>
/// <param name="messageInjectionActive">Whether streaming-time message injection is enabled.</param>
public void Initialize(string title, IEnumerable<string> commandHelpTexts, bool messageInjectionActive)
{
// Set the help text on the mode-and-help bar (persists below the rule).
this._appComponent.Props = this._appComponent.Props! with
{
HelpText = string.Join(", ", commandHelpTexts),
ModeText = this._currentMode,
};
System.Console.Write(AnsiEscapes.EraseEntireScreen);
System.Console.Write(AnsiEscapes.EraseScrollbackBuffer);
this._appComponent.Render();
this.AppendOutputEntries(
new OutputEntry(OutputEntryType.InfoLine, $"=== {title} ===\n", ConsoleColor.White),
new OutputEntry(OutputEntryType.InfoLine, "\n"));
}
/// <summary>
/// Restores the cursor and exits the alternate screen, ending the interactive UI.
/// </summary>
public void Deactivate() => this._appComponent.Deactivate();
/// <summary>
/// Switches the bottom panel to streaming mode and starts the spinner.
/// </summary>
public void BeginStreaming()
{
this._appComponent.Props = this._appComponent.Props! with
{
Mode = BottomPanelMode.Streaming,
ShowSpinner = true,
};
this._appComponent.Render();
}
/// <summary>
/// Stops the spinner without leaving streaming mode. Use between the end of the
/// stream and any observer-driven prompts (e.g. tool approvals).
/// </summary>
public void StopSpinner()
{
this._appComponent.Props = this._appComponent.Props! with { ShowSpinner = false };
this._appComponent.Render();
}
/// <summary>
/// Switches the bottom panel back to text-input mode and stops the spinner.
/// </summary>
public void EndStreaming()
{
this._appComponent.Props = this._appComponent.Props! with
{
Mode = BottomPanelMode.TextInput,
ShowSpinner = false,
};
this._appComponent.Render();
}
/// <summary>
/// Resets per-turn streaming bookkeeping in preparation for a new agent turn.
/// </summary>
public void BeginStreamingOutput()
{
this._hasReceivedAnyText = false;
this._currentStreamingEntry = null;
}
/// <summary>
/// Sets the formatted usage text shown on the agent status bar.
/// </summary>
public void SetUsageText(string usageText)
{
this._appComponent.Props = this._appComponent.Props! with { UsageText = usageText };
this._appComponent.Render();
}
/// <summary>
/// Clears the usage text from the agent status bar.
/// </summary>
public void ClearUsageText()
{
this._appComponent.Props = this._appComponent.Props! with { UsageText = null };
this._appComponent.Render();
}
/// <summary>
/// Replaces the queued-message display with one entry per pending message.
/// </summary>
public void ShowQueuedMessages(IReadOnlyList<ChatMessage> pending)
{
var newQueued = new List<object>(pending.Count);
foreach (var msg in pending)
{
string text = msg.Text ?? string.Empty;
newQueued.Add(new OutputEntry(OutputEntryType.UserInput, $" 💬 {text}\n", ConsoleColor.DarkGray));
}
this._appComponent.Props = this._appComponent.Props! with { QueuedItems = newQueued };
this._appComponent.Render();
}
/// <summary>
/// Echoes a submitted user input as a regular user-input entry in the output area,
/// using the current mode-aware prompt prefix.
/// </summary>
/// <param name="text">The user-entered text.</param>
public void WriteUserInputEcho(string text)
{
this.AppendOutputEntries(new OutputEntry(
OutputEntryType.UserInput,
$"\nYou: {text}\n",
ConsoleColor.Green));
}
/// <summary>
/// Writes informational output as an output entry, without a trailing newline.
/// </summary>
public Task WriteInfoAsync(string text, ConsoleColor? color = null) =>
this.WriteInfoCoreAsync(text, color, newLine: false);
/// <summary>
/// Writes informational output as an output entry, followed by a newline.
/// </summary>
public Task WriteInfoLineAsync(string text, ConsoleColor? color = null) =>
this.WriteInfoCoreAsync(text, color, newLine: true);
private Task WriteInfoCoreAsync(string text, ConsoleColor? color, bool newLine)
{
// Add a blank line separator when transitioning from streaming text or user input.
string prefix = this._lastEntryType is OutputEntryType.StreamingText or OutputEntryType.StreamFooter
? "\n\n "
: " ";
string fullText = newLine ? prefix + text + "\n" : prefix + text;
this.AppendOutputEntries(new OutputEntry(
OutputEntryType.InfoLine,
fullText,
color ?? ModeColors.Get(this.CurrentMode, this._modeColors)));
return Task.CompletedTask;
}
/// <summary>
/// Writes streaming text output from the agent. Successive calls accumulate into a
/// single streaming entry that is re-rendered by the text panel.
/// </summary>
public Task WriteTextAsync(string text, ConsoleColor? color = null)
{
lock (this._outputLock)
{
this._lastEntryType = OutputEntryType.StreamingText;
this._hasReceivedAnyText = true;
ConsoleColor effectiveColor = color ?? ModeColors.Get(this.CurrentMode, this._modeColors);
if (this._currentStreamingEntry is not null)
{
this._currentStreamingEntry = this._currentStreamingEntry with
{
Text = this._currentStreamingEntry.Text + text,
};
this._outputItems[^1] = this._currentStreamingEntry;
}
else
{
const string Prefix = "\n";
this._currentStreamingEntry = new OutputEntry(OutputEntryType.StreamingText, Prefix + text, effectiveColor);
this._outputItems.Add(this._currentStreamingEntry);
}
this._appComponent.Props = this._appComponent.Props! with
{
ScrollItems = new List<object>(this._outputItems),
};
}
this._appComponent.Render();
return Task.CompletedTask;
}
/// <summary>
/// Writes a blank-line separator to visually close the streaming output section.
/// Call before observer completions so their output is visually separated.
/// </summary>
public Task EndStreamingOutputAsync()
{
lock (this._outputLock)
{
this._outputItems.Add(new OutputEntry(OutputEntryType.StreamFooter, "\n"));
this._currentStreamingEntry = null;
this._lastEntryType = OutputEntryType.StreamFooter;
this._appComponent.Props = this._appComponent.Props! with
{
ScrollItems = new List<object>(this._outputItems),
};
}
this._appComponent.Render();
return Task.CompletedTask;
}
/// <summary>
/// Shows a "(no text response from agent)" warning if no text was received
/// and no observer produced follow-up messages. Call after observer completions.
/// </summary>
/// <param name="hasFollowUpMessages">Whether any observer produced follow-up messages.</param>
public Task WriteNoTextWarningAsync(bool hasFollowUpMessages)
{
if (!this._hasReceivedAnyText && !hasFollowUpMessages)
{
this.AppendOutputEntries(new OutputEntry(
OutputEntryType.StreamFooter,
" (no text response from agent)\n",
ConsoleColor.DarkYellow));
}
return Task.CompletedTask;
}
/// <summary>
/// Reads a line of input from the user. If <paramref name="prompt"/> is supplied
/// it is rendered as an info line above the input row before reading.
/// </summary>
public async Task<string?> ReadLineAsync(string? prompt = null, ConsoleColor? promptColor = null)
{
if (prompt is not null)
{
ConsoleColor ruleColor = ModeColors.Get(this.CurrentMode, this._modeColors);
this.AppendOutputEntries(
new OutputEntry(OutputEntryType.InfoLine, "\n", ruleColor),
new OutputEntry(OutputEntryType.InfoLine, $" {prompt}", promptColor ?? ruleColor));
}
this._appComponent.Props = this._appComponent.Props! with { Mode = BottomPanelMode.TextInput };
this._appComponent.Render();
string input = await this.WaitForInputAsync();
this.AppendOutputEntries(new OutputEntry(
OutputEntryType.UserInput,
$"\nYou: {input}\n",
ConsoleColor.Green));
return input;
}
/// <summary>
/// Presents a selection prompt with the given choices and waits for the user's
/// selection. The title is displayed above the list in the bottom panel. After
/// selection the bottom panel is restored to text-input mode and both the question
/// and selection are echoed in the output area.
/// </summary>
public async Task<string> ReadSelectionAsync(string title, IList<string> choices)
{
this._appComponent.Props = this._appComponent.Props! with
{
Mode = BottomPanelMode.ListSelection,
Items = choices.ToList(),
ListTitle = title,
ListCustomTextPlaceholder = "✏️ Type a custom response...",
};
this._appComponent.Render();
string selection = await this.WaitForInputAsync();
this._appComponent.Props = this._appComponent.Props with { Mode = BottomPanelMode.TextInput };
this.AppendOutputEntries(
new OutputEntry(
OutputEntryType.InfoLine,
$"\n {title}\n",
ModeColors.Get(this.CurrentMode, this._modeColors)),
new OutputEntry(
OutputEntryType.UserInput,
$"\nYou: {selection}\n",
ConsoleColor.Green));
return selection;
}
/// <summary>
/// Awaits the next non-streaming user input submission.
/// </summary>
public Task<string> WaitForInputAsync()
{
this._pendingInputTcs = new TaskCompletionSource<string>(TaskCreationOptions.RunContinuationsAsynchronously);
return this._pendingInputTcs.Task;
}
private void OnInputSubmitted(object? sender, InputSubmittedEventArgs e)
{
if (e.Mode == BottomPanelMode.Streaming)
{
this.StreamingInputReceived?.Invoke(this, new StreamingInputReceivedEventArgs(e.Text));
}
else
{
var waiter = this._pendingInputTcs;
this._pendingInputTcs = null;
waiter?.TrySetResult(e.Text);
}
}
/// <inheritdoc/>
public void Dispose()
{
this._appComponent.InputSubmitted -= this.OnInputSubmitted;
this._appComponent.Deactivate();
this._appComponent.Dispose();
}
/// <summary>
/// Renders an <see cref="OutputEntry"/> to a string with ANSI color codes.
/// Used as the render delegate for the <see cref="HarnessAppComponent"/>.
/// </summary>
private static string RenderOutputEntry(object item)
{
if (item is not OutputEntry entry)
{
return item?.ToString() ?? string.Empty;
}
if (entry.Color.HasValue)
{
return $"{AnsiEscapes.SetForegroundColor(entry.Color.Value)}{entry.Text}{AnsiEscapes.ResetAttributes}";
}
return entry.Text;
}
/// <summary>
/// Appends one or more output entries to the output list under lock,
/// updates <see cref="_lastEntryType"/> to the last entry's type, and renders.
/// </summary>
private void AppendOutputEntries(params OutputEntry[] entries)
{
lock (this._outputLock)
{
foreach (OutputEntry entry in entries)
{
this._outputItems.Add(entry);
}
if (entries.Length > 0)
{
this._lastEntryType = entries[^1].Type;
}
this._appComponent.Props = this._appComponent.Props! with
{
ScrollItems = new List<object>(this._outputItems),
};
}
this._appComponent.Render();
}
}
@@ -0,0 +1,120 @@
// Copyright (c) Microsoft. All rights reserved.
using Microsoft.Extensions.AI;
namespace Harness.Shared.Console;
/// <summary>
/// Abstraction over the harness UI state. All callers (observers, command handlers,
/// the agent runner) interact with the UI exclusively through this interface, which
/// internally translates each operation into a <c>SetState</c> call on the underlying
/// reactive component.
/// </summary>
/// <remarks>
/// This interface is intentionally narrow: it does not expose blocking input methods.
/// The agent runner orchestrates input flow via <see cref="FollowUpQuestion"/>
/// objects returned from observers.
/// </remarks>
public interface IUXStateDriver
{
/// <summary>
/// Gets or sets the current agent mode (e.g. "plan", "execute"). Setting also
/// refreshes the rule colour and bottom-panel prompt to match the new mode.
/// </summary>
string? CurrentMode { get; set; }
/// <summary>
/// Echoes a submitted user input as a regular user-input entry in the output area.
/// </summary>
void WriteUserInputEcho(string text);
/// <summary>
/// Writes informational output as an output entry, without a trailing newline.
/// </summary>
Task WriteInfoAsync(string text, ConsoleColor? color = null);
/// <summary>
/// Writes informational output as an output entry, followed by a newline.
/// </summary>
Task WriteInfoLineAsync(string text, ConsoleColor? color = null);
/// <summary>
/// Writes streaming text output from the agent. Successive calls accumulate into a
/// single streaming entry that is re-rendered by the text panel.
/// </summary>
Task WriteTextAsync(string text, ConsoleColor? color = null);
/// <summary>
/// Writes a blank-line separator to visually close the streaming output section.
/// </summary>
Task EndStreamingOutputAsync();
/// <summary>
/// Shows a "(no text response from agent)" warning if no text was received
/// and no observer produced follow-up actions.
/// </summary>
Task WriteNoTextWarningAsync(bool hasFollowUpActions);
/// <summary>
/// Switches the bottom panel to streaming mode and starts the spinner.
/// </summary>
void BeginStreaming();
/// <summary>
/// Stops the spinner without leaving streaming mode.
/// </summary>
void StopSpinner();
/// <summary>
/// Switches the bottom panel back to text-input mode and stops the spinner.
/// </summary>
void EndStreaming();
/// <summary>
/// Resets per-turn streaming bookkeeping in preparation for a new agent turn.
/// </summary>
void BeginStreamingOutput();
/// <summary>
/// Sets the formatted usage text shown on the agent status bar.
/// </summary>
void SetUsageText(string usageText);
/// <summary>
/// Replaces the queued-message display with one entry per pending message.
/// </summary>
void SetQueuedMessages(IReadOnlyList<ChatMessage> pending);
/// <summary>
/// Appends the supplied questions to the pending follow-up question queue in
/// component state. If the queue was empty, the bottom-panel display is
/// reconfigured to present the new head question.
/// </summary>
void QueueFollowUpQuestions(IReadOnlyList<FollowUpQuestion> questions);
/// <summary>
/// Appends a message to the accumulated follow-up response list in component state.
/// Called by the runner for direct <see cref="FollowUpMessage"/> outputs and by
/// the component when a question's continuation produces a response.
/// </summary>
void AddFollowUpResponse(ChatMessage response);
/// <summary>
/// Pops the head of the pending follow-up question queue. Reconfigures the
/// bottom-panel display for the new head, or restores the default text-input
/// mode if the queue is now empty.
/// </summary>
void AdvanceFollowUpQuestion();
/// <summary>
/// Returns the current accumulated follow-up responses and clears them in state.
/// Called by the runner immediately before invoking the next agent turn.
/// </summary>
IReadOnlyList<ChatMessage> TakeFollowUpResponses();
/// <summary>
/// Signals that the application should shut down. Completes the shutdown task
/// on the owning component.
/// </summary>
void RequestShutdown();
}
@@ -18,36 +18,41 @@ public abstract class ConsoleObserver
/// Override to set options such as <see cref="AgentRunOptions.ResponseFormat"/>.
/// </summary>
/// <param name="options">The run options to configure.</param>
public virtual void ConfigureRunOptions(AgentRunOptions options)
/// <param name="agent">The agent being interacted with.</param>
/// <param name="session">The current agent session.</param>
public virtual void ConfigureRunOptions(AgentRunOptions options, AIAgent agent, AgentSession session)
{
}
/// <summary>
/// Called for each <see cref="AIContent"/> item in the response stream.
/// </summary>
/// <param name="ux">The harness UX container, used for rendering output and interacting with the user.</param>
/// <param name="ux">The UX state driver, used for rendering output.</param>
/// <param name="content">The content item from the stream.</param>
public virtual Task OnContentAsync(HarnessUXContainer ux, AIContent content) => Task.CompletedTask;
/// <param name="agent">The agent being interacted with.</param>
/// <param name="session">The current agent session.</param>
public virtual Task OnContentAsync(IUXStateDriver ux, AIContent content, AIAgent agent, AgentSession session) => Task.CompletedTask;
/// <summary>
/// Called for each text update in the response stream.
/// </summary>
/// <param name="ux">The harness UX container, used for rendering output and interacting with the user.</param>
/// <param name="ux">The UX state driver, used for rendering output.</param>
/// <param name="text">The text from the update.</param>
public virtual Task OnTextAsync(HarnessUXContainer ux, string text) => Task.CompletedTask;
/// <summary>
/// Called after the response stream completes. Returns messages to include in the
/// next agent invocation, or <see langword="null"/> if no re-invocation is needed.
/// </summary>
/// <param name="ux">The harness UX container, used for rendering output and interacting with the user.</param>
/// <param name="agent">The agent being interacted with.</param>
/// <param name="session">The current agent session.</param>
/// <param name="options">The console options.</param>
/// <returns>Messages to send to the agent, or <see langword="null"/> if no action is needed.</returns>
public virtual Task<IList<ChatMessage>?> OnStreamCompleteAsync(
HarnessUXContainer ux,
public virtual Task OnTextAsync(IUXStateDriver ux, string text, AIAgent agent, AgentSession session) => Task.CompletedTask;
/// <summary>
/// Called after the response stream completes. Returns a heterogeneous list of
/// follow-up actions (questions to ask the user, and/or messages to add directly to
/// the next agent invocation), or <see langword="null"/> if no follow-up is needed.
/// </summary>
/// <param name="ux">The UX state driver, used for rendering output.</param>
/// <param name="agent">The agent being interacted with.</param>
/// <param name="session">The current agent session.</param>
/// <returns>Follow-up actions to process after the stream completes, or <see langword="null"/>.</returns>
public virtual Task<IList<FollowUpAction>?> OnStreamCompleteAsync(
IUXStateDriver ux,
AIAgent agent,
AgentSession session,
HarnessConsoleOptions options) => Task.FromResult<IList<ChatMessage>?>(null);
AgentSession session) => Task.FromResult<IList<FollowUpAction>?>(null);
}
@@ -1,5 +1,6 @@
// Copyright (c) Microsoft. All rights reserved.
using Microsoft.Agents.AI;
using Microsoft.Extensions.AI;
namespace Harness.Shared.Console.Observers;
@@ -7,10 +8,10 @@ namespace Harness.Shared.Console.Observers;
/// <summary>
/// Displays error content (❌) from the response stream.
/// </summary>
internal sealed class ErrorDisplayObserver : ConsoleObserver
public sealed class ErrorDisplayObserver : ConsoleObserver
{
/// <inheritdoc/>
public override async Task OnContentAsync(HarnessUXContainer ux, AIContent content)
public override async Task OnContentAsync(IUXStateDriver ux, AIContent content, AIAgent agent, AgentSession session)
{
if (content is ErrorContent errorContent)
{
@@ -2,51 +2,77 @@
using System.Text;
using System.Text.Json;
using Harness.ConsoleReactiveComponents;
using Microsoft.Agents.AI;
using Microsoft.Extensions.AI;
namespace Harness.Shared.Console.Observers;
/// <summary>
/// Planning observer that configures structured output, collects streamed text,
/// and deserializes it as a <see cref="PlanningResponse"/>. Renders clarification
/// questions and approval prompts, and manages mode switching when the user approves a plan.
/// Planning observer that is mode-aware: in planning mode it configures structured
/// JSON output, collects streamed text, and deserializes it as a <see cref="PlanningResponse"/>;
/// in execution mode it passes text straight through to <see cref="IUXStateDriver.WriteTextAsync"/>
/// for live streaming display.
/// </summary>
internal sealed class PlanningOutputObserver : ConsoleObserver
public sealed class PlanningOutputObserver : ConsoleObserver
{
private readonly StringBuilder _textCollector = new();
private readonly AgentModeProvider _modeProvider;
private readonly string _planModeName;
private readonly string _executionModeName;
private readonly IReadOnlyDictionary<string, ConsoleColor>? _modeColors;
/// <summary>
/// Initializes a new instance of the <see cref="PlanningOutputObserver"/> class.
/// </summary>
/// <param name="modeProvider">The mode provider for switching modes on approval.</param>
public PlanningOutputObserver(AgentModeProvider modeProvider)
/// <param name="planModeName">The mode name that represents the planning mode.</param>
/// <param name="executionModeName">The mode name to switch to when the user approves a plan.</param>
/// <param name="modeColors">Optional mode-to-color mapping for display.</param>
public PlanningOutputObserver(AgentModeProvider modeProvider, string planModeName, string executionModeName, IReadOnlyDictionary<string, ConsoleColor>? modeColors = null)
{
this._modeProvider = modeProvider;
this._planModeName = planModeName;
this._executionModeName = executionModeName;
this._modeColors = modeColors;
}
/// <inheritdoc/>
public override void ConfigureRunOptions(AgentRunOptions options)
public override void ConfigureRunOptions(AgentRunOptions options, AIAgent agent, AgentSession session)
{
options.ResponseFormat = ChatResponseFormat.ForJsonSchema<PlanningResponse>();
if (this.IsPlanningMode(this._modeProvider.GetMode(session)))
{
options.ResponseFormat = ChatResponseFormat.ForJsonSchema<PlanningResponse>();
}
}
/// <inheritdoc/>
public override Task OnTextAsync(HarnessUXContainer ux, string text)
public override Task OnTextAsync(IUXStateDriver ux, string text, AIAgent agent, AgentSession session)
{
// Collect text silently instead of displaying it.
this._textCollector.Append(text);
return Task.CompletedTask;
if (this.IsPlanningMode(ux.CurrentMode))
{
// Planning mode: collect text silently for JSON parsing after the stream.
this._textCollector.Append(text);
return Task.CompletedTask;
}
// Execution mode: stream text directly to the console.
return ux.WriteTextAsync(text);
}
/// <inheritdoc/>
public override async Task<IList<ChatMessage>?> OnStreamCompleteAsync(
HarnessUXContainer ux,
public override async Task<IList<FollowUpAction>?> OnStreamCompleteAsync(
IUXStateDriver ux,
AIAgent agent,
AgentSession session,
HarnessConsoleOptions options)
AgentSession session)
{
if (!this.IsPlanningMode(ux.CurrentMode))
{
// Execution mode: text was already streamed live; nothing to parse.
this._textCollector.Clear();
return null;
}
// Read collected text from our stream observation.
string collectedText = this._textCollector.ToString();
this._textCollector.Clear();
@@ -75,10 +101,9 @@ internal sealed class PlanningOutputObserver : ConsoleObserver
return null;
}
// Render based on response type.
if (planningResponse.Type == PlanningResponseType.Clarification)
{
return AsUserMessages(await this.RenderClarificationsAndCollectResponsesAsync(ux, planningResponse));
return BuildClarificationActions(planningResponse);
}
if (planningResponse.Type == PlanningResponseType.Approval)
@@ -90,67 +115,87 @@ internal sealed class PlanningOutputObserver : ConsoleObserver
return null;
}
string response = await this.RenderApprovalAndCollectResponseAsync(ux, question, options);
if (response == "Approved")
{
this._modeProvider.SetMode(session, options.ExecutionModeName!);
await ux.WriteInfoLineAsync($"✅ Switched to {options.ExecutionModeName} mode.",
ModeColors.Get(options.ExecutionModeName, options.ModeColors));
}
return AsUserMessages(response);
return new List<FollowUpAction> { this.BuildApprovalAction(question, session) };
}
await ux.WriteInfoLineAsync($"(unexpected response type: {planningResponse.Type})", ConsoleColor.DarkYellow);
return null;
}
private static IList<ChatMessage>? AsUserMessages(string? text) =>
text is not null ? [new ChatMessage(ChatRole.User, text)] : null;
private async Task<string?> RenderClarificationsAndCollectResponsesAsync(HarnessUXContainer ux, PlanningResponse response)
private static List<FollowUpAction> BuildClarificationActions(PlanningResponse response)
{
var answers = new List<string>();
var actions = new List<FollowUpAction>(response.Questions.Count);
foreach (var question in response.Questions)
{
string? answer;
string prompt = question.Message;
async Task<ChatMessage?> Continuation(string answer, IUXStateDriver ux)
{
if (string.IsNullOrWhiteSpace(answer))
{
string noAnswer = $"🔹 {prompt}\n └─ {AnsiEscapes.SetForegroundColor(ConsoleColor.DarkGray)}(no answer){AnsiEscapes.ResetAttributes}";
await ux.WriteInfoLineAsync(noAnswer, ConsoleColor.Gray).ConfigureAwait(false);
return null;
}
string formatted = $"🔹 {prompt}\n └─ {AnsiEscapes.SetForegroundColor(ConsoleColor.Green)}{answer}{AnsiEscapes.ResetAttributes}";
await ux.WriteInfoLineAsync(formatted, ConsoleColor.Gray).ConfigureAwait(false);
return new ChatMessage(ChatRole.User, $"Q: {prompt}\nA: {answer}");
}
if (question.Choices is { Count: > 0 })
{
answer = await ux.ReadSelectionAsync(
question.Message,
question.Choices);
actions.Add(new ChoiceFollowUpQuestion(
Prompt: prompt,
Choices: question.Choices,
AllowCustomText: true,
Continuation: Continuation));
}
else
{
answer = (await ux.ReadLineAsync(question.Message))?.Trim();
}
if (!string.IsNullOrWhiteSpace(answer))
{
answers.Add($"Q: {question.Message}\nA: {answer}");
actions.Add(new TextFollowUpQuestion(
Prompt: prompt,
Continuation: Continuation));
}
}
return answers.Count > 0 ? string.Join("\n\n", answers) : null;
return actions;
}
private async Task<string> RenderApprovalAndCollectResponseAsync(HarnessUXContainer ux, PlanningQuestion question, HarnessConsoleOptions options)
private ChoiceFollowUpQuestion BuildApprovalAction(PlanningQuestion question, AgentSession session)
{
var choices = new List<string>
{
"Approve and switch to execute mode",
};
const string ApproveOption = "Approve and switch to execute mode";
var choices = new List<string> { ApproveOption };
string selection = await ux.ReadSelectionAsync(question.Message, choices);
return new ChoiceFollowUpQuestion(
Prompt: question.Message,
Choices: choices,
AllowCustomText: true,
Continuation: async (selection, ux) =>
{
string formatted = $"🔹 {question.Message}\n └─ {AnsiEscapes.SetForegroundColor(ConsoleColor.Green)}{selection}{AnsiEscapes.ResetAttributes}";
await ux.WriteInfoLineAsync(formatted, ConsoleColor.Gray).ConfigureAwait(false);
if (selection == choices[0])
{
return "Approved";
}
if (selection == ApproveOption)
{
this._modeProvider.SetMode(session, this._executionModeName);
await ux.WriteInfoLineAsync(
$"✅ Switched to {this._executionModeName} mode.",
ModeColors.Get(this._executionModeName, this._modeColors)).ConfigureAwait(false);
return new ChatMessage(ChatRole.User, "Approved");
}
// Custom freeform input — treat as suggested changes.
return selection;
// Custom freeform input — treat as suggested changes.
return new ChatMessage(ChatRole.User, selection);
});
}
/// <summary>
/// Returns <see langword="true"/> when the current mode matches the configured plan mode name.
/// A <see langword="null"/> mode (no mode provider) is also treated as planning mode.
/// </summary>
private bool IsPlanningMode(string? currentMode) =>
currentMode is null || string.Equals(currentMode, this._planModeName, StringComparison.OrdinalIgnoreCase);
}
@@ -1,5 +1,6 @@
// Copyright (c) Microsoft. All rights reserved.
using Microsoft.Agents.AI;
using Microsoft.Extensions.AI;
namespace Harness.Shared.Console.Observers;
@@ -7,10 +8,10 @@ namespace Harness.Shared.Console.Observers;
/// <summary>
/// Displays reasoning content in dark magenta from the response stream.
/// </summary>
internal sealed class ReasoningDisplayObserver : ConsoleObserver
public sealed class ReasoningDisplayObserver : ConsoleObserver
{
/// <inheritdoc/>
public override async Task OnContentAsync(HarnessUXContainer ux, AIContent content)
public override async Task OnContentAsync(IUXStateDriver ux, AIContent content, AIAgent agent, AgentSession session)
{
if (content is TextReasoningContent reasoning && !string.IsNullOrEmpty(reasoning.Text))
{
@@ -1,15 +1,17 @@
// Copyright (c) Microsoft. All rights reserved.
using Microsoft.Agents.AI;
namespace Harness.Shared.Console.Observers;
/// <summary>
/// Streams agent text output directly to the console.
/// Used in normal (non-planning) mode.
/// </summary>
internal sealed class TextOutputObserver : ConsoleObserver
public sealed class TextOutputObserver : ConsoleObserver
{
/// <inheritdoc/>
public override async Task OnTextAsync(HarnessUXContainer ux, string text)
public override async Task OnTextAsync(IUXStateDriver ux, string text, AIAgent agent, AgentSession session)
{
await ux.WriteTextAsync(text);
}
@@ -1,5 +1,7 @@
// Copyright (c) Microsoft. All rights reserved.
using Harness.ConsoleReactiveComponents;
using Harness.Shared.Console.ToolFormatters;
using Microsoft.Agents.AI;
using Microsoft.Extensions.AI;
@@ -7,86 +9,103 @@ namespace Harness.Shared.Console.Observers;
/// <summary>
/// Collects <see cref="ToolApprovalRequestContent"/> items during the response stream,
/// displays approval-needed notifications inline, and prompts the user for approval
/// decisions after the stream completes.
/// displays approval-needed notifications inline, and after the stream completes returns
/// one <see cref="ChoiceFollowUpQuestion"/> per pending approval request. Each question's
/// continuation produces a separate <see cref="ChatMessage"/> carrying the approval
/// response content.
/// </summary>
internal sealed class ToolApprovalObserver : ConsoleObserver
public sealed class ToolApprovalObserver : ConsoleObserver
{
private readonly List<ToolApprovalRequestContent> _approvalRequests = [];
private readonly IReadOnlyList<ToolCallFormatter> _formatters;
/// <summary>
/// Initializes a new instance of the <see cref="ToolApprovalObserver"/> class.
/// </summary>
/// <param name="formatters">Optional list of tool formatters. When <see langword="null"/>,
/// the default formatters from <see cref="ToolCallFormatter.BuildDefaultToolFormatters"/> are used.</param>
public ToolApprovalObserver(IReadOnlyList<ToolCallFormatter>? formatters = null)
{
this._formatters = formatters ?? ToolCallFormatter.BuildDefaultToolFormatters();
}
/// <inheritdoc/>
public override async Task OnContentAsync(HarnessUXContainer ux, AIContent content)
public override async Task OnContentAsync(IUXStateDriver ux, AIContent content, AIAgent agent, AgentSession session)
{
if (content is ToolApprovalRequestContent approvalRequest)
{
this._approvalRequests.Add(approvalRequest);
string toolName = approvalRequest.ToolCall is FunctionCallContent fc
? ToolCallFormatter.Format(fc)
? ToolCallFormatter.Format(this._formatters, fc)
: approvalRequest.ToolCall?.ToString() ?? "unknown";
await ux.WriteInfoLineAsync($"⚠️ Approval needed: {toolName}", ConsoleColor.Yellow);
}
}
/// <inheritdoc/>
public override async Task<IList<ChatMessage>?> OnStreamCompleteAsync(
HarnessUXContainer ux,
public override Task<IList<FollowUpAction>?> OnStreamCompleteAsync(
IUXStateDriver ux,
AIAgent agent,
AgentSession session,
HarnessConsoleOptions options)
AgentSession session)
{
if (this._approvalRequests.Count == 0)
{
return null;
return Task.FromResult<IList<FollowUpAction>?>(null);
}
var actions = new List<FollowUpAction>(this._approvalRequests.Count);
foreach (var request in this._approvalRequests)
{
actions.Add(this.BuildApprovalQuestion(request));
}
var messages = await PromptForApprovalsAsync(ux, this._approvalRequests);
this._approvalRequests.Clear();
return messages;
return Task.FromResult<IList<FollowUpAction>?>(actions);
}
private static async Task<List<ChatMessage>?> PromptForApprovalsAsync(HarnessUXContainer ux, List<ToolApprovalRequestContent> approvalRequests)
private ChoiceFollowUpQuestion BuildApprovalQuestion(ToolApprovalRequestContent request)
{
if (approvalRequests.Count == 0)
string toolName = request.ToolCall is FunctionCallContent fc
? ToolCallFormatter.Format(this._formatters, fc)
: request.ToolCall?.ToString() ?? "unknown";
var choices = new List<string>
{
return null;
}
"Approve this call",
"Always approve this tool (any arguments)",
"Always approve this tool with these arguments",
"Deny",
};
var responses = new List<AIContent>();
foreach (var request in approvalRequests)
{
string toolName = request.ToolCall is FunctionCallContent fc
? ToolCallFormatter.Format(fc)
: request.ToolCall?.ToString() ?? "unknown";
string prompt = $"🔐 Tool approval: {toolName}";
var choices = new List<string>
return new ChoiceFollowUpQuestion(
Prompt: prompt,
Choices: choices,
AllowCustomText: false,
Continuation: async (selection, ux) =>
{
"Approve this call",
"Always approve this tool (any arguments)",
"Always approve this tool with these arguments",
"Deny",
};
AIContent response = selection switch
{
"Always approve this tool (any arguments)" => request.CreateAlwaysApproveToolResponse("User chose to always approve this tool"),
"Always approve this tool with these arguments" => request.CreateAlwaysApproveToolWithArgumentsResponse("User chose to always approve this tool with these arguments"),
"Deny" => request.CreateResponse(approved: false, reason: "User denied"),
_ => request.CreateResponse(approved: true, reason: "User approved"),
};
string selection = await ux.ReadSelectionAsync($"🔐 Tool approval: {toolName}", choices);
AIContent response = selection switch
{
"Always approve this tool (any arguments)" => request.CreateAlwaysApproveToolResponse("User chose to always approve this tool"),
"Always approve this tool with these arguments" => request.CreateAlwaysApproveToolWithArgumentsResponse("User chose to always approve this tool with these arguments"),
"Deny" => request.CreateResponse(approved: false, reason: "User denied"),
_ => request.CreateResponse(approved: true, reason: "User approved"),
};
string action = selection switch
{
"Always approve this tool (any arguments)" => "✅ Always approved (any args)",
"Always approve this tool with these arguments" => "✅ Always approved (these args)",
"Deny" => "❌ Denied",
_ => "✅ Approved",
};
string action = selection switch
{
"Always approve this tool (any arguments)" => "✅ Always approved (any args)",
"Always approve this tool with these arguments" => "✅ Always approved (these args)",
"Deny" => "❌ Denied",
_ => "✅ Approved",
};
await ux.WriteInfoLineAsync($" {action}", ConsoleColor.DarkGray);
ConsoleColor answerColor = selection == "Deny" ? ConsoleColor.Red : ConsoleColor.Green;
string formatted = $"🔹 {prompt}\n └─ {AnsiEscapes.SetForegroundColor(answerColor)}{action}{AnsiEscapes.ResetAttributes}";
await ux.WriteInfoLineAsync(formatted, ConsoleColor.Gray).ConfigureAwait(false);
responses.Add(response);
}
return [new ChatMessage(ChatRole.User, responses)];
return new ChatMessage(ChatRole.User, [response]);
});
}
}
@@ -1,5 +1,7 @@
// Copyright (c) Microsoft. All rights reserved.
using Harness.Shared.Console.ToolFormatters;
using Microsoft.Agents.AI;
using Microsoft.Extensions.AI;
namespace Harness.Shared.Console.Observers;
@@ -8,14 +10,26 @@ namespace Harness.Shared.Console.Observers;
/// Displays tool call notifications (🔧) for <see cref="FunctionCallContent"/>
/// and <see cref="ToolCallContent"/> items in the response stream.
/// </summary>
internal sealed class ToolCallDisplayObserver : ConsoleObserver
public sealed class ToolCallDisplayObserver : ConsoleObserver
{
private readonly IReadOnlyList<ToolCallFormatter> _formatters;
/// <summary>
/// Initializes a new instance of the <see cref="ToolCallDisplayObserver"/> class.
/// </summary>
/// <param name="formatters">Optional list of tool formatters. When <see langword="null"/>,
/// the default formatters from <see cref="ToolCallFormatter.BuildDefaultToolFormatters"/> are used.</param>
public ToolCallDisplayObserver(IReadOnlyList<ToolCallFormatter>? formatters = null)
{
this._formatters = formatters ?? ToolCallFormatter.BuildDefaultToolFormatters();
}
/// <inheritdoc/>
public override async Task OnContentAsync(HarnessUXContainer ux, AIContent content)
public override async Task OnContentAsync(IUXStateDriver ux, AIContent content, AIAgent agent, AgentSession session)
{
if (content is FunctionCallContent functionCall)
{
await ux.WriteInfoLineAsync($"🔧 Calling tool: {ToolCallFormatter.Format(functionCall)}...", ConsoleColor.DarkYellow);
await ux.WriteInfoLineAsync($"🔧 Calling tool: {ToolCallFormatter.Format(this._formatters, functionCall)}...", ConsoleColor.DarkYellow);
}
else if (content is ToolCallContent toolCall)
{
@@ -1,288 +0,0 @@
// Copyright (c) Microsoft. All rights reserved.
using System.Text;
using System.Text.Json;
using Microsoft.Extensions.AI;
namespace Harness.Shared.Console.Observers;
/// <summary>
/// Formats <see cref="FunctionCallContent"/> instances into human-readable strings
/// for console display.
/// </summary>
public static class ToolCallFormatter
{
/// <summary>
/// Returns a formatted string for the given tool call, with human-readable
/// details for known tools (todos, mode, sub-agents, web tools).
/// </summary>
/// <param name="call">The function call content to format.</param>
/// <returns>A formatted string describing the tool call.</returns>
public static string Format(FunctionCallContent call)
{
string? detail = call.Name switch
{
// Todo tools
"TodoList_Add" => FormatAddTodos(call),
"TodoList_Complete" => FormatIdList(call, "ids", "Complete"),
"TodoList_Remove" => FormatIdList(call, "ids", "Remove"),
"TodoList_GetRemaining" => null,
"TodoList_GetAll" => null,
// Mode tools
"AgentMode_Set" => FormatStringArg(call, "mode"),
"AgentMode_Get" => null,
// Sub-agent tools
"SubAgents_StartTask" => FormatStartSubTask(call),
"SubAgents_WaitForFirstCompletion" => FormatIdList(call, "taskIds", "Wait for"),
"SubAgents_GetTaskResults" => FormatSingleId(call, "taskId"),
"SubAgents_GetAllTasks" => null,
"SubAgents_ContinueTask" => FormatContinueTask(call),
"SubAgents_ClearCompletedTask" => FormatSingleId(call, "taskId"),
// File memory tools
"FileMemory_SaveFile" => FormatSaveFile(call),
"FileMemory_ReadFile" => FormatStringArg(call, "fileName"),
"FileMemory_DeleteFile" => FormatStringArg(call, "fileName"),
"FileMemory_ListFiles" => null,
"FileMemory_SearchFiles" => FormatSearchFiles(call),
// External tools
"web_search" => FormatStringArg(call, "query"),
"DownloadUri" => FormatStringArg(call, "uri"),
_ => FormatFallback(call),
};
return detail is not null ? $"{call.Name} {detail}" : call.Name;
}
private static string? FormatAddTodos(FunctionCallContent call)
{
if (call.Arguments?.TryGetValue("todos", out object? todosObj) != true || todosObj is null)
{
return null;
}
var titles = new List<string>();
if (todosObj is JsonElement jsonArray && jsonArray.ValueKind == JsonValueKind.Array)
{
foreach (JsonElement item in jsonArray.EnumerateArray())
{
string? title = item.TryGetProperty("title", out JsonElement titleElement)
? titleElement.GetString()
: null;
if (!string.IsNullOrEmpty(title))
{
titles.Add(title);
}
}
}
if (titles.Count == 0)
{
return null;
}
var sb = new StringBuilder();
sb.Append($"({titles.Count} item{(titles.Count == 1 ? "" : "s")})");
foreach (string title in titles)
{
sb.Append($"\n • {title}");
}
return sb.ToString();
}
private static string? FormatIdList(FunctionCallContent call, string paramName, string verb)
{
List<int>? ids = GetIntList(call, paramName);
if (ids is null || ids.Count == 0)
{
return null;
}
return $"({verb} #{string.Join(", #", ids)})";
}
private static string? FormatSingleId(FunctionCallContent call, string paramName)
{
int? id = GetInt(call, paramName);
return id.HasValue ? $"(task #{id.Value})" : null;
}
private static string? FormatStartSubTask(FunctionCallContent call)
{
string? agentName = GetString(call, "agentName");
string? description = GetString(call, "description");
if (agentName is null && description is null)
{
return null;
}
var sb = new StringBuilder("(");
if (agentName is not null)
{
sb.Append($"agent: {agentName}");
}
if (description is not null)
{
if (agentName is not null)
{
sb.Append(", ");
}
sb.Append($"\"{Truncate(description, 60)}\"");
}
sb.Append(')');
return sb.ToString();
}
private static string? FormatContinueTask(FunctionCallContent call)
{
int? taskId = GetInt(call, "taskId");
string? text = GetString(call, "text");
if (!taskId.HasValue)
{
return null;
}
return text is not null
? $"(task #{taskId.Value}, \"{Truncate(text, 50)}\")"
: $"(task #{taskId.Value})";
}
private static string? FormatSaveFile(FunctionCallContent call)
{
string? fileName = GetString(call, "fileName");
string? description = GetString(call, "description");
if (fileName is null)
{
return null;
}
return string.IsNullOrEmpty(description)
? $"({fileName})"
: $"({fileName}, with description)";
}
private static string? FormatSearchFiles(FunctionCallContent call)
{
string? pattern = GetString(call, "regexPattern");
string? filePattern = GetString(call, "filePattern");
if (pattern is null)
{
return null;
}
return string.IsNullOrEmpty(filePattern)
? $"(/{pattern}/)"
: $"(/{pattern}/ in {filePattern})";
}
private static string? FormatStringArg(FunctionCallContent call, string paramName)
{
string? value = GetString(call, paramName);
return value is not null ? $"({value})" : null;
}
private static string? FormatFallback(FunctionCallContent call)
{
if (call.Arguments is null || call.Arguments.Count == 0)
{
return null;
}
var parts = new List<string>();
foreach (var kvp in call.Arguments)
{
string? stringValue = kvp.Value switch
{
JsonElement je => je.ValueKind switch
{
JsonValueKind.String => je.GetString(),
JsonValueKind.Number => je.GetRawText(),
JsonValueKind.True => "true",
JsonValueKind.False => "false",
_ => null,
},
not null => kvp.Value.ToString(),
_ => null,
};
if (stringValue is not null)
{
parts.Add($"{kvp.Key}: {Truncate(stringValue, 40)}");
}
}
return parts.Count > 0 ? $"({string.Join(", ", parts)})" : null;
}
private static string? GetString(FunctionCallContent call, string paramName)
{
if (call.Arguments?.TryGetValue(paramName, out object? value) != true || value is null)
{
return null;
}
return value switch
{
JsonElement je when je.ValueKind == JsonValueKind.String => je.GetString(),
string s => s,
_ => value.ToString(),
};
}
private static int? GetInt(FunctionCallContent call, string paramName)
{
if (call.Arguments?.TryGetValue(paramName, out object? value) != true || value is null)
{
return null;
}
return value switch
{
JsonElement je when je.ValueKind == JsonValueKind.Number => je.GetInt32(),
int i => i,
_ => int.TryParse(value.ToString(), out int parsed) ? parsed : null,
};
}
private static List<int>? GetIntList(FunctionCallContent call, string paramName)
{
if (call.Arguments?.TryGetValue(paramName, out object? value) != true || value is null)
{
return null;
}
var result = new List<int>();
if (value is JsonElement je && je.ValueKind == JsonValueKind.Array)
{
foreach (JsonElement item in je.EnumerateArray())
{
if (item.ValueKind == JsonValueKind.Number)
{
result.Add(item.GetInt32());
}
}
}
return result.Count > 0 ? result : null;
}
private static string Truncate(string text, int maxLength)
{
return text.Length <= maxLength ? text : string.Concat(text.AsSpan(0, maxLength), "…");
}
}
@@ -1,5 +1,6 @@
// Copyright (c) Microsoft. All rights reserved.
using Microsoft.Agents.AI;
using Microsoft.Extensions.AI;
namespace Harness.Shared.Console.Observers;
@@ -7,7 +8,7 @@ namespace Harness.Shared.Console.Observers;
/// <summary>
/// Displays token usage statistics (📊) from the response stream.
/// </summary>
internal sealed class UsageDisplayObserver : ConsoleObserver
public sealed class UsageDisplayObserver : ConsoleObserver
{
private readonly int? _maxContextWindowTokens;
private readonly int? _maxOutputTokens;
@@ -24,7 +25,7 @@ internal sealed class UsageDisplayObserver : ConsoleObserver
}
/// <inheritdoc/>
public override Task OnContentAsync(HarnessUXContainer ux, AIContent content)
public override Task OnContentAsync(IUXStateDriver ux, AIContent content, AIAgent agent, AgentSession session)
{
if (content is UsageContent usage)
{
@@ -5,7 +5,7 @@ namespace Harness.Shared.Console;
/// <summary>
/// Represents the type of an output entry in the console conversation.
/// </summary>
public enum OutputEntryType
internal enum OutputEntryType
{
/// <summary>User input echo (e.g. "You: hello").</summary>
UserInput,
@@ -25,9 +25,10 @@ public enum OutputEntryType
/// <summary>
/// Represents a single output entry in the console conversation history.
/// These entries are rendered by the <see cref="HarnessAppComponent"/> via its render delegate.
/// Used internally by <see cref="HarnessConsoleUXStateDriver"/> to track
/// the in-progress streaming entry and last-entry type for spacing decisions.
/// </summary>
/// <param name="Type">The type of output entry.</param>
/// <param name="Text">The text content of the entry.</param>
/// <param name="Color">Optional foreground color for rendering.</param>
public record OutputEntry(OutputEntryType Type, string Text, ConsoleColor? Color = null);
internal sealed record OutputEntry(OutputEntryType Type, string Text, ConsoleColor? Color = null);
@@ -0,0 +1,51 @@
// Copyright (c) Microsoft. All rights reserved.
using System.Text.Json;
using Microsoft.Extensions.AI;
namespace Harness.Shared.Console.ToolFormatters;
/// <summary>
/// Catch-all formatter that handles any tool not matched by a more specific formatter.
/// Displays a generic summary of the tool's arguments. This formatter should always be
/// placed last in the formatter list.
/// </summary>
public sealed class FallbackToolFormatter : ToolCallFormatter
{
/// <inheritdoc/>
public override bool CanFormat(FunctionCallContent call) => true;
/// <inheritdoc/>
public override string? FormatDetail(FunctionCallContent call)
{
if (call.Arguments is null || call.Arguments.Count == 0)
{
return null;
}
var parts = new List<string>();
foreach (var kvp in call.Arguments)
{
string? stringValue = kvp.Value switch
{
JsonElement je => je.ValueKind switch
{
JsonValueKind.String => je.GetString(),
JsonValueKind.Number => je.GetRawText(),
JsonValueKind.True => "true",
JsonValueKind.False => "false",
_ => null,
},
not null => kvp.Value.ToString(),
_ => null,
};
if (stringValue is not null)
{
parts.Add($"{kvp.Key}: {Truncate(stringValue, 40)}");
}
}
return parts.Count > 0 ? $"({string.Join(", ", parts)})" : null;
}
}
@@ -0,0 +1,61 @@
// Copyright (c) Microsoft. All rights reserved.
using Microsoft.Extensions.AI;
namespace Harness.Shared.Console.ToolFormatters;
/// <summary>
/// Formats <c>FileMemory_*</c> tool calls, showing file names and search patterns
/// with tree-view corners for save operations.
/// </summary>
public sealed class FileMemoryToolFormatter : ToolCallFormatter
{
/// <inheritdoc/>
public override bool CanFormat(FunctionCallContent call) => call.Name.StartsWith("FileMemory_", StringComparison.Ordinal);
/// <inheritdoc/>
public override string? FormatDetail(FunctionCallContent call) => call.Name switch
{
"FileMemory_SaveFile" => FormatSaveFile(call),
"FileMemory_ReadFile" => FormatStringArg(call, "fileName"),
"FileMemory_DeleteFile" => FormatStringArg(call, "fileName"),
"FileMemory_SearchFiles" => FormatSearchFiles(call),
_ => null,
};
private static string? FormatSaveFile(FunctionCallContent call)
{
string? fileName = GetStringArgumentValue(call, "fileName");
string? description = GetStringArgumentValue(call, "description");
if (fileName is null)
{
return null;
}
return string.IsNullOrEmpty(description)
? $"\n └─ {fileName}"
: $"\n └─ {fileName} (with description)";
}
private static string? FormatSearchFiles(FunctionCallContent call)
{
string? pattern = GetStringArgumentValue(call, "regexPattern");
string? filePattern = GetStringArgumentValue(call, "filePattern");
if (pattern is null)
{
return null;
}
return string.IsNullOrEmpty(filePattern)
? $"(/{pattern}/)"
: $"(/{pattern}/ in {filePattern})";
}
private static string? FormatStringArg(FunctionCallContent call, string paramName)
{
string? value = GetStringArgumentValue(call, paramName);
return value is not null ? $"({value})" : null;
}
}
@@ -0,0 +1,27 @@
// Copyright (c) Microsoft. All rights reserved.
using Microsoft.Extensions.AI;
namespace Harness.Shared.Console.ToolFormatters;
/// <summary>
/// Formats <c>AgentMode_*</c> tool calls, showing the target mode for Set operations.
/// </summary>
public sealed class ModeToolFormatter : ToolCallFormatter
{
/// <inheritdoc/>
public override bool CanFormat(FunctionCallContent call) => call.Name.StartsWith("AgentMode_", StringComparison.Ordinal);
/// <inheritdoc/>
public override string? FormatDetail(FunctionCallContent call) => call.Name switch
{
"AgentMode_Set" => FormatStringArg(call, "mode"),
_ => null,
};
private static string? FormatStringArg(FunctionCallContent call, string paramName)
{
string? value = GetStringArgumentValue(call, paramName);
return value is not null ? $"({value})" : null;
}
}
@@ -0,0 +1,101 @@
// Copyright (c) Microsoft. All rights reserved.
using System.Text;
using Microsoft.Extensions.AI;
namespace Harness.Shared.Console.ToolFormatters;
/// <summary>
/// Formats <c>SubAgents_*</c> tool calls with human-readable details
/// for task start, continue, wait, and result retrieval operations.
/// </summary>
public sealed class SubAgentToolFormatter : ToolCallFormatter
{
/// <inheritdoc/>
public override bool CanFormat(FunctionCallContent call) => call.Name.StartsWith("SubAgents_", StringComparison.Ordinal);
/// <inheritdoc/>
public override string? FormatDetail(FunctionCallContent call) => call.Name switch
{
"SubAgents_StartTask" => FormatStartSubTask(call),
"SubAgents_WaitForFirstCompletion" => FormatIdList(call, "taskIds", "Wait for"),
"SubAgents_GetTaskResults" => FormatSingleId(call, "taskId"),
"SubAgents_ContinueTask" => FormatContinueTask(call),
"SubAgents_ClearCompletedTask" => FormatSingleId(call, "taskId"),
_ => null,
};
private static string? FormatStartSubTask(FunctionCallContent call)
{
string? agentName = GetStringArgumentValue(call, "agentName");
string? description = GetStringArgumentValue(call, "description");
if (agentName is null && description is null)
{
return null;
}
var sb = new StringBuilder();
if (agentName is not null && description is not null)
{
sb.Append($"\n ├─ Agent: {agentName}");
sb.Append($"\n └─ \"{Truncate(description, 80)}\"");
}
else if (agentName is not null)
{
sb.Append($"\n └─ Agent: {agentName}");
}
else
{
sb.Append($"\n └─ \"{Truncate(description!, 80)}\"");
}
return sb.ToString();
}
private static string? FormatIdList(FunctionCallContent call, string paramName, string verb)
{
List<int>? ids = GetIntListArgumentValue(call, paramName);
if (ids is null || ids.Count == 0)
{
return null;
}
var sb = new StringBuilder();
for (int i = 0; i < ids.Count; i++)
{
string connector = i < ids.Count - 1 ? "├─" : "└─";
sb.Append($"\n {connector} {verb} #{ids[i]}");
}
return sb.ToString();
}
private static string? FormatSingleId(FunctionCallContent call, string paramName)
{
int? id = GetIntArgumentValue(call, paramName);
return id.HasValue ? $"(task #{id.Value})" : null;
}
private static string? FormatContinueTask(FunctionCallContent call)
{
int? taskId = GetIntArgumentValue(call, "taskId");
string? text = GetStringArgumentValue(call, "text");
if (!taskId.HasValue)
{
return null;
}
if (text is not null)
{
var sb = new StringBuilder();
sb.Append($"\n ├─ Task #{taskId.Value}");
sb.Append($"\n └─ \"{Truncate(text, 80)}\"");
return sb.ToString();
}
return $"\n └─ Task #{taskId.Value}";
}
}
@@ -0,0 +1,84 @@
// Copyright (c) Microsoft. All rights reserved.
using System.Text;
using System.Text.Json;
using Microsoft.Extensions.AI;
namespace Harness.Shared.Console.ToolFormatters;
/// <summary>
/// Formats <c>TodoList_*</c> tool calls with tree-view output for added items
/// and structured output for complete/remove operations.
/// </summary>
public sealed class TodoToolFormatter : ToolCallFormatter
{
/// <inheritdoc/>
public override bool CanFormat(FunctionCallContent call) => call.Name.StartsWith("TodoList_", StringComparison.Ordinal);
/// <inheritdoc/>
public override string? FormatDetail(FunctionCallContent call) => call.Name switch
{
"TodoList_Add" => FormatAddTodos(call),
"TodoList_Complete" => FormatIdList(call, "ids", "Complete"),
"TodoList_Remove" => FormatIdList(call, "ids", "Remove"),
_ => null,
};
private static string? FormatAddTodos(FunctionCallContent call)
{
if (call.Arguments?.TryGetValue("todos", out object? todosObj) != true || todosObj is null)
{
return null;
}
var titles = new List<string>();
if (todosObj is JsonElement jsonArray && jsonArray.ValueKind == JsonValueKind.Array)
{
foreach (JsonElement item in jsonArray.EnumerateArray())
{
string? title = item.TryGetProperty("title", out JsonElement titleElement)
? titleElement.GetString()
: null;
if (!string.IsNullOrEmpty(title))
{
titles.Add(title);
}
}
}
if (titles.Count == 0)
{
return null;
}
var sb = new StringBuilder();
sb.Append($"({titles.Count} item{(titles.Count == 1 ? "" : "s")})");
for (int i = 0; i < titles.Count; i++)
{
string connector = i < titles.Count - 1 ? "├─" : "└─";
sb.Append($"\n {connector} {titles[i]}");
}
return sb.ToString();
}
private static string? FormatIdList(FunctionCallContent call, string paramName, string verb)
{
List<int>? ids = GetIntListArgumentValue(call, paramName);
if (ids is null || ids.Count == 0)
{
return null;
}
var sb = new StringBuilder();
for (int i = 0; i < ids.Count; i++)
{
string connector = i < ids.Count - 1 ? "├─" : "└─";
sb.Append($"\n {connector} {verb} #{ids[i]}");
}
return sb.ToString();
}
}
@@ -0,0 +1,135 @@
// Copyright (c) Microsoft. All rights reserved.
using System.Text.Json;
using Microsoft.Extensions.AI;
namespace Harness.Shared.Console.ToolFormatters;
/// <summary>
/// Base class for tool call formatters that produce human-readable display strings
/// for <see cref="FunctionCallContent"/> items shown in the console.
/// </summary>
public abstract class ToolCallFormatter
{
/// <summary>
/// Returns <see langword="true"/> if this formatter can handle the given function call.
/// </summary>
/// <param name="call">The function call content to check.</param>
/// <returns><see langword="true"/> if this formatter should be used; otherwise <see langword="false"/>.</returns>
public abstract bool CanFormat(FunctionCallContent call);
/// <summary>
/// Returns the detail portion of the formatted output for the given tool call,
/// or <see langword="null"/> if only the tool name should be displayed.
/// </summary>
/// <param name="call">The function call content to format.</param>
/// <returns>A detail string to append after the tool name, or <see langword="null"/>.</returns>
public abstract string? FormatDetail(FunctionCallContent call);
/// <summary>
/// Formats a tool call using the first matching formatter from the provided list.
/// Returns <c>"{toolName} {detail}"</c> when a formatter produces detail,
/// or just <c>"{toolName}"</c> otherwise.
/// </summary>
internal static string Format(IReadOnlyList<ToolCallFormatter> formatters, FunctionCallContent call)
{
foreach (var formatter in formatters)
{
if (formatter.CanFormat(call))
{
string? detail = formatter.FormatDetail(call);
return detail is not null ? $"{call.Name} {detail}" : call.Name;
}
}
return call.Name;
}
/// <summary>
/// Creates the default list of tool call formatters. The <see cref="FallbackToolFormatter"/>
/// is always last. Users can call this method and combine the result with their own formatters.
/// </summary>
/// <returns>A list of all built-in tool call formatters.</returns>
public static List<ToolCallFormatter> BuildDefaultToolFormatters()
{
return
[
new TodoToolFormatter(),
new ModeToolFormatter(),
new SubAgentToolFormatter(),
new FileMemoryToolFormatter(),
new WebSearchToolFormatter(),
new FallbackToolFormatter(),
];
}
/// <summary>
/// Extracts a string argument value from a function call.
/// </summary>
protected static string? GetStringArgumentValue(FunctionCallContent call, string paramName)
{
if (call.Arguments?.TryGetValue(paramName, out object? value) != true || value is null)
{
return null;
}
return value switch
{
JsonElement je when je.ValueKind == JsonValueKind.String => je.GetString(),
string s => s,
_ => value.ToString(),
};
}
/// <summary>
/// Extracts an integer argument value from a function call.
/// </summary>
protected static int? GetIntArgumentValue(FunctionCallContent call, string paramName)
{
if (call.Arguments?.TryGetValue(paramName, out object? value) != true || value is null)
{
return null;
}
return value switch
{
JsonElement je when je.ValueKind == JsonValueKind.Number => je.GetInt32(),
int i => i,
_ => int.TryParse(value.ToString(), out int parsed) ? parsed : null,
};
}
/// <summary>
/// Extracts a list of integer argument values from a function call.
/// </summary>
protected static List<int>? GetIntListArgumentValue(FunctionCallContent call, string paramName)
{
if (call.Arguments?.TryGetValue(paramName, out object? value) != true || value is null)
{
return null;
}
var result = new List<int>();
if (value is JsonElement je && je.ValueKind == JsonValueKind.Array)
{
foreach (JsonElement item in je.EnumerateArray())
{
if (item.ValueKind == JsonValueKind.Number)
{
result.Add(item.GetInt32());
}
}
}
return result.Count > 0 ? result : null;
}
/// <summary>
/// Truncates a string to the specified maximum length, appending an ellipsis if truncated.
/// </summary>
protected static string Truncate(string text, int maxLength)
{
return text.Length <= maxLength ? text : string.Concat(text.AsSpan(0, maxLength), "…");
}
}
@@ -0,0 +1,22 @@
// Copyright (c) Microsoft. All rights reserved.
using Microsoft.Extensions.AI;
namespace Harness.Shared.Console.ToolFormatters;
/// <summary>
/// Formats <c>web_search</c> tool calls, showing the search query.
/// </summary>
public sealed class WebSearchToolFormatter : ToolCallFormatter
{
/// <inheritdoc/>
public override bool CanFormat(FunctionCallContent call) =>
call.Name is "web_search";
/// <inheritdoc/>
public override string? FormatDetail(FunctionCallContent call)
{
string? value = GetStringArgumentValue(call, "query");
return value is not null ? $"({value})" : null;
}
}
@@ -0,0 +1,23 @@
// Copyright (c) Microsoft. All rights reserved.
using Harness.Shared.Console.ToolFormatters;
using Microsoft.Extensions.AI;
namespace SampleApp;
/// <summary>
/// Formats <c>DownloadUri</c> tool calls, showing the target URI.
/// </summary>
public sealed class DownloadUriToolFormatter : ToolCallFormatter
{
/// <inheritdoc/>
public override bool CanFormat(FunctionCallContent call) =>
call.Name is "DownloadUri";
/// <inheritdoc/>
public override string? FormatDetail(FunctionCallContent call)
{
string? value = GetStringArgumentValue(call, "uri");
return value is not null ? $"({value})" : null;
}
}
@@ -8,7 +8,8 @@
//
// Special commands:
// /todos — Display the current todo list without invoking the agent.
// exit — End the session.
// /mode — Get or set the current agent mode.
// /exit — End the session.
#pragma warning disable OPENAI001 // Suppress experimental API warnings for Responses API usage.
#pragma warning disable MAAI001 // Suppress experimental API warnings for Agents AI experiments.
@@ -16,6 +17,7 @@
using System.ClientModel.Primitives;
using Azure.Identity;
using Harness.Shared.Console;
using Harness.Shared.Console.ToolFormatters;
using Microsoft.Agents.AI;
using Microsoft.Extensions.AI;
using OpenAI;
@@ -158,13 +160,15 @@ AIAgent agent =
// Run the interactive console session using the shared HarnessConsole helper.
await HarnessConsole.RunAgentAsync(
agent,
title: "Research Assistant",
userPrompt: "Enter a research topic to get started.",
new HarnessConsoleOptions
{
MaxContextWindowTokens = MaxContextWindowTokens,
MaxOutputTokens = MaxOutputTokens,
EnablePlanningUx = true,
PlanningModeName = "plan",
ExecutionModeName = "execute"
Observers = HarnessConsoleOptions.BuildObserversWithPlanning(
agent,
planModeName: "plan",
executionModeName: "execute",
maxContextWindowTokens: MaxContextWindowTokens,
maxOutputTokens: MaxOutputTokens,
toolFormatters: [new DownloadUriToolFormatter(), .. ToolCallFormatter.BuildDefaultToolFormatters()]),
CommandHandlers = HarnessConsoleOptions.BuildDefaultCommandHandlers(agent),
});
@@ -6,7 +6,7 @@
// equipped with Foundry's hosted web search tool.
//
// Special commands:
// exit — End the session.
// /exit — End the session.
#pragma warning disable OPENAI001 // Suppress experimental API warnings for Responses API usage.
#pragma warning disable MAAI001 // Suppress experimental API warnings for Agents AI experiments.
@@ -103,5 +103,4 @@ AIAgent parentAgent =
// Run the interactive console session.
await HarnessConsole.RunAgentAsync(
parentAgent,
title: "Stock Price Researcher (SubAgents Demo)",
userPrompt: "Enter a list of stock tickers (e.g., BAC, MSFT, BA):");
@@ -8,7 +8,7 @@
// Ask the agent to analyze the data, produce summaries, or create new output files.
//
// Special commands:
// exit — End the session.
// /exit — End the session.
#pragma warning disable OPENAI001 // Suppress experimental API warnings for Responses API usage.
#pragma warning disable MAAI001 // Suppress experimental API warnings for Agents AI experiments.
@@ -85,5 +85,4 @@ AIAgent agent =
// Run the interactive console session.
await HarnessConsole.RunAgentAsync(
agent,
title: "Data Processing Assistant",
userPrompt: "Ask me to analyze the data files, produce summaries, or create output files.");
@@ -0,0 +1,42 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFrameworks>net10.0</TargetFrameworks>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
</PropertyGroup>
<PropertyGroup>
<InjectIsExternalInitOnLegacy>true</InjectIsExternalInitOnLegacy>
<InjectSharedFoundryAgents>true</InjectSharedFoundryAgents>
<InjectSharedWorkflowsExecution>true</InjectSharedWorkflowsExecution>
<InjectSharedWorkflowsSettings>true</InjectSharedWorkflowsSettings>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Azure.AI.Projects" />
<PackageReference Include="Microsoft.Extensions.Configuration" />
<PackageReference Include="Microsoft.Extensions.Configuration.Binder" />
<PackageReference Include="Microsoft.Extensions.Configuration.EnvironmentVariables" />
<PackageReference Include="Microsoft.Extensions.Configuration.Json" />
<PackageReference Include="Microsoft.Extensions.Configuration.UserSecrets" />
<PackageReference Include="Microsoft.Extensions.DependencyInjection" />
<PackageReference Include="Microsoft.Extensions.Logging" />
<PackageReference Include="OpenAI" />
<PackageReference Include="System.ClientModel" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\..\..\..\src\Microsoft.Agents.AI.Workflows.Declarative\Microsoft.Agents.AI.Workflows.Declarative.csproj" />
<ProjectReference Include="..\..\..\..\src\Microsoft.Agents.AI.Workflows.Declarative.Foundry\Microsoft.Agents.AI.Workflows.Declarative.Foundry.csproj" />
<ProjectReference Include="..\..\..\..\src\Microsoft.Agents.AI.Workflows.Declarative.Mcp\Microsoft.Agents.AI.Workflows.Declarative.Mcp.csproj" />
</ItemGroup>
<ItemGroup>
<None Include="InvokeFoundryToolboxMcp.yaml">
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</None>
</ItemGroup>
</Project>
@@ -0,0 +1,87 @@
#
# This workflow demonstrates invoking MCP tools through a Foundry toolbox MCP proxy.
#
# The toolbox is provisioned with TWO different tool types:
# 1. A Foundry built-in web_search tool
# 2. A Microsoft Learn MCP server (microsoft_docs)
# Both are surfaced through the same MCP-compatible toolbox endpoint.
#
# The workflow:
# 1. Accepts a documentation/web search query as input
# 2. Lists the tools exposed by the Foundry toolbox using reserved toolName: tools/list
# 3. Invokes the microsoft_docs_search MCP tool
# 4. Invokes the built-in web_search tool against the same toolbox endpoint
# 5. Uses an agent to summarize and combine both result sets
#
# Example input:
# How do I use Azure OpenAI with my data?
#
kind: Workflow
trigger:
kind: OnConversationStart
id: workflow_invoke_foundry_toolbox_mcp
actions:
# Set the search query from user input.
- kind: SetVariable
id: set_search_query
variable: Local.SearchQuery
value: =System.LastMessage.Text
# List tools exposed by the Foundry toolbox MCP proxy.
- kind: InvokeMcpTool
id: list_toolbox_tools
serverUrl: =Env.FOUNDRY_TOOLBOX_MCP_SERVER_URL
serverLabel: foundry_toolbox
toolName: tools/list
conversationId: =System.ConversationId
headers:
Foundry-Features: Toolboxes=V1Preview
output:
autoSend: true
result: Local.ToolboxTools
# Invoke a specific tool exposed through the toolbox and add the result to the conversation.
- kind: InvokeMcpTool
id: search_docs_with_toolbox
serverUrl: =Env.FOUNDRY_TOOLBOX_MCP_SERVER_URL
serverLabel: foundry_toolbox
toolName: =Env.FOUNDRY_TOOLBOX_DOCS_SERVER_LABEL & "___microsoft_docs_search"
conversationId: =System.ConversationId
headers:
Foundry-Features: Toolboxes=V1Preview
arguments:
query: =Local.SearchQuery
output:
autoSend: true
result: Local.SearchResult
# Invoke the web_search built-in tool through the same toolbox proxy. The toolbox surfaces
# built-in Foundry tools (like web_search) alongside MCP tools through one MCP-compatible
# endpoint. Note that web_search expects argument 'search_query' (not 'query').
- kind: InvokeMcpTool
id: search_web_with_toolbox
serverUrl: =Env.FOUNDRY_TOOLBOX_MCP_SERVER_URL
serverLabel: foundry_toolbox
toolName: =Env.FOUNDRY_TOOLBOX_WEB_SEARCH_TOOL_NAME
conversationId: =System.ConversationId
headers:
Foundry-Features: Toolboxes=V1Preview
arguments:
search_query: =Local.SearchQuery
output:
autoSend: true
result: Local.WebSearchResult
# Use the agent to summarize what happened and answer from the toolbox result.
- kind: InvokeAzureAgent
id: summarize_toolbox_result
agent:
name: FoundryToolboxMcpAgent
conversationId: =System.ConversationId
input:
messages: =UserMessage("Combine the Microsoft Learn docs results and the Foundry web search results in the conversation to answer the query " & Local.SearchQuery)
output:
autoSend: true
messages: Local.Summary
@@ -0,0 +1,218 @@
// Copyright (c) Microsoft. All rights reserved.
// This sample demonstrates using InvokeMcpTool to call MCP tools through a Foundry toolbox.
// It creates a sample toolbox that exposes Microsoft Learn MCP tools, lists the toolbox tools
// through the reserved tools/list operation, then calls microsoft_docs_search from the workflow.
using System.ClientModel;
using System.ClientModel.Primitives;
using System.Collections.Concurrent;
using System.Net.Http.Headers;
using Azure.AI.Projects;
using Azure.AI.Projects.Agents;
using Azure.Core;
using Azure.Identity;
using Microsoft.Agents.AI.Workflows.Declarative.Mcp;
using Microsoft.Extensions.Configuration;
using OpenAI.Responses;
using Shared.Foundry;
using Shared.Workflows;
#pragma warning disable OPENAI001 // Experimental API
#pragma warning disable AAIP001 // AgentToolboxes is experimental
namespace Demo.Workflows.Declarative.InvokeFoundryToolboxMcp;
/// <summary>
/// Demonstrates a workflow that uses InvokeMcpTool to call MCP tools exposed through a Foundry toolbox.
/// </summary>
/// <remarks>
/// This sample provisions a toolbox with Microsoft Learn MCP tools, uses the reserved
/// <c>tools/list</c> tool name to list the toolbox tools, calls one specific toolbox tool,
/// and has a Foundry agent summarize the results.
/// </remarks>
internal sealed class Program
{
private const string ToolboxNameSetting = "FOUNDRY_TOOLBOX_NAME";
private const string ToolboxApiVersionSetting = "FOUNDRY_AGENT_TOOLSET_API_VERSION";
private const string ToolboxMcpServerUrlSetting = "FOUNDRY_TOOLBOX_MCP_SERVER_URL";
private const string DocsServerLabelSetting = "FOUNDRY_TOOLBOX_DOCS_SERVER_LABEL";
private const string WebSearchToolNameSetting = "FOUNDRY_TOOLBOX_WEB_SEARCH_TOOL_NAME";
private const string DefaultToolboxName = "declarative_foundry_toolbox_mcp";
private const string DefaultToolboxApiVersion = "v1";
private const string DefaultDocsServerLabel = "microsoft_docs";
private const string DefaultWebSearchToolName = "web_search";
public static async Task Main(string[] args)
{
// Initialize configuration
IConfiguration configuration = Application.InitializeConfig();
Uri foundryEndpoint = new(configuration.GetValue(Application.Settings.FoundryEndpoint));
string toolboxName = configuration[ToolboxNameSetting] ?? DefaultToolboxName;
string toolboxApiVersion = configuration[ToolboxApiVersionSetting] ?? DefaultToolboxApiVersion;
string docsServerLabel = configuration[DocsServerLabelSetting] ?? DefaultDocsServerLabel;
string webSearchToolName = configuration[WebSearchToolNameSetting] ?? DefaultWebSearchToolName;
// WARNING: DefaultAzureCredential is convenient for development but requires careful consideration in production.
// In production, consider using a specific credential (e.g., ManagedIdentityCredential) to avoid
// latency issues, unintended credential probing, and potential security risks from fallback mechanisms.
DefaultAzureCredential credential = new();
// Ensure sample toolbox and agent exist in Foundry
string toolboxEndpoint = await CreateSampleToolboxAsync(toolboxName, docsServerLabel, foundryEndpoint, credential);
string toolboxMcpServerUrl = BuildToolboxMcpServerUrl(toolboxEndpoint, toolboxName, toolboxApiVersion);
IConfiguration workflowConfiguration = new ConfigurationBuilder()
.AddConfiguration(configuration)
.AddInMemoryCollection(new Dictionary<string, string?>
{
[ToolboxMcpServerUrlSetting] = toolboxMcpServerUrl,
[DocsServerLabelSetting] = docsServerLabel,
[WebSearchToolNameSetting] = webSearchToolName,
})
.Build();
await CreateAgentAsync(foundryEndpoint, configuration, credential);
// Get input from command line or console
string workflowInput = Application.GetInput(args);
// Create the MCP tool handler for invoking the Foundry toolbox MCP proxy.
ConcurrentBag<HttpClient> createdHttpClients = [];
DefaultMcpToolHandler mcpToolHandler = new(
httpClientProvider: async (serverUrl, _) =>
{
await Task.CompletedTask.ConfigureAwait(false);
if (!string.Equals(serverUrl, toolboxMcpServerUrl, StringComparison.OrdinalIgnoreCase))
{
return null;
}
FoundryToolboxBearerTokenHandler handler = new(credential)
{
InnerHandler = new HttpClientHandler()
};
HttpClient httpClient = new(handler);
createdHttpClients.Add(httpClient);
return httpClient;
});
try
{
// Create the workflow factory with MCP tool provider
WorkflowFactory workflowFactory = new("InvokeFoundryToolboxMcp.yaml", foundryEndpoint)
{
Configuration = workflowConfiguration,
McpToolHandler = mcpToolHandler
};
// Execute the workflow
WorkflowRunner runner = new() { UseJsonCheckpoints = true };
await runner.ExecuteAsync(workflowFactory.CreateWorkflow, workflowInput);
}
finally
{
// Clean up connections and dispose created HttpClients
await mcpToolHandler.DisposeAsync();
foreach (HttpClient httpClient in createdHttpClients)
{
httpClient.Dispose();
}
}
}
private static async Task CreateAgentAsync(Uri foundryEndpoint, IConfiguration configuration, TokenCredential credential)
{
AIProjectClient aiProjectClient = new(foundryEndpoint, credential);
await aiProjectClient.CreateAgentAsync(
agentName: "FoundryToolboxMcpAgent",
agentDefinition: DefineToolboxAgent(configuration),
agentDescription: "Summarizes Foundry toolbox MCP tool results");
}
private static DeclarativeAgentDefinition DefineToolboxAgent(IConfiguration configuration)
{
return new DeclarativeAgentDefinition(configuration.GetValue(Application.Settings.FoundryModel))
{
Instructions =
"""
You are a helpful assistant that explains results produced by tools exposed through a Foundry toolbox.
The conversation history contains output from BOTH a Microsoft Learn documentation search (MCP) and a Foundry web search.
Synthesize an answer that draws on both sources, calls out where they agree or differ, and notes which toolbox tool produced each fact when it is relevant.
Be concise.
"""
};
}
private static async Task<string> CreateSampleToolboxAsync(string name, string serverLabel, Uri foundryEndpoint, TokenCredential credential)
{
AgentAdministrationClientOptions options = new();
options.AddPolicy(new FoundryFeaturesPolicy("Toolboxes=V1Preview"), PipelinePosition.PerCall);
AgentAdministrationClient adminClient = new(foundryEndpoint, credential, options);
AgentToolboxes toolboxClient = adminClient.GetAgentToolboxes();
try
{
await toolboxClient.DeleteToolboxAsync(name);
Console.WriteLine($"Deleted existing toolbox '{name}'");
}
catch (ClientResultException ex) when (ex.Status == 404)
{
// Toolbox does not exist.
}
ProjectsAgentTool webTool = ProjectsAgentTool.AsProjectTool(ResponseTool.CreateWebSearchTool());
ProjectsAgentTool mcpTool = ProjectsAgentTool.AsProjectTool(ResponseTool.CreateMcpTool(
serverLabel: serverLabel,
serverUri: new Uri("https://learn.microsoft.com/api/mcp"),
toolCallApprovalPolicy: new McpToolCallApprovalPolicy(GlobalMcpToolCallApprovalPolicy.NeverRequireApproval)));
ToolboxVersion created = (await toolboxClient.CreateToolboxVersionAsync(
name: name,
tools: [webTool, mcpTool],
description: "Sample toolbox combining Foundry web search with the Microsoft Learn MCP tools for the declarative InvokeFoundryToolboxMcp sample.")).Value;
Console.WriteLine($"Created toolbox '{created.Name}' v{created.Version} ({created.Tools.Count} tool(s))");
return $"{foundryEndpoint.ToString().TrimEnd('/')}/toolboxes";
}
private static string BuildToolboxMcpServerUrl(string toolboxEndpoint, string toolboxName, string apiVersion) =>
$"{toolboxEndpoint.TrimEnd('/')}/{toolboxName}/mcp?api-version={Uri.EscapeDataString(apiVersion)}";
private sealed class FoundryToolboxBearerTokenHandler(TokenCredential credential) : DelegatingHandler
{
private static readonly TokenRequestContext s_tokenContext =
new(["https://ai.azure.com/.default"]);
protected override async Task<HttpResponseMessage> SendAsync(
HttpRequestMessage request,
CancellationToken cancellationToken)
{
AccessToken token = await credential.GetTokenAsync(s_tokenContext, cancellationToken);
request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", token.Token);
return await base.SendAsync(request, cancellationToken);
}
}
private sealed class FoundryFeaturesPolicy(string feature) : PipelinePolicy
{
private const string FeatureHeader = "Foundry-Features";
public override void Process(PipelineMessage message, IReadOnlyList<PipelinePolicy> pipeline, int currentIndex)
{
message.Request.Headers.Add(FeatureHeader, feature);
ProcessNext(message, pipeline, currentIndex);
}
public override ValueTask ProcessAsync(PipelineMessage message, IReadOnlyList<PipelinePolicy> pipeline, int currentIndex)
{
message.Request.Headers.Add(FeatureHeader, feature);
return ProcessNextAsync(message, pipeline, currentIndex);
}
}
}
@@ -17,6 +17,7 @@ namespace Microsoft.Agents.AI;
/// <see cref="HarnessAgent"/> assembles the following pipeline from a caller-supplied <see cref="IChatClient"/>:
/// <list type="number">
/// <item><description><see cref="FunctionInvokingChatClient"/> — automatic function/tool invocation.</description></item>
/// <item><description><see cref="MessageInjectingChatClient"/> — allows external code to inject messages into the conversation mid-stream.</description></item>
/// <item><description><see cref="PerServiceCallChatHistoryPersistingChatClient"/> — persists chat history after every individual service call within a function-invocation loop.</description></item>
/// <item><description><see cref="AIContextProviderChatClient"/> with a <see cref="CompactionProvider"/> — applies context-window compaction before each call so long function-invocation loops do not overflow the context window.</description></item>
/// </list>
@@ -110,6 +111,7 @@ public sealed class HarnessAgent : DelegatingAIAgent
return chatClient
.AsBuilder()
.UseFunctionInvocation()
.UseMessageInjection()
.UsePerServiceCallChatHistoryPersistence()
.UseAIContextProviders(compactionProvider)
.BuildAIAgent(new ChatClientAgentOptions
@@ -3,12 +3,15 @@
using System;
using System.Collections.Generic;
using System.Globalization;
using System.IO;
using System.Linq;
using System.Net.Http;
using System.Text;
using System.Text.Json;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.AI;
using Microsoft.Shared.Diagnostics;
using ModelContextProtocol.Client;
using ModelContextProtocol.Protocol;
@@ -24,6 +27,14 @@ namespace Microsoft.Agents.AI.Workflows.Declarative.Mcp;
/// </remarks>
public sealed class DefaultMcpToolHandler : IMcpToolHandler, IAsyncDisposable
{
/// <summary>
/// Reserved <c>toolName</c> value that maps an <see cref="IMcpToolHandler.InvokeToolAsync"/> request
/// to the MCP protocol <c>tools/list</c> discovery operation.
/// </summary>
public const string ListToolsToolName = "tools/list";
private static readonly JsonWriterOptions s_toolListJsonWriterOptions = new() { Indented = true };
private readonly Func<string, CancellationToken, Task<HttpClient?>>? _httpClientProvider;
private readonly Dictionary<string, McpClient> _clients = [];
private readonly Dictionary<string, HttpClient> _ownedHttpClients = [];
@@ -53,9 +64,18 @@ public sealed class DefaultMcpToolHandler : IMcpToolHandler, IAsyncDisposable
CancellationToken cancellationToken = default)
{
// TODO: Handle connectionName and server label appropriately when Hosted scenario supports them. For now, ignore
McpServerToolResultContent resultContent = new(Guid.NewGuid().ToString());
if (IsListToolsToolName(toolName))
{
ThrowIfListToolsArgumentsSpecified(arguments);
McpClient listToolsClient = await this.GetOrCreateClientAsync(serverUrl, serverLabel, headers, cancellationToken).ConfigureAwait(false);
IList<McpClientTool> tools = await listToolsClient.ListToolsAsync(cancellationToken: cancellationToken).ConfigureAwait(false);
return CreateListToolsResultContent(tools.Select(tool => tool.ProtocolTool));
}
McpClient client = await this.GetOrCreateClientAsync(serverUrl, serverLabel, headers, cancellationToken).ConfigureAwait(false);
McpServerToolResultContent resultContent = new(Guid.NewGuid().ToString());
// Convert IDictionary to IReadOnlyDictionary for CallToolAsync
IReadOnlyDictionary<string, object?>? readOnlyArguments = arguments is null
? null
@@ -72,6 +92,23 @@ public sealed class DefaultMcpToolHandler : IMcpToolHandler, IAsyncDisposable
return resultContent;
}
internal static bool IsListToolsToolName(string toolName) =>
string.Equals(toolName, ListToolsToolName, StringComparison.Ordinal);
internal static McpServerToolResultContent CreateListToolsResultContent(IEnumerable<Tool> tools)
{
Throw.IfNull(tools);
McpServerToolResultContent resultContent = new(Guid.NewGuid().ToString())
{
Outputs = []
};
resultContent.Outputs.Add(new TextContent(SerializeToolsList(tools)));
return resultContent;
}
/// <inheritdoc/>
public async ValueTask DisposeAsync()
{
@@ -183,6 +220,16 @@ public sealed class DefaultMcpToolHandler : IMcpToolHandler, IAsyncDisposable
return hashCode.ToString(CultureInfo.InvariantCulture);
}
private static void ThrowIfListToolsArgumentsSpecified(IDictionary<string, object?>? arguments)
{
if (arguments is { Count: > 0 })
{
throw new ArgumentException(
$"The reserved MCP '{ListToolsToolName}' operation does not accept tool arguments.",
nameof(arguments));
}
}
private static void PopulateResultContent(McpServerToolResultContent resultContent, CallToolResult result)
{
// Ensure Outputs list is initialized
@@ -230,6 +277,17 @@ public sealed class DefaultMcpToolHandler : IMcpToolHandler, IAsyncDisposable
TextContentBlock text => new TextContent(text.Text),
ImageContentBlock image => CreateDataContent(image.Data, image.MimeType ?? "image/*"),
AudioContentBlock audio => CreateDataContent(audio.Data, audio.MimeType ?? "audio/*"),
EmbeddedResourceBlock embedded => ConvertEmbeddedResource(embedded),
_ => new TextContent(block.ToString() ?? string.Empty),
};
}
private static AIContent ConvertEmbeddedResource(EmbeddedResourceBlock block)
{
return block.Resource switch
{
TextResourceContents text => new TextContent(text.Text),
BlobResourceContents blob => CreateDataContent(blob.Blob, blob.MimeType ?? "application/octet-stream"),
_ => new TextContent(block.ToString() ?? string.Empty),
};
}
@@ -255,4 +313,39 @@ public sealed class DefaultMcpToolHandler : IMcpToolHandler, IAsyncDisposable
return new DataContent($"data:{mediaType};base64,{base64}", mediaType);
}
private static string SerializeToolsList(IEnumerable<Tool> tools)
{
using MemoryStream stream = new();
using (Utf8JsonWriter writer = new(stream, s_toolListJsonWriterOptions))
{
writer.WriteStartObject();
writer.WriteStartArray("tools");
foreach (Tool tool in tools)
{
writer.WriteStartObject();
writer.WriteString("name", tool.Name);
writer.WriteString("description", tool.Description);
writer.WritePropertyName("inputSchema");
tool.InputSchema.WriteTo(writer);
writer.WritePropertyName("outputSchema");
if (tool.OutputSchema is JsonElement outputSchema)
{
outputSchema.WriteTo(writer);
}
else
{
writer.WriteNullValue();
}
writer.WriteEndObject();
}
writer.WriteEndArray();
writer.WriteEndObject();
}
return Encoding.UTF8.GetString(stream.GetBuffer(), 0, (int)stream.Length);
}
}
@@ -54,6 +54,8 @@ public class HandoffWorkflowBuilderCore<TBuilder> where TBuilder : HandoffWorkfl
private bool _emitAgentResponseUpdateEvents;
private HandoffToolCallFilteringBehavior _toolCallFilteringBehavior = HandoffToolCallFilteringBehavior.HandoffOnly;
private bool _returnToPrevious;
private string? _name;
private string? _description;
/// <summary>
/// Initializes a new instance of the <see cref="HandoffsWorkflowBuilder"/> class with no handoff relationships.
@@ -97,6 +99,20 @@ public class HandoffWorkflowBuilderCore<TBuilder> where TBuilder : HandoffWorkfl
return (TBuilder)this;
}
/// <inheritdoc cref="WorkflowBuilder.WithName(string)"/>
public TBuilder WithName(string name)
{
this._name = name;
return (TBuilder)this;
}
/// <inheritdoc cref="WorkflowBuilder.WithDescription(string)"/>
public TBuilder WithDescription(string description)
{
this._description = description;
return (TBuilder)this;
}
/// <summary>
/// Sets a value indicating whether agent streaming update events should be emitted during execution.
/// If <see langword="null"/>, the value will be taken from the <see cref="TurnToken"/>
@@ -330,7 +346,16 @@ public class HandoffWorkflowBuilderCore<TBuilder> where TBuilder : HandoffWorkfl
builder.AddEdge(start, executors[this._initialAgent.Id]);
}
// Build the workflow.
if (!string.IsNullOrWhiteSpace(this._name))
{
builder.WithName(this._name);
}
if (!string.IsNullOrWhiteSpace(this._description))
{
builder.WithDescription(this._description);
}
return builder.WithOutputFrom(end).Build();
}
}
@@ -1,5 +1,6 @@
// Copyright (c) Microsoft. All rights reserved.
using System;
using System.Collections.Generic;
using System.Diagnostics.CodeAnalysis;
using System.Threading.Tasks;
@@ -140,7 +141,15 @@ public class MagenticWorkflowBuilder(AIAgent managerAgent)
}
/// <inheritdoc cref="WorkflowBuilder.Build"/>
public Workflow Build() => this.ReduceToWorkflowBuilder().Build();
public Workflow Build()
{
if (this._team.Count == 0)
{
throw new InvalidOperationException("At least one participant must be added via AddParticipants() before building the workflow.");
}
return this.ReduceToWorkflowBuilder().Build();
}
private TaskLimits Limits => new(
MaxRoundCount: this._maxRounds,
@@ -101,6 +101,7 @@ internal class MagenticOrchestrator(AIAgent managerAgent, List<AIAgent> team, Ta
return base.ConfigureProtocol(protocolBuilder)
.SendsMessage<ChatMessage>()
.SendsMessage<ResetChatSignal>()
.YieldsOutput<List<ChatMessage>>()
.ConfigureRoutes(ConfigureRoutes);
void ConfigureRoutes(RouteBuilder routeBuilder) => routeBuilder.AddPortHandler<MagenticPlanReviewRequest, MagenticPlanReviewResponse>(
@@ -109,7 +110,7 @@ internal class MagenticOrchestrator(AIAgent managerAgent, List<AIAgent> team, Ta
out this._planReviewPort);
}
private ValueTask SubmitPlanReviewRequestAsync(MagenticTaskContext taskContext, IWorkflowContext workflowContext)
private ValueTask SubmitPlanReviewRequestAsync(MagenticTaskContext taskContext, IWorkflowContext workflowContext, bool replanAfterStall = false)
{
MagenticProgressLedger? progressLedger = taskContext.ProgressLedger;
if (progressLedger?.IsStarted is not true)
@@ -117,7 +118,7 @@ internal class MagenticOrchestrator(AIAgent managerAgent, List<AIAgent> team, Ta
progressLedger = null;
}
MagenticPlanReviewRequest request = new(taskContext.TaskLedger!.CurrentPlan, progressLedger, taskContext.IsStalled);
MagenticPlanReviewRequest request = new(taskContext.TaskLedger!.CurrentPlan, progressLedger, replanAfterStall);
return this._planReviewPort!.PostRequestAsync(request);
}
@@ -146,7 +147,7 @@ internal class MagenticOrchestrator(AIAgent managerAgent, List<AIAgent> team, Ta
if (this._taskContext.IsTerminated)
{
throw new InvalidOperationException("Magentic Orchestration has already been terminated and cannot process new messages. Please start a new session.");
throw new InvalidOperationException("This Magentic orchestration has already terminated. To process new messages, create a new workflow instance.");
}
if (response.IsApproved)
@@ -161,7 +162,7 @@ internal class MagenticOrchestrator(AIAgent managerAgent, List<AIAgent> team, Ta
}
}
private async ValueTask UpdatePlanAndDelegateAsync(MagenticTaskContext taskContext, IWorkflowContext context, CancellationToken cancellationToken)
private async ValueTask UpdatePlanAndDelegateAsync(MagenticTaskContext taskContext, IWorkflowContext context, CancellationToken cancellationToken, bool replanAfterStall = false)
{
bool isReplan = taskContext.TaskLedger != null;
@@ -177,7 +178,7 @@ internal class MagenticOrchestrator(AIAgent managerAgent, List<AIAgent> team, Ta
if (requirePlanSignoff)
{
await this.SubmitPlanReviewRequestAsync(taskContext, context).ConfigureAwait(false);
await this.SubmitPlanReviewRequestAsync(taskContext, context, replanAfterStall).ConfigureAwait(false);
}
else
{
@@ -187,9 +188,22 @@ internal class MagenticOrchestrator(AIAgent managerAgent, List<AIAgent> team, Ta
protected override async ValueTask TakeTurnAsync(List<ChatMessage> messages, IWorkflowContext context, bool? emitEvents, CancellationToken cancellationToken = default)
{
// First Turn: Initialize the task context and send the initial messages to the planner agent
this._taskContext ??= new(messages, team, limits, emitEvents, []);
await this.UpdatePlanAndDelegateAsync(this._taskContext, context, cancellationToken).ConfigureAwait(false);
if (this._taskContext?.IsTerminated == true)
{
throw new InvalidOperationException("This Magentic orchestration has already terminated. To process new messages, create a new workflow instance.");
}
if (this._taskContext == null)
{
// First Turn: Initialize the task context and create the initial plan
this._taskContext = new(messages, team, limits, emitEvents, []);
await this.UpdatePlanAndDelegateAsync(this._taskContext, context, cancellationToken).ConfigureAwait(false);
}
else
{
// Subsequent turns: agent returned control, go directly to coordination (progress ledger only, no replan)
await this.RunCoordinationRoundAsync(this._taskContext, context, cancellationToken).ConfigureAwait(false);
}
}
private ChatMessage? _fullTaskLedgerMessage;
@@ -288,10 +302,11 @@ internal class MagenticOrchestrator(AIAgent managerAgent, List<AIAgent> team, Ta
private async ValueTask ResetAndReplanAsync(MagenticTaskContext taskContext, IWorkflowContext context, CancellationToken cancellationToken)
{
bool wasStalled = taskContext.IsStalled;
taskContext.Reset();
await context.SendMessageAsync(new ResetChatSignal(), cancellationToken: cancellationToken).ConfigureAwait(false);
await this.UpdatePlanAndDelegateAsync(taskContext, context, cancellationToken).ConfigureAwait(false);
await this.UpdatePlanAndDelegateAsync(taskContext, context, cancellationToken, replanAfterStall: wasStalled).ConfigureAwait(false);
}
private async ValueTask PrepareFinalAnswerAsync(MagenticTaskContext taskContext, IWorkflowContext context, CancellationToken cancellationToken)
@@ -58,7 +58,7 @@ internal class MagenticTaskContext(List<ChatMessage> taskDefinition, List<AIAgen
public bool IsTerminated { get; internal set; }
public bool IsStalled => this.TaskCounters.StallCount >= this.TaskLimits.MaxStallCount;
public bool IsStalled => this.TaskCounters.StallCount > this.TaskLimits.MaxStallCount;
public (bool HitRoundLimit, bool HitResetLimit) CheckLimits()
{
@@ -36,9 +36,13 @@ public sealed class SwitchBuilder
Throw.IfNull(executors);
HashSet<int> indicies = [];
int executorIndex = 0;
foreach (ExecutorBinding executor in executors)
{
// Explicit name: null element inside the collection argument.
Throw.IfNull(executor, $"{nameof(executors)}[{executorIndex++}]");
if (!this._executorIndicies.TryGetValue(executor.Id, out int index))
{
index = this._executors.Count;
@@ -64,8 +68,13 @@ public sealed class SwitchBuilder
{
Throw.IfNull(executors);
int executorIndex = 0;
foreach (ExecutorBinding executor in executors)
{
// Explicit name: null element inside the collection argument.
Throw.IfNull(executor, $"{nameof(executors)}[{executorIndex++}]");
if (!this._executorIndicies.TryGetValue(executor.Id, out int index))
{
index = this._executors.Count;
@@ -25,7 +25,11 @@ public static class WorkflowBuilderExtensions
/// <param name="target">The target executor to which messages will be forwarded.</param>
/// <returns>The updated <see cref="WorkflowBuilder"/> instance.</returns>
public static WorkflowBuilder ForwardMessage<TMessage>(this WorkflowBuilder builder, ExecutorBinding source, ExecutorBinding target)
=> builder.ForwardMessage<TMessage>(source, [target], condition: null);
{
Throw.IfNull(target, nameof(target));
return builder.ForwardMessage<TMessage>(source, [target], condition: null);
}
/// <summary>
/// Adds edges to the workflow that forward messages of the specified type from the source executor to
@@ -52,6 +56,8 @@ public static class WorkflowBuilderExtensions
/// <returns>The updated <see cref="WorkflowBuilder"/> instance.</returns>
public static WorkflowBuilder ForwardMessage<TMessage>(this WorkflowBuilder builder, ExecutorBinding source, IEnumerable<ExecutorBinding> targets, Func<TMessage, bool>? condition = null)
{
Throw.IfNull(builder);
Throw.IfNull(source);
Throw.IfNull(targets);
Func<object?, bool> predicate = WorkflowBuilder.CreateConditionFunc<TMessage>(IsAllowedTypeAndMatchingCondition)!;
@@ -62,14 +68,16 @@ public static class WorkflowBuilderExtensions
if (targets is ICollection<ExecutorBinding> { Count: 1 })
#endif
{
return builder.AddEdge(source, targets.First(), predicate);
return builder.AddEdge(source, Throw.IfNull(targets.First(), nameof(targets)), predicate);
}
return builder.AddSwitch(source, (switch_) => switch_.AddCase(predicate, targets));
return builder.AddSwitch(source, (switch_) => switch_.AddCase(predicate, targets.Select(ValidateTarget)));
// The reason we can check for "not null" here is that CreateConditionFunc<T> will do the correct unwrapping
// logic for PortableValues.
bool IsAllowedTypeAndMatchingCondition(TMessage? message) => message != null && (condition == null || condition(message));
ExecutorBinding ValidateTarget(ExecutorBinding target) => Throw.IfNull(target, nameof(targets));
}
/// <summary>
@@ -81,7 +89,11 @@ public static class WorkflowBuilderExtensions
/// <param name="target">The target executor to which messages, except those of type <typeparamref name="TMessage"/>, will be forwarded.</param>
/// <returns>The updated <see cref="WorkflowBuilder"/> instance with the added edges.</returns>
public static WorkflowBuilder ForwardExcept<TMessage>(this WorkflowBuilder builder, ExecutorBinding source, ExecutorBinding target)
=> builder.ForwardExcept<TMessage>(source, [target]);
{
Throw.IfNull(target, nameof(target));
return builder.ForwardExcept<TMessage>(source, [target]);
}
/// <summary>
/// Adds edges from the specified source to the provided executors, excluding messages of a specified type.
@@ -93,6 +105,8 @@ public static class WorkflowBuilderExtensions
/// <returns>The updated <see cref="WorkflowBuilder"/> instance with the added edges.</returns>
public static WorkflowBuilder ForwardExcept<TMessage>(this WorkflowBuilder builder, ExecutorBinding source, IEnumerable<ExecutorBinding> targets)
{
Throw.IfNull(builder);
Throw.IfNull(source);
Throw.IfNull(targets);
Func<object?, bool> predicate = WorkflowBuilder.CreateConditionFunc<TMessage>((Func<object?, bool>)IsAllowedType)!;
@@ -103,14 +117,16 @@ public static class WorkflowBuilderExtensions
if (targets is ICollection<ExecutorBinding> { Count: 1 })
#endif
{
return builder.AddEdge(source, targets.First(), predicate);
return builder.AddEdge(source, Throw.IfNull(targets.First(), nameof(targets)), predicate);
}
return builder.AddSwitch(source, (switch_) => switch_.AddCase(predicate, targets));
return builder.AddSwitch(source, (switch_) => switch_.AddCase(predicate, targets.Select(ValidateTarget)));
// The reason we can check for "null" here is that CreateConditionFunc<T> will do the correct unwrapping
// logic for PortableValues.
static bool IsAllowedType(object? message) => message is null;
ExecutorBinding ValidateTarget(ExecutorBinding target) => Throw.IfNull(target, nameof(targets));
}
/// <summary>
@@ -129,6 +145,7 @@ public static class WorkflowBuilderExtensions
{
Throw.IfNull(builder);
Throw.IfNull(source);
Throw.IfNull(executors);
HashSet<string> seenExecutors = [source.Id];
@@ -103,6 +103,30 @@ public class HostApplicationBuilderWorkflowExtensionsTests
Assert.Contains(workflowDescriptors, d => (string)d.ServiceKey! == "workflow3");
}
/// <summary>
/// Verifies that a handoff workflow can be named from the DI workflow key.
/// </summary>
[Fact]
public void AddWorkflow_HandoffWorkflowWithName_ResolvesWorkflow()
{
var builder = new HostApplicationBuilder();
const string WorkflowName = "handoffWorkflow";
var mockAgent = new Mock<AIAgent>();
mockAgent.Setup(a => a.Name).Returns("handoffAgent");
#pragma warning disable MAAIW001 // This test covers hosting handoff workflows.
builder.AddWorkflow(WorkflowName, (sp, key) =>
AgentWorkflowBuilder.CreateHandoffBuilderWith(mockAgent.Object)
.WithName(key)
.Build());
#pragma warning restore MAAIW001
var workflow = builder.Build().Services.GetRequiredKeyedService<Workflow>(WorkflowName);
Assert.Equal(WorkflowName, workflow.Name);
}
/// <summary>
/// Verifies that AddWorkflow handles empty strings for name.
/// </summary>
@@ -4,6 +4,7 @@ using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Text;
using System.Text.Json;
using System.Threading;
using System.Threading.Tasks;
using FluentAssertions;
@@ -320,6 +321,92 @@ public sealed class DefaultMcpToolHandlerTests
#endregion
#region Reserved Tools/List Tests
[Fact]
public void IsListToolsToolName_WithReservedName_ShouldReturnTrue()
{
// Act
bool result = DefaultMcpToolHandler.IsListToolsToolName(DefaultMcpToolHandler.ListToolsToolName);
// Assert
result.Should().BeTrue();
}
[Fact]
public void IsListToolsToolName_WithRegularToolName_ShouldReturnFalse()
{
// Act
bool result = DefaultMcpToolHandler.IsListToolsToolName("search");
// Assert
result.Should().BeFalse();
}
[Fact]
public async Task InvokeToolAsync_WithListToolsArguments_ShouldThrowArgumentExceptionAsync()
{
// Arrange
DefaultMcpToolHandler handler = new();
try
{
// Act
Func<Task> act = async () => await handler.InvokeToolAsync(
serverUrl: "http://localhost:12345/mcp",
serverLabel: "test",
toolName: DefaultMcpToolHandler.ListToolsToolName,
arguments: new Dictionary<string, object?> { ["ignored"] = true },
headers: null,
connectionName: null);
// Assert
await act.Should().ThrowAsync<ArgumentException>()
.WithMessage("*does not accept tool arguments*");
}
finally
{
await handler.DisposeAsync();
}
}
[Fact]
public async Task CreateListToolsResultContent_WithTools_ShouldSerializeToolMetadataAsync()
{
// Arrange
JsonElement inputSchema = JsonSerializer.Deserialize<JsonElement>(
"""
{
"type": "object",
"properties": {
"query": {
"type": "string"
}
},
"required": [ "query" ]
}
""");
Tool tool = new()
{
Name = "search",
Description = "Searches documentation.",
InputSchema = inputSchema
};
// Act
McpServerToolResultContent result = DefaultMcpToolHandler.CreateListToolsResultContent([tool]);
// Assert
TextContent text = result.Outputs.Should().ContainSingle().Subject.Should().BeOfType<TextContent>().Subject;
using JsonDocument document = JsonDocument.Parse(text.Text);
JsonElement listedTool = document.RootElement.GetProperty("tools")[0];
listedTool.GetProperty("name").GetString().Should().Be("search");
listedTool.GetProperty("description").GetString().Should().Be("Searches documentation.");
listedTool.GetProperty("inputSchema").GetProperty("properties").GetProperty("query").GetProperty("type").GetString().Should().Be("string");
}
#endregion
#region Interface Implementation Tests
[Fact]
@@ -488,5 +575,75 @@ public sealed class DefaultMcpToolHandlerTests
dataContent.MediaType.Should().Be("audio/*");
}
[Fact]
public void ConvertContentBlock_EmbeddedResourceBlock_WithTextResource_ShouldReturnTextContent()
{
// Arrange
EmbeddedResourceBlock block = new()
{
Resource = new TextResourceContents
{
Text = "embedded text payload",
Uri = "resource://example",
MimeType = "text/plain",
},
};
// Act
AIContent result = DefaultMcpToolHandler.ConvertContentBlock(block);
// Assert
result.Should().BeOfType<TextContent>()
.Which.Text.Should().Be("embedded text payload");
}
[Fact]
public void ConvertContentBlock_EmbeddedResourceBlock_WithBlobResource_ShouldReturnDataContent()
{
// Arrange
byte[] base64Bytes = Encoding.UTF8.GetBytes("UklGRiQA");
EmbeddedResourceBlock block = new()
{
Resource = new BlobResourceContents
{
Blob = new ReadOnlyMemory<byte>(base64Bytes),
Uri = "resource://example.bin",
MimeType = "application/zip",
},
};
// Act
AIContent result = DefaultMcpToolHandler.ConvertContentBlock(block);
// Assert
DataContent dataContent = result.Should().BeOfType<DataContent>().Subject;
dataContent.MediaType.Should().Be("application/zip");
dataContent.Uri.Should().Be("data:application/zip;base64,UklGRiQA");
}
[Fact]
public void ConvertContentBlock_EmbeddedResourceBlock_WithBlobResource_NullMimeType_DefaultsToOctetStream()
{
// Arrange
byte[] base64Bytes = Encoding.UTF8.GetBytes("UklGRiQA");
EmbeddedResourceBlock block = new()
{
Resource = new BlobResourceContents
{
Blob = new ReadOnlyMemory<byte>(base64Bytes),
Uri = "resource://example.bin",
MimeType = null!,
},
};
// Act
AIContent result = DefaultMcpToolHandler.ConvertContentBlock(block);
// Assert
DataContent dataContent = result.Should().BeOfType<DataContent>().Subject;
dataContent.MediaType.Should().Be("application/octet-stream");
dataContent.Uri.Should().Be("data:application/octet-stream;base64,UklGRiQA");
}
#endregion
}
@@ -432,6 +432,44 @@ public sealed class InvokeMcpToolExecutorTest(ITestOutputHelper output) : Workfl
VerifyInvocationEvent(events);
}
[Fact]
public async Task InvokeMcpToolExecuteWithReservedListToolsNameAsync()
{
// Arrange
this.State.InitializeSystem();
const string ListToolsToolName = "tools/list";
string? capturedToolName = null;
InvokeMcpTool model = this.CreateModel(
displayName: nameof(InvokeMcpToolExecuteWithReservedListToolsNameAsync),
serverUrl: TestServerUrl,
toolName: ListToolsToolName);
Mock<IMcpToolHandler> mockProvider = new();
mockProvider.Setup(provider => provider.InvokeToolAsync(
It.IsAny<string>(),
It.IsAny<string?>(),
It.IsAny<string>(),
It.IsAny<IDictionary<string, object?>?>(),
It.IsAny<IDictionary<string, string>?>(),
It.IsAny<string?>(),
It.IsAny<CancellationToken>()))
.Callback<string, string?, string, IDictionary<string, object?>?, IDictionary<string, string>?, string?, CancellationToken>(
(_, _, toolName, _, _, _, _) => capturedToolName = toolName)
.ReturnsAsync(new McpServerToolResultContent("list-tools-call-id")
{
Outputs = [new TextContent("{\"tools\":[]}")]
});
MockAgentProvider mockAgentProvider = new();
InvokeMcpToolExecutor action = new(model, mockProvider.Object, mockAgentProvider.Object, this.State);
// Act
WorkflowEvent[] events = await this.ExecuteAsync(action, isDiscrete: false);
// Assert
VerifyModel(model, action);
VerifyInvocationEvent(events);
Assert.Equal(ListToolsToolName, capturedToolName);
}
[Fact]
public async Task InvokeMcpToolExecuteWithMultipleContentTypesAsync()
{
@@ -86,6 +86,23 @@ public class HandoffOrchestrationTests
target.Reason.Should().Be("instructions");
}
[Fact]
public void BuildHandoffs_WithNameAndDescription_SetsWorkflowMetadata()
{
const string WorkflowName = "handoff-workflow";
const string WorkflowDescription = "A handoff workflow";
DoubleEchoAgent agent = new("agent");
var workflow = AgentWorkflowBuilder.CreateHandoffBuilderWith(agent)
.WithName(WorkflowName)
.WithDescription(WorkflowDescription)
.Build();
Assert.Equal(WorkflowName, workflow.Name);
Assert.Equal(WorkflowDescription, workflow.Description);
}
[Fact]
public async Task Handoffs_NoTransfers_ResponseServedByOriginalAgentAsync()
{
@@ -34,7 +34,13 @@ public sealed class InputWaiterTests : IDisposable
[Fact]
public async Task InputWaiter_WaitForInputAsync_BlocksUntilSignaledAsync()
{
Task waitTask = this._waiter.WaitForInputAsync(TimeSpan.FromSeconds(5));
// Use the no-timeout overload so that the wait can only be released by SignalInput.
// A finite timeout would make this test's logic racy: the component correctly
// honors the timeout, but if the test thread is starved of CPU time (CI load,
// GC pause) long enough for the timeout to fire, waitTask completes before
// SignalInput is called and the "should not complete before signaled" assertion
// flakes. Timeout behavior is covered separately below.
Task waitTask = this._waiter.WaitForInputAsync(CancellationToken.None);
Task completedBeforeSignal = await Task.WhenAny(waitTask, Task.Delay(100));
completedBeforeSignal.Should().NotBeSameAs(
@@ -100,6 +106,21 @@ public sealed class InputWaiterTests : IDisposable
this._waiter.SignalInput();
await this._waiter.WaitForInputAsync(TimeSpan.FromSeconds(1));
}
[Fact]
public async Task InputWaiter_WaitForInputAsync_CompletesWhenTimeoutExpiresAsync()
{
// Verify that a finite timeout releases the block even without a signal.
// We only assert that it *does* complete (within a generous outer bound);
// we intentionally do not assert that it stays blocked until the timeout,
// because that would re-introduce the same wall-clock flakiness
// described in BlocksUntilSignaledAsync (see comment on that test).
Task waitTask = this._waiter.WaitForInputAsync(TimeSpan.FromMilliseconds(300));
Task completed = await Task.WhenAny(waitTask, Task.Delay(TimeSpan.FromSeconds(5)));
completed.Should().BeSameAs(waitTask, "the wait task should complete once the timeout expires");
await waitTask;
}
}
public class OutputFilterTests
@@ -133,31 +133,31 @@ public sealed class ObservabilityTests : IDisposable
activityEvents.Should().Contain(e => e.Name == EventNames.WorkflowCompleted, "activity should have workflow completed event");
}
[Fact(Skip = "Flaky test - temporarily disabled.")]
[Fact]
public async Task CreatesWorkflowEndToEndActivities_WithCorrectName_DefaultAsync()
{
await this.TestWorkflowEndToEndActivitiesAsync("Default");
}
[Fact(Skip = "Flaky test - temporarily disabled.")]
[Fact]
public async Task CreatesWorkflowEndToEndActivities_WithCorrectName_OffThreadAsync()
{
await this.TestWorkflowEndToEndActivitiesAsync("OffThread");
}
[Fact(Skip = "Flaky test - temporarily disabled.")]
[Fact]
public async Task CreatesWorkflowEndToEndActivities_WithCorrectName_ConcurrentAsync()
{
await this.TestWorkflowEndToEndActivitiesAsync("Concurrent");
}
[Fact(Skip = "Flaky test - temporarily disabled.")]
[Fact]
public async Task CreatesWorkflowEndToEndActivities_WithCorrectName_LockstepAsync()
{
await this.TestWorkflowEndToEndActivitiesAsync("Lockstep");
}
[Fact(Skip = "Flaky test - temporarily disabled.")]
[Fact]
public async Task CreatesWorkflowActivities_WithCorrectNameAsync()
{
// Arrange
@@ -182,7 +182,7 @@ public sealed class ObservabilityTests : IDisposable
tags.Should().ContainKey(Tags.WorkflowDefinition);
}
[Fact(Skip = "Flaky test - temporarily disabled.")]
[Fact]
public async Task TelemetryDisabledByDefault_CreatesNoActivitiesAsync()
{
// Arrange
@@ -200,7 +200,7 @@ public sealed class ObservabilityTests : IDisposable
capturedActivities.Should().BeEmpty("No activities should be created when telemetry is disabled (default).");
}
[Fact(Skip = "Flaky test - temporarily disabled.")]
[Fact]
public async Task WithOpenTelemetry_UsesProvidedActivitySourceAsync()
{
// Arrange
@@ -235,7 +235,7 @@ public sealed class ObservabilityTests : IDisposable
"All activities should come from the user-provided ActivitySource.");
}
[Fact(Skip = "Flaky test - temporarily disabled.")]
[Fact]
public async Task DisableWorkflowBuild_PreventsWorkflowBuildActivityAsync()
{
// Arrange
@@ -255,7 +255,7 @@ public sealed class ObservabilityTests : IDisposable
"WorkflowBuild activity should be disabled.");
}
[Fact(Skip = "Flaky test - temporarily disabled.")]
[Fact]
public async Task DisableWorkflowRun_PreventsWorkflowRunActivityAsync()
{
// Arrange
@@ -285,7 +285,7 @@ public sealed class ObservabilityTests : IDisposable
"Other activities should still be created.");
}
[Fact(Skip = "Flaky test - temporarily disabled.")]
[Fact]
public async Task DisableExecutorProcess_PreventsExecutorProcessActivityAsync()
{
// Arrange
@@ -312,7 +312,7 @@ public sealed class ObservabilityTests : IDisposable
"Other activities should still be created.");
}
[Fact(Skip = "Flaky test - temporarily disabled.")]
[Fact]
public async Task DisableEdgeGroupProcess_PreventsEdgeGroupProcessActivityAsync()
{
// Arrange
@@ -333,7 +333,7 @@ public sealed class ObservabilityTests : IDisposable
"Other activities should still be created.");
}
[Fact(Skip = "Flaky test - temporarily disabled.")]
[Fact]
public async Task DisableMessageSend_PreventsMessageSendActivityAsync()
{
// Arrange
@@ -382,7 +382,7 @@ public sealed class ObservabilityTests : IDisposable
return builder.WithOpenTelemetry(configure: opts => opts.DisableMessageSend = true).Build();
}
[Fact(Skip = "Flaky test - temporarily disabled.")]
[Fact]
public async Task EnableSensitiveData_LogsExecutorInputAndOutputAsync()
{
// Arrange
@@ -413,7 +413,7 @@ public sealed class ObservabilityTests : IDisposable
tags[Tags.ExecutorOutput].Should().Contain("HELLO", "Output should contain the transformed value.");
}
[Fact(Skip = "Flaky test - temporarily disabled.")]
[Fact]
public async Task EnableSensitiveData_Disabled_DoesNotLogInputOutputAsync()
{
// Arrange
@@ -442,7 +442,7 @@ public sealed class ObservabilityTests : IDisposable
tags.Should().NotContainKey(Tags.ExecutorOutput, "Output should NOT be logged when EnableSensitiveData is false.");
}
[Fact(Skip = "Flaky test - temporarily disabled.")]
[Fact]
public async Task EnableSensitiveData_LogsMessageSendContentAsync()
{
// Arrange
@@ -474,7 +474,7 @@ public sealed class ObservabilityTests : IDisposable
tags.Should().ContainKey(Tags.MessageSourceId, "Source ID should be logged.");
}
[Fact(Skip = "Flaky test - temporarily disabled.")]
[Fact]
public async Task EnableSensitiveData_Disabled_DoesNotLogMessageContentAsync()
{
// Arrange
@@ -0,0 +1,546 @@
// Copyright (c) Microsoft. All rights reserved.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using FluentAssertions;
using Microsoft.Agents.AI.Workflows.Execution;
namespace Microsoft.Agents.AI.Workflows.UnitTests;
public sealed class RouteBuilderTests
{
public enum HandlerOverload
{
SyncWithCancellation = 0,
SyncWithoutCancellation = 1,
AsyncWithCancellation = 2,
AsyncWithoutCancellation = 3,
}
private sealed record TestPayload(string Value);
private sealed class HandlerInvocation
{
public object? Message { get; private set; }
public IWorkflowContext? Context { get; private set; }
public CancellationToken CancellationToken { get; private set; }
public int InvocationCount { get; private set; }
public void Capture(object? message, IWorkflowContext context, CancellationToken cancellationToken = default)
{
this.Message = message;
this.Context = context;
this.CancellationToken = cancellationToken;
this.InvocationCount++;
}
}
private sealed class TestExternalRequestContext : IExternalRequestContext, IExternalRequestSink
{
public List<RequestPort> RegisteredPorts { get; } = [];
public List<ExternalRequest> PostedRequests { get; } = [];
public IExternalRequestSink RegisterPort(RequestPort port)
{
this.RegisteredPorts.Add(port);
return this;
}
public ValueTask PostAsync(ExternalRequest request)
{
this.PostedRequests.Add(request);
return default;
}
}
[Theory]
[InlineData(HandlerOverload.SyncWithCancellation)]
[InlineData(HandlerOverload.SyncWithoutCancellation)]
[InlineData(HandlerOverload.AsyncWithCancellation)]
[InlineData(HandlerOverload.AsyncWithoutCancellation)]
public async Task AddHandler_VoidOverloads_RouteExpectedMessageAsync(HandlerOverload overload)
{
// Arrange
RouteBuilder routeBuilder = new(null);
HandlerInvocation invocation = new();
CancellationToken cancellationToken = new CancellationTokenSource().Token;
RegisterVoidHandler(routeBuilder, invocation, overload);
MessageRouter router = routeBuilder.Build();
TestWorkflowContext context = new("executor");
// Act
CallResult? result = await router.RouteMessageAsync("hello", context, cancellationToken: cancellationToken);
// Assert
result.Should().NotBeNull();
result!.IsSuccess.Should().BeTrue();
result.IsVoid.Should().BeTrue();
result.Result.Should().BeNull();
invocation.InvocationCount.Should().Be(1);
invocation.Message.Should().Be("hello");
invocation.Context.Should().BeSameAs(context);
if (UsesCancellationToken(overload))
{
invocation.CancellationToken.Should().Be(cancellationToken);
}
}
[Theory]
[InlineData(HandlerOverload.SyncWithCancellation)]
[InlineData(HandlerOverload.SyncWithoutCancellation)]
[InlineData(HandlerOverload.AsyncWithCancellation)]
[InlineData(HandlerOverload.AsyncWithoutCancellation)]
public async Task AddHandler_ResultOverloads_RouteExpectedMessageAsync(HandlerOverload overload)
{
// Arrange
RouteBuilder routeBuilder = new(null);
HandlerInvocation invocation = new();
CancellationToken cancellationToken = new CancellationTokenSource().Token;
RegisterResultHandler(routeBuilder, invocation, overload);
MessageRouter router = routeBuilder.Build();
TestWorkflowContext context = new("executor");
// Act
CallResult? result = await router.RouteMessageAsync("hello", context, cancellationToken: cancellationToken);
// Assert
result.Should().NotBeNull();
result!.IsSuccess.Should().BeTrue();
result.IsVoid.Should().BeFalse();
result.Result.Should().Be("HELLO");
router.DefaultOutputTypes.Should().Contain(typeof(string));
invocation.InvocationCount.Should().Be(1);
invocation.Message.Should().Be("hello");
invocation.Context.Should().BeSameAs(context);
if (UsesCancellationToken(overload))
{
invocation.CancellationToken.Should().Be(cancellationToken);
}
}
[Theory]
[InlineData(HandlerOverload.SyncWithCancellation)]
[InlineData(HandlerOverload.SyncWithoutCancellation)]
[InlineData(HandlerOverload.AsyncWithCancellation)]
[InlineData(HandlerOverload.AsyncWithoutCancellation)]
public async Task AddCatchAll_VoidOverloads_RouteUnexpectedMessageAsync(HandlerOverload overload)
{
// Arrange
RouteBuilder routeBuilder = new(null);
HandlerInvocation invocation = new();
CancellationToken cancellationToken = new CancellationTokenSource().Token;
TestPayload payload = new("hello");
RegisterVoidCatchAll(routeBuilder, invocation, overload);
MessageRouter router = routeBuilder.Build();
TestWorkflowContext context = new("executor");
// Act
CallResult? result = await router.RouteMessageAsync(payload, context, cancellationToken: cancellationToken);
// Assert
result.Should().NotBeNull();
result!.IsSuccess.Should().BeTrue();
result.IsVoid.Should().BeTrue();
result.Result.Should().BeNull();
invocation.InvocationCount.Should().Be(1);
invocation.Message.Should().BeEquivalentTo(new PortableValue(payload));
invocation.Context.Should().BeSameAs(context);
if (UsesCancellationToken(overload))
{
invocation.CancellationToken.Should().Be(cancellationToken);
}
}
[Theory]
[InlineData(HandlerOverload.SyncWithCancellation)]
[InlineData(HandlerOverload.SyncWithoutCancellation)]
[InlineData(HandlerOverload.AsyncWithCancellation)]
[InlineData(HandlerOverload.AsyncWithoutCancellation)]
public async Task AddCatchAll_ResultOverloads_RouteUnexpectedMessageAsync(HandlerOverload overload)
{
// Arrange
RouteBuilder routeBuilder = new(null);
HandlerInvocation invocation = new();
CancellationToken cancellationToken = new CancellationTokenSource().Token;
TestPayload payload = new("hello");
RegisterResultCatchAll(routeBuilder, invocation, overload);
MessageRouter router = routeBuilder.Build();
TestWorkflowContext context = new("executor");
// Act
CallResult? result = await router.RouteMessageAsync(payload, context, cancellationToken: cancellationToken);
// Assert
result.Should().NotBeNull();
result!.IsSuccess.Should().BeTrue();
result.IsVoid.Should().BeFalse();
result.Result.Should().Be("HELLO");
invocation.InvocationCount.Should().Be(1);
invocation.Message.Should().BeEquivalentTo(new PortableValue(payload));
invocation.Context.Should().BeSameAs(context);
if (UsesCancellationToken(overload))
{
invocation.CancellationToken.Should().Be(cancellationToken);
}
}
[Fact]
public async Task AddHandlerUntyped_VoidAndResultOverloads_RouteExpectedMessageAsync()
{
// Arrange
RouteBuilder routeBuilder = new(null);
HandlerInvocation voidInvocation = new();
HandlerInvocation resultInvocation = new();
CancellationToken cancellationToken = new CancellationTokenSource().Token;
routeBuilder.AddHandlerUntyped(typeof(string), (message, context, token) =>
{
voidInvocation.Capture(message, context, token);
return default;
});
routeBuilder.AddHandlerUntyped<int>(typeof(int), (message, context, token) =>
{
resultInvocation.Capture(message, context, token);
return new((int)message + 1);
});
MessageRouter router = routeBuilder.Build();
TestWorkflowContext context = new("executor");
// Act
CallResult? voidResult = await router.RouteMessageAsync("hello", context, cancellationToken: cancellationToken);
CallResult? typedResult = await router.RouteMessageAsync(41, context, cancellationToken: cancellationToken);
// Assert
voidResult.Should().NotBeNull();
voidResult!.IsVoid.Should().BeTrue();
voidInvocation.Message.Should().Be("hello");
voidInvocation.Context.Should().BeSameAs(context);
voidInvocation.CancellationToken.Should().Be(cancellationToken);
typedResult.Should().NotBeNull();
typedResult!.Result.Should().Be(42);
router.DefaultOutputTypes.Should().Contain(typeof(int));
resultInvocation.Message.Should().Be(41);
resultInvocation.Context.Should().BeSameAs(context);
resultInvocation.CancellationToken.Should().Be(cancellationToken);
}
[Fact]
public void AddHandler_ForPortableValue_ThrowsInvalidOperationException()
{
// Arrange
RouteBuilder routeBuilder = new(null);
// Act
Action act = () => routeBuilder.AddHandler<PortableValue>((message, context) => { });
// Assert
act.Should().Throw<InvalidOperationException>()
.WithMessage("*Use AddCatchAll()*");
}
[Fact]
public void AddHandler_DuplicateRegistrationWithoutOverwrite_ThrowsArgumentException()
{
// Arrange
RouteBuilder routeBuilder = new(null);
routeBuilder.AddHandler<string>((message, context) => { });
// Act
Action act = () => routeBuilder.AddHandler<string>((message, context) => { });
// Assert
act.Should().Throw<ArgumentException>()
.WithMessage("*already registered*");
}
[Fact]
public void AddHandler_OverwriteWithoutExistingRegistration_ThrowsArgumentException()
{
// Arrange
RouteBuilder routeBuilder = new(null);
// Act
Action act = () => routeBuilder.AddHandler<string>((message, context) => { }, overwrite: true);
// Assert
act.Should().Throw<ArgumentException>()
.WithMessage("*has not yet been registered*");
}
[Fact]
public async Task AddHandler_OverwriteExistingRegistration_RoutesUpdatedHandlerAsync()
{
// Arrange
RouteBuilder routeBuilder = new(null);
routeBuilder.AddHandler<string>((message, context) => context.SendMessageAsync("first"));
routeBuilder.AddHandler<string>((message, context) => context.SendMessageAsync("second"), overwrite: true);
MessageRouter router = routeBuilder.Build();
TestWorkflowContext context = new("executor");
// Act
_ = await router.RouteMessageAsync("hello", context);
// Assert
context.SentMessages.Should().ContainSingle().Which.Should().Be("second");
}
[Fact]
public void AddCatchAll_DuplicateRegistrationWithoutOverwrite_ThrowsInvalidOperationException()
{
// Arrange
RouteBuilder routeBuilder = new(null);
routeBuilder.AddCatchAll((message, context) => { });
// Act
Action act = () => routeBuilder.AddCatchAll((message, context) => { });
// Assert
act.Should().Throw<InvalidOperationException>()
.WithMessage("*already registered*");
}
[Fact]
public async Task AddCatchAll_OverwriteExistingRegistration_RoutesUpdatedHandlerAsync()
{
// Arrange
RouteBuilder routeBuilder = new(null);
routeBuilder.AddCatchAll((message, context) => context.SendMessageAsync("first"));
routeBuilder.AddCatchAll((message, context) => context.SendMessageAsync("second"), overwrite: true);
MessageRouter router = routeBuilder.Build();
TestWorkflowContext context = new("executor");
// Act
_ = await router.RouteMessageAsync(new TestPayload("hello"), context);
// Assert
context.SentMessages.Should().ContainSingle().Which.Should().Be("second");
}
[Fact]
public void AddPortHandler_WithoutExternalRequestContext_ThrowsInvalidOperationException()
{
// Arrange
RouteBuilder routeBuilder = new(null);
// Act
Action act = () => routeBuilder.AddPortHandler<string, int>("port", (response, context, cancellationToken) => default, out _);
// Assert
act.Should().Throw<InvalidOperationException>()
.WithMessage("*external request context is required*");
}
[Fact]
public async Task AddPortHandler_RoutesMatchingExternalResponseAsync()
{
// Arrange
TestExternalRequestContext externalRequestContext = new();
RouteBuilder routeBuilder = new(externalRequestContext);
HandlerInvocation invocation = new();
routeBuilder.AddPortHandler<string, int>("port", (response, context, cancellationToken) =>
{
invocation.Capture(response, context, cancellationToken);
return default;
}, out PortBinding portBinding);
await portBinding.PostRequestAsync("request", requestId: "req-1");
MessageRouter router = routeBuilder.Build();
TestWorkflowContext context = new("executor");
CancellationToken cancellationToken = new CancellationTokenSource().Token;
ExternalResponse response = externalRequestContext.PostedRequests.Single().CreateResponse(42);
// Act
CallResult? result = await router.RouteMessageAsync(response, context, cancellationToken: cancellationToken);
// Assert
externalRequestContext.RegisteredPorts.Should().ContainSingle(port => port.Id == "port");
externalRequestContext.PostedRequests.Should().ContainSingle(request => request.RequestId == "req-1");
result.Should().NotBeNull();
result!.IsSuccess.Should().BeTrue();
result.Result.Should().BeSameAs(response);
invocation.InvocationCount.Should().Be(1);
invocation.Message.Should().Be(42);
invocation.Context.Should().BeSameAs(context);
invocation.CancellationToken.Should().Be(cancellationToken);
}
[Fact]
public async Task AddPortHandler_UnknownPort_ReturnsExceptionResultAsync()
{
// Arrange
TestExternalRequestContext externalRequestContext = new();
RouteBuilder routeBuilder = new(externalRequestContext);
routeBuilder.AddPortHandler<string, int>("port", (response, context, cancellationToken) => default, out _);
MessageRouter router = routeBuilder.Build();
ExternalRequest request = ExternalRequest.Create(RequestPort.Create<string, int>("other"), "request", requestId: "req-1");
// Act
CallResult? result = await router.RouteMessageAsync(request.CreateResponse(42), new TestWorkflowContext("executor"));
// Assert
result.Should().NotBeNull();
result!.IsSuccess.Should().BeFalse();
result.Exception.Should().BeOfType<InvalidOperationException>();
result.Exception!.Message.Should().Contain("Unknown port");
}
private static void RegisterVoidHandler(RouteBuilder routeBuilder, HandlerInvocation invocation, HandlerOverload overload)
{
switch (overload)
{
case HandlerOverload.SyncWithCancellation:
routeBuilder.AddHandler<string>((message, context, cancellationToken) => invocation.Capture(message, context, cancellationToken));
break;
case HandlerOverload.SyncWithoutCancellation:
routeBuilder.AddHandler<string>((message, context) => invocation.Capture(message, context));
break;
case HandlerOverload.AsyncWithCancellation:
routeBuilder.AddHandler<string>((message, context, cancellationToken) =>
{
invocation.Capture(message, context, cancellationToken);
return default;
});
break;
case HandlerOverload.AsyncWithoutCancellation:
routeBuilder.AddHandler<string>((message, context) =>
{
invocation.Capture(message, context);
return default;
});
break;
default:
throw new ArgumentOutOfRangeException(nameof(overload));
}
}
private static void RegisterResultHandler(RouteBuilder routeBuilder, HandlerInvocation invocation, HandlerOverload overload)
{
switch (overload)
{
case HandlerOverload.SyncWithCancellation:
routeBuilder.AddHandler<string, string>((message, context, cancellationToken) =>
{
invocation.Capture(message, context, cancellationToken);
return NormalizeHandlerResult(message);
});
break;
case HandlerOverload.SyncWithoutCancellation:
routeBuilder.AddHandler<string, string>((message, context) =>
{
invocation.Capture(message, context);
return NormalizeHandlerResult(message);
});
break;
case HandlerOverload.AsyncWithCancellation:
Func<string, IWorkflowContext, CancellationToken, ValueTask<string>> asyncHandlerWithCancellation = (message, context, cancellationToken) =>
{
invocation.Capture(message, context, cancellationToken);
return new ValueTask<string>(NormalizeHandlerResult(message));
};
routeBuilder.AddHandler(asyncHandlerWithCancellation);
break;
case HandlerOverload.AsyncWithoutCancellation:
Func<string, IWorkflowContext, ValueTask<string>> asyncHandler = (message, context) =>
{
invocation.Capture(message, context);
return new ValueTask<string>(NormalizeHandlerResult(message));
};
routeBuilder.AddHandler(asyncHandler);
break;
default:
throw new ArgumentOutOfRangeException(nameof(overload));
}
}
private static void RegisterVoidCatchAll(RouteBuilder routeBuilder, HandlerInvocation invocation, HandlerOverload overload)
{
switch (overload)
{
case HandlerOverload.SyncWithCancellation:
routeBuilder.AddCatchAll((message, context, cancellationToken) => invocation.Capture(message, context, cancellationToken));
break;
case HandlerOverload.SyncWithoutCancellation:
routeBuilder.AddCatchAll((message, context) => invocation.Capture(message, context));
break;
case HandlerOverload.AsyncWithCancellation:
routeBuilder.AddCatchAll((message, context, cancellationToken) =>
{
invocation.Capture(message, context, cancellationToken);
return default;
});
break;
case HandlerOverload.AsyncWithoutCancellation:
routeBuilder.AddCatchAll((message, context) =>
{
invocation.Capture(message, context);
return default;
});
break;
default:
throw new ArgumentOutOfRangeException(nameof(overload));
}
}
private static void RegisterResultCatchAll(RouteBuilder routeBuilder, HandlerInvocation invocation, HandlerOverload overload)
{
switch (overload)
{
case HandlerOverload.SyncWithCancellation:
routeBuilder.AddCatchAll((message, context, cancellationToken) =>
{
invocation.Capture(message, context, cancellationToken);
return NormalizeCatchAllResult(message);
});
break;
case HandlerOverload.SyncWithoutCancellation:
routeBuilder.AddCatchAll((message, context) =>
{
invocation.Capture(message, context);
return NormalizeCatchAllResult(message);
});
break;
case HandlerOverload.AsyncWithCancellation:
Func<PortableValue, IWorkflowContext, CancellationToken, ValueTask<string>> asyncCatchAllWithCancellation = (message, context, cancellationToken) =>
{
invocation.Capture(message, context, cancellationToken);
return new ValueTask<string>(NormalizeCatchAllResult(message));
};
routeBuilder.AddCatchAll(asyncCatchAllWithCancellation);
break;
case HandlerOverload.AsyncWithoutCancellation:
Func<PortableValue, IWorkflowContext, ValueTask<string>> asyncCatchAll = (message, context) =>
{
invocation.Capture(message, context);
return new ValueTask<string>(NormalizeCatchAllResult(message));
};
routeBuilder.AddCatchAll(asyncCatchAll);
break;
default:
throw new ArgumentOutOfRangeException(nameof(overload));
}
}
private static bool UsesCancellationToken(HandlerOverload overload) =>
overload is HandlerOverload.SyncWithCancellation or HandlerOverload.AsyncWithCancellation;
private static string NormalizeHandlerResult(string message) => message.ToUpperInvariant();
private static string NormalizeCatchAllResult(PortableValue message) => GetPayloadValue(message).ToUpperInvariant();
private static string GetPayloadValue(PortableValue message)
{
return message.As<TestPayload>() is TestPayload payload
? payload.Value
: throw new InvalidOperationException("Expected catch-all message payload to deserialize as TestPayload.");
}
}
@@ -1,6 +1,7 @@
// Copyright (c) Microsoft. All rights reserved.
using System;
using System.Collections.Generic;
using FluentAssertions;
namespace Microsoft.Agents.AI.Workflows.UnitTests;
@@ -157,4 +158,301 @@ public partial class WorkflowBuilderSmokeTests
workflow3.Name.Should().Be("Named Only");
workflow3.Description.Should().BeNull();
}
[Fact]
public void ForwardMessage_WithSingleTarget_CreatesDirectEdge()
{
// Arrange
NoOpExecutor source = new("start");
NoOpExecutor target = new("target");
// Act
Workflow workflow = new WorkflowBuilder(source.Id)
.ForwardMessage<string>(source, target)
.Build();
// Assert
Edge edge = GetSingleEdge(workflow, source.Id);
edge.Kind.Should().Be(EdgeKind.Direct);
edge.DirectEdgeData.Should().NotBeNull();
edge.DirectEdgeData!.SourceId.Should().Be(source.Id);
edge.DirectEdgeData!.SinkId.Should().Be(target.Id);
edge.DirectEdgeData.Condition.Should().NotBeNull();
edge.DirectEdgeData.Condition!("message").Should().BeTrue();
edge.DirectEdgeData.Condition!(42).Should().BeFalse();
edge.DirectEdgeData.Condition!(null).Should().BeFalse();
}
[Fact]
public void ForwardMessage_WithMultipleTargets_CreatesFanOutEdge()
{
// Arrange
NoOpExecutor source = new("start");
NoOpExecutor target1 = new("target1");
NoOpExecutor target2 = new("target2");
// Act
Workflow workflow = new WorkflowBuilder(source.Id)
.ForwardMessage<string>(source, [target1, target2], message => message == "match")
.Build();
// Assert
Edge edge = GetSingleEdge(workflow, source.Id);
edge.Kind.Should().Be(EdgeKind.FanOut);
edge.FanOutEdgeData.Should().NotBeNull();
edge.FanOutEdgeData!.SourceId.Should().Be(source.Id);
edge.FanOutEdgeData!.SinkIds.Should().Equal([target1.Id, target2.Id]);
edge.FanOutEdgeData.EdgeAssigner.Should().NotBeNull();
edge.FanOutEdgeData.EdgeAssigner!("match", 2).Should().Equal([0, 1]);
edge.FanOutEdgeData.EdgeAssigner!("other", 2).Should().BeEmpty();
edge.FanOutEdgeData.EdgeAssigner!(42, 2).Should().BeEmpty();
}
[Fact]
public void ForwardExcept_WithSingleTarget_CreatesDirectEdge()
{
// Arrange
NoOpExecutor source = new("start");
NoOpExecutor target = new("target");
// Act
Workflow workflow = new WorkflowBuilder(source.Id)
.ForwardExcept<string>(source, target)
.Build();
// Assert
Edge edge = GetSingleEdge(workflow, source.Id);
edge.Kind.Should().Be(EdgeKind.Direct);
edge.DirectEdgeData.Should().NotBeNull();
edge.DirectEdgeData!.SourceId.Should().Be(source.Id);
edge.DirectEdgeData!.SinkId.Should().Be(target.Id);
edge.DirectEdgeData.Condition.Should().NotBeNull();
edge.DirectEdgeData.Condition!("message").Should().BeFalse();
edge.DirectEdgeData.Condition!(42).Should().BeTrue();
edge.DirectEdgeData.Condition!(null).Should().BeTrue();
}
[Fact]
public void ForwardExcept_WithMultipleTargets_CreatesFanOutEdge()
{
// Arrange
NoOpExecutor source = new("start");
NoOpExecutor target1 = new("target1");
NoOpExecutor target2 = new("target2");
// Act
Workflow workflow = new WorkflowBuilder(source.Id)
.ForwardExcept<string>(source, [target1, target2])
.Build();
// Assert
Edge edge = GetSingleEdge(workflow, source.Id);
edge.Kind.Should().Be(EdgeKind.FanOut);
edge.FanOutEdgeData.Should().NotBeNull();
edge.FanOutEdgeData!.SourceId.Should().Be(source.Id);
edge.FanOutEdgeData!.SinkIds.Should().Equal([target1.Id, target2.Id]);
edge.FanOutEdgeData.EdgeAssigner.Should().NotBeNull();
edge.FanOutEdgeData.EdgeAssigner!(42, 2).Should().Equal([0, 1]);
edge.FanOutEdgeData.EdgeAssigner!("message", 2).Should().BeEmpty();
}
[Fact]
public void AddChain_CreatesSequentialDirectEdges()
{
// Arrange
NoOpExecutor source = new("start");
NoOpExecutor middle = new("middle");
NoOpExecutor end = new("end");
// Act
Workflow workflow = new WorkflowBuilder(source.Id)
.AddChain(source, [middle, end])
.Build();
// Assert
Edge firstEdge = GetSingleEdge(workflow, source.Id);
firstEdge.Kind.Should().Be(EdgeKind.Direct);
firstEdge.DirectEdgeData!.SourceId.Should().Be(source.Id);
firstEdge.DirectEdgeData.SinkId.Should().Be(middle.Id);
Edge secondEdge = GetSingleEdge(workflow, middle.Id);
secondEdge.Kind.Should().Be(EdgeKind.Direct);
secondEdge.DirectEdgeData!.SourceId.Should().Be(middle.Id);
secondEdge.DirectEdgeData.SinkId.Should().Be(end.Id);
}
[Fact]
public void AddChain_WhenExecutorRepeats_Throws()
{
// Arrange
NoOpExecutor source = new("start");
NoOpExecutor middle = new("middle");
// Act
Action act = () => new WorkflowBuilder(source.Id)
.AddChain(source, [middle, source]);
// Assert
act.Should().Throw<ArgumentException>()
.WithParameterName("executors");
}
[Fact]
public void AddExternalCall_CreatesRequestPortAndRoundTripEdges()
{
// Arrange
const string PortId = "port1";
NoOpExecutor source = new("start");
// Act
Workflow workflow = new WorkflowBuilder(source.Id)
.AddExternalCall<string, int>(source, PortId)
.Build();
// Assert
workflow.Ports.Should().ContainKey(PortId);
workflow.Ports[PortId].Request.Should().Be(typeof(string));
workflow.Ports[PortId].Response.Should().Be(typeof(int));
workflow.ExecutorBindings.Should().ContainKey(PortId);
Edge requestEdge = GetSingleEdge(workflow, source.Id);
requestEdge.Kind.Should().Be(EdgeKind.Direct);
requestEdge.DirectEdgeData!.SourceId.Should().Be(source.Id);
requestEdge.DirectEdgeData.SinkId.Should().Be(PortId);
Edge responseEdge = GetSingleEdge(workflow, PortId);
responseEdge.Kind.Should().Be(EdgeKind.Direct);
responseEdge.DirectEdgeData!.SourceId.Should().Be(PortId);
responseEdge.DirectEdgeData.SinkId.Should().Be(source.Id);
}
[Fact]
public void AddSwitch_CreatesFanOutEdgeWithCasesAndDefault()
{
// Arrange
NoOpExecutor source = new("start");
NoOpExecutor stringTarget = new("string-target");
NoOpExecutor intTarget = new("int-target");
NoOpExecutor defaultTarget = new("default-target");
// Act
Workflow workflow = new WorkflowBuilder(source.Id)
.AddSwitch(source, switchBuilder => switchBuilder
.AddCase<string>(message => message == "match", [stringTarget])
.AddCase<int>(message => message > 0, [intTarget])
.WithDefault([defaultTarget]))
.Build();
// Assert
Edge edge = GetSingleEdge(workflow, source.Id);
edge.Kind.Should().Be(EdgeKind.FanOut);
edge.FanOutEdgeData.Should().NotBeNull();
edge.FanOutEdgeData!.SourceId.Should().Be(source.Id);
edge.FanOutEdgeData!.SinkIds.Should().Equal([stringTarget.Id, intTarget.Id, defaultTarget.Id]);
edge.FanOutEdgeData.EdgeAssigner.Should().NotBeNull();
edge.FanOutEdgeData.EdgeAssigner!("match", 3).Should().Equal([0]);
edge.FanOutEdgeData.EdgeAssigner!(2, 3).Should().Equal([1]);
edge.FanOutEdgeData.EdgeAssigner!("other", 3).Should().Equal([2]);
}
[Fact]
public void ForwardMessage_InvalidArguments_Throw()
{
// Arrange
WorkflowBuilder builder = new("start");
NoOpExecutor source = new("start");
NoOpExecutor target = new("target");
// Act/Assert
Assert.Throws<ArgumentNullException>(() => ((WorkflowBuilder)null!).ForwardMessage<string>(source, target));
Assert.Throws<ArgumentNullException>("source", () => builder.ForwardMessage<string>(null!, target));
Assert.Throws<ArgumentNullException>("target", () => builder.ForwardMessage<string>(source, (ExecutorBinding)null!));
Assert.Throws<ArgumentNullException>("targets", () => builder.ForwardMessage<string>(source, (IEnumerable<ExecutorBinding>)null!));
Assert.Throws<ArgumentNullException>("targets", () => builder.ForwardMessage<string>(source, [target, null!]));
Assert.Throws<ArgumentException>("targets", () => builder.ForwardMessage<string>(source, []));
}
[Fact]
public void ForwardExcept_InvalidArguments_Throw()
{
// Arrange
WorkflowBuilder builder = new("start");
NoOpExecutor source = new("start");
NoOpExecutor target = new("target");
// Act/Assert
Assert.Throws<ArgumentNullException>(() => ((WorkflowBuilder)null!).ForwardExcept<string>(source, target));
Assert.Throws<ArgumentNullException>("source", () => builder.ForwardExcept<string>(null!, target));
Assert.Throws<ArgumentNullException>("target", () => builder.ForwardExcept<string>(source, (ExecutorBinding)null!));
Assert.Throws<ArgumentNullException>("targets", () => builder.ForwardExcept<string>(source, (IEnumerable<ExecutorBinding>)null!));
Assert.Throws<ArgumentNullException>("targets", () => builder.ForwardExcept<string>(source, [target, null!]));
Assert.Throws<ArgumentException>("targets", () => builder.ForwardExcept<string>(source, []));
}
[Fact]
public void AddChain_InvalidArguments_Throw()
{
// Arrange
WorkflowBuilder builder = new("start");
NoOpExecutor source = new("start");
NoOpExecutor target = new("target");
NoOpExecutor otherTarget = new("other-target");
// Act/Assert
Assert.Throws<ArgumentNullException>(() => ((WorkflowBuilder)null!).AddChain(source, [target]));
Assert.Throws<ArgumentNullException>("source", () => builder.AddChain(null!, [target]));
Assert.Throws<ArgumentNullException>("executors", () => builder.AddChain(source, null!));
Assert.Throws<ArgumentNullException>("executors", () => builder.AddChain(source, [target, null!]));
Assert.Throws<ArgumentException>("executors", () => builder.AddChain(source, [target, source]));
Assert.Throws<ArgumentException>("executors", () => builder.AddChain(source, [target, otherTarget, target]));
}
[Fact]
public void AddExternalCall_InvalidArguments_Throw()
{
// Arrange
WorkflowBuilder builder = new("start");
NoOpExecutor source = new("start");
// Act/Assert
Assert.Throws<ArgumentNullException>(() => ((WorkflowBuilder)null!).AddExternalCall<string, int>(source, "port"));
Assert.Throws<ArgumentNullException>("source", () => builder.AddExternalCall<string, int>(null!, "port"));
Assert.Throws<ArgumentNullException>("portId", () => builder.AddExternalCall<string, int>(source, null!));
}
[Fact]
public void AddSwitch_InvalidArguments_Throw()
{
// Arrange
WorkflowBuilder builder = new("start");
NoOpExecutor source = new("start");
// Act/Assert
Assert.Throws<ArgumentNullException>(() => ((WorkflowBuilder)null!).AddSwitch(source, _ => { }));
Assert.Throws<ArgumentNullException>("source", () => builder.AddSwitch(null!, _ => { }));
Assert.Throws<ArgumentNullException>("configureSwitch", () => builder.AddSwitch(source, null!));
Assert.Throws<ArgumentException>("targets", () => builder.AddSwitch(source, _ => { }));
Assert.Throws<ArgumentException>("targets", () => builder.AddSwitch(source, switchBuilder => switchBuilder.AddCase<string>(_ => true, [])));
}
[Fact]
public void SwitchBuilder_InvalidArguments_Throw()
{
// Arrange
SwitchBuilder switchBuilder = new();
NoOpExecutor target = new("target");
// Act/Assert
Assert.Throws<ArgumentNullException>("predicate", () => switchBuilder.AddCase<string>(null!, [target]));
Assert.Throws<ArgumentNullException>("executors", () => switchBuilder.AddCase<string>(_ => true, null!));
Assert.Throws<ArgumentNullException>("executors[1]", () => switchBuilder.AddCase<string>(_ => true, [target, null!]));
Assert.Throws<ArgumentNullException>("executors", () => switchBuilder.WithDefault(null!));
Assert.Throws<ArgumentNullException>("executors[1]", () => switchBuilder.WithDefault([target, null!]));
}
/// <summary>
/// Gets the only edge emitted by the specified workflow source.
/// </summary>
private static Edge GetSingleEdge(Workflow workflow, string sourceId)
=> workflow.Edges[sourceId].Should().ContainSingle().Subject;
}
@@ -67,7 +67,7 @@ public sealed class WorkflowRunActivityStopTests : IDisposable
/// Bug: The Activity created by LockstepRunEventStream.TakeEventStreamAsync is never
/// disposed because yield break in async iterators does not trigger using disposal.
/// </summary>
[Fact(Skip = "Flaky test - temporarily disabled.")]
[Fact]
public async Task WorkflowRunActivity_IsStopped_LockstepAsync()
{
// Arrange
@@ -111,7 +111,7 @@ public sealed class WorkflowRunActivityStopTests : IDisposable
/// Verifies that the workflow_invoke Activity is stopped when using the OffThread (Default)
/// execution environment (StreamingRunEventStream).
/// </summary>
[Fact(Skip = "Flaky test - temporarily disabled.")]
[Fact]
public async Task WorkflowRunActivity_IsStopped_OffThreadAsync()
{
// Arrange
@@ -156,7 +156,7 @@ public sealed class WorkflowRunActivityStopTests : IDisposable
/// (StreamingRun.WatchStreamAsync) with the OffThread execution environment.
/// This matches the exact usage pattern described in the issue.
/// </summary>
[Fact(Skip = "Flaky test - temporarily disabled.")]
[Fact]
public async Task WorkflowRunActivity_IsStopped_Streaming_OffThreadAsync()
{
// Arrange
@@ -203,7 +203,7 @@ public sealed class WorkflowRunActivityStopTests : IDisposable
/// streaming invocation, even when using the same workflow in a multi-turn pattern,
/// and that each session gets its own session activity.
/// </summary>
[Fact(Skip = "Flaky test - temporarily disabled.")]
[Fact]
public async Task WorkflowRunActivity_IsStopped_Streaming_OffThread_MultiTurnAsync()
{
// Arrange
@@ -264,7 +264,7 @@ public sealed class WorkflowRunActivityStopTests : IDisposable
/// Verifies that all started activities (not just workflow_invoke) are properly stopped.
/// This ensures no spans are "leaked" without being exported.
/// </summary>
[Fact(Skip = "Flaky test - temporarily disabled.")]
[Fact]
public async Task AllActivities_AreStopped_AfterWorkflowCompletionAsync()
{
// Arrange
@@ -305,7 +305,7 @@ public sealed class WorkflowRunActivityStopTests : IDisposable
/// be parented under the workflow session span. The run activity should
/// still nest correctly under the session.
/// </summary>
[Fact(Skip = "Flaky test - temporarily disabled.")]
[Fact]
public async Task Lockstep_SessionActivity_DoesNotLeak_IntoCaller_ActivityCurrentAsync()
{
// Arrange
+1 -2
View File
@@ -44,7 +44,6 @@ GEMINI_MODEL=""
# Ollama
OLLAMA_ENDPOINT=""
OLLAMA_MODEL=""
# Observability
ENABLE_INSTRUMENTATION=true
# Observability (instrumentation is enabled by default; set "ENABLE_INSTRUMENTATION" to "false" to opt out)
ENABLE_SENSITIVE_DATA=true
OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317/"
+1 -1
View File
@@ -72,7 +72,7 @@ def equal(arg1: str, arg2: str) -> bool:
from agent_framework import Agent, Message, tool
# Components
from agent_framework.observability import enable_instrumentation
from agent_framework.observability import enable_sensitive_telemetry
# Connectors (lazy-loaded)
from agent_framework.openai import OpenAIChatClient
+1 -1
View File
@@ -186,7 +186,7 @@ The package follows a flat import structure:
- **Components**: Import from `agent_framework.<component>`
```python
from agent_framework.observability import enable_instrumentation, configure_otel_providers
from agent_framework.observability import enable_sensitive_telemetry, configure_otel_providers
```
- **Connectors**: Import from `agent_framework.<vendor/platform>`
+1 -1
View File
@@ -16,7 +16,7 @@ Status is grouped into these buckets:
| --- | --- | --- |
| `agent-framework` | `python/` | `released` |
| `agent-framework-a2a` | `python/packages/a2a` | `beta` |
| `agent-framework-ag-ui` | `python/packages/ag-ui` | `beta` |
| `agent-framework-ag-ui` | `python/packages/ag-ui` | `rc` |
| `agent-framework-anthropic` | `python/packages/anthropic` | `beta` |
| `agent-framework-azure-contentunderstanding` | `python/packages/azure-contentunderstanding` | `alpha` |
| `agent-framework-azure-ai-search` | `python/packages/azure-ai-search` | `beta` |
+1 -1
View File
@@ -42,7 +42,7 @@ request_handler = DefaultRequestHandler(
app = Starlette(
routes=[
*create_agent_card_routes(my_agent_card),
*create_jsonrpc_routes(request_handler),
*create_jsonrpc_routes(request_handler, "/"),
]
)
```
@@ -78,7 +78,7 @@ class A2AExecutor(AgentExecutor):
app = Starlette(
routes=[
*create_agent_card_routes(public_agent_card),
*create_jsonrpc_routes(request_handler),
*create_jsonrpc_routes(request_handler, "/"),
],
)
@@ -157,9 +157,7 @@ class A2AAgent(AgentTelemetryLayer, BaseAgent):
self.client = factory.create(agent_card, interceptors=interceptors) # type: ignore
except Exception as transport_error:
# Transport negotiation failed - fall back to minimal agent card with JSONRPC
fallback_url = (
agent_card.supported_interfaces[0].url if agent_card.supported_interfaces else url
)
fallback_url = agent_card.supported_interfaces[0].url if agent_card.supported_interfaces else url
if not fallback_url:
raise ValueError(
"A2A transport negotiation failed and no fallback URL is available. "
@@ -365,6 +363,10 @@ class A2AAgent(AgentTelemetryLayer, BaseAgent):
all_updates: list[AgentResponseUpdate] = []
streamed_artifact_ids_by_task: dict[str, set[str]] = {}
# In non-streaming mode, accumulate intermediate status content so it
# can be surfaced when the terminal event arrives (mirroring v0.3.x
# behavior where the full Task history was available at completion).
pending_updates_by_task: dict[str, list[AgentResponseUpdate]] = {}
async for item in a2a_stream:
payload_type = item.WhichOneof("payload")
if payload_type == "message":
@@ -391,27 +393,55 @@ class A2AAgent(AgentTelemetryLayer, BaseAgent):
)
if task.status.state in TERMINAL_TASK_STATES:
streamed_artifact_ids_by_task.pop(task.id, None)
# If the terminal Task has no content, flush accumulated updates
if not updates or all(not u.contents for u in updates):
pending = pending_updates_by_task.pop(task.id, [])
for update in pending:
all_updates.append(update)
yield update
else:
pending_updates_by_task.pop(task.id, None)
for update in updates:
all_updates.append(update)
yield update
elif payload_type == "status_update":
status_event = item.status_update
updates = self._updates_from_task_update_event(status_event)
is_terminal = status_event.status.state in TERMINAL_TASK_STATES
if emit_intermediate:
for update in updates:
all_updates.append(update)
yield update
elif is_terminal:
if updates:
# Terminal event with content — discard accumulated intermediates
pending_updates_by_task.pop(status_event.task_id, None)
for update in updates:
all_updates.append(update)
yield update
else:
# Terminal event with NO content — flush accumulated updates
pending = pending_updates_by_task.pop(status_event.task_id, [])
for update in pending:
all_updates.append(update)
yield update
else:
# Non-streaming intermediate: accumulate for later
if updates:
pending_updates_by_task.setdefault(status_event.task_id, []).extend(updates)
elif payload_type == "artifact_update":
artifact_event = item.artifact_update
updates = self._updates_from_task_update_event(artifact_event)
# Always yield artifact updates — they carry actual response
# content (files, data). Track IDs so that a subsequent
# terminal Task doesn't duplicate the same artifacts.
if updates:
streamed_artifact_ids_by_task.setdefault(artifact_event.task_id, set()).add(
artifact_event.artifact.artifact_id
)
if emit_intermediate:
for update in updates:
all_updates.append(update)
yield update
for update in updates:
all_updates.append(update)
yield update
else:
raise NotImplementedError(f"Unsupported StreamResponse payload: {payload_type}")
@@ -1570,4 +1570,102 @@ async def test_none_metadata_leaves_additional_properties_empty(
assert not response.additional_properties
async def test_non_streaming_terminal_status_update_surfaces_content(
a2a_agent: A2AAgent, mock_a2a_client: MockA2AClient
) -> None:
"""Non-streaming run() should surface content from terminal status_update events."""
completed_msg = A2AMessage(
message_id="msg-complete",
role=A2ARole.ROLE_AGENT,
parts=[Part(text="Done! Here is your answer.")],
)
status = TaskStatus(state=TaskState.TASK_STATE_COMPLETED, message=completed_msg)
event = TaskStatusUpdateEvent(task_id="task-ts", context_id="ctx-ts", status=status)
mock_a2a_client.responses.append(StreamResponse(status_update=event))
response = await a2a_agent.run("Hello")
assert len(response.messages) == 1
assert response.messages[0].text == "Done! Here is your answer."
async def test_non_streaming_accumulates_working_content_for_empty_terminal(
a2a_agent: A2AAgent, mock_a2a_client: MockA2AClient
) -> None:
"""Non-streaming run() accumulates WORKING content and flushes on empty terminal event."""
# Intermediate WORKING event with content
working_msg = A2AMessage(
message_id="msg-working",
role=A2ARole.ROLE_AGENT,
parts=[Part(text="Here is your answer from working state.")],
)
working_status = TaskStatus(state=TaskState.TASK_STATE_WORKING, message=working_msg)
working_event = TaskStatusUpdateEvent(task_id="task-acc", context_id="ctx-acc", status=working_status)
mock_a2a_client.responses.append(StreamResponse(status_update=working_event))
# Terminal COMPLETED event with NO content
completed_status = TaskStatus(state=TaskState.TASK_STATE_COMPLETED)
completed_event = TaskStatusUpdateEvent(task_id="task-acc", context_id="ctx-acc", status=completed_status)
mock_a2a_client.responses.append(StreamResponse(status_update=completed_event))
response = await a2a_agent.run("Hello")
# The accumulated WORKING content is flushed when terminal arrives empty
assert len(response.messages) == 1
assert response.messages[0].text == "Here is your answer from working state."
async def test_non_streaming_intermediate_discarded_when_terminal_has_content(
a2a_agent: A2AAgent, mock_a2a_client: MockA2AClient
) -> None:
"""Non-streaming: if terminal event has content, intermediate content is discarded."""
# Intermediate WORKING event
working_msg = A2AMessage(
message_id="msg-working",
role=A2ARole.ROLE_AGENT,
parts=[Part(text="Still thinking...")],
)
working_status = TaskStatus(state=TaskState.TASK_STATE_WORKING, message=working_msg)
working_event = TaskStatusUpdateEvent(task_id="task-wi", context_id="ctx-wi", status=working_status)
mock_a2a_client.responses.append(StreamResponse(status_update=working_event))
# Terminal COMPLETED event WITH content
completed_msg = A2AMessage(
message_id="msg-final",
role=A2ARole.ROLE_AGENT,
parts=[Part(text="Final answer")],
)
completed_status = TaskStatus(state=TaskState.TASK_STATE_COMPLETED, message=completed_msg)
completed_event = TaskStatusUpdateEvent(task_id="task-wi", context_id="ctx-wi", status=completed_status)
mock_a2a_client.responses.append(StreamResponse(status_update=completed_event))
response = await a2a_agent.run("Hello")
# Terminal content supersedes accumulated intermediates
assert len(response.messages) == 1
assert response.messages[0].text == "Final answer"
async def test_non_streaming_artifact_update_surfaces_content(
a2a_agent: A2AAgent, mock_a2a_client: MockA2AClient
) -> None:
"""Non-streaming run() should surface content from artifact_update events."""
artifact = Artifact(
artifact_id="art-ns",
parts=[Part(text="Artifact content")],
)
event = TaskArtifactUpdateEvent(task_id="task-anu", context_id="ctx-anu", artifact=artifact, append=False)
mock_a2a_client.responses.append(StreamResponse(artifact_update=event))
# Terminal task with the same artifact ID — should be deduped
mock_a2a_client.add_task_response("task-anu", [{"id": "art-ns", "content": "Artifact content"}])
response = await a2a_agent.run("Hello")
# Artifact update + terminal task with same artifact ID = content emitted once from
# the artifact_update, then the duplicate from the task is filtered by streamed_artifact_ids
assert len(response.messages) == 1
assert response.messages[0].text == "Artifact content"
# endregion
+1 -1
View File
@@ -1,6 +1,6 @@
[project]
name = "agent-framework-ag-ui"
version = "1.0.0b260507"
version = "1.0.0rc1"
description = "AG-UI protocol integration for Agent Framework"
readme = "README.md"
license-files = ["LICENSE"]
@@ -5,7 +5,6 @@
from __future__ import annotations
import logging
import sys
from collections.abc import Awaitable, Callable, Sequence
from agent_framework import (
@@ -30,11 +29,6 @@ from chatkit.types import (
WorkflowItem,
)
if sys.version_info >= (3, 11):
from typing import assert_never # type:ignore # pragma: no cover
else:
from typing_extensions import assert_never # type:ignore # pragma: no cover
logger = logging.getLogger(__name__)
@@ -528,7 +522,10 @@ class ThreadItemConverter:
# TODO(evmattso): Implement generated image handling in a future PR
return []
case _:
assert_never(item)
# Unknown ThreadItem variant (e.g. types added in newer chatkit versions).
# Skip rather than fail so we remain forward-compatible with chatkit upgrades.
logger.debug("Skipping unsupported ThreadItem of type %s", type(item).__name__)
return []
async def to_agent_input(
self,
+9 -3
View File
@@ -261,6 +261,7 @@ class MCPTool:
self.request_timeout = request_timeout
self.client = client
self._functions: list[FunctionTool] = []
self._tool_call_meta_by_name: dict[str, dict[str, Any]] = {}
self.is_connected: bool = False
self._tools_loaded: bool = False
self._prompts_loaded: bool = False
@@ -1026,6 +1027,7 @@ class MCPTool:
# Track existing function names to prevent duplicates
existing_names = {func.name for func in self._functions}
self._tool_call_meta_by_name.clear()
params: types.PaginatedRequestParams | None = None
while True:
@@ -1035,6 +1037,9 @@ class MCPTool:
tool_list = await self.session.list_tools(params=params) # type: ignore[union-attr]
for tool in tool_list.tools:
if tool.meta is not None:
self._tool_call_meta_by_name[tool.name] = dict(tool.meta)
normalized_name = _normalize_mcp_name(tool.name)
local_name = _build_prefixed_mcp_name(normalized_name, self.tool_name_prefix)
@@ -1185,14 +1190,15 @@ class MCPTool:
}
}
# Inject OpenTelemetry trace context into MCP _meta for distributed tracing.
otel_meta = _inject_otel_into_mcp_meta()
# Some MCP proxies require their tools/list metadata to be echoed on tools/call.
tool_meta = self._tool_call_meta_by_name.get(tool_name)
meta = _inject_otel_into_mcp_meta(dict(tool_meta) if tool_meta is not None else None)
parser = self.parse_tool_results or self._parse_tool_result_from_mcp
# Try the operation, reconnecting once if the connection is closed
for attempt in range(2):
try:
result = await self.session.call_tool(tool_name, arguments=filtered_kwargs, meta=otel_meta) # type: ignore
result = await self.session.call_tool(tool_name, arguments=filtered_kwargs, meta=meta) # type: ignore
if result.isError:
parsed = parser(result)
text = (
+84 -29
View File
@@ -289,13 +289,15 @@ class SkillScript(ABC):
return None
@abstractmethod
async def run(self, skill: Skill, args: dict[str, Any] | None = None, **kwargs: Any) -> Any:
async def run(self, skill: Skill, args: dict[str, Any] | list[str] | None = None, **kwargs: Any) -> Any:
"""Run this script.
Args:
skill: The skill that owns this script.
args: Optional keyword arguments for the script, provided by the
agent/LLM.
args: Optional arguments for the script, provided by the
agent/LLM. May be a ``dict`` (named keyword arguments
for inline scripts) or a ``list[str]`` (positional CLI
arguments for file-based scripts).
**kwargs: Runtime keyword arguments forwarded only to script
functions that accept ``**kwargs``.
@@ -361,19 +363,31 @@ class InlineSkillScript(SkillScript):
self._parameters_schema_resolved = True
return self._parameters_schema
async def run(self, skill: Skill, args: dict[str, Any] | None = None, **kwargs: Any) -> Any:
async def run(self, skill: Skill, args: dict[str, Any] | list[str] | None = None, **kwargs: Any) -> Any:
"""Run the script by invoking the callable in-process.
Args:
skill: The skill that owns this script.
args: Optional keyword arguments for the script, provided by the
agent/LLM.
agent/LLM. Must be a ``dict`` or ``None``; passing a
``list`` raises :class:`TypeError` because inline scripts
bind arguments by keyword name.
**kwargs: Runtime keyword arguments forwarded only to script
functions that accept ``**kwargs``.
Returns:
The script execution result.
Raises:
TypeError: If ``args`` is a ``list`` (array-style arguments
are only supported for file-based scripts).
"""
if isinstance(args, list):
raise TypeError(
f"Inline script '{self.name}' requires keyword arguments (dict), "
f"but received a list. Array-style arguments are only supported "
f"for file-based scripts."
)
if self._accepts_kwargs: # noqa: SIM108
result = self.function(**(args or {}), **kwargs)
else:
@@ -431,13 +445,23 @@ class FileSkillScript(SkillScript):
self.full_path = full_path
self._runner = runner
async def run(self, skill: Skill, args: dict[str, Any] | None = None, **kwargs: Any) -> Any:
@property
def parameters_schema(self) -> dict[str, Any] | None:
"""JSON Schema advertising that file scripts accept a string array.
Returns a fixed schema ``{"type": "array", "items": {"type": "string"}}``
so that the LLM knows to pass positional CLI arguments as a JSON array
of strings.
"""
return {"type": "array", "items": {"type": "string"}}
async def run(self, skill: Skill, args: dict[str, Any] | list[str] | None = None, **kwargs: Any) -> Any:
"""Run the script by delegating to the configured runner.
Args:
skill: The skill that owns this script. Must be a
:class:`FileSkill`.
args: Optional keyword arguments for the script.
args: Optional arguments for the script.
**kwargs: Additional runtime keyword arguments (unused).
Returns:
@@ -627,9 +651,7 @@ def _validate_compatibility(compatibility: str | None) -> None:
ValueError: If the value exceeds the maximum allowed length.
"""
if compatibility is not None and len(compatibility) > MAX_COMPATIBILITY_LENGTH:
raise ValueError(
f"Skill compatibility must be {MAX_COMPATIBILITY_LENGTH} characters or fewer."
)
raise ValueError(f"Skill compatibility must be {MAX_COMPATIBILITY_LENGTH} characters or fewer.")
def _build_skill_content(
@@ -709,6 +731,7 @@ class InlineSkill(Skill):
instructions="Use this skill for DB tasks.",
)
@skill.resource
def get_schema() -> str:
return "CREATE TABLE ..."
@@ -1348,6 +1371,7 @@ class FileSkill(Skill):
self.path = path
self._resources: list[SkillResource] = list(resources) if resources is not None else []
self._scripts: list[SkillScript] = list(scripts) if scripts is not None else []
self._cached_content: str | None = None
@property
def frontmatter(self) -> SkillFrontmatter:
@@ -1356,8 +1380,23 @@ class FileSkill(Skill):
@property
def content(self) -> str:
"""The skill content provided at construction time."""
return self._content
"""The skill content with appended scripts block.
When scripts are present, a ``<scripts>`` XML block is appended
to the raw SKILL.md content so that the LLM can discover each
script's ``<parameters_schema>``.
The result is cached after the first access. Adding scripts
after the first access will not be reflected.
"""
if self._cached_content is not None:
return self._cached_content
if not self._scripts:
self._cached_content = self._content
else:
script_lines = "\n".join(_create_script_element(s) for s in self._scripts)
self._cached_content = f"{self._content}\n\n<scripts>\n{script_lines}\n</scripts>"
return self._cached_content
@property
def resources(self) -> list[SkillResource]:
@@ -1392,7 +1431,9 @@ class SkillScriptRunner(Protocol):
satisfies this protocol.
"""
def __call__(self, skill: FileSkill, script: FileSkillScript, args: dict[str, Any] | None = None) -> Any:
def __call__(
self, skill: FileSkill, script: FileSkillScript, args: dict[str, Any] | list[str] | None = None
) -> Any:
"""Run a skill script.
The :class:`SkillsProvider` resolves skill and script names
@@ -1402,7 +1443,7 @@ class SkillScriptRunner(Protocol):
Args:
skill: The file-based skill that owns the script.
script: The file-based script to run.
args: Optional keyword arguments for the script.
args: Optional arguments for the script.
Returns:
The result. May be any type; the framework
@@ -1982,7 +2023,7 @@ class SkillsProvider(ContextProvider):
if include_script_runner_tool:
async def _run_script(
skill_name: str, script_name: str, args: dict[str, Any] | None = None, **kwargs: Any
skill_name: str, script_name: str, args: dict[str, Any] | list[str] | None = None, **kwargs: Any
) -> Any:
return await self._run_skill_script(skills, skill_name, script_name, args, **kwargs)
@@ -2005,12 +2046,31 @@ class SkillsProvider(ContextProvider):
),
},
"args": {
"type": ["object", "null"],
"additionalProperties": True,
"oneOf": [
{
"type": "object",
"additionalProperties": True,
"description": (
"Named arguments as key-value pairs "
'(e.g. {"length": 24, "uppercase": true}).'
),
},
{
"type": "array",
"items": {"type": "string"},
"description": (
"Positional CLI arguments as a string array "
'(e.g. ["input.docx", "--output", "result.idx"]).'
),
},
{"type": "null"},
],
"default": None,
"description": (
"Arguments to pass to the script as key-value pairs. "
"Use parameter names as keys without leading dashes "
"Arguments to pass to the script. "
"Use an array of strings for CLI-style positional arguments "
'(e.g. ["input.docx", "--output", "result.idx"]), '
"or an object for named parameters "
'(e.g. {"length": 24, "uppercase": true}). '
"How these values are mapped to the underlying script "
"is determined by the script implementation or configured runner."
@@ -2060,7 +2120,7 @@ class SkillsProvider(ContextProvider):
skills: Sequence[Skill],
skill_name: str,
script_name: str,
args: dict[str, Any] | None = None,
args: dict[str, Any] | list[str] | None = None,
**kwargs: Any,
) -> Any:
"""Run a named script from a skill.
@@ -2072,9 +2132,8 @@ class SkillsProvider(ContextProvider):
skills: The skills to look up the skill from.
skill_name: The name of the owning skill.
script_name: The script name to look up (case-insensitive).
args: Optional keyword arguments for the script, provided by the
agent/LLM. These are mapped to the function's declared
parameters.
args: Optional arguments for the script, provided by the
agent/LLM.
**kwargs: Runtime keyword arguments forwarded only to script
functions that accept ``**kwargs`` (e.g. arguments passed via
``agent.run(user_id="123")``).
@@ -2254,7 +2313,7 @@ class FileSkillsSource(SkillsSource):
Args:
skill_paths: One or more directory paths to search for file-based
skills. Each path may point to an individual skill folder
skills. Each path may point to an individual skill directory
(containing ``SKILL.md``) or to a parent that contains skill
subdirectories.
@@ -2462,11 +2521,7 @@ class FileSkillsSource(SkillsSource):
# Reject absolute paths (check both POSIX and Windows-style roots
# so validation is consistent regardless of the host OS)
if (
os.path.isabs(directory)
or normalized.startswith("/")
or re.match(r"^[A-Za-z]:[/\\]", directory)
):
if os.path.isabs(directory) or normalized.startswith("/") or re.match(r"^[A-Za-z]:[/\\]", directory):
logger.warning(
"Skipping directory '%s': absolute paths are not allowed.",
directory,
@@ -4,6 +4,8 @@
Commonly used exports:
- enable_instrumentation
- disable_instrumentation
- enable_sensitive_telemetry
- configure_otel_providers
- AgentTelemetryLayer
- ChatTelemetryLayer
@@ -80,7 +82,9 @@ __all__ = [
"configure_otel_providers",
"create_metric_views",
"create_resource",
"disable_instrumentation",
"enable_instrumentation",
"enable_sensitive_telemetry",
"get_meter",
"get_tracer",
]
@@ -643,8 +647,8 @@ class ObservabilitySettings:
Sensitive events should only be enabled on test and development environments.
Keyword Args:
enable_instrumentation: Enable OpenTelemetry diagnostics. Default is False.
Can be set via environment variable ENABLE_INSTRUMENTATION.
enable_instrumentation: Enable OpenTelemetry diagnostics. Default is True.
Can be disabled by setting environment variable ENABLE_INSTRUMENTATION=false.
enable_sensitive_data: Enable OpenTelemetry sensitive events. Default is False.
Can be set via environment variable ENABLE_SENSITIVE_DATA.
enable_console_exporters: Enable console exporters for traces, logs, and metrics.
@@ -659,12 +663,12 @@ class ObservabilitySettings:
from agent_framework import ObservabilitySettings
# Using environment variables
# Set ENABLE_INSTRUMENTATION=true
# Instrumentation is enabled by default; set ENABLE_INSTRUMENTATION=false to disable.
# Set ENABLE_CONSOLE_EXPORTERS=true
settings = ObservabilitySettings()
# Or passing parameters directly
settings = ObservabilitySettings(enable_instrumentation=True, enable_console_exporters=True)
settings = ObservabilitySettings(enable_console_exporters=True)
"""
def __init__(self, **kwargs: Any) -> None:
@@ -677,14 +681,74 @@ class ObservabilitySettings:
env_file_encoding=env_file_encoding,
**kwargs,
)
self.enable_instrumentation: bool = data.get("enable_instrumentation") or False
self.enable_sensitive_data: bool = data.get("enable_sensitive_data") or False
# Sticky-disable flag, set by `disable_instrumentation()`. When True, this
# singleton refuses to be re-enabled by any subsequent assignment to the
# `enable_instrumentation` / `enable_sensitive_data` properties (including
# direct third-party writes). It can only be cleared by an explicit
# `enable_instrumentation(force=True)` / `enable_sensitive_telemetry(force=True)`
# call, which is the user re-stating their intent.
self._user_disabled: bool = False
# `enable_instrumentation` is defaulted to True if not set
instrumentation_value = data.get("enable_instrumentation")
self._enable_instrumentation: bool = True if instrumentation_value is None else instrumentation_value
self._enable_sensitive_data: bool = data.get("enable_sensitive_data") or False
if self._enable_sensitive_data and not self._enable_instrumentation:
logger.warning(
"Sensitive data capture is enabled but instrumentation is disabled. "
"Sensitive data will not be captured. Please enable instrumentation to capture sensitive data."
)
self.enable_console_exporters: bool = data.get("enable_console_exporters") or False
self.vs_code_extension_port: int | None = data.get("vs_code_extension_port")
self.env_file_path = env_file_path
self.env_file_encoding = env_file_encoding
self._executed_setup = False
@property
def enable_instrumentation(self) -> bool:
"""Whether instrumentation is enabled.
Always returns False once ``disable_instrumentation()`` has been called,
regardless of the stored value, until ``enable_instrumentation(force=True)``
clears the sticky disable.
"""
if self._user_disabled:
return False
return self._enable_instrumentation
@enable_instrumentation.setter
def enable_instrumentation(self, value: bool) -> None:
if self._user_disabled and value:
# Defense in depth: a third-party (or internal) write of True is
# silently dropped while the user-disabled flag is set, so the
# sticky disable cannot be circumvented by direct attribute writes.
logger.debug(
"Ignoring enable_instrumentation=True assignment: instrumentation was explicitly disabled via "
"disable_instrumentation(). Call enable_instrumentation(force=True) to clear the disable."
)
return
self._enable_instrumentation = value
@property
def enable_sensitive_data(self) -> bool:
"""Whether sensitive-data capture is enabled.
Always returns False once ``disable_instrumentation()`` has been called.
"""
if self._user_disabled:
return False
return self._enable_sensitive_data
@enable_sensitive_data.setter
def enable_sensitive_data(self, value: bool) -> None:
if self._user_disabled and value:
logger.debug(
"Ignoring enable_sensitive_data=True assignment: instrumentation was explicitly disabled via "
"disable_instrumentation(). Call enable_sensitive_telemetry(force=True) to clear the disable."
)
return
self._enable_sensitive_data = value
@property
def ENABLED(self) -> bool:
"""Check if model diagnostics are enabled.
@@ -706,6 +770,17 @@ class ObservabilitySettings:
"""Check if the setup has been executed."""
return self._executed_setup
@property
def is_user_disabled(self) -> bool:
"""Whether ``disable_instrumentation()`` has been called and the disable is still in effect.
Integrations that perform telemetry setup as a side-effect (e.g. provisioning Azure Monitor
providers from a Foundry project's connection string) should consult this flag before doing
their setup work, so the user's explicit opt-out is respected end-to-end and not just at the
framework's span-emission boundary.
"""
return self._user_disabled
def _configure(
self,
*,
@@ -951,24 +1026,91 @@ def _read_int_env(name: str, *, default: int | None = None) -> int | None:
return default
def enable_sensitive_telemetry(*, force: bool = False) -> None:
"""Enable capture of sensitive data in telemetry for your application.
Instrumentation is enabled by default; this method exists to opt-in to capturing
sensitive event payloads (e.g., chat messages, tool arguments).
This method does not configure exporters or providers. It also ensures that
instrumentation is enabled (in case it was explicitly disabled via the
ENABLE_INSTRUMENTATION environment variable).
Keyword Args:
force: When True, clears any sticky disable previously set by
``disable_instrumentation()`` before enabling. Without it, calls are
no-ops if instrumentation has been explicitly disabled.
Warning:
Sensitive events should only be enabled on test and development environments.
"""
global OBSERVABILITY_SETTINGS
if OBSERVABILITY_SETTINGS._user_disabled and not force: # type: ignore[reportPrivateUsage]
logger.info(
"enable_sensitive_telemetry() ignored: instrumentation was explicitly disabled via "
"disable_instrumentation(). Pass force=True to re-enable."
)
return
if force:
OBSERVABILITY_SETTINGS._user_disabled = False # type: ignore[reportPrivateUsage]
OBSERVABILITY_SETTINGS.enable_instrumentation = True
OBSERVABILITY_SETTINGS.enable_sensitive_data = True
def disable_instrumentation() -> None:
"""Explicitly disable Agent Framework instrumentation for this process.
The disable is **sticky**: subsequent attempts by framework auto-setup paths,
library integrations, ``enable_instrumentation()``, ``enable_sensitive_telemetry()``,
``configure_otel_providers()``, or direct writes to
``OBSERVABILITY_SETTINGS.enable_instrumentation`` are ignored and no spans, metrics,
or logs are emitted by Agent Framework code paths.
To override the disable later, call ``enable_instrumentation(force=True)`` or
``enable_sensitive_telemetry(force=True)``. This makes the user's intent to opt out
win against framework code that would otherwise re-enable instrumentation
automatically.
Note:
Disabling does not tear down already-configured OpenTelemetry providers,
exporters, or in-flight spans; it gates future captures by Agent Framework
instrumentation only. To stop emitting telemetry from third-party
instrumentations as well, configure them separately.
"""
global OBSERVABILITY_SETTINGS
OBSERVABILITY_SETTINGS._user_disabled = True # type: ignore[reportPrivateUsage]
OBSERVABILITY_SETTINGS._enable_instrumentation = False # type: ignore[reportPrivateUsage]
OBSERVABILITY_SETTINGS._enable_sensitive_data = False # type: ignore[reportPrivateUsage]
def enable_instrumentation(
*,
enable_sensitive_data: bool | None = None,
force: bool = False,
) -> None:
"""Enable instrumentation for your application.
"""Enable instrumentation for Microsoft Agent Framework.
Calling this method implies you want to enable observability in your application.
This method does not configure exporters or providers.
It only updates the global variables that trigger the instrumentation code.
If you have already set the environment variable ENABLE_INSTRUMENTATION=true,
calling this method has no effect, unless you want to enable or disable sensitive data events.
Note that instrumentation is enabled by default, so this method is only necessary
if you need a programmatic way to enable it (e.g., if you are not sure whether the
environment variable ENABLE_INSTRUMENTATION is set to True or False and want to
ensure it is enabled).
Keyword Args:
enable_sensitive_data: Enable OpenTelemetry sensitive events. Overrides
the environment variable ENABLE_SENSITIVE_DATA if set. Default is None.
force: When True, clears any sticky disable previously set by
``disable_instrumentation()`` before enabling. Without it, calls are
no-ops if instrumentation has been explicitly disabled.
"""
global OBSERVABILITY_SETTINGS
if OBSERVABILITY_SETTINGS._user_disabled and not force: # type: ignore[reportPrivateUsage]
logger.info(
"enable_instrumentation() ignored: instrumentation was explicitly disabled via "
"disable_instrumentation(). Pass force=True to re-enable."
)
return
if force:
OBSERVABILITY_SETTINGS._user_disabled = False # type: ignore[reportPrivateUsage]
OBSERVABILITY_SETTINGS.enable_instrumentation = True
if enable_sensitive_data is not None:
OBSERVABILITY_SETTINGS.enable_sensitive_data = enable_sensitive_data
@@ -1008,7 +1150,7 @@ def configure_otel_providers(
Since you can only setup one provider per signal type (logs, traces, metrics),
you can choose to use this method and take the exporter and provider that we created.
Alternatively, you can setup the providers yourself, or through another library
(e.g., Azure Monitor) and just call `enable_instrumentation()` to enable instrumentation.
(e.g., Azure Monitor) and just call `enable_sensitive_telemetry()` to opt-in to sensitive data capture.
Note:
By default, the Agent Framework emits metrics with the prefixes `agent_framework`
@@ -1042,7 +1184,6 @@ def configure_otel_providers(
from agent_framework.observability import configure_otel_providers
# Using environment variables (recommended)
# Set ENABLE_INSTRUMENTATION=true
# Set OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
configure_otel_providers()
@@ -1087,18 +1228,25 @@ def configure_otel_providers(
.. code-block:: python
# when azure monitor is installed
from agent_framework.observability import enable_instrumentation
from agent_framework.observability import enable_sensitive_telemetry
from azure.monitor.opentelemetry import configure_azure_monitor
connection_string = "InstrumentationKey=your_instrumentation_key_here;..."
configure_azure_monitor(connection_string=connection_string)
enable_instrumentation()
# Optional: opt into capturing sensitive data
enable_sensitive_telemetry()
References:
- https://opentelemetry.io/docs/languages/sdk-configuration/general/
- https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/
"""
global OBSERVABILITY_SETTINGS
if OBSERVABILITY_SETTINGS._user_disabled: # type: ignore[reportPrivateUsage]
logger.info(
"configure_otel_providers(): instrumentation was explicitly disabled via "
"disable_instrumentation(); providers and exporters will still be configured but "
"Agent Framework will emit no telemetry until enable_instrumentation(force=True) is called."
)
if env_file_path:
# Build kwargs, excluding None values
settings_kwargs: dict[str, Any] = {
@@ -1280,7 +1428,7 @@ class ChatTelemetryLayer(Generic[OptionsCoT]):
if stream:
span = _start_streaming_span(attributes, OtelAttr.REQUEST_MODEL)
if OBSERVABILITY_SETTINGS.SENSITIVE_DATA_ENABLED and messages:
if OBSERVABILITY_SETTINGS.SENSITIVE_DATA_ENABLED and messages and span.is_recording():
_capture_messages(
span=span,
provider_name=provider_name,
@@ -1344,6 +1492,7 @@ class ChatTelemetryLayer(Generic[OptionsCoT]):
OBSERVABILITY_SETTINGS.SENSITIVE_DATA_ENABLED
and isinstance(response, ChatResponse)
and response.messages
and span.is_recording()
):
_capture_messages(
span=span,
@@ -1374,7 +1523,7 @@ class ChatTelemetryLayer(Generic[OptionsCoT]):
async def _get_response() -> ChatResponse:
with _get_span(attributes=attributes, span_name_attribute=OtelAttr.REQUEST_MODEL) as span:
if OBSERVABILITY_SETTINGS.SENSITIVE_DATA_ENABLED and messages:
if OBSERVABILITY_SETTINGS.SENSITIVE_DATA_ENABLED and messages and span.is_recording():
_capture_messages(
span=span,
provider_name=provider_name,
@@ -1408,7 +1557,7 @@ class ChatTelemetryLayer(Generic[OptionsCoT]):
duration=duration,
)
_mark_inner_response_telemetry_captured(response)
if OBSERVABILITY_SETTINGS.SENSITIVE_DATA_ENABLED and response.messages:
if OBSERVABILITY_SETTINGS.SENSITIVE_DATA_ENABLED and response.messages and span.is_recording():
finish_reason = cast(
"FinishReason | None",
response.finish_reason if response.finish_reason in FINISH_REASON_MAP else None,
@@ -1552,7 +1701,7 @@ class AgentTelemetryLayer:
if stream:
span = _start_streaming_span(attributes, OtelAttr.AGENT_NAME)
if OBSERVABILITY_SETTINGS.SENSITIVE_DATA_ENABLED and messages:
if OBSERVABILITY_SETTINGS.SENSITIVE_DATA_ENABLED and messages and span.is_recording():
_capture_messages(
span=span,
provider_name=provider_name,
@@ -1613,6 +1762,7 @@ class AgentTelemetryLayer:
OBSERVABILITY_SETTINGS.SENSITIVE_DATA_ENABLED
and isinstance(response, AgentResponse)
and response.messages
and span.is_recording()
):
_capture_messages(
span=span,
@@ -1645,7 +1795,7 @@ class AgentTelemetryLayer:
async def _run() -> AgentResponse[Any]:
try:
with _get_span(attributes=attributes, span_name_attribute=OtelAttr.AGENT_NAME) as span:
if OBSERVABILITY_SETTINGS.SENSITIVE_DATA_ENABLED and messages:
if OBSERVABILITY_SETTINGS.SENSITIVE_DATA_ENABLED and messages and span.is_recording():
_capture_messages(
span=span,
provider_name=provider_name,
@@ -1669,7 +1819,7 @@ class AgentTelemetryLayer:
)
_apply_accumulated_usage(response_attributes, inner_response_telemetry_captured_fields)
_capture_response(span=span, attributes=response_attributes, duration=duration)
if OBSERVABILITY_SETTINGS.SENSITIVE_DATA_ENABLED and response.messages:
if OBSERVABILITY_SETTINGS.SENSITIVE_DATA_ENABLED and response.messages and span.is_recording():
_capture_messages(
span=span,
provider_name=provider_name,
@@ -4194,6 +4194,55 @@ async def test_mcp_tool_call_tool_otel_meta(use_span, expect_traceparent, span_e
assert meta is None
async def test_mcp_tool_call_tool_forwards_tool_list_meta():
"""call_tool echoes per-tool metadata returned by tools/list."""
from opentelemetry import trace
tool_meta = {
"tool_configuration": {
"name": "WorkIQSharePoint.readSmallBinaryFile",
"type": "foundry_toolbox",
}
}
class TestServer(MCPTool):
async def connect(self):
self.session = Mock(spec=ClientSession)
self.session.list_tools = AsyncMock(
return_value=types.ListToolsResult(
tools=[
types.Tool(
name="WorkIQSharePoint.readSmallBinaryFile",
description="Read a binary file",
inputSchema={
"type": "object",
"properties": {"fileId": {"type": "string"}},
"required": ["fileId"],
},
_meta=tool_meta,
)
]
)
)
self.session.call_tool = AsyncMock(
return_value=types.CallToolResult(content=[types.TextContent(type="text", text="result")])
)
self.session.list_prompts = AsyncMock(return_value=types.ListPromptsResult(prompts=[]))
def get_mcp_client(self) -> _AsyncGeneratorContextManager[Any, None]:
return None
server = TestServer(name="test_server")
async with server:
await server.load_tools()
await server.load_prompts()
with trace.use_span(trace.NonRecordingSpan(trace.INVALID_SPAN_CONTEXT)):
await server.call_tool("WorkIQSharePoint.readSmallBinaryFile", fileId="file-1")
assert server.session.call_tool.call_args.kwargs["meta"] == tool_meta
async def test_mcp_streamable_http_tool_hook_not_duplicated_on_repeated_get_mcp_client():
"""Test that calling get_mcp_client multiple times does not accumulate duplicate hooks."""
tool = MCPStreamableHTTPTool(
@@ -1015,11 +1015,25 @@ def test_observability_settings_is_setup_initial(monkeypatch):
assert settings.is_setup is False
# region Test enable_instrumentation function
def test_enable_sensitive_telemetry_function(monkeypatch):
"""Test enable_sensitive_telemetry function enables instrumentation."""
import importlib
monkeypatch.setenv("ENABLE_INSTRUMENTATION", "false")
monkeypatch.setenv("ENABLE_SENSITIVE_DATA", "false")
observability = importlib.import_module("agent_framework.observability")
importlib.reload(observability)
assert observability.OBSERVABILITY_SETTINGS.enable_instrumentation is False
observability.enable_sensitive_telemetry()
assert observability.OBSERVABILITY_SETTINGS.enable_instrumentation is True
assert observability.OBSERVABILITY_SETTINGS.enable_sensitive_data is True
def test_enable_instrumentation_function(monkeypatch):
"""Test enable_instrumentation function enables instrumentation."""
"""Test enable_instrumentation function enables instrumentation when disabled via env."""
import importlib
monkeypatch.setenv("ENABLE_INSTRUMENTATION", "false")
@@ -1032,10 +1046,12 @@ def test_enable_instrumentation_function(monkeypatch):
observability.enable_instrumentation()
assert observability.OBSERVABILITY_SETTINGS.enable_instrumentation is True
# Sensitive data should remain False when not explicitly enabled
assert observability.OBSERVABILITY_SETTINGS.enable_sensitive_data is False
def test_enable_instrumentation_with_sensitive_data(monkeypatch):
"""Test enable_instrumentation function with sensitive_data parameter."""
"""Test enable_instrumentation function with explicit sensitive_data parameter."""
import importlib
monkeypatch.setenv("ENABLE_INSTRUMENTATION", "false")
@@ -1049,111 +1065,6 @@ def test_enable_instrumentation_with_sensitive_data(monkeypatch):
assert observability.OBSERVABILITY_SETTINGS.enable_sensitive_data is True
def test_enable_instrumentation_reads_env_sensitive_data(monkeypatch):
"""Test enable_instrumentation re-reads ENABLE_SENSITIVE_DATA from os.environ when not explicitly passed."""
import importlib
monkeypatch.setenv("ENABLE_INSTRUMENTATION", "false")
monkeypatch.setenv("ENABLE_SENSITIVE_DATA", "false")
observability = importlib.import_module("agent_framework.observability")
importlib.reload(observability)
assert observability.OBSERVABILITY_SETTINGS.enable_sensitive_data is False
# Simulate load_dotenv() setting env var after import
monkeypatch.setenv("ENABLE_SENSITIVE_DATA", "true")
observability.enable_instrumentation()
assert observability.OBSERVABILITY_SETTINGS.enable_instrumentation is True
assert observability.OBSERVABILITY_SETTINGS.enable_sensitive_data is True
def test_configure_otel_providers_reads_env_sensitive_data(monkeypatch):
"""Test configure_otel_providers re-reads ENABLE_SENSITIVE_DATA from os.environ when not explicitly passed."""
import importlib
monkeypatch.setenv("ENABLE_INSTRUMENTATION", "false")
monkeypatch.setenv("ENABLE_SENSITIVE_DATA", "false")
monkeypatch.delenv("VS_CODE_EXTENSION_PORT", raising=False)
monkeypatch.delenv("ENABLE_CONSOLE_EXPORTERS", raising=False)
for key in [
"OTEL_EXPORTER_OTLP_ENDPOINT",
"OTEL_EXPORTER_OTLP_TRACES_ENDPOINT",
"OTEL_EXPORTER_OTLP_METRICS_ENDPOINT",
"OTEL_EXPORTER_OTLP_LOGS_ENDPOINT",
]:
monkeypatch.delenv(key, raising=False)
observability = importlib.import_module("agent_framework.observability")
importlib.reload(observability)
assert observability.OBSERVABILITY_SETTINGS.enable_sensitive_data is False
# Simulate load_dotenv() setting env var after import
monkeypatch.setenv("ENABLE_SENSITIVE_DATA", "true")
with patch.object(observability.OBSERVABILITY_SETTINGS, "_configure"):
observability.configure_otel_providers()
assert observability.OBSERVABILITY_SETTINGS.enable_instrumentation is True
assert observability.OBSERVABILITY_SETTINGS.enable_sensitive_data is True
def test_configure_otel_providers_reads_env_vs_code_port(monkeypatch):
"""Test configure_otel_providers re-reads VS_CODE_EXTENSION_PORT from os.environ when not explicitly passed."""
import importlib
from unittest.mock import patch as mock_patch
monkeypatch.setenv("ENABLE_INSTRUMENTATION", "false")
monkeypatch.delenv("VS_CODE_EXTENSION_PORT", raising=False)
monkeypatch.delenv("ENABLE_CONSOLE_EXPORTERS", raising=False)
for key in [
"OTEL_EXPORTER_OTLP_ENDPOINT",
"OTEL_EXPORTER_OTLP_TRACES_ENDPOINT",
"OTEL_EXPORTER_OTLP_METRICS_ENDPOINT",
"OTEL_EXPORTER_OTLP_LOGS_ENDPOINT",
]:
monkeypatch.delenv(key, raising=False)
observability = importlib.import_module("agent_framework.observability")
importlib.reload(observability)
assert observability.OBSERVABILITY_SETTINGS.vs_code_extension_port is None
# Simulate load_dotenv() setting env var after import
monkeypatch.setenv("VS_CODE_EXTENSION_PORT", "4317")
# Mock _configure to avoid needing optional OTLP gRPC exporter dependency
with mock_patch.object(observability.OBSERVABILITY_SETTINGS, "_configure"):
observability.configure_otel_providers()
assert observability.OBSERVABILITY_SETTINGS.vs_code_extension_port == 4317
def test_configure_otel_providers_explicit_param_overrides_env(monkeypatch):
"""Test that explicit parameters to configure_otel_providers override env vars."""
import importlib
monkeypatch.setenv("ENABLE_INSTRUMENTATION", "false")
monkeypatch.setenv("ENABLE_SENSITIVE_DATA", "true")
monkeypatch.delenv("VS_CODE_EXTENSION_PORT", raising=False)
monkeypatch.delenv("ENABLE_CONSOLE_EXPORTERS", raising=False)
for key in [
"OTEL_EXPORTER_OTLP_ENDPOINT",
"OTEL_EXPORTER_OTLP_TRACES_ENDPOINT",
"OTEL_EXPORTER_OTLP_METRICS_ENDPOINT",
"OTEL_EXPORTER_OTLP_LOGS_ENDPOINT",
]:
monkeypatch.delenv(key, raising=False)
observability = importlib.import_module("agent_framework.observability")
importlib.reload(observability)
# Explicit False should override the env var True
with patch.object(observability.OBSERVABILITY_SETTINGS, "_configure"):
observability.configure_otel_providers(enable_sensitive_data=False)
assert observability.OBSERVABILITY_SETTINGS.enable_sensitive_data is False
def test_enable_instrumentation_explicit_param_overrides_env(monkeypatch):
"""Test that explicit enable_sensitive_data parameter to enable_instrumentation overrides env var."""
import importlib
@@ -1269,6 +1180,161 @@ def test_enable_instrumentation_preserves_console_exporters_after_env_removed(mo
assert observability.OBSERVABILITY_SETTINGS.enable_console_exporters is True
def test_configure_otel_providers_reads_env_sensitive_data(monkeypatch):
"""Test configure_otel_providers re-reads ENABLE_SENSITIVE_DATA from os.environ when not explicitly passed."""
import importlib
monkeypatch.setenv("ENABLE_INSTRUMENTATION", "false")
monkeypatch.setenv("ENABLE_SENSITIVE_DATA", "false")
monkeypatch.delenv("VS_CODE_EXTENSION_PORT", raising=False)
monkeypatch.delenv("ENABLE_CONSOLE_EXPORTERS", raising=False)
for key in [
"OTEL_EXPORTER_OTLP_ENDPOINT",
"OTEL_EXPORTER_OTLP_TRACES_ENDPOINT",
"OTEL_EXPORTER_OTLP_METRICS_ENDPOINT",
"OTEL_EXPORTER_OTLP_LOGS_ENDPOINT",
]:
monkeypatch.delenv(key, raising=False)
observability = importlib.import_module("agent_framework.observability")
importlib.reload(observability)
assert observability.OBSERVABILITY_SETTINGS.enable_sensitive_data is False
# Simulate load_dotenv() setting env var after import
monkeypatch.setenv("ENABLE_SENSITIVE_DATA", "true")
with patch.object(observability.OBSERVABILITY_SETTINGS, "_configure"):
observability.configure_otel_providers()
assert observability.OBSERVABILITY_SETTINGS.enable_instrumentation is True
assert observability.OBSERVABILITY_SETTINGS.enable_sensitive_data is True
def test_configure_otel_providers_reads_env_vs_code_port(monkeypatch):
"""Test configure_otel_providers re-reads VS_CODE_EXTENSION_PORT from os.environ when not explicitly passed."""
import importlib
from unittest.mock import patch as mock_patch
monkeypatch.setenv("ENABLE_INSTRUMENTATION", "false")
monkeypatch.delenv("VS_CODE_EXTENSION_PORT", raising=False)
monkeypatch.delenv("ENABLE_CONSOLE_EXPORTERS", raising=False)
for key in [
"OTEL_EXPORTER_OTLP_ENDPOINT",
"OTEL_EXPORTER_OTLP_TRACES_ENDPOINT",
"OTEL_EXPORTER_OTLP_METRICS_ENDPOINT",
"OTEL_EXPORTER_OTLP_LOGS_ENDPOINT",
]:
monkeypatch.delenv(key, raising=False)
observability = importlib.import_module("agent_framework.observability")
importlib.reload(observability)
assert observability.OBSERVABILITY_SETTINGS.vs_code_extension_port is None
# Simulate load_dotenv() setting env var after import
monkeypatch.setenv("VS_CODE_EXTENSION_PORT", "4317")
# Mock _configure to avoid needing optional OTLP gRPC exporter dependency
with mock_patch.object(observability.OBSERVABILITY_SETTINGS, "_configure"):
observability.configure_otel_providers()
assert observability.OBSERVABILITY_SETTINGS.vs_code_extension_port == 4317
def test_configure_otel_providers_explicit_param_overrides_env(monkeypatch):
"""Test that explicit parameters to configure_otel_providers override env vars."""
import importlib
monkeypatch.setenv("ENABLE_INSTRUMENTATION", "false")
monkeypatch.setenv("ENABLE_SENSITIVE_DATA", "true")
monkeypatch.delenv("VS_CODE_EXTENSION_PORT", raising=False)
monkeypatch.delenv("ENABLE_CONSOLE_EXPORTERS", raising=False)
for key in [
"OTEL_EXPORTER_OTLP_ENDPOINT",
"OTEL_EXPORTER_OTLP_TRACES_ENDPOINT",
"OTEL_EXPORTER_OTLP_METRICS_ENDPOINT",
"OTEL_EXPORTER_OTLP_LOGS_ENDPOINT",
]:
monkeypatch.delenv(key, raising=False)
observability = importlib.import_module("agent_framework.observability")
importlib.reload(observability)
# Explicit False should override the env var True
with patch.object(observability.OBSERVABILITY_SETTINGS, "_configure"):
observability.configure_otel_providers(enable_sensitive_data=False)
assert observability.OBSERVABILITY_SETTINGS.enable_sensitive_data is False
def test_enable_sensitive_telemetry_does_not_touch_console_exporters(monkeypatch):
"""Test enable_sensitive_telemetry does not modify enable_console_exporters (it is an exporter concern)."""
import importlib
monkeypatch.setenv("ENABLE_INSTRUMENTATION", "false")
monkeypatch.delenv("ENABLE_CONSOLE_EXPORTERS", raising=False)
observability = importlib.import_module("agent_framework.observability")
importlib.reload(observability)
assert observability.OBSERVABILITY_SETTINGS.enable_console_exporters is False
# Simulate load_dotenv() setting env var after import
monkeypatch.setenv("ENABLE_CONSOLE_EXPORTERS", "true")
observability.enable_sensitive_telemetry()
# enable_console_exporters is not managed by enable_sensitive_telemetry;
# it is only read by configure_otel_providers.
assert observability.OBSERVABILITY_SETTINGS.enable_console_exporters is False
def test_enable_sensitive_telemetry_does_not_clobber_console_exporters(monkeypatch):
"""Test enable_sensitive_telemetry does not reset enable_console_exporters set by prior configure call."""
import importlib
monkeypatch.setenv("ENABLE_INSTRUMENTATION", "false")
monkeypatch.delenv("ENABLE_CONSOLE_EXPORTERS", raising=False)
monkeypatch.delenv("ENABLE_SENSITIVE_DATA", raising=False)
monkeypatch.delenv("VS_CODE_EXTENSION_PORT", raising=False)
for key in [
"OTEL_EXPORTER_OTLP_ENDPOINT",
"OTEL_EXPORTER_OTLP_TRACES_ENDPOINT",
"OTEL_EXPORTER_OTLP_METRICS_ENDPOINT",
"OTEL_EXPORTER_OTLP_LOGS_ENDPOINT",
]:
monkeypatch.delenv(key, raising=False)
observability = importlib.import_module("agent_framework.observability")
importlib.reload(observability)
# Set console exporters via configure_otel_providers
with patch.object(observability.OBSERVABILITY_SETTINGS, "_configure"):
observability.configure_otel_providers(enable_console_exporters=True)
assert observability.OBSERVABILITY_SETTINGS.enable_console_exporters is True
# Calling enable_sensitive_telemetry should not clobber the value
observability.enable_sensitive_telemetry()
assert observability.OBSERVABILITY_SETTINGS.enable_console_exporters is True
def test_enable_sensitive_telemetry_preserves_console_exporters_after_env_removed(monkeypatch):
"""Test enable_sensitive_telemetry preserves enable_console_exporters when env var is removed after reload."""
import importlib
monkeypatch.setenv("ENABLE_INSTRUMENTATION", "false")
monkeypatch.setenv("ENABLE_CONSOLE_EXPORTERS", "true")
observability = importlib.import_module("agent_framework.observability")
importlib.reload(observability)
assert observability.OBSERVABILITY_SETTINGS.enable_console_exporters is True
# Remove the env var after reload
monkeypatch.delenv("ENABLE_CONSOLE_EXPORTERS", raising=False)
# enable_sensitive_telemetry should not reset the value
observability.enable_sensitive_telemetry()
assert observability.OBSERVABILITY_SETTINGS.enable_console_exporters is True
def test_configure_otel_providers_reads_env_console_exporters(monkeypatch):
"""Test configure_otel_providers re-reads ENABLE_CONSOLE_EXPORTERS from os.environ when not explicitly passed."""
import importlib
@@ -1321,6 +1387,189 @@ def test_configure_otel_providers_explicit_console_exporters_overrides_env(monke
assert observability.OBSERVABILITY_SETTINGS.enable_console_exporters is False
# region Test default-on instrumentation
def test_observability_settings_defaults_instrumentation_true(monkeypatch):
"""ENABLE_INSTRUMENTATION unset → ObservabilitySettings defaults to True."""
from agent_framework.observability import ObservabilitySettings
monkeypatch.delenv("ENABLE_INSTRUMENTATION", raising=False)
settings = ObservabilitySettings()
assert settings.enable_instrumentation is True
def test_enable_instrumentation_reads_env_sensitive_data(monkeypatch):
"""No-arg enable_instrumentation() re-reads ENABLE_SENSITIVE_DATA from env at call time.
Covers the fallback branch where the env var is set AFTER import (e.g. via load_dotenv()).
"""
import importlib
monkeypatch.setenv("ENABLE_INSTRUMENTATION", "false")
monkeypatch.delenv("ENABLE_SENSITIVE_DATA", raising=False)
observability = importlib.import_module("agent_framework.observability")
importlib.reload(observability)
# Simulate load_dotenv() setting the env var after import
monkeypatch.setenv("ENABLE_SENSITIVE_DATA", "true")
observability.enable_instrumentation()
assert observability.OBSERVABILITY_SETTINGS.enable_instrumentation is True
assert observability.OBSERVABILITY_SETTINGS.enable_sensitive_data is True
# region Test disable_instrumentation sticky behavior
def test_disable_instrumentation_flips_settings_off(monkeypatch):
"""disable_instrumentation() immediately turns instrumentation and sensitive data off."""
import importlib
monkeypatch.delenv("ENABLE_INSTRUMENTATION", raising=False)
monkeypatch.setenv("ENABLE_SENSITIVE_DATA", "true")
observability = importlib.import_module("agent_framework.observability")
importlib.reload(observability)
observability.enable_sensitive_telemetry()
assert observability.OBSERVABILITY_SETTINGS.enable_instrumentation is True
assert observability.OBSERVABILITY_SETTINGS.enable_sensitive_data is True
assert observability.OBSERVABILITY_SETTINGS.SENSITIVE_DATA_ENABLED is True
observability.disable_instrumentation()
assert observability.OBSERVABILITY_SETTINGS.enable_instrumentation is False
assert observability.OBSERVABILITY_SETTINGS.enable_sensitive_data is False
assert observability.OBSERVABILITY_SETTINGS.SENSITIVE_DATA_ENABLED is False
assert observability.OBSERVABILITY_SETTINGS.ENABLED is False
def test_disable_instrumentation_is_sticky_against_enable_instrumentation(monkeypatch):
"""Sticky disable: enable_instrumentation() without force is a no-op after disable."""
import importlib
monkeypatch.delenv("ENABLE_INSTRUMENTATION", raising=False)
monkeypatch.delenv("ENABLE_SENSITIVE_DATA", raising=False)
observability = importlib.import_module("agent_framework.observability")
importlib.reload(observability)
observability.disable_instrumentation()
observability.enable_instrumentation(enable_sensitive_data=True)
assert observability.OBSERVABILITY_SETTINGS.enable_instrumentation is False
assert observability.OBSERVABILITY_SETTINGS.enable_sensitive_data is False
def test_disable_instrumentation_is_sticky_against_enable_sensitive_telemetry(monkeypatch):
"""Sticky disable: enable_sensitive_telemetry() without force is a no-op after disable."""
import importlib
monkeypatch.delenv("ENABLE_INSTRUMENTATION", raising=False)
monkeypatch.delenv("ENABLE_SENSITIVE_DATA", raising=False)
observability = importlib.import_module("agent_framework.observability")
importlib.reload(observability)
observability.disable_instrumentation()
observability.enable_sensitive_telemetry()
assert observability.OBSERVABILITY_SETTINGS.enable_instrumentation is False
assert observability.OBSERVABILITY_SETTINGS.enable_sensitive_data is False
def test_disable_instrumentation_is_sticky_against_configure_otel_providers(monkeypatch):
"""Sticky disable: configure_otel_providers() does not flip instrumentation back on."""
import importlib
monkeypatch.delenv("ENABLE_INSTRUMENTATION", raising=False)
monkeypatch.delenv("ENABLE_SENSITIVE_DATA", raising=False)
observability = importlib.import_module("agent_framework.observability")
importlib.reload(observability)
observability.disable_instrumentation()
with patch.object(observability.OBSERVABILITY_SETTINGS, "_configure"):
observability.configure_otel_providers(enable_sensitive_data=True)
assert observability.OBSERVABILITY_SETTINGS.enable_instrumentation is False
assert observability.OBSERVABILITY_SETTINGS.enable_sensitive_data is False
def test_disable_instrumentation_intercepts_direct_attribute_writes(monkeypatch):
"""Sticky disable: direct OBSERVABILITY_SETTINGS.enable_instrumentation = True is intercepted."""
import importlib
monkeypatch.delenv("ENABLE_INSTRUMENTATION", raising=False)
monkeypatch.delenv("ENABLE_SENSITIVE_DATA", raising=False)
observability = importlib.import_module("agent_framework.observability")
importlib.reload(observability)
observability.disable_instrumentation()
observability.OBSERVABILITY_SETTINGS.enable_instrumentation = True
observability.OBSERVABILITY_SETTINGS.enable_sensitive_data = True
assert observability.OBSERVABILITY_SETTINGS.enable_instrumentation is False
assert observability.OBSERVABILITY_SETTINGS.enable_sensitive_data is False
def test_enable_instrumentation_force_clears_disable(monkeypatch):
"""enable_instrumentation(force=True) clears the sticky disable."""
import importlib
monkeypatch.delenv("ENABLE_INSTRUMENTATION", raising=False)
monkeypatch.delenv("ENABLE_SENSITIVE_DATA", raising=False)
observability = importlib.import_module("agent_framework.observability")
importlib.reload(observability)
observability.disable_instrumentation()
observability.enable_instrumentation(force=True, enable_sensitive_data=True)
assert observability.OBSERVABILITY_SETTINGS.enable_instrumentation is True
assert observability.OBSERVABILITY_SETTINGS.enable_sensitive_data is True
def test_enable_sensitive_telemetry_force_clears_disable(monkeypatch):
"""enable_sensitive_telemetry(force=True) clears the sticky disable."""
import importlib
monkeypatch.delenv("ENABLE_INSTRUMENTATION", raising=False)
monkeypatch.delenv("ENABLE_SENSITIVE_DATA", raising=False)
observability = importlib.import_module("agent_framework.observability")
importlib.reload(observability)
observability.disable_instrumentation()
observability.enable_sensitive_telemetry(force=True)
assert observability.OBSERVABILITY_SETTINGS.enable_instrumentation is True
assert observability.OBSERVABILITY_SETTINGS.enable_sensitive_data is True
def test_disable_instrumentation_persists_after_force_until_redisabled(monkeypatch):
"""After force-enable then disable again, the sticky disable is re-armed."""
import importlib
monkeypatch.delenv("ENABLE_INSTRUMENTATION", raising=False)
monkeypatch.delenv("ENABLE_SENSITIVE_DATA", raising=False)
observability = importlib.import_module("agent_framework.observability")
importlib.reload(observability)
observability.disable_instrumentation()
observability.enable_instrumentation(force=True)
assert observability.OBSERVABILITY_SETTINGS.enable_instrumentation is True
observability.disable_instrumentation()
observability.enable_instrumentation()
assert observability.OBSERVABILITY_SETTINGS.enable_instrumentation is False
def test_disable_instrumentation_in_all(monkeypatch):
"""disable_instrumentation must be re-exported from the module's __all__."""
import agent_framework.observability as observability
assert "disable_instrumentation" in observability.__all__
assert callable(observability.disable_instrumentation)
# region Test _to_otel_part content types
@@ -3797,3 +4046,135 @@ async def test_agent_streaming_execute_failure_closes_span_and_resets_contextvar
agent_spans = [s for s in spans if s.attributes.get(OtelAttr.OPERATION.value) == OtelAttr.AGENT_INVOKE_OPERATION]
assert len(agent_spans) == 1
assert agent_spans[0].status.status_code == StatusCode.ERROR
# region Test heavy operations skipped when span is not recording
#
# When ``ENABLE_INSTRUMENTATION`` is on (the default) but no OpenTelemetry
# tracer provider has been configured, the global provider is the
# ``ProxyTracerProvider`` which returns non-recording spans. The telemetry
# layers gate sensitive-data serialization (``_capture_messages``) on
# ``span.is_recording()`` so that we don't pay the JSON-serialization cost
# when the span is going to be dropped anyway. The tests below verify that
# behavior by patching ``get_tracer`` to return a ``NoOpTracer``.
@pytest.mark.parametrize("enable_sensitive_data", [True], indirect=True)
async def test_chat_capture_messages_skipped_when_span_not_recording(
mock_chat_client, span_exporter: InMemorySpanExporter, enable_sensitive_data
):
"""Heavy message serialization is skipped when no provider is configured (non-streaming)."""
from opentelemetry.trace import NoOpTracer
client = mock_chat_client()
messages = [Message(role="user", contents=["Test"])]
span_exporter.clear()
with (
patch("agent_framework.observability.get_tracer", return_value=NoOpTracer()),
patch("agent_framework.observability._capture_messages") as mock_capture_messages,
patch("agent_framework.observability._capture_response") as mock_capture_response,
):
response = await client.get_response(messages=messages, options={"model": "Test"})
assert response is not None
# Sensitive-data serialization must be skipped because span.is_recording() is False.
assert mock_capture_messages.call_count == 0
# _capture_response still runs so that metric histograms continue to record.
assert mock_capture_response.call_count == 1
@pytest.mark.parametrize("enable_sensitive_data", [True], indirect=True)
async def test_chat_streaming_capture_messages_skipped_when_span_not_recording(
mock_chat_client, span_exporter: InMemorySpanExporter, enable_sensitive_data
):
"""Heavy message serialization is skipped when no provider is configured (streaming)."""
from opentelemetry.trace import NoOpTracer
client = mock_chat_client()
messages = [Message(role="user", contents=["Test"])]
span_exporter.clear()
with (
patch("agent_framework.observability.get_tracer", return_value=NoOpTracer()),
patch("agent_framework.observability._capture_messages") as mock_capture_messages,
patch("agent_framework.observability._capture_response") as mock_capture_response,
):
updates: list[ChatResponseUpdate] = []
stream = client.get_response(messages=messages, stream=True, options={"model": "Test"})
async for update in stream:
updates.append(update)
await stream.get_final_response()
assert len(updates) == 2
assert mock_capture_messages.call_count == 0
assert mock_capture_response.call_count == 1
@pytest.mark.parametrize("enable_sensitive_data", [True], indirect=True)
async def test_agent_capture_messages_skipped_when_span_not_recording(
mock_chat_agent, span_exporter: InMemorySpanExporter, enable_sensitive_data
):
"""Agent heavy serialization is skipped when no provider is configured (non-streaming)."""
from opentelemetry.trace import NoOpTracer
agent = mock_chat_agent()
span_exporter.clear()
with (
patch("agent_framework.observability.get_tracer", return_value=NoOpTracer()),
patch("agent_framework.observability._capture_messages") as mock_capture_messages,
patch("agent_framework.observability._capture_response") as mock_capture_response,
):
response = await agent.run("Test message")
assert response is not None
assert mock_capture_messages.call_count == 0
assert mock_capture_response.call_count == 1
@pytest.mark.parametrize("enable_sensitive_data", [True], indirect=True)
async def test_agent_streaming_capture_messages_skipped_when_span_not_recording(
mock_chat_agent, span_exporter: InMemorySpanExporter, enable_sensitive_data
):
"""Agent heavy serialization is skipped when no provider is configured (streaming)."""
from opentelemetry.trace import NoOpTracer
agent = mock_chat_agent()
span_exporter.clear()
with (
patch("agent_framework.observability.get_tracer", return_value=NoOpTracer()),
patch("agent_framework.observability._capture_messages") as mock_capture_messages,
patch("agent_framework.observability._capture_response") as mock_capture_response,
):
updates: list[Any] = []
stream = agent.run("Test message", stream=True)
async for update in stream:
updates.append(update)
await stream.get_final_response()
assert len(updates) == 2
assert mock_capture_messages.call_count == 0
assert mock_capture_response.call_count == 1
@pytest.mark.parametrize("enable_sensitive_data", [True], indirect=True)
async def test_chat_capture_messages_called_when_span_recording(
mock_chat_client, span_exporter: InMemorySpanExporter, enable_sensitive_data
):
"""Sanity check: with a real recording provider, sensitive-data capture still runs."""
client = mock_chat_client()
messages = [Message(role="user", contents=["Test"])]
span_exporter.clear()
with (
patch("agent_framework.observability._capture_messages") as mock_capture_messages,
patch("agent_framework.observability._capture_response") as mock_capture_response,
):
response = await client.get_response(messages=messages, options={"model": "Test"})
assert response is not None
# Two _capture_messages calls: one for input, one for output messages.
assert mock_capture_messages.call_count == 2
assert mock_capture_response.call_count == 1
+177 -12
View File
@@ -319,9 +319,7 @@ class TestDiscoverResourceFiles:
refs = skill_dir / "references"
refs.mkdir(parents=True)
(refs / "doc.md").write_text("content", encoding="utf-8")
resources = FileSkillsSource._discover_resource_files(
str(skill_dir), directories=("references", "references")
)
resources = FileSkillsSource._discover_resource_files(str(skill_dir), directories=("references", "references"))
assert resources == ["references/doc.md"]
def test_results_are_sorted(self, tmp_path: Path) -> None:
@@ -1675,9 +1673,7 @@ class TestValidateAndNormalizeDirectoryNames:
FileSkillsSource._validate_and_normalize_directory_names([" "])
def test_multiple_directories(self) -> None:
result = FileSkillsSource._validate_and_normalize_directory_names(
[".", "references", "assets", "scripts"]
)
result = FileSkillsSource._validate_and_normalize_directory_names([".", "references", "assets", "scripts"])
assert result == [".", "references", "assets", "scripts"]
def test_default_resource_directories(self) -> None:
@@ -3518,7 +3514,6 @@ class TestSkillsProviderFactories:
await _init_provider(provider)
run_tool = next(t for t in _ctx(provider)[2] if hasattr(t, "name") and t.name == "run_skill_script")
args_desc = run_tool.parameters()["properties"]["args"]["description"]
assert "without leading dashes" in args_desc
assert "script implementation or configured runner" in args_desc
async def test_require_script_approval_sets_approval_mode(self) -> None:
@@ -4744,12 +4739,16 @@ class TestCreateScriptElement:
def test_name_only(self) -> None:
s = FileSkillScript(name="run.py", full_path=f"{_ABS}/test/scripts/run.py")
elem = _create_script_element(s)
assert elem == ' <script name="run.py"/>'
assert 'name="run.py"' in elem
assert "<parameters_schema>" in elem
assert '"type": "array"' in elem
def test_with_description(self) -> None:
s = FileSkillScript(name="run.py", description="Execute script.", full_path=f"{_ABS}/test/scripts/run.py")
elem = _create_script_element(s)
assert elem == ' <script name="run.py" description="Execute script."/>'
assert 'name="run.py"' in elem
assert 'description="Execute script."' in elem
assert "<parameters_schema>" in elem
def test_xml_escapes_name(self) -> None:
s = FileSkillScript(name='script"special', full_path=f"{_ABS}/test/scripts/s.py")
@@ -4776,10 +4775,12 @@ class TestCreateScriptElement:
assert "query" in elem
assert "&quot;" not in elem
def test_no_parameters_for_file_script(self) -> None:
def test_file_script_includes_array_parameters(self) -> None:
s = FileSkillScript(name="run.py", full_path=f"{_ABS}/test/scripts/run.py")
elem = _create_script_element(s)
assert "<parameters_schema>" not in elem
assert "<parameters_schema>" in elem
assert '"type": "array"' in elem
assert '"type": "string"' in elem
# ---------------------------------------------------------------------------
@@ -4800,7 +4801,7 @@ class TestSkillScriptParametersSchema:
def test_none_for_file_based_script(self) -> None:
script = FileSkillScript(name="run.py", full_path=f"{_ABS}/test/scripts/run.py")
assert script.parameters_schema is None
assert script.parameters_schema == {"type": "array", "items": {"type": "string"}}
def test_no_params_function_returns_none(self) -> None:
def noop() -> None:
@@ -5407,3 +5408,167 @@ class TestInlineSkillContentCaching:
second = skill.content
assert first is second # Same object (cached)
assert "<name>test-skill</name>" in first
# ---------------------------------------------------------------------------
# Tests: Array-style (list[str]) script arguments
# ---------------------------------------------------------------------------
class TestArrayStyleScriptArgs:
"""Tests for list[str] arguments on skill scripts (port of .NET PR #5475)."""
async def test_inline_script_rejects_list_args(self) -> None:
"""InlineSkillScript.run() raises TypeError when args is a list."""
script = InlineSkillScript(name="greet", function=lambda name="world": f"hello {name}")
skill = InlineSkill(frontmatter=SkillFrontmatter(name="s", description="d"), instructions="c")
with pytest.raises(TypeError, match="requires keyword arguments"):
await script.run(skill, args=["hello", "--name", "Alice"])
async def test_inline_script_error_message_mentions_script_name(self) -> None:
"""The TypeError message includes the script name for debugging."""
script = InlineSkillScript(name="my-script", function=lambda: None)
skill = InlineSkill(frontmatter=SkillFrontmatter(name="s", description="d"), instructions="c")
with pytest.raises(TypeError, match="my-script"):
await script.run(skill, args=["arg1"])
async def test_file_script_passes_list_to_runner(self) -> None:
"""FileSkillScript.run() passes list[str] args through to the runner."""
captured: dict[str, Any] = {}
def runner(skill: Any, script: Any, args: Any = None) -> str:
captured["args"] = args
return "ok"
script = FileSkillScript(name="run.py", full_path=f"{_ABS}/test/run.py", runner=runner)
skill = FileSkill(
frontmatter=SkillFrontmatter(name="my-skill", description="d"), content="c", path=f"{_ABS}/test"
)
result = await script.run(skill, args=["input.docx", "--output", "result.idx"])
assert result == "ok"
assert captured["args"] == ["input.docx", "--output", "result.idx"]
async def test_file_script_passes_dict_to_runner(self) -> None:
"""FileSkillScript.run() still passes dict args through to the runner."""
captured: dict[str, Any] = {}
def runner(skill: Any, script: Any, args: Any = None) -> str:
captured["args"] = args
return "ok"
script = FileSkillScript(name="run.py", full_path=f"{_ABS}/test/run.py", runner=runner)
skill = FileSkill(
frontmatter=SkillFrontmatter(name="my-skill", description="d"), content="c", path=f"{_ABS}/test"
)
result = await script.run(skill, args={"key": "val"})
assert result == "ok"
assert captured["args"] == {"key": "val"}
async def test_file_script_passes_none_to_runner(self) -> None:
"""FileSkillScript.run() passes None args through to the runner."""
captured: dict[str, Any] = {}
def runner(skill: Any, script: Any, args: Any = None) -> str:
captured["args"] = args
return "ok"
script = FileSkillScript(name="run.py", full_path=f"{_ABS}/test/run.py", runner=runner)
skill = FileSkill(
frontmatter=SkillFrontmatter(name="my-skill", description="d"), content="c", path=f"{_ABS}/test"
)
result = await script.run(skill)
assert result == "ok"
assert captured["args"] is None
def test_file_script_parameters_schema_returns_array(self) -> None:
"""FileSkillScript.parameters_schema returns the string-array JSON schema."""
script = FileSkillScript(name="run.py", full_path=f"{_ABS}/test/run.py")
assert script.parameters_schema == {"type": "array", "items": {"type": "string"}}
async def test_runner_protocol_accepts_list_args(self) -> None:
"""A runner accepting list[str] args satisfies the SkillScriptRunner protocol."""
captured: dict[str, Any] = {}
def my_runner(skill: Any, script: Any, args: Any = None) -> str:
captured["args"] = args
return "ok"
assert isinstance(my_runner, SkillScriptRunner)
skill = FileSkill(frontmatter=SkillFrontmatter(name="s", description="d"), content="c", path=f"{_ABS}/test")
script = FileSkillScript(name="run.py", full_path=f"{_ABS}/test/run.py")
result = my_runner(skill, script, args=["--flag", "value"])
assert result == "ok"
assert captured["args"] == ["--flag", "value"]
async def test_tool_schema_accepts_array_args(self) -> None:
"""The run_skill_script tool schema accepts array-style args via oneOf."""
skill = InlineSkill(frontmatter=SkillFrontmatter(name="my-skill", description="test"), instructions="body")
skill.scripts.append(InlineSkillScript(name="s1", function=lambda: None))
provider = SkillsProvider([skill])
await _init_provider(provider)
run_tool = next(t for t in _ctx(provider)[2] if hasattr(t, "name") and t.name == "run_skill_script")
args_schema = run_tool.parameters()["properties"]["args"]
assert "oneOf" in args_schema
types = [s.get("type") for s in args_schema["oneOf"]]
assert "object" in types
assert "array" in types
assert "null" in types
async def test_run_skill_script_with_list_args_via_provider(self) -> None:
"""End-to-end: list args flow through provider to file-based script runner."""
captured: dict[str, Any] = {}
def runner(skill: Any, script: Any, args: Any = None) -> str:
captured["args"] = args
return "list_result"
script = FileSkillScript(name="run.py", full_path=f"{_ABS}/test/run.py", runner=runner)
skill = FileSkill(
frontmatter=SkillFrontmatter(name="my-skill", description="test"),
content="Body",
path=f"{_ABS}/test",
scripts=[script],
)
provider = SkillsProvider([skill])
await _init_provider(provider)
run_tool = next(t for t in _ctx(provider)[2] if hasattr(t, "name") and t.name == "run_skill_script")
result = await run_tool.func(skill_name="my-skill", script_name="run.py", args=["input.docx", "--verbose"])
assert result == "list_result"
assert captured["args"] == ["input.docx", "--verbose"]
async def test_run_skill_script_inline_with_list_args_returns_error(self) -> None:
"""Inline script called with list args through provider returns error (TypeError caught)."""
skill = InlineSkill(frontmatter=SkillFrontmatter(name="my-skill", description="test"), instructions="body")
skill.scripts.append(InlineSkillScript(name="s1", function=lambda: "ok"))
provider = SkillsProvider([skill])
await _init_provider(provider)
run_tool = next(t for t in _ctx(provider)[2] if hasattr(t, "name") and t.name == "run_skill_script")
result = await run_tool.func(skill_name="my-skill", script_name="s1", args=["arg1"])
assert "Error" in result
assert "Failed to run" in result
def test_file_skill_content_includes_scripts_block(self) -> None:
"""FileSkill.content appends a <scripts> block when scripts are present."""
script = FileSkillScript(name="run.py", full_path=f"{_ABS}/test/run.py")
skill = FileSkill(
frontmatter=SkillFrontmatter(name="my-skill", description="test"),
content="---\nname: my-skill\n---\nBody",
path=f"{_ABS}/test",
scripts=[script],
)
assert "<scripts>" in skill.content
assert 'name="run.py"' in skill.content
assert "<parameters_schema>" in skill.content
assert '"type": "array"' in skill.content
def test_file_skill_content_no_scripts_no_block(self) -> None:
"""FileSkill.content does not append a <scripts> block when no scripts."""
skill = FileSkill(
frontmatter=SkillFrontmatter(name="my-skill", description="test"),
content="---\nname: my-skill\n---\nBody",
path=f"{_ABS}/test",
)
assert "<scripts>" not in skill.content
+6
View File
@@ -34,6 +34,12 @@ devui ./agents
devui --entities my_agent.py
```
## Security Posture
DevUI is a development-only sample app, not a production hosting surface. Authentication is enabled by default.
Unauthenticated mode is allowed only on `localhost` / `127.0.0.1`; `0.0.0.0`, LAN IPs, and hostnames require
`DEVUI_AUTH_TOKEN` or `--auth-token`.
## Import Path
```python
+30 -13
View File
@@ -47,6 +47,9 @@ devui ./agents --port 8080
# → API: http://localhost:8080/v1/*
```
DevUI is auth-enabled by default. Localhost starts with a generated development token logged at startup; pass it as
`Authorization: Bearer <token>` for direct API calls.
When DevUI starts with no discovered entities, it displays a **sample entity gallery** with curated examples from the Agent Framework repository. You can download these samples, review them, and run them locally to get started quickly.
## Using MCP Tools
@@ -137,12 +140,14 @@ For convenience, DevUI provides an OpenAI Responses backend API. This means you
```bash
# Simple - use your entity name as the entity_id in metadata
curl -X POST http://localhost:8080/v1/responses \
-H "Authorization: Bearer <devui-token>" \
-H "Content-Type: application/json" \
-d @- << 'EOF'
{
"metadata": {"entity_id": "weather_agent"},
"input": "Hello world"
}
EOF
```
Or use the OpenAI Python SDK:
@@ -152,7 +157,7 @@ from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8080/v1",
api_key="not-needed" # API key not required for local DevUI
api_key="<devui-token>"
)
response = client.responses.create(
@@ -201,6 +206,7 @@ DevUI provides an **OpenAI Proxy** feature for testing OpenAI models directly th
```bash
curl -X POST http://localhost:8080/v1/responses \
-H "Authorization: Bearer <devui-token>" \
-H "X-Proxy-Backend: openai" \
-d '{"model": "gpt-4.1-mini", "input": "Hello"}'
```
@@ -214,14 +220,14 @@ devui [directory] [options]
Options:
--port, -p Port (default: 8080)
--host Host (default: 127.0.0.1)
--host Host (default: 127.0.0.1; non-loopback hosts require auth)
--headless API only, no UI
--no-open Don't automatically open browser
--instrumentation Enable OpenTelemetry instrumentation
--reload Enable auto-reload
--mode developer|user (default: developer)
--auth Enable Bearer token authentication
--auth-token Custom authentication token
--no-auth Disable auth for loopback-only local development
--auth-token Custom authentication token (required for non-loopback hosts unless DEVUI_AUTH_TOKEN is set)
```
### UI Modes
@@ -233,8 +239,8 @@ Options:
# Development
devui ./agents
# Production (user-facing)
devui ./agents --mode user --auth
# Local-only no-auth development
devui ./agents --no-auth
```
## Key Endpoints
@@ -336,28 +342,39 @@ These custom extensions are clearly namespaced and can be safely ignored by stan
## Security
DevUI is designed as a **sample application for local development** and should not be exposed to untrusted networks without proper authentication.
DevUI is designed as a **sample application for local development** and is not intended for production use. For
production, or for features beyond this sample app, build a custom interface and API server using the Agent Framework SDK.
**For production deployments:**
Auth is enabled by default. Unauthenticated mode is allowed only when DevUI is bound to `localhost` or `127.0.0.1`.
Network-reachable binds such as `0.0.0.0`, LAN IPs, and hostnames require Bearer token authentication with an explicit
token.
**For shared development hosts:**
```bash
# User mode with authentication (recommended)
devui ./agents --mode user --auth --host 0.0.0.0
# Set a token explicitly before binding beyond loopback
DEVUI_AUTH_TOKEN="<secure-dev-token>" devui ./agents --mode user --host 0.0.0.0
# Or pass the token on the command line
devui ./agents --mode user --host 0.0.0.0 --auth-token "<secure-dev-token>"
```
This restricts developer APIs (reload, deployment, entity details) and requires Bearer token authentication.
Do not use `--no-auth` with `0.0.0.0`, LAN IPs, or hostnames. That configuration fails closed before startup.
**Security features:**
- User mode restricts developer-facing APIs
- Optional Bearer token authentication via `--auth`
- Bearer token authentication is enabled by default
- Unauthenticated mode is loopback-only (`localhost` / `127.0.0.1`)
- Non-loopback binds require `DEVUI_AUTH_TOKEN` or `--auth-token`
- Only loads entities from local directories or in-memory registration
- No remote code execution capabilities
- Binds to localhost (127.0.0.1) by default
**Best practices:**
- Use `--mode user --auth` for any deployment exposed to end users
- Do not use DevUI as a production deployment surface
- Use `--mode user` plus `DEVUI_AUTH_TOKEN` or `--auth-token` for shared development hosts
- Review all agent/workflow code before running
- Only load entities from trusted sources
- Use `.env` files for sensitive credentials (never commit them)
@@ -126,29 +126,6 @@ def serve(
if not isinstance(port, int) or not (1 <= port <= 65535):
raise ValueError(f"Invalid port: {port}. Must be integer between 1 and 65535")
# Security check: warn loudly when network-exposed without authentication.
if host not in ("127.0.0.1", "localhost") and not auth_enabled:
logger.warning("WARNING: Exposing DevUI to the network with --no-auth.")
logger.warning("Anyone on your network can read agent metadata and trigger requests.")
logger.warning("Drop --no-auth and DevUI will require Bearer tokens.")
# Refuse to auto-generate a token for network-exposed binds. Auto-generated tokens
# are fine for localhost convenience; for anything else, require an explicit token.
if auth_enabled and not auth_token:
import os
env_token = os.environ.get("DEVUI_AUTH_TOKEN")
if not env_token:
is_production = (
host not in ("127.0.0.1", "localhost")
or os.environ.get("CI") == "true"
or os.environ.get("KUBERNETES_SERVICE_HOST")
)
if is_production:
logger.error("Authentication required but no token provided.")
logger.error("Set DEVUI_AUTH_TOKEN env var or pass auth_token='...' to serve().")
raise ValueError("DEVUI_AUTH_TOKEN required when host is not localhost")
# Enable instrumentation if requested
if instrumentation_enabled:
from agent_framework.observability import enable_instrumentation
@@ -81,13 +81,15 @@ Examples:
parser.add_argument(
"--no-auth",
action="store_true",
help="Disable Bearer token authentication. DevUI is auth-enabled by default; use this to opt out.",
help=(
"Disable Bearer token authentication for loopback-only local development. Non-loopback hosts require auth."
),
)
parser.add_argument(
"--auth-token",
type=str,
help="Custom Bearer token. Auto-generated and logged at startup when omitted.",
help="Custom Bearer token. Required for non-loopback hosts when DEVUI_AUTH_TOKEN is not set.",
)
parser.add_argument("--version", action="version", version=f"Agent Framework DevUI {get_version()}")
@@ -89,7 +89,7 @@ class DevServer:
mode: Server mode - 'developer' (full access, verbose errors) or 'user' (restricted APIs, generic errors)
auth_enabled: Whether to require Bearer token auth on /v1/* endpoints. Defaults to True.
auth_token: Bearer token. If None and auth_enabled, falls back to the DEVUI_AUTH_TOKEN
environment variable, then to an auto-generated token (logged at startup).
environment variable. Loopback binds may use an auto-generated token logged at startup.
"""
self.entities_dir = entities_dir
self.port = port
@@ -106,7 +106,7 @@ class DevServer:
self.ui_enabled = ui_enabled
self.mode = mode
self.auth_enabled = auth_enabled
self.auth_token = self._resolve_auth_token(auth_enabled, auth_token)
self.auth_token = self._resolve_auth_token(host, auth_enabled, auth_token)
self.executor: AgentFrameworkExecutor | None = None
self.openai_executor: OpenAIExecutor | None = None
self.deployment_manager = DeploymentManager()
@@ -118,8 +118,14 @@ class DevServer:
"""Set in-memory entities to register on startup."""
self._pending_entities = entities
_AUTH_LOOPBACK_HOSTS = frozenset({"127.0.0.1", "localhost"})
_LOOPBACK_HOSTS = frozenset({"127.0.0.1", "localhost", "[::1]", "::1"})
@classmethod
def _is_auth_loopback_host(cls, host: str) -> bool:
"""Return True when unauthenticated DevUI may be limited to local loopback."""
return host.lower() in cls._AUTH_LOOPBACK_HOSTS
def _loopback_allowed_hosts(self) -> frozenset[str] | None:
"""Return the Host-header allowlist when bound to a loopback interface, else None.
@@ -131,16 +137,25 @@ class DevServer:
return None
return self._LOOPBACK_HOSTS
@staticmethod
def _resolve_auth_token(auth_enabled: bool, auth_token: str | None) -> str | None:
@classmethod
def _resolve_auth_token(cls, host: str, auth_enabled: bool, auth_token: str | None) -> str | None:
"""Resolve the active Bearer token. Returns None when auth is disabled."""
is_loopback = cls._is_auth_loopback_host(host)
if not auth_enabled:
if not is_loopback:
raise ValueError(
"DevUI authentication cannot be disabled for non-loopback hosts. "
"Bind to 127.0.0.1/localhost for no-auth local development, or enable auth and provide "
"DEVUI_AUTH_TOKEN or auth_token for network-reachable binds."
)
return None
if auth_token:
return auth_token
env_token = os.getenv("DEVUI_AUTH_TOKEN")
if env_token:
return env_token
if not is_loopback:
raise ValueError("DEVUI_AUTH_TOKEN or auth_token is required when DevUI is bound to a non-loopback host.")
generated = secrets.token_urlsafe(32)
logger.info("=" * 70)
logger.info("DevUI authentication enabled with auto-generated token:")
+7
View File
@@ -60,6 +60,9 @@ devui
This launches the UI with all example agents/workflows at http://localhost:8080
DevUI is auth-enabled by default. Copy the generated token from startup logs and pass it as
`Authorization: Bearer <token>` for direct API calls. Use `--no-auth` only for loopback-only local testing.
## 5. What You'll See
- A web interface for testing agents interactively
@@ -74,6 +77,7 @@ You can also test via API calls:
```bash
curl -X POST http://localhost:8080/v1/responses \
-H "Authorization: Bearer <devui-token>" \
-H "Content-Type: application/json" \
-d '{
"model": "weather_agent",
@@ -86,6 +90,7 @@ curl -X POST http://localhost:8080/v1/responses \
```bash
# Create a conversation
curl -X POST http://localhost:8080/v1/conversations \
-H "Authorization: Bearer <devui-token>" \
-H "Content-Type: application/json" \
-d '{"metadata": {"agent_id": "weather_agent"}}'
@@ -93,6 +98,7 @@ curl -X POST http://localhost:8080/v1/conversations \
# Use conversation ID in requests
curl -X POST http://localhost:8080/v1/responses \
-H "Authorization: Bearer <devui-token>" \
-H "Content-Type: application/json" \
-d '{
"model": "weather_agent",
@@ -102,6 +108,7 @@ curl -X POST http://localhost:8080/v1/responses \
# Continue the conversation
curl -X POST http://localhost:8080/v1/responses \
-H "Authorization: Bearer <devui-token>" \
-H "Content-Type: application/json" \
-d '{
"model": "weather_agent",
@@ -4,8 +4,10 @@
import asyncio
import inspect
import sys
import tempfile
from pathlib import Path
from typing import Any
import pytest
from conftest import MockAgent
@@ -492,7 +494,7 @@ def test_devserver_requires_auth_by_default(monkeypatch):
def test_devserver_auth_can_be_explicitly_disabled(monkeypatch):
"""Callers can opt out of auth with auth_enabled=False (escape hatch for tests / trusted hosts)."""
"""Callers can opt out of auth on loopback (escape hatch for tests / trusted local hosts)."""
monkeypatch.delenv("DEVUI_AUTH_TOKEN", raising=False)
server = _server_with_mock_agent(auth_enabled=False)
@@ -504,6 +506,106 @@ def test_devserver_auth_can_be_explicitly_disabled(monkeypatch):
assert response.status_code == 200
def test_devserver_rejects_non_loopback_no_auth(monkeypatch):
"""Non-loopback binds must not be network-reachable without authentication."""
monkeypatch.delenv("DEVUI_AUTH_TOKEN", raising=False)
with pytest.raises(ValueError, match="authentication cannot be disabled"):
DevServer(host="0.0.0.0", auth_enabled=False)
with pytest.raises(ValueError, match="authentication cannot be disabled"):
DevServer(host="devui.example", auth_enabled=False)
def test_devserver_rejects_non_loopback_without_explicit_token(monkeypatch):
"""Network-reachable auth requires an operator-provided token, not a generated token."""
monkeypatch.delenv("DEVUI_AUTH_TOKEN", raising=False)
with pytest.raises(ValueError, match="DEVUI_AUTH_TOKEN or auth_token"):
DevServer(host="0.0.0.0")
def test_devserver_allows_non_loopback_with_explicit_token(monkeypatch):
"""A network-reachable bind is allowed when auth has an explicit token."""
monkeypatch.delenv("DEVUI_AUTH_TOKEN", raising=False)
server = DevServer(host="0.0.0.0", auth_token="s3cret")
assert server.auth_enabled is True
assert server.auth_token == "s3cret"
def test_devserver_allows_non_loopback_with_env_token(monkeypatch):
"""A network-reachable bind is allowed when auth uses DEVUI_AUTH_TOKEN."""
monkeypatch.setenv("DEVUI_AUTH_TOKEN", "env-s3cret")
server = DevServer(host="0.0.0.0")
assert server.auth_enabled is True
assert server.auth_token == "env-s3cret"
def test_devserver_allows_loopback_no_auth(monkeypatch):
"""Unauthenticated DevUI remains available for local-only development and tests."""
monkeypatch.delenv("DEVUI_AUTH_TOKEN", raising=False)
for host in ("127.0.0.1", "localhost"):
server = DevServer(host=host, auth_enabled=False)
assert server.auth_enabled is False
assert server.auth_token is None
def test_devserver_loopback_auth_auto_generates_token(monkeypatch):
"""Loopback auth-enabled usage may still use a generated development token."""
monkeypatch.delenv("DEVUI_AUTH_TOKEN", raising=False)
server = DevServer(host="127.0.0.1")
assert server.auth_enabled is True
assert server.auth_token
def test_serve_rejects_non_loopback_no_auth(monkeypatch):
"""The public serve() helper must inherit the DevServer network-auth invariant."""
monkeypatch.delenv("DEVUI_AUTH_TOKEN", raising=False)
with pytest.raises(ValueError, match="authentication cannot be disabled"):
agent_framework_devui.serve(entities=[], host="0.0.0.0", auth_enabled=False, ui_enabled=False)
def test_serve_rejects_non_loopback_without_explicit_token(monkeypatch):
"""serve() must not maintain a weaker generated-token path for network binds."""
monkeypatch.delenv("DEVUI_AUTH_TOKEN", raising=False)
with pytest.raises(ValueError, match="DEVUI_AUTH_TOKEN or auth_token"):
agent_framework_devui.serve(entities=[], host="0.0.0.0", ui_enabled=False)
def test_serve_allows_non_loopback_with_explicit_token(monkeypatch):
"""serve() accepts a network bind when an explicit token is provided."""
import uvicorn
monkeypatch.delenv("DEVUI_AUTH_TOKEN", raising=False)
run_args = {}
def fake_run(_app, *, host, port, **_kwargs):
run_args["host"] = host
run_args["port"] = port
monkeypatch.setattr(uvicorn, "run", fake_run)
agent_framework_devui.serve(
entities=[],
host="0.0.0.0",
port=9090,
auth_token="s3cret",
auto_open=False,
ui_enabled=False,
)
assert run_args == {"host": "0.0.0.0", "port": 9090}
def test_devserver_accepts_request_with_valid_bearer_token(monkeypatch):
"""When auth is on, supplying the configured Bearer token grants access."""
monkeypatch.delenv("DEVUI_AUTH_TOKEN", raising=False)
@@ -567,8 +669,8 @@ def test_serve_defaults_to_auth_enabled():
)
def test_cli_enables_auth_by_default_and_supports_no_auth_optout():
"""`devui ./agents` must produce auth-enabled config; `--no-auth` is the explicit escape hatch."""
def test_cli_enables_auth_by_default_and_supports_loopback_no_auth_optout():
"""`devui ./agents` must produce auth-enabled config; `--no-auth` is the loopback-only escape hatch."""
from agent_framework_devui._cli import create_cli_parser
parser = create_cli_parser()
@@ -578,3 +680,76 @@ def test_cli_enables_auth_by_default_and_supports_no_auth_optout():
optout_args = parser.parse_args(["--no-auth"])
assert optout_args.no_auth is True
help_text = parser.format_help()
assert "loopback-only" in help_text
assert "Non-loopback hosts require auth" in help_text
def _run_cli_with_fake_uvicorn(monkeypatch, tmp_path: Path, *args: str) -> dict[str, Any]:
"""Run the DevUI CLI without binding a socket."""
import uvicorn
from agent_framework_devui import _cli
run_args: dict[str, Any] = {}
def fake_run(_app, *, host, port, **_kwargs):
run_args["host"] = host
run_args["port"] = port
monkeypatch.setattr(uvicorn, "run", fake_run)
monkeypatch.setattr(sys, "argv", ["devui", str(tmp_path), "--no-open", "--headless", *args])
_cli.main()
return run_args
def test_cli_allows_loopback_no_auth_without_binding_socket(monkeypatch, tmp_path):
"""`devui --no-auth` remains valid on the default loopback host."""
monkeypatch.delenv("DEVUI_AUTH_TOKEN", raising=False)
run_args = _run_cli_with_fake_uvicorn(monkeypatch, tmp_path, "--no-auth")
assert run_args == {"host": "127.0.0.1", "port": 8080}
def test_cli_rejects_non_loopback_no_auth_before_binding_socket(monkeypatch, tmp_path, capsys):
"""`devui --host 0.0.0.0 --no-auth` must fail through shared server validation."""
monkeypatch.delenv("DEVUI_AUTH_TOKEN", raising=False)
with pytest.raises(SystemExit) as exc_info:
_run_cli_with_fake_uvicorn(monkeypatch, tmp_path, "--host", "0.0.0.0", "--no-auth")
assert exc_info.value.code == 1
assert "authentication cannot be disabled" in capsys.readouterr().err
def test_cli_rejects_non_loopback_without_explicit_token_before_binding_socket(monkeypatch, tmp_path, capsys):
"""`devui --host 0.0.0.0` must fail when neither --auth-token nor DEVUI_AUTH_TOKEN is set."""
monkeypatch.delenv("DEVUI_AUTH_TOKEN", raising=False)
with pytest.raises(SystemExit) as exc_info:
_run_cli_with_fake_uvicorn(monkeypatch, tmp_path, "--host", "0.0.0.0")
assert exc_info.value.code == 1
assert "DEVUI_AUTH_TOKEN or auth_token" in capsys.readouterr().err
def test_cli_allows_non_loopback_with_auth_token_without_binding_socket(monkeypatch, tmp_path):
"""`devui --host 0.0.0.0 --auth-token ...` starts with token auth enabled."""
monkeypatch.delenv("DEVUI_AUTH_TOKEN", raising=False)
run_args = _run_cli_with_fake_uvicorn(monkeypatch, tmp_path, "--host", "0.0.0.0", "--auth-token", "s3cret")
assert run_args == {"host": "0.0.0.0", "port": 8080}
def test_cli_allows_non_loopback_with_env_token_without_binding_socket(monkeypatch, tmp_path):
"""`DEVUI_AUTH_TOKEN=... devui --host 0.0.0.0` starts with token auth enabled."""
monkeypatch.setenv("DEVUI_AUTH_TOKEN", "env-s3cret")
run_args = _run_cli_with_fake_uvicorn(monkeypatch, tmp_path, "--host", "0.0.0.0")
assert run_args == {"host": "0.0.0.0", "port": 8080}
@@ -793,8 +793,22 @@ class RawFoundryAgent( # type: ignore[misc]
Raises:
ImportError: If azure-monitor-opentelemetry-exporter is not installed.
"""
from agent_framework.observability import (
OBSERVABILITY_SETTINGS,
create_metric_views,
create_resource,
enable_instrumentation,
)
from azure.core.exceptions import ResourceNotFoundError
if OBSERVABILITY_SETTINGS.is_user_disabled:
logger.info(
"FoundryAgent.configure_azure_monitor(): Skipping setup because instrumentation was "
"explicitly disabled via disable_instrumentation(). Call enable_instrumentation(force=True) "
"to re-enable, then re-invoke configure_azure_monitor()."
)
return
client = self.client
if not isinstance(client, RawFoundryAgentChatClient):
raise TypeError("configure_azure_monitor requires a RawFoundryAgentChatClient-based client.")
@@ -817,8 +831,6 @@ class RawFoundryAgent( # type: ignore[misc]
"Install it with: pip install azure-monitor-opentelemetry"
) from exc
from agent_framework.observability import create_metric_views, create_resource, enable_instrumentation
if "resource" not in kwargs:
kwargs["resource"] = create_resource()
@@ -271,8 +271,22 @@ class RawFoundryChatClient( # type: ignore[misc]
Raises:
ImportError: If azure-monitor-opentelemetry-exporter is not installed.
"""
from agent_framework.observability import (
OBSERVABILITY_SETTINGS,
create_metric_views,
create_resource,
enable_instrumentation,
)
from azure.core.exceptions import ResourceNotFoundError
if OBSERVABILITY_SETTINGS.is_user_disabled:
logger.info(
"FoundryChatClient.configure_azure_monitor(): Skipping setup because instrumentation was "
"explicitly disabled via disable_instrumentation(). Call enable_instrumentation(force=True) "
"to re-enable, then re-invoke configure_azure_monitor()."
)
return
try:
conn_string = await self.project_client.telemetry.get_application_insights_connection_string()
except ResourceNotFoundError:
@@ -291,8 +305,6 @@ class RawFoundryChatClient( # type: ignore[misc]
"Install it with: pip install azure-monitor-opentelemetry"
) from exc
from agent_framework.observability import create_metric_views, create_resource, enable_instrumentation
if "resource" not in kwargs:
kwargs["resource"] = create_resource()
@@ -11,6 +11,7 @@ import tempfile
import threading
from collections.abc import AsyncIterable, AsyncIterator, Generator, Mapping, Sequence
from contextlib import suppress
from pathlib import Path
from typing import Protocol, cast
from agent_framework import (
@@ -205,6 +206,47 @@ class FileBasedFunctionApprovalStorage:
return await asyncio.to_thread(self._load_sync, approval_request_id)
def _checkpoint_storage_for_context(root: str, context_id: str) -> FileCheckpointStorage:
"""Build a ``FileCheckpointStorage`` for ``context_id`` rooted under ``root``.
``context_id`` originates from caller-controlled fields such as
``previous_response_id`` or from server-generated fields such as
``conversation_id`` / ``response_id``. In every case it must be treated as
an untrusted single path segment: path separators, drive letters, parent
references and similar would otherwise let the resulting directory escape
the configured checkpoint root (CWE-22). The check resolves the joined
path and verifies it stays under the resolved root before any directory is
created on disk.
"""
if not isinstance(context_id, str) or not context_id:
raise RuntimeError("Invalid checkpoint context id: must be a non-empty string.")
# Reject any segment that is not a single safe path component. This covers
# POSIX/Windows separators, NUL bytes, drive letters, and all-dot segments
# (``.``, ``..``, ``...``, ...). We deliberately do not URL-decode the id
# here: the hosting layer never decodes context ids before joining them, so
# forms such as ``%2e%2e`` are accepted as literal directory names. Do NOT
# add decoding here without re-validating after the decode -- decode-then-
# join is exactly the pattern that reintroduces traversal. We also do not
# attempt to "sanitize" by stripping characters because that can introduce
# collisions between distinct ids.
if (
"/" in context_id
or "\\" in context_id
or "\x00" in context_id
# All-dot segments (``.``, ``..``, ``...``, ...) reduce to "" after stripping dots.
or context_id.strip(".") == ""
or os.path.isabs(context_id)
or os.path.splitdrive(context_id)[0]
):
raise RuntimeError(f"Invalid checkpoint context id: {context_id!r}")
root_path = Path(root).resolve()
storage_path = (root_path / context_id).resolve()
if not storage_path.is_relative_to(root_path):
raise RuntimeError(f"Invalid checkpoint context id: {context_id!r}")
return FileCheckpointStorage(storage_path)
class ResponsesHostServer(ResponsesAgentServerHost):
"""A responses server host for an agent."""
@@ -400,7 +442,7 @@ class ResponsesHostServer(ResponsesAgentServerHost):
latest_checkpoint_id: str | None = None
restore_storage: FileCheckpointStorage | None = None
if context_id is not None:
restore_storage = FileCheckpointStorage(os.path.join(self._checkpoint_storage_path, context_id))
restore_storage = _checkpoint_storage_for_context(self._checkpoint_storage_path, context_id)
latest_checkpoint = await restore_storage.get_latest(workflow_name=self._agent.workflow.name)
if latest_checkpoint is not None:
latest_checkpoint_id = latest_checkpoint.checkpoint_id
@@ -414,7 +456,7 @@ class ResponsesHostServer(ResponsesAgentServerHost):
# supplied, restore_storage points at the *prior* response's
# directory and write_storage points at the *current* response's.
write_context_id = context.conversation_id or context.response_id
write_storage = FileCheckpointStorage(os.path.join(self._checkpoint_storage_path, write_context_id))
write_storage = _checkpoint_storage_for_context(self._checkpoint_storage_path, write_context_id)
# Multi-turn pattern: when we have a prior checkpoint, restore it
# first (drive the workflow back to idle with prior state intact),
@@ -11,7 +11,7 @@ the registered _handle_create handler.
from __future__ import annotations
import json
from collections.abc import AsyncIterator
from collections.abc import AsyncIterator, Callable
from unittest.mock import AsyncMock, MagicMock
import httpx
@@ -20,6 +20,7 @@ from agent_framework import (
AgentResponse,
AgentResponseUpdate,
Content,
FileCheckpointStorage,
HistoryProvider,
Message,
RawAgent,
@@ -2652,3 +2653,241 @@ class TestFunctionApprovalRoundTrip:
# endregion
# region Checkpoint context path validation
class TestCheckpointContextPathValidation:
"""Regression tests for the path-traversal hardening of checkpoint storage.
These tests guard against CWE-22 in the workflow hosting path. The hosting
code joins caller-supplied identifiers (``previous_response_id``) and
server-generated identifiers (``conversation_id`` / ``response_id``) under
the configured checkpoint root. Without validation, traversal segments
such as ``../../escape`` or absolute paths cause directory creation
outside the intended root.
"""
@staticmethod
def _helper() -> Callable[[str, str], FileCheckpointStorage]:
from agent_framework_foundry_hosting._responses import ( # pyright: ignore[reportPrivateUsage]
_checkpoint_storage_for_context,
)
return _checkpoint_storage_for_context
def test_valid_segment_creates_storage_under_root(self, tmp_path: Any) -> None:
helper = self._helper()
root = tmp_path / "root"
root.mkdir()
storage = helper(str(root), "resp_abc123")
assert storage.storage_path.is_dir()
assert storage.storage_path.parent == root.resolve()
@pytest.mark.parametrize(
"bad_id",
[
# Original MSRC repro: traversal embedded inside an id-shaped value.
# The 14 ``A``s pad the suffix to mimic the exact length of the
# ``api-made-dir<14-char-suffix>`` segment from the original report.
"caresp_x/../../service-data/api-made-dir" + "A" * 14,
# Variant report repros.
"../../escape",
"..",
".",
"...",
"/tmp/escape",
"/absolute/path",
"C:\\temp\\escape",
"..\\..\\escape",
"foo\\..\\bar",
"foo/bar",
"with\x00null",
"",
],
)
def test_traversal_and_separator_payloads_are_rejected(self, tmp_path: Any, bad_id: str) -> None:
helper = self._helper()
# Use a dedicated root *inside* tmp_path so we can assert that nothing
# was created anywhere under tmp_path (root, siblings, or above).
# Asserting against tmp_path.parent would be flaky under parallel test
# execution because tmp_path.parent is shared across tests.
root = tmp_path / "root"
root.mkdir()
before = sorted(p.name for p in tmp_path.iterdir())
with pytest.raises(RuntimeError):
helper(str(root), bad_id)
# No sibling/escape directory should have been created next to the root.
after = sorted(p.name for p in tmp_path.iterdir())
assert before == after, f"Unexpected filesystem artifacts created for payload {bad_id!r}"
# And nothing inside the root either.
assert list(root.iterdir()) == []
def test_non_string_context_id_is_rejected(self, tmp_path: Any) -> None:
helper = self._helper()
with pytest.raises(RuntimeError):
helper(str(tmp_path), None) # type: ignore[arg-type]
def test_url_encoded_traversal_is_treated_as_literal_segment(self, tmp_path: Any) -> None:
"""URL-encoded traversal should not decode to traversal at the filesystem layer.
The hosting layer never URL-decodes ids before using them; the helper
should accept ``%2e%2e`` as a single literal segment (no escape).
"""
helper = self._helper()
root = tmp_path / "root"
root.mkdir()
storage = helper(str(root), "%2e%2e")
assert storage.storage_path.parent == root.resolve()
assert storage.storage_path.name == "%2e%2e"
@pytest.mark.parametrize(
"context_field,bad_id",
[
# Restore sink: caller-controlled previous_response_id.
("previous_response_id", "../../escape"),
("previous_response_id", "/tmp/escape-abs"),
("previous_response_id", "caresp_x/../../service-data/api-made-dir" + "A" * 14),
# Restore sink: server-issued conversation_id (defense in depth).
("conversation_id", "../../escape"),
# Write sink: malicious response_id (defense in depth).
("response_id", "../../escape"),
],
)
async def test_handle_inner_workflow_rejects_malicious_context_id(
self, tmp_path: Any, context_field: str, bad_id: str
) -> None:
"""End-to-end: ``_handle_inner_workflow`` must reject malicious ids on
both the restore sink (``previous_response_id`` / ``conversation_id``)
and the write sink (``response_id``) without creating any directories.
"""
from unittest.mock import patch
from agent_framework import WorkflowAgent
from azure.ai.agentserver.responses import ResponseContext
from azure.ai.agentserver.responses.models import CreateResponse
# Build a mock that satisfies isinstance(agent, WorkflowAgent) and the
# constructor's "no existing checkpointing" guard.
agent = MagicMock(spec=WorkflowAgent)
agent.id = "wf-agent"
agent.name = "wf"
agent.description = ""
agent.context_providers = []
agent.workflow = MagicMock()
agent.workflow.name = "wf"
agent.workflow._runner_context.has_checkpointing = MagicMock(return_value=False)
# Constructor inspects WorkflowAgent.workflow internals; bypass setup
# by feeding a configured mock through a normal init.
server = ResponsesHostServer(agent, store=InMemoryResponseProvider())
# Re-root checkpoint storage at our isolated tmp_path so we can detect
# any escape attempt on the filesystem.
root = tmp_path / "root"
root.mkdir()
server._checkpoint_storage_path = str(root) # pyright: ignore[reportPrivateUsage]
# Build a ResponseContext with the malicious id targeting the chosen sink.
kwargs: dict[str, Any] = {
"response_id": "resp_" + "a" * 48,
"mode_flags": MagicMock(),
}
if context_field == "previous_response_id":
request = CreateResponse(model="m", input="hi", previous_response_id=bad_id)
kwargs["previous_response_id"] = bad_id
elif context_field == "conversation_id":
request = CreateResponse(model="m", input="hi")
kwargs["conversation_id"] = bad_id
else: # response_id (write sink)
request = CreateResponse(model="m", input="hi")
kwargs["response_id"] = bad_id
# Avoid invoking the real input-resolution machinery, which would need
# a configured provider; we never reach the workflow run on rejection.
with patch.object(ResponseContext, "get_input_items", new=AsyncMock(return_value=[])):
context = ResponseContext(**kwargs)
before = sorted(p.name for p in tmp_path.iterdir())
with pytest.raises(RuntimeError, match="Invalid checkpoint context id"):
async for _ in server._handle_inner_workflow(request, context): # pyright: ignore[reportPrivateUsage]
pass
after = sorted(p.name for p in tmp_path.iterdir())
assert before == after, f"Unexpected filesystem artifacts created for {context_field}={bad_id!r}"
assert list(root.iterdir()) == [], f"Checkpoint dir created inside root for {context_field}={bad_id!r}"
@pytest.mark.parametrize(
"context_field,bad_id",
[
# Restore sink: caller-controlled previous_response_id. These are
# rejected by request validation (HTTP 400) before the checkpoint
# code is reached.
("previous_response_id", "../../escape"),
("previous_response_id", "/tmp/escape-abs"),
("previous_response_id", "caresp_x/../../service-data/api-made-dir" + "A" * 14),
# Restore sink: server-issued conversation id (defense in depth).
# Reaches the checkpoint code and is rejected there, surfacing as
# an HTTP 5xx without creating any filesystem artifacts.
("conversation", "../../escape"),
("conversation", "/tmp/escape-abs"),
],
)
async def test_malicious_context_id_rejected_e2e(self, tmp_path: Any, context_field: str, bad_id: str) -> None:
"""End-to-end (ASGI-in-process): malicious context ids must be rejected
through the full HTTP pipeline, and no checkpoint directory may be
created on disk for either the validation-layer rejection
(``previous_response_id``) or the deeper checkpoint-layer rejection
(``conversation``).
The ``response_id`` write-sink is server-generated and not reachable
via the public HTTP surface, so its defense-in-depth check is covered
by the helper-level test above.
"""
from agent_framework import WorkflowAgent
# Build a mock that satisfies isinstance(agent, WorkflowAgent) and the
# constructor's "no existing checkpointing" guard.
agent = MagicMock(spec=WorkflowAgent)
agent.id = "wf-agent"
agent.name = "wf"
agent.description = ""
agent.context_providers = []
agent.workflow = MagicMock()
agent.workflow.name = "wf"
agent.workflow._runner_context.has_checkpointing = MagicMock( # pyright: ignore[reportPrivateUsage]
return_value=False
)
server = ResponsesHostServer(agent, store=InMemoryResponseProvider())
# Re-root checkpoint storage at our isolated tmp_path so we can detect
# any escape attempt on the filesystem.
root = tmp_path / "root"
root.mkdir()
server._checkpoint_storage_path = str(root) # pyright: ignore[reportPrivateUsage]
payload: dict[str, Any] = {"model": "m", "input": "hi"}
if context_field == "previous_response_id":
payload["previous_response_id"] = bad_id
else: # conversation
payload["conversation"] = bad_id
before = sorted(p.name for p in tmp_path.iterdir())
transport = httpx.ASGITransport(app=server)
async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
resp = await client.post("/responses", json=payload)
after = sorted(p.name for p in tmp_path.iterdir())
# The request must not succeed; either request validation rejects it
# (4xx) or the checkpoint layer raises and the server returns 5xx.
# Either way, no successful response may be produced.
assert resp.status_code >= 400, (
f"Expected non-2xx for {context_field}={bad_id!r}, got {resp.status_code}: {resp.text[:200]}"
)
assert before == after, (
f"Unexpected filesystem artifacts under tmp_path for {context_field}={bad_id!r}: "
f"before={before} after={after}"
)
assert list(root.iterdir()) == [], f"Checkpoint directory created inside root for {context_field}={bad_id!r}"
# endregion
@@ -1429,9 +1429,10 @@ class RawOpenAIChatClient( # type: ignore[misc]
props = content.additional_properties or {}
# Local-shell variant serializes as `local_shell_call` carrying a server-issued id;
# plain function_call_output pairs by call_id and is safe under storage.
if (
props.get(OPENAI_SHELL_OUTPUT_TYPE_KEY) == OPENAI_SHELL_OUTPUT_TYPE_LOCAL_SHELL_CALL
and props.get(OPENAI_LOCAL_SHELL_CALL_ITEM_ID_KEY)
if props.get(
OPENAI_SHELL_OUTPUT_TYPE_KEY
) == OPENAI_SHELL_OUTPUT_TYPE_LOCAL_SHELL_CALL and props.get(
OPENAI_LOCAL_SHELL_CALL_ITEM_ID_KEY
):
continue
new_args: dict[str, Any] = {}
@@ -4120,9 +4120,7 @@ async def test_prepare_options_with_conversation_id_strips_server_items_for_mixe
types = [item.get("type") for item in options["input"]]
assert "reasoning" not in types
assert "function_call" not in types
output_call_ids = {
item["call_id"] for item in options["input"] if item.get("type") == "function_call_output"
}
output_call_ids = {item["call_id"] for item in options["input"] if item.get("type") == "function_call_output"}
assert output_call_ids == {"call_history", "call_live"}
assert options["previous_response_id"] == "resp_prev123"
@@ -27,6 +27,9 @@ OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
# Agent Framework specific settings
# ==================================
# Observability is enabled by default. Set to "false" to opt out.
# ENABLE_INSTRUMENTATION=false
# Enable sensitive data logging (prompts, responses, etc.)
# WARNING: Only enable in dev/test environments
ENABLE_SENSITIVE_DATA=true
@@ -34,9 +37,6 @@ ENABLE_SENSITIVE_DATA=true
# Optional: Enable console exporters for debugging
# ENABLE_CONSOLE_EXPORTERS=true
# Optional: Enable observability (automatically enabled if env vars are set or configure_otel_providers() is called)
# ENABLE_INSTRUMENTATION=true
# OpenAI specific variables
# ==========================
OPENAI_API_KEY="..."
+187 -175
View File
@@ -1,12 +1,12 @@
# Agent Framework Observability
This sample folder shows how a Python application can be configured to send Agent Framework observability data to the Application Performance Management (APM) vendor(s) of your choice based on the OpenTelemetry standard.
These samples show how to send Agent Framework observability data to the Application Performance Management (APM) backend of your choice, based on the OpenTelemetry standard.
In this sample, we provide options to send telemetry to [Application Insights](https://learn.microsoft.com/en-us/azure/azure-monitor/app/app-insights-overview), [Aspire Dashboard](https://learn.microsoft.com/en-us/dotnet/aspire/fundamentals/dashboard/overview?tabs=bash) and the console.
The samples target [Application Insights](https://learn.microsoft.com/en-us/azure/azure-monitor/app/app-insights-overview), the [Aspire Dashboard](https://learn.microsoft.com/en-us/dotnet/aspire/fundamentals/dashboard/overview?tabs=bash), and the console, but any OTLP-compatible backend works.
> **Quick Start**: For local development without Azure setup, you can use the [Aspire Dashboard](https://learn.microsoft.com/en-us/dotnet/aspire/fundamentals/dashboard/standalone) which runs locally via Docker and provides an excellent telemetry viewing experience for OpenTelemetry data. Or you can use the built-in tracing module of the [AI Toolkit for VS Code](https://marketplace.visualstudio.com/items?itemName=ms-windows-ai-studio.windows-ai-studio).
> **Quick Start**: For local development without Azure setup, use the [Aspire Dashboard](https://learn.microsoft.com/en-us/dotnet/aspire/fundamentals/dashboard/standalone) (runs locally via Docker), or the built-in tracing module of the [AI Toolkit for VS Code](https://marketplace.visualstudio.com/items?itemName=ms-windows-ai-studio.windows-ai-studio).
> Note that it is also possible to use other Application Performance Management (APM) vendors. An example is [Prometheus](https://prometheus.io/docs/introduction/overview/). Please refer to this [page](https://opentelemetry.io/docs/languages/python/exporters/) to learn more about exporters.
> Other backends such as [Prometheus](https://prometheus.io/docs/introduction/overview/) are also supported. See the [OpenTelemetry Python exporters](https://opentelemetry.io/docs/languages/python/exporters/) page for the full list.
For more information, please refer to the following resources:
@@ -18,19 +18,15 @@ For more information, please refer to the following resources:
## What to expect
The Agent Framework Python SDK is designed to efficiently generate comprehensive logs, traces, and metrics throughout the flow of agent/model invocation and tool execution. This allows you to effectively monitor your AI application's performance and accurately track token consumption. It does so based on the Semantic Conventions for GenAI defined by OpenTelemetry, and the workflows emit their own spans to provide end-to-end visibility.
The Agent Framework Python SDK is **natively instrumented** to emit logs, traces, and metrics throughout agent/model invocation and tool execution, so you can monitor your AI application's performance and track token consumption. Instrumentation follows the OpenTelemetry [Semantic Conventions for GenAI](https://opentelemetry.io/docs/specs/semconv/gen-ai/), and workflows emit their own spans for end-to-end visibility.
Next to what happens in the code when you run, we also make setting up observability as easy as possible. By calling a single function `configure_otel_providers()` from the `agent_framework.observability` module, you can enable telemetry for traces, logs, and metrics. The function automatically reads standard OpenTelemetry environment variables to configure exporters and providers, making it simple to get started.
### MCP trace propagation
Whenever there is an active OpenTelemetry span context, Agent Framework automatically propagates trace context to MCP servers via the `params._meta` field of `tools/call` requests. It uses the globally-configured OpenTelemetry propagator(s) (W3C Trace Context by default, producing `traceparent` and `tracestate`), so custom propagators (B3, Jaeger, etc.) are also supported. This enables distributed tracing across agent-to-MCP-server boundaries, compliant with the [MCP `_meta` specification](https://modelcontextprotocol.io/specification/2025-11-25/basic#_meta).
**Scope:** automatic `_meta` injection applies only to MCP sessions that the agent process itself opens — `MCPStreamableHTTPTool`, `MCPStdioTool`, and `MCPWebsocketTool` (or any other client-opened `MCPTool` subclass). It does **not** apply to hosted/provider-managed MCP tool configurations such as `FoundryChatClient.get_mcp_tool(...)`, `OpenAIChatClient.get_mcp_tool(...)`, `AnthropicClient.get_mcp_tool(...)`, `GeminiChatClient.get_mcp_tool(...)`, or toolbox-fetched tools (for example, `toolbox = await client.get_toolbox(...)`, then passing `toolbox.tools` into `Agent(tools=...)`), because in those cases the `tools/call` message is issued by the provider service runtime rather than by the agent process. As a result, the framework has no opportunity to inject trace context into those requests, and propagating `traceparent`/`tracestate` across that hosted-service boundary is the responsibility of the service runtime, not Agent Framework. If end-to-end distributed tracing to the downstream MCP server is required, use a client-opened MCP transport instead of a hosted connector.
Setting up observability is also easy: a single call to `configure_otel_providers()` from the `agent_framework.observability` module wires up the trace, log, and metric providers. It reads the standard OpenTelemetry environment variables to configure exporters automatically.
### Five patterns for configuring observability
We've identified multiple ways to configure observability in your application, depending on your needs:
> Setting up observability has two parts: (1) **instrumentation**, the code that generates telemetry, and (2) **exporter/provider configuration**, which decides where that telemetry is sent. Agent Framework is natively instrumented and **enabled by default**, so you only need to handle the second part.
There are five common ways to do that, depending on your needs:
**1. Standard otel environment variables, configured for you**
@@ -42,22 +38,29 @@ from agent_framework.observability import configure_otel_providers
# Reads OTEL_EXPORTER_OTLP_* environment variables automatically
configure_otel_providers()
```
Or if you just want console exporters:
```python
from agent_framework.observability import configure_otel_providers
# Enable console exporters via environment variable
configure_otel_providers(enable_console_exporters=True)
# It is also possible to set ENABLE_CONSOLE_EXPORTERS=true in environment
# variables instead of calling `configure_otel_providers()` with the parameter.
# The framework will automatically read that and set up console exporters.
```
This is the **recommended approach** for getting started.
**2. Custom Exporters**
One level more control over the exporters that are created is to do that yourself, and then pass them to `configure_otel_providers()`. We will still create the providers for you, but you can customize the exporters as needed:
For more control, construct exporters yourself and pass them to `configure_otel_providers()`. The framework still creates the providers for you:
```python
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.exporter.otlp.proto.grpc.exporter import Compression
from agent_framework.observability import configure_otel_providers
# Create custom exporters with specific configuration
@@ -67,17 +70,17 @@ exporters = [
OTLPMetricExporter(endpoint="http://localhost:4317"),
]
# These will be added alongside any exporters from environment variables
configure_otel_providers(exporters=exporters, enable_sensitive_data=True)
# These are added alongside any exporters configured from environment variables
configure_otel_providers(exporters=exporters)
```
**3. Third party setup**
**3. Third-party setup**
A lot of third party specific otel package, have their own easy setup methods, for example Azure Monitor has `configure_azure_monitor()`. You can use those methods to setup the third party first, and then call `enable_instrumentation()` from the `agent_framework.observability` module to activate the Agent Framework telemetry code paths. In all these cases, if you already setup observability via environment variables, you don't need to call `enable_instrumentation()` as it will be enabled automatically.
Many third-party OTel packages ship their own setup helpers (for example, Azure Monitor's `configure_azure_monitor()`). You can use those directly — Agent Framework instrumentation is on by default, so no extra wiring is needed. To also capture sensitive data, call `enable_sensitive_telemetry()` from `agent_framework.observability`.
```python
from azure.monitor.opentelemetry import configure_azure_monitor
from agent_framework.observability import create_resource, enable_instrumentation
from agent_framework.observability import create_resource, enable_sensitive_telemetry
# Configure Azure Monitor first
configure_azure_monitor(
@@ -86,10 +89,10 @@ configure_azure_monitor(
enable_live_metrics=True,
)
# Then activate Agent Framework's telemetry code paths
# This is optional if ENABLE_INSTRUMENTATION and or ENABLE_SENSITIVE_DATA are set in env vars
enable_instrumentation(enable_sensitive_data=False)
# Optional: opt in to capturing sensitive data
enable_sensitive_telemetry()
```
For Microsoft Foundry projects, use `client.configure_azure_monitor()` which retrieves the connection string from the project and configures everything:
```python
@@ -110,7 +113,7 @@ Or with [Langfuse](https://langfuse.com/integrations/frameworks/microsoft-agent-
```python
# environment should be setup correctly, with langfuse urls and keys
from agent_framework.observability import enable_instrumentation
from agent_framework.observability import enable_sensitive_telemetry
from langfuse import get_client
langfuse = get_client()
@@ -121,9 +124,9 @@ if langfuse.auth_check():
else:
print("Authentication failed. Please check your credentials and host.")
# Then activate Agent Framework's telemetry code paths
# This is optional if ENABLE_INSTRUMENTATION and or ENABLE_SENSITIVE_DATA are set in env vars
enable_instrumentation(enable_sensitive_data=False)
# Agent Framework instrumentation is on by default.
# Optional: opt in to capturing sensitive data
enable_sensitive_telemetry()
```
Or with [Comet Opik](https://www.comet.com/docs/opik/integrations/microsoft-agent-framework):
@@ -131,53 +134,152 @@ Or with [Comet Opik](https://www.comet.com/docs/opik/integrations/microsoft-agen
```python
import os
from agent_framework.observability import enable_instrumentation
from agent_framework.observability import enable_sensitive_telemetry
# Use Opik OTLP settings from your project settings
os.environ["OTEL_EXPORTER_OTLP_ENDPOINT"] = "<opik_otlp_endpoint>"
os.environ["OTEL_EXPORTER_OTLP_HEADERS"] = "<opik_otlp_headers>"
# Then activate Agent Framework's telemetry code paths
# This is optional if ENABLE_INSTRUMENTATION and or ENABLE_SENSITIVE_DATA are set in env vars
enable_instrumentation(enable_sensitive_data=False)
# Agent Framework instrumentation is on by default.
# Optional: opt in to capturing sensitive data
enable_sensitive_telemetry()
```
**4. Manual setup**
Of course you can also do a complete manual setup of exporters, providers, and instrumentation. Please refer to sample [advanced_manual_setup_console_output.py](./advanced_manual_setup_console_output.py) for a comprehensive example of how to manually setup exporters and providers for traces, logs, and metrics that will get sent to the console. This gives you full control over which exporters and providers to use. We do have a helper function `create_resource()` in the `agent_framework.observability` module that you can use to create a resource with the appropriate service name and version based on environment variables or standard defaults for Agent Framework, this is not used in the sample.
**5. Auto-instrumentation (zero-code)**
You can also use the [OpenTelemetry CLI tool](https://opentelemetry.io/docs/instrumentation/python/getting-started/#automatic-instrumentation) to automatically instrument your application without changing any code. Please refer to sample [advanced_zero_code.py](./advanced_zero_code.py) for an example of how to use the CLI tool to enable instrumentation for Agent Framework applications.
For full control, set up providers and exporters yourself. See [advanced_manual_setup_console_output.py](./advanced_manual_setup_console_output.py) for a complete example that sends traces, logs, and metrics to the console. The `create_resource()` helper in `agent_framework.observability` can build a resource with the appropriate service name and version from environment variables (or sensible defaults), although the sample does not use it.
**5. Zero-code provider/exporter configuration**
Because Agent Framework is **natively instrumented** with OpenTelemetry, you do not need to auto-instrument the framework itself. You can, however, use the [`opentelemetry-instrument`](https://opentelemetry.io/docs/zero-code/python/) CLI wrapper to configure the global tracer/meter providers and exporters from environment variables (or CLI flags) at process startup. Your application code then does not need to call `configure_otel_providers()` — the native spans and metrics from Agent Framework are picked up by the globally configured pipeline. See [advanced_zero_code.py](./advanced_zero_code.py) for an example.
### MCP trace propagation
Whenever there is an active OpenTelemetry span context, Agent Framework automatically propagates trace context to MCP servers via the `params._meta` field of `tools/call` requests. It uses the globally configured OpenTelemetry propagator(s) — W3C Trace Context by default (producing `traceparent` and `tracestate`) — so custom propagators (B3, Jaeger, etc.) are also supported. This enables distributed tracing across agent-to-MCP-server boundaries, compliant with the [MCP `_meta` specification](https://modelcontextprotocol.io/specification/2025-11-25/basic#_meta).
**Scope:** automatic `_meta` injection applies only to MCP sessions that the agent process itself opens — `MCPStreamableHTTPTool`, `MCPStdioTool`, and `MCPWebsocketTool` (or any other client-opened `MCPTool` subclass). It does **not** apply to hosted or provider-managed MCP tool configurations such as `FoundryChatClient.get_mcp_tool(...)`, `OpenAIChatClient.get_mcp_tool(...)`, `AnthropicClient.get_mcp_tool(...)`, `GeminiChatClient.get_mcp_tool(...)`, or toolbox-fetched tools (e.g. `toolbox = await client.get_toolbox(...)` then `Agent(tools=toolbox.tools)`). In those cases the `tools/call` message is issued by the provider service runtime rather than by the agent process, so propagating `traceparent`/`tracestate` across that boundary is the service runtime's responsibility. If you need end-to-end distributed tracing to the downstream MCP server, use a client-opened MCP transport instead of a hosted connector.
## Configuration
### Dependencies
As part of Agent Framework we use the following OpenTelemetry packages:
- `opentelemetry-api`
- `opentelemetry-sdk`
- `opentelemetry-semantic-conventions-ai`
Agent Framework's core depends on **`opentelemetry-api`** only — the API package is enough for the instrumentation hooks (spans, meters, log records) to emit telemetry, and it has no runtime side effects when no provider is configured.
We do not install exporters by default, so you will need to add those yourself, this prevents us from installing unnecessary dependencies. For Application Insights, you will need to install `azure-monitor-opentelemetry`. For Aspire Dashboard or other OTLP compatible backends, you will need to install `opentelemetry-exporter-otlp-proto-grpc`. For HTTP protocol support, you will also need to install `opentelemetry-exporter-otlp-proto-http`.
If you want the framework to set up providers / exporters for you via `configure_otel_providers()` (or to use the `create_resource()` / `create_metric_views()` helpers), you also need the OpenTelemetry SDK:
And for many others, different packages are used, so refer to the documentation of the specific exporter you want to use.
```bash
pip install opentelemetry-sdk
```
If `opentelemetry-sdk` is missing, those helper functions raise a clear `ImportError` telling you to install it. Day-to-day instrumentation still works without the SDK as long as some other component (e.g. `azure-monitor-opentelemetry`, your application bootstrap, an APM agent) has configured the global OpenTelemetry providers.
Exporters are **not** installed by default — install only what you need:
- **Application Insights**: `azure-monitor-opentelemetry`
- **Aspire Dashboard or other OTLP/gRPC backends**: `opentelemetry-exporter-otlp-proto-grpc`
- **OTLP over HTTP**: `opentelemetry-exporter-otlp-proto-http`
For other backends, refer to the documentation of the specific exporter.
### Environment variables
The following environment variables are used to turn on/off observability of the Agent Framework:
Agent Framework reads the following environment variables:
- `ENABLE_INSTRUMENTATION`
- `ENABLE_SENSITIVE_DATA`
- `ENABLE_CONSOLE_EXPORTERS`
| Variable | Default | Purpose |
|----------|---------|---------|
| `ENABLE_INSTRUMENTATION` | `true` | Set to `false` to disable native instrumentation. See [Disabling instrumentation](#disabling-instrumentation) for the programmatic alternative with sticky semantics. |
| `ENABLE_SENSITIVE_DATA` | `false` | Set to `true` to emit sensitive data (prompts, responses, etc.). |
| `ENABLE_CONSOLE_EXPORTERS` | `false` | Set to `true` to add console exporters. Only used by `configure_otel_providers()`. |
| `VS_CODE_EXTENSION_PORT` | unset | Port used by the [AI Toolkit for VS Code](https://marketplace.visualstudio.com/items?itemName=ms-windows-ai-studio.windows-ai-studio#tracing) tracing integration. Only used by `configure_otel_providers()`. |
All of these are booleans and default to `false`.
You can also call `enable_sensitive_telemetry()` from `agent_framework.observability` to opt in to sensitive-data capture programmatically.
Finally we have `VS_CODE_EXTENSION_PORT` which you can set to a port, which can be used to setup the AI Toolkit for VS Code tracing integration. See [here](https://marketplace.visualstudio.com/items?itemName=ms-windows-ai-studio.windows-ai-studio#tracing) for more details.
> **Note**: Sensitive data includes prompts, responses, and tool arguments. Only enable it in development or test environments — it may expose user or system secrets in production.
The framework will emit observability data when the `ENABLE_INSTRUMENTATION` environment variable is set to `true`. If both are `true` then it will also emit sensitive information. When these are not set, or set to false, you can use the `enable_instrumentation()` function from the `agent_framework.observability` module to turn on instrumentation programmatically. This is useful when you want to control this via code instead of environment variables.
### Disabling instrumentation
> **Note**: Sensitive information includes prompts, responses, and more, and should only be enabled in a development or test environment. It is not recommended to enable this in production environments as it may expose sensitive data.
There are two ways to turn Agent Framework's native instrumentation off, and they have **different scopes**:
The two other variables, `ENABLE_CONSOLE_EXPORTERS` and `VS_CODE_EXTENSION_PORT`, are used to configure where the observability data is sent. Those are only activated when calling `configure_otel_providers()`.
| Approach | Scope | Sticky? | When framework code calls `enable_instrumentation()` later, what happens? |
|----------|-------|---------|---------------------------------------------------------------------------|
| `ENABLE_INSTRUMENTATION=false` in the environment | Initial settings only | No | Instrumentation flips back **on**. |
| `disable_instrumentation()` called from code | Process-wide, sticky | Yes | Instrumentation **stays off** — the user-disable intent wins. |
If you want telemetry off **and want it to stay off**, use `disable_instrumentation()`.
#### Sticky semantics — why this matters
Framework integrations and third-party libraries can call `enable_instrumentation()`, `enable_sensitive_telemetry()`, or `configure_otel_providers()` as part of their own setup. For example, `FoundryChatClient.configure_azure_monitor()` calls `enable_instrumentation()` after wiring up Azure Monitor. That's normally what you want — but if **you** have explicitly opted out, you don't want any of those calls to silently re-enable telemetry.
`disable_instrumentation()` solves this by setting a **sticky** flag on `OBSERVABILITY_SETTINGS` that remains in effect until you explicitly clear it. While the flag is set:
1. `OBSERVABILITY_SETTINGS.enable_instrumentation` and `enable_sensitive_data` **read as `False`** regardless of the stored value.
2. `enable_instrumentation()` and `enable_sensitive_telemetry()` are **no-ops** and log an info-level message.
3. `configure_otel_providers()` still configures providers / exporters / views (so a later force-enable can use them), but does not flip instrumentation on.
4. Direct attribute writes like `OBSERVABILITY_SETTINGS.enable_instrumentation = True` from any code are **silently dropped** (defense in depth).
5. Integrations that consult `OBSERVABILITY_SETTINGS.is_user_disabled` (e.g. `FoundryChatClient.configure_azure_monitor()`, `FoundryAgent.configure_azure_monitor()`) **skip their setup entirely**, so global Azure Monitor providers aren't installed unnecessarily.
```python
from agent_framework.observability import disable_instrumentation
# After this call, Agent Framework expresses your intent to opt out of telemetry.
# Library and framework code is expected to honor that intent and not flip
# instrumentation back on (e.g. by calling `enable_instrumentation()`,
# `enable_sensitive_telemetry()`, or writing to public attributes on
# `OBSERVABILITY_SETTINGS`). The framework actively short-circuits the public
# enable paths so the user's intent stays leading. A determined caller can still
# pass `force=True` or mutate private (`_`-prefixed) attributes to bypass it,
# but those are out-of-contract escape hatches that should not be used by
# integrations on the user's behalf.
disable_instrumentation()
```
#### Forcing re-enablement after a disable
To intentionally re-enable telemetry after `disable_instrumentation()`, pass `force=True` to either of the two public enable helpers. This is the only way to clear the sticky disable, so the user's opt-out can only be reversed by a deliberate user opt-in:
```python
from agent_framework.observability import (
disable_instrumentation,
enable_instrumentation,
enable_sensitive_telemetry,
)
disable_instrumentation()
# Without force=True, these are no-ops while the disable is sticky:
enable_instrumentation() # logs info, does nothing
enable_sensitive_telemetry() # logs info, does nothing
# With force=True, the sticky disable is cleared and the call proceeds:
enable_instrumentation(force=True)
# or
enable_sensitive_telemetry(force=True)
# After a force-enable you can `disable_instrumentation()` again to re-arm
# the sticky disable.
```
#### Checking the disable state from integrations
If you're writing an integration that performs telemetry setup as a side effect (e.g. provisioning a third-party exporter), consult the public read-only `is_user_disabled` property and early-return when it's set:
```python
from agent_framework.observability import OBSERVABILITY_SETTINGS
if OBSERVABILITY_SETTINGS.is_user_disabled:
logger.info(
"Skipping telemetry setup because the user called disable_instrumentation()."
)
return
```
This is what the built-in `FoundryChatClient.configure_azure_monitor()` and `FoundryAgent.configure_azure_monitor()` do — so calling `disable_instrumentation()` reliably prevents Azure Monitor's global providers from being installed by those helpers.
#### What `disable_instrumentation()` does **not** do
- It does not tear down OpenTelemetry providers, exporters, or in-flight spans that were already set up before the disable call. It only gates **future** captures by Agent Framework code paths.
- It does not stop telemetry from third-party instrumentations (e.g. `azure-monitor-opentelemetry`'s system metrics) that are wired up outside Agent Framework. Configure those separately if needed.
- It does not persist across processes. Each Python process starts with the disable flag cleared; if you always want telemetry off in a given environment, set `ENABLE_INSTRUMENTATION=false` as an environment variable in addition to (or instead of) the programmatic call.
#### Environment variables for `configure_otel_providers()`
@@ -202,7 +304,8 @@ The `configure_otel_providers()` function automatically reads **standard OpenTel
> **Note**: These are standard OpenTelemetry environment variables. See the [OpenTelemetry spec](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/) for more details.
#### Logging
Use standard Python logging configuration to align logs with telemetry output.
Use standard Python logging configuration to align logs with telemetry output:
```python
import logging
@@ -212,15 +315,14 @@ logging.basicConfig(
datefmt="%Y-%m-%d %H:%M:%S",
)
```
You can control at what level logging happens and thus what logs get exported, you can do this, by adding this:
To control which logs are exported, adjust the root logger level — other loggers inherit from it by default:
```python
import logging
logger = logging.getLogger()
logger.setLevel(logging.NOTSET)
logging.getLogger().setLevel(logging.NOTSET)
```
This gets the root logger and sets the level of that, automatically other loggers inherit from that one, and you will get detailed logs in your telemetry.
## Samples
@@ -228,36 +330,35 @@ This folder contains different samples demonstrating how to use telemetry in var
| Sample | Description |
|--------|-------------|
| [configure_otel_providers_with_parameters.py](./configure_otel_providers_with_parameters.py) | **Recommended starting point**: Shows how to create custom exporters with specific configuration and pass them to `configure_otel_providers()`. Useful for advanced scenarios. |
| [configure_otel_providers_with_env_var.py](./configure_otel_providers_with_env_var.py) | Shows how to setup telemetry using standard OpenTelemetry environment variables (`OTEL_EXPORTER_OTLP_*`). |
| [agent_observability.py](./agent_observability.py) | Shows telemetry collection for an agentic application with tool calls using environment variables. |
| [foundry_tracing.py](./foundry_tracing.py) | Shows Azure Monitor integration with Foundry for any chat client. |
| [advanced_manual_setup_console_output.py](./advanced_manual_setup_console_output.py) | Advanced: Shows manual setup of exporters and providers with console output. Useful for understanding how observability works under the hood. |
| [advanced_zero_code.py](./advanced_zero_code.py) | Advanced: Shows zero-code telemetry setup using the `opentelemetry-enable_instrumentation` CLI tool. |
| [workflow_observability.py](./workflow_observability.py) | Shows telemetry collection for a workflow with multiple executors and message passing. |
| [configure_otel_providers_with_env_var.py](./configure_otel_providers_with_env_var.py) | **Recommended starting point**: configure telemetry using standard OpenTelemetry environment variables (`OTEL_EXPORTER_OTLP_*`). |
| [configure_otel_providers_with_parameters.py](./configure_otel_providers_with_parameters.py) | Create custom exporters with specific configuration and pass them to `configure_otel_providers()`. |
| [agent_observability.py](./agent_observability.py) | Telemetry collection for an agentic application with tool calls. |
| [foundry_tracing.py](./foundry_tracing.py) | Azure Monitor integration with Microsoft Foundry. |
| [workflow_observability.py](./workflow_observability.py) | Telemetry collection for a workflow with multiple executors and message passing. |
| [advanced_manual_setup_console_output.py](./advanced_manual_setup_console_output.py) | Advanced: manual setup of exporters and providers with console output — useful for understanding how observability works under the hood. |
| [advanced_zero_code.py](./advanced_zero_code.py) | Advanced: zero-code provider/exporter setup using the `opentelemetry-instrument` CLI wrapper. |
### Running the samples
1. Open a terminal and navigate to this folder: `python/samples/02-agents/observability/`. This is necessary for the `.env` file to be read correctly.
2. Create a `.env` file if one doesn't already exist in this folder. Please refer to the [example file](./.env.example).
> **Note**: You can start with just `ENABLE_INSTRUMENTATION=true` and add `OTEL_EXPORTER_OTLP_ENDPOINT` or other configuration as needed. If no exporters are configured, you can set `ENABLE_CONSOLE_EXPORTERS=true` for console output.
3. Choose one environment-loading approach:
- **A. Sample-managed loading (current samples):** run from this folder so the sample's `load_dotenv()` call can find `.env`.
- **B. Shell/IDE-managed environment:** set/export environment variables directly, or use an IDE run configuration that injects env vars / `.env`.
- **C. Explicit env file in code:** pass `env_file_path` to APIs like `configure_otel_providers(env_file_path=".env")` (or your own settings loader path).
- **D. CLI-managed env file:** run with `uv` and pass the file explicitly, for example:
`uv run --env-file=.env python configure_otel_providers_with_env_var.py`
4. Activate your python virtual environment, then run a sample (for example `python configure_otel_providers_with_env_var.py`).
1. Open a terminal in this folder (`python/samples/02-agents/observability/`) so that `.env` is found.
2. Create a `.env` file if you don't already have one. See [.env.example](./.env.example).
> Instrumentation is on by default. Set `OTEL_EXPORTER_OTLP_ENDPOINT` (or other configuration) as needed. With no exporters configured, set `ENABLE_CONSOLE_EXPORTERS=true` for console output.
3. Pick an environment-loading approach:
- **A. Sample-managed:** run from this folder so the sample's `load_dotenv()` call can find `.env`.
- **B. Shell/IDE-managed:** export environment variables, or use an IDE run configuration that injects them.
- **C. Explicit env file in code:** pass `env_file_path` to APIs like `configure_otel_providers(env_file_path=".env")`.
- **D. CLI-managed:** run with `uv` and pass the file explicitly, e.g. `uv run --env-file=.env python configure_otel_providers_with_env_var.py`.
4. Activate your virtual environment, then run a sample (e.g. `python configure_otel_providers_with_env_var.py`).
> If you do manual provider setup (e.g., Azure Monitor), call `enable_instrumentation()` to turn on Agent Framework telemetry code paths; if you want Agent Framework to configure exporters/providers for you, call `configure_otel_providers(...)`.
> If you set up providers manually (e.g. Azure Monitor), Agent Framework instrumentation is still on by default. Call `enable_sensitive_telemetry()` if you also want to capture sensitive data. To have Agent Framework configure exporters and providers for you, call `configure_otel_providers(...)`.
> Each sample will print the Operation/Trace ID, which can be used later for filtering logs and traces in Application Insights or Aspire Dashboard.
> Each sample prints its Operation/Trace ID, which you can use to filter logs and traces in Application Insights or the Aspire Dashboard.
# Appendix
## Azure Monitor Queries
When you are in Azure Monitor and want to have a overall view of the span, use this query in the logs section:
For an overall view of a span in Azure Monitor, run this query in the Logs section:
```kusto
dependencies
@@ -280,7 +381,8 @@ dependencies
```
### Grafana dashboards with Application Insights data
Besides the Application Insights native UI, you can also use Grafana to visualize the telemetry data in Application Insights. There are two tailored dashboards for you to get started quickly:
In addition to the native Application Insights UI, you can use Grafana to visualize the same telemetry data. Two tailored dashboards are available to get you started:
#### Agent Overview dashboard
Open dashboard in Azure portal: <https://aka.ms/amg/dash/af-agent>
@@ -292,117 +394,27 @@ Open dashboard in Azure portal: <https://aka.ms/amg/dash/af-workflow>
## Migration Guide
We've done a major update to the observability API in Agent Framework Python SDK. The new API simplifies configuration by relying more on standard OpenTelemetry environment variables and have split the instrumentation from the configuration.
Instrumentation is now **enabled by default** (you no longer have to opt in by calling `enable_instrumentation()` at startup), and the way you opt in to capturing sensitive payloads has its own dedicated function.
If you're updating from a previous version of the Agent Framework, here are the key changes to the observability API:
### Environment Variables
| Old Variable | New Variable | Notes |
|-------------|--------------|-------|
| `OTLP_ENDPOINT` | `OTEL_EXPORTER_OTLP_ENDPOINT` | Standard OpenTelemetry env var |
| `APPLICATIONINSIGHTS_CONNECTION_STRING` | N/A | Use `configure_azure_monitor()` |
| N/A | `ENABLE_CONSOLE_EXPORTERS` | New opt-in flag for console output |
### OTLP Configuration
**Before (Deprecated):**
```
from agent_framework.observability import setup_observability
# Via parameter
setup_observability(otlp_endpoint="http://localhost:4317")
# Via environment variable
# OTLP_ENDPOINT=http://localhost:4317
setup_observability()
```
**After (Current):**
```python
from agent_framework.observability import configure_otel_providers
# Via standard OTEL environment variable (recommended)
# OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
configure_otel_providers()
# Or via custom exporters
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
configure_otel_providers(exporters=[
OTLPSpanExporter(endpoint="http://localhost:4317"),
OTLPLogExporter(endpoint="http://localhost:4317"),
OTLPMetricExporter(endpoint="http://localhost:4317"),
])
```
### Azure Monitor Configuration
**Before (Deprecated):**
```
from agent_framework.observability import setup_observability
setup_observability(
applicationinsights_connection_string="InstrumentationKey=...",
applicationinsights_live_metrics=True,
)
```
**After (Current):**
If your code previously did:
```python
from agent_framework.foundry import FoundryChatClient
from agent_framework.observability import create_resource, enable_instrumentation
from azure.identity import AzureCliCredential
from azure.monitor.opentelemetry import configure_azure_monitor
from agent_framework.observability import enable_instrumentation
async def main():
# For Microsoft Foundry projects
client = FoundryChatClient(
project_endpoint="https://your-project.services.ai.azure.com",
model="gpt-4o",
credential=AzureCliCredential(),
)
await client.configure_azure_monitor(enable_live_metrics=True)
# For non-Azure AI projects
configure_azure_monitor(
connection_string="InstrumentationKey=...",
resource=create_resource(),
enable_live_metrics=True,
)
enable_instrumentation()
enable_instrumentation(enable_sensitive_data=True)
```
### Console Output
replace it with:
**Before (Deprecated):**
```
from agent_framework.observability import setup_observability
# Console was used as automatic fallback
setup_observability() # Would output to console if no exporters configured
```
**After (Current):**
```python
from agent_framework.observability import configure_otel_providers
from agent_framework.observability import enable_sensitive_telemetry
# Console exporters are now opt-in
# ENABLE_CONSOLE_EXPORTERS=true
configure_otel_providers()
# Or programmatically
configure_otel_providers(enable_console_exporters=True)
enable_sensitive_telemetry()
```
### Benefits of New API
`enable_sensitive_telemetry()` ensures that instrumentation is on and turns sensitive-event capture on in one call. `enable_instrumentation()` still exists for the rare case where you want to programmatically force instrumentation on without enabling sensitive data (e.g. to override `ENABLE_INSTRUMENTATION=false`), and it now also accepts `force=True` to clear a previous `disable_instrumentation()` — see [Disabling instrumentation](#disabling-instrumentation).
1. **Standards Compliant**: Uses standard OpenTelemetry environment variables
2. **Simpler**: Less configuration needed, more relies on environment
3. **Flexible**: Easy to add custom exporters alongside environment-based ones
4. **Cleaner Separation**: Azure Monitor setup is in Azure-specific client
5. **Better Compatibility**: Works with any OTEL-compatible tool (Jaeger, Zipkin, Prometheus, etc.)
> **Note**: Sensitive data includes prompts, responses, and tool arguments. Only enable it in development or test environments — it may expose user or system secrets in production.
## Aspire Dashboard
@@ -437,7 +449,7 @@ OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
Or set it as an environment variable when running your samples:
```bash
ENABLE_INSTRUMENTATION=true OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 python configure_otel_providers_with_env_var.py
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 python configure_otel_providers_with_env_var.py
```
### Viewing telemetry data
@@ -7,7 +7,7 @@ from typing import Annotated
from agent_framework import Message, tool
from agent_framework.foundry import FoundryChatClient
from agent_framework.observability import enable_instrumentation
from agent_framework.observability import enable_sensitive_telemetry
from azure.identity import AzureCliCredential
from dotenv import load_dotenv
from opentelemetry._logs import set_logger_provider
@@ -135,7 +135,8 @@ async def main():
setup_logging()
setup_tracing()
setup_metrics()
enable_instrumentation()
# Instrumentation is enabled by default; call this to also capture sensitive data.
enable_sensitive_telemetry()
await run_chat_client()
@@ -19,13 +19,20 @@ if TYPE_CHECKING:
"""
This sample shows how you can configure observability of an application with zero code changes.
It relies on the OpenTelemetry auto-instrumentation capabilities, and the observability setup
is done via environment variables.
Follow the install guidance from https://opentelemetry.io/docs/zero-code/python/ to install the OpenTelemetry CLI tool,
when using `uv` there are some additional steps, so follow the instructions carefully.
Agent Framework is natively instrumented with OpenTelemetry, so no auto-instrumentation of the
framework itself is required. Running the `opentelemetry-instrument` CLI wrapper simply configures
the global tracer/meter providers and exporters from environment variables (or CLI flags) at
process startup, so the application code does not need to set them up explicitly. The native
spans/metrics emitted by Agent Framework are then picked up by that globally configured pipeline.
And setup a local OpenTelemetry Collector instance to receive the traces and metrics (and update the endpoint below).
See: https://opentelemetry.io/docs/zero-code/python/
Install the OpenTelemetry CLI tool following the guidance above (when using `uv` there are some
additional steps, so follow the instructions carefully).
Then setup a local OpenTelemetry Collector instance to receive the traces and metrics (and update
the endpoint below).
Then you can run:
```bash
@@ -10,7 +10,7 @@ import os
# warnings.filterwarnings("ignore", message=r"\[SKILLS\].*", category=FutureWarning)
from textwrap import dedent
from agent_framework import Agent, ClassSkill, SkillsProvider
from agent_framework import Agent, ClassSkill, SkillFrontmatter, SkillsProvider
from agent_framework.foundry import FoundryChatClient
from azure.identity import AzureCliCredential
from dotenv import load_dotenv
@@ -49,10 +49,12 @@ class UnitConverterSkill(ClassSkill):
def __init__(self) -> None:
super().__init__(
name="unit-converter",
description=(
"Convert between common units using a multiplication factor. "
"Use when asked to convert miles, kilometers, pounds, or kilograms."
frontmatter=SkillFrontmatter(
name="unit-converter",
description=(
"Convert between common units using a multiplication factor. "
"Use when asked to convert miles, kilometers, pounds, or kilograms."
),
),
)
@@ -20,30 +20,43 @@ from typing import Any
from agent_framework import FileSkill, FileSkillScript
def subprocess_script_runner(skill: FileSkill, script: FileSkillScript, args: dict[str, Any] | None = None) -> str:
def subprocess_script_runner(
skill: FileSkill, script: FileSkillScript, args: dict[str, Any] | list[str] | None = None
) -> str:
"""Run a skill script as a local Python subprocess.
Uses ``FileSkillScript.full_path`` as the script path, converts the
``args`` dict to CLI flags, and returns captured output.
``args`` to CLI arguments, and returns captured output.
Args:
skill: The file-based skill that owns the script.
script: The file-based script to run.
args: Optional arguments forwarded as CLI flags.
args: Optional arguments. A ``list[str]`` is forwarded as
positional CLI arguments. Passing a ``dict`` or any other
type raises :class:`TypeError` file-based scripts expect
positional arguments as a JSON array of strings.
Returns:
The combined stdout/stderr output, or an error message.
Raises:
TypeError: If ``args`` is not a ``list[str]`` or ``None``, or if
any list element is not a string.
"""
script_path = Path(script.full_path)
if not script_path.is_file():
return f"Error: Script file not found: {script_path}"
cmd = [sys.executable, str(script_path)]
# Convert args dict to CLI flags
if args:
for key, value in args.items():
if isinstance(value, bool):
if value:
cmd.append(f"--{key}")
elif value is not None:
cmd.append(f"--{key}")
cmd.append(str(value))
if isinstance(args, list):
for item in args:
if not isinstance(item, str):
raise TypeError(
f"File-based skill scripts only accept string CLI arguments "
f"but received a {type(item).__name__}. "
f"All array elements must be strings."
)
cmd.extend(args)
elif args is not None:
raise TypeError(
f"Expected a list of CLI arguments but received {type(args).__name__}. "
f"File-based skill scripts expect positional arguments as a list of strings."
)
try:
result = subprocess.run(
cmd,
@@ -22,9 +22,6 @@ What this example shows:
- executor_completed events (type='executor_completed') contain the messages sent via ctx.send_message() in event.data
- How to generically observe all executor I/O through workflow streaming events
This approach allows you to enable_instrumentation any workflow for observability without
changing the executor implementations.
Prerequisites:
- No external services required.
"""
+1 -1
View File
@@ -103,7 +103,7 @@ def main() -> None:
app = Starlette(
routes=[
*create_agent_card_routes(agent_card),
*create_jsonrpc_routes(request_handler),
*create_jsonrpc_routes(request_handler, "/"),
]
)
@@ -8,16 +8,11 @@ AgentCards for the invoice, policy, and logistics agent types.
from __future__ import annotations
from typing import TYPE_CHECKING
from a2a.types import AgentCapabilities, AgentCard, AgentInterface, AgentSkill
from agent_framework import Agent
from agent_framework.foundry import FoundryChatClient
from invoice_data import query_by_invoice_id, query_by_transaction_id, query_invoices
if TYPE_CHECKING:
from agent_framework import Agent
from agent_framework.foundry import FoundryChatClient
# ---------------------------------------------------------------------------
# Agent instructions
# ---------------------------------------------------------------------------
+20 -55
View File
@@ -10,18 +10,12 @@ published back through the a2a-sdk event queue.
from __future__ import annotations
import asyncio
import uuid
from typing import TYPE_CHECKING
from a2a.helpers import new_task_from_user_message
from a2a.server.agent_execution.agent_executor import AgentExecutor
from a2a.types import (
Message,
Part,
Role,
TaskState,
TaskStatus,
TaskStatusUpdateEvent,
)
from a2a.server.tasks import TaskUpdater
from a2a.types import Part, TaskState
if TYPE_CHECKING:
from a2a.server.agent_execution.context import RequestContext
@@ -47,17 +41,17 @@ class AgentFrameworkExecutor(AgentExecutor):
if not user_text:
user_text = "Hello"
task_id = context.task_id or str(uuid.uuid4())
context_id = context.context_id or str(uuid.uuid4())
# v1.0 requires a Task object in the queue before any TaskStatusUpdateEvent
task = context.current_task
if not task and context.message:
task = new_task_from_user_message(context.message)
await event_queue.enqueue_event(task)
task_id = task.id if task else context.task_id
updater = TaskUpdater(event_queue, task_id, context.context_id)
# Signal that the agent is working
await event_queue.enqueue_event(
TaskStatusUpdateEvent(
task_id=task_id,
context_id=context_id,
status=TaskStatus(state=TaskState.TASK_STATE_WORKING),
)
)
await updater.start_work()
try:
response = await self.agent.run(user_text)
@@ -71,48 +65,19 @@ class AgentFrameworkExecutor(AgentExecutor):
if not response_parts:
response_parts.append(Part(text=str(response)))
# Publish the agent's response as a completed message
await event_queue.enqueue_event(
TaskStatusUpdateEvent(
task_id=task_id,
context_id=context_id,
status=TaskStatus(
state=TaskState.TASK_STATE_COMPLETED,
message=Message(
message_id=str(uuid.uuid4()),
role=Role.ROLE_AGENT,
parts=response_parts,
),
),
)
# Publish the agent's response and mark as completed
await updater.complete(
message=updater.new_agent_message(response_parts),
)
except asyncio.CancelledError:
raise
except Exception as e:
await event_queue.enqueue_event(
TaskStatusUpdateEvent(
task_id=task_id,
context_id=context_id,
status=TaskStatus(
state=TaskState.TASK_STATE_FAILED,
message=Message(
message_id=str(uuid.uuid4()),
role=Role.ROLE_AGENT,
parts=[Part(text=f"Agent error: {e}")],
),
),
)
await updater.update_status(
state=TaskState.TASK_STATE_FAILED,
message=updater.new_agent_message([Part(text=f"Agent error: {e}")]),
)
async def cancel(self, context: RequestContext, event_queue: EventQueue) -> None:
"""Handle cancellation by publishing a canceled status."""
task_id = context.task_id or str(uuid.uuid4())
context_id = context.context_id or str(uuid.uuid4())
await event_queue.enqueue_event(
TaskStatusUpdateEvent(
task_id=task_id,
context_id=context_id,
status=TaskStatus(state=TaskState.TASK_STATE_CANCELED),
)
)
updater = TaskUpdater(event_queue, context.task_id, context.context_id)
await updater.update_status(state=TaskState.TASK_STATE_CANCELED)

Some files were not shown because too many files have changed in this diff Show More