Capture meeting notes

Expanded the pipeline documentation to include details about the Dev Harness and added new features related to compaction strategies and test frameworks.

docs/harness/pipeline.md

@@ -0,0 +1,214 @@
+# MAF Brainstorming Session
+> March, 17th 2026
+
+## Reference Documents
+
+- [OpenClaw Agent Harness](https://microsoft-my.sharepoint.com/:w:/p/shahen/IQAU5F524RvtTpzjAZIjMf0BAQqMMPYCMRRPnX0eslRiPb0?e=MLF6CL)
+- [Agent Platform Comparison](https://m365.cloud.microsoft/chat/pages/eyJ1IjoiaHR0cHM6Ly9taWNyb3NvZnQuc2hhcmVwb2ludC5jb20vY29udGVudHN0b3JhZ2UveDhGTk8teHRza3VDUlgyX2ZNVEhMYmRXU2tHOE93Skt2VTBCX3pNTDFaVT9uYXY9Y3owbE1rWmpiMjUwWlc1MGMzUnZjbUZuWlNVeVJuZzRSazVQTFhoMGMydDFRMUpZTWw5bVRWUklUR0prVjFOclJ6aFBkMHBMZGxVd1FsOTZUVXd4V2xVbVpEMWlKVEl4TUhscVMzWlphRU5CVlVkV2IwUndhVlpzZUZKekxXTkNZMjltVm1jME1VSnlabVZpWWxKeVNXdFZlamd5YlRBelJtVnhXVlJaVmtwQlkzQlVkMnc0TUNabVBUQXhXVlJZVWxWVVNqSk1WbFV6UlZsWE4xTmFSek5JVDBGUFRWSkNXRWhLUzFFbVl6MGxNa1ltWVQxTWIyOXdRWEJ3Sm5BOUpUUXdabXgxYVdSNEpUSkdiRzl2Y0Mxd1lXZGxMV052Ym5SaGFXNWxjaVo0UFNVM1FpVXlNbmNsTWpJbE0wRWxNakpVTUZKVVZVaDRkR0ZYVG5saU0wNTJXbTVSZFdNeWFHaGpiVlozWWpKc2RXUkROV3BpTWpFNFdXbEZkMlZYY0V4a2JHeHZVVEJHVmxJeFduWlNTRUp3Vm0xNE5GVnVUWFJaTUVwcVlqSmFWMXA2VVhoUmJrcHRXbGRLYVZWdVNrcGhNVlkyVDBSS2RFMUVUa2RhV0VaYVZrWnNWMU5yUm1walJsSXpZa1JuZDJaRVFYaFhWbEpaVld4V1ZWUldUa0pPTUdSWlZGVk9SbFpVVWs5U2EydDVVMVJqZVZGc2FGWldSa3BDVFRGSkpUTkVKVEl5SlRKREpUSXlhU1V5TWlVelFTVXlNbU5oTkRSbFlUZGtMVEF5TWpjdE5ERm1OQzFpWVRsaUxUUm1OV1kyTlRWaVpEZGpPU1V5TWlVM1JBPT0ifQ?auth=2&ct=1773767195553&or=Teams-HL&LOF=1)
+- [LangChain Deep Agents](https://docs.langchain.com/oss/python/deepagents/overview)
+- [CodeAct](https://arxiv.org/abs/2402.01030)
+- [AI Accelerator - Foundry](https://microsoft.sharepoint.com/:p:/t/CoreAIStudioOutboundProduct/IQAFfbFIu5E7RLMcFmX_7C6EAU2xt1TxnGbR36EN6JTwX0Y?e=9UsKZ6)
+- [Foundry Developer Portal (Hosted Agents)](https://hosted-agents-builder.lemonriver-6a2ef1ee.westus2.azurecontainerapps.io/getting-started)
+
+
+## Next Steps
+
+Demo for MVP session on campus next week.
+
+1. Code Act: How to with MAF
+1. Harness Preview: Single agent with compaction, tools (including shell or file-system), and simple orchestration loop.
+
+Deploy agent with harness as _Foundry Hosted Agent_.
+
+
+## What Is the Agent Harness?
+
+The runtime control plane that enables reliable, long-running agent execution. Not a specific agent. Not a specific set of tools. It's the **infrastructure layer** that any agent can run within.
+
+What it provides:
+
+- **Outer loop** — model ↔ tools ↔ state ↔ repair, running as long as it needs to
+- **State & durability** — checkpoints, resume, memory
+- **Tool plumbing** — registry, schema, invocation, error handling
+- **Governance** — permissions, human-in-the-loop, policies
+- **Context management** — compaction, eviction, externalization
+- **Observability** — traces, transcripts, replay
+
+
+### Where We Stand
+
+MAF compared against DeepAgents, Amplifier, Opencode, Copilot CLI, OpenAI Codex, and Claude Code — every one of them has these capabilities. MAF has some partially, most not at all.
+
+| Priority | Capability | Layer | MAF Today |
+|---|:---|:---|:---|
+| **P0** | Agent/User Orchestration | Harness | 🟡 Workflows or part of `AIAgent`? |
+| **P0** | Session Persistence | Harness | 🟢 `AgentSession`/`AIContextProvider` |
+| **P0** | Context Compaction | Harness | 🟢 In preview |
+| **P0** | Memory | Environment | 🟡 Partial — Python core only |
+| **P0** | Permissions / Scoping | Harness | 🔴 Nothing |
+| **P0** | Tool: Composite Tool Calling | Harness | 🔴 Needs definition |
+| **P0** | Tool: Filesystem | Environment | 🟢 Local access |
+| **P0** | Tool: Shell Execution | Environment | 🟢 Local access |
+| **P0** | Tool: Todo / Planning | Harness | 🔴 Nothing |
+| **P0** | Sub-Agent Delegation | Harness | 🟡 Partial — orchestration exists, state isolation incomplete |
+| **P1** | Skills / Prompt Presets | Persona | 🔴 Nothing |
+| **P1** | Model Routing | Harness | 🔴 Nothing |
+| **P1** | Agent Budgets | Persona | 🔴 Nothing |
+| **P1** | Prompt Caching | Persona | 🔴 Nothing |
+
+### Open Issues
+
+- How does this look in DevUI?
+
+
+
+## Features
+
+### Agent/User Orchestration
+
+How does the harness manage structured, multi-turn data collection from the user? 
+What's the interaction model between the outer loop and user-facing slot-filling prompts? 
+How does it compose with compaction and task management? How does the agent re-ask or repair slot values after partial completion?
+
+### Session Persistence
+
+`AgentSession` is directly serializable and there is also a `ChatHistoryProvider` option for more complex storage needs. 
+
+Open questions: 
+- How to handle schema evolution? 
+- How to version `AgentSession`? 
+- How to support partial loading for long histories?
+- Does a session include more than conversation context (i.e. messages)?
+
+### Compaction Strategy
+
+Two API tiers — **simple for most developers, advanced for full control**.
+
+**Simple (menu-driven)** — developer picks from preset enums:
+```csharp
+builder.AddCompaction(Approach.Balanced, Size.Compact, summarizingChatClient);
+```
+
+**Advanced (pipeline)** — ordered stages, least to most aggressive:
+```csharp
+builder.AddCompaction(
+    new PipelineCompactionStrategy(
+        // 1. Gentle: collapse old tool-call groups into short summaries
+        new ToolResultCompactionStrategy(CompactionTriggers.MessagesExceed(7)),
+
+        // 2. Moderate: use an LLM to summarize older conversation spans into a concise message
+        new SummarizationCompactionStrategy(summarizerChatClient, CompactionTriggers.TokensExceed(0x6000)),
+
+        // 3. Aggressive: keep only the last N user turns and their responses
+        new SlidingWindowCompactionStrategy(CompactionTriggers.TurnsExceed(32)),
+
+        // 4. Emergency: drop oldest groups until under the token budget
+        new TruncationCompactionStrategy(CompactionTriggers.TokensExceed(0x8000))));
+```
+
+1. **Gentle:** Collapse old tool-call groups into short summaries (`ToolResultCompactionStrategy`)
+2. **Moderate:** LLM-based summarization of older conversation spans (`SummarizationCompactionStrategy`)
+3. **Aggressive:** Sliding window — keep only last N user turns (`SlidingWindowCompactionStrategy`)
+
+### Memories
+
+Loads from backend storage, injects into system prompt automatically.
+
+### Permissions / Scoping
+
+Agent Scopes define the explicit boundaries within which an agent is allowed to operate, constraining where it can act 
+(for example, a single folder or service) and what level of access it has (such as read‑only, write, or execute). 
+By enforcing scoped resources and permissions, this feature ensures the agent’s actions remain intentionally limited, 
+predictable, and aligned with least‑privilege principles—preventing overreach even when the agent could otherwise reason 
+about broader options.
+
+### Tools: File System / Shell
+
+Both should follow the same pattern: **interface-based, with pluggable backends**. Local is the default. Remote/sandboxed is supported.
+
+- **File System** — `FilesystemTool` with `read`, `write`, `edit`, `list`, `glob`, `grep`. Backed by a `FilesystemProtocol` with `LocalFilesystem` (direct access) and `HostedFilesystem` (remote sandbox). Must include path validation, traversal prevention, large-file pagination. Consider snapshot/restore for tracking changes during execution.
+
+- **Shell** — `LocalShellTool` / `HostedShellTool`. Configurable timeout, output truncation, working directory, environment variables. Open question: what sandboxing and permission model?
+
+- **Computer Use (TODO)** — same interface pattern for screen/mouse/keyboard interaction. Future work. Security and governance implications are significant.
+
+### Tools: Task Management
+
+`TodoTool` with `write_todos`. `TodoItem` has content + status (pending / in progress / completed). `TodoMiddleware` injects current todos into the system prompt — this is what gives the agent the ability to self-plan and track its own progress.
+
+Open debate: P0 or P1? "An agent _can_ work with only filesystem + shell. But for Claude Code-like complex multi-step tasks, this is P0."
+
+### Data Driven: Input Schema / Structured Data Output
+
+How does the harness support defining what data the agent needs (input schema) and what the agent produces (structured output)? Is this related to or distinct from slot filling? How does the developer define and validate schemas?
+### Sub-Agent Delegation
+
+MAF already has orchestration (Sequential, Concurrent, Group Chat, Magentic, Handoff, Human-in-the-loop). The gap is **state isolation**: sub-agents need their own message history and todo state, isolated from the parent, returning results as tool responses. "A lot of task planners and coding agents use sub-agents. This feels P0."
+
+### Skills / Prompt Presets (P1)
+
+`SkillsMiddleware` loads reusable instruction sets from `SKILL.md` files with YAML frontmatter (Anthropic Skills format). Progressive disclosure — metadata first, content on demand. Skill discovery from filesystem paths.
+
+### Model Routing (P1)
+
+`ModelRouterMiddleware` with strategies: cost-aware (minimize cost for task requirements) and heuristic (rule-based, e.g., stronger model for code tasks).
+
+### Agent Budgets
+
+Agent Budgets define a hard execution limit—such as a maximum number of tokens, turns, or tool calls—within which an agent must 
+decide whether to act and how far to pursue a task, directly shaping planning, delegation, and early stopping behavior. 
+Unlike a context‑management budget, which governs what information is retained or loaded, an agent budget constrains execution itself, 
+informing decisions like skipping steps, reducing depth, or terminating when the remaining budget cannot justify further action.
+
+### Prompt Caching (P1)
+
+Prompt caching is important for coding agents.  
+When an agent is iterating on a coding problem, the same or similar prompts are often repeated.
+We need a great caching story to speed up iteration and reduce costs.
+
+
+## Shape
+
+Everything hangs off a **fluent builder pattern**.  
+Ideally this builder is identical with the agent-builder pattern, so developers 
+can seamlessly transition from "building an agent" to "building a harness for that agent" without learning a new API. 
+
+Example:
+
+```
+builder
+    .AddCompaction(...)
+    .AddTool(filesystemTool)
+    .AddTool(shellTool)
+    .AddMemory(...)
+    .AddTodo(...)
+```
+
+- **Composability** — developers opt in/out of individual capabilities. Minimal harness = just the outer loop. Full harness = everything.
+- **Hosting** — must integrate cleanly with DI and hosting (ASP.NET, Azure Functions).
+- **Presets** — opinionated starters? (`HarnessPresets.CodingAgent`, `HarnessPresets.Conversational`, `HarnessPresets.Research`)
+- **Two-tier deployment** — "develop local, deploy remote":
+  - _Local:_ Direct filesystem/shell, fast iteration, debugging, human-in-the-loop
+  - _Production (Foundry Hosted Agents):_ Managed containers, autoscaling, identity, observability, Teams / M365 Copilot / Web
+- The interface-based tool abstractions (`LocalFilesystem` ↔ `HostedFilesystem`, `LocalShell` ↔ `HostedShell`) are what make the two-tier model work.
+
+
+## Validation
+
+- **Prompt evaluation** — test default harness prompts (compaction, slot filling, etc.) across OpenAI, Azure OpenAI, Anthropic, and other providers
+- **Custom prompt override** — developers can replace default prompts; need to document and validate the override mechanism
+- **Compaction testing** — verify different pipeline configurations produce correct and useful results
+- **Test strategy** — unit tests, integration tests, model-in-the-loop evaluation, benchmarks
+
+
+## Tutorials
+
+Suggested progression:
+
+1. Getting started — minimal harness setup
+2. Adding compaction to a long-running conversation
+3. Using tools (filesystem, shell) within the harness
+4. Slot filling / guided conversations
+5. Task management and structured output
+6. Advanced — custom compaction pipelines, memory, sub-agents
+

dotnet/tests/Microsoft.Agents.AI.Workflows.UnitTests/MessageMergerTests.cs

@@ -17,7 +17,7 @@ public class MessageMergerTests
     [Fact]
     public void Test_MessageMerger_AssemblesMessage()
     {
-        DateTimeOffset creationTime = DateTimeOffset.UtcNow;
+        DateTimeOffset creationTime = DateTimeOffset.UtcNow.Subtract(TimeSpan.FromSeconds(1));
         string responseId = Guid.NewGuid().ToString("N");
         string messageId = Guid.NewGuid().ToString("N");
 

python/packages/ag-ui/agent_framework_ag_ui/_message_adapters.py

@@ -242,8 +242,16 @@ def _deduplicate_messages(messages: list[Message]) -> list[Message]:
             unique_messages.append(msg)
 
         else:
-            content_str = str([str(c) for c in msg.contents]) if msg.contents else ""
-            key = (role_value, hash(content_str))
+            # Use message_id for deduplication when available — two messages with the
+            # same id are definitively the same message (e.g. upstream replays), while
+            # different messages that happen to share identical content (e.g. repeated
+            # "yes" confirmations) will have distinct ids and be preserved.
+            # Fall back to content-hash when message_id is absent or empty.
+            if msg.message_id:
+                key = ("id", msg.message_id)
+            else:
+                content_str = str([str(c) for c in msg.contents]) if msg.contents else ""
+                key = ("content", role_value, hash(content_str))
 
             if key in seen_keys:
                 logger.info(f"Skipping duplicate message at index {idx}: role={role_value}")

python/packages/ag-ui/tests/ag_ui/test_message_adapters.py

@@ -1015,15 +1015,111 @@ def test_deduplicate_assistant_tool_calls():
     assert len(result) == 1
 
 
-def test_deduplicate_general_messages():
-    """Duplicate general user messages are deduplicated."""
+def test_deduplicate_by_message_id():
+    """Messages with the same message_id are deduplicated."""
+    from agent_framework_ag_ui._message_adapters import _deduplicate_messages
+
+    msg1 = Message(role="user", contents=[Content.from_text(text="Hello")])
+    msg1.message_id = "msg-1"
+    msg2 = Message(role="user", contents=[Content.from_text(text="Hello")])
+    msg2.message_id = "msg-1"
+
+    result = _deduplicate_messages([msg1, msg2])
+    assert len(result) == 1
+    assert result == [msg1]
+
+
+def test_deduplicate_preserves_repeated_confirmations_with_distinct_ids():
+    """Identical content with different message_ids is preserved."""
+    from agent_framework_ag_ui._message_adapters import _deduplicate_messages
+
+    assistant = Message(role="assistant", contents=[Content.from_text(text="Are you sure?")])
+    assistant.message_id = "msg-1"
+    confirm1 = Message(role="user", contents=[Content.from_text(text="yes")])
+    confirm1.message_id = "msg-2"
+    confirm2 = Message(role="user", contents=[Content.from_text(text="yes")])
+    confirm2.message_id = "msg-3"
+
+    result = _deduplicate_messages([confirm1, assistant, confirm2])
+    assert result == [confirm1, assistant, confirm2]
+
+
+def test_deduplicate_preserves_repeated_system_messages_with_distinct_ids():
+    """Non-consecutive identical system messages with different ids are preserved."""
+    from agent_framework_ag_ui._message_adapters import _deduplicate_messages
+
+    sys1 = Message(role="system", contents=[Content.from_text(text="You are a helpful assistant.")])
+    sys1.message_id = "msg-1"
+    user_msg = Message(role="user", contents=[Content.from_text(text="Hi")])
+    user_msg.message_id = "msg-2"
+    sys2 = Message(role="system", contents=[Content.from_text(text="You are a helpful assistant.")])
+    sys2.message_id = "msg-3"
+
+    result = _deduplicate_messages([sys1, user_msg, sys2])
+    assert result == [sys1, user_msg, sys2]
+
+
+def test_deduplicate_skips_replayed_system_messages_with_same_id():
+    """System messages replayed with the same message_id are deduplicated."""
+    from agent_framework_ag_ui._message_adapters import _deduplicate_messages
+
+    msgs = []
+    for _ in range(3):
+        m = Message(role="system", contents=[Content.from_text(text="You are a helpful assistant.")])
+        m.message_id = "msg-1"
+        msgs.append(m)
+
+    result = _deduplicate_messages(msgs)
+    assert len(result) == 1
+
+
+def test_deduplicate_without_message_id_uses_content_hash():
+    """Messages without message_id are deduplicated by content hash."""
     from agent_framework_ag_ui._message_adapters import _deduplicate_messages
 
     msg1 = Message(role="user", contents=[Content.from_text(text="Hello")])
     msg2 = Message(role="user", contents=[Content.from_text(text="Hello")])
 
     result = _deduplicate_messages([msg1, msg2])
-    assert len(result) == 1
+    assert result == [msg1]
+
+
+def test_deduplicate_without_message_id_preserves_different_content():
+    """Messages without message_id but different content are preserved."""
+    from agent_framework_ag_ui._message_adapters import _deduplicate_messages
+
+    msg1 = Message(role="user", contents=[Content.from_text(text="Hello")])
+    msg2 = Message(role="user", contents=[Content.from_text(text="World")])
+
+    result = _deduplicate_messages([msg1, msg2])
+    assert result == [msg1, msg2]
+
+
+def test_deduplicate_handles_none_contents():
+    """Messages with contents=None pass through without errors; duplicates are deduped."""
+    from agent_framework_ag_ui._message_adapters import _deduplicate_messages
+
+    msg1 = Message(role="user", contents=None)
+    msg2 = Message(role="assistant", contents=[Content.from_text(text="Hello")])
+    msg3 = Message(role="user", contents=None)
+
+    result = _deduplicate_messages([msg1, msg2, msg3])
+    assert result == [msg1, msg2]
+
+
+def test_deduplicate_mixed_id_and_no_id():
+    """Messages with and without message_id coexist correctly."""
+    from agent_framework_ag_ui._message_adapters import _deduplicate_messages
+
+    msg1 = Message(role="user", contents=[Content.from_text(text="Hello")])
+    msg1.message_id = "msg-1"
+    msg2 = Message(role="user", contents=[Content.from_text(text="Hello")])  # no id
+    msg3 = Message(role="user", contents=[Content.from_text(text="Hello")])
+    msg3.message_id = "msg-1"  # duplicate of msg1
+
+    result = _deduplicate_messages([msg1, msg2, msg3])
+    assert len(result) == 2
+    assert result == [msg1, msg2]
 
 
 def test_deduplicate_replaces_empty_tool_result():
@@ -1038,7 +1134,30 @@ def test_deduplicate_replaces_empty_tool_result():
     assert result[0].contents[0].result == "actual result"
 
 
-# ── Multimodal & content conversion edge cases ──
+def test_deduplicate_empty_string_message_id_falls_back_to_content_hash():
+    """Empty-string message_id is treated as missing; content-hash dedup is used."""
+    from agent_framework_ag_ui._message_adapters import _deduplicate_messages
+
+    msg1 = Message(role="user", contents=[Content.from_text(text="Hello")])
+    msg1.message_id = ""
+    msg2 = Message(role="user", contents=[Content.from_text(text="World")])
+    msg2.message_id = ""
+
+    result = _deduplicate_messages([msg1, msg2])
+    assert result == [msg1, msg2], "Different content with empty IDs should both be preserved"
+
+
+def test_deduplicate_empty_string_message_id_deduplicates_same_content():
+    """Empty-string message_id with identical content should be deduplicated."""
+    from agent_framework_ag_ui._message_adapters import _deduplicate_messages
+
+    msg1 = Message(role="user", contents=[Content.from_text(text="Hello")])
+    msg1.message_id = ""
+    msg2 = Message(role="user", contents=[Content.from_text(text="Hello")])
+    msg2.message_id = ""
+
+    result = _deduplicate_messages([msg1, msg2])
+    assert result == [msg1], "Same content with empty IDs should be deduplicated"
 
 
 def test_convert_agui_content_unknown_source_type_fallback():