Python: DevUI fixes : Add multimodal input support for workflows and refactor chat input (#2593)

* show app version in devui .NET: Python: Improved Versioning for DevUI
Fixes #2059

* feat: Add multimodal input support for workflows and refactor chat input

This PR adds support for multimodal content (images, files) in workflow
inputs and refactors the chat input into a reusable component.

## Multimodal Workflow Support
- Add `isChatMessageSchema()` to detect ChatMessage input schemas
- Update `RunWorkflowButton` to use `ChatMessageInput` for ChatMessage workflows
- Wrap multimodal content in OpenAI message format for backend processing
- Add `_is_openai_multimodal_format()` to detect OpenAI ResponseInputParam
- Update `_parse_workflow_input()` to route multimodal input through
  existing `_convert_input_to_chat_message()` converter

## Reusable ChatMessageInput Component
- Extract chat input logic from agent-view into `ChatMessageInput` component
- Support file upload, drag & drop, paste handling, and attachments
- Add `useDragDrop` hook for parent-level drag handling with full-area
  drop zones
- Refactor agent-view to use the new shared component

## Other Improvements
- Add `isStreaming` prop to executor nodes for animation control
- Clean up unused imports and state variables in agent-view
- Add tests for multimodal workflow input handling

Fixes workflow input not receiving images when using AgentExecutor nodes.

* add self loop edge, fix #2470

* fix test
This commit is contained in:
Victor Dibia
2025-12-03 12:15:51 -08:00
committed by GitHub
Unverified
parent 6835161f2d
commit 411ee7a60f
38 changed files with 3488 additions and 1493 deletions
+53 -2
View File
@@ -418,7 +418,7 @@ async def test_executor_action_events(mapper: MessageMapper, test_request: Agent
async def test_magentic_agent_delta_creates_message_container(
mapper: MessageMapper, test_request: AgentFrameworkRequest
) -> None:
"""Test that MagenticAgentDeltaEvent creates message containers (Option A implementation)."""
"""Test that MagenticAgentDeltaEvent creates message containers when no executor context (fallback)."""
# Create mock MagenticAgentDeltaEvent that mimics the real class
from dataclasses import dataclass
@@ -438,7 +438,7 @@ async def test_magentic_agent_delta_creates_message_container(
agent_id: str
text: str | None = None
# First delta should create message container
# First delta should create message container (no executor context = fallback behavior)
first_delta = MagenticAgentDeltaEvent(agent_id="test_agent", text="Hello ")
events = await mapper.convert_event(first_delta, test_request)
@@ -465,6 +465,57 @@ async def test_magentic_agent_delta_creates_message_container(
assert events[0].item_id == message_id
async def test_magentic_agent_delta_routes_to_executor_item(
mapper: MessageMapper, test_request: AgentFrameworkRequest
) -> None:
"""Test that MagenticAgentDeltaEvent routes to executor item when executor context is present."""
from dataclasses import dataclass
try:
from agent_framework import WorkflowEvent
@dataclass
class ExecutorInvokedEvent(WorkflowEvent):
executor_id: str
@dataclass
class MagenticAgentDeltaEvent(WorkflowEvent):
agent_id: str
text: str | None = None
except ImportError:
@dataclass
class ExecutorInvokedEvent:
executor_id: str
@dataclass
class MagenticAgentDeltaEvent:
agent_id: str
text: str | None = None
# First, simulate executor being invoked (sets current_executor_id in context)
executor_event = ExecutorInvokedEvent(executor_id="agent_writer")
executor_events = await mapper.convert_event(executor_event, test_request)
# Should create executor item
assert len(executor_events) == 1
assert executor_events[0].type == "response.output_item.added"
assert executor_events[0].item.type == "executor_action"
executor_item_id = executor_events[0].item.id
# Now send Magentic delta - should route to executor's item, NOT create new message
delta = MagenticAgentDeltaEvent(agent_id="writer", text="Hello world")
delta_events = await mapper.convert_event(delta, test_request)
# Should only emit 1 event: text delta routed to executor's item
assert len(delta_events) == 1
assert delta_events[0].type == "response.output_text.delta"
assert delta_events[0].item_id == executor_item_id # Routed to executor's item!
assert delta_events[0].delta == "Hello world"
if __name__ == "__main__":
# Simple test runner
async def run_all_tests() -> None: