mirror of
https://github.com/microsoft/agent-framework.git
synced 2026-06-16 21:04:09 +08:00
Python: DevUI: Add OpenAI Responses API proxy support + HIL for Workflows (#1737)
* DevUI: Add OpenAI Responses API proxy support with enhanced UI features This commit adds support for proxying requests to OpenAI's Responses API, allowing DevUI to route conversations to OpenAI models when configured to enable testing. Backend changes: - Add OpenAI proxy executor with conversation routing logic - Enhance event mapper to support OpenAI Responses API format - Extend server endpoints to handle OpenAI proxy mode - Update models with OpenAI-specific response types - Remove emojis from logging and CLI output for cleaner text Frontend changes: - Add settings modal with OpenAI proxy configuration UI - Enhance agent and workflow views with improved state management - Add new UI components (separator, switch) for settings - Update debug panel with better event filtering - Improve message renderers for OpenAI content types - Update types and API client for OpenAI integration * update ui, settings modal and workflow input form, add register cleanup hooks. * add workflow HIL support, user mode, other fixes * feat(devui): add human-in-the-loop (HIL) support with dynamic response schemas Implement HIL workflow support allowing workflows to pause for user input with dynamically generated JSON schemas based on response handler type hints. Key Features: - Automatic response schema extraction from @response_handler decorators - Dynamic form generation in UI based on Pydantic/dataclass response types - Checkpoint-based conversation storage for HIL requests/responses - Resume workflow execution after user provides HIL response Backend Changes: - Add extract_response_type_from_executor() to introspect response handlers - Enrich RequestInfoEvent with response_schema via _enrich_request_info_event_with_response_schema() - Map RequestInfoEvent to response.input.requested OpenAI event format - Store HIL responses in conversation history and restore checkpoints Frontend Changes: - Add HILInputModal component with SchemaFormRenderer for dynamic forms - Support Pydantic BaseModel and dataclass response types - Render enum fields as dropdowns, strings as text/textarea, numbers, booleans, arrays, objects - Display original request context alongside response form Testing: - Add tests for checkpoint storage (test_checkpoints.py) - Add schema generation tests for all input types (test_schema_generation.py) - Validate end-to-end HIL flow with spam workflow sample This enables workflows to seamlessly pause execution and request structured user input with type-safe, validated forms generated automatically from response type annotations. * improve HIL support, improve workflow execution view * ui updates * ui updates * improve HIL for workflows, add auth and view modes * update workflow * security improvements , ui fixes * fix mypy error * update loading spinner in ui --------- Co-authored-by: Mark Wallace <127216156+markwallace-microsoft@users.noreply.github.com>
This commit is contained in:
Victor Dibia
committed by
GitHub
Unverified
GitHub
parent
85484c0259
commit
94eae24082
@@ -62,6 +62,27 @@ serve(entities=[agent])
|
||||
|
||||
MCP tools use lazy initialization and connect automatically on first use. DevUI attempts to clean up connections on shutdown
|
||||
|
||||
## Resource Cleanup
|
||||
|
||||
Register cleanup hooks to properly close credentials and resources on shutdown:
|
||||
|
||||
```python
|
||||
from azure.identity.aio import DefaultAzureCredential
|
||||
from agent_framework import ChatAgent
|
||||
from agent_framework.azure import AzureOpenAIChatClient
|
||||
from agent_framework_devui import register_cleanup, serve
|
||||
|
||||
credential = DefaultAzureCredential()
|
||||
client = AzureOpenAIChatClient()
|
||||
agent = ChatAgent(name="MyAgent", chat_client=client)
|
||||
|
||||
# Register cleanup hook - credential will be closed on shutdown
|
||||
register_cleanup(agent, credential.close)
|
||||
serve(entities=[agent])
|
||||
```
|
||||
|
||||
Works with multiple resources and file-based discovery. See tests for more examples.
|
||||
|
||||
## Directory Structure
|
||||
|
||||
For your agents to be discovered by the DevUI, they must be organized in a directory structure like below. Each agent/workflow must have an `__init__.py` that exports the required variable (`agent` or `workflow`).
|
||||
@@ -150,6 +171,22 @@ response2 = client.responses.create(
|
||||
|
||||
**How it works:** DevUI automatically retrieves the conversation's message history from the stored thread and passes it to the agent. You don't need to manually manage message history - just provide the same `conversation` ID for follow-up requests.
|
||||
|
||||
### OpenAI Proxy Mode
|
||||
|
||||
DevUI provides an **OpenAI Proxy** feature for testing OpenAI models directly through the interface without creating custom agents. Enable via Settings → OpenAI Proxy tab.
|
||||
|
||||
**How it works:** The UI sends requests to the DevUI backend (with `X-Proxy-Backend: openai` header), which then proxies them to OpenAI's Responses API (and Conversations API for multi-turn chats). This proxy approach keeps your `OPENAI_API_KEY` secure on the server—never exposed in the browser or client-side code.
|
||||
|
||||
**Example:**
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8080/v1/responses \
|
||||
-H "X-Proxy-Backend: openai" \
|
||||
-d '{"model": "gpt-4.1-mini", "input": "Hello"}'
|
||||
```
|
||||
|
||||
**Note:** Requires `OPENAI_API_KEY` environment variable configured on the backend.
|
||||
|
||||
## CLI Options
|
||||
|
||||
```bash
|
||||
@@ -162,6 +199,21 @@ Options:
|
||||
--config YAML config file
|
||||
--tracing none|framework|workflow|all
|
||||
--reload Enable auto-reload
|
||||
--mode developer|user (default: developer)
|
||||
--auth Enable Bearer token authentication
|
||||
```
|
||||
|
||||
### UI Modes
|
||||
|
||||
- **developer** (default): Full access - debug panel, entity details, hot reload, deployment
|
||||
- **user**: Simplified UI with restricted APIs - only chat and conversation management
|
||||
|
||||
```bash
|
||||
# Development
|
||||
devui ./agents
|
||||
|
||||
# Production (user-facing)
|
||||
devui ./agents --mode user --auth
|
||||
```
|
||||
|
||||
## Key Endpoints
|
||||
@@ -187,18 +239,23 @@ Given that DevUI offers an OpenAI Responses API, it internally maps messages and
|
||||
| `response.function_result.complete` | `FunctionResultContent` | DevUI |
|
||||
| `response.function_approval.requested` | `FunctionApprovalRequestContent` | DevUI |
|
||||
| `response.function_approval.responded` | `FunctionApprovalResponseContent` | DevUI |
|
||||
| `response.output_item.added` (ResponseOutputImage) | `DataContent` (images) | DevUI |
|
||||
| `response.output_item.added` (ResponseOutputFile) | `DataContent` (files) | DevUI |
|
||||
| `response.output_item.added` (ResponseOutputData) | `DataContent` (other) | DevUI |
|
||||
| `response.output_item.added` (ResponseOutputImage/File) | `UriContent` (images/files) | DevUI |
|
||||
| `error` | `ErrorContent` | OpenAI |
|
||||
| Final `Response.usage` field (not streamed) | `UsageContent` | OpenAI |
|
||||
| | **Workflow Events** | |
|
||||
| `response.output_item.added` (ExecutorActionItem)* | `ExecutorInvokedEvent` | OpenAI |
|
||||
| `response.output_item.done` (ExecutorActionItem)* | `ExecutorCompletedEvent` | OpenAI |
|
||||
| `response.output_item.done` (ExecutorActionItem with error)* | `ExecutorFailedEvent` | OpenAI |
|
||||
| `response.output_item.added` (ResponseOutputMessage) | `WorkflowOutputEvent` | OpenAI |
|
||||
| `response.workflow_event.complete` | `WorkflowEvent` (other) | DevUI |
|
||||
| `response.trace.complete` | `WorkflowStatusEvent` | DevUI |
|
||||
| `response.trace.complete` | `WorkflowWarningEvent` | DevUI |
|
||||
| | **Trace Content** | |
|
||||
| `response.trace.complete` | `DataContent` | DevUI |
|
||||
| `response.trace.complete` | `UriContent` | DevUI |
|
||||
| `response.trace.complete` | `DataContent` (no data/errors) | DevUI |
|
||||
| `response.trace.complete` | `UriContent` (unsupported MIME) | DevUI |
|
||||
| `response.trace.complete` | `HostedFileContent` | DevUI |
|
||||
| `response.trace.complete` | `HostedVectorStoreContent` | DevUI |
|
||||
|
||||
@@ -213,15 +270,19 @@ DevUI follows the OpenAI Responses API specification for maximum compatibility:
|
||||
|
||||
**OpenAI Standard Event Types Used:**
|
||||
|
||||
- `ResponseOutputItemAddedEvent` - Output item notifications (function calls and results)
|
||||
- `ResponseOutputItemAddedEvent` - Output item notifications (function calls, images, files, data)
|
||||
- `ResponseOutputItemDoneEvent` - Output item completion notifications
|
||||
- `Response.usage` - Token usage (in final response, not streamed)
|
||||
- All standard text, reasoning, and function call events
|
||||
|
||||
**Custom DevUI Extensions:**
|
||||
|
||||
- `response.output_item.added` with custom item types:
|
||||
- `ResponseOutputImage` - Agent-generated images (inline display)
|
||||
- `ResponseOutputFile` - Agent-generated files (inline display)
|
||||
- `ResponseOutputData` - Agent-generated structured data (inline display)
|
||||
- `response.function_approval.requested` - Function approval requests (for interactive approval workflows)
|
||||
- `response.function_approval.responded` - Function approval responses (user approval/rejection)
|
||||
- `response.function_result.complete` - Server-side function execution results
|
||||
- `response.workflow_event.complete` - Agent Framework workflow events
|
||||
- `response.trace.complete` - Execution traces and internal content (DataContent, UriContent, hosted files/stores)
|
||||
|
||||
@@ -254,18 +315,28 @@ These custom extensions are clearly namespaced and can be safely ignored by stan
|
||||
|
||||
## Security
|
||||
|
||||
DevUI is designed as a **sample application for local development** and should not be exposed to untrusted networks or used in production environments.
|
||||
DevUI is designed as a **sample application for local development** and should not be exposed to untrusted networks without proper authentication.
|
||||
|
||||
**For production deployments:**
|
||||
|
||||
```bash
|
||||
# User mode with authentication (recommended)
|
||||
devui ./agents --mode user --auth --host 0.0.0.0
|
||||
```
|
||||
|
||||
This restricts developer APIs (reload, deployment, entity details) and requires Bearer token authentication.
|
||||
|
||||
**Security features:**
|
||||
|
||||
- User mode restricts developer-facing APIs
|
||||
- Optional Bearer token authentication via `--auth`
|
||||
- Only loads entities from local directories or in-memory registration
|
||||
- No remote code execution capabilities
|
||||
- Binds to localhost (127.0.0.1) by default
|
||||
- All samples must be manually downloaded and reviewed before running
|
||||
|
||||
**Best practices:**
|
||||
|
||||
- Never expose DevUI to the internet
|
||||
- Use `--mode user --auth` for any deployment exposed to end users
|
||||
- Review all agent/workflow code before running
|
||||
- Only load entities from trusted sources
|
||||
- Use `.env` files for sensitive credentials (never commit them)
|
||||
|
||||
Reference in New Issue
Block a user