Files
agent-framework/dotnet/src/Microsoft.Agents.AI.Hosting.OpenAI/Responses/Models/StreamingResponseEvent.cs
T
7a45929807 Python: .Net: Dotnet devui compatibility fixes (#2026)
* DevUI: Add OpenAI Responses API proxy support with enhanced UI features

This commit adds support for proxying requests to OpenAI's Responses API,
allowing DevUI to route conversations to OpenAI models when configured to enable testing.

Backend changes:
- Add OpenAI proxy executor with conversation routing logic
- Enhance event mapper to support OpenAI Responses API format
- Extend server endpoints to handle OpenAI proxy mode
- Update models with OpenAI-specific response types
- Remove emojis from logging and CLI output for cleaner text

Frontend changes:
- Add settings modal with OpenAI proxy configuration UI
- Enhance agent and workflow views with improved state management
- Add new UI components (separator, switch) for settings
- Update debug panel with better event filtering
- Improve message renderers for OpenAI content types
- Update types and API client for OpenAI integration

* update ui, settings modal and workflow input form, add register cleanup hooks.

* add workflow HIL support, user mode, other fixes

* feat(devui): add human-in-the-loop (HIL) support with dynamic response schemas

Implement  HIL workflow support allowing workflows to pause for user input
with dynamically generated JSON schemas based on response handler type hints.

Key Features:
- Automatic response schema extraction from @response_handler decorators
- Dynamic form generation in UI based on Pydantic/dataclass response types
- Checkpoint-based conversation storage for HIL requests/responses
- Resume workflow execution after user provides HIL response

Backend Changes:
- Add extract_response_type_from_executor() to introspect response handlers
- Enrich RequestInfoEvent with response_schema via _enrich_request_info_event_with_response_schema()
- Map RequestInfoEvent to response.input.requested OpenAI event format
- Store HIL responses in conversation history and restore checkpoints

Frontend Changes:
- Add HILInputModal component with SchemaFormRenderer for dynamic forms
- Support Pydantic BaseModel and dataclass response types
- Render enum fields as dropdowns, strings as text/textarea, numbers, booleans, arrays, objects
- Display original request context alongside response form

Testing:
- Add  tests for checkpoint storage (test_checkpoints.py)
- Add schema generation tests for all input types (test_schema_generation.py)
- Validate end-to-end HIL flow with spam workflow sample

This enables workflows to seamlessly pause execution and request structured user input
with type-safe, validated forms generated automatically from response type annotations.

* improve HIL support, improve workflow execution view

* ui updates

* ui updates

* improve HIL for workflows, add auth and view modes

* update workflow

* security improvements , ui fixes

* fix mypy error

* update loading spinner in ui

* DevUI: Serialize workflow input as string to maintain conformance with OpenAI Responses format

* Phase 1: Add /meta endpoint and fix workflow event naming for .NET DevUI compatibility

* additional fixes for .NET DevUI workflow visualization item ID tracking

**Problem:**
.NET DevUI was generating different item IDs for ExecutorInvokedEvent and
ExecutorCompletedEvent, causing only the first executor to highlight in the
workflow graph. Long executor names and error messages also broke UI layout.

**Changes:**
- Add ExecutorActionItemResource to match Python DevUI implementation
- Track item IDs per executor using dictionary in AgentRunResponseUpdateExtensions
- Reuse same item ID across invoked/completed/failed events for proper pairing
- Add truncateText() utility to workflow-utils.ts
- Truncate executor names to 35 chars in execution timeline
- Truncate error messages to 150 chars in workflow graph nodes

** Details:**
- ExecutorActionItemResource registered with JSON source generation context
- Dictionary cleaned up after executor completion/failure to prevent memory leaks
- Frontend item tracking by unique item.id supports multiple executor runs
- All changes follow existing codebase patterns and conventions

Tested with review-workflow showing correct executor highlighting and state
transitions for sequential and concurrent executors.

* format fixes, remove cors tests

* remove unecessary attributes

---------

Co-authored-by: Mark Wallace <127216156+markwallace-microsoft@users.noreply.github.com>
Co-authored-by: Reuben Bond <reuben.bond@gmail.com>
2025-11-10 18:56:31 +00:00

702 lines
23 KiB
C#

// Copyright (c) Microsoft. All rights reserved.
using System.Collections.Generic;
using System.Text.Json;
using System.Text.Json.Serialization;
namespace Microsoft.Agents.AI.Hosting.OpenAI.Responses.Models;
/// <summary>
/// Abstract base class for all streaming response events in the OpenAI Responses API.
/// Provides common properties shared across all streaming event types.
/// </summary>
[JsonPolymorphic(TypeDiscriminatorPropertyName = "type", UnknownDerivedTypeHandling = JsonUnknownDerivedTypeHandling.FailSerialization)]
[JsonDerivedType(typeof(StreamingResponseCreated), StreamingResponseCreated.EventType)]
[JsonDerivedType(typeof(StreamingResponseInProgress), StreamingResponseInProgress.EventType)]
[JsonDerivedType(typeof(StreamingResponseCompleted), StreamingResponseCompleted.EventType)]
[JsonDerivedType(typeof(StreamingResponseIncomplete), StreamingResponseIncomplete.EventType)]
[JsonDerivedType(typeof(StreamingResponseFailed), StreamingResponseFailed.EventType)]
[JsonDerivedType(typeof(StreamingResponseCancelled), StreamingResponseCancelled.EventType)]
[JsonDerivedType(typeof(StreamingOutputItemAdded), StreamingOutputItemAdded.EventType)]
[JsonDerivedType(typeof(StreamingOutputItemDone), StreamingOutputItemDone.EventType)]
[JsonDerivedType(typeof(StreamingContentPartAdded), StreamingContentPartAdded.EventType)]
[JsonDerivedType(typeof(StreamingContentPartDone), StreamingContentPartDone.EventType)]
[JsonDerivedType(typeof(StreamingOutputTextDelta), StreamingOutputTextDelta.EventType)]
[JsonDerivedType(typeof(StreamingOutputTextDone), StreamingOutputTextDone.EventType)]
[JsonDerivedType(typeof(StreamingFunctionCallArgumentsDelta), StreamingFunctionCallArgumentsDelta.EventType)]
[JsonDerivedType(typeof(StreamingFunctionCallArgumentsDone), StreamingFunctionCallArgumentsDone.EventType)]
[JsonDerivedType(typeof(StreamingReasoningSummaryTextDelta), StreamingReasoningSummaryTextDelta.EventType)]
[JsonDerivedType(typeof(StreamingReasoningSummaryTextDone), StreamingReasoningSummaryTextDone.EventType)]
[JsonDerivedType(typeof(StreamingWorkflowEventComplete), StreamingWorkflowEventComplete.EventType)]
[JsonDerivedType(typeof(StreamingFunctionApprovalRequested), StreamingFunctionApprovalRequested.EventType)]
[JsonDerivedType(typeof(StreamingFunctionApprovalResponded), StreamingFunctionApprovalResponded.EventType)]
internal abstract class StreamingResponseEvent
{
/// <summary>
/// Gets the type identifier for the streaming response event.
/// This property is used to discriminate between different event types during serialization.
/// </summary>
[JsonIgnore]
public abstract string Type { get; }
/// <summary>
/// Gets the sequence number of this event in the streaming response.
/// Events are numbered sequentially starting from 1 to maintain ordering.
/// </summary>
[JsonPropertyName("sequence_number")]
public int SequenceNumber { get; init; }
}
/// <summary>
/// Denotes an <see cref="StreamingResponseEvent"/> instance which contains an update to the <see cref="Models.Response"/> instance.
/// </summary>
internal interface IStreamingResponseEventWithResponse
{
/// <summary>
/// Gets the response object associated with this streaming event.
/// </summary>
Response Response { get; }
}
/// <summary>
/// Represents a streaming response event indicating that a new response has been created and streaming has begun.
/// This is typically the first event sent in a streaming response sequence.
/// </summary>
internal sealed class StreamingResponseCreated : StreamingResponseEvent, IStreamingResponseEventWithResponse
{
/// <summary>
/// The constant event type identifier for response created events.
/// </summary>
public const string EventType = "response.created";
/// <inheritdoc/>
[JsonIgnore]
public override string Type => EventType;
/// <summary>
/// Gets or sets the response object that was created.
/// This contains metadata about the response including ID, creation timestamp, and other properties.
/// </summary>
[JsonPropertyName("response")]
public required Response Response { get; init; }
}
/// <summary>
/// Represents a streaming response event indicating that the response is in progress.
/// </summary>
internal sealed class StreamingResponseInProgress : StreamingResponseEvent, IStreamingResponseEventWithResponse
{
/// <summary>
/// The constant event type identifier for response in progress events.
/// </summary>
public const string EventType = "response.in_progress";
/// <inheritdoc/>
[JsonIgnore]
public override string Type => EventType;
/// <summary>
/// Gets or sets the response object that is in progress.
/// </summary>
[JsonPropertyName("response")]
public required Response Response { get; init; }
}
/// <summary>
/// Represents a streaming response event indicating that the response has been completed.
/// This is typically the last event sent in a streaming response sequence.
/// </summary>
internal sealed class StreamingResponseCompleted : StreamingResponseEvent, IStreamingResponseEventWithResponse
{
/// <summary>
/// The constant event type identifier for response completed events.
/// </summary>
public const string EventType = "response.completed";
/// <inheritdoc/>
[JsonIgnore]
public override string Type => EventType;
/// <summary>
/// Gets or sets the completed response object.
/// This contains the final state of the response including all generated content and metadata.
/// </summary>
[JsonPropertyName("response")]
public required Response Response { get; init; }
}
/// <summary>
/// Represents a streaming response event indicating that the response finished as incomplete.
/// </summary>
internal sealed class StreamingResponseIncomplete : StreamingResponseEvent, IStreamingResponseEventWithResponse
{
/// <summary>
/// The constant event type identifier for response incomplete events.
/// </summary>
public const string EventType = "response.incomplete";
/// <inheritdoc/>
[JsonIgnore]
public override string Type => EventType;
/// <summary>
/// Gets or sets the incomplete response object.
/// </summary>
[JsonPropertyName("response")]
public required Response Response { get; init; }
}
/// <summary>
/// Represents a streaming response event indicating that the response has failed.
/// </summary>
internal sealed class StreamingResponseFailed : StreamingResponseEvent, IStreamingResponseEventWithResponse
{
/// <summary>
/// The constant event type identifier for response failed events.
/// </summary>
public const string EventType = "response.failed";
/// <inheritdoc/>
[JsonIgnore]
public override string Type => EventType;
/// <summary>
/// Gets or sets the failed response object.
/// </summary>
[JsonPropertyName("response")]
public required Response Response { get; init; }
}
/// <summary>
/// Represents a streaming response event indicating that the response has been cancelled.
/// Only responses created with background=true can be cancelled.
/// </summary>
internal sealed class StreamingResponseCancelled : StreamingResponseEvent, IStreamingResponseEventWithResponse
{
/// <summary>
/// The constant event type identifier for response cancelled events.
/// </summary>
public const string EventType = "response.cancelled";
/// <inheritdoc/>
[JsonIgnore]
public override string Type => EventType;
/// <summary>
/// Gets or sets the cancelled response object.
/// </summary>
[JsonPropertyName("response")]
public required Response Response { get; init; }
}
/// <summary>
/// Represents a streaming response event indicating that a new output item has been added to the response.
/// This event is sent when the AI agent produces a new piece of content during streaming.
/// </summary>
internal sealed class StreamingOutputItemAdded : StreamingResponseEvent
{
/// <summary>
/// The constant event type identifier for output item added events.
/// </summary>
public const string EventType = "response.output_item.added";
/// <inheritdoc/>
[JsonIgnore]
public override string Type => EventType;
/// <summary>
/// Gets or sets the index of the output in the response where this item was added.
/// Multiple outputs can exist in a single response, and this identifies which one.
/// </summary>
[JsonPropertyName("output_index")]
public int OutputIndex { get; init; }
/// <summary>
/// Gets or sets the output item that was added.
/// This contains the actual content or data produced by the AI agent.
/// </summary>
[JsonPropertyName("item")]
public required ItemResource Item { get; init; }
}
/// <summary>
/// Represents a streaming response event indicating that an output item has been completed.
/// This event is sent when the AI agent finishes producing a particular piece of content.
/// </summary>
internal sealed class StreamingOutputItemDone : StreamingResponseEvent
{
/// <summary>
/// The constant event type identifier for output item done events.
/// </summary>
public const string EventType = "response.output_item.done";
/// <inheritdoc/>
[JsonIgnore]
public override string Type => EventType;
/// <summary>
/// Gets or sets the index of the output in the response where this item was completed.
/// This corresponds to the same output index from the associated <see cref="StreamingOutputItemAdded"/>.
/// </summary>
[JsonPropertyName("output_index")]
public int OutputIndex { get; init; }
/// <summary>
/// Gets or sets the completed output item.
/// This contains the final version of the content produced by the AI agent.
/// </summary>
[JsonPropertyName("item")]
public required ItemResource Item { get; init; }
}
/// <summary>
/// Represents a streaming response event indicating that a new content part has been added to an output item.
/// </summary>
internal sealed class StreamingContentPartAdded : StreamingResponseEvent
{
/// <summary>
/// The constant event type identifier for content part added events.
/// </summary>
public const string EventType = "response.content_part.added";
/// <inheritdoc/>
[JsonIgnore]
public override string Type => EventType;
/// <summary>
/// Gets or sets the item ID.
/// </summary>
[JsonPropertyName("item_id")]
public required string ItemId { get; init; }
/// <summary>
/// Gets or sets the output index.
/// </summary>
[JsonPropertyName("output_index")]
public int OutputIndex { get; init; }
/// <summary>
/// Gets or sets the content index.
/// </summary>
[JsonPropertyName("content_index")]
public int ContentIndex { get; init; }
/// <summary>
/// Gets or sets the content part that was added.
/// </summary>
[JsonPropertyName("part")]
public required ItemContent Part { get; init; }
}
/// <summary>
/// Represents a streaming response event indicating that a content part has been completed.
/// </summary>
internal sealed class StreamingContentPartDone : StreamingResponseEvent
{
/// <summary>
/// The constant event type identifier for content part done events.
/// </summary>
public const string EventType = "response.content_part.done";
/// <inheritdoc/>
[JsonIgnore]
public override string Type => EventType;
/// <summary>
/// Gets or sets the item ID.
/// </summary>
[JsonPropertyName("item_id")]
public required string ItemId { get; init; }
/// <summary>
/// Gets or sets the output index.
/// </summary>
[JsonPropertyName("output_index")]
public int OutputIndex { get; init; }
/// <summary>
/// Gets or sets the content index.
/// </summary>
[JsonPropertyName("content_index")]
public int ContentIndex { get; init; }
/// <summary>
/// Gets or sets the completed content part.
/// </summary>
[JsonPropertyName("part")]
public required ItemContent Part { get; init; }
}
/// <summary>
/// Represents a streaming response event containing a text delta (incremental text chunk).
/// </summary>
internal sealed class StreamingOutputTextDelta : StreamingResponseEvent
{
/// <summary>
/// The constant event type identifier for output text delta events.
/// </summary>
public const string EventType = "response.output_text.delta";
/// <inheritdoc/>
[JsonIgnore]
public override string Type => EventType;
/// <summary>
/// Gets or sets the item ID.
/// </summary>
[JsonPropertyName("item_id")]
public required string ItemId { get; init; }
/// <summary>
/// Gets or sets the output index.
/// </summary>
[JsonPropertyName("output_index")]
public int OutputIndex { get; init; }
/// <summary>
/// Gets or sets the content index.
/// </summary>
[JsonPropertyName("content_index")]
public int ContentIndex { get; init; }
/// <summary>
/// Gets or sets the text delta (incremental chunk of text).
/// </summary>
[JsonPropertyName("delta")]
public required string Delta { get; init; }
/// <summary>
/// Gets or sets the log probability information for the output tokens.
/// </summary>
[JsonPropertyName("logprobs")]
public List<JsonElement> Logprobs { get; init; } = [];
}
/// <summary>
/// Represents a streaming response event indicating that output text has been completed.
/// </summary>
internal sealed class StreamingOutputTextDone : StreamingResponseEvent
{
/// <summary>
/// The constant event type identifier for output text done events.
/// </summary>
public const string EventType = "response.output_text.done";
/// <inheritdoc/>
[JsonIgnore]
public override string Type => EventType;
/// <summary>
/// Gets or sets the item ID.
/// </summary>
[JsonPropertyName("item_id")]
public required string ItemId { get; init; }
/// <summary>
/// Gets or sets the output index.
/// </summary>
[JsonPropertyName("output_index")]
public int OutputIndex { get; init; }
/// <summary>
/// Gets or sets the content index.
/// </summary>
[JsonPropertyName("content_index")]
public int ContentIndex { get; init; }
/// <summary>
/// Gets or sets the complete text.
/// </summary>
[JsonPropertyName("text")]
public required string Text { get; init; }
}
/// <summary>
/// Represents a streaming response event containing a function call arguments delta.
/// </summary>
internal sealed class StreamingFunctionCallArgumentsDelta : StreamingResponseEvent
{
/// <summary>
/// The constant event type identifier for function call arguments delta events.
/// </summary>
public const string EventType = "response.function_call_arguments.delta";
/// <inheritdoc/>
[JsonIgnore]
public override string Type => EventType;
/// <summary>
/// Gets or sets the item ID.
/// </summary>
[JsonPropertyName("item_id")]
public required string ItemId { get; init; }
/// <summary>
/// Gets or sets the output index.
/// </summary>
[JsonPropertyName("output_index")]
public int OutputIndex { get; init; }
/// <summary>
/// Gets or sets the function arguments delta.
/// </summary>
[JsonPropertyName("delta")]
public required string Delta { get; init; }
}
/// <summary>
/// Represents a streaming response event indicating that function call arguments are complete.
/// </summary>
internal sealed class StreamingFunctionCallArgumentsDone : StreamingResponseEvent
{
/// <summary>
/// The constant event type identifier for function call arguments done events.
/// </summary>
public const string EventType = "response.function_call_arguments.done";
/// <inheritdoc/>
[JsonIgnore]
public override string Type => EventType;
/// <summary>
/// Gets or sets the item ID.
/// </summary>
[JsonPropertyName("item_id")]
public required string ItemId { get; init; }
/// <summary>
/// Gets or sets the output index.
/// </summary>
[JsonPropertyName("output_index")]
public int OutputIndex { get; init; }
/// <summary>
/// Gets or sets the complete function arguments.
/// </summary>
[JsonPropertyName("arguments")]
public required string Arguments { get; init; }
}
/// <summary>
/// Represents a streaming response event containing a reasoning summary text delta (incremental text chunk).
/// </summary>
internal sealed class StreamingReasoningSummaryTextDelta : StreamingResponseEvent
{
/// <summary>
/// The constant event type identifier for reasoning summary text delta events.
/// </summary>
public const string EventType = "response.reasoning_summary_text.delta";
/// <inheritdoc/>
[JsonIgnore]
public override string Type => EventType;
/// <summary>
/// Gets or sets the item ID this summary text delta is associated with.
/// </summary>
[JsonPropertyName("item_id")]
public required string ItemId { get; init; }
/// <summary>
/// Gets or sets the output index.
/// </summary>
[JsonPropertyName("output_index")]
public int OutputIndex { get; init; }
/// <summary>
/// Gets or sets the index of the summary part within the reasoning summary.
/// </summary>
[JsonPropertyName("summary_index")]
public int SummaryIndex { get; init; }
/// <summary>
/// Gets or sets the text delta that was added to the summary.
/// </summary>
[JsonPropertyName("delta")]
public required string Delta { get; init; }
}
/// <summary>
/// Represents a streaming response event indicating that reasoning summary text has been completed.
/// </summary>
internal sealed class StreamingReasoningSummaryTextDone : StreamingResponseEvent
{
/// <summary>
/// The constant event type identifier for reasoning summary text done events.
/// </summary>
public const string EventType = "response.reasoning_summary_text.done";
/// <inheritdoc/>
[JsonIgnore]
public override string Type => EventType;
/// <summary>
/// Gets or sets the item ID this summary text is associated with.
/// </summary>
[JsonPropertyName("item_id")]
public required string ItemId { get; init; }
/// <summary>
/// Gets or sets the output index.
/// </summary>
[JsonPropertyName("output_index")]
public int OutputIndex { get; init; }
/// <summary>
/// Gets or sets the index of the summary part within the reasoning summary.
/// </summary>
[JsonPropertyName("summary_index")]
public int SummaryIndex { get; init; }
/// <summary>
/// Gets or sets the full text of the completed reasoning summary.
/// </summary>
[JsonPropertyName("text")]
public required string Text { get; init; }
}
/// <summary>
/// Represents a streaming response event containing a workflow event.
/// This event is sent during workflow execution to provide observability into workflow steps,
/// executor invocations, errors, and other workflow lifecycle events.
/// </summary>
internal sealed class StreamingWorkflowEventComplete : StreamingResponseEvent
{
/// <summary>
/// The constant event type identifier for workflow event events.
/// </summary>
public const string EventType = "response.workflow_event.completed";
/// <inheritdoc/>
[JsonIgnore]
public override string Type => EventType;
/// <summary>
/// Gets or sets the index of the output in the response.
/// </summary>
[JsonPropertyName("output_index")]
public int OutputIndex { get; set; }
/// <summary>
/// Gets or sets the workflow event data containing event type, executor ID, and event-specific data.
/// </summary>
[JsonPropertyName("data")]
public JsonElement? Data { get; set; }
/// <summary>
/// Gets or sets the executor ID if this is an executor-scoped event.
/// </summary>
[JsonPropertyName("executor_id")]
public string? ExecutorId { get; set; }
/// <summary>
/// Gets or sets the item ID for tracking purposes.
/// </summary>
[JsonPropertyName("item_id")]
public string? ItemId { get; set; }
}
/// <summary>
/// Represents a streaming response event indicating a function approval has been requested.
/// This is a non-standard DevUI extension for human-in-the-loop scenarios.
/// </summary>
internal sealed class StreamingFunctionApprovalRequested : StreamingResponseEvent
{
/// <summary>
/// The constant event type identifier for function approval requested events.
/// </summary>
public const string EventType = "response.function_approval.requested";
/// <inheritdoc/>
[JsonIgnore]
public override string Type => EventType;
/// <summary>
/// Gets or sets the unique identifier for the approval request.
/// </summary>
[JsonPropertyName("request_id")]
public required string RequestId { get; init; }
/// <summary>
/// Gets or sets the function call that requires approval.
/// </summary>
[JsonPropertyName("function_call")]
public required FunctionCallInfo FunctionCall { get; init; }
/// <summary>
/// Gets or sets the item ID for tracking purposes.
/// </summary>
[JsonPropertyName("item_id")]
public required string ItemId { get; init; }
/// <summary>
/// Gets or sets the output index.
/// </summary>
[JsonPropertyName("output_index")]
public int OutputIndex { get; init; }
}
/// <summary>
/// Represents a streaming response event indicating a function approval has been responded to.
/// This is a non-standard DevUI extension for human-in-the-loop scenarios.
/// </summary>
internal sealed class StreamingFunctionApprovalResponded : StreamingResponseEvent
{
/// <summary>
/// The constant event type identifier for function approval responded events.
/// </summary>
public const string EventType = "response.function_approval.responded";
/// <inheritdoc/>
[JsonIgnore]
public override string Type => EventType;
/// <summary>
/// Gets or sets the unique identifier of the approval request being responded to.
/// </summary>
[JsonPropertyName("request_id")]
public required string RequestId { get; init; }
/// <summary>
/// Gets or sets a value indicating whether the function call was approved.
/// </summary>
[JsonPropertyName("approved")]
public bool Approved { get; init; }
/// <summary>
/// Gets or sets the item ID for tracking purposes.
/// </summary>
[JsonPropertyName("item_id")]
public required string ItemId { get; init; }
/// <summary>
/// Gets or sets the output index.
/// </summary>
[JsonPropertyName("output_index")]
public int OutputIndex { get; init; }
}
/// <summary>
/// Represents function call information for approval events.
/// </summary>
internal sealed class FunctionCallInfo
{
/// <summary>
/// Gets or sets the function call ID.
/// </summary>
[JsonPropertyName("id")]
public required string Id { get; init; }
/// <summary>
/// Gets or sets the function name.
/// </summary>
[JsonPropertyName("name")]
public required string Name { get; init; }
/// <summary>
/// Gets or sets the function arguments.
/// </summary>
[JsonPropertyName("arguments")]
public required JsonElement Arguments { get; init; }
}