.NET: Workflow telemetry opt in (#3467)

* feat(workflows): Make telemetry opt-in via WithOpenTelemetry()

- Add WorkflowTelemetryOptions class with EnableSensitiveData property
- Add WorkflowTelemetryContext to manage ActivitySource lifecycle
- Add WithOpenTelemetry() extension method on WorkflowBuilder
- Update all workflow components to use telemetry context:
  - WorkflowBuilder, Workflow, Executor
  - InProcessRunnerContext, InProcessRunner
  - LockstepRunEventStream, StreamingRunEventStream
  - All edge runners (Direct, FanIn, FanOut, Response)
- Telemetry is now disabled by default
- Users must call WithOpenTelemetry() to enable spans/activities

BREAKING CHANGE: Workflow telemetry is now opt-in. Users who relied on
automatic telemetry must add .WithOpenTelemetry() to their workflow builder.

* refactor: Pass telemetry context as parameter instead of via interface

- Remove IWorkflowContextWithTelemetry interface
- Add internal ExecuteAsync overload that accepts WorkflowTelemetryContext
- Public ExecuteAsync delegates with WorkflowTelemetryContext.Disabled
- InProcessRunner passes TelemetryContext when calling ExecuteAsync
- BoundContext now implements IWorkflowContext (not the removed interface)

* Add optional ActivitySource parameter to WithOpenTelemetry

Allow users to provide their own ActivitySource when enabling telemetry,
giving them better control over the ActivitySource lifecycle. When not
provided, the framework creates one internally (existing behavior).

Changes:
- Add optional activitySource parameter to WithOpenTelemetry() extension
- Update WorkflowTelemetryContext to accept external ActivitySource
- Add unit test for user-provided ActivitySource scenario

* Add component-level telemetry control with disable flags

Allow users to selectively disable specific activity types via
WorkflowTelemetryOptions. All activities are enabled by default.

New disable flags:
- DisableWorkflowBuild: Disables workflow.build activities
- DisableWorkflowRun: Disables workflow_invoke activities
- DisableExecutorProcess: Disables executor.process activities
- DisableEdgeGroupProcess: Disables edge_group.process activities
- DisableMessageSend: Disables message.send activities

Added helper methods to WorkflowTelemetryContext for each activity type
and updated all activity creation sites to use them.

* Implement EnableSensitiveData to log executor input/output

When EnableSensitiveData is true in WorkflowTelemetryOptions, executor
input and output are logged as JSON-serialized attributes in the
executor.process activity.

New activity tags:
- executor.input: JSON serialized input message
- executor.output: JSON serialized output result (non-void only)

Added suppression attributes for AOT/trimming warnings since this is
an opt-in feature for debugging/diagnostics.

* Refactor activity start methods to centralize tagging logic

Move tagging logic into WorkflowTelemetryContext methods:
- StartExecutorProcessActivity now accepts executorId, executorType,
  messageType, and message; sets all tags including executor.input
  when EnableSensitiveData is true
- Added SetExecutorOutput method to set executor.output after execution
- StartMessageSendActivity now accepts sourceId, targetId, and message;
  sets all tags including message.content when EnableSensitiveData is true

Simplified Executor.cs and InProcessRunnerContext.cs by removing
inline tagging code. Added message.content tag constant.

* Revert Python changes

* Update samples and code cleanup

* Fix file formatting

* Add comment

* Add telemetry configuration to declarative workflow

* Remove delays in tests

* Address comments
This commit is contained in:
Tao Chen
2026-02-09 15:10:50 -08:00
committed by GitHub
Unverified
parent 80cb6edc8d
commit e3b4b6662b
28 changed files with 1041 additions and 47 deletions
@@ -6,7 +6,6 @@ using Azure.AI.Projects;
using Azure.AI.Projects.OpenAI;
using Azure.Identity;
using Microsoft.Agents.AI;
using Microsoft.Extensions.AI;
string endpoint = Environment.GetEnvironmentVariable("AZURE_FOUNDRY_PROJECT_ENDPOINT") ?? throw new InvalidOperationException("AZURE_FOUNDRY_PROJECT_ENDPOINT is not set.");
string deploymentName = Environment.GetEnvironmentVariable("AZURE_FOUNDRY_PROJECT_DEPLOYMENT_NAME") ?? "gpt-4o-mini";
@@ -35,8 +35,10 @@ public static class Program
using var traceProvider = Sdk.CreateTracerProviderBuilder()
.SetResourceBuilder(resourceBuilder)
.AddSource("Microsoft.Agents.AI.Workflows*")
.AddSource(SourceName)
// The following source is only required if not specifying
// the `activitySource` in the WithOpenTelemetry call below
.AddSource("Microsoft.Agents.AI.Workflows*")
.AddAzureMonitorTraceExporter(options => options.ConnectionString = applicationInsightsConnectionString)
.Build();
@@ -51,6 +53,10 @@ public static class Program
// Build the workflow by connecting executors sequentially
var workflow = new WorkflowBuilder(uppercase)
.AddEdge(uppercase, reverse)
.WithOpenTelemetry(
// Set `EnableSensitiveData` to true to include message content in traces
configure: cfg => cfg.EnableSensitiveData = true,
activitySource: s_activitySource)
.Build();
// Execute the workflow with input data
@@ -37,8 +37,10 @@ public static class Program
using var traceProvider = Sdk.CreateTracerProviderBuilder()
.SetResourceBuilder(resourceBuilder)
.AddSource("Microsoft.Agents.AI.Workflows*")
.AddSource(SourceName)
// The following source is only required if not specifying
// the `activitySource` in the WithOpenTelemetry call below
.AddSource("Microsoft.Agents.AI.Workflows*")
.AddOtlpExporter(options => options.Endpoint = new Uri(otlpEndpoint))
.Build();
@@ -53,6 +55,10 @@ public static class Program
// Build the workflow by connecting executors sequentially
var workflow = new WorkflowBuilder(uppercase)
.AddEdge(uppercase, reverse)
.WithOpenTelemetry(
// Set `EnableSensitiveData` to true to include message content in traces
configure: cfg => cfg.EnableSensitiveData = true,
activitySource: s_activitySource)
.Build();
// Execute the workflow with input data
@@ -1,5 +1,8 @@
// Copyright (c) Microsoft. All rights reserved.
using System;
using System.Diagnostics;
using Microsoft.Agents.AI.Workflows.Observability;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
@@ -41,4 +44,23 @@ public sealed class DeclarativeWorkflowOptions(WorkflowAgentProvider agentProvid
/// Gets the <see cref="ILoggerFactory"/> used to create loggers for workflow components.
/// </summary>
public ILoggerFactory LoggerFactory { get; init; } = NullLoggerFactory.Instance;
/// <summary>
/// Gets the callback to configure telemetry options.
/// </summary>
public Action<WorkflowTelemetryOptions>? ConfigureTelemetry { get; init; }
/// <summary>
/// Gets an optional <see cref="ActivitySource"/> for telemetry.
/// If provided, the caller retains ownership and is responsible for disposal.
/// If <see langword="null"/> but <see cref="ConfigureTelemetry"/> is set, a shared default
/// activity source named "Microsoft.Agents.AI.Workflows" will be used.
/// </summary>
public ActivitySource? TelemetryActivitySource { get; init; }
/// <summary>
/// Gets a value indicating whether telemetry is enabled.
/// Telemetry is enabled when either <see cref="ConfigureTelemetry"/> or <see cref="TelemetryActivitySource"/> is set.
/// </summary>
internal bool IsTelemetryEnabled => this.ConfigureTelemetry is not null || this.TelemetryActivitySource is not null;
}
@@ -51,6 +51,14 @@ internal sealed class WorkflowActionVisitor : DialogActionVisitor
this._workflowModel.Build(builder);
// Apply telemetry if configured
if (this._workflowOptions.IsTelemetryEnabled)
{
builder.WorkflowBuilder.WithOpenTelemetry(
this._workflowOptions.ConfigureTelemetry,
this._workflowOptions.TelemetryActivitySource);
}
// Build final workflow
return builder.WorkflowBuilder.Build(validateOrphans: false);
}
@@ -14,7 +14,7 @@ internal sealed class DirectEdgeRunner(IRunnerContext runContext, DirectEdgeData
protected internal override async ValueTask<DeliveryMapping?> ChaseEdgeAsync(MessageEnvelope envelope, IStepTracer? stepTracer)
{
using var activity = s_activitySource.StartActivity(ActivityNames.EdgeGroupProcess);
using var activity = this.StartActivity();
activity?
.SetTag(Tags.EdgeGroupType, nameof(DirectEdgeRunner))
.SetTag(Tags.MessageSourceId, this.EdgeData.SourceId)
@@ -14,9 +14,6 @@ internal interface IStatefulEdgeRunner
internal abstract class EdgeRunner
{
protected static readonly string s_namespace = typeof(EdgeRunner).Namespace!;
protected static readonly ActivitySource s_activitySource = new(s_namespace);
// TODO: Can this be sync?
protected internal abstract ValueTask<DeliveryMapping?> ChaseEdgeAsync(MessageEnvelope envelope, IStepTracer? stepTracer);
}
@@ -26,4 +23,6 @@ internal abstract class EdgeRunner<TEdgeData>(
{
protected IRunnerContext RunContext { get; } = Throw.IfNull(runContext);
protected TEdgeData EdgeData { get; } = Throw.IfNull(edgeData);
protected Activity? StartActivity() => this.RunContext.TelemetryContext.StartEdgeGroupProcessActivity();
}
@@ -19,7 +19,7 @@ internal sealed class FanInEdgeRunner(IRunnerContext runContext, FanInEdgeData e
{
Debug.Assert(!envelope.IsExternal, "FanIn edges should never be chased from external input");
using var activity = s_activitySource.StartActivity(ActivityNames.EdgeGroupProcess);
using var activity = this.StartActivity();
activity?
.SetTag(Tags.EdgeGroupType, nameof(FanInEdgeRunner))
.SetTag(Tags.MessageTargetId, this.EdgeData.SinkId);
@@ -13,7 +13,7 @@ internal sealed class FanOutEdgeRunner(IRunnerContext runContext, FanOutEdgeData
{
protected internal override async ValueTask<DeliveryMapping?> ChaseEdgeAsync(MessageEnvelope envelope, IStepTracer? stepTracer)
{
using var activity = s_activitySource.StartActivity(ActivityNames.EdgeGroupProcess);
using var activity = this.StartActivity();
activity?
.SetTag(Tags.EdgeGroupType, nameof(FanOutEdgeRunner))
.SetTag(Tags.MessageSourceId, this.EdgeData.SourceId);
@@ -3,11 +3,14 @@
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Agents.AI.Workflows.Observability;
namespace Microsoft.Agents.AI.Workflows.Execution;
internal interface IRunnerContext : IExternalRequestSink, ISuperStepJoinContext
{
WorkflowTelemetryContext TelemetryContext { get; }
ValueTask AddEventAsync(WorkflowEvent workflowEvent, CancellationToken cancellationToken = default);
ValueTask SendMessageAsync(string sourceId, object message, string? targetId = null, CancellationToken cancellationToken = default);
@@ -3,6 +3,7 @@
using System;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Agents.AI.Workflows.Observability;
namespace Microsoft.Agents.AI.Workflows.Execution;
@@ -12,6 +13,8 @@ internal interface ISuperStepRunner
string StartExecutorId { get; }
WorkflowTelemetryContext TelemetryContext { get; }
bool HasUnservicedRequests { get; }
bool HasUnprocessedMessages { get; }
@@ -13,9 +13,6 @@ namespace Microsoft.Agents.AI.Workflows.Execution;
internal sealed class LockstepRunEventStream : IRunEventStream
{
private static readonly string s_namespace = typeof(LockstepRunEventStream).Namespace!;
private static readonly ActivitySource s_activitySource = new(s_namespace);
private readonly CancellationTokenSource _stopCancellation = new();
private readonly InputWaiter _inputWaiter = new();
private int _isDisposed;
@@ -53,7 +50,7 @@ internal sealed class LockstepRunEventStream : IRunEventStream
this._stepRunner.OutgoingEvents.EventRaised += OnWorkflowEventAsync;
using Activity? activity = s_activitySource.StartActivity(ActivityNames.WorkflowRun);
using Activity? activity = this._stepRunner.TelemetryContext.StartWorkflowRunActivity();
activity?.SetTag(Tags.WorkflowId, this._stepRunner.StartExecutorId).SetTag(Tags.RunId, this._stepRunner.RunId);
try
@@ -25,7 +25,7 @@ internal sealed class ResponseEdgeRunner(IRunnerContext runContext, string execu
{
Debug.Assert(envelope.IsExternal, "Input edges should only be chased from external input");
using var activity = s_activitySource.StartActivity(ActivityNames.EdgeGroupProcess);
using var activity = this.StartActivity();
activity?
.SetTag(Tags.EdgeGroupType, nameof(ResponseEdgeRunner))
.SetTag(Tags.MessageSourceId, envelope.SourceId)
@@ -17,9 +17,6 @@ namespace Microsoft.Agents.AI.Workflows.Execution;
/// </summary>
internal sealed class StreamingRunEventStream : IRunEventStream
{
private static readonly string s_namespace = typeof(StreamingRunEventStream).Namespace!;
private static readonly ActivitySource s_activitySource = new(s_namespace);
private readonly Channel<WorkflowEvent> _eventChannel;
private readonly ISuperStepRunner _stepRunner;
private readonly InputWaiter _inputWaiter;
@@ -63,7 +60,7 @@ internal sealed class StreamingRunEventStream : IRunEventStream
// Subscribe to events - they will flow directly to the channel as they're raised
this._stepRunner.OutgoingEvents.EventRaised += OnEventRaisedAsync;
using Activity? activity = s_activitySource.StartActivity(ActivityNames.WorkflowRun);
using Activity? activity = this._stepRunner.TelemetryContext.StartWorkflowRunActivity();
activity?.SetTag(Tags.WorkflowId, this._stepRunner.StartExecutorId).SetTag(Tags.RunId, this._stepRunner.RunId);
try
@@ -26,9 +26,6 @@ public abstract class Executor : IIdentified
/// </summary>
public string Id { get; }
private static readonly string s_namespace = typeof(Executor).Namespace!;
private static readonly ActivitySource s_activitySource = new(s_namespace);
// TODO: Add overloads for binding with a configuration/options object once the Configured<T> hierarchy goes away.
/// <summary>
@@ -142,13 +139,13 @@ public abstract class Executor : IIdentified
/// <returns>A ValueTask representing the asynchronous operation, wrapping the output from the executor.</returns>
/// <exception cref="NotSupportedException">No handler found for the message type.</exception>
/// <exception cref="TargetInvocationException">An exception is generated while handling the message.</exception>
public async ValueTask<object?> ExecuteAsync(object message, TypeId messageType, IWorkflowContext context, CancellationToken cancellationToken = default)
public ValueTask<object?> ExecuteAsync(object message, TypeId messageType, IWorkflowContext context, CancellationToken cancellationToken = default)
=> this.ExecuteAsync(message, messageType, context, WorkflowTelemetryContext.Disabled, cancellationToken);
internal async ValueTask<object?> ExecuteAsync(object message, TypeId messageType, IWorkflowContext context, WorkflowTelemetryContext telemetryContext, CancellationToken cancellationToken = default)
{
using var activity = s_activitySource.StartActivity(ActivityNames.ExecutorProcess, ActivityKind.Internal);
activity?.SetTag(Tags.ExecutorId, this.Id)
.SetTag(Tags.ExecutorType, this.GetType().FullName)
.SetTag(Tags.MessageType, messageType.TypeName)
.CreateSourceLinks(context.TraceContext);
using var activity = telemetryContext.StartExecutorProcessActivity(this.Id, this.GetType().FullName, messageType.TypeName, message);
activity?.CreateSourceLinks(context.TraceContext);
await context.AddEventAsync(new ExecutorInvokedEvent(this.Id, message), cancellationToken).ConfigureAwait(false);
@@ -183,6 +180,11 @@ public abstract class Executor : IIdentified
return null; // Void result.
}
// Output is not available if executor does not return anything, in which case
// messages sent in the handlers of this executor will be set in the message
// send activities.
telemetryContext.SetExecutorOutput(activity, result.Result);
// If we had a real return type, raise it as a SendMessage; TODO: Should we have a way to disable this behaviour?
if (result.Result is not null && this.Options.AutoSendMessageHandlerResultObject)
{
@@ -9,6 +9,7 @@ using System.Threading;
using System.Threading.Tasks;
using Microsoft.Agents.AI.Workflows.Checkpointing;
using Microsoft.Agents.AI.Workflows.Execution;
using Microsoft.Agents.AI.Workflows.Observability;
using Microsoft.Shared.Diagnostics;
namespace Microsoft.Agents.AI.Workflows.InProc;
@@ -70,6 +71,9 @@ internal sealed class InProcessRunner : ISuperStepRunner, ICheckpointingHandle
/// <inheritdoc cref="ISuperStepRunner.StartExecutorId"/>
public string StartExecutorId { get; }
/// <inheritdoc cref="ISuperStepRunner.TelemetryContext"/>
public WorkflowTelemetryContext TelemetryContext => this.Workflow.TelemetryContext;
private readonly HashSet<Type> _knownValidInputTypes;
public async ValueTask<bool> IsValidInputTypeAsync(Type messageType, CancellationToken cancellationToken = default)
{
@@ -201,6 +205,7 @@ internal sealed class InProcessRunner : ISuperStepRunner, ICheckpointingHandle
envelope.Message,
envelope.MessageType,
this.RunContext.BindWorkflowContext(receiverId, envelope.TraceContext),
this.TelemetryContext,
cancellationToken
).ConfigureAwait(false);
}
@@ -70,6 +70,7 @@ internal sealed class InProcessRunnerContext : IRunnerContext
this.ConcurrentRunsEnabled = enableConcurrentRuns;
this.OutgoingEvents = outgoingEvents;
}
public WorkflowTelemetryContext TelemetryContext => this._workflow.TelemetryContext;
public IExternalRequestSink RegisterPort(string executorId, RequestPort port)
{
@@ -195,12 +196,10 @@ internal sealed class InProcessRunnerContext : IRunnerContext
return this.OutgoingEvents.EnqueueAsync(workflowEvent);
}
private static readonly string s_namespace = typeof(IWorkflowContext).Namespace!;
private static readonly ActivitySource s_activitySource = new(s_namespace);
public async ValueTask SendMessageAsync(string sourceId, object message, string? targetId = null, CancellationToken cancellationToken = default)
{
using Activity? activity = s_activitySource.StartActivity(ActivityNames.MessageSend, ActivityKind.Producer);
using Activity? activity = this._workflow.TelemetryContext.StartMessageSendActivity(sourceId, targetId, message);
// Create a carrier for trace context propagation
var traceContext = activity is null ? null : new Dictionary<string, string>();
if (traceContext is not null)
@@ -5,7 +5,7 @@ namespace Microsoft.Agents.AI.Workflows.Observability;
internal static class ActivityNames
{
public const string WorkflowBuild = "workflow.build";
public const string WorkflowRun = "workflow.run";
public const string WorkflowRun = "workflow_invoke";
public const string MessageSend = "message.send";
public const string ExecutorProcess = "executor.process";
public const string EdgeGroupProcess = "edge_group.process";
@@ -14,7 +14,10 @@ internal static class Tags
public const string RunId = "run.id";
public const string ExecutorId = "executor.id";
public const string ExecutorType = "executor.type";
public const string ExecutorInput = "executor.input";
public const string ExecutorOutput = "executor.output";
public const string MessageType = "message.type";
public const string MessageContent = "message.content";
public const string EdgeGroupType = "edge_group.type";
public const string MessageSourceId = "message.source_id";
public const string MessageTargetId = "message.target_id";
@@ -0,0 +1,216 @@
// Copyright (c) Microsoft. All rights reserved.
using System.Diagnostics;
using System.Diagnostics.CodeAnalysis;
using System.Text.Json;
namespace Microsoft.Agents.AI.Workflows.Observability;
/// <summary>
/// Internal context for workflow telemetry, holding the enabled state and configuration options.
/// </summary>
internal sealed class WorkflowTelemetryContext
{
private const string DefaultSourceName = "Microsoft.Agents.AI.Workflows";
private static readonly ActivitySource s_defaultActivitySource = new(DefaultSourceName);
/// <summary>
/// Gets a shared instance representing disabled telemetry.
/// </summary>
public static WorkflowTelemetryContext Disabled { get; } = new();
/// <summary>
/// Gets a value indicating whether telemetry is enabled.
/// </summary>
public bool IsEnabled { get; }
/// <summary>
/// Gets the telemetry options.
/// </summary>
public WorkflowTelemetryOptions Options { get; }
/// <summary>
/// Gets the activity source used for creating telemetry spans.
/// </summary>
public ActivitySource ActivitySource { get; }
private WorkflowTelemetryContext()
{
this.IsEnabled = false;
this.Options = new WorkflowTelemetryOptions();
this.ActivitySource = s_defaultActivitySource;
}
/// <summary>
/// Initializes a new instance of the <see cref="WorkflowTelemetryContext"/> class with telemetry enabled.
/// </summary>
/// <param name="options">The telemetry options.</param>
/// <param name="activitySource">
/// An optional activity source to use. If provided, this activity source will be used directly
/// and the caller retains ownership (responsible for disposal). If <see langword="null"/>, the
/// shared default activity source will be used.
/// </param>
public WorkflowTelemetryContext(WorkflowTelemetryOptions options, ActivitySource? activitySource = null)
{
this.IsEnabled = true;
this.Options = options;
this.ActivitySource = activitySource ?? s_defaultActivitySource;
}
/// <summary>
/// Starts an activity if telemetry is enabled, otherwise returns null.
/// </summary>
/// <param name="name">The activity name.</param>
/// <param name="kind">The activity kind.</param>
/// <returns>An activity if telemetry is enabled and the activity is sampled, otherwise null.</returns>
public Activity? StartActivity(string name, ActivityKind kind = ActivityKind.Internal)
{
if (!this.IsEnabled)
{
return null;
}
return this.ActivitySource.StartActivity(name, kind);
}
/// <summary>
/// Starts a workflow build activity if enabled.
/// </summary>
/// <returns>An activity if workflow build telemetry is enabled, otherwise null.</returns>
public Activity? StartWorkflowBuildActivity()
{
if (!this.IsEnabled || this.Options.DisableWorkflowBuild)
{
return null;
}
return this.ActivitySource.StartActivity(ActivityNames.WorkflowBuild);
}
/// <summary>
/// Starts a workflow run activity if enabled.
/// </summary>
/// <returns>An activity if workflow run telemetry is enabled, otherwise null.</returns>
public Activity? StartWorkflowRunActivity()
{
if (!this.IsEnabled || this.Options.DisableWorkflowRun)
{
return null;
}
return this.ActivitySource.StartActivity(ActivityNames.WorkflowRun);
}
/// <summary>
/// Starts an executor process activity if enabled, with all standard tags set.
/// </summary>
/// <param name="executorId">The executor identifier.</param>
/// <param name="executorType">The executor type name.</param>
/// <param name="messageType">The message type name.</param>
/// <param name="message">The input message. Logged only when <see cref="WorkflowTelemetryOptions.EnableSensitiveData"/> is true.</param>
/// <returns>An activity if executor process telemetry is enabled, otherwise null.</returns>
public Activity? StartExecutorProcessActivity(string executorId, string? executorType, string messageType, object? message)
{
if (!this.IsEnabled || this.Options.DisableExecutorProcess)
{
return null;
}
Activity? activity = this.ActivitySource.StartActivity(ActivityNames.ExecutorProcess + " " + executorId);
if (activity is null)
{
return null;
}
activity.SetTag(Tags.ExecutorId, executorId)
.SetTag(Tags.ExecutorType, executorType)
.SetTag(Tags.MessageType, messageType);
if (this.Options.EnableSensitiveData)
{
activity.SetTag(Tags.ExecutorInput, SerializeForTelemetry(message));
}
return activity;
}
/// <summary>
/// Sets the executor output tag on an activity when sensitive data logging is enabled.
/// </summary>
/// <param name="activity">The activity to set the output on.</param>
/// <param name="output">The output value to log.</param>
public void SetExecutorOutput(Activity? activity, object? output)
{
if (activity is not null && this.Options.EnableSensitiveData)
{
activity.SetTag(Tags.ExecutorOutput, SerializeForTelemetry(output));
}
}
/// <summary>
/// Starts an edge group process activity if enabled.
/// </summary>
/// <returns>An activity if edge group process telemetry is enabled, otherwise null.</returns>
public Activity? StartEdgeGroupProcessActivity()
{
if (!this.IsEnabled || this.Options.DisableEdgeGroupProcess)
{
return null;
}
return this.ActivitySource.StartActivity(ActivityNames.EdgeGroupProcess);
}
/// <summary>
/// Starts a message send activity if enabled, with all standard tags set.
/// </summary>
/// <param name="sourceId">The source executor identifier.</param>
/// <param name="targetId">The target executor identifier, if any.</param>
/// <param name="message">The message being sent. Logged only when <see cref="WorkflowTelemetryOptions.EnableSensitiveData"/> is true.</param>
/// <returns>An activity if message send telemetry is enabled, otherwise null.</returns>
public Activity? StartMessageSendActivity(string sourceId, string? targetId, object? message)
{
if (!this.IsEnabled || this.Options.DisableMessageSend)
{
return null;
}
Activity? activity = this.ActivitySource.StartActivity(ActivityNames.MessageSend, ActivityKind.Producer);
if (activity is null)
{
return null;
}
activity.SetTag(Tags.MessageSourceId, sourceId);
if (targetId is not null)
{
activity.SetTag(Tags.MessageTargetId, targetId);
}
if (this.Options.EnableSensitiveData)
{
activity.SetTag(Tags.MessageContent, SerializeForTelemetry(message));
}
return activity;
}
[UnconditionalSuppressMessage("ReflectionAnalysis", "IL3050:RequiresDynamicCode", Justification = "Telemetry serialization is optional and only used when explicitly enabled.")]
[UnconditionalSuppressMessage("Trimming", "IL2026:Members annotated with 'RequiresUnreferencedCodeAttribute' require dynamic access", Justification = "Telemetry serialization is optional and only used when explicitly enabled.")]
private static string? SerializeForTelemetry(object? value)
{
if (value is null)
{
return null;
}
try
{
return JsonSerializer.Serialize(value, value.GetType());
}
catch (JsonException)
{
return $"[Unserializable: {value.GetType().FullName}]";
}
}
}
@@ -0,0 +1,68 @@
// Copyright (c) Microsoft. All rights reserved.
namespace Microsoft.Agents.AI.Workflows.Observability;
/// <summary>
/// Configuration options for workflow telemetry.
/// </summary>
public sealed class WorkflowTelemetryOptions
{
/// <summary>
/// Gets or sets a value indicating whether potentially sensitive information should be included in telemetry.
/// </summary>
/// <value>
/// <see langword="true"/> if potentially sensitive information should be included in telemetry;
/// <see langword="false"/> if telemetry shouldn't include raw inputs and outputs.
/// The default value is <see langword="false"/>.
/// </value>
/// <remarks>
/// By default, telemetry includes metadata but not raw inputs and outputs,
/// such as message content and executor data.
/// </remarks>
public bool EnableSensitiveData { get; set; }
/// <summary>
/// Gets or sets a value indicating whether workflow build activities should be disabled.
/// </summary>
/// <value>
/// <see langword="true"/> to disable <c>workflow.build</c> activities;
/// <see langword="false"/> to enable them. The default value is <see langword="false"/>.
/// </value>
public bool DisableWorkflowBuild { get; set; }
/// <summary>
/// Gets or sets a value indicating whether workflow run activities should be disabled.
/// </summary>
/// <value>
/// <see langword="true"/> to disable <c>workflow_invoke</c> activities;
/// <see langword="false"/> to enable them. The default value is <see langword="false"/>.
/// </value>
public bool DisableWorkflowRun { get; set; }
/// <summary>
/// Gets or sets a value indicating whether executor process activities should be disabled.
/// </summary>
/// <value>
/// <see langword="true"/> to disable <c>executor.process</c> activities;
/// <see langword="false"/> to enable them. The default value is <see langword="false"/>.
/// </value>
public bool DisableExecutorProcess { get; set; }
/// <summary>
/// Gets or sets a value indicating whether edge group process activities should be disabled.
/// </summary>
/// <value>
/// <see langword="true"/> to disable <c>edge_group.process</c> activities;
/// <see langword="false"/> to enable them. The default value is <see langword="false"/>.
/// </value>
public bool DisableEdgeGroupProcess { get; set; }
/// <summary>
/// Gets or sets a value indicating whether message send activities should be disabled.
/// </summary>
/// <value>
/// <see langword="true"/> to disable <c>message.send</c> activities;
/// <see langword="false"/> to enable them. The default value is <see langword="false"/>.
/// </value>
public bool DisableMessageSend { get; set; }
}
@@ -0,0 +1,69 @@
// Copyright (c) Microsoft. All rights reserved.
using System;
using System.Diagnostics;
using Microsoft.Agents.AI.Workflows.Observability;
using Microsoft.Shared.Diagnostics;
namespace Microsoft.Agents.AI.Workflows;
/// <summary>
/// Provides extension methods for adding OpenTelemetry instrumentation to <see cref="WorkflowBuilder"/> instances.
/// </summary>
public static class OpenTelemetryWorkflowBuilderExtensions
{
/// <summary>
/// Enables OpenTelemetry instrumentation for the workflow, providing comprehensive observability for workflow operations.
/// </summary>
/// <param name="builder">The <see cref="WorkflowBuilder"/> to which OpenTelemetry support will be added.</param>
/// <param name="configure">
/// An optional callback that provides additional configuration of the <see cref="WorkflowTelemetryOptions"/> instance.
/// This allows for fine-tuning telemetry behavior such as enabling sensitive data collection.
/// </param>
/// <param name="activitySource">
/// An optional <see cref="ActivitySource"/> to use for telemetry. If provided, this activity source will be used
/// directly and the caller retains ownership (responsible for disposal). If <see langword="null"/>, a shared
/// default activity source named "Microsoft.Agents.AI.Workflows" will be used.
/// </param>
/// <returns>The <see cref="WorkflowBuilder"/> with OpenTelemetry instrumentation enabled, enabling method chaining.</returns>
/// <exception cref="ArgumentNullException"><paramref name="builder"/> is <see langword="null"/>.</exception>
/// <remarks>
/// <para>
/// This extension adds comprehensive telemetry capabilities to workflows, including:
/// <list type="bullet">
/// <item><description>Distributed tracing of workflow execution</description></item>
/// <item><description>Executor invocation and processing spans</description></item>
/// <item><description>Edge routing and message delivery spans</description></item>
/// <item><description>Workflow build and validation spans</description></item>
/// <item><description>Error tracking and exception details</description></item>
/// </list>
/// </para>
/// <para>
/// By default, workflow telemetry is disabled. Call this method to enable telemetry collection.
/// </para>
/// </remarks>
/// <example>
/// <code>
/// var workflow = new WorkflowBuilder(startExecutor)
/// .AddEdge(executor1, executor2)
/// .WithOpenTelemetry(cfg => cfg.EnableSensitiveData = true)
/// .Build();
/// </code>
/// </example>
public static WorkflowBuilder WithOpenTelemetry(
this WorkflowBuilder builder,
Action<WorkflowTelemetryOptions>? configure = null,
ActivitySource? activitySource = null)
{
Throw.IfNull(builder);
WorkflowTelemetryOptions options = new();
configure?.Invoke(options);
WorkflowTelemetryContext context = new(options, activitySource);
builder.SetTelemetryContext(context);
return builder;
}
}
@@ -8,6 +8,7 @@ using System.Threading;
using System.Threading.Tasks;
using Microsoft.Agents.AI.Workflows.Checkpointing;
using Microsoft.Agents.AI.Workflows.Execution;
using Microsoft.Agents.AI.Workflows.Observability;
using Microsoft.Shared.Diagnostics;
namespace Microsoft.Agents.AI.Workflows;
@@ -76,6 +77,11 @@ public class Workflow
/// </summary>
public string? Description { get; internal init; }
/// <summary>
/// Gets the telemetry context for the workflow.
/// </summary>
internal WorkflowTelemetryContext TelemetryContext { get; }
internal bool AllowConcurrent => this.ExecutorBindings.Values.All(registration => registration.SupportsConcurrentSharedExecution);
internal IEnumerable<string> NonConcurrentExecutorIds =>
@@ -88,11 +94,13 @@ public class Workflow
/// <param name="startExecutorId">The unique identifier of the starting executor for the workflow. Cannot be <c>null</c>.</param>
/// <param name="name">Optional human-readable name for the workflow.</param>
/// <param name="description">Optional description of what the workflow does.</param>
internal Workflow(string startExecutorId, string? name = null, string? description = null)
/// <param name="telemetryContext">Optional telemetry context for the workflow.</param>
internal Workflow(string startExecutorId, string? name = null, string? description = null, WorkflowTelemetryContext? telemetryContext = null)
{
this.StartExecutorId = Throw.IfNull(startExecutorId);
this.Name = name;
this.Description = description;
this.TelemetryContext = telemetryContext ?? WorkflowTelemetryContext.Disabled;
}
private bool _needsReset;
@@ -38,9 +38,7 @@ public class WorkflowBuilder
private readonly string _startExecutorId;
private string? _name;
private string? _description;
private static readonly string s_namespace = typeof(WorkflowBuilder).Namespace!;
private static readonly ActivitySource s_activitySource = new(s_namespace);
private WorkflowTelemetryContext _telemetryContext = WorkflowTelemetryContext.Disabled;
/// <summary>
/// Initializes a new instance of the WorkflowBuilder class with the specified starting executor.
@@ -137,6 +135,15 @@ public class WorkflowBuilder
return this;
}
/// <summary>
/// Sets the telemetry context for the workflow.
/// </summary>
/// <param name="context">The telemetry context to use.</param>
internal void SetTelemetryContext(WorkflowTelemetryContext context)
{
this._telemetryContext = Throw.IfNull(context);
}
/// <summary>
/// Binds the specified executor (via registration) to the workflow, allowing it to participate in workflow execution.
/// </summary>
@@ -563,7 +570,7 @@ public class WorkflowBuilder
activity?.AddEvent(new ActivityEvent(EventNames.BuildValidationCompleted));
var workflow = new Workflow(this._startExecutorId, this._name, this._description)
var workflow = new Workflow(this._startExecutorId, this._name, this._description, this._telemetryContext)
{
ExecutorBindings = this._executorBindings,
Edges = this._edges,
@@ -601,7 +608,7 @@ public class WorkflowBuilder
/// or if the start executor is not bound.</exception>
public Workflow Build(bool validateOrphans = true)
{
using Activity? activity = s_activitySource.StartActivity(ActivityNames.WorkflowBuild);
using Activity? activity = this._telemetryContext.StartWorkflowBuildActivity();
var workflow = this.BuildInternal(validateOrphans, activity);
@@ -1,7 +1,5 @@
// Copyright (c) Microsoft. All rights reserved.
using Xunit;
namespace Microsoft.Agents.AI.CosmosNoSql.UnitTests;
/// <summary>
@@ -0,0 +1,259 @@
// Copyright (c) Microsoft. All rights reserved.
using System;
using System.Collections.Concurrent;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Agents.AI.Workflows.Observability;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.Logging.Abstractions;
using Moq;
namespace Microsoft.Agents.AI.Workflows.Declarative.UnitTests;
/// <summary>
/// Tests for <see cref="DeclarativeWorkflowOptions"/> telemetry configuration.
/// </summary>
[Collection("DeclarativeWorkflowOptionsTest")]
public sealed class DeclarativeWorkflowOptionsTest : IDisposable
{
// These constants mirror Microsoft.Agents.AI.Workflows.Observability.ActivityNames
// which is internal and not accessible from this test project.
private const string WorkflowBuildActivityName = "workflow.build";
private const string WorkflowRunActivityName = "workflow_invoke";
// The default activity source name used by the workflow telemetry context.
private const string DefaultTelemetrySourceName = "Microsoft.Agents.AI.Workflows";
private const string SimpleWorkflowYaml = """
kind: Workflow
trigger:
kind: OnConversationStart
id: test_workflow
actions:
- kind: EndConversation
id: end_all
""";
private readonly ActivitySource _activitySource = new("TestSource");
private readonly ActivityListener _activityListener;
private readonly ConcurrentBag<Activity> _capturedActivities = [];
public DeclarativeWorkflowOptionsTest()
{
this._activityListener = new ActivityListener
{
ShouldListenTo = source =>
source.Name == DefaultTelemetrySourceName ||
source.Name == "TestSource",
Sample = (ref ActivityCreationOptions<ActivityContext> options) => ActivitySamplingResult.AllData,
ActivityStarted = activity => this._capturedActivities.Add(activity),
};
ActivitySource.AddActivityListener(this._activityListener);
}
public void Dispose()
{
this._activityListener.Dispose();
this._activitySource.Dispose();
}
[Fact]
public void ConfigureTelemetry_DefaultIsNull()
{
// Arrange
Mock<WorkflowAgentProvider> mockProvider = CreateMockProvider();
// Act
DeclarativeWorkflowOptions options = new(mockProvider.Object);
// Assert
Assert.Null(options.ConfigureTelemetry);
}
[Fact]
public void ConfigureTelemetry_CanBeSet()
{
// Arrange
Mock<WorkflowAgentProvider> mockProvider = CreateMockProvider();
bool callbackInvoked = false;
// Act
DeclarativeWorkflowOptions options = new(mockProvider.Object)
{
ConfigureTelemetry = opt =>
{
callbackInvoked = true;
opt.EnableSensitiveData = true;
}
};
// Assert
Assert.NotNull(options.ConfigureTelemetry);
WorkflowTelemetryOptions telemetryOptions = new();
options.ConfigureTelemetry(telemetryOptions);
Assert.True(callbackInvoked);
Assert.True(telemetryOptions.EnableSensitiveData);
}
[Fact]
public void TelemetryActivitySource_DefaultIsNull()
{
// Arrange
Mock<WorkflowAgentProvider> mockProvider = CreateMockProvider();
// Act
DeclarativeWorkflowOptions options = new(mockProvider.Object);
// Assert
Assert.Null(options.TelemetryActivitySource);
}
[Fact]
public void TelemetryActivitySource_CanBeSet()
{
// Arrange
Mock<WorkflowAgentProvider> mockProvider = CreateMockProvider();
// Act
DeclarativeWorkflowOptions options = new(mockProvider.Object)
{
TelemetryActivitySource = this._activitySource
};
// Assert
Assert.Same(this._activitySource, options.TelemetryActivitySource);
}
[Fact]
public async Task BuildWorkflow_WithDefaultTelemetry_AppliesTelemetryAsync()
{
// Arrange
using Activity testActivity = new Activity("DefaultTelemetryTest").Start()!;
Mock<WorkflowAgentProvider> mockProvider = CreateMockProvider();
DeclarativeWorkflowOptions options = new(mockProvider.Object)
{
ConfigureTelemetry = _ => { },
LoggerFactory = NullLoggerFactory.Instance
};
// Act
using StringReader reader = new(SimpleWorkflowYaml);
Workflow workflow = DeclarativeWorkflowBuilder.Build<string>(reader, options);
await using Run run = await InProcessExecution.RunAsync(workflow, "test input");
// Assert
Activity[] capturedActivities = this._capturedActivities
.Where(a => a.RootId == testActivity.RootId && a.Source.Name == DefaultTelemetrySourceName)
.ToArray();
Assert.NotEmpty(capturedActivities);
Assert.Contains(capturedActivities, a => a.OperationName.StartsWith(WorkflowBuildActivityName, StringComparison.Ordinal));
Assert.Contains(capturedActivities, a => a.OperationName.StartsWith(WorkflowRunActivityName, StringComparison.Ordinal));
}
[Fact]
public async Task BuildWorkflow_WithTelemetryActivitySource_AppliesTelemetryAsync()
{
// Arrange
using Activity testActivity = new Activity("TelemetryActivitySourceTest").Start()!;
Mock<WorkflowAgentProvider> mockProvider = CreateMockProvider();
DeclarativeWorkflowOptions options = new(mockProvider.Object)
{
TelemetryActivitySource = this._activitySource,
LoggerFactory = NullLoggerFactory.Instance
};
// Act
using StringReader reader = new(SimpleWorkflowYaml);
Workflow workflow = DeclarativeWorkflowBuilder.Build<string>(reader, options);
await using Run run = await InProcessExecution.RunAsync(workflow, "test input");
// Assert
Activity[] capturedActivities = this._capturedActivities
.Where(a => a.RootId == testActivity.RootId && a.Source.Name == "TestSource")
.ToArray();
Assert.NotEmpty(capturedActivities);
Assert.All(capturedActivities, a => Assert.Equal("TestSource", a.Source.Name));
}
[Fact]
public async Task BuildWorkflow_WithConfigureTelemetry_AppliesConfigurationAsync()
{
// Arrange
using Activity testActivity = new Activity("ConfigureTelemetryTest").Start()!;
Mock<WorkflowAgentProvider> mockProvider = CreateMockProvider();
bool configureInvoked = false;
DeclarativeWorkflowOptions options = new(mockProvider.Object)
{
ConfigureTelemetry = opt =>
{
configureInvoked = true;
opt.EnableSensitiveData = true;
},
LoggerFactory = NullLoggerFactory.Instance
};
// Act
using StringReader reader = new(SimpleWorkflowYaml);
Workflow workflow = DeclarativeWorkflowBuilder.Build<string>(reader, options);
await using Run run = await InProcessExecution.RunAsync(workflow, "test input");
// Assert
Assert.True(configureInvoked);
Activity[] capturedActivities = this._capturedActivities
.Where(a => a.RootId == testActivity.RootId && a.Source.Name == DefaultTelemetrySourceName)
.ToArray();
Assert.NotEmpty(capturedActivities);
Assert.Contains(capturedActivities, a => a.OperationName.StartsWith(WorkflowBuildActivityName, StringComparison.Ordinal));
Assert.Contains(capturedActivities, a => a.OperationName.StartsWith(WorkflowRunActivityName, StringComparison.Ordinal));
}
[Fact]
public async Task BuildWorkflow_WithoutTelemetry_DoesNotCreateActivitiesAsync()
{
// Arrange
using Activity testActivity = new Activity("NoTelemetryTest").Start()!;
Mock<WorkflowAgentProvider> mockProvider = CreateMockProvider();
DeclarativeWorkflowOptions options = new(mockProvider.Object)
{
LoggerFactory = NullLoggerFactory.Instance
};
// Act
using StringReader reader = new(SimpleWorkflowYaml);
Workflow workflow = DeclarativeWorkflowBuilder.Build<string>(reader, options);
await using Run run = await InProcessExecution.RunAsync(workflow, "test input");
// Assert - No workflow activities should be created when telemetry is disabled
Activity[] capturedActivities = this._capturedActivities
.Where(a => a.RootId == testActivity.RootId &&
(a.OperationName.StartsWith(WorkflowBuildActivityName, StringComparison.Ordinal) ||
a.OperationName.StartsWith(WorkflowRunActivityName, StringComparison.Ordinal)))
.ToArray();
Assert.Empty(capturedActivities);
}
private static Mock<WorkflowAgentProvider> CreateMockProvider()
{
Mock<WorkflowAgentProvider> mockAgentProvider = new(MockBehavior.Strict);
mockAgentProvider
.Setup(provider => provider.CreateConversationAsync(It.IsAny<CancellationToken>()))
.Returns(() => Task.FromResult(Guid.NewGuid().ToString("N")));
mockAgentProvider
.Setup(provider => provider.CreateMessageAsync(It.IsAny<string>(), It.IsAny<ChatMessage>(), It.IsAny<CancellationToken>()))
.Returns(Task.FromResult(new ChatMessage(ChatRole.Assistant, "Test response")));
return mockAgentProvider;
}
}
@@ -67,7 +67,7 @@ public sealed class ObservabilityTests : IDisposable
WorkflowBuilder builder = new(uppercase);
builder.AddEdge(uppercase, reverse).WithOutputFrom(reverse);
return builder.Build();
return builder.WithOpenTelemetry().Build();
}
private static Dictionary<string, int> GetExpectedActivityNameCounts() =>
@@ -111,8 +111,6 @@ public sealed class ObservabilityTests : IDisposable
Run run = await executionEnvironment.RunAsync(workflow, "Hello, World!");
await run.DisposeAsync();
await Task.Delay(100); // Allow time for activities to be captured
// Assert
var capturedActivities = this._capturedActivities.Where(a => a.RootId == testActivity.RootId).ToList();
capturedActivities.Should().HaveCount(8, "Exactly 8 activities should be created.");
@@ -122,12 +120,12 @@ public sealed class ObservabilityTests : IDisposable
{
var activityName = kvp.Key;
var expectedCount = kvp.Value;
var actualCount = capturedActivities.Count(a => a.OperationName == activityName);
var actualCount = capturedActivities.Count(a => a.OperationName.StartsWith(activityName, StringComparison.Ordinal));
actualCount.Should().Be(expectedCount, $"Activity '{activityName}' should occur {expectedCount} times.");
}
// Verify WorkflowRun activity events include workflow lifecycle events
var workflowRunActivity = capturedActivities.First(a => a.OperationName == ActivityNames.WorkflowRun);
var workflowRunActivity = capturedActivities.First(a => a.OperationName.StartsWith(ActivityNames.WorkflowRun, StringComparison.Ordinal));
var activityEvents = workflowRunActivity.Events.ToList();
activityEvents.Should().Contain(e => e.Name == EventNames.WorkflowStarted, "activity should have workflow started event");
activityEvents.Should().Contain(e => e.Name == EventNames.WorkflowCompleted, "activity should have workflow completed event");
@@ -166,8 +164,6 @@ public sealed class ObservabilityTests : IDisposable
// Act
CreateWorkflow();
await Task.Delay(100); // Allow time for activities to be captured
// Assert
var capturedActivities = this._capturedActivities.Where(a => a.RootId == testActivity.RootId).ToList();
capturedActivities.Should().HaveCount(1, "Exactly 1 activity should be created.");
@@ -183,4 +179,325 @@ public sealed class ObservabilityTests : IDisposable
tags.Should().ContainKey(Tags.WorkflowId);
tags.Should().ContainKey(Tags.WorkflowDefinition);
}
[Fact]
public async Task TelemetryDisabledByDefault_CreatesNoActivitiesAsync()
{
// Arrange
// Create a test activity to correlate captured activities
using var testActivity = new Activity("ObservabilityTest").Start();
// Act - Build workflow WITHOUT calling WithOpenTelemetry()
Func<string, string> uppercaseFunc = s => s.ToUpperInvariant();
var uppercase = uppercaseFunc.BindAsExecutor("UppercaseExecutor");
WorkflowBuilder builder = new(uppercase);
builder.Build(); // No WithOpenTelemetry() call
// Assert - No activities should be created
var capturedActivities = this._capturedActivities.Where(a => a.RootId == testActivity.RootId).ToList();
capturedActivities.Should().BeEmpty("No activities should be created when telemetry is disabled (default).");
}
[Fact]
public async Task WithOpenTelemetry_UsesProvidedActivitySourceAsync()
{
// Arrange
using var testActivity = new Activity("ObservabilityTest").Start();
using var userActivitySource = new ActivitySource("UserProvidedSource");
// Set up a separate listener for the user-provided source
ConcurrentBag<Activity> userActivities = [];
using var userListener = new ActivityListener
{
ShouldListenTo = source => source.Name == "UserProvidedSource",
Sample = (ref ActivityCreationOptions<ActivityContext> options) => ActivitySamplingResult.AllData,
ActivityStarted = activity => userActivities.Add(activity),
};
ActivitySource.AddActivityListener(userListener);
Func<string, string> uppercaseFunc = s => s.ToUpperInvariant();
var uppercase = uppercaseFunc.BindAsExecutor("UppercaseExecutor");
// Act
WorkflowBuilder builder = new(uppercase);
var workflow = builder.WithOpenTelemetry(activitySource: userActivitySource).Build();
Run run = await InProcessExecution.Default.RunAsync(workflow, "Hello");
await run.DisposeAsync();
// Assert
var capturedActivities = userActivities.Where(a => a.RootId == testActivity.RootId).ToList();
capturedActivities.Should().NotBeEmpty("Activities should be created with user-provided ActivitySource.");
capturedActivities.Should().OnlyContain(
a => a.Source.Name == "UserProvidedSource",
"All activities should come from the user-provided ActivitySource.");
}
[Fact]
public async Task DisableWorkflowBuild_PreventsWorkflowBuildActivityAsync()
{
// Arrange
using var testActivity = new Activity("ObservabilityTest").Start();
Func<string, string> uppercaseFunc = s => s.ToUpperInvariant();
var uppercase = uppercaseFunc.BindAsExecutor("UppercaseExecutor");
// Act
WorkflowBuilder builder = new(uppercase);
builder.WithOpenTelemetry(configure: opts => opts.DisableWorkflowBuild = true).Build();
// Assert
var capturedActivities = this._capturedActivities.Where(a => a.RootId == testActivity.RootId).ToList();
capturedActivities.Should().NotContain(
a => a.OperationName.StartsWith(ActivityNames.WorkflowBuild, StringComparison.Ordinal),
"WorkflowBuild activity should be disabled.");
}
[Fact]
public async Task DisableWorkflowRun_PreventsWorkflowRunActivityAsync()
{
// Arrange
using var testActivity = new Activity("ObservabilityTest").Start();
Func<string, string> uppercaseFunc = s => s.ToUpperInvariant();
var uppercase = uppercaseFunc.BindAsExecutor("UppercaseExecutor");
// Act
WorkflowBuilder builder = new(uppercase);
builder.WithOutputFrom(uppercase);
var workflow = builder.WithOpenTelemetry(configure: opts => opts.DisableWorkflowRun = true).Build();
Run run = await InProcessExecution.Default.RunAsync(workflow, "Hello");
await run.DisposeAsync();
// Assert
var capturedActivities = this._capturedActivities.Where(a => a.RootId == testActivity.RootId).ToList();
capturedActivities.Should().NotContain(
a => a.OperationName.StartsWith(ActivityNames.WorkflowRun, StringComparison.Ordinal),
"WorkflowRun activity should be disabled.");
capturedActivities.Should().Contain(
a => a.OperationName.StartsWith(ActivityNames.WorkflowBuild, StringComparison.Ordinal),
"Other activities should still be created.");
}
[Fact]
public async Task DisableExecutorProcess_PreventsExecutorProcessActivityAsync()
{
// Arrange
using var testActivity = new Activity("ObservabilityTest").Start();
Func<string, string> uppercaseFunc = s => s.ToUpperInvariant();
var uppercase = uppercaseFunc.BindAsExecutor("UppercaseExecutor");
// Act
WorkflowBuilder builder = new(uppercase);
builder.WithOutputFrom(uppercase);
var workflow = builder.WithOpenTelemetry(configure: opts => opts.DisableExecutorProcess = true).Build();
Run run = await InProcessExecution.Default.RunAsync(workflow, "Hello");
await run.DisposeAsync();
// Assert
var capturedActivities = this._capturedActivities.Where(a => a.RootId == testActivity.RootId).ToList();
capturedActivities.Should().NotContain(
a => a.OperationName.StartsWith(ActivityNames.ExecutorProcess, StringComparison.Ordinal),
"ExecutorProcess activity should be disabled.");
capturedActivities.Should().Contain(
a => a.OperationName.StartsWith(ActivityNames.WorkflowRun, StringComparison.Ordinal),
"Other activities should still be created.");
}
[Fact]
public async Task DisableEdgeGroupProcess_PreventsEdgeGroupProcessActivityAsync()
{
// Arrange
using var testActivity = new Activity("ObservabilityTest").Start();
var workflow = CreateWorkflowWithDisabledEdges();
// Act
Run run = await InProcessExecution.Default.RunAsync(workflow, "Hello");
await run.DisposeAsync();
// Assert
var capturedActivities = this._capturedActivities.Where(a => a.RootId == testActivity.RootId).ToList();
capturedActivities.Should().NotContain(
a => a.OperationName.StartsWith(ActivityNames.EdgeGroupProcess, StringComparison.Ordinal),
"EdgeGroupProcess activity should be disabled.");
capturedActivities.Should().Contain(
a => a.OperationName.StartsWith(ActivityNames.ExecutorProcess, StringComparison.Ordinal),
"Other activities should still be created.");
}
[Fact]
public async Task DisableMessageSend_PreventsMessageSendActivityAsync()
{
// Arrange
using var testActivity = new Activity("ObservabilityTest").Start();
var workflow = CreateWorkflowWithDisabledMessages();
// Act
Run run = await InProcessExecution.Default.RunAsync(workflow, "Hello");
await run.DisposeAsync();
// Assert
var capturedActivities = this._capturedActivities.Where(a => a.RootId == testActivity.RootId).ToList();
capturedActivities.Should().NotContain(
a => a.OperationName.StartsWith(ActivityNames.MessageSend, StringComparison.Ordinal),
"MessageSend activity should be disabled.");
capturedActivities.Should().Contain(
a => a.OperationName.StartsWith(ActivityNames.ExecutorProcess, StringComparison.Ordinal),
"Other activities should still be created.");
}
private static Workflow CreateWorkflowWithDisabledEdges()
{
Func<string, string> uppercaseFunc = s => s.ToUpperInvariant();
var uppercase = uppercaseFunc.BindAsExecutor("UppercaseExecutor");
Func<string, string> reverseFunc = s => new string(s.Reverse().ToArray());
var reverse = reverseFunc.BindAsExecutor("ReverseTextExecutor");
WorkflowBuilder builder = new(uppercase);
builder.AddEdge(uppercase, reverse).WithOutputFrom(reverse);
return builder.WithOpenTelemetry(configure: opts => opts.DisableEdgeGroupProcess = true).Build();
}
private static Workflow CreateWorkflowWithDisabledMessages()
{
Func<string, string> uppercaseFunc = s => s.ToUpperInvariant();
var uppercase = uppercaseFunc.BindAsExecutor("UppercaseExecutor");
Func<string, string> reverseFunc = s => new string(s.Reverse().ToArray());
var reverse = reverseFunc.BindAsExecutor("ReverseTextExecutor");
WorkflowBuilder builder = new(uppercase);
builder.AddEdge(uppercase, reverse).WithOutputFrom(reverse);
return builder.WithOpenTelemetry(configure: opts => opts.DisableMessageSend = true).Build();
}
[Fact]
public async Task EnableSensitiveData_LogsExecutorInputAndOutputAsync()
{
// Arrange
using var testActivity = new Activity("ObservabilityTest").Start();
Func<string, string> uppercaseFunc = s => s.ToUpperInvariant();
var uppercase = uppercaseFunc.BindAsExecutor("UppercaseExecutor");
// Act
WorkflowBuilder builder = new(uppercase);
builder.WithOutputFrom(uppercase);
var workflow = builder.WithOpenTelemetry(configure: opts => opts.EnableSensitiveData = true).Build();
Run run = await InProcessExecution.Default.RunAsync(workflow, "hello");
await run.DisposeAsync();
// Assert
var capturedActivities = this._capturedActivities.Where(a => a.RootId == testActivity.RootId).ToList();
var executorActivity = capturedActivities.FirstOrDefault(
a => a.OperationName.StartsWith(ActivityNames.ExecutorProcess, StringComparison.Ordinal));
executorActivity.Should().NotBeNull("ExecutorProcess activity should be created.");
var tags = executorActivity!.Tags.ToDictionary(t => t.Key, t => t.Value);
tags.Should().ContainKey(Tags.ExecutorInput, "Input should be logged when EnableSensitiveData is true.");
tags.Should().ContainKey(Tags.ExecutorOutput, "Output should be logged when EnableSensitiveData is true.");
tags[Tags.ExecutorInput].Should().Contain("hello", "Input should contain the input value.");
tags[Tags.ExecutorOutput].Should().Contain("HELLO", "Output should contain the transformed value.");
}
[Fact]
public async Task EnableSensitiveData_Disabled_DoesNotLogInputOutputAsync()
{
// Arrange
using var testActivity = new Activity("ObservabilityTest").Start();
Func<string, string> uppercaseFunc = s => s.ToUpperInvariant();
var uppercase = uppercaseFunc.BindAsExecutor("UppercaseExecutor");
// Act - EnableSensitiveData is false by default
WorkflowBuilder builder = new(uppercase);
builder.WithOutputFrom(uppercase);
var workflow = builder.WithOpenTelemetry().Build();
Run run = await InProcessExecution.Default.RunAsync(workflow, "hello");
await run.DisposeAsync();
// Assert
var capturedActivities = this._capturedActivities.Where(a => a.RootId == testActivity.RootId).ToList();
var executorActivity = capturedActivities.FirstOrDefault(
a => a.OperationName.StartsWith(ActivityNames.ExecutorProcess, StringComparison.Ordinal));
executorActivity.Should().NotBeNull("ExecutorProcess activity should be created.");
var tags = executorActivity!.Tags.ToDictionary(t => t.Key, t => t.Value);
tags.Should().NotContainKey(Tags.ExecutorInput, "Input should NOT be logged when EnableSensitiveData is false.");
tags.Should().NotContainKey(Tags.ExecutorOutput, "Output should NOT be logged when EnableSensitiveData is false.");
}
[Fact]
public async Task EnableSensitiveData_LogsMessageSendContentAsync()
{
// Arrange
using var testActivity = new Activity("ObservabilityTest").Start();
Func<string, string> uppercaseFunc = s => s.ToUpperInvariant();
var uppercase = uppercaseFunc.BindAsExecutor("UppercaseExecutor");
Func<string, string> reverseFunc = s => new string(s.Reverse().ToArray());
var reverse = reverseFunc.BindAsExecutor("ReverseTextExecutor");
// Act
WorkflowBuilder builder = new(uppercase);
builder.AddEdge(uppercase, reverse).WithOutputFrom(reverse);
var workflow = builder.WithOpenTelemetry(configure: opts => opts.EnableSensitiveData = true).Build();
Run run = await InProcessExecution.Default.RunAsync(workflow, "hello");
await run.DisposeAsync();
// Assert
var capturedActivities = this._capturedActivities.Where(a => a.RootId == testActivity.RootId).ToList();
var messageSendActivity = capturedActivities.FirstOrDefault(
a => a.OperationName.StartsWith(ActivityNames.MessageSend, StringComparison.Ordinal));
messageSendActivity.Should().NotBeNull("MessageSend activity should be created.");
var tags = messageSendActivity!.Tags.ToDictionary(t => t.Key, t => t.Value);
tags.Should().ContainKey(Tags.MessageContent, "Message content should be logged when EnableSensitiveData is true.");
tags.Should().ContainKey(Tags.MessageSourceId, "Source ID should be logged.");
}
[Fact]
public async Task EnableSensitiveData_Disabled_DoesNotLogMessageContentAsync()
{
// Arrange
using var testActivity = new Activity("ObservabilityTest").Start();
Func<string, string> uppercaseFunc = s => s.ToUpperInvariant();
var uppercase = uppercaseFunc.BindAsExecutor("UppercaseExecutor");
Func<string, string> reverseFunc = s => new string(s.Reverse().ToArray());
var reverse = reverseFunc.BindAsExecutor("ReverseTextExecutor");
// Act - EnableSensitiveData is false by default
WorkflowBuilder builder = new(uppercase);
builder.AddEdge(uppercase, reverse).WithOutputFrom(reverse);
var workflow = builder.WithOpenTelemetry().Build();
Run run = await InProcessExecution.Default.RunAsync(workflow, "hello");
await run.DisposeAsync();
// Assert
var capturedActivities = this._capturedActivities.Where(a => a.RootId == testActivity.RootId).ToList();
var messageSendActivity = capturedActivities.FirstOrDefault(
a => a.OperationName.StartsWith(ActivityNames.MessageSend, StringComparison.Ordinal));
messageSendActivity.Should().NotBeNull("MessageSend activity should be created.");
var tags = messageSendActivity!.Tags.ToDictionary(t => t.Key, t => t.Value);
tags.Should().NotContainKey(Tags.MessageContent, "Message content should NOT be logged when EnableSensitiveData is false.");
tags.Should().ContainKey(Tags.MessageSourceId, "Source ID should still be logged.");
}
}
@@ -6,6 +6,7 @@ using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Agents.AI.Workflows.Execution;
using Microsoft.Agents.AI.Workflows.Observability;
namespace Microsoft.Agents.AI.Workflows.UnitTests;
@@ -133,6 +134,8 @@ public class TestRunContext : IRunnerContext
public bool WithCheckpointing => false;
public bool ConcurrentRunsEnabled => false;
WorkflowTelemetryContext IRunnerContext.TelemetryContext => WorkflowTelemetryContext.Disabled;
ValueTask<Executor> IRunnerContext.EnsureExecutorAsync(string executorId, IStepTracer? tracer, CancellationToken cancellationToken) =>
new(this.Executors[executorId]);