Python: Tracing for workflows (#480)

* workflow tracing design doc * add tracing implementation for workflow * fix bug caused by double wrapping of sub workflow request * add unit tests for tracing * add documentation for workflow tracing * remove unnecessary file * update aspire command * fix tests * proper serialization of subworkflows and add workflow.definition * add serialization test * fix subworkflow serialization * workflow_id --> id * update workflow sample to address comments * update naming; use costant * use NoOpTracer instead of nullcontext * use span event instead of attribtutes for status * fix typing * add workflow.build span * rename methods for clarity * ensure all source trace contexts are propagated in fan in
2026-06-16 21:04:09 +08:00 · 2025-08-26 22:09:16 -07:00
parent 78125f019a
commit 379e3b9a00
14 changed files with 1533 additions and 140 deletions
@@ -2,3 +2,4 @@ CONNECTION_STRING="..."
 OTLP_ENDPOINT="http://localhost:4317/"
 AGENT_FRAMEWORK_GENAI_ENABLE_OTEL_DIAGNOSTICS=true
 AGENT_FRAMEWORK_GENAI_ENABLE_OTEL_DIAGNOSTICS_SENSITIVE=true
+AGENT_FRAMEWORK_WORKFLOW_ENABLE_OTEL_DIAGNOSTICS=true
@@ -4,9 +4,12 @@ This sample project shows how a Python application can be configured to send Age

 In this sample, we provide options to send telemetry to [Application Insights](https://learn.microsoft.com/en-us/azure/azure-monitor/app/app-insights-overview), [Aspire Dashboard](https://learn.microsoft.com/en-us/dotnet/aspire/fundamentals/dashboard/overview?tabs=bash), and console output.

+> **Quick Start**: For local development without Azure setup, you can use the [Aspire Dashboard](https://learn.microsoft.com/en-us/dotnet/aspire/fundamentals/dashboard/standalone) which runs locally via Docker and provides an excellent telemetry viewing experience for OpenTelemetry data.
+
 > Note that it is also possible to use other Application Performance Management (APM) vendors. An example is [Prometheus](https://prometheus.io/docs/introduction/overview/). Please refer to this [link](https://opentelemetry.io/docs/languages/python/exporters/) to learn more about exporters.

 For more information, please refer to the following resources:
+
 1. [Azure Monitor OpenTelemetry Exporter](https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/monitor/azure-monitor-opentelemetry-exporter)
 2. [Aspire Dashboard for Python Apps](https://learn.microsoft.com/en-us/dotnet/aspire/fundamentals/dashboard/standalone-for-python?tabs=flask%2Cwindows)
 3. [Python Logging](https://docs.python.org/3/library/logging.html)
@@ -19,13 +22,18 @@ The Agent Framework Python SDK is designed to efficiently generate comprehensive
 ## Configuration

 ### Required resources
+
 2. OpenAI or [Azure OpenAI](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal)
+
 ### Optional resources
+
 1. [Application Insights](https://learn.microsoft.com/en-us/azure/azure-monitor/app/create-workspace-resource)
 2. [Aspire Dashboard](https://learn.microsoft.com/en-us/dotnet/aspire/fundamentals/dashboard/standalone-for-python?tabs=flask%2Cwindows#start-the-aspire-dashboard)

 ### Dependencies
+
 You will also need to install the following dependencies to your virtual environment to run this sample:
+
 ```bash
 # For Azure ApplicationInsights/AzureMonitor
 uv pip install azure-monitor-opentelemetry azure-monitor-opentelemetry-exporter
@@ -39,16 +47,18 @@ uv pip install opentelemetry-exporter-otlp-proto-grpc
 2. Create a `.env` file if one doesn't already exist in this folder. Please refer to the [example file](./.env.example).
    > Note that `CONNECTION_STRING` and `SAMPLE_OTLP_ENDPOINT` are optional. If you don't configure them, everything will get outputted to the console.
    > Set `AGENT_FRAMEWORK_GENAI_ENABLE_OTEL_DIAGNOSTICS=true` to enable basic telemetry and `AGENT_FRAMEWORK_GENAI_ENABLE_OTEL_DIAGNOSTICS_SENSITIVE=true` to include sensitive information like prompts and responses.
+    > Set `AGENT_FRAMEWORK_WORKFLOW_ENABLE_OTEL_DIAGNOSTICS=true` to enable workflow telemetry for the workflow samples.
        > Sensitive information should only be enabled in a development or test environment. It is not recommended to enable this in production environments as it may expose sensitive data.
-3. Activate your python virtual environment, and then run `python scenarios.py` or `python interactive.py`.
+3. Activate your python virtual environment, and then run `python scenarios.py`, `python interactive.py`, `python agent.py`, or `python workflow.py`.

 > This will output the Operation/Trace ID, which can be used later for filtering.

 ### Scenarios

-This sample includes two different applications demonstrating Agent Framework telemetry:
+This sample includes multiple applications demonstrating Agent Framework telemetry:

 #### scenarios.py
+
 Organized into specific scenarios where the framework will generate useful telemetry data:

 - `chat_client`: This is when a chat client is invoked directly (i.e. not streaming) with a weather tool function. **Information about the call to the underlying model and tool usage will be recorded**.
@@ -58,8 +68,24 @@ Organized into specific scenarios where the framework will generate useful telem
 By default, running `python scenarios.py` will run all three scenarios. To run individual scenarios, use the `--scenario` command line argument. For example, `python scenarios.py --scenario chat_client`. For more information, please run `python scenarios.py -h`.

 #### interactive.py
+
 An interactive chat application that demonstrates telemetry collection in a conversational context. This sample includes the same `get_weather` tool function and allows for multi-turn conversations. Run `python interactive.py` and start chatting. Type 'exit' to quit the application. This sample only logs at the `WARNING` level, so you will not see as much telemetry data as in the `scenarios.py` sample.

+#### agent.py
+
+A sample demonstrating Agent Framework telemetry collection for agent-based workflows. This shows how telemetry is captured when using the Agent Framework's agent abstraction layer, including agent initialization, message processing, and tool execution within an agent context.
+
+By default, running `python agent.py` will run all agent scenarios. To run individual scenarios, use the `--scenario` command line argument. For example, `python agent.py --scenario basic`. For more information, please run `python agent.py -h`.
+
+#### workflow.py
+
+A sample demonstrating workflow telemetry collection for the Agent Framework's workflow execution engine. This includes two scenarios:
+
+- `sequential`: A simple sequential workflow that processes text through two connected executors (uppercase conversion followed by text reversal). **Information about workflow execution, executor processing, and message passing between executors will be recorded**.
+- `sub_workflow`: A more complex scenario demonstrating sub-workflow patterns with a parent workflow orchestrating multiple text processing tasks via sub-workflows. **Information about parent workflow execution, sub-workflow invocation, and cross-workflow communication will be recorded**.
+
+By default, running `python workflow.py` will run all workflow scenarios. To run individual scenarios, use the `--scenario` command line argument. For example, `python workflow.py --scenario sequential`. For more information, please run `python workflow.py -h`.
+
 ## Application Insights/Azure Monitor

 ### Logs and traces
@@ -100,15 +126,52 @@ dependencies

 ## Aspire Dashboard

+The [Aspire Dashboard](https://learn.microsoft.com/en-us/dotnet/aspire/fundamentals/dashboard/standalone) is a local telemetry viewing tool that provides an excellent experience for viewing OpenTelemetry data without requiring Azure setup.
+
+### Setting up Aspire Dashboard with Docker
+
+The easiest way to run the Aspire Dashboard locally is using Docker:
+
+```bash
+# Pull and run the Aspire Dashboard container
+docker run --rm -it -d \
+    -p 18888:18888 \
+    -p 4317:18889 \
+    --name aspire-dashboard \
+    mcr.microsoft.com/dotnet/aspire-dashboard:latest
+```
+
+This will start the dashboard with:
+
+- **Web UI**: Available at <http://localhost:18888>
+- **OTLP endpoint**: Available at `http://localhost:4317` for your applications to send telemetry data
+
+### Configuring your application
+
+Make sure your `.env` file includes the OTLP endpoint:
+
+```bash
+OTLP_ENDPOINT=http://localhost:4317
+```
+
+Or set it as an environment variable when running your samples:
+
+```bash
+OTLP_ENDPOINT=http://localhost:4317 python scenarios.py
+```
+
+### Viewing telemetry data
+
 > Make sure you have the dashboard running to receive telemetry data.

-Once the the sample finishes running, navigate to http://localhost:18888 in a web browser to see the telemetry data. Follow the instructions [here](https://learn.microsoft.com/en-us/dotnet/aspire/fundamentals/dashboard/explore) to authenticate to the dashboard and start exploring!
+Once your sample finishes running, navigate to <http://localhost:18888> in a web browser to see the telemetry data. Follow the [Aspire Dashboard exploration guide](https://learn.microsoft.com/en-us/dotnet/aspire/fundamentals/dashboard/explore) to authenticate to the dashboard and start exploring your traces, logs, and metrics!

 ## Console output

 You won't have to deploy an Application Insights resource or install Docker to run Aspire Dashboard if you choose to inspect telemetry data in a console. However, it is difficult to navigate through all the spans and logs produced, so **this method is only recommended when you are just getting started**.

 We recommend you to get started with the `chat_client` scenario as this generates the least amount of telemetry data. Below is similar to what you will see when you run `python scenarios.py --scenario chat_client`:
+
 ```Json
 {
    "name": "chat.completions gpt-4o",
@@ -0,0 +1,253 @@
+# Copyright (c) Microsoft. All rights reserved.
+# type: ignore
+import asyncio
+import logging
+from typing import Any
+
+from agent_framework.workflow import (
+    Executor,
+    WorkflowBuilder,
+    WorkflowCompletedEvent,
+    WorkflowContext,
+    handler,
+)
+from azure.monitor.opentelemetry import configure_azure_monitor
+from opentelemetry import trace
+from opentelemetry._logs import set_logger_provider
+from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter
+from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
+from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
+from opentelemetry.metrics import set_meter_provider
+from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
+from opentelemetry.sdk._logs.export import BatchLogRecordProcessor, ConsoleLogExporter
+from opentelemetry.sdk.metrics import MeterProvider
+from opentelemetry.sdk.metrics.export import ConsoleMetricExporter, PeriodicExportingMetricReader
+from opentelemetry.sdk.metrics.view import DropAggregation, View
+from opentelemetry.sdk.resources import Resource
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter
+from opentelemetry.semconv.attributes import service_attributes
+from opentelemetry.trace import SpanKind, set_tracer_provider
+from opentelemetry.trace.span import format_trace_id
+from pydantic_settings import BaseSettings
+
+"""Telemetry sample demonstrating OpenTelemetry integration with Agent Framework workflows.
+
+This sample runs a simple sequential workflow with telemetry collection,
+showing telemetry collection for workflow execution, executor processing,
+and message publishing between executors.
+"""
+
+
+class TelemetrySampleSettings(BaseSettings):
+    """Settings for the telemetry sample application.
+
+    Optional settings are:
+    - connection_string: str - The connection string for the Application Insights resource.
+                This value can be found in the Overview section when examining
+                your resource from the Azure portal.
+                (Env var CONNECTION_STRING)
+    - otlp_endpoint: str - The OTLP endpoint to send telemetry data to.
+                Depending on the exporter used, you may find this value in different places.
+                (Env var OTLP_ENDPOINT)
+
+    If no connection string or OTLP endpoint is provided, the telemetry data will be
+    exported to the console.
+    """
+
+    connection_string: str | None = None
+    otlp_endpoint: str | None = None
+
+
+# Load settings
+settings = TelemetrySampleSettings()
+
+# Create a resource to represent the service/sample
+resource = Resource.create({service_attributes.SERVICE_NAME: "WorkflowTelemetryExample"})
+
+if settings.connection_string:
+    configure_azure_monitor(
+        connection_string=settings.connection_string,
+        enable_live_metrics=True,
+        logger_name="agent_framework",
+    )
+
+
+def set_up_logging():
+    class LogFilter(logging.Filter):
+        """A filter to not process records from several subpackages."""
+
+        # These are the namespaces that we want to exclude from logging for the purposes of this demo.
+        namespaces_to_exclude: list[str] = [
+            "httpx",
+            "openai",
+        ]
+
+        def filter(self, record):
+            return not any([record.name.startswith(namespace) for namespace in self.namespaces_to_exclude])
+
+    exporters = []
+    if settings.otlp_endpoint:
+        exporters.append(OTLPLogExporter(endpoint=settings.otlp_endpoint))
+    if not exporters:
+        exporters.append(ConsoleLogExporter())
+
+    # Create and set a global logger provider for the application.
+    logger_provider = LoggerProvider(resource=resource)
+    # Log processors are initialized with an exporter which is responsible
+    # for sending the telemetry data to a particular backend.
+    for log_exporter in exporters:
+        logger_provider.add_log_record_processor(BatchLogRecordProcessor(log_exporter))
+    # Sets the global default logger provider
+    set_logger_provider(logger_provider)
+
+    # Create a logging handler to write logging records, in OTLP format, to the exporter.
+    handler = LoggingHandler()
+    handler.addFilter(LogFilter())
+    # Attach the handler to the root logger. `getLogger()` with no arguments returns the root logger.
+    # Events from all child loggers will be processed by this handler.
+    logger = logging.getLogger()
+    logger.addHandler(handler)
+    # Set the logging level to NOTSET to allow all records to be processed by the handler.
+    logger.setLevel(logging.NOTSET)
+
+
+def set_up_tracing():
+    exporters = []
+    if settings.otlp_endpoint:
+        exporters.append(OTLPSpanExporter(endpoint=settings.otlp_endpoint))
+    if not exporters:
+        exporters.append(ConsoleSpanExporter())
+
+    # Initialize a trace provider for the application. This is a factory for creating tracers.
+    tracer_provider = TracerProvider(resource=resource)
+    # Span processors are initialized with an exporter which is responsible
+    # for sending the telemetry data to a particular backend.
+    for exporter in exporters:
+        tracer_provider.add_span_processor(BatchSpanProcessor(exporter))
+    # Sets the global default tracer provider
+    set_tracer_provider(tracer_provider)
+
+
+def set_up_metrics():
+    exporters = []
+    if settings.otlp_endpoint:
+        exporters.append(OTLPMetricExporter(endpoint=settings.otlp_endpoint))
+    if not exporters:
+        exporters.append(ConsoleMetricExporter())
+
+    # Initialize a metric provider for the application. This is a factory for creating meters.
+    metric_readers = [
+        PeriodicExportingMetricReader(metric_exporter, export_interval_millis=5000) for metric_exporter in exporters
+    ]
+    meter_provider = MeterProvider(
+        metric_readers=metric_readers,
+        resource=resource,
+        views=[
+            # Dropping all instrument names except for those starting with "agent_framework"
+            View(instrument_name="*", aggregation=DropAggregation()),
+            View(instrument_name="agent_framework*"),
+        ],
+    )
+    # Sets the global default meter provider
+    set_meter_provider(meter_provider)
+
+
+# Executors for sequential workflow
+class UpperCaseExecutor(Executor):
+    """An executor that converts text to uppercase."""
+
+    @handler
+    async def to_upper_case(self, text: str, ctx: WorkflowContext[str]) -> None:
+        """Execute the task by converting the input string to uppercase."""
+        print(f"UpperCaseExecutor: Processing '{text}'")
+        result = text.upper()
+        print(f"UpperCaseExecutor: Result '{result}'")
+
+        # Send the result to the next executor in the workflow.
+        await ctx.send_message(result)
+
+
+class ReverseTextExecutor(Executor):
+    """An executor that reverses text."""
+
+    @handler
+    async def reverse_text(self, text: str, ctx: WorkflowContext[Any]) -> None:
+        """Execute the task by reversing the input string."""
+        print(f"ReverseTextExecutor: Processing '{text}'")
+        result = text[::-1]
+        print(f"ReverseTextExecutor: Result '{result}'")
+
+        # Send the result with a workflow completion event.
+        await ctx.add_event(WorkflowCompletedEvent(result))
+
+
+async def run_sequential_workflow() -> None:
+    """Run a simple sequential workflow demonstrating telemetry collection.
+
+    This workflow processes a string through two executors in sequence:
+    1. UpperCaseExecutor converts the input to uppercase
+    2. ReverseTextExecutor reverses the string and completes the workflow
+
+    Telemetry data collected includes:
+    - Overall workflow execution spans
+    - Individual executor processing spans
+    - Message publishing between executors
+    - Workflow completion events
+    """
+
+    tracer = trace.get_tracer(__name__)
+    with tracer.start_as_current_span("Scenario: Sequential Workflow", kind=SpanKind.CLIENT) as current_span:
+        print("Running scenario: Sequential Workflow")
+        try:
+            # Step 1: Create the executors.
+            upper_case_executor = UpperCaseExecutor(id="upper_case_executor")
+            reverse_text_executor = ReverseTextExecutor(id="reverse_text_executor")
+
+            # Step 2: Build the workflow with the defined edges.
+            workflow = (
+                WorkflowBuilder()
+                .add_edge(upper_case_executor, reverse_text_executor)
+                .set_start_executor(upper_case_executor)
+                .build()
+            )
+
+            # Step 3: Run the workflow with an initial message.
+            input_text = "hello world"
+            print(f"Starting workflow with input: '{input_text}'")
+
+            completion_event = None
+            async for event in workflow.run_streaming(input_text):
+                print(f"Event: {event}")
+                if isinstance(event, WorkflowCompletedEvent):
+                    # The WorkflowCompletedEvent contains the final result.
+                    completion_event = event
+
+            if completion_event:
+                print(f"Workflow completed with result: '{completion_event.data}'")
+            else:
+                print("Workflow completed without a completion event")
+
+        except Exception as e:
+            current_span.record_exception(e)
+            print(f"Error running workflow: {e}")
+
+
+async def main():
+    """Run the telemetry sample with a simple sequential workflow."""
+    # Set up the providers
+    # This must be done before any other telemetry calls
+    set_up_logging()
+    set_up_tracing()
+    set_up_metrics()
+
+    tracer = trace.get_tracer("agent_framework")
+    with tracer.start_as_current_span("Sequential Workflow Scenario", kind=SpanKind.CLIENT) as current_span:
+        print(f"Trace ID: {format_trace_id(current_span.get_span_context().trace_id)}")
+
+        # Run the sequential workflow scenario
+        await run_sequential_workflow()
+
+
+if __name__ == "__main__":
+    asyncio.run(main())