mirror of
https://github.com/microsoft/agent-framework.git
synced 2026-06-16 21:04:09 +08:00
Python: Improved telemetry setup (#421)
* test with stack and simplified names * quick demo of agent decorator * moved builder to protocol to enhance functionality * undid chatclientAgent -> agent rename * one more * reverted AIAgent rename * final reverts * fixed foundry import * revert changes * streamlined otel and fcc decorators * cleanup of telemetry * further refinement * lots of updates * fixed typing * fix for mypy * added input and output atttributes * fix import * initial work on baking in otel * major update to telemetry * final fixes after rename * fix * fix test * updated tests * fix for tests * fixes for tests * updated based on comments * removed agent decorator * fix for Python: ServiceResponseException when using multiple tools Fixes #649 * addressed comments * fix tests * fix tests * fix tools tests * fix for conversation_id in assistants client * fix responses test * fix tests and mypy * updated test * foundry fix --------- Co-authored-by: Chris <66376200+crickman@users.noreply.github.com>
This commit is contained in:
Eduard van Valkenburg
committed by
GitHub
Unverified
GitHub
parent
6aa746d891
commit
82ca4065cb
@@ -1,8 +1,8 @@
|
||||
# Agent Framework Python Telemetry
|
||||
|
||||
This sample project shows how a Python application can be configured to send Agent Framework telemetry to the Application Performance Management (APM) vendors of your choice.
|
||||
This sample folder shows how a Python application can be configured to send Agent Framework telemetry to the Application Performance Management (APM) vendors of your choice.
|
||||
|
||||
In this sample, we provide options to send telemetry to [Application Insights](https://learn.microsoft.com/en-us/azure/azure-monitor/app/app-insights-overview), [Aspire Dashboard](https://learn.microsoft.com/en-us/dotnet/aspire/fundamentals/dashboard/overview?tabs=bash), and console output.
|
||||
In this sample, we provide options to send telemetry to [Application Insights](https://learn.microsoft.com/en-us/azure/azure-monitor/app/app-insights-overview) and [Aspire Dashboard](https://learn.microsoft.com/en-us/dotnet/aspire/fundamentals/dashboard/overview?tabs=bash).
|
||||
|
||||
> **Quick Start**: For local development without Azure setup, you can use the [Aspire Dashboard](https://learn.microsoft.com/en-us/dotnet/aspire/fundamentals/dashboard/standalone) which runs locally via Docker and provides an excellent telemetry viewing experience for OpenTelemetry data.
|
||||
|
||||
@@ -24,6 +24,7 @@ The Agent Framework Python SDK is designed to efficiently generate comprehensive
|
||||
### Required resources
|
||||
|
||||
2. OpenAI or [Azure OpenAI](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal)
|
||||
2. [Foundry project](https://ai.azure.com/doc/azure/ai-foundry/what-is-azure-ai-foundry)
|
||||
|
||||
### Optional resources
|
||||
|
||||
@@ -31,61 +32,51 @@ The Agent Framework Python SDK is designed to efficiently generate comprehensive
|
||||
2. [Aspire Dashboard](https://learn.microsoft.com/en-us/dotnet/aspire/fundamentals/dashboard/standalone-for-python?tabs=flask%2Cwindows#start-the-aspire-dashboard)
|
||||
|
||||
### Dependencies
|
||||
No additional dependencies are required to enable telemetry. The necessary packages are included as part of the `agent-framework` package. Unless you want to use a different APM vendor, in which case you will need to install the appropriate OpenTelemetry exporter package.
|
||||
|
||||
You will also need to install the following dependencies to your virtual environment to run this sample:
|
||||
### Environment variables
|
||||
The following environment variables can be set to configure telemetry, the first two set the basic configuration:
|
||||
|
||||
```bash
|
||||
# For Azure ApplicationInsights/AzureMonitor
|
||||
uv pip install azure-monitor-opentelemetry azure-monitor-opentelemetry-exporter
|
||||
# For OTLP endpoint
|
||||
uv pip install opentelemetry-exporter-otlp-proto-grpc
|
||||
```
|
||||
- AGENT_FRAMEWORK_ENABLE_OTEL=true
|
||||
- AGENT_FRAMEWORK_ENABLE_SENSITIVE_DATA=true
|
||||
|
||||
## Running the sample
|
||||
Next we need to know where to send the telemetry, for that you can use either a OTLP endpoint or a connection string for Application Insights:
|
||||
- AGENT_FRAMEWORK_OTLP_ENDPOINT="<url to OTLP endpoint>"
|
||||
or
|
||||
- AGENT_FRAMEWORK_MONITOR_CONNECTION_STRING="<connection string>"
|
||||
Finally, you can enable live metrics streaming to Application Insights:
|
||||
- AGENT_FRAMEWORK_MONITOR_LIVE_METRICS=true
|
||||
|
||||
> IMPORTANT - If both OTLP endpoint and connection string are set, the connection string will take precedence and there will be no trace to the OTLP endpoint.
|
||||
|
||||
## Samples
|
||||
This folder contains different samples demonstrating how to use telemetry in various scenarios.
|
||||
|
||||
### [01 - zero_code](./01-zero_code.py):
|
||||
A simple example showing how to enable telemetry in a zero-touch scenario. When the above environment variables are set, telemetry will be automatically enabled, however since you do not define any overarching tracer, you will only see the spans for the specific calls to the chat client and tools.
|
||||
|
||||
### [02a](./02a-generic_chat_client.py) and [02b](./02b-foundry_chat_client.py) Chat Clients:
|
||||
These two samples show how to first setup the telemetry by manually importing the `setup_telemetry` function from the `agent_framework.telemetry` module and calling it. After this is done, the trace that get's created will live in the same context as the chat client calls, allowing you to see the end-to-end flow of your application. For Foundry, there is a method in the Foundry project client to get the telemetry url for your project, the `.setup_foundry_telemetry()` method in the `FoundryChatClient` class will use this url to configure telemetry and you then do not have to import and call `setup_telemetry()` manually.
|
||||
Because of the way OpenTelemetry works, you can only call `setup_telemetry()` once per application run, so make sure you do that in the right place.
|
||||
|
||||
### [03a](./03a-generic_agent.py) and [03b](./03b-foundry_agent.py) Agents:
|
||||
These two samples show how to setup telemetry when using the Agent Framework's agent abstraction layer. They are similar to the chat client samples, but also show how to create an agent and invoke it. The same rules apply for setting up telemetry, you can either call `setup_telemetry()` manually, or use the `setup_foundry_telemetry()` method in the `FoundryChatClient` class.
|
||||
|
||||
### [04 - workflow](./04-workflow.py) Workflow:
|
||||
This sample shows how to setup telemetry when using the Agent Framework's workflow execution engine. It demonstrates a simple workflow scenario with telemetry.
|
||||
|
||||
|
||||
## Running the samples
|
||||
|
||||
1. Open a terminal and navigate to this folder: `python/samples/getting_started/telemetry/`. This is necessary for the `.env` file to be read correctly.
|
||||
2. Create a `.env` file if one doesn't already exist in this folder. Please refer to the [example file](./.env.example).
|
||||
> Note that `CONNECTION_STRING` and `SAMPLE_OTLP_ENDPOINT` are optional. If you don't configure them, everything will get outputted to the console.
|
||||
> Set `AGENT_FRAMEWORK_GENAI_ENABLE_OTEL_DIAGNOSTICS=true` to enable basic telemetry and `AGENT_FRAMEWORK_GENAI_ENABLE_OTEL_DIAGNOSTICS_SENSITIVE=true` to include sensitive information like prompts and responses.
|
||||
> Set `AGENT_FRAMEWORK_WORKFLOW_ENABLE_OTEL_DIAGNOSTICS=true` to enable workflow telemetry for the workflow samples.
|
||||
> Set `AGENT_FRAMEWORK_ENABLE_OTEL=true` to enable basic telemetry and `AGENT_FRAMEWORK_ENABLE_SENSITIVE_DATA=true` to include sensitive information like prompts and responses.
|
||||
> Sensitive information should only be enabled in a development or test environment. It is not recommended to enable this in production environments as it may expose sensitive data.
|
||||
3. Activate your python virtual environment, and then run `python scenarios.py`, `python interactive.py`, `python agent.py`, or `python workflow.py`.
|
||||
|
||||
> Set `AGENT_FRAMEWORK_WORKFLOW_ENABLE_OTEL=true` to enable workflow telemetry for the workflow samples.
|
||||
3. Activate your python virtual environment, and then run `python 01-zero_code.py` or others.
|
||||
> This will output the Operation/Trace ID, which can be used later for filtering.
|
||||
|
||||
### Scenarios
|
||||
|
||||
This sample includes multiple applications demonstrating Agent Framework telemetry:
|
||||
|
||||
#### scenarios.py
|
||||
|
||||
Organized into specific scenarios where the framework will generate useful telemetry data:
|
||||
|
||||
- `chat_client`: This is when a chat client is invoked directly (i.e. not streaming) with a weather tool function. **Information about the call to the underlying model and tool usage will be recorded**.
|
||||
- `chat_client_stream`: This is when a chat client is invoked with streaming enabled and a weather tool function. **Information about the streaming call to the underlying model and tool usage will be recorded**.
|
||||
- `ai_function`: This is when an AI function (`get_weather`) is invoked directly. **Information about the AI function and the call to the underlying model will be recorded**.
|
||||
|
||||
By default, running `python scenarios.py` will run all three scenarios. To run individual scenarios, use the `--scenario` command line argument. For example, `python scenarios.py --scenario chat_client`. For more information, please run `python scenarios.py -h`.
|
||||
|
||||
#### interactive.py
|
||||
|
||||
An interactive chat application that demonstrates telemetry collection in a conversational context. This sample includes the same `get_weather` tool function and allows for multi-turn conversations. Run `python interactive.py` and start chatting. Type 'exit' to quit the application. This sample only logs at the `WARNING` level, so you will not see as much telemetry data as in the `scenarios.py` sample.
|
||||
|
||||
#### agent.py
|
||||
|
||||
A sample demonstrating Agent Framework telemetry collection for agent-based workflows. This shows how telemetry is captured when using the Agent Framework's agent abstraction layer, including agent initialization, message processing, and tool execution within an agent context.
|
||||
|
||||
By default, running `python agent.py` will run all agent scenarios. To run individual scenarios, use the `--scenario` command line argument. For example, `python agent.py --scenario basic`. For more information, please run `python agent.py -h`.
|
||||
|
||||
#### workflow.py
|
||||
|
||||
A sample demonstrating workflow telemetry collection for the Agent Framework's workflow execution engine. This includes two scenarios:
|
||||
|
||||
- `sequential`: A simple sequential workflow that processes text through two connected executors (uppercase conversion followed by text reversal). **Information about workflow execution, executor processing, and message passing between executors will be recorded**.
|
||||
- `sub_workflow`: A more complex scenario demonstrating sub-workflow patterns with a parent workflow orchestrating multiple text processing tasks via sub-workflows. **Information about parent workflow execution, sub-workflow invocation, and cross-workflow communication will be recorded**.
|
||||
|
||||
By default, running `python workflow.py` will run all workflow scenarios. To run individual scenarios, use the `--scenario` command line argument. For example, `python workflow.py --scenario sequential`. For more information, please run `python workflow.py -h`.
|
||||
|
||||
## Application Insights/Azure Monitor
|
||||
|
||||
### Logs and traces
|
||||
@@ -151,13 +142,13 @@ This will start the dashboard with:
|
||||
Make sure your `.env` file includes the OTLP endpoint:
|
||||
|
||||
```bash
|
||||
OTLP_ENDPOINT=http://localhost:4317
|
||||
AGENT_FRAMEWORK_OTLP_ENDPOINT=http://localhost:4317
|
||||
```
|
||||
|
||||
Or set it as an environment variable when running your samples:
|
||||
|
||||
```bash
|
||||
OTLP_ENDPOINT=http://localhost:4317 python scenarios.py
|
||||
AGENT_FRAMEWORK_ENABLE_OTEL=true AGENT_FRAMEWORK_OTLP_ENDPOINT=http://localhost:4317 python 01-zero_code.py
|
||||
```
|
||||
|
||||
### Viewing telemetry data
|
||||
@@ -170,138 +161,4 @@ Once your sample finishes running, navigate to <http://localhost:18888> in a web
|
||||
|
||||
You won't have to deploy an Application Insights resource or install Docker to run Aspire Dashboard if you choose to inspect telemetry data in a console. However, it is difficult to navigate through all the spans and logs produced, so **this method is only recommended when you are just getting started**.
|
||||
|
||||
We recommend you to get started with the `chat_client` scenario as this generates the least amount of telemetry data. Below is similar to what you will see when you run `python scenarios.py --scenario chat_client`:
|
||||
|
||||
```Json
|
||||
{
|
||||
"name": "chat.completions gpt-4o",
|
||||
"context": {
|
||||
"trace_id": "0xbda1d9efcd65435653d18fa37aef7dd3",
|
||||
"span_id": "0xcd443e1917510385",
|
||||
"trace_state": "[]"
|
||||
},
|
||||
"kind": "SpanKind.INTERNAL",
|
||||
"parent_id": "0xeca0a2ca7b7a8191",
|
||||
"start_time": "2024-09-09T23:13:14.625156Z",
|
||||
"end_time": "2024-09-09T23:13:17.311909Z",
|
||||
"status": {
|
||||
"status_code": "UNSET"
|
||||
},
|
||||
"attributes": {
|
||||
"gen_ai.operation.name": "chat.completions",
|
||||
"gen_ai.system": "openai",
|
||||
"gen_ai.request.model": "gpt-4o",
|
||||
"gen_ai.response.id": "chatcmpl-A5hrG13nhtFsOgx4ziuoskjNscHtT",
|
||||
"gen_ai.response.finish_reason": "FinishReason.STOP",
|
||||
"gen_ai.response.prompt_tokens": 16,
|
||||
"gen_ai.response.completion_tokens": 28
|
||||
},
|
||||
"events": [
|
||||
{
|
||||
"name": "gen_ai.content.prompt",
|
||||
"timestamp": "2024-09-09T23:13:14.625156Z",
|
||||
"attributes": {
|
||||
"gen_ai.prompt": "[{\"role\": \"user\", \"content\": \"Why is the sky blue in one sentence?\"}]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "gen_ai.content.completion",
|
||||
"timestamp": "2024-09-09T23:13:17.311909Z",
|
||||
"attributes": {
|
||||
"gen_ai.completion": "[{\"role\": \"assistant\", \"content\": \"The sky appears blue because molecules in the Earth's atmosphere scatter shorter wavelengths of sunlight, such as blue, more effectively than longer wavelengths like red.\"}]"
|
||||
}
|
||||
}
|
||||
],
|
||||
"links": [],
|
||||
"resource": {
|
||||
"attributes": {
|
||||
"telemetry.sdk.language": "python",
|
||||
"telemetry.sdk.name": "opentelemetry",
|
||||
"telemetry.sdk.version": "1.26.0",
|
||||
"service.name": "TelemetryExample"
|
||||
},
|
||||
"schema_url": ""
|
||||
}
|
||||
}
|
||||
{
|
||||
"name": "Scenario: Chat Client",
|
||||
"context": {
|
||||
"trace_id": "0xbda1d9efcd65435653d18fa37aef7dd3",
|
||||
"span_id": "0xeca0a2ca7b7a8191",
|
||||
"trace_state": "[]"
|
||||
},
|
||||
"kind": "SpanKind.INTERNAL",
|
||||
"parent_id": "0x48af7ad55f2f64b5",
|
||||
"start_time": "2024-09-09T23:13:14.625156Z",
|
||||
"end_time": "2024-09-09T23:13:17.312910Z",
|
||||
"status": {
|
||||
"status_code": "UNSET"
|
||||
},
|
||||
"attributes": {},
|
||||
"events": [],
|
||||
"links": [],
|
||||
"resource": {
|
||||
"attributes": {
|
||||
"telemetry.sdk.language": "python",
|
||||
"telemetry.sdk.name": "opentelemetry",
|
||||
"telemetry.sdk.version": "1.26.0",
|
||||
"service.name": "TelemetryExample"
|
||||
},
|
||||
"schema_url": ""
|
||||
}
|
||||
}
|
||||
{
|
||||
"name": "Scenario's",
|
||||
"context": {
|
||||
"trace_id": "0xbda1d9efcd65435653d18fa37aef7dd3",
|
||||
"span_id": "0x48af7ad55f2f64b5",
|
||||
"trace_state": "[]"
|
||||
},
|
||||
"kind": "SpanKind.INTERNAL",
|
||||
"parent_id": null,
|
||||
"start_time": "2024-09-09T23:13:13.840481Z",
|
||||
"end_time": "2024-09-09T23:13:17.312910Z",
|
||||
"status": {
|
||||
"status_code": "UNSET"
|
||||
},
|
||||
"attributes": {},
|
||||
"events": [],
|
||||
"links": [],
|
||||
"resource": {
|
||||
"attributes": {
|
||||
"telemetry.sdk.language": "python",
|
||||
"telemetry.sdk.name": "opentelemetry",
|
||||
"telemetry.sdk.version": "1.26.0",
|
||||
"service.name": "TelemetryExample"
|
||||
},
|
||||
"schema_url": ""
|
||||
}
|
||||
}
|
||||
{
|
||||
"body": "Agent Framework usage: CompletionUsage(completion_tokens=28, prompt_tokens=16, total_tokens=44)",
|
||||
"severity_number": "<SeverityNumber.INFO: 9>",
|
||||
"severity_text": "INFO",
|
||||
"attributes": {
|
||||
"code.filepath": "/path/to/agent_framework/openai/chat_client.py",
|
||||
"code.function": "store_usage",
|
||||
"code.lineno": 81
|
||||
},
|
||||
"dropped_attributes": 0,
|
||||
"timestamp": "2024-09-09T23:13:17.311909Z",
|
||||
"observed_timestamp": "2024-09-09T23:13:17.311909Z",
|
||||
"trace_id": "0xbda1d9efcd65435653d18fa37aef7dd3",
|
||||
"span_id": "0xcd443e1917510385",
|
||||
"trace_flags": 1,
|
||||
"resource": {
|
||||
"attributes": {
|
||||
"telemetry.sdk.language": "python",
|
||||
"telemetry.sdk.name": "opentelemetry",
|
||||
"telemetry.sdk.version": "1.26.0",
|
||||
"service.name": "TelemetryExample"
|
||||
},
|
||||
"schema_url": ""
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
In the output, you will find three spans: `Scenario's`, `Scenario: Chat Client`, and `chat.completions gpt-4o`, each representing a different layer in the sample. In particular, `chat.completions gpt-4o` is generated by the chat client. Inside it, you will find information about the call, such as the timestamp of the operation, the response id and the finish reason. You will also find sensitive information such as the prompt and response to and from the model (only if you have `AGENT_FRAMEWORK__GENAI_ENABLE_OTEL_DIAGNOSTICS_SENSITIVE` set to true). If you use Application Insights or Aspire Dashboard, these information will be available to you in an interactive UI.
|
||||
Use the guides from OpenTelemetry to setup exporters for [the console](https://opentelemetry.io/docs/languages/python/getting-started/), or use [manual_setup_console_output](./manual_setup_console_output.py) as a reference, just know that there are a lot of options you can setup and this is not a comprehensive example.
|
||||
|
||||
Reference in New Issue
Block a user