mirror of
https://github.com/microsoft/agent-framework.git
synced 2026-06-16 21:04:09 +08:00
Python telemetry (#223)
* initial work on telemetry * moved tool operation const * missing quotes * working otel with samples * updated readme and other assets * added tests * added tests * small updates * updated genaiattributes docs * updated tests * additional warning * cleanup of tests
This commit is contained in:
Eduard van Valkenburg
committed by
GitHub
Unverified
GitHub
parent
3ee9dddfa2
commit
0ce8eb1e2f
@@ -0,0 +1,4 @@
|
||||
CONNECTION_STRING="..."
|
||||
OTLP_ENDPOINT="http://localhost:4317/"
|
||||
AGENT_FRAMEWORK_GENAI_ENABLE_OTEL_DIAGNOSTICS=true
|
||||
AGENT_FRAMEWORK_GENAI_ENABLE_OTEL_DIAGNOSTICS_SENSITIVE=true
|
||||
@@ -0,0 +1,244 @@
|
||||
# Agent Framework Python Telemetry
|
||||
|
||||
This sample project shows how a Python application can be configured to send Agent Framework telemetry to the Application Performance Management (APM) vendors of your choice.
|
||||
|
||||
In this sample, we provide options to send telemetry to [Application Insights](https://learn.microsoft.com/en-us/azure/azure-monitor/app/app-insights-overview), [Aspire Dashboard](https://learn.microsoft.com/en-us/dotnet/aspire/fundamentals/dashboard/overview?tabs=bash), and console output.
|
||||
|
||||
> Note that it is also possible to use other Application Performance Management (APM) vendors. An example is [Prometheus](https://prometheus.io/docs/introduction/overview/). Please refer to this [link](https://opentelemetry.io/docs/languages/python/exporters/) to learn more about exporters.
|
||||
|
||||
For more information, please refer to the following resources:
|
||||
1. [Azure Monitor OpenTelemetry Exporter](https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/monitor/azure-monitor-opentelemetry-exporter)
|
||||
2. [Aspire Dashboard for Python Apps](https://learn.microsoft.com/en-us/dotnet/aspire/fundamentals/dashboard/standalone-for-python?tabs=flask%2Cwindows)
|
||||
3. [Python Logging](https://docs.python.org/3/library/logging.html)
|
||||
4. [Observability in Python](https://www.cncf.io/blog/2022/04/22/opentelemetry-and-python-a-complete-instrumentation-guide/)
|
||||
|
||||
## What to expect
|
||||
|
||||
The Agent Framework Python SDK is designed to efficiently generate comprehensive logs, traces, and metrics throughout the flow of function execution and model invocation. This allows you to effectively monitor your AI application's performance and accurately track token consumption.
|
||||
|
||||
## Configuration
|
||||
|
||||
### Required resources
|
||||
2. OpenAI or [Azure OpenAI](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal)
|
||||
### Optional resources
|
||||
1. [Application Insights](https://learn.microsoft.com/en-us/azure/azure-monitor/app/create-workspace-resource)
|
||||
2. [Aspire Dashboard](https://learn.microsoft.com/en-us/dotnet/aspire/fundamentals/dashboard/standalone-for-python?tabs=flask%2Cwindows#start-the-aspire-dashboard)
|
||||
|
||||
### Dependencies
|
||||
You will also need to install the following dependencies to your virtual environment to run this sample:
|
||||
```bash
|
||||
# For Azure ApplicationInsights/AzureMonitor
|
||||
uv pip install azure-monitor-opentelemetry azure-monitor-opentelemetry-exporter
|
||||
# For OTLP endpoint
|
||||
uv pip install opentelemetry-exporter-otlp-proto-grpc
|
||||
```
|
||||
|
||||
## Running the sample
|
||||
|
||||
1. Open a terminal and navigate to this folder: `python/samples/getting_started/telemetry/`. This is necessary for the `.env` file to be read correctly.
|
||||
2. Create a `.env` file if one doesn't already exist in this folder. Please refer to the [example file](./.env.example).
|
||||
> Note that `CONNECTION_STRING` and `SAMPLE_OTLP_ENDPOINT` are optional. If you don't configure them, everything will get outputted to the console.
|
||||
> Set `AGENT_FRAMEWORK_GENAI_ENABLE_OTEL_DIAGNOSTICS=true` to enable basic telemetry and `AGENT_FRAMEWORK_GENAI_ENABLE_OTEL_DIAGNOSTICS_SENSITIVE=true` to include sensitive information like prompts and responses.
|
||||
> Sensitive information should only be enabled in a development or test environment. It is not recommended to enable this in production environments as it may expose sensitive data.
|
||||
3. Activate your python virtual environment, and then run `python scenarios.py` or `python interactive.py`.
|
||||
|
||||
> This will output the Operation/Trace ID, which can be used later for filtering.
|
||||
|
||||
### Scenarios
|
||||
|
||||
This sample includes two different applications demonstrating Agent Framework telemetry:
|
||||
|
||||
#### scenarios.py
|
||||
Organized into specific scenarios where the framework will generate useful telemetry data:
|
||||
|
||||
- `chat_client`: This is when a chat client is invoked directly (i.e. not streaming) with a weather tool function. **Information about the call to the underlying model and tool usage will be recorded**.
|
||||
- `chat_client_stream`: This is when a chat client is invoked with streaming enabled and a weather tool function. **Information about the streaming call to the underlying model and tool usage will be recorded**.
|
||||
- `ai_function`: This is when an AI function (`get_weather`) is invoked directly. **Information about the AI function and the call to the underlying model will be recorded**.
|
||||
|
||||
By default, running `python scenarios.py` will run all three scenarios. To run individual scenarios, use the `--scenario` command line argument. For example, `python scenarios.py --scenario chat_client`. For more information, please run `python scenarios.py -h`.
|
||||
|
||||
#### interactive.py
|
||||
An interactive chat application that demonstrates telemetry collection in a conversational context. This sample includes the same `get_weather` tool function and allows for multi-turn conversations. Run `python interactive.py` and start chatting. Type 'exit' to quit the application. This sample only logs at the `WARNING` level, so you will not see as much telemetry data as in the `scenarios.py` sample.
|
||||
|
||||
## Application Insights/Azure Monitor
|
||||
|
||||
### Logs and traces
|
||||
|
||||
Go to your Application Insights instance, click on _Transaction search_ on the left menu. Use the operation id output by the program to search for the logs and traces associated with the operation. Click on any of the search result to view the end-to-end transaction details. Read more [here](https://learn.microsoft.com/en-us/azure/azure-monitor/app/transaction-search-and-diagnostics?tabs=transaction-search).
|
||||
|
||||
### Metrics
|
||||
|
||||
Running the application once will only generate one set of measurements (for each metrics). Run the application a couple times to generate more sets of measurements.
|
||||
|
||||
> Note: Make sure not to run the program too frequently. Otherwise, you may get throttled.
|
||||
|
||||
Please refer to here on how to analyze metrics in [Azure Monitor](https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/analyze-metrics).
|
||||
|
||||
## Logs
|
||||
|
||||
When you are in Azure Monitor and want to have a overall view of the span, use this query in the logs section:
|
||||
|
||||
```kusto
|
||||
dependencies
|
||||
| where operation_Id in (dependencies
|
||||
| project operation_Id, timestamp
|
||||
| order by timestamp desc
|
||||
| summarize operations = make_set(operation_Id), timestamp = max(timestamp) by operation_Id
|
||||
| order by timestamp desc
|
||||
| project operation_Id
|
||||
| take 2)
|
||||
| evaluate bag_unpack(customDimensions)
|
||||
| extend tool_call_id = tostring(["gen_ai.tool.call.id"])
|
||||
| join kind=leftouter (customMetrics
|
||||
| extend tool_call_id = tostring(customDimensions['gen_ai.tool.call.id'])
|
||||
| where isnotempty(tool_call_id)
|
||||
| project tool_call_duration = value, tool_call_id)
|
||||
on tool_call_id
|
||||
| project-keep timestamp, target, operation_Id, tool_call_duration, duration, gen_ai*
|
||||
| order by timestamp asc
|
||||
```
|
||||
|
||||
## Aspire Dashboard
|
||||
|
||||
> Make sure you have the dashboard running to receive telemetry data.
|
||||
|
||||
Once the the sample finishes running, navigate to http://localhost:18888 in a web browser to see the telemetry data. Follow the instructions [here](https://learn.microsoft.com/en-us/dotnet/aspire/fundamentals/dashboard/explore) to authenticate to the dashboard and start exploring!
|
||||
|
||||
## Console output
|
||||
|
||||
You won't have to deploy an Application Insights resource or install Docker to run Aspire Dashboard if you choose to inspect telemetry data in a console. However, it is difficult to navigate through all the spans and logs produced, so **this method is only recommended when you are just getting started**.
|
||||
|
||||
We recommend you to get started with the `chat_client` scenario as this generates the least amount of telemetry data. Below is similar to what you will see when you run `python scenarios.py --scenario chat_client`:
|
||||
```Json
|
||||
{
|
||||
"name": "chat.completions gpt-4o",
|
||||
"context": {
|
||||
"trace_id": "0xbda1d9efcd65435653d18fa37aef7dd3",
|
||||
"span_id": "0xcd443e1917510385",
|
||||
"trace_state": "[]"
|
||||
},
|
||||
"kind": "SpanKind.INTERNAL",
|
||||
"parent_id": "0xeca0a2ca7b7a8191",
|
||||
"start_time": "2024-09-09T23:13:14.625156Z",
|
||||
"end_time": "2024-09-09T23:13:17.311909Z",
|
||||
"status": {
|
||||
"status_code": "UNSET"
|
||||
},
|
||||
"attributes": {
|
||||
"gen_ai.operation.name": "chat.completions",
|
||||
"gen_ai.system": "openai",
|
||||
"gen_ai.request.model": "gpt-4o",
|
||||
"gen_ai.response.id": "chatcmpl-A5hrG13nhtFsOgx4ziuoskjNscHtT",
|
||||
"gen_ai.response.finish_reason": "FinishReason.STOP",
|
||||
"gen_ai.response.prompt_tokens": 16,
|
||||
"gen_ai.response.completion_tokens": 28
|
||||
},
|
||||
"events": [
|
||||
{
|
||||
"name": "gen_ai.content.prompt",
|
||||
"timestamp": "2024-09-09T23:13:14.625156Z",
|
||||
"attributes": {
|
||||
"gen_ai.prompt": "[{\"role\": \"user\", \"content\": \"Why is the sky blue in one sentence?\"}]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "gen_ai.content.completion",
|
||||
"timestamp": "2024-09-09T23:13:17.311909Z",
|
||||
"attributes": {
|
||||
"gen_ai.completion": "[{\"role\": \"assistant\", \"content\": \"The sky appears blue because molecules in the Earth's atmosphere scatter shorter wavelengths of sunlight, such as blue, more effectively than longer wavelengths like red.\"}]"
|
||||
}
|
||||
}
|
||||
],
|
||||
"links": [],
|
||||
"resource": {
|
||||
"attributes": {
|
||||
"telemetry.sdk.language": "python",
|
||||
"telemetry.sdk.name": "opentelemetry",
|
||||
"telemetry.sdk.version": "1.26.0",
|
||||
"service.name": "TelemetryExample"
|
||||
},
|
||||
"schema_url": ""
|
||||
}
|
||||
}
|
||||
{
|
||||
"name": "Scenario: Chat Client",
|
||||
"context": {
|
||||
"trace_id": "0xbda1d9efcd65435653d18fa37aef7dd3",
|
||||
"span_id": "0xeca0a2ca7b7a8191",
|
||||
"trace_state": "[]"
|
||||
},
|
||||
"kind": "SpanKind.INTERNAL",
|
||||
"parent_id": "0x48af7ad55f2f64b5",
|
||||
"start_time": "2024-09-09T23:13:14.625156Z",
|
||||
"end_time": "2024-09-09T23:13:17.312910Z",
|
||||
"status": {
|
||||
"status_code": "UNSET"
|
||||
},
|
||||
"attributes": {},
|
||||
"events": [],
|
||||
"links": [],
|
||||
"resource": {
|
||||
"attributes": {
|
||||
"telemetry.sdk.language": "python",
|
||||
"telemetry.sdk.name": "opentelemetry",
|
||||
"telemetry.sdk.version": "1.26.0",
|
||||
"service.name": "TelemetryExample"
|
||||
},
|
||||
"schema_url": ""
|
||||
}
|
||||
}
|
||||
{
|
||||
"name": "Scenario's",
|
||||
"context": {
|
||||
"trace_id": "0xbda1d9efcd65435653d18fa37aef7dd3",
|
||||
"span_id": "0x48af7ad55f2f64b5",
|
||||
"trace_state": "[]"
|
||||
},
|
||||
"kind": "SpanKind.INTERNAL",
|
||||
"parent_id": null,
|
||||
"start_time": "2024-09-09T23:13:13.840481Z",
|
||||
"end_time": "2024-09-09T23:13:17.312910Z",
|
||||
"status": {
|
||||
"status_code": "UNSET"
|
||||
},
|
||||
"attributes": {},
|
||||
"events": [],
|
||||
"links": [],
|
||||
"resource": {
|
||||
"attributes": {
|
||||
"telemetry.sdk.language": "python",
|
||||
"telemetry.sdk.name": "opentelemetry",
|
||||
"telemetry.sdk.version": "1.26.0",
|
||||
"service.name": "TelemetryExample"
|
||||
},
|
||||
"schema_url": ""
|
||||
}
|
||||
}
|
||||
{
|
||||
"body": "Agent Framework usage: CompletionUsage(completion_tokens=28, prompt_tokens=16, total_tokens=44)",
|
||||
"severity_number": "<SeverityNumber.INFO: 9>",
|
||||
"severity_text": "INFO",
|
||||
"attributes": {
|
||||
"code.filepath": "/path/to/agent_framework/openai/chat_client.py",
|
||||
"code.function": "store_usage",
|
||||
"code.lineno": 81
|
||||
},
|
||||
"dropped_attributes": 0,
|
||||
"timestamp": "2024-09-09T23:13:17.311909Z",
|
||||
"observed_timestamp": "2024-09-09T23:13:17.311909Z",
|
||||
"trace_id": "0xbda1d9efcd65435653d18fa37aef7dd3",
|
||||
"span_id": "0xcd443e1917510385",
|
||||
"trace_flags": 1,
|
||||
"resource": {
|
||||
"attributes": {
|
||||
"telemetry.sdk.language": "python",
|
||||
"telemetry.sdk.name": "opentelemetry",
|
||||
"telemetry.sdk.version": "1.26.0",
|
||||
"service.name": "TelemetryExample"
|
||||
},
|
||||
"schema_url": ""
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
In the output, you will find three spans: `Scenario's`, `Scenario: Chat Client`, and `chat.completions gpt-4o`, each representing a different layer in the sample. In particular, `chat.completions gpt-4o` is generated by the chat client. Inside it, you will find information about the call, such as the timestamp of the operation, the response id and the finish reason. You will also find sensitive information such as the prompt and response to and from the model (only if you have `AGENT_FRAMEWORK__GENAI_ENABLE_OTEL_DIAGNOSTICS_SENSITIVE` set to true). If you use Application Insights or Aspire Dashboard, these information will be available to you in an interactive UI.
|
||||
@@ -0,0 +1,186 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
# type: ignore
|
||||
import asyncio
|
||||
import logging
|
||||
from random import randint
|
||||
from typing import Annotated
|
||||
|
||||
from agent_framework import ChatMessage, ChatResponse, ChatResponseUpdate
|
||||
from agent_framework.openai import OpenAIChatClient
|
||||
from azure.monitor.opentelemetry import configure_azure_monitor
|
||||
from opentelemetry import trace
|
||||
from opentelemetry._logs import set_logger_provider
|
||||
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter
|
||||
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
|
||||
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
|
||||
from opentelemetry.metrics import set_meter_provider
|
||||
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
|
||||
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor, ConsoleLogExporter
|
||||
from opentelemetry.sdk.metrics import MeterProvider
|
||||
from opentelemetry.sdk.metrics.export import ConsoleMetricExporter, PeriodicExportingMetricReader
|
||||
from opentelemetry.sdk.metrics.view import DropAggregation, View
|
||||
from opentelemetry.sdk.resources import Resource
|
||||
from opentelemetry.sdk.trace import TracerProvider
|
||||
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter
|
||||
from opentelemetry.semconv.attributes import service_attributes
|
||||
from opentelemetry.trace import set_tracer_provider
|
||||
from pydantic import Field
|
||||
from pydantic_settings import BaseSettings
|
||||
|
||||
|
||||
class TelemetrySampleSettings(BaseSettings):
|
||||
"""Settings for the telemetry sample application.
|
||||
|
||||
Optional settings are:
|
||||
- connection_string: str - The connection string for the Application Insights resource.
|
||||
This value can be found in the Overview section when examining
|
||||
your resource from the Azure portal.
|
||||
(Env var CONNECTION_STRING)
|
||||
- otlp_endpoint: str - The OTLP endpoint to send telemetry data to.
|
||||
Depending on the exporter used, you may find this value in different places.
|
||||
(Env var OTLP_ENDPOINT)
|
||||
|
||||
If no connection string or OTLP endpoint is provided, the telemetry data will be
|
||||
exported to the console.
|
||||
"""
|
||||
|
||||
connection_string: str | None = None
|
||||
otlp_endpoint: str | None = None
|
||||
|
||||
|
||||
# Load settings
|
||||
settings = TelemetrySampleSettings()
|
||||
|
||||
# Create a resource to represent the service/sample
|
||||
resource = Resource.create({service_attributes.SERVICE_NAME: "TelemetryExample"})
|
||||
|
||||
# Define the scenarios that can be run
|
||||
SCENARIOS = ["ai_service", "kernel_function", "auto_function_invocation", "all"]
|
||||
|
||||
if settings.connection_string:
|
||||
configure_azure_monitor(
|
||||
connection_string=settings.connection_string, enable_live_metrics=True, logger_name="agent_framework"
|
||||
)
|
||||
|
||||
|
||||
def set_up_logging():
|
||||
class LogFilter(logging.Filter):
|
||||
"""A filter to not process records from several subpackages."""
|
||||
|
||||
# These are the namespaces that we want to exclude from logging for the purposes of this demo.
|
||||
namespaces_to_exclude: list[str] = [
|
||||
"httpx",
|
||||
"openai",
|
||||
]
|
||||
|
||||
def filter(self, record):
|
||||
return not any([record.name.startswith(namespace) for namespace in self.namespaces_to_exclude])
|
||||
|
||||
exporters = []
|
||||
if settings.otlp_endpoint:
|
||||
exporters.append(OTLPLogExporter(endpoint=settings.otlp_endpoint))
|
||||
if not exporters:
|
||||
exporters.append(ConsoleLogExporter())
|
||||
|
||||
# Create and set a global logger provider for the application.
|
||||
logger_provider = LoggerProvider(resource=resource)
|
||||
# Log processors are initialized with an exporter which is responsible
|
||||
# for sending the telemetry data to a particular backend.
|
||||
for log_exporter in exporters:
|
||||
logger_provider.add_log_record_processor(BatchLogRecordProcessor(log_exporter))
|
||||
# Sets the global default logger provider
|
||||
set_logger_provider(logger_provider)
|
||||
|
||||
# Create a logging handler to write logging records, in OTLP format, to the exporter.
|
||||
handler = LoggingHandler()
|
||||
handler.addFilter(LogFilter())
|
||||
# Attach the handler to the root logger. `getLogger()` with no arguments returns the root logger.
|
||||
# Events from all child loggers will be processed by this handler.
|
||||
logger = logging.getLogger()
|
||||
logger.addHandler(handler)
|
||||
# Set the logging level to NOTSET to allow all records to be processed by the handler.
|
||||
logger.setLevel(logging.WARNING)
|
||||
|
||||
|
||||
def set_up_tracing():
|
||||
exporters = []
|
||||
if settings.otlp_endpoint:
|
||||
exporters.append(OTLPSpanExporter(endpoint=settings.otlp_endpoint))
|
||||
if not exporters:
|
||||
exporters.append(ConsoleSpanExporter())
|
||||
|
||||
# Initialize a trace provider for the application. This is a factory for creating tracers.
|
||||
tracer_provider = TracerProvider(resource=resource)
|
||||
# Span processors are initialized with an exporter which is responsible
|
||||
# for sending the telemetry data to a particular backend.
|
||||
for exporter in exporters:
|
||||
tracer_provider.add_span_processor(BatchSpanProcessor(exporter))
|
||||
# Sets the global default tracer provider
|
||||
set_tracer_provider(tracer_provider)
|
||||
|
||||
|
||||
def set_up_metrics():
|
||||
exporters = []
|
||||
if settings.otlp_endpoint:
|
||||
exporters.append(OTLPMetricExporter(endpoint=settings.otlp_endpoint))
|
||||
if not exporters:
|
||||
exporters.append(ConsoleMetricExporter())
|
||||
|
||||
# Initialize a metric provider for the application. This is a factory for creating meters.
|
||||
metric_readers = [
|
||||
PeriodicExportingMetricReader(metric_exporter, export_interval_millis=5000) for metric_exporter in exporters
|
||||
]
|
||||
meter_provider = MeterProvider(
|
||||
metric_readers=metric_readers,
|
||||
resource=resource,
|
||||
views=[
|
||||
# Dropping all instrument names except for those starting with "agent_framework"
|
||||
View(instrument_name="*", aggregation=DropAggregation()),
|
||||
View(instrument_name="agent_framework*"),
|
||||
],
|
||||
)
|
||||
# Sets the global default meter provider
|
||||
set_meter_provider(meter_provider)
|
||||
|
||||
|
||||
def get_weather(
|
||||
location: Annotated[str, Field(description="The location to get the weather for.")],
|
||||
) -> str:
|
||||
"""Get the weather for a given location."""
|
||||
conditions = ["sunny", "cloudy", "rainy", "stormy"]
|
||||
return f"The weather in {location} is {conditions[randint(0, 3)]} with a high of {randint(10, 30)}°C."
|
||||
|
||||
|
||||
async def main():
|
||||
# Set up the providers
|
||||
# This must be done before any other telemetry calls
|
||||
set_up_logging()
|
||||
set_up_tracing()
|
||||
set_up_metrics()
|
||||
|
||||
tracer = trace.get_tracer("agent_framework")
|
||||
with tracer.start_as_current_span("Scenario: Interactive Chat") as current_span:
|
||||
print("Running scenario: Interactive Chat")
|
||||
print("Welcome to the chat, type 'exit' to quit.")
|
||||
client = OpenAIChatClient()
|
||||
messages: list[ChatMessage] = []
|
||||
message = input("User: ")
|
||||
try:
|
||||
while message.lower() != "exit":
|
||||
messages.append(ChatMessage(role="user", text=message))
|
||||
print("Assistant: ", end="")
|
||||
updates: list[ChatResponseUpdate] = []
|
||||
async for update in client.get_streaming_response(messages, tools=get_weather):
|
||||
updates.append(update)
|
||||
if update.text:
|
||||
print(update.text, end="")
|
||||
print("")
|
||||
messages.extend(ChatResponse.from_chat_response_updates(updates).messages)
|
||||
message = input("User: ")
|
||||
except Exception as e:
|
||||
current_span.record_exception(e)
|
||||
print(f"\nError running interactive chat: {e}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,249 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
# type: ignore
|
||||
import argparse
|
||||
import asyncio
|
||||
import logging
|
||||
from random import randint
|
||||
from typing import Annotated, Literal
|
||||
|
||||
from agent_framework import ai_function
|
||||
from agent_framework.openai import OpenAIChatClient
|
||||
from azure.monitor.opentelemetry import configure_azure_monitor
|
||||
from opentelemetry import trace
|
||||
from opentelemetry._logs import set_logger_provider
|
||||
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter
|
||||
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
|
||||
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
|
||||
from opentelemetry.metrics import set_meter_provider
|
||||
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
|
||||
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor, ConsoleLogExporter
|
||||
from opentelemetry.sdk.metrics import MeterProvider
|
||||
from opentelemetry.sdk.metrics.export import ConsoleMetricExporter, PeriodicExportingMetricReader
|
||||
from opentelemetry.sdk.metrics.view import DropAggregation, View
|
||||
from opentelemetry.sdk.resources import Resource
|
||||
from opentelemetry.sdk.trace import TracerProvider
|
||||
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter
|
||||
from opentelemetry.semconv.attributes import service_attributes
|
||||
from opentelemetry.trace import set_tracer_provider
|
||||
from opentelemetry.trace.span import format_trace_id
|
||||
from pydantic import Field
|
||||
from pydantic_settings import BaseSettings
|
||||
|
||||
|
||||
class TelemetrySampleSettings(BaseSettings):
|
||||
"""Settings for the telemetry sample application.
|
||||
|
||||
Optional settings are:
|
||||
- connection_string: str - The connection string for the Application Insights resource.
|
||||
This value can be found in the Overview section when examining
|
||||
your resource from the Azure portal.
|
||||
(Env var CONNECTION_STRING)
|
||||
- otlp_endpoint: str - The OTLP endpoint to send telemetry data to.
|
||||
Depending on the exporter used, you may find this value in different places.
|
||||
(Env var OTLP_ENDPOINT)
|
||||
|
||||
If no connection string or OTLP endpoint is provided, the telemetry data will be
|
||||
exported to the console.
|
||||
"""
|
||||
|
||||
connection_string: str | None = None
|
||||
otlp_endpoint: str | None = None
|
||||
|
||||
|
||||
# Load settings
|
||||
settings = TelemetrySampleSettings()
|
||||
|
||||
# Create a resource to represent the service/sample
|
||||
resource = Resource.create({service_attributes.SERVICE_NAME: "TelemetryExample"})
|
||||
|
||||
# Define the scenarios that can be run
|
||||
SCENARIOS = ["chat_client", "chat_client_stream", "ai_function", "all"]
|
||||
|
||||
if settings.connection_string:
|
||||
configure_azure_monitor(
|
||||
connection_string=settings.connection_string,
|
||||
enable_live_metrics=True,
|
||||
logger_name="agent_framework",
|
||||
)
|
||||
|
||||
|
||||
def set_up_logging():
|
||||
class LogFilter(logging.Filter):
|
||||
"""A filter to not process records from several subpackages."""
|
||||
|
||||
# These are the namespaces that we want to exclude from logging for the purposes of this demo.
|
||||
namespaces_to_exclude: list[str] = [
|
||||
"httpx",
|
||||
"openai",
|
||||
]
|
||||
|
||||
def filter(self, record):
|
||||
return not any([record.name.startswith(namespace) for namespace in self.namespaces_to_exclude])
|
||||
|
||||
exporters = []
|
||||
if settings.otlp_endpoint:
|
||||
exporters.append(OTLPLogExporter(endpoint=settings.otlp_endpoint))
|
||||
if not exporters:
|
||||
exporters.append(ConsoleLogExporter())
|
||||
|
||||
# Create and set a global logger provider for the application.
|
||||
logger_provider = LoggerProvider(resource=resource)
|
||||
# Log processors are initialized with an exporter which is responsible
|
||||
# for sending the telemetry data to a particular backend.
|
||||
for log_exporter in exporters:
|
||||
logger_provider.add_log_record_processor(BatchLogRecordProcessor(log_exporter))
|
||||
# Sets the global default logger provider
|
||||
set_logger_provider(logger_provider)
|
||||
|
||||
# Create a logging handler to write logging records, in OTLP format, to the exporter.
|
||||
handler = LoggingHandler()
|
||||
handler.addFilter(LogFilter())
|
||||
# Attach the handler to the root logger. `getLogger()` with no arguments returns the root logger.
|
||||
# Events from all child loggers will be processed by this handler.
|
||||
logger = logging.getLogger()
|
||||
logger.addHandler(handler)
|
||||
# Set the logging level to NOTSET to allow all records to be processed by the handler.
|
||||
logger.setLevel(logging.NOTSET)
|
||||
|
||||
|
||||
def set_up_tracing():
|
||||
exporters = []
|
||||
if settings.otlp_endpoint:
|
||||
exporters.append(OTLPSpanExporter(endpoint=settings.otlp_endpoint))
|
||||
if not exporters:
|
||||
exporters.append(ConsoleSpanExporter())
|
||||
|
||||
# Initialize a trace provider for the application. This is a factory for creating tracers.
|
||||
tracer_provider = TracerProvider(resource=resource)
|
||||
# Span processors are initialized with an exporter which is responsible
|
||||
# for sending the telemetry data to a particular backend.
|
||||
for exporter in exporters:
|
||||
tracer_provider.add_span_processor(BatchSpanProcessor(exporter))
|
||||
# Sets the global default tracer provider
|
||||
set_tracer_provider(tracer_provider)
|
||||
|
||||
|
||||
def set_up_metrics():
|
||||
exporters = []
|
||||
if settings.otlp_endpoint:
|
||||
exporters.append(OTLPMetricExporter(endpoint=settings.otlp_endpoint))
|
||||
if not exporters:
|
||||
exporters.append(ConsoleMetricExporter())
|
||||
|
||||
# Initialize a metric provider for the application. This is a factory for creating meters.
|
||||
metric_readers = [
|
||||
PeriodicExportingMetricReader(metric_exporter, export_interval_millis=5000) for metric_exporter in exporters
|
||||
]
|
||||
meter_provider = MeterProvider(
|
||||
metric_readers=metric_readers,
|
||||
resource=resource,
|
||||
views=[
|
||||
# Dropping all instrument names except for those starting with "agent_framework"
|
||||
View(instrument_name="*", aggregation=DropAggregation()),
|
||||
View(instrument_name="agent_framework*"),
|
||||
],
|
||||
)
|
||||
# Sets the global default meter provider
|
||||
set_meter_provider(meter_provider)
|
||||
|
||||
|
||||
def get_weather(
|
||||
location: Annotated[str, Field(description="The location to get the weather for.")],
|
||||
) -> str:
|
||||
"""Get the weather for a given location."""
|
||||
conditions = ["sunny", "cloudy", "rainy", "stormy"]
|
||||
return f"The weather in {location} is {conditions[randint(0, 3)]} with a high of {randint(10, 30)}°C."
|
||||
|
||||
|
||||
async def run_chat_client(stream: bool = False) -> None:
|
||||
"""Run an AI service.
|
||||
|
||||
This function runs an AI service and prints the output.
|
||||
Telemetry will be collected for the service execution behind the scenes,
|
||||
and the traces will be sent to the configured telemetry backend.
|
||||
|
||||
The telemetry will include information about the AI service execution.
|
||||
|
||||
Args:
|
||||
stream (bool): Whether to use streaming for the plugin
|
||||
"""
|
||||
|
||||
tracer = trace.get_tracer(__name__)
|
||||
with tracer.start_as_current_span(
|
||||
"Scenario: Chat Client Stream" if stream else "Scenario: Chat Client"
|
||||
) as current_span:
|
||||
print("Running scenario: Chat Client" if not stream else "Running scenario: Chat Client Stream")
|
||||
try:
|
||||
client = OpenAIChatClient()
|
||||
message = "What's the weather in Amsterdam and in Paris?"
|
||||
print(f"User: {message}")
|
||||
if stream:
|
||||
print("Assistant: ", end="")
|
||||
async for chunk in client.get_streaming_response(message, tools=get_weather):
|
||||
if str(chunk):
|
||||
print(str(chunk), end="")
|
||||
print("")
|
||||
else:
|
||||
response = await client.get_response(message, tools=get_weather)
|
||||
print(f"Assistant: {response}")
|
||||
except Exception as e:
|
||||
current_span.record_exception(e)
|
||||
print(f"Error running AI service: {e}")
|
||||
|
||||
|
||||
async def run_ai_function() -> None:
|
||||
"""Run a AI function.
|
||||
|
||||
This function runs a AI function and prints the output.
|
||||
Telemetry will be collected for the function execution behind the scenes,
|
||||
and the traces will be sent to the configured telemetry backend.
|
||||
|
||||
The telemetry will include information about the AI function execution
|
||||
and the AI service execution.
|
||||
"""
|
||||
|
||||
tracer = trace.get_tracer(__name__)
|
||||
with tracer.start_as_current_span("Scenario: AI Function") as current_span:
|
||||
print("Running scenario: AI Function")
|
||||
try:
|
||||
func = ai_function(get_weather)
|
||||
weather = await func.invoke(location="Amsterdam")
|
||||
print(f"Weather in Amsterdam:\n{weather}")
|
||||
except Exception as e:
|
||||
current_span.record_exception(e)
|
||||
print(f"Error running kernel plugin: {e}")
|
||||
|
||||
|
||||
async def main(scenario: Literal["chat_client", "chat_client_stream", "ai_function", "all"] = "all"):
|
||||
# Set up the providers
|
||||
# This must be done before any other telemetry calls
|
||||
set_up_logging()
|
||||
set_up_tracing()
|
||||
set_up_metrics()
|
||||
|
||||
tracer = trace.get_tracer("agent_framework")
|
||||
with tracer.start_as_current_span("Scenario's") as current_span:
|
||||
print(f"Trace ID: {format_trace_id(current_span.get_span_context().trace_id)}")
|
||||
|
||||
# Scenarios where telemetry is collected in the SDK, from the most basic to the most complex.
|
||||
if scenario == "chat_client" or scenario == "all":
|
||||
await run_chat_client(stream=False)
|
||||
if scenario == "chat_client_stream" or scenario == "all":
|
||||
await run_chat_client(stream=True)
|
||||
if scenario == "ai_function" or scenario == "all":
|
||||
await run_ai_function()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
arg_parser = argparse.ArgumentParser()
|
||||
|
||||
arg_parser.add_argument(
|
||||
"--scenario",
|
||||
type=str,
|
||||
choices=SCENARIOS,
|
||||
default="all",
|
||||
help="The scenario to run. Default is all.",
|
||||
)
|
||||
|
||||
args = arg_parser.parse_args()
|
||||
asyncio.run(main(args.scenario))
|
||||
Reference in New Issue
Block a user