[BREAKING] Python: Enable instrumentation by default (#5865)

* Enable instrumentation by default

* Update samples

* Optimization when span is not recording

* Address Copilot comments

* Revert uv.lock

* Add warning

* Formatting

* Fix mypy

* Add disable_instrumentation() with sticky user-intent semantics

Add a public disable_instrumentation() entry point so users can explicitly opt
out of Agent Framework telemetry, with a sticky-disable flag that makes the
user's intent "leading" — no framework code path (foundry's
configure_azure_monitor, configure_otel_providers, enable_instrumentation,
enable_sensitive_telemetry, or direct OBSERVABILITY_SETTINGS.enable_*
writes) can re-enable instrumentation until the user explicitly clears the
disable with enable_instrumentation(force=True) /
enable_sensitive_telemetry(force=True).

Also addresses the two remaining unresolved review threads on the PR:
1. test_observability_settings_defaults_instrumentation_true pins the new
   "ENABLE_INSTRUMENTATION defaults to True when env unset" behavior.
2. test_enable_instrumentation_reads_env_sensitive_data restores coverage
   for the post-import load_dotenv() fallback path.

Implementation:
- ObservabilitySettings.enable_instrumentation / enable_sensitive_data become
  properties backed by _enable_*. While _user_disabled is True, the getters
  return False and the setters drop True writes (defense in depth so third-
  party writes can't subvert the disable).
- Public is_user_disabled read-only property lets integrations (e.g. foundry's
  configure_azure_monitor) cheaply check the disable state without poking at
  privates.
- enable_instrumentation() and enable_sensitive_telemetry() short-circuit with
  an info log when disabled; gain a force=True kwarg that clears the disable.
- configure_otel_providers() still creates providers / exporters / views so a
  later force-enable can use them, but logs an info message when called while
  disabled.
- Foundry's FoundryChatClient.configure_azure_monitor and
  FoundryAgent.configure_azure_monitor early-return when the user has
  disabled, so Azure Monitor's global providers aren't installed unnecessarily.

Tests: 11 new tests covering default-on, env re-read at call time, sticky
behavior against each re-enable surface (enable_instrumentation,
enable_sensitive_telemetry, configure_otel_providers, direct attribute
writes), force=True override, re-arming the disable, and the __all__ export.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: document disable_instrumentation() and force=True paths

Add a "Disabling instrumentation" section to the observability sample README
that walks through:

- The distinction between the ENABLE_INSTRUMENTATION env var (initial,
  non-sticky) and disable_instrumentation() (process-wide, sticky).
- Why the sticky semantics matter: framework integrations like
  FoundryChatClient.configure_azure_monitor() can call
  enable_instrumentation() as part of their setup, and the user's opt-out
  needs to win.
- All five surfaces guarded by the sticky disable (property reads, public
  enable functions, configure_otel_providers, direct attribute writes,
  is_user_disabled-aware integrations).
- The force=True escape hatch on both enable_instrumentation() and
  enable_sensitive_telemetry().
- How third-party integrations should consult OBSERVABILITY_SETTINGS.is_user_disabled.
- The limits of the disable (does not tear down existing providers /
  in-flight spans / third-party instrumentation, does not persist across
  processes).

Cross-links the new section from the ENABLE_INSTRUMENTATION row in the env
vars table.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: soften disable_instrumentation() overclaim about telemetry guarantees

Replace 'no telemetry will be emitted no matter what' (which is too strong,
since callers can still pass force=True or mutate private attributes) with
language framing the disable as a user-intent contract that library and
framework code is expected to honor: the framework actively short-circuits
the public enable paths, force=True and private-attribute writes are
acknowledged as out-of-contract escape hatches that integrations should
not use on the user's behalf.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: correct observability Dependencies section

- opentelemetry-sdk is no longer a hard dependency; it is lazily imported by
  create_resource(), create_metric_views(), and configure_otel_providers()
  with a clear ImportError when missing. Day-to-day instrumentation works
  with opentelemetry-api alone provided some other component configures the
  global OpenTelemetry providers (Azure Monitor, an APM agent, application
  bootstrap, etc.).
- opentelemetry-semantic-conventions-ai is no longer used anywhere in the
  source; remove it from the listed dependencies.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: replace stale observability migration guide with current PR's only relevant migration

The old guide documented the move away from setup_observability(otlp_endpoint=...)
which was an earlier-release API change unrelated to this PR and stale enough that
it's more confusing than helpful at this point. Replace it with a short note on the
single migration this PR introduces: callers of
enable_instrumentation(enable_sensitive_data=True) should switch to
enable_sensitive_telemetry(). Cross-link to the Disabling instrumentation section
for the rare 'force on without enabling sensitive data' use case where
enable_instrumentation() still applies.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

python/samples/04-hosting/foundry-hosted-agents/responses/07_observability/.env.example

@@ -1,4 +1,3 @@
 FOUNDRY_PROJECT_ENDPOINT="..."
 AZURE_AI_MODEL_DEPLOYMENT_NAME="..."
-ENABLE_INSTRUMENTATION=true
 ENABLE_SENSITIVE_DATA=true

python/samples/04-hosting/foundry-hosted-agents/responses/07_observability/README.md

@@ -16,7 +16,7 @@ The agent is hosted using the [Agent Framework](https://github.com/microsoft/age
 
 ### Instrumentation
 
-Agent Framework is [**natively instrumented**](https://learn.microsoft.com/en-us/agent-framework/agents/observability?pivots=programming-language-python) to capture diagnostics and telemetry for agent execution, but it's turned off by default. This sample demonstrates how to enable instrumentation via environment variables in `agent.manifest.yaml` and `agent.yaml`. The relevant environment variables are `ENABLE_INSTRUMENTATION` and `ENABLE_SENSITIVE_DATA`, which can be set to `true` to enable diagnostics and capture sensitive events respectively.
+Agent Framework is [**natively instrumented**](https://learn.microsoft.com/en-us/agent-framework/agents/observability?pivots=programming-language-python) to capture diagnostics and telemetry for agent execution. Instrumentation is enabled by default. To also capture sensitive event payloads (prompts, tool arguments, etc.) set `ENABLE_SENSITIVE_DATA=true`. This sample demonstrates how to manage these settings via environment variables in `agent.manifest.yaml` and `agent.yaml`.
 
 Foundry Hosted Agent has built-in observability thus you don't need to set up exporters manually to capture telemetry from your code. The traces, metrics, and logs generated by the agent are automatically collected and made available through Foundry's observability stack via Azure Monitor/Application Insights. The `APPLICATIONINSIGHTS_CONNECTION_STRING` environment variable is injected when the agent is deployed to Foundry, however it is still required to be set in your environment if you want to run the agent host locally and have telemetry sent to Application Insights from your local environment.
 

python/samples/04-hosting/foundry-hosted-agents/responses/07_observability/agent.manifest.yaml

@@ -17,8 +17,6 @@ template:
   environment_variables:
     - name: AZURE_AI_MODEL_DEPLOYMENT_NAME
       value: "{{AZURE_AI_MODEL_DEPLOYMENT_NAME}}"
-    - name: ENABLE_INSTRUMENTATION
-      value: true
     - name: ENABLE_SENSITIVE_DATA
       value: true
 resources:

python/samples/04-hosting/foundry-hosted-agents/responses/07_observability/agent.yaml

@@ -5,12 +5,10 @@ protocols:
   - protocol: responses
     version: 1.0.0
 resources:
-  cpu: '0.25'
-  memory: '0.5Gi'
+  cpu: "0.25"
+  memory: "0.5Gi"
 environment_variables:
   - name: AZURE_AI_MODEL_DEPLOYMENT_NAME
     value: ${AZURE_AI_MODEL_DEPLOYMENT_NAME}
-  - name: ENABLE_INSTRUMENTATION
-    value: true
   - name: ENABLE_SENSITIVE_DATA
-    value: true
+    value: true