Files
codex/codex-rs/otel
jifandGitHub 5f4d06ef18 Encrypt multi-agent v2 message payloads (#26210)
## Why

Multi-agent v2 currently routes agent instructions through normal tool
arguments and inter-agent context. That means the parent model can emit
plaintext task text, Codex can persist it in history/rollouts, and the
recipient can receive it as ordinary assistant-message JSON.

This changes the v2 path so agent instructions stay encrypted between
model calls: Responses encrypts the `message` argument returned by the
model, Codex forwards only that ciphertext, and Responses decrypts it
internally for the recipient model.

## What changed

- Mark the v2 `message` parameter as encrypted for `spawn_agent`,
`send_message`, and `followup_task`.
- Treat multi-agent v2 tool `message` values as ciphertext
unconditionally.
- Store v2 inter-agent task text in
`InterAgentCommunication.encrypted_content` with empty plaintext
`content`.
- Convert encrypted inter-agent communications into the Responses
`agent_message` input item before sending the child request.
- Preserve `agent_message` items across history, rollout, compaction,
telemetry, and app-server schema paths.
- Leave multi-agent v1 unchanged.

## Message shape

The model still calls the v2 tools with a `message` argument, but that
value is now ciphertext:

```json
{
  "name": "spawn_agent",
  "arguments": {
    "task_name": "worker",
    "message": "<ciphertext>"
  }
}
```

Codex stores the task as encrypted inter-agent communication:

```json
{
  "author": "/root",
  "recipient": "/root/worker",
  "content": "",
  "encrypted_content": "<ciphertext>",
  "trigger_turn": true
}
```

When Codex builds the recipient request, it forwards the ciphertext
using the new Responses input item:

```json
{
  "type": "agent_message",
  "author": "/root",
  "recipient": "/root/worker",
  "content": [
    {
      "type": "encrypted_content",
      "encrypted_content": "<ciphertext>"
    }
  ]
}
```

Responses decrypts that item internally for the recipient model.

## Context impact

- Parent context no longer carries plaintext v2 agent task instructions
from these tool arguments.
- Codex rollout/history stores ciphertext for v2 agent instructions.
- Recipient requests receive an `agent_message` item instead of
assistant commentary JSON for encrypted task delivery.
- Plaintext completion/status notifications are still plaintext because
they are Codex-generated status messages, not encrypted model tool
arguments.

## Validation

- `just test -p codex-tools`
- `just test -p codex-protocol`
- `just test -p codex-rollout`
- `just test -p codex-rollout-trace`
- `just test -p codex-otel`
- `just write-app-server-schema`
2026-06-05 10:25:57 +02:00
..

codex-otel

codex-otel is the OpenTelemetry integration crate for Codex. It provides:

  • Provider wiring for log/trace/metric exporters (codex_otel::OtelProvider and codex_otel::provider).
  • Session-scoped business event emission via codex_otel::SessionTelemetry.
  • Low-level metrics APIs via codex_otel::metrics.
  • Trace-context helpers via codex_otel::trace_context and crate-root re-exports.

Tracing and logs

Create an OTEL provider from OtelSettings. The provider also configures metrics (when enabled), then attach its layers to your tracing_subscriber registry:

use codex_otel::config::OtelExporter;
use codex_otel::config::OtelHttpProtocol;
use codex_otel::config::OtelSettings;
use codex_otel::OtelProvider;
use tracing_subscriber::prelude::*;

let settings = OtelSettings {
    environment: "dev".to_string(),
    service_name: "codex-cli".to_string(),
    service_version: env!("CARGO_PKG_VERSION").to_string(),
    codex_home: std::path::PathBuf::from("/tmp"),
    exporter: OtelExporter::OtlpHttp {
        endpoint: "https://otlp.example.com".to_string(),
        headers: std::collections::HashMap::new(),
        protocol: OtelHttpProtocol::Binary,
        tls: None,
    },
    trace_exporter: OtelExporter::OtlpHttp {
        endpoint: "https://otlp.example.com".to_string(),
        headers: std::collections::HashMap::new(),
        protocol: OtelHttpProtocol::Binary,
        tls: None,
    },
    metrics_exporter: OtelExporter::None,
    span_attributes: std::collections::BTreeMap::new(),
    tracestate: std::collections::BTreeMap::new(),
};

if let Some(provider) = OtelProvider::from(&settings)? {
    let registry = tracing_subscriber::registry()
        .with(provider.logger_layer())
        .with(provider.tracing_layer());
    registry.init();
}

Configured span attributes and W3C tracestate member fields are applied to exported trace spans and propagated trace context:

[otel.span_attributes]
"example.trace_attr" = "enabled"

[otel.tracestate.example]
alpha = "one"
beta = "two"

Configured tracestate members and encoded values must be valid W3C tracestate. Each nested table is encoded as semicolon-separated key:value fields inside that member. If propagated trace context already has the named member, Codex upserts configured fields and preserves other fields in that member. This config shape does not support setting opaque tracestate member values. Invalid trace metadata entries are ignored during config load and reported as startup warnings.

SessionTelemetry (events)

SessionTelemetry adds consistent metadata to tracing events and helps record Codex-specific session events. Rich session/business events should go through SessionTelemetry; subsystem-owned audit events can stay with the owning subsystem.

use codex_otel::SessionTelemetry;

let manager = SessionTelemetry::new(
    conversation_id,
    model,
    slug,
    account_id,
    account_email,
    auth_mode,
    originator,
    log_user_prompts,
    terminal_type,
    session_source,
);

manager.user_prompt(&prompt_items);

Metrics (OTLP or in-memory)

Modes:

  • OTLP: exports metrics via the OpenTelemetry OTLP exporter (HTTP or gRPC).
  • In-memory: records via opentelemetry_sdk::metrics::InMemoryMetricExporter for tests/assertions; call shutdown() to flush.

codex-otel also provides OtelExporter::Statsig, a shorthand for exporting OTLP/HTTP JSON metrics to Statsig using Codex-internal defaults.

Statsig ingestion (OTLP/HTTP JSON) example:

use codex_otel::config::{OtelExporter, OtelHttpProtocol};

let metrics = MetricsClient::new(MetricsConfig::otlp(
    "dev",
    "codex-cli",
    env!("CARGO_PKG_VERSION"),
    OtelExporter::OtlpHttp {
        endpoint: "https://api.statsig.com/otlp".to_string(),
        headers: std::collections::HashMap::from([(
            "statsig-api-key".to_string(),
            std::env::var("STATSIG_SERVER_SDK_SECRET")?,
        )]),
        protocol: OtelHttpProtocol::Json,
        tls: None,
    },
))?;

metrics.counter("codex.session_started", 1, &[("source", "tui")])?;
metrics.histogram("codex.request_latency", 83, &[("route", "chat")])?;

In-memory (tests):

let exporter = InMemoryMetricExporter::default();
let metrics = MetricsClient::new(MetricsConfig::in_memory(
    "test",
    "codex-cli",
    env!("CARGO_PKG_VERSION"),
    exporter.clone(),
))?;
metrics.counter("codex.turns", 1, &[("model", "gpt-5.1")])?;
metrics.shutdown()?; // flushes in-memory exporter

Trace context

Trace propagation helpers remain separate from the session event emitter:

use codex_otel::current_span_w3c_trace_context;
use codex_otel::set_parent_from_w3c_trace_context;

Shutdown

  • OtelProvider::shutdown() stops the OTEL exporter.
  • SessionTelemetry::shutdown_metrics() flushes and shuts down the metrics provider.

Both are optional because drop performs best-effort shutdown, but calling them explicitly gives deterministic flushing (or a shutdown error if flushing does not complete in time).