Add auth 401 observability to client bug reports (#14611)

CXC-392 [With 401](https://openai.sentry.io/issues/7333870443/?project=4510195390611458&query=019ce8f8-560c-7f10-a00a-c59553740674&referrer=issue-stream) <img width="1909" height="555" alt="401 auth tags in Sentry" src="https://github.com/user-attachments/assets/412ea950-61c4-4780-9697-15c270971ee3" /> - auth_401_*: preserved facts from the latest unauthorized response snapshot - auth_*: latest auth-related facts from the latest request attempt - auth_recovery_*: unauthorized recovery state and follow-up result Without 401 <img width="1917" height="522" alt="happy-path auth tags in Sentry" src="https://github.com/user-attachments/assets/3381ed28-8022-43b0-b6c0-623a630e679f" /> ###### Summary - Add client-visible 401 diagnostics for auth attachment, upstream auth classification, and 401 request id / cf-ray correlation. - Record unauthorized recovery mode, phase, outcome, and retry/follow-up status without changing auth behavior. - Surface the highest-signal auth and recovery fields on uploaded client bug reports so they are usable in Sentry. - Preserve original unauthorized evidence under `auth_401_*` while keeping follow-up result tags separate. ###### Rationale (from spec findings) - The dominant bucket needed proof of whether the client attached auth before send or upstream still classified the request as missing auth. - Client uploads needed to show whether unauthorized recovery ran and what the client tried next. - Request id and cf-ray needed to be preserved on the unauthorized response so server-side correlation is immediate. - The bug-report path needed the same auth evidence as the request telemetry path, otherwise the observability would not be operationally useful. ###### Scope - Add auth 401 and unauthorized-recovery observability in `codex-rs/core`, `codex-rs/codex-api`, and `codex-rs/otel`, including feedback-tag surfacing. - Keep auth semantics, refresh behavior, retry behavior, endpoint classification, and geo-denial follow-up work out of this PR. ###### Trade-offs - This exports only safe auth evidence: header presence/name, upstream auth classification, request ids, and recovery state. It does not export token values or raw upstream bodies. - This keeps websocket connection reuse as a transport clue because it can help distinguish stale reused sessions from fresh reconnects. - Misroute/base-url classification and geo-denial are intentionally deferred to a separate follow-up PR so this review stays focused on the dominant auth 401 bucket. ###### Client follow-up - PR 2 will add misroute/provider and geo-denial observability plus the matching feedback-tag surfacing. - A separate host/app-server PR should log auth-decision inputs so pre-send host auth state can be correlated with client request evidence. - `device_id` remains intentionally separate until there is a safe existing source on the feedback upload path. ###### Testing - `cargo test -p codex-core refresh_available_models_sorts_by_priority` - `cargo test -p codex-core emit_feedback_request_tags_` - `cargo test -p codex-core emit_feedback_auth_recovery_tags_` - `cargo test -p codex-core auth_request_telemetry_context_tracks_attached_auth_and_retry_phase` - `cargo test -p codex-core extract_response_debug_context_decodes_identity_headers` - `cargo test -p codex-core identity_auth_details` - `cargo test -p codex-core telemetry_error_messages_preserve_non_http_details` - `cargo test -p codex-core --all-features --no-run` - `cargo test -p codex-otel otel_export_routing_policy_routes_api_request_auth_observability` - `cargo test -p codex-otel otel_export_routing_policy_routes_websocket_connect_auth_observability` - `cargo test -p codex-otel otel_export_routing_policy_routes_websocket_request_transport_observability`
2026-07-01 00:31:56 +08:00 · 2026-03-14 15:38:51 -07:00
parent 9060dc7557
commit d692b74007
19 changed files with 1997 additions and 51 deletions
@@ -3,6 +3,7 @@ use crate::api_bridge::auth_provider_from_auth;
 use crate::api_bridge::map_api_error;
 use crate::auth::AuthManager;
 use crate::auth::AuthMode;
+use crate::auth::CodexAuth;
 use crate::config::Config;
 use crate::default_client::build_reqwest_client;
 use crate::error::CodexErr;
@@ -11,8 +12,15 @@ use crate::model_provider_info::ModelProviderInfo;
 use crate::models_manager::collaboration_mode_presets::CollaborationModesConfig;
 use crate::models_manager::collaboration_mode_presets::builtin_collaboration_mode_presets;
 use crate::models_manager::model_info;
+use crate::response_debug_context::extract_response_debug_context;
+use crate::response_debug_context::telemetry_transport_error_message;
+use crate::util::FeedbackRequestTags;
+use crate::util::emit_feedback_request_tags;
 use codex_api::ModelsClient;
+use codex_api::RequestTelemetry;
 use codex_api::ReqwestTransport;
+use codex_api::TransportError;
+use codex_otel::TelemetryAuthMode;
 use codex_protocol::config_types::CollaborationModeMask;
 use codex_protocol::openai_models::ModelInfo;
 use codex_protocol::openai_models::ModelPreset;
@@ -32,6 +40,82 @@ use tracing::instrument;
 const MODEL_CACHE_FILE: &str = "models_cache.json";
 const DEFAULT_MODEL_CACHE_TTL: Duration = Duration::from_secs(300);
 const MODELS_REFRESH_TIMEOUT: Duration = Duration::from_secs(5);
+const MODELS_ENDPOINT: &str = "/models";
+#[derive(Clone)]
+struct ModelsRequestTelemetry {
+    auth_mode: Option<String>,
+    auth_header_attached: bool,
+    auth_header_name: Option<&'static str>,
+}
+
+impl RequestTelemetry for ModelsRequestTelemetry {
+    fn on_request(
+        &self,
+        attempt: u64,
+        status: Option<http::StatusCode>,
+        error: Option<&TransportError>,
+        duration: Duration,
+    ) {
+        let success = status.is_some_and(|code| code.is_success()) && error.is_none();
+        let error_message = error.map(telemetry_transport_error_message);
+        let response_debug = error
+            .map(extract_response_debug_context)
+            .unwrap_or_default();
+        let status = status.map(|status| status.as_u16());
+        tracing::event!(
+            target: "codex_otel.log_only",
+            tracing::Level::INFO,
+            event.name = "codex.api_request",
+            duration_ms = %duration.as_millis(),
+            http.response.status_code = status,
+            success = success,
+            error.message = error_message.as_deref(),
+            attempt = attempt,
+            endpoint = MODELS_ENDPOINT,
+            auth.header_attached = self.auth_header_attached,
+            auth.header_name = self.auth_header_name,
+            auth.request_id = response_debug.request_id.as_deref(),
+            auth.cf_ray = response_debug.cf_ray.as_deref(),
+            auth.error = response_debug.auth_error.as_deref(),
+            auth.error_code = response_debug.auth_error_code.as_deref(),
+            auth.mode = self.auth_mode.as_deref(),
+        );
+        tracing::event!(
+            target: "codex_otel.trace_safe",
+            tracing::Level::INFO,
+            event.name = "codex.api_request",
+            duration_ms = %duration.as_millis(),
+            http.response.status_code = status,
+            success = success,
+            error.message = error_message.as_deref(),
+            attempt = attempt,
+            endpoint = MODELS_ENDPOINT,
+            auth.header_attached = self.auth_header_attached,
+            auth.header_name = self.auth_header_name,
+            auth.request_id = response_debug.request_id.as_deref(),
+            auth.cf_ray = response_debug.cf_ray.as_deref(),
+            auth.error = response_debug.auth_error.as_deref(),
+            auth.error_code = response_debug.auth_error_code.as_deref(),
+            auth.mode = self.auth_mode.as_deref(),
+        );
+        emit_feedback_request_tags(&FeedbackRequestTags {
+            endpoint: MODELS_ENDPOINT,
+            auth_header_attached: self.auth_header_attached,
+            auth_header_name: self.auth_header_name,
+            auth_mode: self.auth_mode.as_deref(),
+            auth_retry_after_unauthorized: None,
+            auth_recovery_mode: None,
+            auth_recovery_phase: None,
+            auth_connection_reused: None,
+            auth_request_id: response_debug.request_id.as_deref(),
+            auth_cf_ray: response_debug.cf_ray.as_deref(),
+            auth_error: response_debug.auth_error.as_deref(),
+            auth_error_code: response_debug.auth_error_code.as_deref(),
+            auth_recovery_followup_success: None,
+            auth_recovery_followup_status: None,
+        });
+    }
+}

 /// Strategy for refreshing available models.
 #[derive(Debug, Clone, Copy, PartialEq, Eq)]
@@ -330,11 +414,17 @@ impl ModelsManager {
        let _timer =
            codex_otel::start_global_timer("codex.remote_models.fetch_update.duration_ms", &[]);
        let auth = self.auth_manager.auth().await;
-        let auth_mode = self.auth_manager.auth_mode();
+        let auth_mode = auth.as_ref().map(CodexAuth::auth_mode);
        let api_provider = self.provider.to_api_provider(auth_mode)?;
        let api_auth = auth_provider_from_auth(auth.clone(), &self.provider)?;
        let transport = ReqwestTransport::new(build_reqwest_client());
-        let client = ModelsClient::new(transport, api_provider, api_auth);
+        let request_telemetry: Arc<dyn RequestTelemetry> = Arc::new(ModelsRequestTelemetry {
+            auth_mode: auth_mode.map(|mode| TelemetryAuthMode::from(mode).to_string()),
+            auth_header_attached: api_auth.auth_header_attached(),
+            auth_header_name: api_auth.auth_header_name(),
+        });
+        let client = ModelsClient::new(transport, api_provider, api_auth)
+            .with_telemetry(Some(request_telemetry));

        let client_version = crate::models_manager::client_version_to_whole();
        let (models, etag) = timeout(