Commit Graph

10 Commits

  • [codex-analytics] guardian review analytics events emission (#17693)
    ## Why
    
    Guardian approvals now run as review sessions, but Codex analytics did
    not have a terminal event for those reviews. That made it hard to
    measure approval outcomes, failure modes, Guardian session reuse, model
    metadata, token usage, and timing separately from the parent turn.
    
    ## What changed
    
    Adds `codex_guardian_review` analytics emission for Guardian approval
    reviews. The event is emitted from the Guardian review path with review
    identity, target item id, approval request source, a PII-minimized
    reviewed-action shape, terminal decision/status, failure reason,
    Guardian assessment fields, Guardian session metadata, token usage, and
    timing metadata.
    
    The reviewed-action payload intentionally omits high-risk fields such as
    shell commands, working directories, argv, file paths, network
    targets/hosts, rationale, retry reason, and permission justifications.
    It also classifies prompt-build failures separately from Guardian
    session/runtime failures so fail-closed cases are distinguishable in
    analytics.
    
    ## Verification
    
    - Guardian review analytics tests cover terminal success,
    timeout/cancel/fail-closed paths, session metadata, and token usage
    plumbing.
    - `cargo clippy -p codex-core --lib --tests -- -D warnings`
    
    ---
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/17693).
    * #17696
    * #17695
    * __->__ #17693
  • [codex-analytics] guardian review analytics schema polishing (#17692)
    ## Why
    
    Guardian review analytics needs a Rust event shape that matches the
    backend schema while avoiding unnecessary PII exposure from reviewed
    tool calls. This PR narrows the analytics payload to the fields we
    intend to emit and keeps shared Guardian assessment enums in protocol
    instead of duplicating equivalent analytics-only enums.
    
    ## What changed
    
    - Uses protocol Guardian enums directly for `risk_level`,
    `user_authorization`, `outcome`, and command source values.
    - Removes high-risk reviewed-action fields from the analytics payload,
    including raw commands, display strings, working directories, file
    paths, network targets/hosts, justification text, retry reason, and
    rationale text.
    - Makes `target_item_id` and `tool_call_count` nullable so the Codex
    event can represent cases where the app-server protocol or producer does
    not have those values.
    - Keeps lower-risk structured reviewed-action metadata such as sandbox
    permissions, permission profile, `tty`, `execve` source/program, network
    protocol/port, and MCP connector/tool labels.
    - Adds an analytics reducer/client test covering `codex_guardian_review`
    serialization with an optional `target_item_id` and absent removed
    fields.
    
    ## Verification
    
    - `cargo test -p codex-analytics
    guardian_review_event_ingests_custom_fact_with_optional_target_item`
    - `cargo fmt --check`
    
    ---
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/17692).
    * #17696
    * #17695
    * #17693
    * __->__ #17692
  • Add codex_hook_run analytics event (#17996)
    # Why
    Add product analytics for hook handler executions so we can understand
    which hooks are running, where they came from, and whether they
    completed, failed, stopped, or blocked work.
    
    # What
    - add the new `codex_hook_run` analytics event and payload plumbing in
    `codex-rs/analytics`
    - emit hook-run analytics from the shared hook completion path in
    `codex-rs/core`
    - classify hook source from the loaded hook path as `system`, `user`,
    `project`, or `unknown`
    
    ```
    {
      "event_type": "codex_hook_run",
      "event_params": {
        "thread_id": "string",
        "turn_id": "string",
        "model_slug": "string",
        "hook_name": "string, // any HookEventName
        "hook_source": "system | user | project | unknown",
        "status": "completed | failed | stopped | blocked"
      }
    }
    ```
    
    ---------
    
    Co-authored-by: Codex <noreply@openai.com>
  • [codex-analytics] feature plumbing and emittance (#16640)
    ---
    [//]: # (BEGIN SAPLING FOOTER)
    Stack created with [Sapling](https://sapling-scm.com). Best reviewed
    with [ReviewStack](https://reviewstack.dev/openai/codex/pull/16640).
    * #16870
    * #16706
    * #16641
    * __->__ #16640
  • feat(analytics): add guardian review event schema (#17055)
    Just the analytics schema definition for guardian evaluations. No wiring
    done yet.
  • [codex-analytics] add compaction analytics event (#17155)
    - event for compaction analytics
    - introduces thread-connection and thread metadata caches for data
    denormalization, expected to be useful for denormalization onto core
    emitted events in general
    - threads analytics event client into core (mirrors approved
    implementation in #16640)
    - denormalizes key thread metadata: thread_source, subagent_source,
    parent_thread_id, as well as app-server client and runtime metadata)
    - compaction strategy defaults to memento, forward compatible with
    expected prefill_compaction strategy
    
    1. Manual standalone compact, local
    `INFO | 2026-04-09 17:35:50 | codex_backend.routers.analytics_events |
    analytics_events.track_analytics_events:526 | Tracked
    codex_compaction_event event params={'thread_id':
    '019d74d0-5cfb-70c0-bef9-165c3bf9b2df', 'turn_id':
    '019d74d0-d7f6-7c81-acc6-aae2030243d6', 'product_surface': 'codex',
    'app_server_client': {'product_client_id': 'CODEX_CLI', 'client_name':
    'codex-tui', 'client_version': '0.0.0', 'rpc_transport': 'in_process',
    'experimental_api_enabled': True}, 'runtime': {'codex_rs_version':
    '0.0.0', 'runtime_os': 'macos', 'runtime_os_version': '26.4.0',
    'runtime_arch': 'aarch64'}, 'trigger': 'manual', 'reason':
    'user_requested', 'implementation': 'responses', 'phase':
    'standalone_turn', 'strategy': 'memento', 'status': 'completed',
    'active_context_tokens_before': 20170, 'active_context_tokens_after':
    4830, 'started_at': 1775781337, 'completed_at': 1775781350,
    'thread_source': 'user', 'subagent_source': None, 'parent_thread_id':
    None, 'error': None, 'duration_ms': 13524} | `
    
    2. Auto pre-turn compact, local
    `INFO | 2026-04-09 17:37:30 | codex_backend.routers.analytics_events |
    analytics_events.track_analytics_events:526 | Tracked
    codex_compaction_event event params={'thread_id':
    '019d74d2-45ef-71d1-9c93-23cc0c13d988', 'turn_id':
    '019d74d2-7b42-7372-9f0e-c0da3f352328', 'product_surface': 'codex',
    'app_server_client': {'product_client_id': 'CODEX_CLI', 'client_name':
    'codex-tui', 'client_version': '0.0.0', 'rpc_transport': 'in_process',
    'experimental_api_enabled': True}, 'runtime': {'codex_rs_version':
    '0.0.0', 'runtime_os': 'macos', 'runtime_os_version': '26.4.0',
    'runtime_arch': 'aarch64'}, 'trigger': 'auto', 'reason':
    'context_limit', 'implementation': 'responses', 'phase': 'pre_turn',
    'strategy': 'memento', 'status': 'completed',
    'active_context_tokens_before': 20063, 'active_context_tokens_after':
    4822, 'started_at': 1775781444, 'completed_at': 1775781449,
    'thread_source': 'user', 'subagent_source': None, 'parent_thread_id':
    None, 'error': None, 'duration_ms': 5497} | `
    
    3. Auto mid-turn compact, local
    `INFO | 2026-04-09 17:38:28 | codex_backend.routers.analytics_events |
    analytics_events.track_analytics_events:526 | Tracked
    codex_compaction_event event params={'thread_id':
    '019d74d3-212f-7a20-8c0a-4816a978675e', 'turn_id':
    '019d74d3-3ee1-7462-89f6-2ffbeefcd5e3', 'product_surface': 'codex',
    'app_server_client': {'product_client_id': 'CODEX_CLI', 'client_name':
    'codex-tui', 'client_version': '0.0.0', 'rpc_transport': 'in_process',
    'experimental_api_enabled': True}, 'runtime': {'codex_rs_version':
    '0.0.0', 'runtime_os': 'macos', 'runtime_os_version': '26.4.0',
    'runtime_arch': 'aarch64'}, 'trigger': 'auto', 'reason':
    'context_limit', 'implementation': 'responses', 'phase': 'mid_turn',
    'strategy': 'memento', 'status': 'completed',
    'active_context_tokens_before': 20325, 'active_context_tokens_after':
    14641, 'started_at': 1775781500, 'completed_at': 1775781508,
    'thread_source': 'user', 'subagent_source': None, 'parent_thread_id':
    None, 'error': None, 'duration_ms': 7507} | `
    
    4. Remote /responses/compact, manual standalone
    `INFO | 2026-04-09 17:40:20 | codex_backend.routers.analytics_events |
    analytics_events.track_analytics_events:526 | Tracked
    codex_compaction_event event params={'thread_id':
    '019d74d4-7a11-78a1-89f7-0535a1149416', 'turn_id':
    '019d74d4-e087-7183-9c20-b1e40b7578c0', 'product_surface': 'codex',
    'app_server_client': {'product_client_id': 'CODEX_CLI', 'client_name':
    'codex-tui', 'client_version': '0.0.0', 'rpc_transport': 'in_process',
    'experimental_api_enabled': True}, 'runtime': {'codex_rs_version':
    '0.0.0', 'runtime_os': 'macos', 'runtime_os_version': '26.4.0',
    'runtime_arch': 'aarch64'}, 'trigger': 'manual', 'reason':
    'user_requested', 'implementation': 'responses_compact', 'phase':
    'standalone_turn', 'strategy': 'memento', 'status': 'completed',
    'active_context_tokens_before': 23461, 'active_context_tokens_after':
    6171, 'started_at': 1775781601, 'completed_at': 1775781620,
    'thread_source': 'user', 'subagent_source': None, 'parent_thread_id':
    None, 'error': None, 'duration_ms': 18971} | `
  • [codex-analytics] subagent analytics (#15915)
    - creates custom event that emits subagent thread analytics from core
    - wires client metadata (`product_client_id, client_name,
    client_version`), through from app-server
    - creates `created_at `timestamp in core
    - subagent analytics are behind `FeatureFlag::GeneralAnalytics`
    
    PR stack
    - [[telemetry] thread events
    #15690](https://github.com/openai/codex/pull/15690)
    - --> [[telemetry] subagent events
    #15915](https://github.com/openai/codex/pull/15915)
    - [[telemetry] turn events
    #15591](https://github.com/openai/codex/pull/15591)
    - [[telemetry] steer events
    #15697](https://github.com/openai/codex/pull/15697)
    - [[telemetry] queued prompt data
    #15804](https://github.com/openai/codex/pull/15804)
    
    Notes:
    - core does not spawn a subagent thread for compact, but represented in
    mapping for consistency
    
    `INFO | 2026-04-01 13:08:12 | codex_backend.routers.analytics_events |
    analytics_events.track_analytics_events:399 | Tracked
    codex_thread_initialized event params={'thread_id':
    '019d4aa9-233b-70f2-a958-c3dbae1e30fa', 'product_surface': 'codex',
    'app_server_client': {'product_client_id': 'CODEX_CLI', 'client_name':
    'codex-tui', 'client_version': '0.0.0', 'rpc_transport': 'in_process',
    'experimental_api_enabled': None}, 'runtime': {'codex_rs_version':
    '0.0.0', 'runtime_os': 'macos', 'runtime_os_version': '26.4.0',
    'runtime_arch': 'aarch64'}, 'model': 'gpt-5.3-codex', 'ephemeral':
    False, 'initialization_mode': 'new', 'created_at': 1775074091,
    'thread_source': 'subagent', 'subagent_source': 'thread_spawn',
    'parent_thread_id': '019d4aa8-51ec-77e3-bafb-2c1b8e29e385'} | `
    
    `INFO | 2026-04-01 13:08:41 | codex_backend.routers.analytics_events |
    analytics_events.track_analytics_events:399 | Tracked
    codex_thread_initialized event params={'thread_id':
    '019d4aa9-94e3-75f1-8864-ff8ad0e55e1e', 'product_surface': 'codex',
    'app_server_client': {'product_client_id': 'CODEX_CLI', 'client_name':
    'codex-tui', 'client_version': '0.0.0', 'rpc_transport': 'in_process',
    'experimental_api_enabled': None}, 'runtime': {'codex_rs_version':
    '0.0.0', 'runtime_os': 'macos', 'runtime_os_version': '26.4.0',
    'runtime_arch': 'aarch64'}, 'model': 'gpt-5.3-codex', 'ephemeral':
    False, 'initialization_mode': 'new', 'created_at': 1775074120,
    'thread_source': 'subagent', 'subagent_source': 'review',
    'parent_thread_id': None} | `
    
    ---------
    
    Co-authored-by: jif-oai <jif@openai.com>
    Co-authored-by: Michael Bolin <mbolin@openai.com>
  • [codex-analytics] thread events (#15690)
    - add event for thread initialization
    - thread/start, thread/fork, thread/resume
    - feature flagged behind `FeatureFlag::GeneralAnalytics`
    - does not yet support threads started by subagents
    
    PR stack:
    - --> [[telemetry] thread events
    #15690](https://github.com/openai/codex/pull/15690)
    - [[telemetry] subagent events
    #15915](https://github.com/openai/codex/pull/15915)
    - [[telemetry] turn events
    #15591](https://github.com/openai/codex/pull/15591)
    - [[telemetry] steer events
    #15697](https://github.com/openai/codex/pull/15697)
    - [[telemetry] queued prompt data
    #15804](https://github.com/openai/codex/pull/15804)
    
    
    Sample extracted logs in Codex-backend
    ```
    INFO     | 2026-03-29 16:39:37 | codex_backend.routers.analytics_events | analytics_events.track_analytics_events:398 | Tracked analytics event codex_thread_initialized thread_id=019d3bf7-9f5f-7f82-9877-6d48d1052531 product_surface=codex product_client_id=CODEX_CLI client_name=codex-tui client_version=0.0.0 rpc_transport=in_process experimental_api_enabled=True codex_rs_version=0.0.0 runtime_os=macos runtime_os_version=26.4.0 runtime_arch=aarch64 model=gpt-5.3-codex ephemeral=False thread_source=user initialization_mode=new subagent_source=None parent_thread_id=None created_at=1774827577 | 
    INFO     | 2026-03-29 16:45:46 | codex_backend.routers.analytics_events | analytics_events.track_analytics_events:398 | Tracked analytics event codex_thread_initialized thread_id=019d3b84-5731-79d0-9b3b-9c6efe5f5066 product_surface=codex product_client_id=CODEX_CLI client_name=codex-tui client_version=0.0.0 rpc_transport=in_process experimental_api_enabled=True codex_rs_version=0.0.0 runtime_os=macos runtime_os_version=26.4.0 runtime_arch=aarch64 model=gpt-5.3-codex ephemeral=False thread_source=user initialization_mode=resumed subagent_source=None parent_thread_id=None created_at=1774820022 | 
    INFO     | 2026-03-29 16:45:49 | codex_backend.routers.analytics_events | analytics_events.track_analytics_events:398 | Tracked analytics event codex_thread_initialized thread_id=019d3bfd-4cd6-7c12-a13e-48cef02e8c4d product_surface=codex product_client_id=CODEX_CLI client_name=codex-tui client_version=0.0.0 rpc_transport=in_process experimental_api_enabled=True codex_rs_version=0.0.0 runtime_os=macos runtime_os_version=26.4.0 runtime_arch=aarch64 model=gpt-5.3-codex ephemeral=False thread_source=user initialization_mode=forked subagent_source=None parent_thread_id=None created_at=1774827949 | 
    INFO     | 2026-03-29 17:20:29 | codex_backend.routers.analytics_events | analytics_events.track_analytics_events:398 | Tracked analytics event codex_thread_initialized thread_id=019d3c1d-0412-7ed2-ad24-c9c0881a36b0 product_surface=codex product_client_id=CODEX_SERVICE_EXEC client_name=codex_exec client_version=0.0.0 rpc_transport=in_process experimental_api_enabled=True codex_rs_version=0.0.0 runtime_os=macos runtime_os_version=26.4.0 runtime_arch=aarch64 model=gpt-5.3-codex ephemeral=False thread_source=user initialization_mode=new subagent_source=None parent_thread_id=None created_at=1774830027 | 
    ```
    
    Notes
    - `product_client_id` gets canonicalized in codex-backend
    - subagent threads are addressed in a following pr
  • [codex-analytics] refactor analytics to use reducer architecture (#16225)
    - rework codex analytics crate to use reducer / publish architecture
    - in anticipation of extensive codex analytics
  • Extract codex-analytics crate (#15748)
    ## Summary
    - move the analytics events client into codex-analytics
    - update codex-core and app-server callsites to use the new crate
    
    ## Testing
    - CI
    
    ---------
    
    Co-authored-by: Codex <noreply@openai.com>