Separate local and remote plugin analytics IDs (#29495)

## Why

Plugin analytics overloaded `plugin_id`: most events used the Codex
`<plugin>@<marketplace>` identity, while remote install events used the
backend plugin ID. That makes the same field change meaning across event
types and complicates downstream identity resolution.

This change makes the contract unambiguous:

- `plugin_id`: the local Codex `<plugin>@<marketplace>` identity, when
resolved
- `remote_plugin_id`: the backend plugin identity, when available

For a remote install failure that happens before plugin details resolve,
`plugin_id` is `null` and `remote_plugin_id` remains populated.

## What changed

All six plugin analytics events use the same identity contract:

- `codex_plugin_installed`
- `codex_plugin_install_failed`
- `codex_plugin_uninstalled`
- `codex_plugin_enabled`
- `codex_plugin_disabled`
- `codex_plugin_used`

Remote identity is resolved from the current installed-plugin snapshot
first, with persisted install metadata as fallback. The telemetry
metadata type keeps local identity optional for failures that occur
before remote details are available.

The app-server test client's manual analytics smokes now find remote
mutation events through `remote_plugin_id` and validate that `plugin_id`
remains local.

## Remote uninstall

Resolve and capture telemetry metadata before removing the local plugin
cache, then emit `codex_plugin_uninstalled` after the backend confirms
success. The event is also emitted when backend uninstall succeeds but
local cache cleanup reports `CacheRemove`.

If a concurrent remote-cache refresh removes the local bundle before
telemetry capture, the already-fetched remote plugin detail supplies
fallback capability metadata.

## Validation

- `just test -p codex-analytics` — 82 passed
- `just test -p codex-core-plugins` — 271 passed
- `just test -p codex-app-server-test-client` — 5 passed
- `just test -p codex-plugin` — 3 passed
- `just test -p codex-app-server plugin_install` — 37 passed
- `just test -p codex-app-server plugin_uninstall` — 10 passed

The production app-server install/uninstall flow was also exercised
against `plugins~Plugin_f1b845ac33888191ac156169c58733c2`
(`build-ios-apps@openai-curated-remote`), and the plugin's original
uninstalled state was restored.
This commit is contained in:
jameswt-oai
2026-06-23 12:27:14 -07:00
committed by GitHub
parent c5a9a95ab6
commit ff50b47dce
15 changed files with 427 additions and 91 deletions
+13 -10
View File
@@ -46,12 +46,13 @@ cargo run -p codex-app-server-test-client -- \
Use `--capture-file /tmp/plugin-analytics.jsonl` to select the output path.
The command validates one `codex_plugin_disabled`, `codex_plugin_enabled`, and
`codex_plugin_used` event with the expected local plugin identity and capability
metadata. The enabled and disabled events come from successful writes to the
temporary config; the command does not mutate the remote enabled state. It
prints the events and leaves the JSONL file in place for inspection. It does not
install or uninstall plugins and does not modify the profile's persistent
config.
`codex_plugin_used` event with the expected local and remote plugin identities
and capability metadata. Each event includes the local ID in `plugin_id` and the
backend ID in `remote_plugin_id`. The enabled and disabled events come from
successful writes to the temporary config; the command does not mutate the
remote enabled state. It prints the events and leaves the JSONL file in place
for inspection. It does not install or uninstall plugins and does not modify
the profile's persistent config.
### Testing remote install and uninstall analytics
@@ -63,9 +64,11 @@ or CI.
Choose a remote plugin that is available to the active account and is not
currently installed. The command refuses to run when the plugin is already
installed, installs it, validates `codex_plugin_installed`, uninstalls it, and
verifies that the original uninstalled state was restored. The current install
event uses the backend ID as `plugin_id`. Uninstall is part of cleanup but is
not yet an analytics assertion.
validates `codex_plugin_uninstalled`, and verifies that the original
uninstalled state was restored.
The mutation events include the local Codex ID in `plugin_id` and the backend ID
in `remote_plugin_id`.
`--remote-plugin-id` takes the backend ID, such as `plugins~Plugin_...`, not the
local `<plugin>@<marketplace>` ID.
@@ -83,7 +86,7 @@ Analytics use the normal queue, reduction, batching, and serialization path,
but the debug capture destination suppresses analytics network delivery. The
command prints one of these final states:
- `PASS`: the install event validated and the plugin is uninstalled.
- `PASS`: the install and uninstall events validated and the plugin is uninstalled.
- `FAIL-CLEAN`: validation failed, but the original uninstalled state was
restored.
- `FAIL-LOCAL-CACHE`: the backend is uninstalled, but local cleanup reported
@@ -35,7 +35,7 @@ pub(super) fn read_events_for_remote_plugin(
matching.extend(
events
.iter()
.filter(|event| event["event_params"]["plugin_id"] == remote_plugin_id)
.filter(|event| event["event_params"]["remote_plugin_id"] == remote_plugin_id)
.cloned(),
);
}
@@ -44,6 +44,7 @@ pub(super) fn read_events_for_remote_plugin(
pub(super) struct PluginEventIdentity<'a> {
pub(super) plugin_id: &'a str,
pub(super) remote_plugin_id: &'a str,
pub(super) plugin_name: &'a str,
pub(super) marketplace_name: &'a str,
}
@@ -52,25 +53,29 @@ pub(super) fn validate_mutation_events(
events: Vec<Value>,
expected: PluginEventIdentity<'_>,
) -> Result<Vec<Value>> {
let event_type = "codex_plugin_installed";
let matching = events
.iter()
.filter(|event| event["event_type"] == event_type)
.collect::<Vec<_>>();
let [event] = matching.as_slice() else {
bail!(
"expected exactly one `{event_type}` event for `{}`, found {}",
expected.plugin_id,
matching.len()
);
};
validate_event(event, &expected)?;
Ok(vec![(*event).clone()])
let mut validated = Vec::new();
for event_type in ["codex_plugin_installed", "codex_plugin_uninstalled"] {
let matching = events
.iter()
.filter(|event| event["event_type"] == event_type)
.collect::<Vec<_>>();
let [event] = matching.as_slice() else {
bail!(
"expected exactly one `{event_type}` event for `{}`, found {}",
expected.remote_plugin_id,
matching.len()
);
};
validate_event(event, &expected)?;
validated.push((*event).clone());
}
Ok(validated)
}
fn validate_event(event: &Value, expected: &PluginEventIdentity<'_>) -> Result<()> {
let params = &event["event_params"];
require_string(params, "plugin_id", expected.plugin_id)?;
require_string(params, "remote_plugin_id", expected.remote_plugin_id)?;
require_string(params, "plugin_name", expected.plugin_name)?;
require_string(params, "marketplace_name", expected.marketplace_name)?;
for field in [
@@ -14,15 +14,17 @@ const REMOTE_PLUGIN_ID: &str = "plugins~Plugin_test";
fn reads_and_validates_remote_plugin_mutation_events() {
let path = unique_capture_path("valid");
let installed = mutation_event("codex_plugin_installed");
let uninstalled = mutation_event("codex_plugin_uninstalled");
let unrelated = json!({
"event_type": "codex_plugin_installed",
"event_params": {
"plugin_id": "plugins~Plugin_other"
"plugin_id": "other@openai-curated-remote",
"remote_plugin_id": "plugins~Plugin_other"
}
});
let contents = [
json!({"events": [unrelated]}),
json!({"events": [installed]}),
json!({"events": [installed, uninstalled]}),
]
.into_iter()
.map(|payload| serde_json::to_string(&payload).expect("serialize capture payload"))
@@ -35,7 +37,7 @@ fn reads_and_validates_remote_plugin_mutation_events() {
let validated =
validate_mutation_events(events, expected_identity()).expect("validate mutation events");
assert_eq!(validated, vec![installed]);
assert_eq!(validated, vec![installed, uninstalled]);
fs::remove_file(path).expect("remove capture file");
}
@@ -62,7 +64,8 @@ fn mutation_event(event_type: &str) -> Value {
json!({
"event_type": event_type,
"event_params": {
"plugin_id": REMOTE_PLUGIN_ID,
"plugin_id": "sample@openai-curated-remote",
"remote_plugin_id": REMOTE_PLUGIN_ID,
"plugin_name": "sample",
"marketplace_name": "openai-curated-remote",
"has_skills": true,
@@ -75,7 +78,8 @@ fn mutation_event(event_type: &str) -> Value {
fn expected_identity() -> PluginEventIdentity<'static> {
PluginEventIdentity {
plugin_id: REMOTE_PLUGIN_ID,
plugin_id: "sample@openai-curated-remote",
remote_plugin_id: REMOTE_PLUGIN_ID,
plugin_name: "sample",
marketplace_name: "openai-curated-remote",
}
@@ -312,13 +312,19 @@ fn run_mutation_sequence(
state_err
}
})?;
wait_for_remote_plugin_event(
capture_path,
&expected.remote_plugin_id,
"codex_plugin_uninstalled",
)?;
let captured_events =
read_events_for_remote_plugin(capture_path, &expected.remote_plugin_id)?;
let events = validate_mutation_events(
captured_events,
PluginEventIdentity {
plugin_id: &expected.remote_plugin_id,
plugin_id: &expected.plugin_id,
remote_plugin_id: &expected.remote_plugin_id,
plugin_name: &expected.plugin_name,
marketplace_name: &expected.marketplace_name,
},
@@ -159,6 +159,7 @@ fn wait_for_plugin_usage(
#[derive(Debug)]
struct ExpectedPlugin {
plugin_id: String,
remote_plugin_id: String,
plugin_name: String,
marketplace_name: String,
}
@@ -208,13 +209,15 @@ fn expected_plugin(response: &PluginInstalledResponse, plugin_id: &str) -> Resul
plugin.availability
);
}
plugin
let remote_plugin_id = plugin
.remote_plugin_id
.as_ref()
.with_context(|| format!("plugin `{plugin_id}` does not have a remote plugin id"))?;
.with_context(|| format!("plugin `{plugin_id}` does not have a remote plugin id"))?
.clone();
Ok(ExpectedPlugin {
plugin_id: plugin.id.clone(),
remote_plugin_id,
plugin_name: plugin.name.clone(),
marketplace_name: marketplace.name.clone(),
})
@@ -444,6 +447,7 @@ fn event_count(events: &[Value], event_type: &str) -> usize {
fn validate_identity(event: &Value, expected: &ExpectedPlugin) -> Result<()> {
let params = &event["event_params"];
require_string(params, "plugin_id", &expected.plugin_id)?;
require_string(params, "remote_plugin_id", &expected.remote_plugin_id)?;
require_string(params, "plugin_name", &expected.plugin_name)?;
require_string(params, "marketplace_name", &expected.marketplace_name)
}