mirror of
https://github.com/pchuan98/codex.git
synced 2026-07-01 00:31:56 +08:00
9cd11e9e62
## Stack - Base: #27191 - This PR is the third vertical and should be reviewed against `jif/external-plugins-2`, not `main`. ## Why #27191 moves the host-owned Apps MCP registration behind an extension contributor, but deliberately preserves the existing endpoint-selection feature while that contribution contract lands. App-server can therefore resolve the server through extensions, yet the hosted plugin endpoint is still selected through temporary `apps_mcp_path_override` plumbing. That is not the long-term plugin model. A plugin can bundle skills, connectors, MCP servers, and hooks, and those components do not all need the same source or execution environment. In particular, an authenticated HTTP MCP server can expose plugin capabilities directly from a backend without an executor or an orchestrator filesystem. This PR completes that hosted vertical. App-server's MCP extension now owns the aggregate hosted plugin runtime at `/ps/mcp`. Connector actions continue to arrive as MCP tools, while backend-provided skills arrive as MCP resources and use Codex's existing resource list/read paths. No second backend client, skill filesystem, or generic plugin activation framework is introduced. The backend route remains the hosted implementation. This change replaces Codex's temporary endpoint-selection mechanism, not the service behind the endpoint. ## What changed ### Hosted plugin runtime The MCP extension now contributes `codex_apps` as the hosted plugin runtime rather than as a configurable Apps endpoint: - `https://chatgpt.com` resolves to `https://chatgpt.com/backend-api/ps/mcp`; - a bare custom ChatGPT base resolves to `/api/codex/ps/mcp`; - the existing product-SKU header and ChatGPT authentication behavior are preserved; - executor availability is never consulted for this streamable HTTP transport. The same MCP connection carries both component shapes supported by the hosted endpoint: - connector actions are discovered and invoked as MCP tools; - hosted skills are enumerated and read as MCP resources through the existing `list_mcp_resources` and `read_mcp_resource` paths. This keeps component access in the subsystem that already owns the protocol instead of downloading backend skills into an orchestrator filesystem or inventing a parallel hosted-skill client. ### Explicit runtime ordering `McpManager` now resolves the reserved `codex_apps` entry in three ordered phases: 1. install the legacy Apps fallback for compatibility; 2. apply ordered extension `Set` or `Remove` overlays; 3. apply the final ChatGPT-auth gate without synthesizing the server again. This ordering is important: - an ordinary configured or plugin MCP server cannot claim the auth-bearing `codex_apps` name; - an extension-contributed hosted runtime wins over the fallback; - an extension `Remove` remains authoritative; - a host without the MCP extension retains the legacy Apps endpoint and current local-only behavior. The temporary `legacy_apps_mcp_loader_enabled` coordination flag is no longer needed. ### Remove the path override The `apps_mcp_path_override` feature and its runtime plumbing are removed, including: - the feature registry entry and structured feature config; - `Config` and `McpConfig` fields; - config schema output; - config-lock materialization; - URL override handling in `codex-mcp`. Existing boolean and structured forms still deserialize as ignored compatibility input. They are omitted from new serialized config, and config-lock comparison normalizes the removed input so older locks remain replayable. ### App-server coverage App-server MCP fixtures now serve the hosted route at `/api/codex/ps/mcp`. Existing resource-read and tool/elicitation flows therefore exercise the extension-owned endpoint rather than succeeding through the legacy fallback. The stack also adds the missing `codex_chatgpt::connectors` re-export for the manager-backed connector helper introduced in #27191. ## Compatibility - App-server installs the extension and uses `/ps/mcp` for the hosted runtime. - CLI and other hosts that do not install the extension retain the legacy Apps endpoint. - Apps disabled or non-ChatGPT authentication removes `codex_apps` from the effective runtime view. - Existing local plugins, local skills, executor-selected skills, configured MCP servers, and MCP OAuth behavior are otherwise unchanged. - Backend plugin enablement remains account/workspace state owned by the hosted endpoint; this PR does not add thread-local backend plugin selection. ## Architectural fit The stack now proves two independent runtime shapes: 1. #27184 resolves filesystem-backed skills through the executor that owns a selected root. 2. #27191 and this PR resolve a backend-hosted HTTP MCP through an extension with no executor. Together they preserve the intended separation: - selection identifies a plugin/root when explicit selection is needed; - each component's owning extension resolves its concrete access mechanism; - execution stays with the runtime required by that component; - existing skills, MCP, connector, and hook subsystems remain the downstream consumers. ## Planned follow-ups 1. **Executor stdio MCP:** selecting an executor plugin registers a manifest-declared stdio MCP server and executes it in the environment that owns the plugin. 2. **Optional backend selection:** only if CCA needs thread-local selection distinct from backend account/workspace enablement, add a concrete backend-owned capability location and surface those selected skills through the skills catalog. 3. **Connector metadata and hooks:** activate those plugin components through their existing owning subsystems, with executor hooks remaining environment-bound. 4. **Propagation and persistence:** define explicit resume, fork, subagent, refresh, and environment-removal semantics once selected roots have multiple real consumers. 5. **Local convergence:** migrate legacy local skill, MCP, connector, and hook paths behind their owning extensions one vertical at a time, then remove duplicate core managers and compatibility plumbing after parity. ## Verification Coverage in this change exercises: - extension-owned `/backend-api/ps/mcp` registration without an executor; - preservation of the legacy endpoint in hosts without the extension; - extension `Set` and `Remove` precedence over the legacy fallback; - ChatGPT-auth gating for the reserved server; - hosted MCP resource reads with and without an active thread; - connector tool invocation and MCP elicitation through the hosted route; - ignored boolean and structured forms of the removed path override; - config-lock replay compatibility for the removed feature. `cargo check -p codex-features -p codex-mcp-extension -p codex-app-server` passes. Tests and Clippy were not run locally under the current development instruction; CI provides the full validation pass.
387 lines
15 KiB
Rust
387 lines
15 KiB
Rust
use anyhow::Context;
|
|
use codex_config::config_toml::ConfigLockfileToml;
|
|
use codex_config::config_toml::ConfigToml;
|
|
use codex_config::types::MemoriesToml;
|
|
use codex_features::Feature;
|
|
use codex_features::FeatureToml;
|
|
use codex_features::FeaturesToml;
|
|
use codex_features::MultiAgentV2ConfigToml;
|
|
use codex_protocol::ThreadId;
|
|
|
|
use crate::config::Config;
|
|
use crate::config_lock::ConfigLockReplayOptions;
|
|
use crate::config_lock::clear_config_lock_debug_controls;
|
|
use crate::config_lock::config_lockfile;
|
|
use crate::config_lock::toml_round_trip;
|
|
use crate::config_lock::validate_config_lock_replay;
|
|
|
|
use super::SessionConfiguration;
|
|
|
|
pub(crate) async fn validate_config_lock_if_configured(
|
|
session_configuration: &SessionConfiguration,
|
|
) -> anyhow::Result<()> {
|
|
if session_configuration.session_source.is_non_root_agent() {
|
|
return Ok(());
|
|
}
|
|
let Some(expected) = session_configuration
|
|
.original_config_do_not_use
|
|
.config_lock_toml
|
|
.as_ref()
|
|
else {
|
|
return Ok(());
|
|
};
|
|
let actual = session_configuration.to_config_lockfile_toml()?;
|
|
let config = session_configuration.original_config_do_not_use.as_ref();
|
|
let options = ConfigLockReplayOptions {
|
|
allow_codex_version_mismatch: config.config_lock_allow_codex_version_mismatch,
|
|
};
|
|
validate_config_lock_replay(expected, &actual, options)
|
|
.context("config lock replay validation failed")?;
|
|
Ok(())
|
|
}
|
|
|
|
pub(crate) async fn export_config_lock_if_configured(
|
|
session_configuration: &SessionConfiguration,
|
|
conversation_id: ThreadId,
|
|
) -> anyhow::Result<()> {
|
|
let config = session_configuration.original_config_do_not_use.as_ref();
|
|
let Some(export_dir) = config.config_lock_export_dir.as_ref() else {
|
|
return Ok(());
|
|
};
|
|
|
|
let lock = session_configuration.to_config_lockfile_toml()?;
|
|
let lock = toml::to_string_pretty(&lock).context("failed to serialize config lock")?;
|
|
let path = export_dir.join(format!("{conversation_id}.config.lock.toml"));
|
|
|
|
tokio::fs::create_dir_all(export_dir)
|
|
.await
|
|
.with_context(|| {
|
|
format!(
|
|
"failed to create config lock export directory {}",
|
|
export_dir.display()
|
|
)
|
|
})?;
|
|
tokio::fs::write(&path, lock)
|
|
.await
|
|
.with_context(|| format!("failed to write config lock to {}", path.display()))?;
|
|
|
|
Ok(())
|
|
}
|
|
|
|
impl SessionConfiguration {
|
|
pub(crate) fn to_config_lockfile_toml(&self) -> anyhow::Result<ConfigLockfileToml> {
|
|
Ok(config_lockfile(session_configuration_to_lock_config_toml(
|
|
self,
|
|
)?))
|
|
}
|
|
}
|
|
|
|
fn session_configuration_to_lock_config_toml(
|
|
sc: &SessionConfiguration,
|
|
) -> anyhow::Result<ConfigToml> {
|
|
let config = sc.original_config_do_not_use.as_ref();
|
|
// Start from the resolved layer stack, then patch in values that are only
|
|
// known after session setup. Export and replay validation both use this
|
|
// path, so every field here is part of the lockfile contract.
|
|
let mut lock_config: ConfigToml = config
|
|
.config_layer_stack
|
|
.effective_config()
|
|
.try_into()
|
|
.context("failed to deserialize effective config for config lock")?;
|
|
|
|
if config.config_lock_save_fields_resolved_from_model_catalog {
|
|
save_session_resolved_fields(sc, &mut lock_config);
|
|
}
|
|
|
|
save_config_resolved_fields(config, &mut lock_config)?;
|
|
drop_lockfile_inputs(&mut lock_config);
|
|
|
|
Ok(lock_config)
|
|
}
|
|
|
|
/// Saves values chosen during session construction from the model catalog,
|
|
/// collaboration mode, and resolved prompt setup.
|
|
///
|
|
/// These values are not always present in the raw layer stack, so copy them
|
|
/// from the live session when the lockfile should be fully self-contained.
|
|
fn save_session_resolved_fields(sc: &SessionConfiguration, lock_config: &mut ConfigToml) {
|
|
lock_config.model = Some(sc.collaboration_mode.model().to_string());
|
|
lock_config.model_reasoning_effort = sc.collaboration_mode.reasoning_effort();
|
|
lock_config.model_reasoning_summary = sc.model_reasoning_summary;
|
|
lock_config.service_tier = sc.service_tier.clone();
|
|
lock_config.instructions = Some(sc.base_instructions.clone());
|
|
lock_config.developer_instructions = sc.developer_instructions.clone();
|
|
lock_config.compact_prompt = sc.compact_prompt.clone();
|
|
lock_config.personality = sc.personality;
|
|
lock_config.approval_policy = Some(sc.approval_policy.value());
|
|
lock_config.approvals_reviewer = Some(sc.approvals_reviewer);
|
|
}
|
|
|
|
/// Saves values stored on `Config` after higher-level resolution,
|
|
/// normalization, defaulting, or feature materialization.
|
|
///
|
|
/// Persist the resolved representation so replay compares against the behavior
|
|
/// Codex actually ran with, not only the user-authored TOML inputs.
|
|
fn save_config_resolved_fields(
|
|
config: &Config,
|
|
lock_config: &mut ConfigToml,
|
|
) -> anyhow::Result<()> {
|
|
lock_config.web_search = Some(config.web_search_mode.value());
|
|
lock_config.model_provider = Some(config.model_provider_id.clone());
|
|
lock_config.plan_mode_reasoning_effort = config.plan_mode_reasoning_effort.clone();
|
|
lock_config.model_verbosity = config.model_verbosity;
|
|
lock_config.include_permissions_instructions = Some(config.include_permissions_instructions);
|
|
lock_config.include_apps_instructions = Some(config.include_apps_instructions);
|
|
lock_config.include_collaboration_mode_instructions =
|
|
Some(config.include_collaboration_mode_instructions);
|
|
lock_config.include_environment_context = Some(config.include_environment_context);
|
|
lock_config.background_terminal_max_timeout = Some(config.background_terminal_max_timeout);
|
|
|
|
// Feature aliases and feature configs need to be written in their resolved
|
|
// form; otherwise replay can drift when a legacy key maps to the same
|
|
// runtime feature.
|
|
let features = lock_config
|
|
.features
|
|
.get_or_insert_with(FeaturesToml::default);
|
|
features.materialize_resolved_enabled(config.features.get());
|
|
let mut multi_agent_v2: MultiAgentV2ConfigToml =
|
|
resolved_config_to_toml(&config.multi_agent_v2, "features.multi_agent_v2")?;
|
|
multi_agent_v2.enabled = Some(config.features.enabled(Feature::MultiAgentV2));
|
|
features.multi_agent_v2 = Some(FeatureToml::Config(multi_agent_v2));
|
|
lock_config.memories = Some(resolved_config_to_toml::<MemoriesToml>(
|
|
&config.memories,
|
|
"memories",
|
|
)?);
|
|
|
|
let agents = lock_config.agents.get_or_insert_with(Default::default);
|
|
// Multi-agent v2 owns thread fanout through its feature config. Preserve
|
|
// the legacy agents.max_threads setting only when v2 is disabled.
|
|
agents.max_threads = if config.features.enabled(Feature::MultiAgentV2) {
|
|
None
|
|
} else {
|
|
config.agent_max_threads
|
|
};
|
|
agents.max_depth = Some(config.agent_max_depth);
|
|
agents.job_max_runtime_seconds = config.agent_job_max_runtime_seconds;
|
|
agents.interrupt_message = Some(config.agent_interrupt_message_enabled);
|
|
|
|
lock_config
|
|
.skills
|
|
.get_or_insert_with(Default::default)
|
|
.include_instructions = Some(config.include_skill_instructions);
|
|
|
|
Ok(())
|
|
}
|
|
|
|
fn drop_lockfile_inputs(lock_config: &mut ConfigToml) {
|
|
// The lockfile should contain replayable values, not the profile,
|
|
// debug-control, file-include, and environment-specific inputs that
|
|
// produced those values in the original session.
|
|
lock_config.profile = None;
|
|
lock_config.profiles.clear();
|
|
clear_config_lock_debug_controls(lock_config);
|
|
lock_config.model_instructions_file = None;
|
|
lock_config.experimental_compact_prompt_file = None;
|
|
lock_config.model_catalog_json = None;
|
|
lock_config.sandbox_mode = None;
|
|
lock_config.sandbox_workspace_write = None;
|
|
lock_config.default_permissions = None;
|
|
lock_config.permissions = None;
|
|
lock_config.experimental_use_unified_exec_tool = None;
|
|
}
|
|
|
|
fn resolved_config_to_toml<Toml>(
|
|
value: &impl serde::Serialize,
|
|
label: &'static str,
|
|
) -> anyhow::Result<Toml>
|
|
where
|
|
Toml: serde::de::DeserializeOwned + serde::Serialize,
|
|
{
|
|
toml_round_trip(value, label).map_err(anyhow::Error::from)
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
use pretty_assertions::assert_eq;
|
|
use std::sync::Arc;
|
|
|
|
#[tokio::test]
|
|
async fn lock_contains_prompts_and_materializes_features() {
|
|
let mut sc = crate::session::tests::make_session_configuration_for_tests().await;
|
|
sc.base_instructions = "resolved instructions".to_string();
|
|
sc.developer_instructions = Some("resolved developer instructions".to_string());
|
|
sc.compact_prompt = Some("resolved compact prompt".to_string());
|
|
|
|
let lockfile = sc.to_config_lockfile_toml().expect("lock should serialize");
|
|
let lock = &lockfile.config;
|
|
|
|
assert_eq!(lock.instructions, Some(sc.base_instructions.clone()));
|
|
assert_eq!(lock.developer_instructions, sc.developer_instructions);
|
|
assert_eq!(lock.compact_prompt, sc.compact_prompt);
|
|
assert_eq!(lock.model, Some(sc.collaboration_mode.model().to_string()));
|
|
assert_eq!(
|
|
lock.model_reasoning_effort,
|
|
sc.collaboration_mode.reasoning_effort()
|
|
);
|
|
assert_eq!(lock.profile, None);
|
|
assert!(lock.profiles.is_empty());
|
|
assert!(
|
|
lock.debug
|
|
.as_ref()
|
|
.is_none_or(|debug| debug.config_lockfile.is_none())
|
|
);
|
|
assert!(lock.memories.is_some());
|
|
|
|
let features = lock
|
|
.features
|
|
.as_ref()
|
|
.expect("lock should materialize feature states");
|
|
let feature_entries = features.entries();
|
|
for spec in codex_features::FEATURES {
|
|
assert_eq!(
|
|
feature_entries.get(spec.key),
|
|
Some(&sc.original_config_do_not_use.features.enabled(spec.id)),
|
|
"{}",
|
|
spec.key
|
|
);
|
|
}
|
|
assert_eq!(
|
|
features.code_mode,
|
|
Some(FeatureToml::Enabled(
|
|
sc.original_config_do_not_use
|
|
.features
|
|
.enabled(Feature::CodeMode)
|
|
))
|
|
);
|
|
|
|
let multi_agent_v2 = features
|
|
.multi_agent_v2
|
|
.as_ref()
|
|
.expect("multi_agent_v2 config should be materialized");
|
|
assert!(matches!(
|
|
multi_agent_v2,
|
|
FeatureToml::Config(MultiAgentV2ConfigToml {
|
|
enabled: Some(false),
|
|
max_concurrent_threads_per_session: Some(_),
|
|
min_wait_timeout_ms: Some(_),
|
|
max_wait_timeout_ms: Some(_),
|
|
default_wait_timeout_ms: Some(_),
|
|
usage_hint_enabled: Some(_),
|
|
hide_spawn_agent_metadata: Some(_),
|
|
..
|
|
})
|
|
));
|
|
|
|
assert_eq!(lockfile.version, crate::config_lock::CONFIG_LOCK_VERSION);
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn lock_skips_session_values_when_model_catalog_fields_are_not_saved() {
|
|
let mut sc = crate::session::tests::make_session_configuration_for_tests().await;
|
|
let mut config = (*sc.original_config_do_not_use).clone();
|
|
config.config_lock_save_fields_resolved_from_model_catalog = false;
|
|
sc.original_config_do_not_use = Arc::new(config);
|
|
sc.base_instructions = "catalog instructions".to_string();
|
|
sc.developer_instructions = Some("catalog developer instructions".to_string());
|
|
sc.compact_prompt = Some("catalog compact prompt".to_string());
|
|
sc.service_tier = Some("flex".to_string());
|
|
|
|
let lockfile = sc.to_config_lockfile_toml().expect("lock should serialize");
|
|
let lock = &lockfile.config;
|
|
|
|
assert_eq!(lock.model, None);
|
|
assert_eq!(lock.model_reasoning_effort, None);
|
|
assert_eq!(lock.model_reasoning_summary, None);
|
|
assert_eq!(lock.service_tier, None);
|
|
assert_eq!(lock.instructions, None);
|
|
assert_eq!(lock.developer_instructions, None);
|
|
assert_eq!(lock.compact_prompt, None);
|
|
assert_eq!(lock.personality, None);
|
|
assert_eq!(lock.approval_policy, None);
|
|
assert_eq!(lock.approvals_reviewer, None);
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn lock_validation_reports_config_diff() {
|
|
let sc = crate::session::tests::make_session_configuration_for_tests().await;
|
|
let expected = sc.to_config_lockfile_toml().expect("lock should serialize");
|
|
let mut actual = expected.clone();
|
|
actual.config.model = Some("different-model".to_string());
|
|
|
|
let error =
|
|
validate_config_lock_replay(&expected, &actual, ConfigLockReplayOptions::default())
|
|
.expect_err("config drift should fail");
|
|
let message = error.to_string();
|
|
assert!(
|
|
message.contains("replayed effective config does not match config lock"),
|
|
"{message}"
|
|
);
|
|
assert!(message.contains("model = "), "{message}");
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn lock_validation_ignores_removed_apps_mcp_path_override() {
|
|
let sc = crate::session::tests::make_session_configuration_for_tests().await;
|
|
let actual = sc.to_config_lockfile_toml().expect("lock should serialize");
|
|
let mut expected_value = toml::Value::try_from(&actual).expect("lock should become TOML");
|
|
expected_value["config"]["features"]
|
|
.as_table_mut()
|
|
.expect("features should be a table")
|
|
.insert(
|
|
"apps_mcp_path_override".to_string(),
|
|
toml::Value::Table(toml::Table::from_iter([
|
|
("enabled".to_string(), toml::Value::Boolean(true)),
|
|
(
|
|
"path".to_string(),
|
|
toml::Value::String("/custom/mcp".to_string()),
|
|
),
|
|
])),
|
|
);
|
|
let expected: ConfigLockfileToml = expected_value
|
|
.try_into()
|
|
.expect("lock with removed input should deserialize");
|
|
|
|
validate_config_lock_replay(&expected, &actual, ConfigLockReplayOptions::default())
|
|
.expect("removed compatibility input should not cause lock drift");
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn lock_validation_rejects_codex_version_mismatch_by_default() {
|
|
let sc = crate::session::tests::make_session_configuration_for_tests().await;
|
|
let mut expected = sc.to_config_lockfile_toml().expect("lock should serialize");
|
|
expected.codex_version = "older-version".to_string();
|
|
let actual = sc.to_config_lockfile_toml().expect("lock should serialize");
|
|
|
|
let error =
|
|
validate_config_lock_replay(&expected, &actual, ConfigLockReplayOptions::default())
|
|
.expect_err("version drift should fail");
|
|
let message = error.to_string();
|
|
assert!(
|
|
message.contains("config lock Codex version mismatch"),
|
|
"{message}"
|
|
);
|
|
assert!(
|
|
message.contains("debug.config_lockfile.allow_codex_version_mismatch=true"),
|
|
"{message}"
|
|
);
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn lock_validation_can_ignore_codex_version_mismatch() {
|
|
let sc = crate::session::tests::make_session_configuration_for_tests().await;
|
|
let mut expected = sc.to_config_lockfile_toml().expect("lock should serialize");
|
|
expected.codex_version = "older-version".to_string();
|
|
let actual = sc.to_config_lockfile_toml().expect("lock should serialize");
|
|
|
|
validate_config_lock_replay(
|
|
&expected,
|
|
&actual,
|
|
ConfigLockReplayOptions {
|
|
allow_codex_version_mismatch: true,
|
|
},
|
|
)
|
|
.expect("version drift should be ignored");
|
|
}
|
|
}
|