mirror of https://github.com/pchuan98/codex.git synced 2026-07-01 00:31:56 +08:00

Files

T

jif 89ac3ec27c Load selected executor skills through extensions (#27184 )

## Why

CCA is moving toward a split runtime where the orchestrator may not have
a filesystem, while executors can expose preinstalled plugins and
skills. A thread therefore needs to select capabilities without asking
app-server or core to interpret executor-owned paths through the
orchestrator's filesystem.

The longer-term model is broader than executor skills:

- A plugin is a bundle of skills, MCP servers, connectors/apps, and
hooks.
- A plugin root can be local, executor-owned, or hosted by a backend.
- Components inside one plugin can use different access and execution
mechanisms. A skill may be read from a filesystem or through backend
tools; an HTTP MCP server can run without an executor; a stdio MCP
server or hook needs an execution environment.
- Core should carry generic extension initialization data. The extension
that owns a component should discover it, expose it to the model, and
invoke it through the appropriate runtime.

This PR establishes that architecture through one complete vertical:
selecting a root on an executor, discovering the skills beneath it,
exposing those skills to the model, and reading an explicitly invoked
`SKILL.md` through the same executor.

## Contract

`thread/start` gains an experimental `selectedCapabilityRoots` field:

```json
{
  "selectedCapabilityRoots": [
    {
      "id": "deploy-plugin@1",
      "location": {
        "type": "environment",
        "environmentId": "workspace",
        "path": "/opt/codex/plugins/deploy"
      }
    }
  ]
}
```

The root is intentionally not classified as a "plugin" or "skill" in the
API. It can point at a standalone skill, a directory containing several
skills, or a plugin containing skills and other components. This PR only
teaches the skills extension how to consume it; later extensions can
resolve MCP, connector, and hook components from the same selection.

The platform-supplied `id` is stable selection identity. The location
says which runtime owns the root and gives that runtime an opaque path.
App-server does not inspect or canonicalize the path.

## What changed

### Generic thread extension initialization

App-server converts selected roots into `ExtensionDataInit`. Core
carries that generic initialization value until the final thread ID is
known, then creates thread-scoped `ExtensionData` before lifecycle
contributors run.

This keeps `Session` and core independent of the capability-selection
contract. The initialization value is consumed during construction; it
is not retained as another long-lived `Session` field.

### Executor-backed skills

The skills extension now owns an `ExecutorSkillProvider` that:

- resolves the selected environment through `EnvironmentManager`
- discovers, canonicalizes, and reads skills through that environment's
`ExecutorFileSystem`
- contributes the bounded selected-skill catalog as stable developer
context
- reads an explicitly invoked skill body through the authority that
listed it
- warns when an environment or root is unavailable
- never falls back to the orchestrator filesystem for an executor-owned
root

Skill catalog and instruction fragments have hard byte bounds, which
also bound them below the 10K-token per-item context limit. If a
selected executor skill has the same name as a legacy local skill, the
executor selection owns that invocation and the local body is not
injected a second time.

Existing local and bundled skill loading remains in place. Omitting
`selectedCapabilityRoots` therefore preserves current local-only
behavior.

## Current semantics

- Only environment-owned locations are represented in this first
contract.
- Roots are resolved by the destination extension, not by app-server or
core.
- An unavailable executor or invalid root produces a warning and no
capabilities from that root; it does not trigger a local-filesystem
fallback.
- Selection applies to a newly started active thread.
- MCP servers, connectors, and hooks beneath a selected plugin root are
not activated yet.
- Selection is not yet persisted or inherited across resume, fork, or
subagent creation. Existing local capabilities continue to behave as
they do today in those flows.

## Planned vertical follow-ups

1. **Hosted HTTP MCP:** add an extension-backed HTTP MCP source that
works without an executor, then replace the special-purpose MCP plugins
loader with that implementation.
2. **Executor MCP:** register and execute stdio MCP servers through the
environment that owns the selected plugin root.
3. **Backend skills:** add a hosted skill source whose catalog and
bodies are accessed through extension tools rather than a filesystem.
4. **Connectors and hooks:** activate those components through their
owning extensions, using the same selected-root boundary and
component-specific runtime.
5. **Durable selection:** define the desired-selection lifecycle,
persist it, and make resume, fork, and subagent inheritance explicit
rather than accidental.
6. **Local convergence:** incrementally route existing local plugin,
skill, and MCP loading through the same extension model while preserving
current local behavior.

Each follow-up remains reviewable as an end-to-end capability. The
platform selects roots, generic thread extension data carries the
selection, and the owning extension resolves and operates its component.

## Verification

Coverage added for:

- app-server end-to-end discovery and explicit invocation of a skill
inside an executor-selected plugin root
- exclusive invocation when a selected executor skill collides with a
local skill name
- executor filesystem authority for discovery, canonicalization, and
reads
- thread extension initialization before lifecycle contributors run
- stable executor catalog context, explicit invocation, context
rebuilding, hidden skills, and preserved host/remote catalog behavior

Targeted protocol, core-skills, skills-extension, core lifecycle, and
app-server executor-skill tests were run during development.

89ac3ec27c · 2026-06-09 19:51:54 +02:00

History

read

chore: move memory prompt builder into extension (#24558 )

2026-05-26 11:53:47 +02:00

write

Load selected executor skills through extensions (#27184 )

2026-06-09 19:51:54 +02:00

README.md

chore: drop orphaned codex memories MCP crate (#24555 )

2026-05-26 11:29:37 +02:00

README.md

Memories

This directory owns reusable memory crates and the memory pipeline documentation.

Runtime orchestration for Phase 1 and Phase 2 still lives in codex-core under codex-rs/core/src/memories/.

Crates

codex-rs/memories/read (codex-memories-read) owns the read path: memory developer-instruction injection, memory citation parsing, and read-usage telemetry classification.
codex-rs/memories/write (codex-memories-write) owns the write path: Phase 1 and Phase 2 prompt rendering, filesystem artifact helpers, workspace diff helpers, and extension resource pruning.

Prompt Templates

Memory prompt templates live with the crate that uses them:

The undated template files are the canonical latest versions used at runtime:
- read/templates/memories/read_path.md
- write/templates/memories/stage_one_system.md
- write/templates/memories/stage_one_input.md
- write/templates/memories/consolidation.md
In codex, edit those undated template files in place.
The dated snapshot-copy workflow is used in the separate openai/project/agent_memory/write harness repo, not here.

When it runs

The pipeline is triggered when a root session starts, and only if:

the session is not ephemeral
the memory feature is enabled
the session is not a sub-agent session
the state DB is available

It runs asynchronously in the background and executes two phases in order: Phase 1, then Phase 2.

Phase 1: Rollout Extraction (per-thread)

Phase 1 finds recent eligible rollouts and extracts a structured memory from each one.

Eligible rollouts are selected from the state DB using startup claim rules. In practice this means the pipeline only considers rollouts that are:

from allowed interactive session sources
within the configured age window
idle long enough (to avoid summarizing still-active/fresh rollouts)
not already owned by another in-flight phase-1 worker
within startup scan/claim limits (bounded work per startup)

What it does:

claims a bounded set of rollout jobs from the state DB (startup claim)
filters rollout content down to memory-relevant response items
sends each rollout to a model (in parallel, with a concurrency cap)
expects structured output containing:
- a detailed raw_memory
- a compact rollout_summary
- an optional rollout_slug
redacts secrets from the generated memory fields
stores successful outputs back into the state DB as stage-1 outputs

Concurrency / coordination:

Phase 1 runs multiple extraction jobs in parallel (with a fixed concurrency cap) so startup memory generation can process several rollouts at once.
Each job is leased/claimed in the state DB before processing, which prevents duplicate work across concurrent workers/startups.
Failed jobs are marked with retry backoff, so they are retried later instead of hot-looping.

Job outcomes:

succeeded (memory produced)
succeeded_no_output (valid run but nothing useful generated)
failed (with retry backoff/lease handling in DB)

Phase 1 is the stage that turns individual rollouts into DB-backed memory records.

Phase 2: Global Consolidation

Phase 2 consolidates the latest stage-1 outputs into the filesystem memory artifacts and then runs a dedicated consolidation agent.

What it does:

claims a single global phase-2 lock before touching the memories root (so only one consolidation inspects or mutates the workspace at a time)
loads a bounded set of stage-1 outputs from the state DB using phase-2 selection rules:
- ignores memories whose last_usage falls outside the configured max_unused_days window
- for memories with no last_usage, falls back to generated_at so fresh never-used memories can still be selected
- ranks eligible memories by usage_count first, then by the most recent last_usage / generated_at
computes a completion watermark from the claimed watermark + newest input timestamps
syncs local memory artifacts under the memories root:
- raw_memories.md (merged raw memories, stable ascending thread-id order)
- rollout_summaries/ (one summary file per selected rollout)
keeps the memories root itself as a git-baseline directory, initialized under ~/.codex/memories/.git by codex-git-utils
prunes stale rollout summaries that are no longer selected
prunes memory extension resource files older than the extension retention window, so cleanup appears in the workspace diff
writes phase2_workspace_diff.md in the memories root with the git-style diff from the previous successful Phase 2 baseline to the current worktree
if the memory workspace has no changes after artifact sync/pruning, marks the job successful and exits

If the memory workspace has changes, it then:

spawns an internal consolidation sub-agent
builds the Phase 2 prompt with the path to the generated workspace diff
points the agent at phase2_workspace_diff.md for the detailed diff context
runs it with no approvals, no network, and local write access only
disables collab for that agent (to prevent recursive delegation)
watches the agent status and heartbeats the global job lease while it runs
resets the memory git baseline after the agent completes successfully; the generated diff file is removed before this reset so deleted content is not kept in the prompt artifact or unreachable git objects
marks the phase-2 job success/failure in the state DB when the agent finishes

Selection and workspace-diff behavior:

successful Phase 2 runs mark the exact stage-1 snapshots they consumed with selected_for_phase2 = 1 and persist the matching selected_for_phase2_source_updated_at
Phase 1 upserts preserve the previous selected_for_phase2 baseline until the next successful Phase 2 run rewrites it
Phase 2 loads only the current top-N selected stage-1 inputs, syncs rollout_summaries/ directly to that selection, renders raw_memories.md in stable ascending thread-id order to avoid usage-rank churn, then lets the git-style workspace diff surface additions, modifications, and deletions against the previous successful memory baseline
when the selected input set is empty, stale rollout_summaries/ files are removed and raw_memories.md is rewritten to the empty-input placeholder; consolidated outputs such as MEMORY.md, memory_summary.md, and skills/ are left for the agent to update

Watermark behavior:

The global phase-2 lock does not use DB watermarks as a dirty check; git workspace dirtiness decides whether an agent needs to run.
The global phase-2 job row still tracks an input watermark as bookkeeping for the latest DB input timestamp known when the job was claimed.
Phase 2 recomputes a new_watermark using the max of:
- the claimed watermark
- the newest source_updated_at timestamp in the stage-1 inputs it actually loaded
On success, Phase 2 stores that completion watermark in the DB.
This avoids moving the recorded completion watermark backwards, but does not decide whether Phase 2 has work.

In practice, this phase is responsible for refreshing the on-disk memory workspace and producing/updating the higher-level consolidated memory outputs.

Why it is split into two phases

Phase 1 scales across many rollouts and produces normalized per-rollout memory records.
Phase 2 serializes global consolidation so the shared memory artifacts are updated safely and consistently.