core: cache the tool search handler per session (#27258)

## Why

Tool router construction rebuilds the deferred-tool BM25 index during
session initialization and before each sampling continuation, even when
the searchable tool metadata is unchanged. Local profiling measured
`append_tool_search_executor` at roughly 113 ms per continuation, making
repeated index construction the largest measured router-building cost.

## What changed

- Add a session-scoped `ToolSearchHandlerCache` so continuations and
user turns can reuse the existing handler.
- Key reuse on the complete ordered `Vec<ToolSearchInfo>`, rebuilding
when searchable text, loadable tool specs, source metadata, or ordering
changes.
- Build handlers outside the cache lock and recheck before publishing
them, avoiding holding the mutex during index construction.

## Verification

- `cache_reuses_identical_search_infos_and_rebuilds_changed_inputs`
covers exact cache reuse and invalidation when the ordered search
metadata changes.
- Local rollout profiling showed the initial router build populating the
cache and unchanged later continuations reusing it:
  - uncached: 118 ms median across 14 spans from 3 rollouts
  - cached: 4 ms median across 12 spans from 3 rollouts
This commit is contained in:
mchen-oai
2026-06-15 14:48:30 -07:00
committed by GitHub
parent fbbe7706d6
commit af99f6a72f
12 changed files with 169 additions and 29 deletions
+2 -2
View File
@@ -6,13 +6,13 @@ use crate::ToolSearchSourceInfo;
use crate::ToolSpec;
use crate::default_namespace_description;
#[derive(Clone)]
#[derive(Clone, PartialEq)]
pub struct ToolSearchEntry {
pub search_text: String,
pub output: LoadableToolSpec,
}
#[derive(Clone)]
#[derive(Clone, PartialEq)]
pub struct ToolSearchInfo {
pub entry: ToolSearchEntry,
pub source_info: Option<ToolSearchSourceInfo>,