app-server: keep the model cache warm (#28699)

## Why

The app server is long-lived, but its shared model cache otherwise
refreshes only when a caller needs it. Once the five-minute cache
expires, starting a thread or calling `model/list` can wait for
`/models` on the request path.

Refresh the cache in the background before it expires so foreground
callers normally use fresh local state.

## What changed

- Start an app-server worker that refreshes models immediately and then
every three minutes using the existing models-manager API.
- Hold only a weak reference to the models manager between refreshes, so
the worker does not extend its lifetime.
- Stop scheduling refreshes when the app-server lifecycle handle is shut
down or dropped. A refresh already in progress is allowed to finish.
- Adjust affected app-server test fixtures to distinguish the background
`/models` probe from the connection they are testing.

The existing models-manager cache, refresh strategies, auth handling,
ETag behavior, and concurrency semantics are unchanged.

## Testing

-
`models_refresh_worker::tests::refreshes_immediately_periodically_and_stops_when_dropped`
-
`suite::v2::remote_control::listen_off_honors_persisted_remote_control_enable`
-
`suite::v2::attestation::attestation_generate_round_trip_adds_header_to_responses_websocket_handshake`
This commit is contained in:
jif
2026-06-17 15:18:39 +01:00
committed by GitHub
Unverified
parent 45f603302c
commit 5935a90619
7 changed files with 238 additions and 58 deletions
@@ -92,6 +92,8 @@ use tokio::time::timeout;
use tokio_util::sync::CancellationToken;
use tracing::Instrument;
use crate::models_refresh_worker::ModelsRefreshWorker;
const EXTERNAL_AUTH_REFRESH_TIMEOUT: Duration = Duration::from_secs(10);
const CONNECTION_RPC_DRAIN_TIMEOUT: Duration = Duration::from_secs(/*secs*/ 30);
@@ -182,6 +184,7 @@ impl ExternalAuth for ExternalAuthRefreshBridge {
pub(crate) struct MessageProcessor {
outgoing: Arc<OutgoingMessageSender>,
models_refresh_worker: ModelsRefreshWorker,
skills_watcher: Arc<SkillsWatcher>,
account_processor: AccountRequestProcessor,
apps_processor: AppsRequestProcessor,
@@ -371,6 +374,8 @@ impl MessageProcessor {
)),
)
});
let models_manager = thread_manager.get_models_manager();
let models_refresh_worker = crate::models_refresh_worker::spawn(&models_manager);
thread_manager
.plugins_manager()
.set_analytics_events_client(analytics_events_client.clone());
@@ -537,6 +542,7 @@ impl MessageProcessor {
Self {
outgoing,
models_refresh_worker,
skills_watcher,
account_processor,
apps_processor,
@@ -566,6 +572,7 @@ impl MessageProcessor {
pub(crate) fn clear_runtime_references(&self) {
self.account_processor.clear_external_auth();
self.apps_processor.shutdown();
self.models_refresh_worker.shutdown();
self.skills_watcher.shutdown();
}
@@ -742,6 +749,7 @@ impl MessageProcessor {
}
pub(crate) async fn drain_background_tasks(&self) {
self.models_refresh_worker.shutdown();
self.thread_processor.drain_background_tasks().await;
}