mirror of
https://github.com/pchuan98/codex.git
synced 2026-07-01 00:31:56 +08:00
app-server: keep the model cache warm (#28699)
## Why The app server is long-lived, but its shared model cache otherwise refreshes only when a caller needs it. Once the five-minute cache expires, starting a thread or calling `model/list` can wait for `/models` on the request path. Refresh the cache in the background before it expires so foreground callers normally use fresh local state. ## What changed - Start an app-server worker that refreshes models immediately and then every three minutes using the existing models-manager API. - Hold only a weak reference to the models manager between refreshes, so the worker does not extend its lifetime. - Stop scheduling refreshes when the app-server lifecycle handle is shut down or dropped. A refresh already in progress is allowed to finish. - Adjust affected app-server test fixtures to distinguish the background `/models` probe from the connection they are testing. The existing models-manager cache, refresh strategies, auth handling, ETag behavior, and concurrency semantics are unchanged. ## Testing - `models_refresh_worker::tests::refreshes_immediately_periodically_and_stops_when_dropped` - `suite::v2::remote_control::listen_off_honors_persisted_remote_control_enable` - `suite::v2::attestation::attestation_generate_round_trip_adds_header_to_responses_websocket_handshake`
This commit is contained in:
@@ -92,6 +92,8 @@ use tokio::time::timeout;
|
||||
use tokio_util::sync::CancellationToken;
|
||||
use tracing::Instrument;
|
||||
|
||||
use crate::models_refresh_worker::ModelsRefreshWorker;
|
||||
|
||||
const EXTERNAL_AUTH_REFRESH_TIMEOUT: Duration = Duration::from_secs(10);
|
||||
const CONNECTION_RPC_DRAIN_TIMEOUT: Duration = Duration::from_secs(/*secs*/ 30);
|
||||
|
||||
@@ -182,6 +184,7 @@ impl ExternalAuth for ExternalAuthRefreshBridge {
|
||||
|
||||
pub(crate) struct MessageProcessor {
|
||||
outgoing: Arc<OutgoingMessageSender>,
|
||||
models_refresh_worker: ModelsRefreshWorker,
|
||||
skills_watcher: Arc<SkillsWatcher>,
|
||||
account_processor: AccountRequestProcessor,
|
||||
apps_processor: AppsRequestProcessor,
|
||||
@@ -371,6 +374,8 @@ impl MessageProcessor {
|
||||
)),
|
||||
)
|
||||
});
|
||||
let models_manager = thread_manager.get_models_manager();
|
||||
let models_refresh_worker = crate::models_refresh_worker::spawn(&models_manager);
|
||||
thread_manager
|
||||
.plugins_manager()
|
||||
.set_analytics_events_client(analytics_events_client.clone());
|
||||
@@ -537,6 +542,7 @@ impl MessageProcessor {
|
||||
|
||||
Self {
|
||||
outgoing,
|
||||
models_refresh_worker,
|
||||
skills_watcher,
|
||||
account_processor,
|
||||
apps_processor,
|
||||
@@ -566,6 +572,7 @@ impl MessageProcessor {
|
||||
pub(crate) fn clear_runtime_references(&self) {
|
||||
self.account_processor.clear_external_auth();
|
||||
self.apps_processor.shutdown();
|
||||
self.models_refresh_worker.shutdown();
|
||||
self.skills_watcher.shutdown();
|
||||
}
|
||||
|
||||
@@ -742,6 +749,7 @@ impl MessageProcessor {
|
||||
}
|
||||
|
||||
pub(crate) async fn drain_background_tasks(&self) {
|
||||
self.models_refresh_worker.shutdown();
|
||||
self.thread_processor.drain_background_tasks().await;
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user