feat: 重组项目结构并完善运行管理

This commit is contained in:
chuan
2026-08-10 15:00:39 +08:00
parent ea4625e5da
commit 28fa69e614
127 changed files with 3927 additions and 1079 deletions
+49
View File
@@ -0,0 +1,49 @@
# Changelog
本项目遵循语义化版本。0.2.1 是包含公开 API 变更的 breaking release。
## Unreleased
### Added
- BEP-51 `sample_infohashes` 主动发现、按节点 interval/退避的采样 actor,以及有界 Hash
去重。
- 采样 Hash 优先查询来源节点,再通过迭代式 `get_peers` 补充 Peer。
### Changed
- Metadata 最大 info 字典大小改为可通过 `max_metadata_size_bytes` 配置,并拒绝负数文件大小和总大小溢出。
- 主动 `get_peers` ingress 改为有界排队,避免突发采样在速率预算耗尽时直接丢弃。
- 默认主动 Peer lookup 提升到每秒 128 个、最多 256 个并发 lookup。
- 空节点池的 Bootstrap 默认改为 30 秒重试、每轮最多 16 个端点,降低坏 DNS
地址导致冷启动停滞的概率。
## 0.2.1 - 2026-07-30
### Breaking changes
-`DHTOptions` 的 Metadata 和 crawl 参数改为嵌套结构:`MetadataOptions`
`CrawlOptions``RateLimitOptions``PoolOptions``BootstrapOptions`
`TargetOptions``SchedulerOptions`
- 删除旧的 `metadata_timeout``max_metadata_queue_size`
`max_metadata_worker_count``node_queue_capacity` 等扁平字段。
- 删除旧 active/candidate frontier 和 sharded queue 实现,改用单所有者严格 FIFO
节点池、recent-probe set 与 responsive-node ring。
- Metadata 调度改为有界、按 InfoHash 去重、最多三个 Peer、60 秒 freshness TTL。
### Added
- 独立的主动爬取 QPS、新目标、节点替换、回复包/字节、单来源回复、总在途和子网在途限制。
- 根据 Metadata 队列压力自动降低实际 `find_node` QPS。
- Bootstrap 来源退避、低水位触发和响应节点快照。
- `on_torrent_with_ack``on_metadata_fetch_complete`
`MetadataFetchCompletionStatus` 和真实 Peer `attempts`
-`SocketAddr` 缓存 Metadata Peer 的 timeout/connect failure。
- `DhtRuntimeStats::snapshot()``observability_snapshot()` 和三组固定桶直方图。
- DHT、UDP、节点池、Metadata scheduler/fetcher 的低基数 Prometheus 指标。
### Changed
- Metadata timeout 现在覆盖连接、握手、传输、SHA1 和解析的完整 Peer 尝试。
- UDP ingress、crawl events 和 Metadata queues 全部有界,并暴露 drop/depth 指标。
- DHT 回复增加总包、总字节、单来源限流,以及 `ping`/`get_peers` 10% 保底预算。
+54
View File
@@ -0,0 +1,54 @@
# 定义可复用 DHT 爬虫基础库的包元数据依赖和功能开关
[package]
name = "dht-crawler"
version = "0.2.1"
edition.workspace = true
authors.workspace = true
description = "高性能的 Rust DHT 爬虫基础库"
license.workspace = true
documentation = "https://docs.rs/dht-crawler"
repository.workspace = true
keywords = ["dht", "bittorrent", "crawler", "p2p", "torrent"]
categories = ["network-programming", "asynchronous"]
readme = "README.md"
[lib]
name = "dht_crawler"
crate-type = ["rlib"]
[dependencies]
ahash = "0.8"
arc-swap = "1.7"
async-channel = "2.5.0"
bytes = "1.0"
crossbeam-queue = "0.3"
hex = "0.4"
log = "0.4"
metrics = { version = "0.24", optional = true }
rand = "0.10.2"
rbit = "0.2"
serde.workspace = true
serde_bencode = "0.2"
serde_bytes = "0.11.19"
sha1 = "0.11.0"
socket2 = { version = "0.6.5", features = ["all"] }
thiserror.workspace = true
tokio = { workspace = true, features = ["rt", "rt-multi-thread", "net", "sync", "time", "macros"] }
tokio-util.workspace = true
[dev-dependencies]
metrics-exporter-prometheus = { version = "0.18", default-features = false, features = ["http-listener"] }
mimalloc = "0.1"
tokio = { workspace = true, features = ["signal"] }
tracing.workspace = true
tracing-subscriber = { workspace = true, features = ["env-filter", "fmt", "tracing-log"] }
[features]
default = []
metrics = ["dep:metrics"]
mimalloc = []
[[example]]
name = "dht_crawler_example"
path = "examples/main.rs"
+272
View File
@@ -0,0 +1,272 @@
# dht-crawler
[![Crates.io](https://img.shields.io/crates/v/dht-crawler.svg)](https://crates.io/crates/dht-crawler)
[![Documentation](https://docs.rs/dht-crawler/badge.svg)](https://docs.rs/dht-crawler)
[![License](https://img.shields.io/crates/l/dht-crawler.svg)](../../LICENSE)
基于 Rust 和 Tokio 的 BitTorrent DHT 爬虫库。它参与 BEP-5 DHT 网络,通过 BEP-51
`sample_infohashes` 主动发现 InfoHash,也接收 `announce_peer`,并通过 BEP-9
`ut_metadata` 获取、校验和解析 torrent 元数据。
`dht-crawler` 提供:
- IPv4、IPv6 和双栈 DHT
- 主动节点发现、BEP-51 InfoHash 采样与 `get_peers` 查询;
- 有界、去重的 Metadata 下载队列;
- InfoHash 过滤、异步准入、结果交付和完成通知;
- 默认可用的运行时统计,以及可选的 `metrics` 集成。
## 安装
```bash
cargo add dht-crawler
cargo add tokio --features rt-multi-thread,macros,signal
```
或在 `Cargo.toml` 中添加:
```toml
[dependencies]
dht-crawler = "0.2"
tokio = { version = "1", features = ["rt-multi-thread", "macros", "signal"] }
```
## 快速开始
```rust
use dht_crawler::prelude::*;
#[tokio::main]
async fn main() -> Result<()> {
let server = DHTServer::new(DHTOptions {
port: 6881,
netmode: NetMode::Ipv4Only,
..Default::default()
})
.await?;
server.on_torrent(|torrent| {
println!(
"{} {} {}",
torrent.info_hash,
torrent.name,
torrent.format_size()
);
});
server.on_error(|error| {
eprintln!("DHT runtime error: {error}");
});
let shutdown = server.clone();
tokio::spawn(async move {
if tokio::signal::ctrl_c().await.is_ok() {
shutdown.shutdown();
}
});
// 一直运行,直到 shutdown() 被调用。
server.start().await
}
```
运行仓库中的完整示例:
```bash
cargo run --release --example dht_crawler_example
```
## 核心 API
`DHTServer` 是主要入口:
| API | 用途 |
|---|---|
| `DHTServer::new(options)` | 校验配置、绑定 UDP Socket 并创建内部管道 |
| `start().await` | 启动 DHT 与爬取任务,等待 `shutdown()` |
| `shutdown()` | 停止 UDP、爬取和 Metadata 任务;可重复调用 |
| `filter(callback)` | 在 InfoHash 进入队列前执行同步过滤 |
| `on_metadata_fetch(callback)` | 在第一次 Peer 下载前执行异步准入 |
| `on_torrent(callback)` | 接收已校验的 `TorrentInfo` |
| `on_torrent_with_ack(callback)` | 接收结果并显式确认是否接受交付 |
| `on_metadata_fetch_complete(callback)` | 接收已准入任务的最终状态 |
| `on_error(callback)` | 接收运行期错误 |
| `runtime_stats()` | 获取可复制的运行时统计句柄 |
同类回调重复注册时,新回调会替换旧回调。
### 过滤与准入
`filter` 是同步的早期过滤器,适合拦截已处理过的 InfoHash:
```rust
server.filter(|info_hash| !already_exists(info_hash));
```
`on_metadata_fetch` 是异步准入回调,在实际连接 Peer 前调用:
```rust
server.on_metadata_fetch(|info_hash| async move {
should_download(&info_hash).await
});
```
返回 `false` 会终止任务,不下载 Metadata,也不会触发 torrent 或 completion 回调。
未注册准入回调时默认允许下载。
### 交付确认
不需要确认下游是否接收时使用 `on_torrent`。需要确认下游是否成功接收时使用
`on_torrent_with_ack`
```rust
server.on_torrent_with_ack(|torrent| {
output.try_send(torrent).is_ok()
});
server.on_metadata_fetch_complete(|completion| {
println!(
"{}: {:?}, attempts={}",
completion.info_hash,
completion.status,
completion.attempts
);
});
```
完成状态:
| 状态 | 含义 |
|---|---|
| `Accepted` | Metadata 下载成功,结果已被回调接受 |
| `FetchFailed` | 所有可用 Peer 尝试均失败 |
| `DeliveryRejected` | Metadata 下载成功,但结果未被回调接受 |
`attempts` 只统计实际发起的 Peer 网络请求。异步准入拒绝不会产生 completion 事件。
## 配置
大多数调用方可以从 `DHTOptions::default()` 开始,只覆盖监听方式和容量限制:
```rust
let options = DHTOptions {
port: 6881,
netmode: NetMode::DualStack,
hash_queue_capacity: 20_000,
metadata: MetadataOptions {
timeout_secs: 5,
max_queue_size: 20_000,
max_worker_count: 8,
max_connects_per_second: 2,
max_metadata_size_bytes: 10 * 1024 * 1024,
..Default::default()
},
crawl: CrawlOptions {
rate_limit: RateLimitOptions {
max_find_node_rate_per_sec: 6,
max_in_flight: 12,
..Default::default()
},
..Default::default()
},
..Default::default()
};
```
配置分组:
| 类型 | 控制内容 |
|---|---|
| `DHTOptions` | 监听端口、网络模式、顶层队列和主动 UDP 查询总预算 |
| `MetadataOptions` | 下载超时、队列、并发、每秒 TCP 建连和失败 Peer 缓存 |
| `PeerLookupOptions` | 主动 `get_peers` 的速率与并发 |
| `SampleInfohashesOptions` | BEP-51 采样速率、并发、新节点稳定分流、有界候选队列、超时、退避和 Hash 去重容量 |
| `RateLimitOptions` | `find_node`、在途请求和 UDP 回复预算 |
| `PoolOptions` | 节点池、最近探测记录和响应节点缓存 |
| `BootstrapOptions` | Bootstrap 节点与失败退避 |
| `TargetOptions` | 主动爬取目标生成策略 |
| `SchedulerOptions` | 内部事件队列、批处理与快照限制 |
完整字段和默认值以 [docs.rs API 文档](https://docs.rs/dht-crawler) 为准。需要注意:
- BEP-51 采样 hash 可以通过 `DHTServer::on_sampled_hashes` 批量异步准入
- 采样准入队列有固定容量并在压力升高时暂停新的 BEP-51 查询
- `new_node_sample_percent` 大于零时按节点地址稳定分流并优先直接采样 避免同一次发现同时执行 `find_node`
- 新节点采样通道满时回退到抓取池 不会依靠无限队列维持吞吐
- 带首选节点的 Peer Lookup 先执行单点查询只有失败后才进入有限迭代查找
- `DHTOptions::default()` 使用 `Ipv4Only`
- `NetMode::DualStack` 会分别绑定 IPv4 和 IPv6 Socket
- `DHTServer::new()` 会立即在所有可用接口上绑定配置的 UDP 端口;
- 空节点池默认每 30 秒重新尝试 Bootstrap,每轮最多使用 16 个已解析端点;
- `MetadataOptions::timeout_secs` 是单个 Peer 尝试的端到端期限;
- `PeerLookupOptions::max_lookups_per_second = 0` 会关闭主动 `get_peers`
- `SampleInfohashesOptions::max_queries_per_second = 0` 会关闭 BEP-51 主动采样;
- Metadata 和爬取队列都是有界的,容量应与下游处理能力一起调整。
## 数据与运行语义
`TorrentInfo` 包含 `info_hash``magnet_link``name``total_size``files`
`piece_length``peers``timestamp`。只有通过 SHA1 校验并成功解析的 Metadata
才会交付给 torrent 回调。
库使用有界队列控制内存占用。队列满或速率预算耗尽时,新事件可能被拒绝、淘汰或计入
drop 指标。Metadata 队列按 InfoHash 去重,每个任务可尝试多个候选 Peer;连接超时和
连接失败的 Peer 会被短期缓存,避免反复占用 worker。
`start()` 返回后,当前实例不能再次启动。如需重新运行,请创建新的 `DHTServer`
## 可观测性
运行时快照无需启用 Cargo feature
```rust
let stats = server.runtime_stats();
let snapshot = stats.snapshot();
println!(
"nodes={} metadata={}/{} workers={}",
snapshot.node_pool_size,
snapshot.metadata_queue_depth,
snapshot.metadata_queue_max,
snapshot.metadata_in_flight,
);
```
`observability_snapshot()` 提供 UDP、查询、队列、Metadata 失败原因和固定桶直方图。
这些快照面向监控,读取时不是跨字段事务视图。
启用 `metrics` 后,库通过 [`metrics`](https://crates.io/crates/metrics) facade 记录
指标,但不会安装 recorder 或启动 HTTP 服务:
```toml
[dependencies]
dht-crawler = { version = "0.2", features = ["metrics"] }
```
指标名称、类型、标签和单位见 [docs/metrics.md](docs/metrics.md)。
## Cargo features
默认不启用任何 feature。
| Feature | 用途 |
|---|---|
| `metrics` | 通过 `metrics` facade 记录指标 |
| `mimalloc` | 将 mimalloc 注册为全局分配器 |
启用 `mimalloc` 前,请确认最终二进制没有注册其他全局分配器。
## 开发
```bash
cargo fmt --all --check
cargo check --all-targets --all-features
cargo clippy --all-targets --all-features -- -D warnings
cargo test --all-features
cargo doc --no-deps --all-features
```
## 许可证
[MIT](LICENSE)
+110
View File
@@ -0,0 +1,110 @@
# dht-crawler 指标参考
启用 Cargo feature `metrics` 后,库通过 `metrics` facade 记录以下指标。库不安装
recorder、不监听端口,也不依赖任何特定导出协议;Prometheus exporter 应由最终应用安装。
所有 `*_total` 都是进程生命周期累计 counter。Gauge 是当前值。Histogram 的记录值
使用下表标出的单位。
## UDP 与 KRPC
| 指标 | 类型 | 标签 | 单位/含义 |
|---|---|---|---|
| `dht_udp_bytes_received_total` | counter | — | Socket 接收字节 |
| `dht_udp_packets_received_total` | counter | `status=ok|dropped_size|dropped_magic|queue_full` | UDP ingress 结果 |
| `dht_udp_bytes_sent_total` | counter | — | 成功发送字节 |
| `dht_udp_packets_sent_total` | counter | `type=query|response` | 成功发送包 |
| `dht_udp_query_size_bytes` | histogram | — | find_node query 编码长度,bytes |
| `dht_messages_processed_total` | counter | `type=q|r|e|unknown` | 成功解析的 KRPC 消息类型 |
| `dht_messages_parse_error_total` | counter | — | bencode/KRPC 解析失败 |
| `dht_queries_total` | counter | `q=ping|find_node|get_peers|announce_peer|vote|other_or_invalid` | 入站查询类型 |
| `dht_udp_responses_dropped_total` | counter | `reason=rate_limit` | 最终未发送的限流回复 |
| `dht_udp_responses_priority_reserved_total` | counter | `query=ping|get_peers` | 使用 10% 保底预算的回复 |
`dht_udp_bytes_received_total` 包含后续被判定为 invalid/queue-full 的 Datagram;发送侧只在
`send_to` 成功后累计。
## 主动爬取与节点池
| 指标 | 类型 | 标签 | 含义 |
|---|---|---|---|
| `dht_node_pool_size` | gauge | — | 当前 FIFO 节点数 |
| `dht_node_pool_oldest_age_seconds` | gauge | — | FIFO 最老节点年龄 |
| `dht_node_pool_admissions_total` | counter | — | 新准入节点 |
| `dht_node_pool_replacements_total` | counter | — | 满池替换 |
| `dht_node_pool_dropped_total` | counter | `reason=duplicate|rate_limit|invalid` | 节点拒绝原因 |
| `dht_find_node_in_flight` | gauge | — | 当前在途 find_node |
| `dht_find_node_effective_rate_per_second` | gauge | — | Metadata 压力调整后的实际预算 |
| `dht_crawl_queries_sent_total` | counter | `kind=new|revisit|bootstrap` | 已交给 egress 的查询用途;发送失败另计 |
| `dht_find_node_responses_total` | counter | — | 与 pending transaction 匹配的回复 |
| `dht_find_node_response_unmatched_total` | counter | — | 无匹配 pending 的回复 |
| `dht_find_node_timeouts_total` | counter | — | pending 超时 |
| `dht_find_node_send_failures_total` | counter | — | UDP query 发送失败 |
| `dht_crawl_events_dropped_total` | counter | `kind=discovered|response` | 有界 actor channel 丢弃 |
| `dht_metadata_queue_pressure_ratio` | gauge | — | Metadata depth/capacity,范围 0..1 |
actor 每秒把内部增量 flush 到 counter,因此 exporter 看到的 counter 可能最多延迟约一秒。
## BEP-51 InfoHash 采样
| 指标 | 类型 | 标签 | 含义 |
|---|---|---|---|
| `dht_sample_infohashes_queries_total` | counter | — | 已发送的 BEP-51 查询 |
| `dht_sample_infohashes_responses_total` | counter | — | 匹配的 BEP-51 响应 |
| `dht_sample_infohashes_timeouts_total` | counter | — | 超时的 BEP-51 请求 |
| `dht_sample_infohashes_hashes_total` | counter | — | 已接受并送往 Peer lookup 的新 Hash |
| `dht_sample_infohashes_in_flight` | gauge | — | 当前在途采样请求 |
| `dht_sample_infohashes_dropped_total` | counter | `reason=response_queue_full|peer_lookup_queue_full` | 有界队列丢弃 |
## announce 与 Metadata ingress
| 指标 | 类型 | 标签 | 含义 |
|---|---|---|---|
| `dht_announce_peer_blocked_total` | counter | `reason=invalid_token|filtered` | announce 拒绝原因 |
| `dht_info_hashes_discovered_total` | counter | — | token/hash/filter 校验通过的 InfoHash |
| `dht_metadata_ingress_dropped_total` | counter | `reason=queue_full` | Hash ingress 满导致的丢弃 |
## Metadata scheduler
| 指标 | 类型 | 标签 | 单位/含义 |
|---|---|---|---|
| `dht_metadata_queue_depth` | gauge | — | Pending Hash 数 |
| `dht_metadata_in_flight` | gauge | — | 当前 job 数 |
| `dht_metadata_queue_events_total` | counter | `result=inserted|deduplicated|evicted_oldest|stale|expired` | 队列事件 |
| `dht_metadata_queue_wait_seconds` | histogram | — | Hash 从最近发现到首次分派的秒数 |
| `dht_metadata_jobs_dispatched_total` | counter | — | 分派 job 数 |
| `dht_metadata_jobs_completed_total` | counter | `result=accepted|fetch_failed|delivery_rejected|gate_rejected` | job 终态 |
| `dht_metadata_worker_join_error_total` | counter | — | worker task join 失败 |
| `dht_metadata_completion_callback_panics_total` | counter | — | 完成回调 panic |
## Metadata Peer 下载
| 指标 | 类型 | 标签 | 单位/含义 |
|---|---|---|---|
| `dht_metadata_fetch_attempts_total` | counter | — | 实际 Peer 尝试 |
| `dht_metadata_peer_attempts_total` | counter | — | 与 fetch attempts 相同的 Peer 尝试计数 |
| `dht_metadata_fetch_success_total` | counter | — | 成功下载和解析 |
| `dht_metadata_fetch_result_total` | counter | `result=success|failed|timeout` | Peer 尝试结果 |
| `dht_metadata_fetch_fail_total` | counter | `reason=timeout|send_error|size_limit|sha1_mismatch|parse_error` | 详细失败原因 |
| `dht_metadata_connection_result_total` | counter | `result=success|failed` | TCP/BitTorrent connect 结果 |
| `dht_metadata_handshake_result_total` | counter | `result=success|no_extension_support` | 扩展能力/最终校验结果 |
| `dht_metadata_fetch_duration_seconds` | histogram | — | 端到端 Peer 尝试秒数 |
| `dht_metadata_size_bytes` | histogram | — | 完整 bencoded info payload 字节数 |
| `dht_metadata_bytes_downloaded_total` | counter | — | 收到的 Metadata piece 数据字节数 |
| `dht_metadata_peer_failure_cache_hits_total` | counter | `reason=timeout|connect_failed` | 坏 Peer 缓存命中 |
| `dht_metadata_peer_failure_cache_inserts_total` | counter | `reason=timeout|connect_failed` | 坏 Peer 缓存写入 |
| `dht_metadata_peer_failure_cache_entries` | gauge | — | 当前缓存条目数 |
`dht_metadata_fetch_result_total{result="failed"}` 汇总非 timeout 的失败,不适合单独用于
分析具体原因;详细原因应结合 `fetch_fail`、connection 和 handshake 指标。
## 与原子快照的关系
`DhtRuntimeStats` 始终可用,与 `metrics` feature 无关:
- `snapshot()` 提供队列、节点池、crawl、Peer 和 UDP 运行状态。
- `observability_snapshot()` 提供 UDP 字节/包、入站查询分类、announce、节点准入、
Metadata 失败分类、failure cache 分类和固定桶。
两套出口在同一事件点更新,但读取时都不是跨字段事务快照;短时间内可能相差一个并发
事件,Prometheus 的 crawl actor counter 还可能有最多约一秒 flush 延迟。
+151
View File
@@ -0,0 +1,151 @@
// 负责演示基础 DHT 爬虫的配置启动监控和安全停止方式
#[cfg(feature = "mimalloc")]
#[global_allocator]
static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;
use dht_crawler::prelude::*;
#[cfg(feature = "metrics")]
use metrics_exporter_prometheus::PrometheusBuilder;
#[cfg(feature = "metrics")]
use std::net::SocketAddr;
use std::sync::Arc;
use std::sync::atomic::{AtomicUsize, Ordering};
use tracing_subscriber::EnvFilter;
#[tokio::main]
async fn main() -> Result<()> {
let filter = EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info"));
tracing_subscriber::fmt()
.with_env_filter(filter)
.with_ansi(true)
.init();
// 初始化 Prometheus metrics 导出器
#[cfg(feature = "metrics")]
{
let addr: SocketAddr = "0.0.0.0:9000"
.parse()
.map_err(|e| DHTError::Init(format!("无效的 metrics 监听地址: {e}")))?;
PrometheusBuilder::new()
.with_http_listener(addr)
.install()
.map_err(|e| DHTError::Init(format!("无法安装 Prometheus metrics 导出器: {e}")))?;
log::info!("📊 Prometheus metrics 导出器已启动,访问 http://localhost:9000/metrics");
}
let options = DHTOptions {
port: 12313,
netmode: NetMode::Ipv4Only,
metadata: MetadataOptions {
timeout_secs: 4,
max_queue_size: 10_000,
max_worker_count: 256,
..MetadataOptions::default()
},
..Default::default()
};
// 统计计数器
let torrent_count = Arc::new(AtomicUsize::new(0));
let torrent_count_clone = torrent_count.clone();
// 🚀 初始化 DHT Server
log::info!("🔧 正在初始化 DHT Server...");
let server = DHTServer::new(options.clone()).await?;
log::info!("🚀 DHT Server 启动,监听端口: {}", options.port);
// 注册错误回调,将运行时错误输出而不是 panic
server.on_error(|err| {
log::error!("DHT 运行时错误: {}", err);
});
// 设置 torrent 回调
server.on_torrent(move |_torrent| {
let _count = torrent_count_clone.fetch_add(1, Ordering::Relaxed) + 1;
// 🔇 取消打印 torrent 信息,减少日志输出
// let total_size: u64 = torrent.files.iter().map(|f| f.size).sum();
// let files_display = if torrent.files.len() <= 3 {
// torrent.files.iter()
// .map(|f| format!("{} ({})", f.path, format_size(f.size)))
// .collect::<Vec<_>>()
// .join(", ")
// } else {
// format!("{}个文件", torrent.files.len())
// };
//
// log::info!(
// "🎉 [{}] {} ({}, {})",
// count,
// torrent.name,
// format_size(total_size),
// files_display
// );
});
// 设置元数据获取前的检查回调
server.on_metadata_fetch(|_hash| async move { true });
// 一个通过 gate 的 Hash 最终只会收到一次完成状态。
server.on_metadata_fetch_complete(|completion| {
log::debug!(
"Metadata complete: hash={}, status={:?}, peer_attempts={}",
completion.info_hash,
completion.status,
completion.attempts
);
});
// 启动监控任务
let count_monitor = torrent_count.clone();
let runtime_stats = server.runtime_stats();
tokio::spawn(async move {
let mut interval = tokio::time::interval(std::time::Duration::from_secs(5));
let start_time = std::time::Instant::now();
loop {
interval.tick().await;
let success_fetch = count_monitor.load(Ordering::Relaxed);
let uptime = start_time.elapsed().as_secs();
let runtime = runtime_stats.snapshot();
// ✅ 监控:爬虫运行状态
log::info!(
"📊 [监控] 时长: {}s | 成功抓取: ✨ {} | 节点: {} | BEP51: hash={}, resp={}, timeout={} | Lookup: peer={} | Fetch: ok={}, fail={}, connect={}, timeout={}, noext={} | Metadata: {}/{} | worker: {}",
uptime,
success_fetch,
runtime.node_pool_size,
runtime.sample_infohashes_hashes_discovered,
runtime.sample_infohashes_responses,
runtime.sample_infohashes_timeouts,
runtime.peer_lookup_peers_found,
runtime.metadata_peer_succeeded,
runtime.metadata_peer_failed,
runtime.metadata_connect_failed,
runtime.metadata_peer_timeouts,
runtime.metadata_no_extension,
runtime.metadata_queue_depth,
runtime.metadata_queue_max,
runtime.metadata_in_flight,
);
if uptime > 0 && success_fetch > 0 {
let speed = (success_fetch as f64) / (uptime as f64 / 60.0);
log::info!("📈 平均抓取速度: {:.2} 种子/分钟", speed);
}
}
});
let shutdown_server = server.clone();
tokio::spawn(async move {
if tokio::signal::ctrl_c().await.is_ok() {
log::info!("收到 Ctrl-C,正在停止 DHT Server");
shutdown_server.shutdown();
}
});
server.start().await?;
Ok(())
}
+73
View File
@@ -0,0 +1,73 @@
// 负责校验解析和规范化 DHT 节点网络地址
use crate::types::NetMode;
use std::net::{IpAddr, Ipv4Addr, Ipv6Addr, SocketAddr};
pub(crate) fn addr_allowed_by_netmode(addr: &SocketAddr, netmode: NetMode) -> bool {
match netmode {
NetMode::Ipv4Only => addr.is_ipv4(),
NetMode::Ipv6Only => addr.is_ipv6(),
NetMode::DualStack => true,
}
}
pub(crate) fn is_valid_node_addr(addr: &SocketAddr) -> bool {
if addr.port() == 0 {
return false;
}
match addr.ip() {
IpAddr::V4(ip) => is_valid_ipv4_node_addr(ip),
IpAddr::V6(ip) => is_valid_ipv6_node_addr(ip),
}
}
fn is_valid_ipv4_node_addr(ip: Ipv4Addr) -> bool {
let octets = ip.octets();
let is_cgnat = octets[0] == 100 && (octets[1] & 0b1100_0000) == 64;
let is_benchmark = octets[0] == 198 && (octets[1] == 18 || octets[1] == 19);
let is_reserved = octets[0] >= 240;
!ip.is_unspecified()
&& !ip.is_loopback()
&& !ip.is_private()
&& !ip.is_link_local()
&& !ip.is_multicast()
&& !ip.is_broadcast()
&& !ip.is_documentation()
&& !is_cgnat
&& !is_benchmark
&& !is_reserved
}
fn is_valid_ipv6_node_addr(ip: Ipv6Addr) -> bool {
let octets = ip.octets();
let is_unique_local = (octets[0] & 0xfe) == 0xfc;
let is_unicast_link_local = octets[0] == 0xfe && (octets[1] & 0xc0) == 0x80;
let is_documentation =
octets[0] == 0x20 && octets[1] == 0x01 && octets[2] == 0x0d && octets[3] == 0xb8;
!ip.is_unspecified()
&& !ip.is_loopback()
&& !ip.is_multicast()
&& !is_unique_local
&& !is_unicast_link_local
&& !is_documentation
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn invalid_node_addresses_are_filtered() {
assert!(is_valid_node_addr(&"8.8.8.8:6881".parse().unwrap()));
assert!(is_valid_node_addr(
&"[2001:4860:4860::8888]:6881".parse().unwrap()
));
assert!(!is_valid_node_addr(&"8.8.8.8:0".parse().unwrap()));
assert!(!is_valid_node_addr(&"10.0.0.1:6881".parse().unwrap()));
assert!(!is_valid_node_addr(&"127.0.0.1:6881".parse().unwrap()));
assert!(!is_valid_node_addr(&"[fc00::1]:6881".parse().unwrap()));
}
}
+223
View File
@@ -0,0 +1,223 @@
// 负责管理 DHT 引导节点解析重试退避和准入调度
use crate::addr::{addr_allowed_by_netmode, is_valid_node_addr};
use crate::crawl_config::ResolvedCrawlConfig;
use crate::types::NetMode;
use std::collections::{HashMap, HashSet};
use std::net::SocketAddr;
use std::time::{Duration, Instant};
pub(crate) struct BootstrapGate {
last_bootstrap: Option<Instant>,
}
impl BootstrapGate {
pub(crate) fn new() -> Self {
Self {
last_bootstrap: None,
}
}
pub(crate) fn should_bootstrap(
&mut self,
pool_len: usize,
config: &ResolvedCrawlConfig,
now: Instant,
) -> bool {
if config.bootstrap_max_nodes_per_round == 0 {
return false;
}
if pool_len >= config.low_watermark {
return false;
}
if let Some(last) = self.last_bootstrap
&& now.checked_duration_since(last).unwrap_or_default() < config.bootstrap_interval
{
return false;
}
self.last_bootstrap = Some(now);
true
}
}
#[derive(Default)]
struct BootstrapSourceState {
last_attempt: Option<Instant>,
last_success: Option<Instant>,
fail_count: u32,
backoff_until: Option<Instant>,
}
pub(crate) struct BootstrapSourcePool {
pub(crate) hosts: Vec<String>,
states: HashMap<SocketAddr, BootstrapSourceState>,
backoff_base: Duration,
backoff_max: Duration,
}
impl BootstrapSourcePool {
pub(crate) fn new(hosts: Vec<String>, backoff_base: Duration, backoff_max: Duration) -> Self {
Self {
hosts,
states: HashMap::new(),
backoff_base,
backoff_max,
}
}
pub(crate) fn select(
&mut self,
candidates: Vec<SocketAddr>,
max_nodes: usize,
now: Instant,
) -> Vec<SocketAddr> {
let mut selected = Vec::with_capacity(max_nodes);
let mut seen = HashSet::with_capacity(candidates.len());
let mut earliest_backoff: Option<(SocketAddr, Instant)> = None;
for addr in candidates {
if !seen.insert(addr) {
continue;
}
let state = self.states.entry(addr).or_default();
if let Some(backoff_until) = state.backoff_until
&& backoff_until > now
{
if earliest_backoff.is_none_or(|(_, current)| backoff_until < current) {
earliest_backoff = Some((addr, backoff_until));
}
continue;
}
selected.push(addr);
if selected.len() >= max_nodes {
return selected;
}
}
if selected.is_empty()
&& max_nodes > 0
&& let Some((addr, _)) = earliest_backoff
{
selected.push(addr);
}
selected
}
pub(crate) fn mark_attempt(&mut self, addr: SocketAddr, now: Instant) {
self.states.entry(addr).or_default().last_attempt = Some(now);
}
pub(crate) fn mark_success(&mut self, addr: SocketAddr, now: Instant) {
let state = self.states.entry(addr).or_default();
state.last_success = Some(now);
state.fail_count = 0;
state.backoff_until = None;
}
pub(crate) fn mark_timeout(&mut self, addr: SocketAddr, now: Instant) {
let state = self.states.entry(addr).or_default();
state.fail_count = state.fail_count.saturating_add(1);
let multiplier = 1u32
.checked_shl(state.fail_count.saturating_sub(1).min(16))
.unwrap_or(u32::MAX);
let backoff = self
.backoff_base
.saturating_mul(multiplier)
.min(self.backoff_max);
state.backoff_until = Some(now + backoff);
}
}
pub(crate) async fn resolve_bootstrap_nodes(hosts: &[String], netmode: NetMode) -> Vec<SocketAddr> {
let mut resolved = Vec::new();
for host in hosts {
if let Ok(addrs) = tokio::net::lookup_host(host).await {
for addr in addrs {
if !addr_allowed_by_netmode(&addr, netmode) || !is_valid_node_addr(&addr) {
continue;
}
resolved.push(addr);
}
}
}
resolved
}
#[cfg(test)]
mod tests {
use super::*;
use crate::types::CrawlOptions;
fn test_config() -> ResolvedCrawlConfig {
ResolvedCrawlConfig::from_options(&CrawlOptions::default())
}
#[test]
fn bootstrap_pool_backs_off_dead_sources_without_spending_quota() {
let start = Instant::now();
let addr1: SocketAddr = "8.8.8.8:6881".parse().unwrap();
let addr2: SocketAddr = "1.1.1.1:6881".parse().unwrap();
let mut pool = BootstrapSourcePool::new(
vec!["example.invalid:6881".to_string()],
Duration::from_secs(300),
Duration::from_secs(3600),
);
pool.mark_timeout(addr1, start);
let selected = pool.select(vec![addr1, addr2], 1, start + Duration::from_secs(1));
assert_eq!(selected, vec![addr2]);
pool.mark_success(addr1, start + Duration::from_secs(2));
let selected = pool.select(vec![addr1], 1, start + Duration::from_secs(3));
assert_eq!(selected, vec![addr1]);
}
#[test]
fn bootstrap_pool_forces_one_retry_when_all_sources_backed_off() {
let start = Instant::now();
let addr1: SocketAddr = "8.8.8.8:6881".parse().unwrap();
let addr2: SocketAddr = "1.1.1.1:6881".parse().unwrap();
let mut pool = BootstrapSourcePool::new(
vec!["example.invalid:6881".to_string()],
Duration::from_secs(300),
Duration::from_secs(3600),
);
pool.mark_timeout(addr1, start);
pool.mark_timeout(addr2, start);
let selected = pool.select(vec![addr1, addr2], 3, start + Duration::from_secs(1));
assert_eq!(selected.len(), 1);
}
#[test]
fn bootstrap_pool_deduplicates_resolved_addresses() {
let start = Instant::now();
let addr: SocketAddr = "8.8.8.8:6881".parse().unwrap();
let mut pool = BootstrapSourcePool::new(
vec!["example.invalid:6881".to_string()],
Duration::from_secs(300),
Duration::from_secs(3600),
);
let selected = pool.select(vec![addr, addr], 10, start);
assert_eq!(selected, vec![addr]);
}
#[test]
fn bootstrap_gate_uses_pool_low_water_mark() {
let start = Instant::now();
let config = test_config();
let mut gate = BootstrapGate::new();
assert!(!gate.should_bootstrap(config.low_watermark, &config, start));
assert!(gate.should_bootstrap(0, &config, start));
assert!(!gate.should_bootstrap(
999,
&config,
start + config.bootstrap_interval - Duration::from_secs(1)
));
assert!(gate.should_bootstrap(999, &config, start + config.bootstrap_interval));
}
}
+182
View File
@@ -0,0 +1,182 @@
// 负责实现网络查询共享速率预算和突发容量控制
use std::{
sync::{Arc, Mutex},
time::{Duration, Instant},
};
/// Single-owner token bucket. It deliberately contains no atomics or locks.
pub(crate) struct RateBucket {
rate_per_sec: f64,
capacity: f64,
tokens: f64,
last_refill: Instant,
}
#[derive(Clone)]
pub(crate) struct SharedRateBudget {
bucket: Arc<Mutex<RateBucket>>,
}
impl SharedRateBudget {
pub(crate) fn per_second(rate_per_sec: u32, burst: u32, initially_full: bool) -> Self {
Self {
bucket: Arc::new(Mutex::new(RateBucket::per_second(
rate_per_sec,
burst,
initially_full,
Instant::now(),
))),
}
}
pub(crate) fn try_take_one(&self, now: Instant) -> bool {
self.bucket
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.try_take_one(now)
}
pub(crate) fn refund_one(&self) {
self.bucket
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.refund_one();
}
}
impl RateBucket {
pub(crate) fn per_second(
rate_per_sec: u32,
burst: u32,
initially_full: bool,
now: Instant,
) -> Self {
Self::new(
f64::from(rate_per_sec),
f64::from(burst),
initially_full,
now,
)
}
pub(crate) fn per_minute(
rate_per_minute: u32,
burst: u32,
initially_full: bool,
now: Instant,
) -> Self {
Self::new(
f64::from(rate_per_minute) / 60.0,
f64::from(burst),
initially_full,
now,
)
}
fn new(rate_per_sec: f64, capacity: f64, initially_full: bool, now: Instant) -> Self {
let capacity = if rate_per_sec <= 0.0 {
0.0
} else {
capacity.max(1.0)
};
Self {
rate_per_sec,
capacity,
tokens: if initially_full { capacity } else { 0.0 },
last_refill: now,
}
}
pub(crate) fn set_per_second_rate(&mut self, rate_per_sec: u32, now: Instant) {
self.refill(now);
self.rate_per_sec = f64::from(rate_per_sec);
if self.rate_per_sec <= 0.0 {
self.tokens = 0.0;
} else {
self.tokens = self.tokens.min(self.capacity);
}
}
pub(crate) fn try_take_one(&mut self, now: Instant) -> bool {
self.try_take_exact(1, now)
}
pub(crate) fn try_take_exact(&mut self, count: usize, now: Instant) -> bool {
if count == 0 {
return true;
}
if self.rate_per_sec <= 0.0 || self.capacity <= 0.0 {
return false;
}
self.refill(now);
if self.tokens < count as f64 {
return false;
}
self.tokens -= count as f64;
true
}
pub(crate) fn try_take(&mut self, max: usize, now: Instant) -> usize {
if max == 0 || self.rate_per_sec <= 0.0 || self.capacity <= 0.0 {
return 0;
}
self.refill(now);
let taken = (self.tokens.floor() as usize).min(max);
self.tokens -= taken as f64;
taken
}
pub(crate) fn refund_one(&mut self) {
self.refund(1);
}
pub(crate) fn refund(&mut self, count: usize) {
self.tokens = (self.tokens + count as f64).min(self.capacity);
}
fn refill(&mut self, now: Instant) {
let elapsed = now
.checked_duration_since(self.last_refill)
.unwrap_or(Duration::ZERO)
.as_secs_f64();
if elapsed > 0.0 {
self.tokens = (self.tokens + elapsed * self.rate_per_sec).min(self.capacity);
self.last_refill = now;
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn second_bucket_is_smooth_and_bounded() {
let start = Instant::now();
let mut bucket = RateBucket::per_second(100, 20, false, start);
assert_eq!(bucket.try_take(100, start), 0);
assert_eq!(bucket.try_take(100, start + Duration::from_millis(100)), 10);
assert_eq!(bucket.try_take(100, start + Duration::from_secs(10)), 20);
}
#[test]
fn minute_bucket_refills_fractionally() {
let start = Instant::now();
let mut bucket = RateBucket::per_minute(600, 10, false, start);
assert_eq!(bucket.try_take(100, start + Duration::from_millis(500)), 5);
}
#[test]
fn shared_budget_is_global_across_clones() {
let budget = SharedRateBudget::per_second(10, 2, true);
let clone = budget.clone();
let now = Instant::now();
assert!(budget.try_take_one(now));
assert!(clone.try_take_one(now));
assert!(!budget.try_take_one(now));
clone.refund_one();
assert!(budget.try_take_one(now));
}
}
+139
View File
@@ -0,0 +1,139 @@
// 负责解析抓取配置并生成运行时资源限制
use crate::types::CrawlOptions;
use std::time::Duration;
#[derive(Debug, Clone)]
pub(crate) struct ResolvedCrawlConfig {
pub(crate) max_find_node_rate_per_sec: u32,
pub(crate) burst: u32,
pub(crate) max_in_flight: usize,
pub(crate) request_timeout: Duration,
pub(crate) max_new_destinations_per_minute: u32,
pub(crate) max_response_rate_per_sec: u32,
pub(crate) max_response_bytes_per_sec: u64,
pub(crate) max_response_rate_per_source: u32,
pub(crate) metadata_pressure_floor_percent: u8,
pub(crate) pool_capacity: usize,
pub(crate) max_replacements_per_minute: u32,
pub(crate) recent_probe_ttl: Duration,
pub(crate) responsive_capacity: usize,
pub(crate) responsive_ttl: Duration,
pub(crate) low_watermark: usize,
pub(crate) max_in_flight_per_subnet: usize,
pub(crate) bootstrap_nodes: Vec<String>,
pub(crate) bootstrap_interval: Duration,
pub(crate) bootstrap_max_nodes_per_round: usize,
pub(crate) bootstrap_backoff_base: Duration,
pub(crate) bootstrap_backoff_max: Duration,
pub(crate) random_walk_percent: u8,
pub(crate) sparse_bucket_percent: u8,
pub(crate) neighbor_sender_id: bool,
pub(crate) priority_event_channel_capacity: usize,
pub(crate) discovery_event_channel_capacity: usize,
pub(crate) event_batch_limit: usize,
pub(crate) node_batch_limit: usize,
pub(crate) routing_snapshot_size: usize,
pub(crate) snapshot_refresh: Duration,
}
impl ResolvedCrawlConfig {
pub(crate) fn from_options(options: &CrawlOptions) -> Self {
let capacity = options.pool.capacity.max(1);
Self {
max_find_node_rate_per_sec: options.rate_limit.max_find_node_rate_per_sec,
burst: if options.rate_limit.max_find_node_rate_per_sec == 0 {
0
} else {
options.rate_limit.burst.max(1)
},
max_in_flight: options.rate_limit.max_in_flight.max(1),
request_timeout: Duration::from_secs(options.rate_limit.request_timeout_secs.max(1)),
max_new_destinations_per_minute: options.rate_limit.max_new_destinations_per_minute,
max_response_rate_per_sec: options.rate_limit.max_response_rate_per_sec,
max_response_bytes_per_sec: options.rate_limit.max_response_bytes_per_sec,
max_response_rate_per_source: options.rate_limit.max_response_rate_per_source,
metadata_pressure_floor_percent: options
.rate_limit
.metadata_pressure_floor_percent
.min(100),
pool_capacity: capacity,
max_replacements_per_minute: options.rate_limit.max_replacements_per_minute,
recent_probe_ttl: Duration::from_secs(options.pool.recent_probe_ttl_secs.max(1)),
responsive_capacity: options.pool.responsive_capacity.max(1),
responsive_ttl: Duration::from_secs(options.pool.responsive_ttl_secs.max(1)),
low_watermark: options.pool.low_watermark.min(capacity),
max_in_flight_per_subnet: options.rate_limit.max_in_flight_per_subnet.max(1),
bootstrap_nodes: if options.bootstrap.nodes.is_empty() {
crate::types::BootstrapOptions::default().nodes
} else {
options.bootstrap.nodes.clone()
},
bootstrap_interval: Duration::from_secs(options.bootstrap.interval_secs),
bootstrap_max_nodes_per_round: options.bootstrap.max_nodes_per_round,
bootstrap_backoff_base: Duration::from_secs(
options.bootstrap.source_backoff_base_secs.max(1),
),
bootstrap_backoff_max: Duration::from_secs(
options.bootstrap.source_backoff_max_secs.max(1),
),
random_walk_percent: options.target.random_walk_percent.min(100),
sparse_bucket_percent: options.target.sparse_bucket_percent.min(100),
neighbor_sender_id: options.target.neighbor_sender_id,
priority_event_channel_capacity: options
.scheduler
.priority_event_channel_capacity
.max(1),
discovery_event_channel_capacity: options
.scheduler
.discovery_event_channel_capacity
.max(1),
event_batch_limit: options.scheduler.event_batch_limit.max(1),
node_batch_limit: options.scheduler.node_batch_limit.max(1),
routing_snapshot_size: options.scheduler.routing_snapshot_size.max(1),
snapshot_refresh: Duration::from_millis(
options.scheduler.snapshot_refresh_millis.max(100),
),
}
}
pub(crate) fn rate_for_metadata_pressure(&self, pressure: f64) -> u32 {
let max = f64::from(self.max_find_node_rate_per_sec);
if pressure < 0.80 {
return self.max_find_node_rate_per_sec;
}
let floor = max * f64::from(self.metadata_pressure_floor_percent) / 100.0;
if pressure >= 0.95 {
return floor.round() as u32;
}
let progress = ((pressure - 0.80) / 0.15).clamp(0.0, 1.0);
(max - (max - floor) * progress).round() as u32
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn resolves_pool_and_rate_limit() {
let mut options = CrawlOptions::default();
options.rate_limit.max_find_node_rate_per_sec = 200;
options.rate_limit.metadata_pressure_floor_percent = 25;
options.pool.capacity = 42;
let resolved = ResolvedCrawlConfig::from_options(&options);
assert_eq!(resolved.pool_capacity, 42);
assert_eq!(resolved.rate_for_metadata_pressure(0.79), 200);
assert_eq!(resolved.rate_for_metadata_pressure(0.95), 50);
assert_eq!(resolved.rate_for_metadata_pressure(1.0), 50);
}
}
+974
View File
@@ -0,0 +1,974 @@
// 负责协调节点探测 Peer 查找采样发现和 Metadata 调度
use crate::bootstrap::{BootstrapGate, BootstrapSourcePool, resolve_bootstrap_nodes};
use crate::budget::{RateBucket, SharedRateBudget};
use crate::crawl_config::ResolvedCrawlConfig;
use crate::krpc::{for_each_response_node, send_find_node_query};
use crate::node_id::{
TransactionId, bucket_index, neighbor_node_id, random_node_id, target_for_bucket,
};
use crate::node_pool::{AdmissionOutcome, NodePool, ResponsiveReservoir, SubnetKey};
use crate::protocol::DhtResponse;
use crate::routing_snapshot::RoutingSnapshot;
#[cfg(test)]
use crate::runtime_stats::DhtRuntimeLimits;
use crate::runtime_stats::DhtRuntimeStats;
use crate::sample_infohashes::SampleCandidateRouter;
use crate::types::{NetMode, NodeTuple};
use ahash::AHashMap;
use arc_swap::ArcSwap;
use bytes::BytesMut;
#[cfg(feature = "metrics")]
use metrics::{counter, gauge};
use rand::RngExt;
use std::collections::{HashMap, VecDeque};
use std::net::SocketAddr;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
use tokio::net::UdpSocket;
use tokio::sync::mpsc;
use tokio_util::sync::CancellationToken;
const SCHEDULE_INTERVAL: Duration = Duration::from_millis(5);
const MAX_POOL_SCAN_PER_SCHEDULE: usize = 256;
#[derive(Debug, Clone, Copy, Hash, PartialEq, Eq)]
struct PendingKey {
addr: SocketAddr,
tid: TransactionId,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum ProbePurpose {
New,
Revisit,
Bootstrap,
}
#[derive(Debug, Clone, Copy)]
struct PendingRequest {
node: NodeTuple,
purpose: ProbePurpose,
deadline: Instant,
subnet: SubnetKey,
}
enum PriorityEvent {
Response {
remote_addr: SocketAddr,
tid: TransactionId,
response: DhtResponse,
},
SendFailed(PendingKey),
BootstrapResolved(Vec<SocketAddr>),
}
struct OutboundRequest {
key: PendingKey,
node: NodeTuple,
target: [u8; 20],
sender_id: [u8; 20],
}
pub(crate) struct CrawlEngine {
config: ResolvedCrawlConfig,
priority_tx: mpsc::Sender<PriorityEvent>,
discovery_tx: mpsc::Sender<NodeTuple>,
/// One-shot handoff used only by `DHTServer::start`; never touched by the crawl hot path.
receivers: Mutex<Option<(mpsc::Receiver<PriorityEvent>, mpsc::Receiver<NodeTuple>)>>,
pub(crate) snapshot: Arc<ArcSwap<RoutingSnapshot>>,
pub(crate) node_count: Arc<AtomicUsize>,
runtime_stats: DhtRuntimeStats,
outbound_query_budget: SharedRateBudget,
sample_candidate_router: Option<SampleCandidateRouter>,
}
impl CrawlEngine {
pub(crate) fn new(
config: ResolvedCrawlConfig,
runtime_stats: DhtRuntimeStats,
outbound_query_budget: SharedRateBudget,
sample_candidate_router: Option<SampleCandidateRouter>,
) -> Self {
let (priority_tx, priority_rx) = mpsc::channel(config.priority_event_channel_capacity);
let (discovery_tx, discovery_rx) = mpsc::channel(config.discovery_event_channel_capacity);
Self {
config,
priority_tx,
discovery_tx,
receivers: Mutex::new(Some((priority_rx, discovery_rx))),
snapshot: Arc::new(ArcSwap::from_pointee(RoutingSnapshot::default())),
node_count: Arc::new(AtomicUsize::new(0)),
runtime_stats,
outbound_query_budget,
sample_candidate_router,
}
}
pub(crate) fn route_discovered(&self, node: NodeTuple) {
if self
.sample_candidate_router
.as_ref()
.is_some_and(|router| router.route(node))
{
return;
}
let enqueue_result = self.discovery_tx.try_send(node);
self.runtime_stats.set_crawl_discovery_queue_depth(
self.discovery_tx
.max_capacity()
.saturating_sub(self.discovery_tx.capacity()),
);
if enqueue_result.is_err() {
self.runtime_stats.crawl_event_dropped_discovered();
#[cfg(feature = "metrics")]
counter!("dht_crawl_events_dropped_total", "kind" => "discovered").increment(1);
}
}
pub(crate) fn route_response(
&self,
remote_addr: SocketAddr,
tid: TransactionId,
response: DhtResponse,
) {
let enqueue_result = self.priority_tx.try_send(PriorityEvent::Response {
remote_addr,
tid,
response,
});
self.runtime_stats.set_crawl_priority_queue_depth(
self.priority_tx
.max_capacity()
.saturating_sub(self.priority_tx.capacity()),
);
if enqueue_result.is_err() {
self.runtime_stats.crawl_event_dropped_response();
#[cfg(feature = "metrics")]
counter!("dht_crawl_events_dropped_total", "kind" => "response").increment(1);
}
}
pub(crate) fn spawn(
&self,
netmode: NetMode,
local_id: [u8; 20],
sockets: &HashMap<SocketAddr, Arc<UdpSocket>>,
metadata_queue_len: Arc<AtomicUsize>,
max_metadata_queue_size: usize,
shutdown: CancellationToken,
) {
let Some((priority_rx, discovery_rx)) = self
.receivers
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.take()
else {
return;
};
let mut egress_v4 = None;
let mut egress_v6 = None;
for (bind_addr, socket) in sockets.iter() {
let (tx, rx) = mpsc::channel(self.config.max_in_flight.max(1));
spawn_egress(
socket.clone(),
rx,
self.priority_tx.clone(),
self.runtime_stats.clone(),
shutdown.clone(),
);
if bind_addr.is_ipv4() {
egress_v4 = Some(tx);
} else {
egress_v6 = Some(tx);
}
}
let actor = CrawlActor::new(CrawlActorInit {
config: self.config.clone(),
netmode,
local_id,
priority_rx,
discovery_rx,
priority_tx: self.priority_tx.clone(),
egress_v4,
egress_v6,
snapshot: self.snapshot.clone(),
node_count: self.node_count.clone(),
metadata_queue_len,
max_metadata_queue_size,
runtime_stats: self.runtime_stats.clone(),
outbound_query_budget: self.outbound_query_budget.clone(),
shutdown,
});
tokio::spawn(actor.run());
}
}
fn spawn_egress(
socket: Arc<UdpSocket>,
mut rx: mpsc::Receiver<OutboundRequest>,
priority_tx: mpsc::Sender<PriorityEvent>,
runtime_stats: DhtRuntimeStats,
shutdown: CancellationToken,
) {
tokio::spawn(async move {
let mut buffer = BytesMut::with_capacity(128);
loop {
tokio::select! {
_ = shutdown.cancelled() => break,
request = rx.recv() => {
let Some(request) = request else { break };
if !send_find_node_query(
&request.node.addr,
&request.key.tid,
&request.target,
&request.sender_id,
&socket,
&mut buffer,
).await {
let _ = priority_tx.try_send(PriorityEvent::SendFailed(request.key));
runtime_stats.set_crawl_priority_queue_depth(
priority_tx
.max_capacity()
.saturating_sub(priority_tx.capacity()),
);
} else {
runtime_stats.udp_sent(buffer.len());
}
}
}
}
});
}
#[derive(Default)]
struct ActorMetrics {
admitted: u64,
replaced: u64,
duplicate: u64,
admission_limited: u64,
invalid: u64,
queries_new: u64,
queries_revisit: u64,
queries_bootstrap: u64,
responses: u64,
timeouts: u64,
send_failures: u64,
unmatched_responses: u64,
}
impl ActorMetrics {
fn record_admission(&mut self, outcome: AdmissionOutcome) {
match outcome {
AdmissionOutcome::Admitted => self.admitted += 1,
AdmissionOutcome::Replaced => self.replaced += 1,
AdmissionOutcome::Duplicate => self.duplicate += 1,
AdmissionOutcome::RateLimited => self.admission_limited += 1,
AdmissionOutcome::Invalid => self.invalid += 1,
}
}
}
fn record_runtime_admission(stats: &DhtRuntimeStats, outcome: AdmissionOutcome) {
match outcome {
AdmissionOutcome::Admitted => stats.node_admitted(),
AdmissionOutcome::Replaced => stats.node_replaced(),
AdmissionOutcome::Duplicate => stats.node_dropped_duplicate(),
AdmissionOutcome::RateLimited => stats.node_dropped_rate_limited(),
AdmissionOutcome::Invalid => stats.node_dropped_invalid(),
}
}
struct CrawlActorInit {
config: ResolvedCrawlConfig,
netmode: NetMode,
local_id: [u8; 20],
priority_rx: mpsc::Receiver<PriorityEvent>,
discovery_rx: mpsc::Receiver<NodeTuple>,
priority_tx: mpsc::Sender<PriorityEvent>,
egress_v4: Option<mpsc::Sender<OutboundRequest>>,
egress_v6: Option<mpsc::Sender<OutboundRequest>>,
snapshot: Arc<ArcSwap<RoutingSnapshot>>,
node_count: Arc<AtomicUsize>,
metadata_queue_len: Arc<AtomicUsize>,
max_metadata_queue_size: usize,
runtime_stats: DhtRuntimeStats,
outbound_query_budget: SharedRateBudget,
shutdown: CancellationToken,
}
struct CrawlActor {
config: ResolvedCrawlConfig,
netmode: NetMode,
local_id: [u8; 20],
priority_rx: mpsc::Receiver<PriorityEvent>,
discovery_rx: mpsc::Receiver<NodeTuple>,
priority_tx: mpsc::Sender<PriorityEvent>,
egress_v4: Option<mpsc::Sender<OutboundRequest>>,
egress_v6: Option<mpsc::Sender<OutboundRequest>>,
pool: NodePool,
responsive: ResponsiveReservoir,
pending: AHashMap<PendingKey, PendingRequest>,
pending_expiry: VecDeque<(Instant, PendingKey)>,
subnet_in_flight: AHashMap<SubnetKey, usize>,
query_budget: RateBucket,
destination_budget: RateBucket,
bootstrap_gate: BootstrapGate,
bootstrap_pool: BootstrapSourcePool,
bootstrap_queue: VecDeque<SocketAddr>,
snapshot: Arc<ArcSwap<RoutingSnapshot>>,
node_count: Arc<AtomicUsize>,
metadata_queue_len: Arc<AtomicUsize>,
max_metadata_queue_size: usize,
next_tid: u64,
metrics: ActorMetrics,
runtime_stats: DhtRuntimeStats,
outbound_query_budget: SharedRateBudget,
shutdown: CancellationToken,
}
impl CrawlActor {
fn new(init: CrawlActorInit) -> Self {
let CrawlActorInit {
config,
netmode,
local_id,
priority_rx,
discovery_rx,
priority_tx,
egress_v4,
egress_v6,
snapshot,
node_count,
metadata_queue_len,
max_metadata_queue_size,
runtime_stats,
outbound_query_budget,
shutdown,
} = init;
let now = Instant::now();
let destination_burst = config.max_new_destinations_per_minute.div_ceil(60).max(1);
Self {
pool: NodePool::new(
config.pool_capacity,
config.max_replacements_per_minute,
config.recent_probe_ttl,
now,
),
responsive: ResponsiveReservoir::new(config.responsive_capacity, config.responsive_ttl),
pending: AHashMap::with_capacity(config.max_in_flight),
pending_expiry: VecDeque::with_capacity(config.max_in_flight),
subnet_in_flight: AHashMap::new(),
query_budget: RateBucket::per_second(
config.max_find_node_rate_per_sec,
config.burst,
false,
now,
),
destination_budget: RateBucket::per_minute(
config.max_new_destinations_per_minute,
destination_burst,
false,
now,
),
bootstrap_pool: BootstrapSourcePool::new(
config.bootstrap_nodes.clone(),
config.bootstrap_backoff_base,
config.bootstrap_backoff_max,
),
config,
netmode,
local_id,
priority_rx,
discovery_rx,
priority_tx,
egress_v4,
egress_v6,
bootstrap_gate: BootstrapGate::new(),
bootstrap_queue: VecDeque::new(),
snapshot,
node_count,
metadata_queue_len,
max_metadata_queue_size,
next_tid: 1,
metrics: ActorMetrics::default(),
runtime_stats,
outbound_query_budget,
shutdown,
}
}
async fn run(mut self) {
let mut schedule_tick = tokio::time::interval(SCHEDULE_INTERVAL);
schedule_tick.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
let mut snapshot_tick = tokio::time::interval(self.config.snapshot_refresh);
snapshot_tick.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
let mut metrics_tick = tokio::time::interval(Duration::from_secs(1));
metrics_tick.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
loop {
tokio::select! {
biased;
_ = self.shutdown.cancelled() => break,
_ = schedule_tick.tick() => self.on_schedule_tick(Instant::now()),
_ = snapshot_tick.tick() => self.publish_snapshot(Instant::now()),
_ = metrics_tick.tick() => self.flush_metrics(Instant::now()),
event = self.priority_rx.recv() => {
self.runtime_stats
.set_crawl_priority_queue_depth(self.priority_rx.len());
let Some(event) = event else { break };
self.handle_priority(event, Instant::now());
self.drain_events();
}
node = self.discovery_rx.recv() => {
self.runtime_stats
.set_crawl_discovery_queue_depth(self.discovery_rx.len());
let Some(node) = node else { break };
self.admit(node, Instant::now());
self.drain_events();
}
}
}
self.runtime_stats.set_crawl_priority_queue_depth(0);
self.runtime_stats.set_crawl_discovery_queue_depth(0);
}
fn drain_events(&mut self) {
let mut events = 1;
let mut nodes = 0;
while events < self.config.event_batch_limit && nodes < self.config.node_batch_limit {
if events % 8 == 0
&& let Ok(node) = self.discovery_rx.try_recv()
{
self.runtime_stats
.set_crawl_discovery_queue_depth(self.discovery_rx.len());
self.admit(node, Instant::now());
events += 1;
nodes += 1;
continue;
}
if let Ok(event) = self.priority_rx.try_recv() {
self.runtime_stats
.set_crawl_priority_queue_depth(self.priority_rx.len());
self.handle_priority(event, Instant::now());
events += 1;
continue;
}
if let Ok(node) = self.discovery_rx.try_recv() {
self.runtime_stats
.set_crawl_discovery_queue_depth(self.discovery_rx.len());
self.admit(node, Instant::now());
events += 1;
nodes += 1;
continue;
}
break;
}
}
fn handle_priority(&mut self, event: PriorityEvent, now: Instant) {
match event {
PriorityEvent::Response {
remote_addr,
tid,
response,
} => self.handle_response(remote_addr, tid, response, now),
PriorityEvent::SendFailed(key) => {
if let Some(pending) = self.pending.remove(&key) {
self.release_pending(pending);
self.metrics.send_failures += 1;
self.runtime_stats.send_failure();
if pending.purpose == ProbePurpose::Bootstrap {
self.bootstrap_pool.mark_timeout(pending.node.addr, now);
}
}
}
PriorityEvent::BootstrapResolved(candidates) => {
let selected = self.bootstrap_pool.select(
candidates,
self.config.bootstrap_max_nodes_per_round,
now,
);
self.bootstrap_queue.extend(selected);
}
}
}
fn handle_response(
&mut self,
remote_addr: SocketAddr,
tid: TransactionId,
response: DhtResponse,
now: Instant,
) {
let key = PendingKey {
addr: remote_addr,
tid,
};
let Some(pending) = self.pending.remove(&key) else {
self.metrics.unmatched_responses += 1;
self.runtime_stats.unmatched_response();
return;
};
self.release_pending(pending);
self.metrics.responses += 1;
self.runtime_stats.response();
if pending.purpose == ProbePurpose::Bootstrap {
self.bootstrap_pool.mark_success(remote_addr, now);
}
let mut responsive_node = pending.node;
if let Some(id) = response.id.as_ref()
&& let Ok(id) = <[u8; 20]>::try_from(id.as_slice())
{
responsive_node.id = id;
}
self.responsive.record(responsive_node, now);
let pool = &mut self.pool;
let metrics = &mut self.metrics;
let runtime_stats = &self.runtime_stats;
for_each_response_node(&response, self.netmode, |node| {
let outcome = pool.admit(node, now);
metrics.record_admission(outcome);
record_runtime_admission(runtime_stats, outcome);
});
self.sync_node_count();
}
fn admit(&mut self, node: NodeTuple, now: Instant) {
let outcome = self.pool.admit(node, now);
self.metrics.record_admission(outcome);
record_runtime_admission(&self.runtime_stats, outcome);
self.sync_node_count();
}
fn on_schedule_tick(&mut self, now: Instant) {
self.expire_pending(now);
self.maybe_resolve_bootstrap(now);
let rate = self
.config
.rate_for_metadata_pressure(self.metadata_pressure());
self.runtime_stats.set_find_node_effective_rate(rate);
self.query_budget.set_per_second_rate(rate, now);
let budget = self.query_budget.try_take(self.config.burst as usize, now);
for _ in 0..budget {
if self.pending.len() >= self.config.max_in_flight {
self.query_budget.refund_one();
break;
}
if !self.schedule_one(now) {
self.query_budget.refund_one();
break;
}
}
self.sync_node_count();
}
fn schedule_one(&mut self, now: Instant) -> bool {
if !self.outbound_query_budget.try_take_one(now) {
return false;
}
let scheduled = self.schedule_one_with_budget(now);
if !scheduled {
self.outbound_query_budget.refund_one();
}
scheduled
}
fn schedule_one_with_budget(&mut self, now: Instant) -> bool {
if self.pool.len() < self.config.low_watermark
&& let Some(addr) = self.bootstrap_queue.front().copied()
{
let is_new = !self.pool.contains_recent(&addr, now);
if is_new && !self.destination_budget.try_take_one(now) {
return self.schedule_revisit(now);
}
let node = NodeTuple {
id: self.local_id,
addr,
};
if self.try_dispatch(node, ProbePurpose::Bootstrap, now) {
self.bootstrap_queue.pop_front();
self.pool.record_probe(addr, now);
self.bootstrap_pool.mark_attempt(addr, now);
self.metrics.queries_bootstrap += 1;
self.runtime_stats.query_bootstrap();
return true;
}
if is_new {
self.destination_budget.refund_one();
}
}
let scan_limit = self.pool.len().min(MAX_POOL_SCAN_PER_SCHEDULE);
for _ in 0..scan_limit {
let Some(node) = self.pool.front() else {
break;
};
let subnet = SubnetKey::from_addr(&node.addr);
if self.subnet_count(&subnet) >= self.config.max_in_flight_per_subnet {
self.pool.rotate_front_to_back();
continue;
}
if !self.destination_budget.try_take_one(now) {
return self.schedule_revisit(now);
}
let node = self
.pool
.take_front_for_probe(now)
.expect("front node exists");
if self.try_dispatch(node, ProbePurpose::New, now) {
self.metrics.queries_new += 1;
self.runtime_stats.query_new();
return true;
}
self.pool.restore_front(node, now);
self.destination_budget.refund_one();
break;
}
self.schedule_revisit(now)
}
fn schedule_revisit(&mut self, now: Instant) -> bool {
let Some(node) = self.responsive.next_revisit(now) else {
return false;
};
let was_recent = self.pool.contains_recent(&node.addr, now);
if !was_recent && !self.destination_budget.try_take_one(now) {
return false;
}
if self.try_dispatch(node, ProbePurpose::Revisit, now) {
self.pool.record_probe(node.addr, now);
self.metrics.queries_revisit += 1;
self.runtime_stats.query_revisit();
return true;
}
if !was_recent {
self.destination_budget.refund_one();
}
false
}
fn try_dispatch(&mut self, node: NodeTuple, purpose: ProbePurpose, now: Instant) -> bool {
if self.pending.len() >= self.config.max_in_flight {
return false;
}
let subnet = SubnetKey::from_addr(&node.addr);
if self.subnet_count(&subnet) >= self.config.max_in_flight_per_subnet {
return false;
}
let tx = if node.addr.is_ipv4() {
self.egress_v4.clone()
} else {
self.egress_v6.clone()
};
let Some(tx) = tx else {
return false;
};
let Ok(permit) = tx.try_reserve() else {
return false;
};
let tid = self.next_tid.to_be_bytes();
self.next_tid = self.next_tid.wrapping_add(1).max(1);
let key = PendingKey {
addr: node.addr,
tid,
};
let deadline = now + self.config.request_timeout;
self.pending.insert(
key,
PendingRequest {
node,
purpose,
deadline,
subnet,
},
);
self.pending_expiry.push_back((deadline, key));
*self.subnet_in_flight.entry(subnet).or_insert(0) += 1;
self.runtime_stats
.set_find_node_in_flight(self.pending.len());
let sender_id = if self.config.neighbor_sender_id {
let generated = neighbor_node_id(&node.id, &self.local_id);
generated
.as_slice()
.try_into()
.expect("neighbor id is always 20 bytes")
} else {
self.local_id
};
permit.send(OutboundRequest {
key,
node,
target: self.choose_target(&node),
sender_id,
});
true
}
fn choose_target(&self, node: &NodeTuple) -> [u8; 20] {
let total = self
.config
.random_walk_percent
.saturating_add(self.config.sparse_bucket_percent)
.max(1);
if rand::rng().random_range(0..total) < self.config.sparse_bucket_percent {
target_for_bucket(&self.local_id, bucket_index(&node.id, &self.local_id))
} else {
random_node_id()
}
}
fn expire_pending(&mut self, now: Instant) {
while let Some((deadline, key)) = self.pending_expiry.front().copied() {
if deadline > now {
break;
}
self.pending_expiry.pop_front();
let should_remove = self
.pending
.get(&key)
.is_some_and(|pending| pending.deadline == deadline);
if !should_remove {
continue;
}
let pending = self.pending.remove(&key).expect("pending entry exists");
self.release_pending(pending);
self.metrics.timeouts += 1;
self.runtime_stats.timeout();
if pending.purpose == ProbePurpose::Bootstrap {
self.bootstrap_pool.mark_timeout(pending.node.addr, now);
}
}
}
fn release_pending(&mut self, pending: PendingRequest) {
if let Some(count) = self.subnet_in_flight.get_mut(&pending.subnet) {
*count = count.saturating_sub(1);
if *count == 0 {
self.subnet_in_flight.remove(&pending.subnet);
}
}
self.runtime_stats
.set_find_node_in_flight(self.pending.len());
}
fn subnet_count(&self, subnet: &SubnetKey) -> usize {
self.subnet_in_flight.get(subnet).copied().unwrap_or(0)
}
fn maybe_resolve_bootstrap(&mut self, now: Instant) {
if !self.bootstrap_queue.is_empty()
|| !self
.bootstrap_gate
.should_bootstrap(self.pool.len(), &self.config, now)
{
return;
}
let hosts = self.bootstrap_pool.hosts.clone();
let netmode = self.netmode;
let priority_tx = self.priority_tx.clone();
let runtime_stats = self.runtime_stats.clone();
tokio::spawn(async move {
let resolved = resolve_bootstrap_nodes(&hosts, netmode).await;
let _ = priority_tx.try_send(PriorityEvent::BootstrapResolved(resolved));
runtime_stats.set_crawl_priority_queue_depth(
priority_tx
.max_capacity()
.saturating_sub(priority_tx.capacity()),
);
});
}
fn publish_snapshot(&self, now: Instant) {
let nodes = self
.responsive
.snapshot(self.config.routing_snapshot_size, now);
self.snapshot.store(Arc::new(RoutingSnapshot::from_nodes(
nodes,
self.config.routing_snapshot_size,
)));
}
fn metadata_pressure(&self) -> f64 {
if self.max_metadata_queue_size == 0 {
1.0
} else {
(self.metadata_queue_len.load(Ordering::Relaxed) as f64
/ self.max_metadata_queue_size as f64)
.min(1.0)
}
}
fn sync_node_count(&self) {
self.node_count.store(self.pool.len(), Ordering::Relaxed);
self.runtime_stats.set_node_pool_size(self.pool.len());
}
fn flush_metrics(&mut self, now: Instant) {
#[cfg(feature = "metrics")]
{
let metadata_pressure = self.metadata_pressure();
gauge!("dht_node_pool_size").set(self.pool.len() as f64);
gauge!("dht_node_pool_oldest_age_seconds").set(self.pool.oldest_age(now).as_secs_f64());
gauge!("dht_find_node_in_flight").set(self.pending.len() as f64);
gauge!("dht_metadata_queue_pressure_ratio").set(metadata_pressure);
gauge!("dht_find_node_effective_rate_per_second")
.set(self.config.rate_for_metadata_pressure(metadata_pressure) as f64);
counter!("dht_node_pool_admissions_total").increment(self.metrics.admitted);
counter!("dht_node_pool_replacements_total").increment(self.metrics.replaced);
counter!("dht_node_pool_dropped_total", "reason" => "duplicate")
.increment(self.metrics.duplicate);
counter!("dht_node_pool_dropped_total", "reason" => "rate_limit")
.increment(self.metrics.admission_limited);
counter!("dht_node_pool_dropped_total", "reason" => "invalid")
.increment(self.metrics.invalid);
counter!("dht_crawl_queries_sent_total", "kind" => "new")
.increment(self.metrics.queries_new);
counter!("dht_crawl_queries_sent_total", "kind" => "revisit")
.increment(self.metrics.queries_revisit);
counter!("dht_crawl_queries_sent_total", "kind" => "bootstrap")
.increment(self.metrics.queries_bootstrap);
counter!("dht_find_node_responses_total").increment(self.metrics.responses);
counter!("dht_find_node_timeouts_total").increment(self.metrics.timeouts);
counter!("dht_find_node_send_failures_total").increment(self.metrics.send_failures);
counter!("dht_find_node_response_unmatched_total")
.increment(self.metrics.unmatched_responses);
}
#[cfg(not(feature = "metrics"))]
let _ = now;
self.metrics = ActorMetrics::default();
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::types::CrawlOptions;
fn node(id: u8, addr: &str) -> NodeTuple {
NodeTuple {
id: [id; 20],
addr: addr.parse().unwrap(),
}
}
fn test_actor(
config: ResolvedCrawlConfig,
) -> (CrawlActor, mpsc::Receiver<OutboundRequest>, DhtRuntimeStats) {
let (priority_tx, priority_rx) = mpsc::channel(16);
let (_discovery_tx, discovery_rx) = mpsc::channel(16);
let (egress_tx, egress_rx) = mpsc::channel(config.max_in_flight);
let runtime_stats = DhtRuntimeStats::with_limits(DhtRuntimeLimits {
metadata_queue: 100_000,
node_pool: config.pool_capacity,
node_pool_low_watermark: config.low_watermark,
find_node_in_flight: config.max_in_flight,
initial_find_node_rate: config.max_find_node_rate_per_sec,
hash_ingress_queue: 0,
crawl_priority_queue: config.priority_event_channel_capacity,
crawl_discovery_queue: config.discovery_event_channel_capacity,
});
let actor = CrawlActor::new(CrawlActorInit {
config,
netmode: NetMode::Ipv4Only,
local_id: [7; 20],
priority_rx,
discovery_rx,
priority_tx,
egress_v4: Some(egress_tx),
egress_v6: None,
snapshot: Arc::new(ArcSwap::from_pointee(RoutingSnapshot::default())),
node_count: Arc::new(AtomicUsize::new(0)),
metadata_queue_len: Arc::new(AtomicUsize::new(0)),
max_metadata_queue_size: 100_000,
runtime_stats: runtime_stats.clone(),
outbound_query_budget: SharedRateBudget::per_second(10_000, 10_000, true),
shutdown: CancellationToken::new(),
});
(actor, egress_rx, runtime_stats)
}
#[test]
fn saturated_head_subnet_does_not_block_later_node() {
let config = ResolvedCrawlConfig::from_options(&CrawlOptions::default());
let max_per_subnet = config.max_in_flight_per_subnet;
let max_in_flight = config.max_in_flight;
let (mut actor, mut egress_rx, runtime_stats) = test_actor(config);
let now = Instant::now() + Duration::from_secs(1);
let blocked = node(1, "8.8.8.8:6881");
let eligible = node(2, "1.1.1.1:6881");
assert_eq!(actor.pool.admit(blocked, now), AdmissionOutcome::Admitted);
assert_eq!(actor.pool.admit(eligible, now), AdmissionOutcome::Admitted);
actor
.subnet_in_flight
.insert(SubnetKey::from_addr(&blocked.addr), max_per_subnet);
assert!(actor.schedule_one(now));
let request = egress_rx.try_recv().expect("eligible node was dispatched");
assert_eq!(request.node, eligible);
assert_eq!(actor.pool.front(), Some(blocked));
assert_eq!(actor.pool.admit(blocked, now), AdmissionOutcome::Duplicate);
assert!(!actor.pool.contains_recent(&blocked.addr, now));
assert!(actor.pool.contains_recent(&eligible.addr, now));
let snapshot = runtime_stats.snapshot();
assert_eq!(snapshot.queries_new, 1);
assert_eq!(snapshot.find_node_in_flight, 1);
assert_eq!(snapshot.find_node_in_flight_max, max_in_flight);
}
#[test]
fn runtime_stats_count_dropped_crawl_events() {
let mut options = CrawlOptions::default();
options.scheduler.priority_event_channel_capacity = 1;
options.scheduler.discovery_event_channel_capacity = 1;
let config = ResolvedCrawlConfig::from_options(&options);
let stats = DhtRuntimeStats::with_limits(DhtRuntimeLimits {
metadata_queue: 100,
node_pool: config.pool_capacity,
node_pool_low_watermark: config.low_watermark,
find_node_in_flight: config.max_in_flight,
initial_find_node_rate: config.max_find_node_rate_per_sec,
hash_ingress_queue: 0,
crawl_priority_queue: config.priority_event_channel_capacity,
crawl_discovery_queue: config.discovery_event_channel_capacity,
});
let engine = CrawlEngine::new(
config,
stats.clone(),
SharedRateBudget::per_second(10_000, 10_000, true),
None,
);
engine.route_discovered(node(1, "8.8.8.8:1"));
engine.route_discovered(node(2, "1.1.1.1:2"));
let response = || DhtResponse {
id: None,
nodes: None,
nodes6: None,
values: None,
samples: None,
num: None,
interval: None,
};
engine.route_response("8.8.8.8:1".parse().unwrap(), [1; 8], response());
engine.route_response("1.1.1.1:2".parse().unwrap(), [2; 8], response());
let snapshot = stats.snapshot();
assert_eq!(snapshot.crawl_events_dropped_discovered, 1);
assert_eq!(snapshot.crawl_events_dropped_response, 1);
assert_eq!(snapshot.crawl_discovery_queue_depth, 1);
assert_eq!(snapshot.crawl_discovery_queue_capacity, 1);
assert_eq!(snapshot.crawl_priority_queue_depth, 1);
assert_eq!(snapshot.crawl_priority_queue_capacity, 1);
}
}
+30
View File
@@ -0,0 +1,30 @@
// 负责定义 DHT 基础库的统一错误类型和结果别名
use thiserror::Error;
#[derive(Error, Debug)]
/// Error returned during DHT initialization or execution.
pub enum DHTError {
/// Socket or other network I/O failed.
#[error("网络错误: {0}")]
Network(#[from] std::io::Error),
/// A shared lock was poisoned.
#[error("锁中毒: {0}")]
LockPoisoned(String),
/// Server initialization failed, for example because a socket could not bind.
#[error("初始化错误: {0}")]
Init(String),
/// An internal invariant or worker operation failed.
#[error("内部错误: {0}")]
Internal(String),
/// Another error represented by a human-readable message.
#[error("{0}")]
Other(String),
}
/// Result type used by the crate's public APIs.
pub type Result<T> = std::result::Result<T, DHTError>;
+310
View File
@@ -0,0 +1,310 @@
// 负责编码 DHT KRPC 查询响应和紧凑节点数据
use crate::addr::{addr_allowed_by_netmode, is_valid_node_addr};
use crate::node_id::TransactionId;
use crate::protocol::DhtResponse;
use crate::types::{NetMode, NodeTuple};
use bytes::BytesMut;
#[cfg(feature = "metrics")]
use metrics::{counter, histogram};
use std::net::{IpAddr, Ipv4Addr, Ipv6Addr, SocketAddr};
use std::sync::Arc;
use tokio::net::UdpSocket;
pub(crate) fn for_each_response_node(
response: &DhtResponse,
netmode: NetMode,
mut visit: impl FnMut(NodeTuple),
) -> usize {
let mut count = 0;
if netmode != NetMode::Ipv6Only
&& let Some(nodes) = response.nodes.as_deref()
&& nodes.len() % 26 == 0
{
for chunk in nodes.chunks_exact(26) {
let id: [u8; 20] = chunk[..20].try_into().expect("compact v4 id is 20 bytes");
let ip = Ipv4Addr::new(chunk[20], chunk[21], chunk[22], chunk[23]);
let port = u16::from_be_bytes([chunk[24], chunk[25]]);
let addr = SocketAddr::new(IpAddr::V4(ip), port);
if is_valid_node_addr(&addr) {
visit(NodeTuple { id, addr });
count += 1;
}
}
}
if netmode != NetMode::Ipv4Only
&& let Some(nodes) = response.nodes6.as_deref()
&& nodes.len() % 38 == 0
{
for chunk in nodes.chunks_exact(38) {
let id: [u8; 20] = chunk[..20].try_into().expect("compact v6 id is 20 bytes");
let ip_bytes: [u8; 16] = chunk[20..36]
.try_into()
.expect("compact v6 address is 16 bytes");
let port = u16::from_be_bytes([chunk[36], chunk[37]]);
let addr = SocketAddr::new(IpAddr::V6(Ipv6Addr::from(ip_bytes)), port);
if is_valid_node_addr(&addr) {
visit(NodeTuple { id, addr });
count += 1;
}
}
}
count
}
pub(crate) fn for_each_response_peer(
response: &DhtResponse,
netmode: NetMode,
mut visit: impl FnMut(SocketAddr),
) -> usize {
let mut count = 0;
let Some(values) = response.values.as_ref() else {
return count;
};
for value in values {
let bytes = value.as_ref();
let addr = match bytes.len() {
6 => {
let ip = Ipv4Addr::new(bytes[0], bytes[1], bytes[2], bytes[3]);
let port = u16::from_be_bytes([bytes[4], bytes[5]]);
SocketAddr::new(IpAddr::V4(ip), port)
}
18 => {
let ip_bytes: [u8; 16] = bytes[..16]
.try_into()
.expect("compact IPv6 Peer address is 16 bytes");
let port = u16::from_be_bytes([bytes[16], bytes[17]]);
SocketAddr::new(IpAddr::V6(Ipv6Addr::from(ip_bytes)), port)
}
_ => continue,
};
if addr_allowed_by_netmode(&addr, netmode) && is_valid_node_addr(&addr) {
visit(addr);
count += 1;
}
}
count
}
pub(crate) fn encode_find_node_query(
buffer: &mut BytesMut,
tid: &TransactionId,
target: &[u8; 20],
sender_id: &[u8; 20],
) {
buffer.clear();
buffer.reserve(112);
buffer.extend_from_slice(b"d1:ad2:id20:");
buffer.extend_from_slice(sender_id);
buffer.extend_from_slice(b"6:target20:");
buffer.extend_from_slice(target);
buffer.extend_from_slice(b"e1:q9:find_node1:t8:");
buffer.extend_from_slice(tid);
buffer.extend_from_slice(b"1:y1:qe");
}
pub(crate) fn encode_get_peers_query(
buffer: &mut BytesMut,
tid: &TransactionId,
info_hash: &[u8; 20],
sender_id: &[u8; 20],
) {
buffer.clear();
buffer.reserve(111);
buffer.extend_from_slice(b"d1:ad2:id20:");
buffer.extend_from_slice(sender_id);
buffer.extend_from_slice(b"9:info_hash20:");
buffer.extend_from_slice(info_hash);
buffer.extend_from_slice(b"e1:q9:get_peers1:t8:");
buffer.extend_from_slice(tid);
buffer.extend_from_slice(b"1:y1:qe");
}
pub(crate) fn encode_sample_infohashes_query(
buffer: &mut BytesMut,
tid: &TransactionId,
target: &[u8; 20],
sender_id: &[u8; 20],
) {
buffer.clear();
buffer.reserve(128);
buffer.extend_from_slice(b"d1:ad2:id20:");
buffer.extend_from_slice(sender_id);
buffer.extend_from_slice(b"6:target20:");
buffer.extend_from_slice(target);
buffer.extend_from_slice(b"e1:q17:sample_infohashes1:t8:");
buffer.extend_from_slice(tid);
buffer.extend_from_slice(b"1:y1:qe");
}
pub(crate) fn encode_response(
buffer: &mut BytesMut,
tid: &[u8],
node_id: &[u8; 20],
token: &[u8; 8],
nodes: &[NodeTuple],
ipv6: bool,
) {
buffer.clear();
buffer.reserve(384);
buffer.extend_from_slice(b"d1:rd2:id20:");
buffer.extend_from_slice(node_id);
let compact_len = if ipv6 {
nodes.iter().filter(|node| node.addr.is_ipv6()).count() * 38
} else {
nodes.iter().filter(|node| node.addr.is_ipv4()).count() * 26
};
if compact_len > 0 {
if ipv6 {
buffer.extend_from_slice(b"6:nodes6");
} else {
buffer.extend_from_slice(b"5:nodes");
}
push_usize(buffer, compact_len);
buffer.extend_from_slice(b":");
for node in nodes {
match node.addr.ip() {
IpAddr::V4(ip) if !ipv6 => {
buffer.extend_from_slice(&node.id);
buffer.extend_from_slice(&ip.octets());
buffer.extend_from_slice(&node.addr.port().to_be_bytes());
}
IpAddr::V6(ip) if ipv6 => {
buffer.extend_from_slice(&node.id);
buffer.extend_from_slice(&ip.octets());
buffer.extend_from_slice(&node.addr.port().to_be_bytes());
}
_ => {}
}
}
}
buffer.extend_from_slice(b"5:token8:");
buffer.extend_from_slice(token);
buffer.extend_from_slice(b"e1:t");
push_usize(buffer, tid.len());
buffer.extend_from_slice(b":");
buffer.extend_from_slice(tid);
buffer.extend_from_slice(b"1:y1:re");
}
fn push_usize(buffer: &mut BytesMut, mut value: usize) {
let mut digits = [0u8; 20];
let mut cursor = digits.len();
loop {
cursor -= 1;
digits[cursor] = b'0' + (value % 10) as u8;
value /= 10;
if value == 0 {
break;
}
}
buffer.extend_from_slice(&digits[cursor..]);
}
pub(crate) async fn send_find_node_query(
addr: &SocketAddr,
tid: &TransactionId,
target: &[u8; 20],
sender_id: &[u8; 20],
socket: &Arc<UdpSocket>,
buffer: &mut BytesMut,
) -> bool {
encode_find_node_query(buffer, tid, target, sender_id);
match socket.send_to(buffer, addr).await {
Ok(len) => {
#[cfg(feature = "metrics")]
{
counter!("dht_udp_bytes_sent_total").increment(len as u64);
counter!("dht_udp_packets_sent_total", "type" => "query").increment(1);
histogram!("dht_udp_query_size_bytes").record(len as f64);
}
#[cfg(not(feature = "metrics"))]
let _ = len;
true
}
Err(_) => false,
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::protocol::DhtMessage;
#[test]
fn manual_find_node_encoding_round_trips() {
let mut buffer = BytesMut::new();
let tid = [1; 8];
let target = [2; 20];
let sender = [3; 20];
encode_find_node_query(&mut buffer, &tid, &target, &sender);
let message: DhtMessage = serde_bencode::from_bytes(&buffer).unwrap();
assert_eq!(message.t.as_ref(), &tid);
assert_eq!(message.q.as_deref(), Some("find_node"));
assert_eq!(message.a.unwrap().target.unwrap().as_ref(), &target);
}
#[test]
fn manual_get_peers_encoding_round_trips() {
let mut buffer = BytesMut::new();
let tid = [1; 8];
let info_hash = [2; 20];
let sender = [3; 20];
encode_get_peers_query(&mut buffer, &tid, &info_hash, &sender);
let message: DhtMessage = serde_bencode::from_bytes(&buffer).unwrap();
assert_eq!(message.t.as_ref(), &tid);
assert_eq!(message.q.as_deref(), Some("get_peers"));
assert_eq!(message.a.unwrap().info_hash.unwrap().as_ref(), &info_hash);
}
#[test]
fn manual_sample_infohashes_encoding_round_trips() {
let mut buffer = BytesMut::new();
let tid = [1; 8];
let target = [2; 20];
let sender = [3; 20];
encode_sample_infohashes_query(&mut buffer, &tid, &target, &sender);
let message: DhtMessage = serde_bencode::from_bytes(&buffer).unwrap();
assert_eq!(message.t.as_ref(), &tid);
assert_eq!(message.q.as_deref(), Some("sample_infohashes"));
assert_eq!(message.a.unwrap().target.unwrap().as_ref(), &target);
}
#[test]
fn manual_response_encoding_round_trips() {
let mut buffer = BytesMut::new();
let nodes = [NodeTuple {
id: [4; 20],
addr: "8.8.8.8:6881".parse().unwrap(),
}];
encode_response(&mut buffer, &[1, 2], &[2; 20], &[3; 8], &nodes, false);
let message: DhtMessage = serde_bencode::from_bytes(&buffer).unwrap();
let response = message.r.unwrap();
assert_eq!(response.nodes.unwrap().len(), 26);
}
#[test]
fn compact_get_peers_values_are_validated() {
let response = DhtResponse {
id: None,
nodes: None,
nodes6: None,
values: Some(vec![
serde_bytes::ByteBuf::from(vec![8, 8, 8, 8, 0x1a, 0xe1]),
serde_bytes::ByteBuf::from(vec![10, 0, 0, 1, 0x1a, 0xe1]),
serde_bytes::ByteBuf::from(vec![1, 2, 3]),
]),
samples: None,
num: None,
interval: None,
};
let mut peers = Vec::new();
assert_eq!(
for_each_response_peer(&response, NetMode::Ipv4Only, |peer| peers.push(peer)),
1
);
assert_eq!(peers[0], "8.8.8.8:6881".parse().unwrap());
}
}
+63
View File
@@ -0,0 +1,63 @@
// 负责导出可复用 DHT 抓取 Metadata 和运行观测能力
//! High-throughput BitTorrent DHT crawler with bounded crawl and Metadata pipelines.
//!
//! [`DHTServer`] is the primary entry point. Configure it with [`DHTOptions`], register
//! callbacks, then await [`DHTServer::start`] until another task calls [`DHTServer::shutdown`].
//! Runtime counters are available through [`DHTServer::runtime_stats`] without enabling any
//! exporter. See the repository README for scheduling, backpressure and persistence details.
mod addr;
mod bootstrap;
mod budget;
mod crawl_config;
mod crawl_engine;
mod error;
mod krpc;
/// BEP-9 Metadata fetch support.
pub mod metadata;
mod node_id;
mod node_pool;
mod peer_lookup;
/// Serializable BEP-5 KRPC wire types.
pub mod protocol;
mod response_limiter;
mod routing_snapshot;
mod runtime_stats;
mod sample_infohashes;
/// Bounded, deduplicating Metadata scheduler.
pub mod scheduler;
mod server;
/// Public configuration, callback payload and network types.
pub mod types;
mod udp_buffer;
mod udp_ingress;
pub use error::{DHTError, Result};
pub use runtime_stats::{
DhtObservabilitySnapshot, DhtRuntimeSnapshot, DhtRuntimeStats, FixedHistogramSnapshot,
};
pub use scheduler::{MetadataScheduler, MetadataSchedulerCallbacks, MetadataSchedulerLimits};
pub use server::DHTServer;
pub use types::{
BootstrapOptions, CrawlOptions, DHTOptions, DiscoverySource, FileInfo, HashDiscovered,
MetadataFetchCompletion, MetadataFetchCompletionStatus, MetadataOptions, NetMode, NodeTuple,
PeerLookupOptions, PeerLookupResult, PoolOptions, RateLimitOptions, SampleInfohashesOptions,
SchedulerOptions, TargetOptions, TorrentInfo,
};
/// Common server, configuration and callback payload imports.
pub mod prelude {
pub use crate::error::{DHTError, Result};
pub use crate::runtime_stats::{DhtRuntimeSnapshot, DhtRuntimeStats};
pub use crate::scheduler::{
MetadataScheduler, MetadataSchedulerCallbacks, MetadataSchedulerLimits,
};
pub use crate::server::DHTServer;
pub use crate::types::{
BootstrapOptions, CrawlOptions, DHTOptions, DiscoverySource, FileInfo, HashDiscovered,
MetadataFetchCompletion, MetadataFetchCompletionStatus, MetadataOptions, NetMode,
NodeTuple, PeerLookupOptions, PeerLookupResult, PoolOptions, RateLimitOptions,
SampleInfohashesOptions, SchedulerOptions, TargetOptions, TorrentInfo,
};
}
+740
View File
@@ -0,0 +1,740 @@
// 负责执行 BitTorrent 握手 Metadata 下载校验解析和 Peer 失败缓存
use crate::runtime_stats::DhtRuntimeStats;
use crate::types::FileInfo;
use ahash::AHashMap;
use bytes::Bytes;
#[cfg(feature = "metrics")]
use metrics::{counter, gauge, histogram};
use rbit::peer::ExtensionMessage;
use rbit::{
ExtensionHandshake, Message, MetadataMessage, MetadataMessageType, PeerConnection, PeerId,
metadata_piece_count,
};
use sha1::{Digest, Sha1};
use std::collections::{BTreeMap, VecDeque};
use std::net::SocketAddr;
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
use tokio::time::timeout;
pub(crate) type FetchedMetadata = (String, u64, Vec<FileInfo>, u64);
pub(crate) enum MetadataFetchOutcome {
Fetched(FetchedMetadata),
Failed,
SkippedCached,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum MetadataFetchFailure {
Connect,
NoExtension,
Send,
SizeLimit,
Sha1,
Parse,
Other,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum PeerFailureReason {
Timeout,
ConnectFailed,
}
struct ConnectRateLimiter {
interval: Duration,
next_start: Mutex<Instant>,
}
impl ConnectRateLimiter {
fn per_second(rate: u32) -> Self {
Self {
interval: Duration::from_secs_f64(1.0 / f64::from(rate.max(1))),
next_start: Mutex::new(Instant::now()),
}
}
async fn acquire(&self) {
let delay = {
let mut next_start = self
.next_start
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
let now = Instant::now();
let reserved = (*next_start).max(now);
*next_start = reserved + self.interval;
reserved.saturating_duration_since(now)
};
if !delay.is_zero() {
tokio::time::sleep(delay).await;
}
}
}
#[cfg(feature = "metrics")]
impl PeerFailureReason {
fn as_str(self) -> &'static str {
match self {
Self::Timeout => "timeout",
Self::ConnectFailed => "connect_failed",
}
}
}
#[derive(Debug, Clone, Copy)]
struct PeerFailureEntry {
expires_at: Instant,
reason: PeerFailureReason,
}
#[derive(Default)]
struct PeerFailureCacheInner {
entries: AHashMap<SocketAddr, PeerFailureEntry>,
expiry: VecDeque<(Instant, SocketAddr)>,
}
struct PeerFailureCache {
inner: Mutex<PeerFailureCacheInner>,
capacity: usize,
ttl: Duration,
}
impl PeerFailureCache {
fn new(capacity: usize, ttl: Duration) -> Self {
Self {
inner: Mutex::new(PeerFailureCacheInner {
entries: AHashMap::with_capacity(capacity.min(16_384)),
expiry: VecDeque::with_capacity(capacity.min(16_384)),
}),
capacity,
ttl,
}
}
fn get(&self, addr: SocketAddr, now: Instant) -> (Option<PeerFailureReason>, usize) {
if self.capacity == 0 || self.ttl.is_zero() {
return (None, 0);
}
let mut inner = self
.inner
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
Self::expire(&mut inner, now);
(
inner.entries.get(&addr).map(|entry| entry.reason),
inner.entries.len(),
)
}
fn insert(&self, addr: SocketAddr, reason: PeerFailureReason, now: Instant) -> usize {
if self.capacity == 0 || self.ttl.is_zero() {
return 0;
}
let mut inner = self
.inner
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
Self::expire(&mut inner, now);
while inner.entries.len() >= self.capacity && !inner.entries.contains_key(&addr) {
let Some((expires_at, oldest_addr)) = inner.expiry.pop_front() else {
break;
};
if inner
.entries
.get(&oldest_addr)
.is_some_and(|entry| entry.expires_at == expires_at)
{
inner.entries.remove(&oldest_addr);
}
}
let expires_at = now + self.ttl;
inner
.entries
.insert(addr, PeerFailureEntry { expires_at, reason });
inner.expiry.push_back((expires_at, addr));
inner.entries.len()
}
fn remove(&self, addr: &SocketAddr, now: Instant) -> usize {
if self.capacity == 0 || self.ttl.is_zero() {
return 0;
}
let mut inner = self
.inner
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
Self::expire(&mut inner, now);
inner.entries.remove(addr);
inner.entries.len()
}
fn expire(inner: &mut PeerFailureCacheInner, now: Instant) {
while let Some((expires_at, addr)) = inner.expiry.front().copied() {
if expires_at > now {
break;
}
inner.expiry.pop_front();
if inner
.entries
.get(&addr)
.is_some_and(|entry| entry.expires_at == expires_at)
{
inner.entries.remove(&addr);
}
}
}
}
#[derive(Clone)]
/// BEP-9 Metadata fetcher with an end-to-end timeout and shared Peer failure cache.
pub struct RbitFetcher {
total_timeout: Duration,
max_metadata_size_bytes: usize,
runtime_stats: DhtRuntimeStats,
peer_failure_cache: Arc<PeerFailureCache>,
connect_rate_limiter: Arc<ConnectRateLimiter>,
}
impl RbitFetcher {
/// Creates a standalone fetcher with the default failure-cache capacity and TTL.
///
/// [`DHTServer`](crate::DHTServer) normally constructs this component from
/// [`MetadataOptions`](crate::MetadataOptions).
pub fn new(timeout_secs: u64) -> Self {
Self::new_with_runtime_stats(
timeout_secs,
32,
10 * 1024 * 1024,
200_000,
60,
DhtRuntimeStats::default(),
)
}
pub(crate) fn new_with_runtime_stats(
timeout_secs: u64,
max_connects_per_second: u32,
max_metadata_size_bytes: usize,
peer_failure_cache_capacity: usize,
peer_failure_ttl_secs: u64,
runtime_stats: DhtRuntimeStats,
) -> Self {
Self {
total_timeout: Duration::from_secs(if timeout_secs == 0 { 15 } else { timeout_secs }),
max_metadata_size_bytes: max_metadata_size_bytes.max(1),
runtime_stats,
peer_failure_cache: Arc::new(PeerFailureCache::new(
peer_failure_cache_capacity,
Duration::from_secs(peer_failure_ttl_secs),
)),
connect_rate_limiter: Arc::new(ConnectRateLimiter::per_second(max_connects_per_second)),
}
}
/// Fetch metadata from one peer under a single end-to-end deadline.
///
/// The deadline covers TCP connect, both BitTorrent handshakes, all metadata
/// piece I/O, hash validation and bencode parsing. Inner library timeouts can
/// therefore never stack on top of the configured metadata timeout.
#[cfg(test)]
pub(crate) async fn fetch(
&self,
info_hash: &[u8; 20],
peer_addr: SocketAddr,
) -> MetadataFetchOutcome {
self.fetch_with_attempt_observer(info_hash, peer_addr, || {})
.await
}
pub(crate) async fn fetch_with_attempt_observer<F>(
&self,
info_hash: &[u8; 20],
peer_addr: SocketAddr,
on_attempt: F,
) -> MetadataFetchOutcome
where
F: FnOnce() + Send,
{
let (cached_reason, cache_entries) = self.peer_failure_cache.get(peer_addr, Instant::now());
self.set_peer_failure_cache_entries(cache_entries);
if let Some(reason) = cached_reason {
self.runtime_stats.metadata_peer_failure_cache_hit();
match reason {
PeerFailureReason::Timeout => self.runtime_stats.peer_cache_hit_timeout(),
PeerFailureReason::ConnectFailed => self.runtime_stats.peer_cache_hit_connect(),
}
#[cfg(feature = "metrics")]
counter!("dht_metadata_peer_failure_cache_hits_total", "reason" => reason.as_str())
.increment(1);
#[cfg(not(feature = "metrics"))]
let _ = reason;
return MetadataFetchOutcome::SkippedCached;
}
self.connect_rate_limiter.acquire().await;
on_attempt();
self.runtime_stats.metadata_peer_attempt();
#[cfg(feature = "metrics")]
{
counter!("dht_metadata_fetch_attempts_total").increment(1);
counter!("dht_metadata_peer_attempts_total").increment(1);
}
let started = Instant::now();
let result = timeout(
self.total_timeout,
self.fetch_with_peer(info_hash, peer_addr),
)
.await;
#[cfg(feature = "metrics")]
histogram!("dht_metadata_fetch_duration_seconds").record(started.elapsed().as_secs_f64());
self.runtime_stats.observe_metadata_fetch_duration(
started.elapsed().as_millis().min(u128::from(u64::MAX)) as u64,
);
match result {
Ok(Ok(metadata)) => {
let cache_entries = self.peer_failure_cache.remove(&peer_addr, Instant::now());
self.set_peer_failure_cache_entries(cache_entries);
self.runtime_stats.metadata_peer_succeeded();
#[cfg(feature = "metrics")]
{
counter!("dht_metadata_fetch_success_total").increment(1);
counter!("dht_metadata_fetch_result_total", "result" => "success").increment(1);
}
MetadataFetchOutcome::Fetched(metadata)
}
Ok(Err(reason)) => {
self.runtime_stats.metadata_peer_failed();
match reason {
MetadataFetchFailure::Connect => self.runtime_stats.metadata_failure_connect(),
MetadataFetchFailure::NoExtension => {
self.runtime_stats.metadata_failure_no_extension()
}
MetadataFetchFailure::Send => self.runtime_stats.metadata_failure_send(),
MetadataFetchFailure::SizeLimit => {
self.runtime_stats.metadata_failure_size_limit()
}
MetadataFetchFailure::Sha1 => self.runtime_stats.metadata_failure_sha1(),
MetadataFetchFailure::Parse => self.runtime_stats.metadata_failure_parse(),
MetadataFetchFailure::Other => self.runtime_stats.metadata_failure_other(),
}
#[cfg(feature = "metrics")]
counter!("dht_metadata_fetch_result_total", "result" => "failed").increment(1);
MetadataFetchOutcome::Failed
}
Err(_) => {
self.record_peer_failure(peer_addr, PeerFailureReason::Timeout);
self.runtime_stats.metadata_peer_failed();
self.runtime_stats.metadata_peer_timeout();
self.runtime_stats.metadata_failure_timeout();
#[cfg(feature = "metrics")]
{
counter!("dht_metadata_fetch_fail_total", "reason" => "timeout").increment(1);
counter!("dht_metadata_fetch_result_total", "result" => "timeout").increment(1);
}
MetadataFetchOutcome::Failed
}
}
}
pub(crate) async fn verify_handshake(
&self,
info_hash: &[u8; 20],
peer_addr: SocketAddr,
) -> bool {
let (cached_reason, _) = self.peer_failure_cache.get(peer_addr, Instant::now());
if cached_reason.is_some() {
return false;
}
self.connect_rate_limiter.acquire().await;
let peer_id = PeerId::generate();
match timeout(
self.total_timeout,
PeerConnection::connect(peer_addr, *info_hash, *peer_id.as_bytes()),
)
.await
{
Ok(Ok(_)) => {
self.peer_failure_cache.remove(&peer_addr, Instant::now());
true
}
Ok(Err(_)) => {
self.record_peer_failure(peer_addr, PeerFailureReason::ConnectFailed);
false
}
Err(_) => {
self.record_peer_failure(peer_addr, PeerFailureReason::Timeout);
false
}
}
}
fn record_peer_failure(&self, peer_addr: SocketAddr, reason: PeerFailureReason) {
let cache_entries = self
.peer_failure_cache
.insert(peer_addr, reason, Instant::now());
self.set_peer_failure_cache_entries(cache_entries);
#[cfg(feature = "metrics")]
counter!("dht_metadata_peer_failure_cache_inserts_total", "reason" => reason.as_str())
.increment(1);
}
fn set_peer_failure_cache_entries(&self, count: usize) {
self.runtime_stats
.set_metadata_peer_failure_cache_entries(count);
#[cfg(feature = "metrics")]
gauge!("dht_metadata_peer_failure_cache_entries").set(count as f64);
}
async fn fetch_with_peer(
&self,
info_hash: &[u8; 20],
peer_addr: SocketAddr,
) -> Result<FetchedMetadata, MetadataFetchFailure> {
let peer_id = PeerId::generate();
let mut conn = match PeerConnection::connect(peer_addr, *info_hash, *peer_id.as_bytes())
.await
{
Ok(conn) => {
#[cfg(feature = "metrics")]
counter!("dht_metadata_connection_result_total", "result" => "success")
.increment(1);
conn
}
Err(_) => {
self.record_peer_failure(peer_addr, PeerFailureReason::ConnectFailed);
self.runtime_stats.metadata_connect_failed();
#[cfg(feature = "metrics")]
counter!("dht_metadata_connection_result_total", "result" => "failed").increment(1);
return Err(MetadataFetchFailure::Connect);
}
};
if !conn.supports_extension {
self.runtime_stats.metadata_no_extension();
#[cfg(feature = "metrics")]
counter!("dht_metadata_handshake_result_total", "result" => "no_extension_support")
.increment(1);
return Err(MetadataFetchFailure::NoExtension);
}
let my_ut_metadata_id = 1;
let handshake = ExtensionHandshake::with_extensions(&[("ut_metadata", my_ut_metadata_id)]);
let handshake_bytes = handshake.encode().map_err(|_| MetadataFetchFailure::Send)?;
if conn
.send(Message::Extended {
id: 0,
payload: handshake_bytes,
})
.await
.is_err()
{
#[cfg(feature = "metrics")]
counter!("dht_metadata_fetch_fail_total", "reason" => "send_error").increment(1);
return Err(MetadataFetchFailure::Send);
}
let mut metadata_size = 0;
let mut remote_ut_metadata_id = 0;
let mut pieces: BTreeMap<u32, Bytes> = BTreeMap::new();
let mut total_received = 0usize;
let mut request_sent = false;
let info_bytes = loop {
let msg = conn
.receive()
.await
.map_err(|_| MetadataFetchFailure::Other)?;
let Message::Extended { id, payload } = msg else {
continue;
};
if id == 0 {
if let Ok(ExtensionMessage::Handshake(remote_hs)) =
ExtensionMessage::decode(id, &payload)
{
if let Some(size) = remote_hs.metadata_size {
metadata_size = size as u32;
}
if let Some(ext_id) = remote_hs.get_extension_id("ut_metadata") {
remote_ut_metadata_id = ext_id;
}
}
if metadata_size > 0 && remote_ut_metadata_id > 0 && !request_sent {
if metadata_size as usize > self.max_metadata_size_bytes {
#[cfg(feature = "metrics")]
counter!("dht_metadata_fetch_fail_total", "reason" => "size_limit")
.increment(1);
return Err(MetadataFetchFailure::SizeLimit);
}
let count = metadata_piece_count(metadata_size as usize);
for piece in 0..count {
let encoded = MetadataMessage::request(piece as u32)
.encode()
.map_err(|_| MetadataFetchFailure::Send)?;
if conn
.send(Message::Extended {
id: remote_ut_metadata_id,
payload: encoded,
})
.await
.is_err()
{
#[cfg(feature = "metrics")]
counter!("dht_metadata_fetch_fail_total", "reason" => "send_error")
.increment(1);
return Err(MetadataFetchFailure::Send);
}
}
request_sent = true;
}
continue;
}
if id != my_ut_metadata_id {
continue;
}
let Ok(meta_msg) = MetadataMessage::decode(&payload) else {
continue;
};
if meta_msg.msg_type != MetadataMessageType::Data {
continue;
}
let Some(data) = meta_msg.data else {
continue;
};
#[cfg(feature = "metrics")]
counter!("dht_metadata_bytes_downloaded_total").increment(data.len() as u64);
self.runtime_stats.metadata_bytes_downloaded(data.len());
let data_len = data.len();
if let Some(previous) = pieces.insert(meta_msg.piece, data) {
total_received = total_received.saturating_sub(previous.len());
}
total_received = total_received.saturating_add(data_len);
if metadata_size == 0 || total_received < metadata_size as usize {
continue;
}
let count = metadata_piece_count(metadata_size as usize);
let mut full_data = Vec::with_capacity(metadata_size as usize);
for piece in 0..count {
let data = pieces
.get(&(piece as u32))
.ok_or(MetadataFetchFailure::Other)?;
full_data.extend_from_slice(data);
}
let info_hash_copy = *info_hash;
let validated = tokio::task::spawn_blocking(move || {
let mut hasher = Sha1::new();
hasher.update(&full_data);
let digest: [u8; 20] = hasher.finalize().into();
(digest == info_hash_copy).then_some(full_data)
})
.await
.ok()
.flatten();
match validated {
Some(data) => {
#[cfg(feature = "metrics")]
counter!("dht_metadata_handshake_result_total", "result" => "success")
.increment(1);
break data;
}
None => {
#[cfg(feature = "metrics")]
counter!("dht_metadata_fetch_fail_total", "reason" => "sha1_mismatch")
.increment(1);
return Err(MetadataFetchFailure::Sha1);
}
}
};
self.runtime_stats.observe_metadata_size(info_bytes.len());
match parse_metadata(&info_bytes) {
Some(metadata) => {
#[cfg(feature = "metrics")]
histogram!("dht_metadata_size_bytes").record(info_bytes.len() as f64);
Ok(metadata)
}
None => {
#[cfg(feature = "metrics")]
counter!("dht_metadata_fetch_fail_total", "reason" => "parse_error").increment(1);
Err(MetadataFetchFailure::Parse)
}
}
}
}
fn parse_metadata(info_bytes: &[u8]) -> Option<FetchedMetadata> {
let value = rbit::decode(info_bytes).ok()?;
let dict = value.as_dict()?;
let name = dict
.get(&b"name"[..])
.and_then(|value| value.as_str())
.unwrap_or("Unknown")
.to_string();
let piece_length = dict
.get(&b"piece length"[..])
.and_then(|value| value.as_integer())
.and_then(|value| u64::try_from(value).ok())
.unwrap_or(0);
let mut total_size = 0_u64;
let mut file_list = Vec::new();
if let Some(files) = dict.get(&b"files"[..]).and_then(|value| value.as_list()) {
for file in files {
let Some(file_dict) = file.as_dict() else {
continue;
};
let Some(length) = file_dict
.get(&b"length"[..])
.and_then(|value| value.as_integer())
else {
continue;
};
let length = u64::try_from(length).ok()?;
total_size = total_size.checked_add(length)?;
let parts = file_dict
.get(&b"path"[..])
.and_then(|value| value.as_list())?;
let path = parts
.iter()
.map(|part| part.as_str())
.collect::<Option<Vec<_>>>()?
.join("/");
file_list.push(FileInfo { path, size: length });
}
} else if let Some(length) = dict
.get(&b"length"[..])
.and_then(|value| value.as_integer())
{
total_size = u64::try_from(length).ok()?;
file_list.push(FileInfo {
path: name.clone(),
size: total_size,
});
}
(total_size > 0).then_some((name, total_size, file_list, piece_length))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn parser_rejects_negative_file_size() {
assert!(parse_metadata(b"d6:lengthi-1e4:name1:ae").is_none());
}
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpListener;
#[tokio::test]
async fn connection_rate_limiter_spaces_attempts() {
let limiter = ConnectRateLimiter::per_second(20);
limiter.acquire().await;
let started = Instant::now();
limiter.acquire().await;
assert!(started.elapsed() >= Duration::from_millis(40));
}
#[test]
fn peer_failure_cache_is_socket_specific_and_expires() {
let start = Instant::now();
let cache = PeerFailureCache::new(10, Duration::from_secs(60));
let first: SocketAddr = "127.0.0.1:1000".parse().unwrap();
let same_ip_other_port: SocketAddr = "127.0.0.1:1001".parse().unwrap();
assert_eq!(cache.insert(first, PeerFailureReason::Timeout, start), 1);
assert_eq!(cache.get(first, start).0, Some(PeerFailureReason::Timeout));
assert_eq!(cache.get(same_ip_other_port, start).0, None);
assert_eq!(cache.get(first, start + Duration::from_secs(61)), (None, 0));
}
#[test]
fn peer_failure_cache_evicts_oldest_at_capacity() {
let start = Instant::now();
let cache = PeerFailureCache::new(1, Duration::from_secs(60));
let first: SocketAddr = "127.0.0.1:1000".parse().unwrap();
let second: SocketAddr = "127.0.0.1:1001".parse().unwrap();
cache.insert(first, PeerFailureReason::Timeout, start);
cache.insert(second, PeerFailureReason::ConnectFailed, start);
assert_eq!(cache.get(first, start).0, None);
assert_eq!(
cache.get(second, start).0,
Some(PeerFailureReason::ConnectFailed)
);
}
#[tokio::test]
async fn total_timeout_covers_peer_handshake() {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let accept_task = tokio::spawn(async move {
let (_stream, _) = listener.accept().await.unwrap();
std::future::pending::<()>().await;
});
let stats = DhtRuntimeStats::default();
let fetcher =
RbitFetcher::new_with_runtime_stats(1, 10, 10 * 1024 * 1024, 10, 60, stats.clone());
let started = Instant::now();
assert!(matches!(
fetcher.fetch(&[7; 20], addr).await,
MetadataFetchOutcome::Failed
));
assert!(started.elapsed() < Duration::from_secs(2));
let cached_started = Instant::now();
assert!(matches!(
fetcher.fetch(&[8; 20], addr).await,
MetadataFetchOutcome::SkippedCached
));
assert!(cached_started.elapsed() < Duration::from_millis(100));
let snapshot = stats.snapshot();
assert_eq!(snapshot.metadata_peer_attempts, 1);
assert_eq!(snapshot.metadata_peer_failed, 1);
assert_eq!(snapshot.metadata_peer_timeouts, 1);
assert_eq!(snapshot.metadata_peer_failure_cache_hits, 1);
assert_eq!(snapshot.metadata_peer_failure_cache_entries, 1);
accept_task.abort();
}
#[tokio::test]
async fn handshake_verifier_accepts_matching_bittorrent_peer() {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let peer = tokio::spawn(async move {
let (mut stream, _) = listener.accept().await.unwrap();
let mut handshake = [0_u8; 68];
stream.read_exact(&mut handshake).await.unwrap();
stream.write_all(&handshake).await.unwrap();
});
let fetcher = RbitFetcher::new_with_runtime_stats(
1,
10,
10 * 1024 * 1024,
10,
60,
DhtRuntimeStats::default(),
);
assert!(fetcher.verify_handshake(&[7; 20], addr).await);
peer.await.unwrap();
}
}
+84
View File
@@ -0,0 +1,84 @@
// 负责生成 DHT 节点标识事务标识和距离目标
use rand::RngExt;
pub(crate) type TransactionId = [u8; 8];
pub(crate) fn transaction_id_from_bytes(bytes: &[u8]) -> Option<TransactionId> {
if bytes.len() != 8 {
return None;
}
let mut tid = [0u8; 8];
tid.copy_from_slice(bytes);
Some(tid)
}
pub(crate) fn random_node_id() -> [u8; 20] {
let mut id = [0u8; 20];
rand::rng().fill(&mut id);
id
}
pub(crate) fn neighbor_node_id(remote_id: &[u8], local_id: &[u8]) -> Vec<u8> {
let mut id = Vec::with_capacity(20);
let prefix_len = remote_id.len().min(6);
id.extend_from_slice(&remote_id[..prefix_len]);
if local_id.len() > prefix_len {
id.extend_from_slice(&local_id[prefix_len..]);
}
while id.len() < 20 {
id.push(rand::random());
}
id.truncate(20);
id
}
pub(crate) fn bucket_index(id: &[u8], local_id: &[u8; 20]) -> usize {
for bit in 0..160 {
let byte = bit / 8;
if byte >= id.len() {
break;
}
let mask = 1 << (7 - (bit % 8));
if (id[byte] ^ local_id[byte]) & mask != 0 {
return bit;
}
}
159
}
pub(crate) fn target_for_bucket(local_id: &[u8; 20], bucket: usize) -> [u8; 20] {
let mut id = *local_id;
let bucket = bucket.min(159);
let byte = bucket / 8;
let bit = 7 - (bucket % 8);
id[byte] ^= 1 << bit;
for item in id.iter_mut().skip(byte + 1) {
*item = rand::random();
}
id
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn transaction_ids_are_eight_bytes() {
let first = 1u64.to_be_bytes();
assert_eq!(transaction_id_from_bytes(&first), Some(first));
assert!(transaction_id_from_bytes(&[1, 2]).is_none());
}
#[test]
fn neighbor_id_keeps_remote_prefix_and_local_suffix() {
let remote = [1u8; 20];
let local = [2u8; 20];
let id = neighbor_node_id(&remote, &local);
assert_eq!(&id[..6], &[1u8; 6]);
assert_eq!(&id[6..], &[2u8; 14]);
}
}
+392
View File
@@ -0,0 +1,392 @@
// 负责维护有界 DHT 节点池去重淘汰和重访状态
use crate::addr::is_valid_node_addr;
use crate::budget::RateBucket;
use crate::types::NodeTuple;
use ahash::{AHashMap, AHashSet};
use std::collections::VecDeque;
use std::net::{IpAddr, SocketAddr};
use std::time::{Duration, Instant};
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum AdmissionOutcome {
Admitted,
Replaced,
Duplicate,
RateLimited,
Invalid,
}
#[derive(Debug, Clone, Copy)]
struct QueuedNode {
node: NodeTuple,
#[cfg_attr(not(feature = "metrics"), allow(dead_code))]
queued_at: Instant,
}
pub(crate) struct NodePool {
queue: VecDeque<QueuedNode>,
queued: AHashSet<SocketAddr>,
recent: AHashMap<SocketAddr, Instant>,
recent_expiry: VecDeque<(Instant, SocketAddr)>,
replacement_budget: RateBucket,
recent_ttl: Duration,
capacity: usize,
warmed: bool,
}
impl NodePool {
pub(crate) fn new(
capacity: usize,
replacements_per_minute: u32,
recent_ttl: Duration,
now: Instant,
) -> Self {
let capacity = capacity.max(1);
let replacement_burst = replacements_per_minute.div_ceil(60).max(1);
Self {
queue: VecDeque::with_capacity(capacity),
queued: AHashSet::with_capacity(capacity),
recent: AHashMap::with_capacity(capacity),
recent_expiry: VecDeque::with_capacity(capacity),
replacement_budget: RateBucket::per_minute(
replacements_per_minute,
replacement_burst,
true,
now,
),
recent_ttl,
capacity,
warmed: false,
}
}
pub(crate) fn admit(&mut self, node: NodeTuple, now: Instant) -> AdmissionOutcome {
if !is_valid_node_addr(&node.addr) {
return AdmissionOutcome::Invalid;
}
self.expire_recent(now);
if self.queued.contains(&node.addr) || self.recent.contains_key(&node.addr) {
return AdmissionOutcome::Duplicate;
}
if self.warmed && !self.replacement_budget.try_take_one(now) {
return AdmissionOutcome::RateLimited;
}
let replaced = if self.queue.len() >= self.capacity {
self.pop_front_internal().is_some()
} else {
false
};
self.queued.insert(node.addr);
self.queue.push_back(QueuedNode {
node,
queued_at: now,
});
if self.queue.len() >= self.capacity {
self.warmed = true;
}
if replaced {
AdmissionOutcome::Replaced
} else {
AdmissionOutcome::Admitted
}
}
pub(crate) fn front(&self) -> Option<NodeTuple> {
self.queue.front().map(|entry| entry.node)
}
/// Move the FIFO head behind the remaining queued nodes without marking
/// it as probed. The address stays in `queued` and is not added to `recent`.
pub(crate) fn rotate_front_to_back(&mut self) -> bool {
if self.queue.len() <= 1 {
return false;
}
self.queue.rotate_left(1);
true
}
pub(crate) fn take_front_for_probe(&mut self, now: Instant) -> Option<NodeTuple> {
let entry = self.pop_front_internal()?;
let expires_at = now + self.recent_ttl;
self.recent.insert(entry.node.addr, expires_at);
self.recent_expiry.push_back((expires_at, entry.node.addr));
Some(entry.node)
}
pub(crate) fn restore_front(&mut self, node: NodeTuple, queued_at: Instant) {
self.recent.remove(&node.addr);
self.queued.insert(node.addr);
self.queue.push_front(QueuedNode { node, queued_at });
}
pub(crate) fn contains_recent(&mut self, addr: &SocketAddr, now: Instant) -> bool {
self.expire_recent(now);
self.recent.contains_key(addr)
}
pub(crate) fn record_probe(&mut self, addr: SocketAddr, now: Instant) {
let expires_at = now + self.recent_ttl;
self.recent.insert(addr, expires_at);
self.recent_expiry.push_back((expires_at, addr));
}
pub(crate) fn len(&self) -> usize {
self.queue.len()
}
#[cfg_attr(not(feature = "metrics"), allow(dead_code))]
pub(crate) fn oldest_age(&self, now: Instant) -> Duration {
self.queue
.front()
.and_then(|entry| now.checked_duration_since(entry.queued_at))
.unwrap_or_default()
}
#[cfg(test)]
fn is_warmed(&self) -> bool {
self.warmed
}
fn pop_front_internal(&mut self) -> Option<QueuedNode> {
let entry = self.queue.pop_front()?;
self.queued.remove(&entry.node.addr);
Some(entry)
}
fn expire_recent(&mut self, now: Instant) {
while let Some((expires_at, addr)) = self.recent_expiry.front().copied() {
if expires_at > now {
break;
}
self.recent_expiry.pop_front();
if self.recent.get(&addr).copied() == Some(expires_at) {
self.recent.remove(&addr);
}
}
}
}
#[derive(Debug, Clone, Copy)]
struct ResponsiveEntry {
node: NodeTuple,
expires_at: Instant,
}
/// Fixed-size responsive-node ring. The crawl actor is the only writer.
pub(crate) struct ResponsiveReservoir {
slots: Vec<Option<ResponsiveEntry>>,
index: AHashMap<SocketAddr, usize>,
write_cursor: usize,
revisit_cursor: usize,
ttl: Duration,
}
impl ResponsiveReservoir {
pub(crate) fn new(capacity: usize, ttl: Duration) -> Self {
let capacity = capacity.max(1);
Self {
slots: vec![None; capacity],
index: AHashMap::with_capacity(capacity),
write_cursor: 0,
revisit_cursor: 0,
ttl,
}
}
pub(crate) fn record(&mut self, node: NodeTuple, now: Instant) {
let entry = ResponsiveEntry {
node,
expires_at: now + self.ttl,
};
if let Some(slot) = self.index.get(&node.addr).copied() {
self.slots[slot] = Some(entry);
return;
}
let slot = self.write_cursor;
if let Some(old) = self.slots[slot]
&& self.index.get(&old.node.addr).copied() == Some(slot)
{
self.index.remove(&old.node.addr);
}
self.slots[slot] = Some(entry);
self.index.insert(node.addr, slot);
self.write_cursor = (self.write_cursor + 1) % self.slots.len();
}
pub(crate) fn next_revisit(&mut self, now: Instant) -> Option<NodeTuple> {
for _ in 0..self.slots.len() {
let slot = self.revisit_cursor;
self.revisit_cursor = (self.revisit_cursor + 1) % self.slots.len();
let Some(entry) = self.slots[slot] else {
continue;
};
if entry.expires_at <= now {
if self.index.get(&entry.node.addr).copied() == Some(slot) {
self.index.remove(&entry.node.addr);
}
self.slots[slot] = None;
continue;
}
return Some(entry.node);
}
None
}
pub(crate) fn snapshot(&self, limit: usize, now: Instant) -> Vec<NodeTuple> {
self.slots
.iter()
.filter_map(|entry| {
entry
.filter(|entry| entry.expires_at > now)
.map(|entry| entry.node)
})
.take(limit)
.collect()
}
}
#[derive(Debug, Clone, Copy, Hash, PartialEq, Eq)]
pub(crate) enum SubnetKey {
V4([u8; 3]),
V6([u8; 8]),
}
impl SubnetKey {
pub(crate) fn from_addr(addr: &SocketAddr) -> Self {
match addr.ip() {
IpAddr::V4(ip) => {
let octets = ip.octets();
Self::V4([octets[0], octets[1], octets[2]])
}
IpAddr::V6(ip) => {
let octets = ip.octets();
Self::V6(octets[..8].try_into().expect("IPv6 prefix has eight bytes"))
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::net::{IpAddr, Ipv4Addr};
fn node(id: u8, addr: &str) -> NodeTuple {
NodeTuple {
id: [id; 20],
addr: addr.parse().unwrap(),
}
}
#[test]
fn strict_fifo_replaces_oldest_after_warmup() {
let start = Instant::now();
let mut pool = NodePool::new(2, 600, Duration::from_secs(60), start);
let first = node(1, "8.8.8.8:1");
let second = node(2, "1.1.1.1:2");
let third = node(3, "9.9.9.9:3");
assert_eq!(pool.admit(first, start), AdmissionOutcome::Admitted);
assert_eq!(pool.admit(second, start), AdmissionOutcome::Admitted);
assert!(pool.is_warmed());
assert_eq!(pool.admit(third, start), AdmissionOutcome::Replaced);
assert_eq!(pool.front(), Some(second));
}
#[test]
fn duplicate_does_not_reorder_fifo() {
let start = Instant::now();
let mut pool = NodePool::new(3, 600, Duration::from_secs(60), start);
let first = node(1, "8.8.8.8:1");
let second = node(2, "1.1.1.1:2");
pool.admit(first, start);
pool.admit(second, start);
assert_eq!(pool.admit(first, start), AdmissionOutcome::Duplicate);
assert_eq!(pool.front(), Some(first));
}
#[test]
fn rotating_front_preserves_queued_dedup_and_recent_state() {
let start = Instant::now();
let mut pool = NodePool::new(3, 600, Duration::from_secs(60), start);
let first = node(1, "8.8.8.8:1");
let second = node(2, "1.1.1.1:2");
assert_eq!(pool.admit(first, start), AdmissionOutcome::Admitted);
assert_eq!(pool.admit(second, start), AdmissionOutcome::Admitted);
assert!(pool.rotate_front_to_back());
assert_eq!(pool.front(), Some(second));
assert_eq!(pool.admit(first, start), AdmissionOutcome::Duplicate);
assert!(!pool.contains_recent(&first.addr, start));
}
#[test]
fn warmed_pool_enforces_replacement_rate() {
let start = Instant::now();
let mut pool = NodePool::new(2, 60, Duration::from_secs(60), start);
pool.admit(node(1, "8.8.8.8:1"), start);
pool.admit(node(2, "1.1.1.1:2"), start);
assert_eq!(
pool.admit(node(3, "9.9.9.9:3"), start),
AdmissionOutcome::Replaced
);
assert_eq!(
pool.admit(node(4, "208.67.222.222:4"), start),
AdmissionOutcome::RateLimited
);
assert_eq!(
pool.admit(node(4, "208.67.222.222:4"), start + Duration::from_secs(1)),
AdmissionOutcome::Replaced
);
}
#[test]
fn recent_probe_blocks_readmission_until_expiry() {
let start = Instant::now();
let mut pool = NodePool::new(3, 600, Duration::from_secs(10), start);
let first = node(1, "8.8.8.8:1");
pool.admit(first, start);
assert_eq!(pool.take_front_for_probe(start), Some(first));
assert_eq!(pool.admit(first, start), AdmissionOutcome::Duplicate);
assert_eq!(
pool.admit(first, start + Duration::from_secs(11)),
AdmissionOutcome::Admitted
);
}
#[test]
fn responsive_ring_overwrites_without_growing() {
let start = Instant::now();
let mut reservoir = ResponsiveReservoir::new(2, Duration::from_secs(10));
reservoir.record(node(1, "8.8.8.8:1"), start);
reservoir.record(node(2, "1.1.1.1:2"), start);
reservoir.record(node(3, "9.9.9.9:3"), start);
let snapshot = reservoir.snapshot(10, start);
assert_eq!(snapshot.len(), 2);
assert!(!snapshot.iter().any(|entry| entry.id == [1; 20]));
}
#[test]
#[ignore = "release-only FIFO throughput smoke test"]
fn million_fifo_operations() {
let start = Instant::now();
let mut pool = NodePool::new(100_000, u32::MAX, Duration::from_secs(600), start);
for value in 0..1_000_000u32 {
let octets = value.to_be_bytes();
let node = NodeTuple {
id: [octets[3]; 20],
addr: SocketAddr::new(
IpAddr::V4(Ipv4Addr::new(11, octets[1], octets[2], octets[3])),
(value % 65_534 + 1) as u16,
),
};
let outcome = pool.admit(node, start);
assert!(!matches!(outcome, AdmissionOutcome::RateLimited));
}
assert_eq!(pool.len(), 100_000);
eprintln!("1,000,000 FIFO admissions in {:?}", start.elapsed());
}
}
+911
View File
@@ -0,0 +1,911 @@
// 负责执行有界主动 Peer 查找并向采集和调用方返回结果
use crate::budget::{RateBucket, SharedRateBudget};
use crate::krpc::{encode_get_peers_query, for_each_response_node, for_each_response_peer};
use crate::node_id::TransactionId;
use crate::protocol::DhtResponse;
use crate::routing_snapshot::{RoutingSnapshot, xor_distance_cmp};
use crate::runtime_stats::DhtRuntimeStats;
use crate::types::{
DiscoverySource, HashDiscovered, NetMode, NodeTuple, PeerLookupOptions, PeerLookupResult,
};
use ahash::{AHashMap, AHashSet};
use arc_swap::ArcSwap;
use bytes::BytesMut;
#[cfg(feature = "metrics")]
use metrics::counter;
use std::collections::VecDeque;
use std::net::SocketAddr;
use std::sync::Arc;
use std::time::{Duration, Instant};
use tokio::net::UdpSocket;
use tokio::sync::{mpsc, oneshot};
use tokio_util::sync::CancellationToken;
const LOOKUP_TID_TAG: u8 = 0xa5;
const MAX_QUERIES_PER_LOOKUP: usize = 12;
const MAX_CONCURRENT_QUERIES_PER_LOOKUP: usize = 4;
const MAX_FRONTIER_NODES: usize = 64;
const MAX_PEERS_PER_LOOKUP: usize = 12;
const LOOKUP_TIMEOUT: Duration = Duration::from_secs(2);
const QUERY_TIMEOUT: Duration = Duration::from_millis(500);
const MAINTENANCE_INTERVAL: Duration = Duration::from_millis(25);
const REQUEST_CHANNEL_CAPACITY: usize = 16_384;
const RESPONSE_CHANNEL_CAPACITY: usize = 4_096;
#[derive(Debug, Clone, Copy, Hash, PartialEq, Eq)]
struct PendingKey {
addr: SocketAddr,
tid: TransactionId,
}
#[derive(Debug, Clone, Copy)]
struct PendingQuery {
lookup_id: u64,
deadline: Instant,
preferred_phase: bool,
}
struct LookupState {
info_hash: [u8; 20],
info_hash_hex: String,
frontier: Vec<NodeTuple>,
preferred: Option<NodeTuple>,
seen_nodes: AHashSet<SocketAddr>,
peers: AHashSet<SocketAddr>,
queried: usize,
outstanding: usize,
deadline: Instant,
preferred_phase: bool,
allow_iterative_fallback: bool,
source: DiscoverySource,
completion: Option<oneshot::Sender<PeerLookupResult>>,
}
impl LookupState {
fn pop_closest(&mut self) -> Option<NodeTuple> {
if let Some(preferred) = self.preferred.take() {
return Some(preferred);
}
let index = self
.frontier
.iter()
.enumerate()
.min_by(|(_, left), (_, right)| xor_distance_cmp(&left.id, &right.id, &self.info_hash))
.map(|(index, _)| index)?;
Some(self.frontier.swap_remove(index))
}
fn is_complete(&self, now: Instant) -> bool {
self.deadline <= now
|| self.peers.len() >= MAX_PEERS_PER_LOOKUP
|| (self.outstanding == 0
&& (self.queried >= MAX_QUERIES_PER_LOOKUP || self.frontier.is_empty()))
}
}
struct LookupResponse {
remote_addr: SocketAddr,
tid: TransactionId,
response: DhtResponse,
}
pub(crate) struct PeerLookupRequest {
pub(crate) info_hash: [u8; 20],
pub(crate) preferred_node: Option<NodeTuple>,
pub(crate) allow_iterative_fallback: bool,
pub(crate) source: DiscoverySource,
pub(crate) completion: Option<oneshot::Sender<PeerLookupResult>>,
}
impl PeerLookupRequest {
pub(crate) fn new(info_hash: [u8; 20]) -> Self {
Self {
info_hash,
preferred_node: None,
allow_iterative_fallback: true,
source: DiscoverySource::ActiveLookup,
completion: None,
}
}
}
#[derive(Clone)]
pub(crate) struct PeerLookupHandle {
request_tx: mpsc::Sender<PeerLookupRequest>,
response_tx: mpsc::Sender<LookupResponse>,
runtime_stats: DhtRuntimeStats,
}
pub(crate) struct PeerLookupRuntime {
pub(crate) options: PeerLookupOptions,
pub(crate) stats: DhtRuntimeStats,
pub(crate) outbound_query_budget: SharedRateBudget,
pub(crate) shutdown: CancellationToken,
}
impl PeerLookupHandle {
pub(crate) fn request_sender(&self) -> mpsc::Sender<PeerLookupRequest> {
self.request_tx.clone()
}
pub(crate) async fn lookup(
&self,
info_hash: [u8; 20],
) -> Result<PeerLookupResult, &'static str> {
let (completion, receiver) = oneshot::channel();
self.request_tx
.send(PeerLookupRequest {
info_hash,
preferred_node: None,
allow_iterative_fallback: true,
source: DiscoverySource::ActiveLookup,
completion: Some(completion),
})
.await
.map_err(|_| "Peer Lookup 已停止")?;
receiver.await.map_err(|_| "Peer Lookup 未返回结果")
}
pub(crate) fn route_response(
&self,
remote_addr: SocketAddr,
tid: TransactionId,
response: DhtResponse,
) {
if self
.response_tx
.try_send(LookupResponse {
remote_addr,
tid,
response,
})
.is_err()
{
self.runtime_stats.peer_lookup_response_dropped();
#[cfg(feature = "metrics")]
counter!("dht_peer_lookup_dropped_total", "reason" => "response_queue_full")
.increment(1);
}
}
}
pub(crate) fn is_peer_lookup_tid(tid: &TransactionId) -> bool {
tid[0] == LOOKUP_TID_TAG
}
pub(crate) fn spawn_peer_lookup(
netmode: NetMode,
local_id: [u8; 20],
sockets: &std::collections::HashMap<SocketAddr, Arc<UdpSocket>>,
snapshot: Arc<ArcSwap<RoutingSnapshot>>,
hash_tx: mpsc::Sender<HashDiscovered>,
runtime: PeerLookupRuntime,
) -> PeerLookupHandle {
let PeerLookupRuntime {
options,
stats,
outbound_query_budget,
shutdown,
} = runtime;
let (request_tx, request_rx) = mpsc::channel(REQUEST_CHANNEL_CAPACITY);
let (response_tx, response_rx) = mpsc::channel(RESPONSE_CHANNEL_CAPACITY);
let socket_v4 = sockets
.iter()
.find_map(|(addr, socket)| addr.is_ipv4().then(|| socket.clone()));
let socket_v6 = sockets
.iter()
.find_map(|(addr, socket)| addr.is_ipv6().then(|| socket.clone()));
let actor = PeerLookupActor {
netmode,
local_id,
socket_v4,
socket_v6,
snapshot,
hash_tx,
request_rx,
response_rx,
request_budget: RateBucket::per_second(
options.max_lookups_per_second,
options.burst,
true,
Instant::now(),
),
max_active_lookups: options.max_active_lookups,
enabled: options.max_lookups_per_second > 0 && options.max_active_lookups > 0,
queued: VecDeque::new(),
queued_hashes: AHashSet::new(),
active_hashes: AHashSet::new(),
active: AHashMap::new(),
pending: AHashMap::new(),
pending_expiry: VecDeque::new(),
next_lookup_id: 1,
next_tid: 1,
runtime_stats: stats.clone(),
outbound_query_budget,
shutdown,
};
tokio::spawn(actor.run());
PeerLookupHandle {
request_tx,
response_tx,
runtime_stats: stats,
}
}
struct PeerLookupActor {
netmode: NetMode,
local_id: [u8; 20],
socket_v4: Option<Arc<UdpSocket>>,
socket_v6: Option<Arc<UdpSocket>>,
snapshot: Arc<ArcSwap<RoutingSnapshot>>,
hash_tx: mpsc::Sender<HashDiscovered>,
request_rx: mpsc::Receiver<PeerLookupRequest>,
response_rx: mpsc::Receiver<LookupResponse>,
request_budget: RateBucket,
max_active_lookups: usize,
enabled: bool,
queued: VecDeque<PeerLookupRequest>,
queued_hashes: AHashSet<[u8; 20]>,
active_hashes: AHashSet<[u8; 20]>,
active: AHashMap<u64, LookupState>,
pending: AHashMap<PendingKey, PendingQuery>,
pending_expiry: VecDeque<(Instant, PendingKey)>,
next_lookup_id: u64,
next_tid: u64,
runtime_stats: DhtRuntimeStats,
outbound_query_budget: SharedRateBudget,
shutdown: CancellationToken,
}
impl PeerLookupActor {
async fn run(mut self) {
let mut maintenance = tokio::time::interval(MAINTENANCE_INTERVAL);
maintenance.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
loop {
tokio::select! {
biased;
_ = self.shutdown.cancelled() => break,
response = self.response_rx.recv() => {
let Some(response) = response else { break };
self.handle_response(response, Instant::now()).await;
}
_ = maintenance.tick() => self.expire(Instant::now()).await,
request = self.request_rx.recv() => {
let Some(request) = request else { break };
self.queue_request(request);
}
}
}
}
fn queue_request(&mut self, request: PeerLookupRequest) {
self.runtime_stats.peer_lookup_requested();
#[cfg(feature = "metrics")]
counter!("dht_peer_lookup_requests_total").increment(1);
if !self.enabled {
if let Some(completion) = request.completion {
let _ = completion.send(PeerLookupResult {
peers: Vec::new(),
queries: 0,
});
}
return;
}
if self.queued_hashes.contains(&request.info_hash)
|| self.active_hashes.contains(&request.info_hash)
{
if let Some(completion) = request.completion {
let _ = completion.send(PeerLookupResult {
peers: Vec::new(),
queries: 0,
});
}
return;
}
if self.queued.len() >= REQUEST_CHANNEL_CAPACITY {
self.runtime_stats.peer_lookup_rate_limited();
#[cfg(feature = "metrics")]
counter!("dht_peer_lookup_dropped_total", "reason" => "request_queue_full")
.increment(1);
if let Some(completion) = request.completion {
let _ = completion.send(PeerLookupResult {
peers: Vec::new(),
queries: 0,
});
}
return;
}
self.queued_hashes.insert(request.info_hash);
self.queued.push_back(request);
}
async fn start_queued(&mut self, now: Instant) {
while self.active.len() < self.max_active_lookups
&& !self.queued.is_empty()
&& self.request_budget.try_take_one(now)
{
let request = self.queued.pop_front().expect("queued request exists");
self.queued_hashes.remove(&request.info_hash);
if !self.start_lookup(request, now).await {
self.request_budget.refund_one();
}
}
}
async fn start_lookup(&mut self, request: PeerLookupRequest, now: Instant) -> bool {
let info_hash = request.info_hash;
let filter_ipv6 = match (self.socket_v4.is_some(), self.socket_v6.is_some()) {
(true, false) => Some(false),
(false, true) => Some(true),
_ => None,
};
let mut frontier =
self.snapshot
.load()
.closest_nodes(&info_hash, MAX_QUERIES_PER_LOOKUP, filter_ipv6);
let preferred = request.preferred_node;
if let Some(preferred) = preferred {
frontier.retain(|node| node.addr != preferred.addr);
}
if frontier.is_empty() && preferred.is_none() {
self.runtime_stats.peer_lookup_empty();
return false;
}
let lookup_id = self.next_lookup_id;
self.next_lookup_id = self.next_lookup_id.wrapping_add(1).max(1);
let mut seen_nodes: AHashSet<_> = frontier.iter().map(|node| node.addr).collect();
if let Some(preferred) = preferred {
seen_nodes.insert(preferred.addr);
}
self.active.insert(
lookup_id,
LookupState {
info_hash,
info_hash_hex: hex::encode(info_hash),
frontier,
preferred,
seen_nodes,
peers: AHashSet::new(),
queried: 0,
outstanding: 0,
deadline: now + LOOKUP_TIMEOUT,
preferred_phase: preferred.is_some(),
allow_iterative_fallback: request.allow_iterative_fallback,
source: request.source,
completion: request.completion,
},
);
self.active_hashes.insert(info_hash);
self.runtime_stats.peer_lookup_started();
#[cfg(feature = "metrics")]
counter!("dht_peer_lookup_started_total").increment(1);
self.dispatch_more(lookup_id, now).await;
true
}
async fn dispatch_more(&mut self, lookup_id: u64, now: Instant) {
loop {
if !self.outbound_query_budget.try_take_one(now) {
break;
}
let next = self.active.get_mut(&lookup_id).and_then(|state| {
if state.deadline <= now
|| state.peers.len() >= MAX_PEERS_PER_LOOKUP
|| state.queried >= MAX_QUERIES_PER_LOOKUP
|| state.outstanding >= MAX_CONCURRENT_QUERIES_PER_LOOKUP
|| (state.preferred_phase && state.outstanding > 0)
{
return None;
}
let preferred_phase = state.preferred_phase;
let node = state.pop_closest()?;
state.queried += 1;
Some((node, state.info_hash, preferred_phase))
});
let Some((node, info_hash, preferred_phase)) = next else {
self.outbound_query_budget.refund_one();
break;
};
let tid = self.next_transaction_id();
let key = PendingKey {
addr: node.addr,
tid,
};
let mut buffer = BytesMut::with_capacity(128);
encode_get_peers_query(&mut buffer, &tid, &info_hash, &self.local_id);
let socket = if node.addr.is_ipv4() {
self.socket_v4.clone()
} else {
self.socket_v6.clone()
};
let sent = match socket {
Some(socket) => socket.send_to(&buffer, node.addr).await.is_ok(),
None => false,
};
if !sent {
self.outbound_query_budget.refund_one();
self.runtime_stats.peer_lookup_send_failed();
if preferred_phase && let Some(state) = self.active.get_mut(&lookup_id) {
if !state.allow_iterative_fallback {
self.finish_lookup(lookup_id);
return;
}
state.preferred_phase = false;
state.deadline = now + LOOKUP_TIMEOUT;
self.runtime_stats.peer_lookup_fallback();
}
continue;
}
let deadline = now + QUERY_TIMEOUT;
self.pending.insert(
key,
PendingQuery {
lookup_id,
deadline,
preferred_phase,
},
);
self.pending_expiry.push_back((deadline, key));
if let Some(state) = self.active.get_mut(&lookup_id) {
state.outstanding += 1;
}
self.runtime_stats.udp_sent(buffer.len());
self.runtime_stats.peer_lookup_query();
#[cfg(feature = "metrics")]
counter!("dht_peer_lookup_queries_total").increment(1);
}
self.finish_if_complete(lookup_id, now);
}
async fn handle_response(&mut self, event: LookupResponse, now: Instant) {
let key = PendingKey {
addr: event.remote_addr,
tid: event.tid,
};
let Some(pending) = self.pending.remove(&key) else {
return;
};
let Some(state) = self.active.get_mut(&pending.lookup_id) else {
return;
};
state.outstanding = state.outstanding.saturating_sub(1);
if pending.preferred_phase {
state.preferred_phase = false;
state.deadline = now + LOOKUP_TIMEOUT;
}
self.runtime_stats.peer_lookup_response();
let mut discovered = Vec::new();
for_each_response_peer(&event.response, self.netmode, |peer| {
if state.peers.len() < MAX_PEERS_PER_LOOKUP && state.peers.insert(peer) {
discovered.push(peer);
}
});
let mut response_nodes = Vec::new();
for_each_response_node(&event.response, self.netmode, |node| {
response_nodes.push(node)
});
for node in response_nodes {
if state.frontier.len() >= MAX_FRONTIER_NODES {
break;
}
if state.seen_nodes.insert(node.addr) {
state.frontier.push(node);
}
}
let hash = state.info_hash_hex.clone();
let lookup_id = pending.lookup_id;
let preferred_succeeded = pending.preferred_phase && !discovered.is_empty();
let preferred_without_fallback = pending.preferred_phase && !state.allow_iterative_fallback;
let source = state.source;
let _ = state;
for peer in discovered {
let event = HashDiscovered {
info_hash: hash.clone(),
peer_addr: peer,
source,
discovered_at: now,
};
if self.hash_tx.try_send(event).is_ok() {
self.runtime_stats.peer_lookup_peer_found();
#[cfg(feature = "metrics")]
counter!("dht_peer_lookup_peers_found_total").increment(1);
} else {
self.runtime_stats.peer_lookup_output_dropped();
#[cfg(feature = "metrics")]
counter!("dht_peer_lookup_dropped_total", "reason" => "hash_queue_full")
.increment(1);
}
}
if preferred_succeeded {
self.runtime_stats.peer_lookup_preferred_succeeded();
self.finish_lookup(lookup_id);
return;
}
if preferred_without_fallback {
self.finish_lookup(lookup_id);
return;
}
if pending.preferred_phase {
self.runtime_stats.peer_lookup_fallback();
}
self.dispatch_more(lookup_id, now).await;
}
async fn expire(&mut self, now: Instant) {
let mut affected = AHashSet::new();
let mut finish_without_fallback = AHashSet::new();
while let Some((deadline, key)) = self.pending_expiry.front().copied() {
if deadline > now {
break;
}
self.pending_expiry.pop_front();
let should_remove = self
.pending
.get(&key)
.is_some_and(|pending| pending.deadline == deadline);
if !should_remove {
continue;
}
let pending = self.pending.remove(&key).expect("pending lookup exists");
if let Some(state) = self.active.get_mut(&pending.lookup_id) {
state.outstanding = state.outstanding.saturating_sub(1);
if pending.preferred_phase {
state.preferred_phase = false;
if state.allow_iterative_fallback {
state.deadline = now + LOOKUP_TIMEOUT;
self.runtime_stats.peer_lookup_fallback();
} else {
finish_without_fallback.insert(pending.lookup_id);
}
}
affected.insert(pending.lookup_id);
}
self.runtime_stats.peer_lookup_timeout();
}
for lookup_id in finish_without_fallback {
affected.remove(&lookup_id);
self.finish_lookup(lookup_id);
}
for lookup_id in affected {
self.dispatch_more(lookup_id, now).await;
}
let expired: Vec<_> = self
.active
.iter()
.filter_map(|(lookup_id, state)| (state.deadline <= now).then_some(*lookup_id))
.collect();
for lookup_id in expired {
self.finish_lookup(lookup_id);
}
self.start_queued(now).await;
}
fn finish_if_complete(&mut self, lookup_id: u64, now: Instant) {
if self
.active
.get(&lookup_id)
.is_some_and(|state| state.is_complete(now))
{
self.finish_lookup(lookup_id);
}
}
fn finish_lookup(&mut self, lookup_id: u64) {
if let Some(mut state) = self.active.remove(&lookup_id) {
self.active_hashes.remove(&state.info_hash);
if let Some(completion) = state.completion.take() {
let mut peers: Vec<_> = state.peers.into_iter().collect();
peers.sort_unstable();
let _ = completion.send(PeerLookupResult {
peers,
queries: state.queried,
});
}
}
self.pending
.retain(|_, pending| pending.lookup_id != lookup_id);
}
fn next_transaction_id(&mut self) -> TransactionId {
let mut tid = self.next_tid.to_be_bytes();
tid[0] = LOOKUP_TID_TAG;
self.next_tid = self.next_tid.wrapping_add(1).max(1);
tid
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::protocol::DhtMessage;
#[test]
fn lookup_transaction_ids_have_a_reserved_tag() {
let tid = [LOOKUP_TID_TAG, 1, 2, 3, 4, 5, 6, 7];
assert!(is_peer_lookup_tid(&tid));
assert!(!is_peer_lookup_tid(&[0; 8]));
}
#[tokio::test]
async fn lookup_response_feeds_discovered_peer_back_to_metadata_scheduler() {
let local_socket = Arc::new(UdpSocket::bind("127.0.0.1:0").await.unwrap());
let remote_socket = UdpSocket::bind("127.0.0.1:0").await.unwrap();
let fallback_socket = UdpSocket::bind("127.0.0.1:0").await.unwrap();
let remote_addr = remote_socket.local_addr().unwrap();
let fallback_addr = fallback_socket.local_addr().unwrap();
let local_addr = local_socket.local_addr().unwrap();
let sockets = std::collections::HashMap::from([(local_addr, local_socket)]);
let snapshot = Arc::new(ArcSwap::from_pointee(RoutingSnapshot::from_nodes(
vec![NodeTuple {
id: [8; 20],
addr: fallback_addr,
}],
1,
)));
let (hash_tx, mut hash_rx) = mpsc::channel(4);
let stats = DhtRuntimeStats::default();
let shutdown = CancellationToken::new();
let handle = spawn_peer_lookup(
NetMode::Ipv4Only,
[7; 20],
&sockets,
snapshot,
hash_tx,
PeerLookupRuntime {
options: PeerLookupOptions::default(),
stats: stats.clone(),
outbound_query_budget: SharedRateBudget::per_second(10_000, 10_000, true),
shutdown: shutdown.clone(),
},
);
let (completion, result_rx) = oneshot::channel();
handle
.request_tx
.send(PeerLookupRequest {
info_hash: [3; 20],
preferred_node: Some(NodeTuple {
id: [9; 20],
addr: remote_addr,
}),
allow_iterative_fallback: true,
source: DiscoverySource::SampleDirect,
completion: Some(completion),
})
.await
.unwrap();
let mut buffer = [0u8; 512];
let (len, source) =
tokio::time::timeout(Duration::from_secs(1), remote_socket.recv_from(&mut buffer))
.await
.unwrap()
.unwrap();
assert_eq!(source, local_addr);
let query: DhtMessage = serde_bencode::from_bytes(&buffer[..len]).unwrap();
assert_eq!(query.q.as_deref(), Some("get_peers"));
let tid: TransactionId = query.t.as_ref().try_into().unwrap();
tokio::time::sleep(Duration::from_millis(100)).await;
assert_eq!(stats.snapshot().peer_lookup_queries, 1);
assert!(fallback_socket.try_recv_from(&mut buffer).is_err());
handle.route_response(
remote_addr,
tid,
DhtResponse {
id: Some(serde_bytes::ByteBuf::from(vec![9; 20])),
nodes: None,
nodes6: None,
values: Some(vec![serde_bytes::ByteBuf::from(vec![
8, 8, 4, 4, 0x1a, 0xe1,
])]),
samples: None,
num: None,
interval: None,
},
);
let event = tokio::time::timeout(Duration::from_secs(1), hash_rx.recv())
.await
.unwrap()
.unwrap();
assert_eq!(event.info_hash, hex::encode([3; 20]));
assert_eq!(event.peer_addr, "8.8.4.4:6881".parse().unwrap());
assert_eq!(event.source, DiscoverySource::SampleDirect);
let result = tokio::time::timeout(Duration::from_secs(1), result_rx)
.await
.unwrap()
.unwrap();
assert_eq!(result.peers, vec!["8.8.4.4:6881".parse().unwrap()]);
assert_eq!(result.queries, 1);
let snapshot = stats.snapshot();
assert_eq!(snapshot.peer_lookup_started, 1);
assert_eq!(snapshot.peer_lookup_queries, 1);
assert_eq!(snapshot.peer_lookup_responses, 1);
assert_eq!(snapshot.peer_lookup_peers_found, 1);
assert_eq!(snapshot.peer_lookup_preferred_succeeded, 1);
assert_eq!(snapshot.peer_lookup_fallbacks, 0);
shutdown.cancel();
}
#[tokio::test]
async fn preferred_node_without_peers_falls_back_to_iterative_lookup() {
let local_socket = Arc::new(UdpSocket::bind("127.0.0.1:0").await.unwrap());
let preferred_socket = UdpSocket::bind("127.0.0.1:0").await.unwrap();
let fallback_socket = UdpSocket::bind("127.0.0.1:0").await.unwrap();
let local_addr = local_socket.local_addr().unwrap();
let preferred_addr = preferred_socket.local_addr().unwrap();
let fallback_addr = fallback_socket.local_addr().unwrap();
let sockets = std::collections::HashMap::from([(local_addr, local_socket)]);
let snapshot = Arc::new(ArcSwap::from_pointee(RoutingSnapshot::from_nodes(
vec![NodeTuple {
id: [8; 20],
addr: fallback_addr,
}],
1,
)));
let (hash_tx, _hash_rx) = mpsc::channel(4);
let stats = DhtRuntimeStats::default();
let shutdown = CancellationToken::new();
let handle = spawn_peer_lookup(
NetMode::Ipv4Only,
[7; 20],
&sockets,
snapshot,
hash_tx,
PeerLookupRuntime {
options: PeerLookupOptions::default(),
stats: stats.clone(),
outbound_query_budget: SharedRateBudget::per_second(10_000, 10_000, true),
shutdown: shutdown.clone(),
},
);
handle
.request_tx
.send(PeerLookupRequest {
info_hash: [3; 20],
preferred_node: Some(NodeTuple {
id: [9; 20],
addr: preferred_addr,
}),
allow_iterative_fallback: true,
source: DiscoverySource::SampleDirect,
completion: None,
})
.await
.unwrap();
let mut buffer = [0u8; 512];
let (len, _) = tokio::time::timeout(
Duration::from_secs(1),
preferred_socket.recv_from(&mut buffer),
)
.await
.unwrap()
.unwrap();
let query: DhtMessage = serde_bencode::from_bytes(&buffer[..len]).unwrap();
let tid: TransactionId = query.t.as_ref().try_into().unwrap();
handle.route_response(
preferred_addr,
tid,
DhtResponse {
id: Some(serde_bytes::ByteBuf::from(vec![9; 20])),
nodes: None,
nodes6: None,
values: None,
samples: None,
num: None,
interval: None,
},
);
tokio::time::timeout(
Duration::from_secs(1),
fallback_socket.recv_from(&mut buffer),
)
.await
.unwrap()
.unwrap();
let snapshot = stats.snapshot();
assert_eq!(snapshot.peer_lookup_queries, 2);
assert_eq!(snapshot.peer_lookup_preferred_succeeded, 0);
assert_eq!(snapshot.peer_lookup_fallbacks, 1);
shutdown.cancel();
}
#[tokio::test]
async fn preferred_node_without_peers_can_finish_without_iterative_lookup() {
let local_socket = Arc::new(UdpSocket::bind("127.0.0.1:0").await.unwrap());
let preferred_socket = UdpSocket::bind("127.0.0.1:0").await.unwrap();
let fallback_socket = UdpSocket::bind("127.0.0.1:0").await.unwrap();
let local_addr = local_socket.local_addr().unwrap();
let preferred_addr = preferred_socket.local_addr().unwrap();
let fallback_addr = fallback_socket.local_addr().unwrap();
let sockets = std::collections::HashMap::from([(local_addr, local_socket)]);
let snapshot = Arc::new(ArcSwap::from_pointee(RoutingSnapshot::from_nodes(
vec![NodeTuple {
id: [8; 20],
addr: fallback_addr,
}],
1,
)));
let (hash_tx, _hash_rx) = mpsc::channel(4);
let stats = DhtRuntimeStats::default();
let shutdown = CancellationToken::new();
let handle = spawn_peer_lookup(
NetMode::Ipv4Only,
[7; 20],
&sockets,
snapshot,
hash_tx,
PeerLookupRuntime {
options: PeerLookupOptions::default(),
stats: stats.clone(),
outbound_query_budget: SharedRateBudget::per_second(10_000, 10_000, true),
shutdown: shutdown.clone(),
},
);
let (completion, result_rx) = oneshot::channel();
handle
.request_tx
.send(PeerLookupRequest {
info_hash: [3; 20],
preferred_node: Some(NodeTuple {
id: [9; 20],
addr: preferred_addr,
}),
allow_iterative_fallback: false,
source: DiscoverySource::SampleDirect,
completion: Some(completion),
})
.await
.unwrap();
let mut buffer = [0u8; 512];
let (len, _) = tokio::time::timeout(
Duration::from_secs(1),
preferred_socket.recv_from(&mut buffer),
)
.await
.unwrap()
.unwrap();
let query: DhtMessage = serde_bencode::from_bytes(&buffer[..len]).unwrap();
let tid: TransactionId = query.t.as_ref().try_into().unwrap();
handle.route_response(
preferred_addr,
tid,
DhtResponse {
id: Some(serde_bytes::ByteBuf::from(vec![9; 20])),
nodes: None,
nodes6: None,
values: None,
samples: None,
num: None,
interval: None,
},
);
let result = tokio::time::timeout(Duration::from_secs(1), result_rx)
.await
.unwrap()
.unwrap();
assert!(result.peers.is_empty());
assert_eq!(result.queries, 1);
assert!(
tokio::time::timeout(
Duration::from_millis(200),
fallback_socket.recv_from(&mut buffer),
)
.await
.is_err()
);
let snapshot = stats.snapshot();
assert_eq!(snapshot.peer_lookup_queries, 1);
assert_eq!(snapshot.peer_lookup_fallbacks, 0);
shutdown.cancel();
}
}
+61
View File
@@ -0,0 +1,61 @@
// 负责定义 DHT KRPC 消息查询参数和响应字段的反序列化模型
use serde::Deserialize;
#[derive(Deserialize, Debug, Clone)]
/// Decoded KRPC envelope.
pub struct DhtMessage {
/// Transaction ID bytes.
pub t: serde_bytes::ByteBuf,
/// Message kind (`q`, `r`, or `e`).
pub y: String,
/// Query method when `y == q`.
pub q: Option<String>,
/// Query arguments.
pub a: Option<DhtArgs>,
/// Response dictionary.
pub r: Option<DhtResponse>,
}
#[derive(Deserialize, Debug, Clone)]
/// Supported BEP-5 query arguments.
pub struct DhtArgs {
/// Sender node ID.
pub id: Option<serde_bytes::ByteBuf>,
/// find_node target ID.
pub target: Option<serde_bytes::ByteBuf>,
/// get_peers/announce InfoHash.
pub info_hash: Option<serde_bytes::ByteBuf>,
/// announce validation token.
pub token: Option<serde_bytes::ByteBuf>,
/// Explicit announced Peer port.
pub port: Option<u16>,
/// Non-zero means use the UDP source port.
pub implied_port: Option<u8>,
}
#[derive(Deserialize, Debug, Clone)]
/// Supported BEP-5 response fields.
pub struct DhtResponse {
#[serde(default)]
/// Responder node ID.
pub id: Option<serde_bytes::ByteBuf>,
#[serde(default)]
/// Compact IPv4 node tuples.
pub nodes: Option<serde_bytes::ByteBuf>,
#[serde(default)]
/// Compact IPv6 node tuples.
pub nodes6: Option<serde_bytes::ByteBuf>,
#[serde(default)]
/// Compact Peer endpoints returned by `get_peers`.
pub values: Option<Vec<serde_bytes::ByteBuf>>,
#[serde(default)]
/// BEP-51 concatenated 20-byte sampled InfoHashes.
pub samples: Option<serde_bytes::ByteBuf>,
#[serde(default)]
/// BEP-51 estimated number of InfoHashes held by the responder.
pub num: Option<u64>,
#[serde(default)]
/// BEP-51 requested delay before sampling this node again, in seconds.
pub interval: Option<u64>,
}
+268
View File
@@ -0,0 +1,268 @@
// 负责限制 DHT 响应的包速率字节速率单来源速率和优先保留预算
use std::{
collections::VecDeque,
net::SocketAddr,
time::{Duration, Instant},
};
use ahash::AHashMap;
use crate::budget::RateBucket;
#[derive(Clone, Copy)]
struct SourceResponseWindow {
started_at: Instant,
last_seen: Instant,
count: u32,
}
pub(crate) struct WorkerResponseLimiter {
regular_packets: RateBucket,
regular_bytes: RateBucket,
priority_packets: RateBucket,
priority_bytes: RateBucket,
per_source_rate: u32,
sources: AHashMap<SocketAddr, SourceResponseWindow>,
source_expiry: VecDeque<(Instant, SocketAddr)>,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum ResponsePermit {
Regular,
PriorityReserve,
Rejected,
}
impl WorkerResponseLimiter {
pub(crate) fn new(
packet_rate: u32,
byte_rate: u64,
per_source_rate: u32,
now: Instant,
) -> Self {
let byte_rate = byte_rate.min(u32::MAX as u64) as u32;
let priority_packet_rate = reserve_quota(packet_rate);
let priority_byte_rate = reserve_quota(byte_rate);
let packet_rate = packet_rate.saturating_sub(priority_packet_rate);
let byte_rate = byte_rate.saturating_sub(priority_byte_rate);
Self {
regular_packets: RateBucket::per_second(
packet_rate,
packet_rate.div_ceil(5).max(1),
true,
now,
),
regular_bytes: RateBucket::per_second(
byte_rate,
byte_rate.div_ceil(5).max(512),
true,
now,
),
priority_packets: RateBucket::per_second(
priority_packet_rate,
priority_packet_rate.div_ceil(5).max(1),
true,
now,
),
priority_bytes: RateBucket::per_second(
priority_byte_rate,
priority_byte_rate.div_ceil(5).max(512),
true,
now,
),
per_source_rate,
sources: AHashMap::new(),
source_expiry: VecDeque::new(),
}
}
pub(crate) fn acquire(
&mut self,
addr: SocketAddr,
encoded_len: usize,
is_priority: bool,
now: Instant,
) -> ResponsePermit {
self.expire_sources(now);
if Self::take_budget(
&mut self.regular_packets,
&mut self.regular_bytes,
encoded_len,
now,
) {
if self.acquire_source_slot(addr, now) {
return ResponsePermit::Regular;
}
Self::refund_budget(
&mut self.regular_packets,
&mut self.regular_bytes,
encoded_len,
);
return ResponsePermit::Rejected;
}
if is_priority
&& Self::take_budget(
&mut self.priority_packets,
&mut self.priority_bytes,
encoded_len,
now,
)
{
if self.acquire_source_slot(addr, now) {
return ResponsePermit::PriorityReserve;
}
Self::refund_budget(
&mut self.priority_packets,
&mut self.priority_bytes,
encoded_len,
);
}
ResponsePermit::Rejected
}
fn take_budget(
packets: &mut RateBucket,
bytes: &mut RateBucket,
encoded_len: usize,
now: Instant,
) -> bool {
if !packets.try_take_one(now) {
return false;
}
if !bytes.try_take_exact(encoded_len, now) {
packets.refund_one();
return false;
}
true
}
fn refund_budget(packets: &mut RateBucket, bytes: &mut RateBucket, encoded_len: usize) {
packets.refund_one();
bytes.refund(encoded_len);
}
fn acquire_source_slot(&mut self, addr: SocketAddr, now: Instant) -> bool {
if self.per_source_rate == 0 {
return false;
}
let entry = self.sources.entry(addr).or_insert(SourceResponseWindow {
started_at: now,
last_seen: now,
count: 0,
});
if now
.checked_duration_since(entry.started_at)
.unwrap_or_default()
>= Duration::from_secs(1)
{
entry.started_at = now;
entry.count = 0;
}
if entry.count >= self.per_source_rate {
return false;
}
entry.count += 1;
entry.last_seen = now;
self.source_expiry
.push_back((now + Duration::from_secs(60), addr));
true
}
fn expire_sources(&mut self, now: Instant) {
while let Some((deadline, addr)) = self.source_expiry.front().copied() {
if deadline > now {
break;
}
self.source_expiry.pop_front();
if self.sources.get(&addr).is_some_and(|entry| {
now.checked_duration_since(entry.last_seen)
.unwrap_or_default()
>= Duration::from_secs(60)
}) {
self.sources.remove(&addr);
}
}
}
}
pub(crate) fn split_u32_quota(total: u32, workers: usize, worker: usize) -> u32 {
let workers = workers.max(1) as u32;
total / workers + u32::from((worker as u32) < total % workers)
}
pub(crate) fn split_u64_quota(total: u64, workers: usize, worker: usize) -> u64 {
let workers = workers.max(1) as u64;
total / workers + u64::from((worker as u64) < total % workers)
}
fn reserve_quota(total: u32) -> u32 {
if total == 0 {
0
} else {
total.div_ceil(10).min(total)
}
}
#[cfg(test)]
mod tests {
use std::{
net::{IpAddr, Ipv4Addr},
time::Duration,
};
use super::*;
#[test]
fn enforces_packet_byte_and_source_limits() {
let start = Instant::now();
let addr = SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 1);
let mut limiter = WorkerResponseLimiter::new(2, 200, 1, start);
assert_eq!(
limiter.acquire(addr, 50, false, start),
ResponsePermit::Regular
);
assert_eq!(
limiter.acquire(addr, 50, false, start),
ResponsePermit::Rejected
);
assert_eq!(
limiter.acquire(addr, 50, false, start + Duration::from_secs(1)),
ResponsePermit::Regular
);
assert_eq!(
limiter.acquire(addr, 513, false, start + Duration::from_secs(2)),
ResponsePermit::Rejected
);
}
#[test]
fn priority_queries_can_use_the_reserve() {
let start = Instant::now();
let mut limiter = WorkerResponseLimiter::new(10, 1_000, 100, start);
let first = SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 1);
let second = SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 2);
let priority = SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 3);
assert_eq!(
limiter.acquire(first, 50, false, start),
ResponsePermit::Regular
);
assert_eq!(
limiter.acquire(second, 50, false, start),
ResponsePermit::Regular
);
assert_eq!(
limiter.acquire(priority, 50, false, start),
ResponsePermit::Rejected
);
assert_eq!(
limiter.acquire(priority, 50, true, start),
ResponsePermit::PriorityReserve
);
let exhausted = SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 4);
assert_eq!(
limiter.acquire(exhausted, 100, true, start),
ResponsePermit::Rejected
);
}
}
+109
View File
@@ -0,0 +1,109 @@
// 负责提供并发可读的 DHT 路由节点快照和距离查询
use crate::types::NodeTuple;
use rand::seq::IndexedRandom;
#[derive(Default)]
pub(crate) struct RoutingSnapshot {
v4: Vec<NodeTuple>,
v6: Vec<NodeTuple>,
}
impl RoutingSnapshot {
pub(crate) fn from_nodes(nodes: Vec<NodeTuple>, limit: usize) -> Self {
let mut v4 = Vec::new();
let mut v6 = Vec::new();
for node in nodes.into_iter().take(limit) {
if node.addr.is_ipv6() {
v6.push(node);
} else {
v4.push(node);
}
}
Self { v4, v6 }
}
pub(crate) fn random_nodes(&self, count: usize, filter_ipv6: Option<bool>) -> Vec<NodeTuple> {
let mut rng = rand::rng();
match filter_ipv6 {
Some(true) => self.v6.sample(&mut rng, count).cloned().collect(),
Some(false) => self.v4.sample(&mut rng, count).cloned().collect(),
None => {
let mut all = Vec::with_capacity(self.v4.len() + self.v6.len());
all.extend_from_slice(&self.v4);
all.extend_from_slice(&self.v6);
all.sample(&mut rng, count).cloned().collect()
}
}
}
pub(crate) fn closest_nodes(
&self,
target: &[u8; 20],
count: usize,
filter_ipv6: Option<bool>,
) -> Vec<NodeTuple> {
if count == 0 {
return Vec::new();
}
let mut nodes = match filter_ipv6 {
Some(true) => self.v6.clone(),
Some(false) => self.v4.clone(),
None => {
let mut all = Vec::with_capacity(self.v4.len() + self.v6.len());
all.extend_from_slice(&self.v4);
all.extend_from_slice(&self.v6);
all
}
};
let compare =
|left: &NodeTuple, right: &NodeTuple| xor_distance_cmp(&left.id, &right.id, target);
if nodes.len() > count {
nodes.select_nth_unstable_by(count, compare);
nodes.truncate(count);
}
nodes.sort_unstable_by(compare);
nodes
}
}
pub(crate) fn xor_distance_cmp(
left: &[u8; 20],
right: &[u8; 20],
target: &[u8; 20],
) -> std::cmp::Ordering {
for index in 0..20 {
let ordering = (left[index] ^ target[index]).cmp(&(right[index] ^ target[index]));
if ordering != std::cmp::Ordering::Equal {
return ordering;
}
}
std::cmp::Ordering::Equal
}
#[cfg(test)]
mod tests {
use super::*;
use std::net::SocketAddr;
fn node(id: u8, port: u16) -> NodeTuple {
NodeTuple {
id: [id; 20],
addr: SocketAddr::from(([8, 8, 8, 8], port)),
}
}
#[test]
fn closest_nodes_orders_by_xor_distance_and_limits_results() {
let snapshot =
RoutingSnapshot::from_nodes(vec![node(0xf0, 1), node(0x01, 2), node(0x10, 3)], 3);
let closest = snapshot.closest_nodes(&[0; 20], 2, Some(false));
assert_eq!(
closest
.iter()
.map(|node| node.addr.port())
.collect::<Vec<_>>(),
vec![2, 3]
);
}
}
File diff suppressed because it is too large Load Diff
+728
View File
@@ -0,0 +1,728 @@
// 负责执行有界 BEP-51 采样并将准入后的 infohash 交给 Peer 查找
use crate::addr::is_valid_node_addr;
use crate::budget::{RateBucket, SharedRateBudget};
use crate::crawl_engine::CrawlEngine;
use crate::krpc::{encode_sample_infohashes_query, for_each_response_node};
use crate::node_id::{TransactionId, random_node_id};
use crate::peer_lookup::{PeerLookupHandle, PeerLookupRequest};
use crate::protocol::DhtResponse;
use crate::routing_snapshot::RoutingSnapshot;
use crate::runtime_stats::DhtRuntimeStats;
use crate::types::{DiscoverySource, NetMode, NodeTuple, SampleInfohashesOptions};
use ahash::{AHashMap, AHashSet};
use arc_swap::{ArcSwap, ArcSwapOption};
use bytes::BytesMut;
#[cfg(feature = "metrics")]
use metrics::{counter, gauge};
use std::collections::VecDeque;
use std::future::Future;
use std::net::SocketAddr;
use std::sync::Arc;
use std::time::{Duration, Instant};
use tokio::net::UdpSocket;
use tokio::sync::mpsc;
use tokio_util::sync::CancellationToken;
const SAMPLE_TID_TAG: u8 = 0x51;
const RESPONSE_CHANNEL_CAPACITY: usize = 4_096;
const ADMISSION_BATCH_CHANNEL_CAPACITY: usize = 64;
const MAINTENANCE_INTERVAL: Duration = Duration::from_millis(25);
const PRODUCTIVE_REVISIT: Duration = Duration::from_secs(60);
const MAX_PROTOCOL_INTERVAL: Duration = Duration::from_secs(6 * 60 * 60);
#[derive(Debug, Clone, Copy, Hash, PartialEq, Eq)]
struct PendingKey {
addr: SocketAddr,
tid: TransactionId,
}
#[derive(Debug, Clone, Copy)]
struct PendingRequest {
deadline: Instant,
source: DiscoverySource,
}
struct SampleResponse {
remote_addr: SocketAddr,
tid: TransactionId,
response: DhtResponse,
}
struct SampleAdmissionBatch {
preferred_node: NodeTuple,
source: DiscoverySource,
hashes: Vec<[u8; 20]>,
}
pub(crate) type SampleHashAdmissionCallback = Box<
dyn Fn(Vec<[u8; 20]>) -> std::pin::Pin<Box<dyn Future<Output = Vec<[u8; 20]>> + Send>>
+ Send
+ Sync
+ 'static,
>;
#[derive(Clone)]
pub(crate) struct SampleInfohashesHandle {
response_tx: mpsc::Sender<SampleResponse>,
runtime_stats: DhtRuntimeStats,
}
#[derive(Clone)]
pub(crate) struct SampleCandidateRouter {
sender: mpsc::Sender<NodeTuple>,
sample_percent: u8,
runtime_stats: DhtRuntimeStats,
}
impl SampleCandidateRouter {
pub(crate) fn route(&self, node: NodeTuple) -> bool {
if self.sample_percent == 0
|| !is_valid_node_addr(&node.addr)
|| stable_address_bucket(node.addr) >= self.sample_percent
{
return false;
}
match self.sender.try_send(node) {
Ok(()) => {
self.runtime_stats.sample_candidate_routed();
self.runtime_stats.set_sample_candidate_queue_depth(
self.sender
.max_capacity()
.saturating_sub(self.sender.capacity()),
);
true
}
Err(mpsc::error::TrySendError::Full(_)) => {
self.runtime_stats.sample_candidate_fallback();
self.runtime_stats
.set_sample_candidate_queue_depth(self.sender.max_capacity());
false
}
Err(mpsc::error::TrySendError::Closed(_)) => false,
}
}
}
pub(crate) fn sample_candidate_lane(
options: &SampleInfohashesOptions,
runtime_stats: DhtRuntimeStats,
) -> (SampleCandidateRouter, mpsc::Receiver<NodeTuple>) {
let capacity = options.candidate_queue_capacity.max(1);
let (sender, receiver) = mpsc::channel(capacity);
runtime_stats.configure_sample_candidate_queue(capacity);
(
SampleCandidateRouter {
sender,
sample_percent: options.new_node_sample_percent.min(100),
runtime_stats,
},
receiver,
)
}
fn stable_address_bucket(addr: SocketAddr) -> u8 {
let mut hash = 0xcbf2_9ce4_8422_2325u64;
let mut mix = |byte: u8| {
hash ^= u64::from(byte);
hash = hash.wrapping_mul(0x0000_0100_0000_01b3);
};
match addr.ip() {
std::net::IpAddr::V4(ip) => ip.octets().into_iter().for_each(&mut mix),
std::net::IpAddr::V6(ip) => ip.octets().into_iter().for_each(&mut mix),
}
addr.port().to_be_bytes().into_iter().for_each(&mut mix);
(hash % 100) as u8
}
impl SampleInfohashesHandle {
pub(crate) fn route_response(
&self,
remote_addr: SocketAddr,
tid: TransactionId,
response: DhtResponse,
) {
if self
.response_tx
.try_send(SampleResponse {
remote_addr,
tid,
response,
})
.is_err()
{
self.runtime_stats.sample_response_dropped();
#[cfg(feature = "metrics")]
counter!("dht_sample_infohashes_dropped_total", "reason" => "response_queue_full")
.increment(1);
}
}
}
pub(crate) fn is_sample_infohashes_tid(tid: &TransactionId) -> bool {
tid[0] == SAMPLE_TID_TAG
}
pub(crate) struct SampleInfohashesRuntime {
pub(crate) options: SampleInfohashesOptions,
pub(crate) stats: DhtRuntimeStats,
pub(crate) outbound_query_budget: SharedRateBudget,
pub(crate) hash_admission: Arc<ArcSwapOption<SampleHashAdmissionCallback>>,
pub(crate) shutdown: CancellationToken,
}
pub(crate) struct SampleInfohashesInputs<'a> {
pub(crate) sockets: &'a std::collections::HashMap<SocketAddr, Arc<UdpSocket>>,
pub(crate) snapshot: Arc<ArcSwap<RoutingSnapshot>>,
pub(crate) candidate_rx: mpsc::Receiver<NodeTuple>,
pub(crate) crawl_engine: Arc<CrawlEngine>,
pub(crate) peer_lookup: PeerLookupHandle,
}
pub(crate) fn spawn_sample_infohashes(
netmode: NetMode,
local_id: [u8; 20],
inputs: SampleInfohashesInputs<'_>,
runtime: SampleInfohashesRuntime,
) -> SampleInfohashesHandle {
let SampleInfohashesInputs {
sockets,
snapshot,
candidate_rx,
crawl_engine,
peer_lookup,
} = inputs;
let SampleInfohashesRuntime {
options,
stats,
outbound_query_budget,
hash_admission,
shutdown,
} = runtime;
let (response_tx, response_rx) = mpsc::channel(RESPONSE_CHANNEL_CAPACITY);
let (admission_tx, admission_rx) = mpsc::channel(ADMISSION_BATCH_CHANNEL_CAPACITY);
let socket_v4 = sockets
.iter()
.find_map(|(addr, socket)| addr.is_ipv4().then(|| socket.clone()));
let socket_v6 = sockets
.iter()
.find_map(|(addr, socket)| addr.is_ipv6().then(|| socket.clone()));
let now = Instant::now();
spawn_sample_admission(
admission_rx,
hash_admission,
peer_lookup.request_sender(),
options.fallback_to_iterative,
stats.clone(),
shutdown.clone(),
);
let actor = SampleInfohashesActor {
netmode,
local_id,
socket_v4,
socket_v6,
snapshot,
candidate_rx,
direct_only: options.new_node_sample_percent > 0,
crawl_engine,
admission_tx,
response_rx,
query_budget: RateBucket::per_second(
options.max_queries_per_second,
options.burst,
true,
now,
),
max_in_flight: options.max_in_flight,
request_timeout: Duration::from_millis(options.request_timeout_millis.max(100)),
unsupported_backoff: Duration::from_secs(options.unsupported_backoff_secs.max(1)),
dedup_capacity: options.dedup_capacity,
seen_hashes: AHashSet::new(),
seen_order: VecDeque::new(),
next_allowed: AHashMap::new(),
pending_addrs: AHashSet::new(),
pending: AHashMap::new(),
pending_expiry: VecDeque::new(),
next_tid: 1,
runtime_stats: stats.clone(),
outbound_query_budget,
shutdown,
};
tokio::spawn(actor.run());
SampleInfohashesHandle {
response_tx,
runtime_stats: stats,
}
}
struct SampleInfohashesActor {
netmode: NetMode,
local_id: [u8; 20],
socket_v4: Option<Arc<UdpSocket>>,
socket_v6: Option<Arc<UdpSocket>>,
snapshot: Arc<ArcSwap<RoutingSnapshot>>,
candidate_rx: mpsc::Receiver<NodeTuple>,
direct_only: bool,
crawl_engine: Arc<CrawlEngine>,
admission_tx: mpsc::Sender<SampleAdmissionBatch>,
response_rx: mpsc::Receiver<SampleResponse>,
query_budget: RateBucket,
max_in_flight: usize,
request_timeout: Duration,
unsupported_backoff: Duration,
dedup_capacity: usize,
seen_hashes: AHashSet<[u8; 20]>,
seen_order: VecDeque<[u8; 20]>,
next_allowed: AHashMap<SocketAddr, Instant>,
pending_addrs: AHashSet<SocketAddr>,
pending: AHashMap<PendingKey, PendingRequest>,
pending_expiry: VecDeque<(Instant, PendingKey)>,
next_tid: u64,
runtime_stats: DhtRuntimeStats,
outbound_query_budget: SharedRateBudget,
shutdown: CancellationToken,
}
fn spawn_sample_admission(
mut receiver: mpsc::Receiver<SampleAdmissionBatch>,
hash_admission: Arc<ArcSwapOption<SampleHashAdmissionCallback>>,
peer_lookup_tx: mpsc::Sender<PeerLookupRequest>,
fallback_to_iterative: bool,
runtime_stats: DhtRuntimeStats,
shutdown: CancellationToken,
) {
tokio::spawn(async move {
loop {
let batch = tokio::select! {
_ = shutdown.cancelled() => break,
batch = receiver.recv() => {
let Some(batch) = batch else { break };
batch
}
};
let input_len = batch.hashes.len();
let admitted = match hash_admission.load_full() {
Some(callback) => callback(batch.hashes).await,
None => batch.hashes,
};
runtime_stats.sample_hash_filtered(input_len.saturating_sub(admitted.len()));
for info_hash in admitted {
let request = PeerLookupRequest {
info_hash,
preferred_node: Some(batch.preferred_node),
allow_iterative_fallback: fallback_to_iterative,
source: batch.source,
completion: None,
};
let sent = tokio::select! {
_ = shutdown.cancelled() => false,
result = peer_lookup_tx.send(request) => result.is_ok(),
};
if !sent {
return;
}
runtime_stats.sample_hash_discovered();
}
}
});
}
impl SampleInfohashesActor {
async fn run(mut self) {
let mut maintenance = tokio::time::interval(MAINTENANCE_INTERVAL);
maintenance.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
loop {
tokio::select! {
biased;
_ = self.shutdown.cancelled() => break,
response = self.response_rx.recv() => {
let Some(response) = response else { break };
self.handle_response(response, Instant::now());
}
_ = maintenance.tick() => {
let now = Instant::now();
self.expire(now);
self.dispatch(now).await;
}
}
}
self.runtime_stats.set_sample_candidate_queue_depth(0);
}
async fn dispatch(&mut self, now: Instant) {
if self.max_in_flight == 0
|| self.pending.len() >= self.max_in_flight
|| self.admission_tx.capacity() == 0
{
return;
}
let available = self.max_in_flight.saturating_sub(self.pending.len());
let budget = self.query_budget.try_take(available, now);
if budget == 0 {
return;
}
let filter_ipv6 = match (self.socket_v4.is_some(), self.socket_v6.is_some()) {
(true, false) => Some(false),
(false, true) => Some(true),
_ => None,
};
let candidates = self
.snapshot
.load()
.random_nodes((budget * 8).max(64), filter_ipv6);
let mut sent_count = 0usize;
while sent_count < budget {
let Some(node) = self.next_direct_candidate(now) else {
break;
};
if self
.send_query(node, DiscoverySource::SampleDirect, now)
.await
{
sent_count += 1;
}
}
if self.direct_only {
if sent_count < budget {
self.query_budget.refund(budget - sent_count);
}
return;
}
for node in candidates {
if sent_count >= budget {
break;
}
if self.pending_addrs.contains(&node.addr)
|| self
.next_allowed
.get(&node.addr)
.is_some_and(|deadline| *deadline > now)
{
continue;
}
if self
.send_query(node, DiscoverySource::SampleSnapshot, now)
.await
{
sent_count += 1;
}
}
if sent_count < budget {
self.query_budget.refund(budget - sent_count);
}
}
fn next_direct_candidate(&mut self, now: Instant) -> Option<NodeTuple> {
for _ in 0..64 {
let node = self.candidate_rx.try_recv().ok()?;
self.runtime_stats
.set_sample_candidate_queue_depth(self.candidate_rx.len());
if self.pending_addrs.contains(&node.addr)
|| self
.next_allowed
.get(&node.addr)
.is_some_and(|deadline| *deadline > now)
{
continue;
}
return Some(node);
}
None
}
async fn send_query(&mut self, node: NodeTuple, source: DiscoverySource, now: Instant) -> bool {
let socket = if node.addr.is_ipv4() {
self.socket_v4.clone()
} else {
self.socket_v6.clone()
};
let Some(socket) = socket else {
return false;
};
if !self.outbound_query_budget.try_take_one(now) {
return false;
}
let tid = self.next_transaction_id();
let mut buffer = BytesMut::with_capacity(128);
encode_sample_infohashes_query(&mut buffer, &tid, &random_node_id(), &self.local_id);
if socket.send_to(&buffer, node.addr).await.is_err() {
self.outbound_query_budget.refund_one();
self.runtime_stats.sample_send_failed();
self.next_allowed
.insert(node.addr, now + self.unsupported_backoff);
return false;
}
let key = PendingKey {
addr: node.addr,
tid,
};
let deadline = now + self.request_timeout;
self.pending
.insert(key, PendingRequest { deadline, source });
self.pending_expiry.push_back((deadline, key));
self.pending_addrs.insert(node.addr);
self.runtime_stats.udp_sent(buffer.len());
self.runtime_stats
.sample_query(source == DiscoverySource::SampleDirect);
#[cfg(feature = "metrics")]
{
counter!("dht_sample_infohashes_queries_total").increment(1);
gauge!("dht_sample_infohashes_in_flight").set(self.pending.len() as f64);
}
true
}
fn handle_response(&mut self, event: SampleResponse, now: Instant) {
let key = PendingKey {
addr: event.remote_addr,
tid: event.tid,
};
let Some(pending) = self.pending.remove(&key) else {
return;
};
self.pending_addrs.remove(&event.remote_addr);
self.runtime_stats
.sample_response(pending.source == DiscoverySource::SampleDirect);
for_each_response_node(&event.response, self.netmode, |node| {
self.crawl_engine.route_discovered(node)
});
let responder_id = event
.response
.id
.as_deref()
.and_then(|id| <[u8; 20]>::try_from(id.as_slice()).ok())
.unwrap_or([0; 20]);
let preferred_node = NodeTuple {
id: responder_id,
addr: event.remote_addr,
};
let mut hashes = Vec::new();
let mut batch_hashes = AHashSet::new();
if let Some(samples) = event.response.samples.as_deref()
&& samples.len() % 20 == 0
{
for chunk in samples.chunks_exact(20) {
let hash: [u8; 20] = chunk.try_into().expect("sample hash is 20 bytes");
if self.seen_hashes.contains(&hash) {
self.runtime_stats.sample_hash_duplicate();
continue;
}
if batch_hashes.insert(hash) {
hashes.push(hash);
}
}
}
let candidate_count = hashes.len();
let admitted_to_triage = if hashes.is_empty() {
false
} else {
let hashes_to_remember = hashes.clone();
if self
.admission_tx
.try_send(SampleAdmissionBatch {
preferred_node,
source: pending.source,
hashes,
})
.is_ok()
{
for hash in hashes_to_remember {
self.remember_hash(hash);
}
true
} else {
for _ in 0..candidate_count {
self.runtime_stats.sample_hash_dropped();
}
#[cfg(feature = "metrics")]
counter!("dht_sample_infohashes_dropped_total", "reason" => "admission_queue_full")
.increment(candidate_count as u64);
false
}
};
let discovered = usize::from(admitted_to_triage) * candidate_count;
let protocol_interval = Duration::from_secs(event.response.interval.unwrap_or(300))
.clamp(Duration::from_secs(10), MAX_PROTOCOL_INTERVAL);
let delay = if discovered > 0 {
protocol_interval.min(PRODUCTIVE_REVISIT)
} else {
protocol_interval.saturating_add(self.unsupported_backoff)
};
self.next_allowed.insert(event.remote_addr, now + delay);
#[cfg(feature = "metrics")]
{
counter!("dht_sample_infohashes_responses_total").increment(1);
counter!("dht_sample_infohashes_hashes_total").increment(discovered as u64);
gauge!("dht_sample_infohashes_in_flight").set(self.pending.len() as f64);
}
}
fn remember_hash(&mut self, hash: [u8; 20]) {
if self.dedup_capacity == 0 {
return;
}
if self.seen_hashes.insert(hash) {
self.seen_order.push_back(hash);
}
while self.seen_order.len() > self.dedup_capacity {
if let Some(expired) = self.seen_order.pop_front() {
self.seen_hashes.remove(&expired);
}
}
}
fn expire(&mut self, now: Instant) {
while let Some((deadline, key)) = self.pending_expiry.front().copied() {
if deadline > now {
break;
}
self.pending_expiry.pop_front();
if !self
.pending
.get(&key)
.is_some_and(|pending| pending.deadline == deadline)
{
continue;
}
self.pending.remove(&key);
self.pending_addrs.remove(&key.addr);
self.next_allowed
.insert(key.addr, now + self.unsupported_backoff);
self.runtime_stats.sample_timeout();
#[cfg(feature = "metrics")]
counter!("dht_sample_infohashes_timeouts_total").increment(1);
}
#[cfg(feature = "metrics")]
gauge!("dht_sample_infohashes_in_flight").set(self.pending.len() as f64);
}
fn next_transaction_id(&mut self) -> TransactionId {
let mut tid = self.next_tid.to_be_bytes();
tid[0] = SAMPLE_TID_TAG;
self.next_tid = self.next_tid.wrapping_add(1).max(1);
tid
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::protocol::DhtMessage;
fn node_for_lane(want_direct: bool, percent: u8) -> NodeTuple {
(1..=u16::MAX)
.map(|port| NodeTuple {
id: [port as u8; 20],
addr: SocketAddr::from(([8, 8, 8, 8], port)),
})
.find(|node| (stable_address_bucket(node.addr) < percent) == want_direct)
.expect("both routing lanes have an address")
}
#[test]
fn sample_transaction_ids_have_a_reserved_tag() {
assert!(is_sample_infohashes_tid(&[
SAMPLE_TID_TAG,
1,
2,
3,
4,
5,
6,
7
]));
assert!(!is_sample_infohashes_tid(&[0; 8]));
}
#[test]
fn bep51_response_fields_decode() {
let bytes = b"d1:rd2:id20:aaaaaaaaaaaaaaaaaaaa8:intervali60e3:numi2e7:samples40:bbbbbbbbbbbbbbbbbbbbcccccccccccccccccccce1:t8:123456781:y1:re";
let message: DhtMessage = serde_bencode::from_bytes(bytes).unwrap();
let response = message.r.unwrap();
assert_eq!(response.interval, Some(60));
assert_eq!(response.num, Some(2));
assert_eq!(response.samples.unwrap().len(), 40);
}
#[test]
fn new_node_routing_is_stable_and_bounded() {
let options = SampleInfohashesOptions {
new_node_sample_percent: 50,
candidate_queue_capacity: 1,
..SampleInfohashesOptions::default()
};
let stats = DhtRuntimeStats::default();
let (router, mut receiver) = sample_candidate_lane(&options, stats.clone());
let direct = node_for_lane(true, 50);
let crawl = node_for_lane(false, 50);
assert!(router.route(direct));
assert!(!router.route(crawl));
assert!(!router.route(direct));
assert_eq!(receiver.try_recv().unwrap(), direct);
let snapshot = stats.snapshot();
assert_eq!(snapshot.sample_candidate_queue_capacity, 1);
assert_eq!(snapshot.sample_candidates_routed, 1);
assert_eq!(snapshot.sample_candidates_fallback, 1);
}
#[test]
fn zero_percent_preserves_snapshot_only_sampling() {
let options = SampleInfohashesOptions {
new_node_sample_percent: 0,
..SampleInfohashesOptions::default()
};
let (router, mut receiver) = sample_candidate_lane(&options, DhtRuntimeStats::default());
assert!(!router.route(node_for_lane(false, 0)));
assert!(receiver.try_recv().is_err());
}
#[tokio::test]
async fn batch_admission_only_forwards_application_approved_hashes() {
let (batch_tx, batch_rx) = mpsc::channel(1);
let (lookup_tx, mut lookup_rx) = mpsc::channel(2);
let admission = Arc::new(ArcSwapOption::empty());
let callback: Arc<SampleHashAdmissionCallback> = Arc::new(Box::new(|hashes| {
Box::pin(async move { hashes.into_iter().filter(|hash| *hash == [2; 20]).collect() })
}));
admission.store(Some(callback));
let stats = DhtRuntimeStats::default();
let shutdown = CancellationToken::new();
spawn_sample_admission(
batch_rx,
admission,
lookup_tx,
false,
stats.clone(),
shutdown.clone(),
);
batch_tx
.send(SampleAdmissionBatch {
preferred_node: NodeTuple {
id: [9; 20],
addr: "127.0.0.1:6881".parse().unwrap(),
},
source: DiscoverySource::SampleDirect,
hashes: vec![[1; 20], [2; 20]],
})
.await
.unwrap();
let request = tokio::time::timeout(Duration::from_secs(1), lookup_rx.recv())
.await
.unwrap()
.unwrap();
assert_eq!(request.info_hash, [2; 20]);
assert!(!request.allow_iterative_fallback);
assert_eq!(request.source, DiscoverySource::SampleDirect);
let snapshot = stats.snapshot();
assert_eq!(snapshot.sample_infohashes_hashes_filtered, 1);
assert_eq!(snapshot.sample_infohashes_hashes_discovered, 1);
shutdown.cancel();
}
}
File diff suppressed because it is too large Load Diff
+808
View File
@@ -0,0 +1,808 @@
// 负责组合 DHT 网络组件回调生命周期和公开运行接口
use crate::addr::is_valid_node_addr;
use crate::budget::SharedRateBudget;
use crate::crawl_config::ResolvedCrawlConfig;
use crate::crawl_engine::CrawlEngine;
use crate::error::Result;
use crate::krpc::encode_response;
use crate::metadata::RbitFetcher;
use crate::node_id::{neighbor_node_id, random_node_id, transaction_id_from_bytes};
use crate::peer_lookup::{
PeerLookupHandle, PeerLookupRuntime, is_peer_lookup_tid, spawn_peer_lookup,
};
use crate::protocol::{DhtArgs, DhtMessage};
use crate::response_limiter::{
ResponsePermit, WorkerResponseLimiter, split_u32_quota, split_u64_quota,
};
use crate::runtime_stats::{DhtRuntimeLimits, DhtRuntimeStats};
use crate::sample_infohashes::{
SampleHashAdmissionCallback, SampleInfohashesHandle, SampleInfohashesInputs,
SampleInfohashesRuntime, is_sample_infohashes_tid, sample_candidate_lane,
spawn_sample_infohashes,
};
use crate::scheduler::{
MetadataCompletionCallback, MetadataFetchCallback, MetadataScheduler,
MetadataSchedulerCallbacks, MetadataSchedulerLimits, MetadataSchedulerRuntime,
TorrentAckCallback,
};
use crate::types::{
DHTOptions, DiscoverySource, HashDiscovered, MetadataFetchCompletion, NetMode, NodeTuple,
TorrentInfo,
};
use crate::udp_buffer::UdpBufferPool;
use crate::udp_ingress::{WorkerHandle, spawn_udp_listener};
use arc_swap::ArcSwapOption;
use bytes::BytesMut;
#[cfg(feature = "metrics")]
use metrics::counter;
use rand::RngExt;
use socket2::{Domain, Protocol, Socket, Type};
use std::collections::HashMap;
use std::future::Future;
use std::hash::{Hash, Hasher};
use std::net::{IpAddr, Ipv4Addr, Ipv6Addr, SocketAddr};
use std::sync::Arc;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::{Duration, Instant};
use tokio::net::UdpSocket;
use tokio::sync::mpsc;
use tokio_util::sync::CancellationToken;
type FilterCallback = Box<dyn Fn(&str) -> bool + Send + Sync + 'static>;
type ErrorCallback = Box<dyn Fn(crate::error::DHTError) + Send + Sync + 'static>;
struct QueryResponse<'a> {
transaction_id: &'a [u8],
remote_addr: SocketAddr,
local_addr: SocketAddr,
query_type: &'a str,
sender_id: Option<&'a [u8]>,
target_id: Option<&'a [u8]>,
}
#[derive(Clone)]
/// Cloneable BEP-5 server handle and primary crate entry point.
pub struct DHTServer {
options: DHTOptions,
crawl_config: ResolvedCrawlConfig,
node_id: [u8; 20],
sockets_by_bind_addr: Arc<HashMap<SocketAddr, Arc<UdpSocket>>>,
token_secret: [u8; 10],
torrent_callback: Arc<ArcSwapOption<TorrentAckCallback>>,
hash_filter: Arc<ArcSwapOption<FilterCallback>>,
on_metadata_fetch: Arc<ArcSwapOption<MetadataFetchCallback>>,
sample_hash_admission: Arc<ArcSwapOption<SampleHashAdmissionCallback>>,
metadata_completion_callback: Arc<ArcSwapOption<MetadataCompletionCallback>>,
error_callback: Arc<ArcSwapOption<ErrorCallback>>,
crawl_engine: Arc<CrawlEngine>,
peer_lookup: PeerLookupHandle,
metadata_fetcher: Arc<RbitFetcher>,
sample_infohashes: SampleInfohashesHandle,
hash_events_tx: mpsc::Sender<HashDiscovered>,
metadata_queue_len: Arc<AtomicUsize>,
max_metadata_queue_size: usize,
runtime_stats: DhtRuntimeStats,
shutdown: CancellationToken,
}
fn create_udp_sock(domain: Domain, ty: Type, addr: SocketAddr) -> std::io::Result<UdpSocket> {
let sock = Socket::new(domain, ty, Some(Protocol::UDP))?;
#[cfg(not(windows))]
{
sock.set_reuse_port(true)?;
if addr.is_ipv6() {
sock.set_only_v6(true)?;
}
}
let _ = sock.set_reuse_address(true);
sock.set_nonblocking(true)?;
let _ = sock.set_recv_buffer_size(32 * 1024 * 1024);
let _ = sock.set_send_buffer_size(8 * 1024 * 1024);
sock.bind(&addr.into())?;
UdpSocket::from_std(sock.into())
}
impl DHTServer {
/// Validates options, binds configured UDP sockets and constructs bounded pipelines.
///
/// Background Metadata scheduling begins during construction. Active crawling and UDP receive
/// loops begin when [`Self::start`] is awaited.
pub async fn new(options: DHTOptions) -> Result<Self> {
let crawl_config = ResolvedCrawlConfig::from_options(&options.crawl);
let runtime_stats = DhtRuntimeStats::with_limits(DhtRuntimeLimits {
metadata_queue: options.metadata.max_queue_size.max(1),
node_pool: crawl_config.pool_capacity,
node_pool_low_watermark: crawl_config.low_watermark,
find_node_in_flight: crawl_config.max_in_flight,
initial_find_node_rate: crawl_config.max_find_node_rate_per_sec,
hash_ingress_queue: options.hash_queue_capacity,
crawl_priority_queue: crawl_config.priority_event_channel_capacity,
crawl_discovery_queue: crawl_config.discovery_event_channel_capacity,
});
const ANY_V4_ADDR: SocketAddr =
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(0, 0, 0, 0)), 8080);
const ANY_V6_ADDR: SocketAddr =
SocketAddr::new(IpAddr::V6(Ipv6Addr::new(0, 0, 0, 0, 0, 0, 0, 0)), 8080);
let mut sockets_by_bind_addr = HashMap::new();
match options.netmode {
NetMode::Ipv4Only => {
let mut addr = ANY_V4_ADDR;
addr.set_port(options.port);
let sock = create_udp_sock(Domain::IPV4, Type::DGRAM, addr)?;
sockets_by_bind_addr.insert(addr, Arc::new(sock));
}
NetMode::Ipv6Only => {
let mut addr = ANY_V6_ADDR;
addr.set_port(options.port);
let sock = create_udp_sock(Domain::IPV6, Type::DGRAM, addr)?;
sockets_by_bind_addr.insert(addr, Arc::new(sock));
}
NetMode::DualStack => {
let mut addr_v4 = ANY_V4_ADDR;
addr_v4.set_port(options.port);
let sock_v4 = create_udp_sock(Domain::IPV4, Type::DGRAM, addr_v4)?;
sockets_by_bind_addr.insert(addr_v4, Arc::new(sock_v4));
let mut addr_v6 = ANY_V6_ADDR;
addr_v6.set_port(options.port);
let sock_v6 = create_udp_sock(Domain::IPV6, Type::DGRAM, addr_v6)?;
sockets_by_bind_addr.insert(addr_v6, Arc::new(sock_v6));
}
}
let node_id = random_node_id();
let mut token_secret = [0u8; 10];
rand::rng().fill(&mut token_secret);
let (hash_events_tx, hash_rx) =
mpsc::channel::<HashDiscovered>(options.hash_queue_capacity);
let fetcher = Arc::new(RbitFetcher::new_with_runtime_stats(
options.metadata.timeout_secs,
options.metadata.max_connects_per_second,
options.metadata.max_metadata_size_bytes,
options.metadata.peer_failure_cache_capacity,
options.metadata.peer_failure_ttl_secs,
runtime_stats.clone(),
));
let torrent_callback = Arc::new(ArcSwapOption::empty());
let on_metadata_fetch = Arc::new(ArcSwapOption::empty());
let sample_hash_admission = Arc::new(ArcSwapOption::empty());
let metadata_completion_callback = Arc::new(ArcSwapOption::empty());
let metadata_queue_len = Arc::new(AtomicUsize::new(0));
let shutdown = CancellationToken::new();
let outbound_query_budget = SharedRateBudget::per_second(
options.max_outbound_queries_per_second,
options.outbound_query_burst,
true,
);
let (sample_candidate_router, sample_candidate_rx) =
sample_candidate_lane(&options.sample_infohashes, runtime_stats.clone());
let crawl_engine = Arc::new(CrawlEngine::new(
crawl_config.clone(),
runtime_stats.clone(),
outbound_query_budget.clone(),
Some(sample_candidate_router),
));
let peer_lookup = spawn_peer_lookup(
options.netmode,
node_id,
&sockets_by_bind_addr,
crawl_engine.snapshot.clone(),
hash_events_tx.clone(),
PeerLookupRuntime {
options: options.peer_lookup.clone(),
stats: runtime_stats.clone(),
outbound_query_budget: outbound_query_budget.clone(),
shutdown: shutdown.clone(),
},
);
let sample_infohashes = spawn_sample_infohashes(
options.netmode,
node_id,
SampleInfohashesInputs {
sockets: &sockets_by_bind_addr,
snapshot: crawl_engine.snapshot.clone(),
candidate_rx: sample_candidate_rx,
crawl_engine: crawl_engine.clone(),
peer_lookup: peer_lookup.clone(),
},
SampleInfohashesRuntime {
options: options.sample_infohashes.clone(),
stats: runtime_stats.clone(),
outbound_query_budget,
hash_admission: sample_hash_admission.clone(),
shutdown: shutdown.clone(),
},
);
let scheduler = MetadataScheduler::new_with_runtime(
hash_rx,
fetcher.clone(),
MetadataSchedulerLimits {
queue_size: options.metadata.max_queue_size,
concurrency: options.metadata.max_worker_count,
},
MetadataSchedulerCallbacks {
torrent: torrent_callback.clone(),
fetch_gate: on_metadata_fetch.clone(),
completion: metadata_completion_callback.clone(),
},
metadata_queue_len.clone(),
shutdown.clone(),
MetadataSchedulerRuntime {
stats: runtime_stats.clone(),
peer_lookup_tx: Some(peer_lookup.request_sender()),
},
);
tokio::spawn(scheduler.run());
let max_metadata_queue_size = options.metadata.max_queue_size;
Ok(Self {
options,
crawl_config: crawl_config.clone(),
node_id,
sockets_by_bind_addr: Arc::new(sockets_by_bind_addr),
token_secret,
torrent_callback,
hash_filter: Arc::new(ArcSwapOption::empty()),
on_metadata_fetch,
sample_hash_admission,
metadata_completion_callback,
error_callback: Arc::new(ArcSwapOption::empty()),
crawl_engine,
peer_lookup,
metadata_fetcher: fetcher,
sample_infohashes,
hash_events_tx,
metadata_queue_len,
max_metadata_queue_size,
runtime_stats,
shutdown,
})
}
/// Registers the asynchronous admission gate invoked before the first real Peer attempt.
///
/// Returning `false` rejects the job without downloading Metadata and without emitting a
/// [`MetadataFetchCompletion`]. If no gate is registered, jobs are admitted.
pub fn on_metadata_fetch<F, Fut>(&self, callback: F)
where
F: Fn(String) -> Fut + Send + Sync + 'static,
Fut: Future<Output = bool> + Send + 'static,
{
let callback: Arc<MetadataFetchCallback> =
Arc::new(Box::new(move |hash| Box::pin(callback(hash))));
self.on_metadata_fetch.store(Some(callback));
}
/// Registers an asynchronous batch admission callback for BEP-51 sampled InfoHashes.
///
/// Only hashes returned by the callback proceed to active Peer lookup. The callback should
/// preserve hashes it cannot classify so transient application errors fail open.
pub fn on_sampled_hashes<F, Fut>(&self, callback: F)
where
F: Fn(Vec<[u8; 20]>) -> Fut + Send + Sync + 'static,
Fut: Future<Output = Vec<[u8; 20]>> + Send + 'static,
{
let callback: Arc<SampleHashAdmissionCallback> =
Arc::new(Box::new(move |hashes| Box::pin(callback(hashes))));
self.sample_hash_admission.store(Some(callback));
}
/// Registers a torrent callback whose return is implicitly treated as accepted delivery.
///
/// Registering a new torrent callback replaces the previous one.
pub fn on_torrent<F>(&self, callback: F)
where
F: Fn(TorrentInfo) + Send + Sync + 'static,
{
let callback: Arc<TorrentAckCallback> = Arc::new(Box::new(move |torrent| {
callback(torrent);
true
}));
self.torrent_callback.store(Some(callback));
}
/// Registers a torrent callback that acknowledges application delivery.
///
/// Returning `true` produces [`crate::MetadataFetchCompletionStatus::Accepted`]. Returning
/// `false` produces [`crate::MetadataFetchCompletionStatus::DeliveryRejected`]: the Metadata
/// download was
/// valid, but the application did not accept it. A callback panic is caught and treated as
/// rejected delivery. Registering this callback replaces any previous torrent callback.
pub fn on_torrent_with_ack<F>(&self, callback: F)
where
F: Fn(TorrentInfo) -> bool + Send + Sync + 'static,
{
let callback: Arc<TorrentAckCallback> = Arc::new(Box::new(callback));
self.torrent_callback.store(Some(callback));
}
/// Registers a callback invoked exactly once when an admitted Metadata job terminates.
///
/// Gate rejection does not emit a completion. `attempts` counts real Peer network attempts;
/// failure-cache skips are excluded. Registering a new callback replaces the previous one.
pub fn on_metadata_fetch_complete<F>(&self, callback: F)
where
F: Fn(MetadataFetchCompletion) + Send + Sync + 'static,
{
let callback: Arc<MetadataCompletionCallback> = Arc::new(Box::new(callback));
self.metadata_completion_callback.store(Some(callback));
}
/// Registers an early synchronous InfoHash filter for valid `announce_peer` queries.
///
/// Returning `false` prevents the Hash from entering the bounded ingress queue. Registering a
/// new filter replaces the previous one.
pub fn filter<F>(&self, filter: F)
where
F: Fn(&str) -> bool + Send + Sync + 'static,
{
let filter: Arc<FilterCallback> = Arc::new(Box::new(filter));
self.hash_filter.store(Some(filter));
}
/// Alias for [`Self::filter`].
pub fn set_filter<F>(&self, filter: F)
where
F: Fn(&str) -> bool + Send + Sync + 'static,
{
self.filter(filter);
}
/// Registers a runtime error callback, replacing any previous callback.
///
/// Initialization and `start` errors are still returned through [`Result`].
pub fn on_error<F>(&self, callback: F)
where
F: Fn(crate::error::DHTError) + Send + Sync + 'static,
{
let callback: Arc<ErrorCallback> = Arc::new(Box::new(callback));
self.error_callback.store(Some(callback));
}
fn emit_error(&self, error: crate::error::DHTError) {
if let Some(callback) = self.error_callback.load_full() {
callback(error);
}
}
/// Returns the current strict FIFO crawl-pool size.
pub fn get_node_pool_size(&self) -> usize {
self.crawl_engine.node_count.load(Ordering::Relaxed)
}
/// Returns a cheap cloneable handle to transport-neutral atomic runtime statistics.
pub fn runtime_stats(&self) -> DhtRuntimeStats {
self.runtime_stats.clone()
}
/// Performs one bounded active `get_peers` lookup and returns its unique Peer addresses.
pub async fn lookup_peers(&self, info_hash: [u8; 20]) -> Result<crate::PeerLookupResult> {
tokio::time::timeout(Duration::from_secs(10), self.peer_lookup.lookup(info_hash))
.await
.map_err(|_| crate::error::DHTError::Other("Peer Lookup 等待超时".to_owned()))?
.map_err(|message| crate::error::DHTError::Other(message.to_owned()))
}
/// Verifies that a Peer completes the BitTorrent handshake for the requested InfoHash.
pub async fn verify_peer_handshake(&self, info_hash: [u8; 20], peer: SocketAddr) -> bool {
self.metadata_fetcher
.verify_handshake(&info_hash, peer)
.await
}
/// Starts crawl and UDP background tasks, then waits until [`Self::shutdown`] is called.
///
/// A shut-down server cannot be restarted. Construct a new [`DHTServer`] for another run.
pub async fn start(&self) -> Result<()> {
if self.shutdown.is_cancelled() {
return Err(crate::error::DHTError::Other(
"DHT server is already shut down".to_string(),
));
}
self.crawl_engine.spawn(
self.options.netmode,
self.node_id,
&self.sockets_by_bind_addr,
self.metadata_queue_len.clone(),
self.max_metadata_queue_size,
self.shutdown.clone(),
);
let buffer_pool = UdpBufferPool::new();
let workers = self.spawn_workers(buffer_pool.clone());
for sock in self.sockets_by_bind_addr.values().cloned() {
spawn_udp_listener(
sock,
workers.clone(),
self.shutdown.clone(),
buffer_pool.clone(),
self.runtime_stats.clone(),
)?;
}
self.shutdown.cancelled().await;
Ok(())
}
/// Cancels UDP, crawl and Metadata tasks. This method is safe to call more than once.
pub fn shutdown(&self) {
self.shutdown.cancel();
}
fn spawn_workers(&self, buffer_pool: UdpBufferPool) -> Vec<WorkerHandle> {
let server = self.clone();
let shutdown = self.shutdown.clone();
let num_workers = std::thread::available_parallelism()
.map(|n| n.get())
.unwrap_or(8);
let queue_size = 5_000;
let mut workers: Vec<WorkerHandle> = Vec::with_capacity(num_workers);
for worker_id in 0..num_workers {
let (tx, mut rx) = mpsc::channel(queue_size);
workers.push(tx);
let server_clone = server.clone();
let cancellation_token = shutdown.clone();
let pool = buffer_pool.clone();
let packet_rate = split_u32_quota(
self.crawl_config.max_response_rate_per_sec,
num_workers,
worker_id,
);
let byte_rate = split_u64_quota(
self.crawl_config.max_response_bytes_per_sec,
num_workers,
worker_id,
);
let per_source_rate = self.crawl_config.max_response_rate_per_source;
tokio::spawn(async move {
let mut response_limiter = WorkerResponseLimiter::new(
packet_rate,
byte_rate,
per_source_rate,
Instant::now(),
);
let mut response_buffer = BytesMut::with_capacity(512);
loop {
tokio::select! {
_ = cancellation_token.cancelled() => break,
msg = rx.recv() => {
match msg {
Some((packet, remote_addr, local_addr)) => {
if let Err(e) = server_clone
.handle_message(
packet.payload(),
remote_addr,
local_addr,
&mut response_limiter,
&mut response_buffer,
)
.await
{
server_clone.emit_error(e);
}
pool.release(packet.buf);
}
None => break,
}
}
}
}
});
}
workers
}
async fn handle_message(
&self,
data: &[u8],
remote_addr: SocketAddr,
local_addr: SocketAddr,
response_limiter: &mut WorkerResponseLimiter,
response_buffer: &mut BytesMut,
) -> Result<()> {
if self.sockets_by_bind_addr.get(&local_addr).is_none() {
return Ok(());
}
let msg: DhtMessage = match serde_bencode::from_bytes(data) {
Ok(m) => m,
Err(_) => {
#[cfg(feature = "metrics")]
counter!("dht_messages_parse_error_total").increment(1);
return Ok(());
}
};
#[cfg(feature = "metrics")]
{
let label = match msg.y.as_str() {
"q" => "q",
"r" => "r",
"e" => "e",
_ => "unknown",
};
counter!("dht_messages_processed_total", "type" => label).increment(1);
}
match msg.y.as_str() {
"q" => {
if let Some(q_type) = &msg.q {
self.handle_query(
&msg,
q_type.as_bytes(),
remote_addr,
local_addr,
response_limiter,
response_buffer,
)
.await?;
}
}
"r" => {
if let Some(response) = msg.r
&& let Some(tid) = transaction_id_from_bytes(&msg.t)
{
if is_sample_infohashes_tid(&tid) {
self.sample_infohashes
.route_response(remote_addr, tid, response);
} else if is_peer_lookup_tid(&tid) {
self.peer_lookup.route_response(remote_addr, tid, response);
} else {
self.crawl_engine.route_response(remote_addr, tid, response);
}
}
}
_ => {}
}
Ok(())
}
async fn handle_query(
&self,
msg: &DhtMessage,
query_type: &[u8],
remote_addr: SocketAddr,
local_addr: SocketAddr,
response_limiter: &mut WorkerResponseLimiter,
response_buffer: &mut BytesMut,
) -> Result<()> {
let args = match &msg.a {
Some(a) => a,
None => return Ok(()),
};
let transaction_id = &msg.t;
let sender_id: Option<&[u8]> = args.id.as_deref().map(|v| v.as_slice());
let target_id_fallback: Option<&[u8]> = args
.target
.as_deref()
.or(args.info_hash.as_deref())
.map(|v| v.as_slice());
let q_str = std::str::from_utf8(query_type).unwrap_or("");
self.runtime_stats.inbound_query(q_str);
if let Some(sender_id) = sender_id
&& sender_id.len() == 20
&& is_valid_node_addr(&remote_addr)
{
self.crawl_engine.route_discovered(NodeTuple {
id: sender_id
.try_into()
.expect("validated DHT sender id contains 20 bytes"),
addr: remote_addr,
});
}
#[cfg(feature = "metrics")]
{
let label = match q_str {
"ping" => "ping",
"find_node" => "find_node",
"get_peers" => "get_peers",
"announce_peer" => "announce_peer",
"vote" => "vote",
_ => "other_or_invalid",
};
counter!("dht_queries_total", "q" => label).increment(1);
}
if q_str == "announce_peer" {
self.handle_announce_peer(args, remote_addr).await?;
}
self.send_response(
QueryResponse {
transaction_id,
remote_addr,
local_addr,
query_type: q_str,
sender_id,
target_id: target_id_fallback,
},
response_limiter,
response_buffer,
)
.await?;
Ok(())
}
async fn handle_announce_peer(&self, args: &DhtArgs, addr: SocketAddr) -> Result<()> {
if let Some(token) = &args.token {
if !self.validate_token(token, addr) {
self.runtime_stats.announce_invalid_token();
#[cfg(feature = "metrics")]
counter!("dht_announce_peer_blocked_total", "reason" => "invalid_token")
.increment(1);
return Ok(());
}
} else {
self.runtime_stats.announce_invalid_token();
return Ok(());
}
if let Some(info_hash) = &args.info_hash {
let info_hash_arr: [u8; 20] = match info_hash.as_ref().try_into() {
Ok(arr) => arr,
Err(_) => {
self.runtime_stats.announce_invalid_token();
return Ok(());
}
};
let hash_hex = hex::encode(info_hash_arr);
if let Some(filter) = self.hash_filter.load_full()
&& !filter(&hash_hex)
{
self.runtime_stats.announce_filtered();
#[cfg(feature = "metrics")]
counter!("dht_announce_peer_blocked_total", "reason" => "filtered").increment(1);
return Ok(());
}
#[cfg(feature = "metrics")]
counter!("dht_info_hashes_discovered_total").increment(1);
let port = if let Some(implied) = args.implied_port {
if implied != 0 {
addr.port()
} else {
args.port.unwrap_or(0)
}
} else {
args.port.unwrap_or(addr.port())
};
if port > 0 {
self.runtime_stats.announce_accepted();
self.runtime_stats.hash_received();
let event = HashDiscovered {
info_hash: hash_hex,
peer_addr: SocketAddr::new(addr.ip(), port),
source: DiscoverySource::AnnouncePeer,
discovered_at: std::time::Instant::now(),
};
let enqueue_result = self.hash_events_tx.try_send(event);
self.runtime_stats.set_hash_ingress_queue_depth(
self.hash_events_tx
.max_capacity()
.saturating_sub(self.hash_events_tx.capacity()),
);
if enqueue_result.is_err() {
self.runtime_stats.hash_ingress_dropped();
#[cfg(feature = "metrics")]
counter!("dht_metadata_ingress_dropped_total", "reason" => "queue_full")
.increment(1);
#[cfg(debug_assertions)]
log::debug!("Hash queue is full; dropping hash");
}
}
}
Ok(())
}
async fn send_response(
&self,
response: QueryResponse<'_>,
response_limiter: &mut WorkerResponseLimiter,
response_buffer: &mut BytesMut,
) -> Result<()> {
let socket = match self.sockets_by_bind_addr.get(&response.local_addr) {
Some(sock) => sock,
None => return Ok(()),
};
let reference_id = response.sender_id.or(response.target_id);
let my_id = if let Some(target) = reference_id {
let generated = neighbor_node_id(target, &self.node_id);
<[u8; 20]>::try_from(generated.as_slice()).expect("neighbor id is always 20 bytes")
} else {
self.node_id
};
let token = self.generate_token(response.remote_addr);
let include_nodes =
response.query_type == "get_peers" || response.query_type == "find_node";
let requestor_is_ipv6 = response.remote_addr.is_ipv6();
let nodes = if include_nodes {
let filter_ipv6 = match self.options.netmode {
NetMode::Ipv4Only => Some(false),
NetMode::Ipv6Only => Some(true),
NetMode::DualStack => Some(requestor_is_ipv6),
};
let snapshot = self.crawl_engine.snapshot.load();
snapshot.random_nodes(8, filter_ipv6)
} else {
Vec::new()
};
encode_response(
response_buffer,
response.transaction_id,
&my_id,
&token,
&nodes,
requestor_is_ipv6,
);
let is_priority = response.query_type == "ping" || response.query_type == "get_peers";
match response_limiter.acquire(
response.remote_addr,
response_buffer.len(),
is_priority,
Instant::now(),
) {
ResponsePermit::Regular => self.runtime_stats.response_normal(),
ResponsePermit::PriorityReserve => {
self.runtime_stats.udp_response_priority_reserved();
#[cfg(feature = "metrics")]
counter!(
"dht_udp_responses_priority_reserved_total",
"query" => if response.query_type == "ping" { "ping" } else { "get_peers" }
)
.increment(1);
}
ResponsePermit::Rejected => {
self.runtime_stats.udp_response_rate_limited();
#[cfg(feature = "metrics")]
counter!("dht_udp_responses_dropped_total", "reason" => "rate_limit").increment(1);
return Ok(());
}
}
match socket.send_to(response_buffer, response.remote_addr).await {
Ok(len) => {
self.runtime_stats.udp_sent(len);
#[cfg(feature = "metrics")]
{
counter!("dht_udp_bytes_sent_total").increment(len as u64);
counter!("dht_udp_packets_sent_total", "type" => "response").increment(1);
}
}
Err(_) => self.runtime_stats.response_send_failed(),
}
Ok(())
}
fn generate_token(&self, addr: SocketAddr) -> [u8; 8] {
let mut hasher = ahash::AHasher::default();
match addr.ip() {
IpAddr::V4(ip) => ip.octets().hash(&mut hasher),
IpAddr::V6(ip) => ip.octets().hash(&mut hasher),
}
self.token_secret.hash(&mut hasher);
hasher.finish().to_le_bytes()
}
fn validate_token(&self, token: &[u8], addr: SocketAddr) -> bool {
if token.len() != 8 {
return false;
}
let expected = self.generate_token(addr);
token == expected
}
}
+447
View File
@@ -0,0 +1,447 @@
// 负责定义 DHT 配置回调载荷和公开网络数据类型
use serde::{Deserialize, Serialize};
use std::net::SocketAddr;
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
/// IP families on which the DHT server listens and crawls.
pub enum NetMode {
/// Bind and crawl IPv4 only.
Ipv4Only,
/// Bind and crawl IPv6 only.
Ipv6Only,
#[default]
/// Bind separate IPv4 and IPv6 sockets.
DualStack,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
/// Validated torrent metadata delivered to the application callback.
pub struct TorrentInfo {
/// Lowercase hexadecimal SHA1 of the bencoded info dictionary.
pub info_hash: String,
/// Magnet URI containing the InfoHash.
pub magnet_link: String,
/// Torrent display name.
pub name: String,
/// Sum of file sizes in bytes.
pub total_size: u64,
/// Files described by the torrent.
pub files: Vec<FileInfo>,
/// Torrent piece length in bytes, or zero if absent.
pub piece_length: u64,
/// Peer addresses used to obtain the Metadata.
pub peers: Vec<String>,
/// Completion time as Unix seconds.
pub timestamp: u64,
}
/// Final outcome of a metadata fetch that passed the admission callback.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum MetadataFetchCompletionStatus {
/// Metadata was fetched and accepted by the torrent callback.
Accepted,
/// All available peer candidates failed.
FetchFailed,
/// Metadata was fetched, but the application did not accept it.
DeliveryRejected,
}
/// Report emitted exactly once after an admitted metadata fetch finishes.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct MetadataFetchCompletion {
/// InfoHash that reached a terminal state.
pub info_hash: String,
/// Final download/delivery status.
pub status: MetadataFetchCompletionStatus,
/// Real Peer network attempts; failure-cache skips are excluded.
pub attempts: usize,
}
impl MetadataFetchCompletion {
/// Returns true only for [`MetadataFetchCompletionStatus::Accepted`].
pub fn is_success(&self) -> bool {
self.status == MetadataFetchCompletionStatus::Accepted
}
}
#[derive(Debug, Clone, Serialize, Deserialize)]
/// One file entry from the validated info dictionary.
pub struct FileInfo {
/// Slash-separated relative path.
pub path: String,
/// File size in bytes.
pub size: u64,
}
#[derive(Debug, Clone, Copy, Hash, PartialEq, Eq)]
/// Compact DHT node tuple used by crawl and routing code.
pub struct NodeTuple {
/// Twenty-byte DHT node ID.
pub id: [u8; 20],
/// Public UDP endpoint.
pub addr: SocketAddr,
}
impl TorrentInfo {
/// Formats [`Self::total_size`] using binary thresholds and a short unit suffix.
pub fn format_size(&self) -> String {
format_bytes(self.total_size)
}
}
impl FileInfo {
/// Formats [`Self::size`] using binary thresholds and a short unit suffix.
pub fn format_size(&self) -> String {
format_bytes(self.size)
}
}
fn format_bytes(bytes: u64) -> String {
const UNITS: &[&str] = &["B", "KB", "MB", "GB", "TB"];
let mut size = bytes as f64;
let mut unit_index = 0;
while size >= 1024.0 && unit_index < UNITS.len() - 1 {
size /= 1024.0;
unit_index += 1;
}
format!("{size:.2} {}", UNITS[unit_index])
}
#[derive(Debug, Clone)]
/// Complete server configuration.
pub struct DHTOptions {
/// UDP listen port.
pub port: u16,
/// Enabled IP families.
pub netmode: NetMode,
/// Capacity between announce processing and the Metadata scheduler.
pub hash_queue_capacity: usize,
/// Maximum combined active find_node get_peers and sample_infohashes queries per second.
pub max_outbound_queries_per_second: u32,
/// Maximum shared outbound query budget consumed immediately after an idle period.
pub outbound_query_burst: u32,
/// Metadata download and Peer-cache limits.
pub metadata: MetadataOptions,
/// Active get_peers lookup rate and concurrency limits.
pub peer_lookup: PeerLookupOptions,
/// Active BEP-51 InfoHash sampling limits.
pub sample_infohashes: SampleInfohashesOptions,
/// Active crawl, node-pool and scheduler limits.
pub crawl: CrawlOptions,
}
#[derive(Debug, Clone)]
/// Metadata download and failure-cache limits.
pub struct MetadataOptions {
/// End-to-end timeout for one Peer attempt, in seconds.
pub timeout_secs: u64,
/// Maximum number of deduplicated pending InfoHashes.
pub max_queue_size: usize,
/// Maximum number of concurrent Metadata jobs.
pub max_worker_count: usize,
/// Maximum real TCP connection attempts started per second.
pub max_connects_per_second: u32,
/// Maximum accepted BEP-9 info dictionary size in bytes.
pub max_metadata_size_bytes: usize,
/// Maximum number of cached bad Peer socket addresses.
pub peer_failure_cache_capacity: usize,
/// Timeout/connect failure cache lifetime in seconds.
pub peer_failure_ttl_secs: u64,
}
#[derive(Debug, Clone)]
/// Active get_peers lookup budgets used to discover additional Metadata Peers.
pub struct PeerLookupOptions {
/// Maximum new InfoHash lookups started per second. Zero disables active lookup.
pub max_lookups_per_second: u32,
/// Maximum lookup budget consumed immediately after an idle period.
pub burst: u32,
/// Maximum InfoHash lookups kept active at the same time.
pub max_active_lookups: usize,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
/// Network path that supplied a Peer candidate for Metadata download.
pub enum DiscoverySource {
/// A remote node announced itself as a Peer.
AnnouncePeer,
/// A Peer was found for a Hash sampled directly from a newly discovered node.
SampleDirect,
/// A Peer was found for a Hash sampled from the responsive-node snapshot.
SampleSnapshot,
/// A Peer was found by the Metadata scheduler's delayed active lookup.
ActiveLookup,
}
#[derive(Debug, Clone)]
/// InfoHash and announcing Peer submitted to the Metadata scheduler.
pub struct HashDiscovered {
/// Lowercase hexadecimal InfoHash.
pub info_hash: String,
/// Peer endpoint derived from announce `port`/`implied_port`.
pub peer_addr: SocketAddr,
/// Network path that supplied this Peer candidate.
pub source: DiscoverySource,
/// Monotonic discovery time used for freshness and queue ordering.
pub discovered_at: std::time::Instant,
}
#[derive(Debug, Clone, PartialEq, Eq)]
/// Terminal result of one bounded active `get_peers` lookup.
pub struct PeerLookupResult {
/// Unique Peer socket addresses returned by the DHT.
pub peers: Vec<SocketAddr>,
/// UDP queries sent for this lookup.
pub queries: usize,
}
#[derive(Debug, Clone)]
/// Active BEP-51 `sample_infohashes` discovery limits.
pub struct SampleInfohashesOptions {
/// Maximum BEP-51 queries started per second. Zero disables sampling.
pub max_queries_per_second: u32,
/// Maximum sampling budget consumed immediately after an idle period.
pub burst: u32,
/// Maximum outstanding BEP-51 requests.
pub max_in_flight: usize,
/// Percentage of newly discovered node addresses routed directly to BEP-51 sampling.
pub new_node_sample_percent: u8,
/// Capacity of the bounded newly discovered node sampling lane.
pub candidate_queue_capacity: usize,
/// Whether a sampled Hash should fall back to iterative get_peers after the sampling node
/// returns no Peer.
pub fallback_to_iterative: bool,
/// Per-request timeout in milliseconds.
pub request_timeout_millis: u64,
/// Retry delay for timeouts or nodes that do not return samples, in seconds.
pub unsupported_backoff_secs: u64,
/// Maximum sampled InfoHashes retained for bounded in-memory deduplication.
pub dedup_capacity: usize,
}
#[derive(Debug, Clone, Default)]
/// Active crawl configuration grouped by responsibility.
pub struct CrawlOptions {
/// FIFO node-pool and responsive-ring limits.
pub pool: PoolOptions,
/// Query, replacement and response budgets.
pub rate_limit: RateLimitOptions,
/// Bootstrap sources and retry policy.
pub bootstrap: BootstrapOptions,
/// Target-generation policy.
pub target: TargetOptions,
/// Internal bounded-channel and snapshot limits.
pub scheduler: SchedulerOptions,
}
#[derive(Debug, Clone)]
/// Independent crawl and UDP-response budgets.
pub struct RateLimitOptions {
/// Maximum active find_node queries scheduled per second.
pub max_find_node_rate_per_sec: u32,
/// Maximum query budget consumed in one scheduler tick.
pub burst: u32,
/// Maximum total pending find_node transactions.
pub max_in_flight: usize,
/// Pending find_node timeout in seconds.
pub request_timeout_secs: u64,
/// Maximum never-before-probed destinations per minute.
pub max_new_destinations_per_minute: u32,
/// Maximum outbound DHT response packets per second.
pub max_response_rate_per_sec: u32,
/// Maximum encoded outbound DHT response bytes per second.
pub max_response_bytes_per_sec: u64,
/// Maximum response packets per source address per second.
pub max_response_rate_per_source: u32,
/// Remaining query-rate percentage when Metadata pressure reaches 95%.
pub metadata_pressure_floor_percent: u8,
/// Maximum FIFO replacements per minute after the pool has warmed.
pub max_replacements_per_minute: u32,
/// Maximum pending find_node transactions per IP subnet.
pub max_in_flight_per_subnet: usize,
}
#[derive(Debug, Clone)]
/// FIFO crawl-pool and responsive-node reservoir limits.
pub struct PoolOptions {
/// Maximum queued crawl nodes.
pub capacity: usize,
/// How long a probed endpoint is blocked from readmission, in seconds.
pub recent_probe_ttl_secs: u64,
/// Maximum nodes retained for replies and revisit traffic.
pub responsive_capacity: usize,
/// Responsive-node lifetime in seconds.
pub responsive_ttl_secs: u64,
/// Pool size below which bootstrap is considered.
pub low_watermark: usize,
}
#[derive(Debug, Clone)]
/// Bootstrap hostnames and retry timing.
pub struct BootstrapOptions {
/// Host:port sources resolved when bootstrap is needed.
pub nodes: Vec<String>,
/// Minimum interval between bootstrap rounds, in seconds.
pub interval_secs: u64,
/// Maximum resolved endpoints selected in one round.
pub max_nodes_per_round: usize,
/// Initial failed-source backoff in seconds.
pub source_backoff_base_secs: u64,
/// Maximum failed-source backoff in seconds.
pub source_backoff_max_secs: u64,
}
#[derive(Debug, Clone)]
/// Distribution used to generate find_node targets and sender IDs.
pub struct TargetOptions {
/// Percentage of targets that are fully random.
pub random_walk_percent: u8,
/// Percentage of targets chosen from sparse routing buckets.
pub sparse_bucket_percent: u8,
/// Whether outbound sender IDs borrow the target's prefix.
pub neighbor_sender_id: bool,
}
#[derive(Debug, Clone)]
/// Capacities and batch limits for the crawl actor.
pub struct SchedulerOptions {
/// Capacity for response/bootstrap priority events.
pub priority_event_channel_capacity: usize,
/// Capacity for newly discovered node events.
pub discovery_event_channel_capacity: usize,
/// Maximum events drained per actor iteration.
pub event_batch_limit: usize,
/// Maximum discovery nodes drained per actor iteration.
pub node_batch_limit: usize,
/// Maximum responsive nodes published in the lock-free snapshot.
pub routing_snapshot_size: usize,
/// Snapshot publication interval in milliseconds.
pub snapshot_refresh_millis: u64,
}
impl Default for DHTOptions {
fn default() -> Self {
Self {
port: 6881,
netmode: NetMode::Ipv4Only,
hash_queue_capacity: 10_000,
max_outbound_queries_per_second: 10,
outbound_query_burst: 2,
metadata: MetadataOptions::default(),
peer_lookup: PeerLookupOptions::default(),
sample_infohashes: SampleInfohashesOptions::default(),
crawl: CrawlOptions::default(),
}
}
}
impl Default for MetadataOptions {
fn default() -> Self {
Self {
timeout_secs: 4,
max_queue_size: 10_000,
max_worker_count: 8,
max_connects_per_second: 2,
max_metadata_size_bytes: 10 * 1024 * 1024,
peer_failure_cache_capacity: 200_000,
peer_failure_ttl_secs: 60,
}
}
}
impl Default for PeerLookupOptions {
fn default() -> Self {
Self {
max_lookups_per_second: 1,
burst: 1,
max_active_lookups: 4,
}
}
}
impl Default for SampleInfohashesOptions {
fn default() -> Self {
Self {
max_queries_per_second: 1,
burst: 1,
max_in_flight: 4,
new_node_sample_percent: 0,
candidate_queue_capacity: 4_096,
fallback_to_iterative: true,
request_timeout_millis: 1_500,
unsupported_backoff_secs: 300,
dedup_capacity: 1_000_000,
}
}
}
impl Default for RateLimitOptions {
fn default() -> Self {
Self {
max_find_node_rate_per_sec: 6,
burst: 2,
max_in_flight: 12,
request_timeout_secs: 2,
max_new_destinations_per_minute: 60,
max_response_rate_per_sec: 500,
max_response_bytes_per_sec: 1024 * 1024,
max_response_rate_per_source: 40,
metadata_pressure_floor_percent: 25,
max_replacements_per_minute: 25_000,
max_in_flight_per_subnet: 8,
}
}
}
impl Default for PoolOptions {
fn default() -> Self {
Self {
capacity: 100_000,
recent_probe_ttl_secs: 600,
responsive_capacity: 16_384,
responsive_ttl_secs: 900,
low_watermark: 10_000,
}
}
}
impl Default for BootstrapOptions {
fn default() -> Self {
Self {
nodes: vec![
"router.bittorrent.com:6881".to_string(),
"dht.transmissionbt.com:6881".to_string(),
"router.utorrent.com:6881".to_string(),
"dht.aelitis.com:6881".to_string(),
],
interval_secs: 30,
max_nodes_per_round: 16,
source_backoff_base_secs: 60,
source_backoff_max_secs: 3_600,
}
}
}
impl Default for TargetOptions {
fn default() -> Self {
Self {
random_walk_percent: 70,
sparse_bucket_percent: 30,
neighbor_sender_id: true,
}
}
}
impl Default for SchedulerOptions {
fn default() -> Self {
Self {
priority_event_channel_capacity: 8_192,
discovery_event_channel_capacity: 16_384,
event_batch_limit: 256,
node_batch_limit: 4_096,
routing_snapshot_size: 4_096,
snapshot_refresh_millis: 1_000,
}
}
}
+81
View File
@@ -0,0 +1,81 @@
// 负责复用有界 UDP 接收缓冲区并控制内存占用
//! UDP 收包缓冲区池:避免每包 `to_owned()` 拷贝。
//!
//! 单线程 listener 从池中取出固定大小缓冲区,`recv_from` 直接写入;
//! 通过 channel 将缓冲区所有权交给 worker,处理完毕后归还池中复用。
use crossbeam_queue::ArrayQueue;
use std::sync::Arc;
/// 与 `process_udp_packet` 中丢弃阈值一致
pub const MAX_DHT_UDP_PACKET: usize = 8192;
/// 预分配缓冲区数量(约等于高峰在途包数)
const INITIAL_POOL_SIZE: usize = 512;
/// 池上限,防止极端背压下无限增长
const MAX_POOL_SIZE: usize = 4096;
/// 在途 UDP 包:固定容量缓冲区 + 有效长度
pub struct UdpPacket {
pub buf: Box<[u8]>,
pub len: usize,
}
impl UdpPacket {
#[inline]
pub fn payload(&self) -> &[u8] {
&self.buf[..self.len]
}
}
/// 固定 8KiB 缓冲区的对象池(`recv_from` 零拷贝移交 worker
#[derive(Clone)]
pub struct UdpBufferPool {
inner: Arc<PoolInner>,
}
struct PoolInner {
free: ArrayQueue<Box<[u8]>>,
buf_capacity: usize,
}
impl UdpBufferPool {
pub fn new() -> Self {
let free = ArrayQueue::new(MAX_POOL_SIZE);
for _ in 0..INITIAL_POOL_SIZE {
let _ = free.push(alloc_buffer(MAX_DHT_UDP_PACKET));
}
Self {
inner: Arc::new(PoolInner {
free,
buf_capacity: MAX_DHT_UDP_PACKET,
}),
}
}
/// 取一块缓冲区;池空时分配新块(背压或突发流量)
pub fn acquire(&self) -> Box<[u8]> {
self.inner
.free
.pop()
.unwrap_or_else(|| alloc_buffer(self.inner.buf_capacity))
}
/// 归还缓冲区;池满时直接丢弃,由 GC 回收
pub fn release(&self, buf: Box<[u8]>) {
if buf.len() != self.inner.buf_capacity {
return;
}
let _ = self.inner.free.push(buf);
}
pub fn buf_capacity(&self) -> usize {
self.inner.buf_capacity
}
}
fn alloc_buffer(capacity: usize) -> Box<[u8]> {
let v = vec![0; capacity];
v.into_boxed_slice()
}
+347
View File
@@ -0,0 +1,347 @@
// 负责将 UDP 数据包分发到有界工作队列并处理过载
use crate::error::DHTError;
use crate::runtime_stats::DhtRuntimeStats;
use crate::udp_buffer::{MAX_DHT_UDP_PACKET, UdpBufferPool, UdpPacket};
#[cfg(feature = "metrics")]
use metrics::counter;
use std::hash::{Hash, Hasher};
use std::net::SocketAddr;
use std::sync::Arc;
use std::time::Duration;
use tokio::net::UdpSocket;
use tokio::sync::mpsc;
use tokio_util::sync::CancellationToken;
pub(crate) type WorkerHandle = mpsc::Sender<(UdpPacket, SocketAddr, SocketAddr)>;
pub(crate) fn spawn_udp_listener(
socket: Arc<UdpSocket>,
mut workers: Vec<WorkerHandle>,
shutdown: CancellationToken,
buffer_pool: UdpBufferPool,
runtime_stats: DhtRuntimeStats,
) -> crate::error::Result<()> {
let local_addr = socket
.local_addr()
.map_err(|e| DHTError::Init(format!("socket local addr failed: {e}")))?;
if workers.is_empty() {
return Err(DHTError::Init(
"spawn_udp_listener: no worker provided".to_string(),
));
}
tokio::spawn(async move {
loop {
let mut buf = buffer_pool.acquire();
let recv_buf = &mut buf[..buffer_pool.buf_capacity()];
tokio::select! {
_ = shutdown.cancelled() => {
buffer_pool.release(buf);
break;
}
result = socket.recv_from(recv_buf) => {
match result {
Ok((size, origin_addr)) => {
if let Err(ProcessUdpPacketError::NoLiveWorkers) =
process_udp_packet(buf, size, origin_addr, local_addr, &buffer_pool, &runtime_stats, &mut workers)
{
log::warn!("Socket {socket:?} is closing because no worker can process packets.");
break
}
}
Err(_) => {
buffer_pool.release(buf);
tokio::select! {
_ = shutdown.cancelled() => break,
_ = tokio::time::sleep(Duration::from_millis(1)) => {},
}
}
}
}
}
}
});
Ok(())
}
enum ProcessUdpPacketError {
PacketTooLarge,
InvalidPacket,
ChokedWorkers,
NoLiveWorkers,
}
fn process_udp_packet(
buf: Box<[u8]>,
size: usize,
origin_addr: SocketAddr,
local_addr: SocketAddr,
buffer_pool: &UdpBufferPool,
runtime_stats: &DhtRuntimeStats,
workers: &mut Vec<WorkerHandle>,
) -> std::result::Result<(), ProcessUdpPacketError> {
runtime_stats.udp_received();
runtime_stats.udp_received_bytes(size);
#[cfg(feature = "metrics")]
counter!("dht_udp_bytes_received_total").increment(size as u64);
if size > MAX_DHT_UDP_PACKET {
runtime_stats.udp_invalid();
#[cfg(feature = "metrics")]
counter!("dht_udp_packets_received_total", "status" => "dropped_size").increment(1);
buffer_pool.release(buf);
return Err(ProcessUdpPacketError::PacketTooLarge);
}
if size == 0 || buf[0] != b'd' {
runtime_stats.udp_invalid();
#[cfg(feature = "metrics")]
counter!("dht_udp_packets_received_total", "status" => "dropped_magic").increment(1);
buffer_pool.release(buf);
return Err(ProcessUdpPacketError::InvalidPacket);
}
let mut packet = UdpPacket { buf, len: size };
let mut hasher = ahash::AHasher::default();
origin_addr.hash(&mut hasher);
let origin_hash = hasher.finish() as usize;
'select_worker: loop {
if workers.is_empty() {
buffer_pool.release(packet.buf);
return Err(ProcessUdpPacketError::NoLiveWorkers);
}
let worker_count = workers.len();
let preferred_index = origin_hash % worker_count;
for offset in 0..worker_count {
let worker_index = (preferred_index + offset) % worker_count;
match workers[worker_index].try_send((packet, origin_addr, local_addr)) {
Ok(_) => {
#[cfg(feature = "metrics")]
counter!("dht_udp_packets_received_total", "status" => "ok").increment(1);
return Ok(());
}
Err(mpsc::error::TrySendError::Full((p, _, _))) => {
packet = p;
}
Err(mpsc::error::TrySendError::Closed((p, _, _))) => {
packet = p;
log::warn!("UDP worker dropped.");
workers.swap_remove(worker_index);
continue 'select_worker;
}
}
}
#[cfg(feature = "metrics")]
counter!("dht_udp_packets_received_total", "status" => "queue_full").increment(1);
runtime_stats.udp_queue_full();
buffer_pool.release(packet.buf);
return Err(ProcessUdpPacketError::ChokedWorkers);
}
}
#[cfg(test)]
mod tests {
use super::*;
fn addresses() -> (SocketAddr, SocketAddr) {
(
"8.8.8.8:6881".parse().unwrap(),
"0.0.0.0:12313".parse().unwrap(),
)
}
fn buffer(pool: &UdpBufferPool, first: u8) -> Box<[u8]> {
let mut buf = pool.acquire();
buf[0] = first;
buf
}
fn packet(pool: &UdpBufferPool, first: u8) -> UdpPacket {
UdpPacket {
buf: buffer(pool, first),
len: 1,
}
}
fn preferred_index(origin_addr: SocketAddr, worker_count: usize) -> usize {
let mut hasher = ahash::AHasher::default();
origin_addr.hash(&mut hasher);
(hasher.finish() as usize) % worker_count
}
#[test]
fn available_preferred_worker_is_used_first() {
let pool = UdpBufferPool::new();
let stats = DhtRuntimeStats::default();
let (origin_addr, local_addr) = addresses();
let (tx0, mut rx0) = mpsc::channel(1);
let (tx1, mut rx1) = mpsc::channel(1);
let mut workers = vec![tx0, tx1];
let preferred = preferred_index(origin_addr, workers.len());
assert!(
process_udp_packet(
buffer(&pool, b'd'),
1,
origin_addr,
local_addr,
&pool,
&stats,
&mut workers,
)
.is_ok()
);
let (preferred_rx, fallback_rx) = if preferred == 0 {
(&mut rx0, &mut rx1)
} else {
(&mut rx1, &mut rx0)
};
let (forwarded, _, _) = preferred_rx.try_recv().unwrap();
assert_eq!(forwarded.payload(), b"d");
assert!(matches!(
fallback_rx.try_recv(),
Err(mpsc::error::TryRecvError::Empty)
));
pool.release(forwarded.buf);
}
#[test]
fn full_preferred_worker_falls_back_to_available_worker() {
let pool = UdpBufferPool::new();
let stats = DhtRuntimeStats::default();
let (origin_addr, local_addr) = addresses();
let (tx0, mut rx0) = mpsc::channel(1);
let (tx1, mut rx1) = mpsc::channel(1);
let mut workers = vec![tx0, tx1];
let preferred = preferred_index(origin_addr, workers.len());
workers[preferred]
.try_send((packet(&pool, b'x'), origin_addr, local_addr))
.unwrap();
assert!(
process_udp_packet(
buffer(&pool, b'd'),
1,
origin_addr,
local_addr,
&pool,
&stats,
&mut workers,
)
.is_ok()
);
let (preferred_rx, fallback_rx) = if preferred == 0 {
(&mut rx0, &mut rx1)
} else {
(&mut rx1, &mut rx0)
};
let (queued, _, _) = preferred_rx.try_recv().unwrap();
let (forwarded, _, _) = fallback_rx.try_recv().unwrap();
assert_eq!(queued.payload(), b"x");
assert_eq!(forwarded.payload(), b"d");
pool.release(queued.buf);
pool.release(forwarded.buf);
}
#[test]
fn closed_preferred_worker_is_removed_before_fallback() {
let pool = UdpBufferPool::new();
let stats = DhtRuntimeStats::default();
let (origin_addr, local_addr) = addresses();
let (closed_tx, closed_rx) = mpsc::channel(1);
drop(closed_rx);
let (open_tx, mut open_rx) = mpsc::channel(1);
let preferred = preferred_index(origin_addr, 2);
let mut workers = if preferred == 0 {
vec![closed_tx, open_tx]
} else {
vec![open_tx, closed_tx]
};
assert!(
process_udp_packet(
buffer(&pool, b'd'),
1,
origin_addr,
local_addr,
&pool,
&stats,
&mut workers,
)
.is_ok()
);
assert_eq!(workers.len(), 1);
let (forwarded, _, _) = open_rx.try_recv().unwrap();
assert_eq!(forwarded.payload(), b"d");
pool.release(forwarded.buf);
}
#[test]
fn packet_is_dropped_only_after_all_live_workers_are_full() {
let pool = UdpBufferPool::new();
let stats = DhtRuntimeStats::default();
let (origin_addr, local_addr) = addresses();
let (tx0, mut rx0) = mpsc::channel(1);
let (tx1, mut rx1) = mpsc::channel(1);
let mut workers = vec![tx0, tx1];
for worker in &workers {
worker
.try_send((packet(&pool, b'x'), origin_addr, local_addr))
.unwrap();
}
let result = process_udp_packet(
buffer(&pool, b'd'),
1,
origin_addr,
local_addr,
&pool,
&stats,
&mut workers,
);
assert!(matches!(result, Err(ProcessUdpPacketError::ChokedWorkers)));
let snapshot = stats.snapshot();
assert_eq!(snapshot.udp_received, 1);
assert_eq!(snapshot.udp_queue_full, 1);
assert_eq!(snapshot.udp_invalid, 0);
assert_eq!(workers.len(), 2);
for receiver in [&mut rx0, &mut rx1] {
let (queued, _, _) = receiver.try_recv().unwrap();
assert_eq!(queued.payload(), b"x");
pool.release(queued.buf);
}
}
#[test]
fn invalid_packet_updates_runtime_stats() {
let pool = UdpBufferPool::new();
let stats = DhtRuntimeStats::default();
let (origin_addr, local_addr) = addresses();
let mut workers = Vec::new();
let result = process_udp_packet(
buffer(&pool, b'x'),
1,
origin_addr,
local_addr,
&pool,
&stats,
&mut workers,
);
assert!(matches!(result, Err(ProcessUdpPacketError::InvalidPacket)));
let snapshot = stats.snapshot();
assert_eq!(snapshot.udp_received, 1);
assert_eq!(snapshot.udp_invalid, 1);
assert_eq!(snapshot.udp_queue_full, 0);
}
}
+59
View File
@@ -0,0 +1,59 @@
# 定义 DHT 元数据搜索应用的独立依赖和构建入口
[package]
name = "dht-search"
version = "0.1.0"
edition.workspace = true
authors.workspace = true
license.workspace = true
repository.workspace = true
publish = false
[features]
default = ["rocksdb-storage"]
rocksdb-storage = ["dep:rocksdb", "dep:rusqlite"]
[dependencies]
axum = "0.8.9"
blake3 = "1.8.5"
clap = { version = "4.5", features = ["derive"] }
dht-crawler = { path = "../crawler", features = ["metrics"] }
dunce = "1.0"
hex = "0.4"
fs2 = "0.4"
rocksdb = { version = "0.24.0", default-features = false, features = ["bindgen-runtime", "lz4"], optional = true }
rmp-serde = "1.3"
regex = "1.12"
rusqlite = { version = "0.40.2", features = ["bundled"], optional = true }
serde.workspace = true
serde_json = "1.0"
tantivy = "0.26.1"
thiserror.workspace = true
tokio = { workspace = true, features = ["macros", "rt-multi-thread", "signal", "sync", "time"] }
tokio-util.workspace = true
toml = "0.9"
tracing.workspace = true
tracing-appender = "0.2"
tracing-subscriber = { workspace = true, features = ["env-filter", "fmt"] }
tower-http = { version = "0.6", features = ["fs"] }
unicode-normalization = "0.1"
[dev-dependencies]
tempfile = "3.27"
tower = { version = "0.5", features = ["util"] }
[target.'cfg(windows)'.dependencies]
windows-sys = { version = "0.61.2", features = ["Win32_Foundation", "Win32_Storage_FileSystem", "Win32_System_Diagnostics_ToolHelp", "Win32_System_ProcessStatus", "Win32_System_Threading"] }
[target.'cfg(unix)'.dependencies]
libc = "0.2"
[[bin]]
name = "dht-search"
path = "src/main.rs"
required-features = ["rocksdb-storage"]
[[bin]]
name = "dht-benchmark"
path = "src/bin/dht-benchmark.rs"
required-features = ["rocksdb-storage"]
+340
View File
@@ -0,0 +1,340 @@
# dht-search
`dht-search` 是集 DHT Metadata 采集 RocksDB 持久化 Tantivy 搜索索引和 HTTP API 于一体的应用
## 构建准备
RocksDB 包含 C++ 代码并在构建时使用 bindgen 因此需要 libclang
Windows 可以把 libclang 安装到工作区本地目录
```powershell
python -m pip install --target .tools\libclang libclang
$env:LIBCLANG_PATH = "$PWD\.tools\libclang\clang\native"
cargo build -p dht-search --release
```
`.tools` 只用于本地构建不会部署到运行设备
## 配置
复制根目录的配置模板
```powershell
Copy-Item dht-search.example.toml dht-search.toml
```
相对 `data_dir` 以配置文件所在目录为基准解析
`content_filter_file` 指向独立的无效文件过滤配置 相对路径同样以主配置文件所在目录为基准解析 推荐直接使用根目录的 `content-filters.toml`
也可以通过命令行覆盖数据目录和本次运行时长
```powershell
cargo run -p dht-search -- --data-dir D:\data\dht-search --run-duration-secs 3600
```
不设置 `run-duration-secs` 时服务持续运行直到收到 Ctrl+C SIGINT 或 SIGTERM
生产 Web 页面需要先在 `src/web` 目录执行 `bun run build` Axum 会从 `http.web_dir` 提供构建结果
### 当前运行模板网络配置
`dht-search.example.toml` 默认使用经过本机一分钟资源测试的 Bitmagnet 等效激进配置 主动 DHT 查询仍共享 `max_outbound_queries_per_second` 总预算 所有队列保持有界 可以按设备和网络条件主动下调
| 配置项 | 运行模板值 | 作用 |
|---|---:|---|
| `max_outbound_queries_per_second` | `1000` | 三类主动 DHT UDP 查询的合计每秒速率硬上限 |
| `outbound_query_burst` | `200` | 空闲后允许立即消费的 UDP 查询数 |
| `find_node_queries_per_second` | `10` | `find_node` 自身速率上限 |
| `find_node_max_in_flight` | `100` | 同时等待响应的 `find_node` 数量 |
| `new_destinations_per_minute` | `12000` | 每分钟首次探测的新 UDP 目标数量 |
| `peer_lookups_per_second` | `200` | 每秒启动的 infohash Peer 查找数量 |
| `peer_lookup_max_active` | `200` | 同时运行的 Peer 查找数量 |
| `sample_queries_per_second` | `60` | BEP-51 采样查询速率 |
| `sample_max_in_flight` | `100` | 同时等待响应的 BEP-51 采样请求数量 |
| `sample_new_node_percent` | `50` | 按节点地址稳定分流到直接采样通道的比例 |
| `sample_candidate_queue_capacity` | `8192` | 新发现节点直接采样通道的有界容量 |
| `sample_fallback_to_iterative` | `false` | 采样节点没有返回 Peer 时快速结束以扩大覆盖面 |
| `metadata_workers` | `400` | 同时处理的 Metadata 任务数量 |
| `metadata_connects_per_second` | `400` | 每秒真正开始的 Peer TCP 连接硬上限 |
Metadata 下载和可用性握手共用 `metadata_connects_per_second` 预算不会各自叠加
### 按需可用性验证
搜索结果和详情访问只会把已过冷却期的种子异步加入持久化验证队列 HTTP 响应不会等待 DHT 或 Peer 网络
| 配置项 | 默认值 | 作用 |
|---|---:|---|
| `verification.enabled` | `true` | 是否启用按需可用性验证 |
| `verification.queue_capacity` | `10000` | 持久化验证队列容量 |
| `verification.max_active` | `8` | 同时验证的种子数量 |
| `verification.max_peer_attempts` | `3` | 每个种子最多握手的 Peer 数量 |
| `verification.lease_secs` | `60` | 异常退出后验证任务重新可领取的租约时间 |
| `verification.poll_interval_millis` | `250` | 持久化队列轮询间隔 |
详情访问使用高优先级 搜索结果使用普通优先级 队列满时高优先级可以替换最旧普通任务
可用性分为 `unknown` `active``possibly_stale` 一次或多次验证失败只表示当前可能没有可连接 Peer 不会删除种子
新抓取记录会把成功下载 Metadata 的来源 Peer 视为一次有效验证 旧记录按需复查时会同时使用新 DHT 结果和已保存的成功来源 Peer
热度是近期 DHT 发现强度最近出现时间和可连接 Peer 数的综合活跃度分数 不代表全球下载量
这组参数仍保留全局速率限制 如果同机代理或路由器再次出现不稳定应优先降低总 UDP 预算和 Metadata 建连速率
Windows 下索引每五秒批量提交 临时文件占用会自动指数退避重试且不会停止采集 HTTP 服务或丢失 RocksDB 待索引状态
### 磁盘空间保护
应用默认每十秒检查 `data_dir` 所在磁盘的剩余空间 低于保护阈值时先停止接收新 Metadata DHT 状态更新索引任务和可用性验证任务 已经进入有界持久化队列的记录会继续排空 随后进入只读保护
只读保护期间 RocksDB 和 Tantivy 不再产生业务写入 现有搜索详情健康检查和 Web 页面仍然可用 剩余空间达到独立恢复阈值后自动恢复采集 使用两个阈值可以避免临界空间附近反复暂停和恢复
| 配置项 | 默认值 | 作用 |
|---|---:|---|
| `disk_guard.enabled` | `true` | 是否启用磁盘空间保护 |
| `disk_guard.check_interval_secs` | `10` | 剩余空间检查间隔 |
| `disk_guard.minimum_free_bytes` | `5368709120` | 低于 5 GiB 时停止接收新任务 |
| `disk_guard.resume_free_bytes` | `6442450944` | 恢复到 6 GiB 时重新接受写入 |
磁盘空间探测失败时采用保守策略进入保护状态 `/stats` 返回 `disk_state` `disk_available_bytes` 阈值 活跃写入数 探测失败数 状态转换数和拒绝任务数 Web 运行状态使用绿色或黄色状态点展示正常与保护状态
### 日志轮转和保留
应用会在读取配置后初始化日志 默认只写入 `data/logs` 的滚动文件而不重复输出到终端 因此用脚本或后台进程启动时不需要再把标准错误重定向到长期增长的日志文件
| 配置项 | 默认值 | 作用 |
|---|---:|---|
| `logging.directory` | `data/logs` | 日志文件目录 相对主配置文件解析 |
| `logging.file_enabled` | `true` | 启用滚动文件日志 |
| `logging.console_enabled` | `false` | 同时输出到当前终端 |
| `logging.rotation` | `daily` | 轮转周期 支持 `minutely` `hourly` `daily``never` |
| `logging.retain_files` | `7` | 最多保留的匹配日志文件数量 |
| `logging.file_prefix` | `dht-search` | 日志文件名前缀 |
默认按天轮转时保留 7 个文件约等于保留最近 7 天 日志组件只清理同目录中同时匹配前缀和 `.log` 后缀的普通文件 不删除目录和符号链接 清理失败会输出错误但不会让服务退出
开发时需要直接观察终端日志可以设置 `console_enabled = true` 文件日志和终端日志不能同时关闭
### RocksDB 检查点备份和恢复
RocksDB 是唯一权威数据源 应用使用 RocksDB 原生 Checkpoint API 在线生成一致快照 备份期间采集可以继续运行 Tantivy 不进入备份因为它可以从 RocksDB 完整重建
| 配置项 | 默认值 | 作用 |
|---|---:|---|
| `backup.enabled` | `true` | 是否启用自动检查点 |
| `backup.directory` | `data/backups` | 检查点目录 相对主配置文件解析 |
| `backup.interval_secs` | `21600` | 每 6 小时创建一次检查点 |
| `backup.retain_checkpoints` | `3` | 保留最近 3 个自动检查点 |
| `backup.create_on_start` | `true` | 每次启动后立即创建一次检查点 |
备份目录与数据目录位于同一磁盘时 RocksDB 会尽量通过硬链接减少复制开销 放到其他磁盘时可能复制全部数据库文件 创建前会检查备份磁盘剩余空间 磁盘保护期间自动跳过而不会阻塞服务
`/stats` 返回检查点成功失败跳过清理数量 最近成功时间耗时和记录数量 自动清理只处理名称严格匹配 `checkpoint-` 加二十位时间戳的直接子目录 不会删除手工目录文件或符号链接
恢复必须在服务停止后执行 数据目录独占锁会阻止运行中的服务和恢复命令同时操作
```powershell
target\release\dht-search.exe `
--config dht-search.example.toml `
--restore-checkpoint data\backups\checkpoint-00000001775400000000
```
恢复命令会先以只读方式校验源检查点 再复制到数据目录内的暂存目录并二次校验 然后切换 `rocksdb` 目录 原数据库不会删除而是保留为 `rocksdb.pre-restore-*` 方便人工回滚 Tantivy 作为派生数据会被删除 下次正常启动自动从恢复后的 RocksDB 重建
确认恢复数据无误后可以人工删除 `rocksdb.pre-restore-*` 释放空间 不要在服务运行时移动或删除这些目录
### 运行诊断历史
应用把可删除的运行资源采样写入独立的 `data/diagnostics.sqlite3` RocksDB 仍然是唯一业务权威数据源 删除诊断数据库不会影响种子数据搜索索引或恢复
| 配置项 | 默认值 | 作用 |
|---|---:|---|
| `diagnostics.enabled` | `true` | 是否采集并保存运行诊断历史 |
| `diagnostics.database` | `data/diagnostics.sqlite3` | 独立 SQLite 数据库路径 |
| `diagnostics.sample_interval_secs` | `10` | 实时资源采样间隔 |
| `diagnostics.raw_retention_hours` | `24` | 原始采样保留时间 |
| `diagnostics.minute_retention_days` | `30` | 每分钟快照保留时间 |
| `diagnostics.queue_capacity` | `128` | SQLite writer 有界队列容量 |
SQLite 使用 WAL 和单独 writer 线程 原始采样超过 24 小时自动删除 同一分钟只保留最新快照且超过 30 天自动删除 写入失败不会停止采集和搜索核心服务
诊断快照包含进程内存 CPU 时间线程句柄 DHT 流量队列深度 RocksDB Block Cache MemTable Compaction 和 SST 状态 Tantivy Writer 预算和提交耗时以及 HTTP 并发错误分类平均 P95 和最大延迟
### 配置管理
配置文件只是强类型配置 DTO 的 TOML 持久化形式 服务通过 `GET /config` 返回当前文件配置和稳定修订号 通过 `PUT /config` 接受完整 DTO
更新前会执行与启动时相同的完整校验 保存时先在同目录写入并同步临时文件再原子替换目标文件 旧修订号返回 `409 Conflict` 防止多个页面互相覆盖
当前版本不在线修改正在运行的 DHT 存储索引和监听器 保存成功后返回 `restart_required = true` 重启服务后统一生效 命令行覆盖字段也会单独返回并继续优先于文件配置
通过 Web 保存会按 DTO 重新生成 TOML 原有手写注释不会保留 管理接口默认随 HTTP 服务提供 因此生产部署不应把 `/config` 暴露到不受信任的公网入口
### Metadata 安全限制
应用会在 Metadata 下载和进入 RocksDB 前执行两层资源与结构校验
| 配置项 | 默认值 | 作用 |
|---|---:|---|
| `metadata_limits.max_metadata_bytes` | `10485760` | 下载阶段允许的最大 info 字典字节数 |
| `metadata_limits.max_files` | `20000` | 单个种子允许的最大文件数量 |
| `metadata_limits.max_name_bytes` | `1024` | 种子名称最大 UTF-8 字节数 |
| `metadata_limits.max_path_bytes` | `4096` | 单个文件路径最大 UTF-8 字节数 |
| `metadata_limits.max_path_depth` | `64` | 单个文件路径最大目录层级 |
空名称 空文件列表 控制字符 空路径段 `.` `..` 大小溢出和声明总大小不一致会被分类拒绝
通过完整 Metadata 校验后被拒绝的 infohash 只在 RocksDB 保存原因规则指纹时间和次数 不保存名称或文件列表 相同规则下再次发现时不会重复下载 修改限制后规则指纹变化并允许重新判断
`/stats` 返回 `metadata_filtered` 总数以及 `metadata_filtered_*` 分类计数 Web 运行状态展示本次运行的过滤总数
可以使用独立数据目录和严格限制运行五分钟测试 不会污染正式数据目录
```powershell
cargo run -p dht-search -- --config dht-search.filter-test.toml
Invoke-RestMethod http://127.0.0.1:8080/stats | ConvertTo-Json -Depth 5
```
严格测试配置仅用于观察过滤效果 不应作为正式采集配置
### 无效文件隐藏规则
`content-filters.toml` 控制哪些文件不参与详情展示 搜索 文件数量 有效大小和内容聚合 默认规则会隐藏 BitComet `_____padding_file_` 文件以及 `.pad``.____padding_file` 填充目录
RocksDB 始终保存完整原始 Metadata 隐藏规则不会删除文件或种子 修改或回滚规则后应用会根据规则指纹重新计算内容组并从 RocksDB 重建 Tantivy
每条规则包含稳定 `id` 开关 匹配字段 匹配方式 值 大小写选项和可读原因 当前字段支持 `file-name``file-path` 匹配方式支持 `exact` `prefix` `suffix` `contains` `wildcard``regex` 动作只允许安全的 `hide`
通配符中 `*` 表示任意长度字符 `?` 表示一个字符并匹配完整字段 正则表达式使用 Rust `regex` 语法 文件路径在匹配前统一使用 `/` 分隔符
如果一个 Metadata 的全部文件都被隐藏 原始记录仍保留在 RocksDB 但不会进入搜索索引或公开详情
### 采样去重和 Peer 查找
BEP-51 返回的 infohash 会先进入有界批量准入队列并由 RocksDB 精确判断
新发现节点按地址稳定分流 同一地址只进入直接采样或 `find_node` 通道 直接采样队列满时会安全回退到抓取池 `/stats` 会分别报告候选队列直接采样请求响应 Hash 去重丢弃以及成功 Metadata 的网络来源
已有 infohash 只更新最后发现时间和发现次数不会再次执行 Peer Lookup
未知 infohash 首先只向返回样本的 DHT 节点查询一次 `get_peers` 只有单点查询没有返回 Peer 时才降级为有限迭代查找
`/stats` 中的 `sampled_hashes_filtered` `peer_lookup_preferred_succeeded``peer_lookup_fallbacks` 用于观察提前去重和单点优先效果
## Linux 资源限制
生产环境仍可能同时使用较多 TCP socket
Linux 生产运行必须把文件描述符上限提高到至少 65536
```bash
prlimit --nofile=65536:65536 -- \
/opt/dht-search/dht-search --config /opt/dht-search/dht-search.toml
```
systemd 服务需要设置
```ini
[Service]
LimitNOFILE=65536
```
文件描述符上限过低时 DHT 连接会挤占 Tantivy 和 RocksDB 打开文件所需的描述符并导致服务安全停止
## API
默认只监听 `127.0.0.1:8080`
```text
GET /health
GET /ready
GET /stats
GET /diagnostics/current
GET /diagnostics/history?range_secs=3600&resolution=raw
GET /diagnostics/history?range_secs=86400&resolution=minute
GET /config
PUT /config
GET /search?q=ubuntu&offset=0&limit=20
GET /search?q=%2A.iso
GET /search?q=&min_size=1048576&max_size=10737418240&extension=mkv
GET /search?q=流浪地球&min_files=1&availability=active&heat=hot&sort=heat
GET /search?q=%5ES%5Cd%7B2%7DE%5Cd%7B2%7D&mode=regex
GET /contents/{content_key}?offset=0&limit=20
GET /torrents/{infohash}?file_offset=0&file_limit=100
```
`limit` 被限制在 1 到 100 之间且 `offset` 最大为 10000
搜索支持中文英文数字和文件名片段匹配
默认搜索会自动识别不区分大小写的通配符并匹配名称 别名和文件路径 `*` 表示任意长度字符 `?` 表示一个字符 例如 `*.iso` 匹配所有以 `.iso` 结尾的已索引名称或文件路径 不含通配符时保持普通关键词和片段搜索
设置 `mode=regex` 后查询文本作为不区分大小写的正则表达式匹配名称 别名和文件路径 通配符和正则最长 256 字节并由 Tantivy 有限状态自动机执行
过滤参数还包括 `min_files` `max_files` `first_seen_after` `first_seen_before` `last_seen_after` `last_seen_before` `availability``heat`
排序支持 `relevance` `latest` `oldest` `heat` `size_desc` `size_asc``discoveries`
有关键词时默认按相关性排序 空查询默认按最近发现排序
搜索结果按 `content_key` 精确折叠并通过 `variant_count` 返回变体数量 `/contents/{content_key}` 用于分页查看全部 infohash 和磁力链接
搜索响应包含 `heat``availability` 摘要 详情响应包含完整验证时间 Peer 数和连续失败次数
详情文件列表默认返回 100 条且单次最多 200 条 使用 `file_offset` 翻页避免超大种子一次向浏览器返回全部文件
生产 Web 页面使用 `/` `/system``/settings` 三个路由 分别提供搜索运行诊断和配置管理 API 路径继续保持独立避免单页回退冲突
## 数据恢复
RocksDB 是权威数据源而 Tantivy 是可重建索引
当 Tantivy 目录不存在或结构不匹配时应用会直接创建新索引并从 RocksDB 的内容组状态完成全量重建
项目当前处于开发阶段 持久化结构变化时直接清理测试数据重新采集 不维护旧测试数据库兼容层
正常退出会先停止 DHT 和诊断采样 再排空持久化队列提交剩余索引最后关闭 HTTP 服务
## 规模基准
`dht-benchmark` 使用确定性数据调用真实 RocksDB 写入内容聚合待索引状态 Tantivy 索引和搜索接口
默认每十条记录生成一个相同内容的不同 infohash 用于同时覆盖精确去重和内容折叠场景
性能测量必须使用 release 构建并从小规模逐步增加
```powershell
$env:LIBCLANG_PATH = "$PWD\.tools\libclang\clang\native"
cargo run --release -p dht-search --bin dht-benchmark -- `
--records 10000 `
--cleanup
cargo run --release -p dht-search --bin dht-benchmark -- `
--records 100000 `
--query-iterations 100 `
--cleanup
cargo run --release -p dht-search --bin dht-benchmark -- `
--records 1000000 `
--query-iterations 100
```
| 参数 | 默认值 | 作用 |
|---|---:|---|
| `--records` | `10000` | 生成记录数量 上限一千万 |
| `--generation-batch-size` | `1000` | 单批生成并暂存在内存的记录数量 |
| `--index-batch-size` | `1000` | 每次 Tantivy 提交的内容文档数量 |
| `--index-max-retries` | `20` | Windows 临时 IO 错误的最大连续重试次数 |
| `--query-iterations` | `50` | 每类查询正式采样次数 |
| `--query-warmup` | `5` | 每类查询预热次数 |
| `--duplicate-every` | `10` | 每多少条创建一个相同内容的不同 infohash 零表示禁用 |
| `--output-dir` | `benchmark-data` | 独立运行数据和 JSON 报告根目录 |
| `--cleanup` | 不启用 | 报告写入后删除本次 RocksDB 和 Tantivy 数据 |
终端和 JSON 报告包含写入吞吐索引吞吐查询平均值与 P50/P95/P99 RocksDB 与 Tantivy 字节占用每条平均占用和进程峰值内存
`benchmark-data/runs` 保存每次未清理的数据库和索引 `benchmark-data/reports` 始终保留 JSON 报告 两者均不进入 Git
+422
View File
@@ -0,0 +1,422 @@
// 负责处理搜索详情统计和健康检查请求
use std::{
str::FromStr,
time::{SystemTime, UNIX_EPOCH},
};
use crate::{
domain::{InfoHash, MetadataRejectionReason},
search::{SearchMode, SearchOptions, SearchPage, SearchSort},
storage::VerificationPriority,
};
use axum::{
Json,
extract::{Path, Query, State},
http::StatusCode,
response::{IntoResponse, Response},
};
use super::{
ApiState,
request::{ContentVariantsRequest, DiagnosticHistoryRequest, SearchRequest, TorrentRequest},
response::{
ContentVariantsResponse, ErrorResponse, StatsResponse, StatusResponse, TorrentResponse,
TorrentVariantResponse,
},
};
use crate::config::{ConfigServiceError, ConfigSnapshot, ConfigUpdateRequest};
use crate::diagnostics::{CurrentDiagnosticsResponse, DiagnosticHistory, HistoryResolution};
pub(crate) async fn health() -> Json<StatusResponse> {
Json(StatusResponse { status: "ok" })
}
pub(crate) async fn ready() -> Json<StatusResponse> {
Json(StatusResponse { status: "ready" })
}
pub(crate) async fn stats(State(state): State<ApiState>) -> Json<StatsResponse> {
let dht = state.dht_stats.snapshot();
let observability = state.dht_stats.observability_snapshot();
let persistence = state.persistence.snapshot();
let disk = state.disk_guard.snapshot();
let backup = state.backup_stats.snapshot();
let http = state.http_stats.snapshot();
let filtered = persistence.filtered;
let verification = state
.verification
.as_ref()
.map(|ingress| ingress.stats().snapshot());
Json(StatsResponse {
http_active_requests: http.active_requests,
http_requests: http.requests,
http_client_errors: http.client_errors,
http_server_errors: http.server_errors,
http_latency_average_micros: http.latency_average_micros,
http_latency_p95_millis: http.latency_p95_millis,
http_latency_max_millis: http.latency_max_millis,
backup_created: backup.created,
backup_failed: backup.failed,
backup_skipped: backup.skipped,
backup_pruned: backup.pruned,
backup_last_success_at: backup.last_success_at,
backup_last_duration_millis: backup.last_duration_millis,
backup_latest_records: backup.latest_records,
disk_state: disk.mode.as_str(),
disk_available_bytes: disk.available_bytes,
disk_minimum_free_bytes: disk.minimum_free_bytes,
disk_resume_free_bytes: disk.resume_free_bytes,
disk_active_writes: disk.active_writes,
disk_probe_failed: disk.probe_failed,
disk_probe_failures: disk.probe_failures,
disk_transitions: disk.transitions,
disk_rejected_new_work: disk.rejected_new_work,
nodes: dht.node_pool_size,
udp_tx_packets: observability.udp_tx_packets,
find_node_queries: dht
.queries_new
.saturating_add(dht.queries_revisit)
.saturating_add(dht.queries_bootstrap),
peer_lookup_queries: dht.peer_lookup_queries,
peer_lookup_preferred_succeeded: dht.peer_lookup_preferred_succeeded,
peer_lookup_fallbacks: dht.peer_lookup_fallbacks,
sample_candidate_queue: dht.sample_candidate_queue_depth,
sample_candidate_queue_capacity: dht.sample_candidate_queue_capacity,
sample_candidates_routed: dht.sample_candidates_routed,
sample_candidates_fallback: dht.sample_candidates_fallback,
sample_queries: dht.sample_infohashes_queries,
sample_direct_queries: dht.sample_infohashes_direct_queries,
sample_snapshot_queries: dht.sample_infohashes_snapshot_queries,
sample_responses: dht.sample_infohashes_responses,
sample_direct_responses: dht.sample_infohashes_direct_responses,
sample_snapshot_responses: dht.sample_infohashes_snapshot_responses,
sample_timeouts: dht.sample_infohashes_timeouts,
sampled_hashes: dht.sample_infohashes_hashes_discovered,
sampled_hashes_filtered: dht.sample_infohashes_hashes_filtered,
sampled_hashes_duplicate: dht.sample_infohashes_hashes_duplicate,
sampled_hashes_dropped: dht.sample_infohashes_hashes_dropped,
metadata_peer_attempts: dht.metadata_peer_attempts,
metadata_in_flight: dht.metadata_in_flight,
metadata_ok: dht.metadata_peer_succeeded,
metadata_ok_from_announce: dht.metadata_success_from_announce,
metadata_ok_from_sample_direct: dht.metadata_success_from_sample_direct,
metadata_ok_from_sample_snapshot: dht.metadata_success_from_sample_snapshot,
metadata_ok_from_active_lookup: dht.metadata_success_from_active_lookup,
metadata_failed: dht.metadata_peer_failed,
metadata_filtered: observability
.metadata_failure_size_limit
.saturating_add(filtered.total()),
metadata_filtered_too_large: observability.metadata_failure_size_limit,
metadata_filtered_invalid_info_hash: filtered
.count(MetadataRejectionReason::InvalidInfoHash),
metadata_filtered_empty_name: filtered.count(MetadataRejectionReason::EmptyName),
metadata_filtered_name_too_long: filtered.count(MetadataRejectionReason::NameTooLong),
metadata_filtered_invalid_name: filtered.count(MetadataRejectionReason::InvalidName),
metadata_filtered_empty_file_list: filtered.count(MetadataRejectionReason::EmptyFileList),
metadata_filtered_too_many_files: filtered.count(MetadataRejectionReason::TooManyFiles),
metadata_filtered_empty_path: filtered.count(MetadataRejectionReason::EmptyPath),
metadata_filtered_path_too_long: filtered.count(MetadataRejectionReason::PathTooLong),
metadata_filtered_path_too_deep: filtered.count(MetadataRejectionReason::PathTooDeep),
metadata_filtered_invalid_path: filtered.count(MetadataRejectionReason::InvalidPath),
metadata_filtered_size_overflow: filtered.count(MetadataRejectionReason::SizeOverflow),
metadata_filtered_total_size_mismatch: filtered
.count(MetadataRejectionReason::TotalSizeMismatch),
persistence_accepted: persistence.accepted,
persistence_inserted: persistence.inserted,
persistence_updated: persistence.updated,
persistence_rejected_full: persistence.rejected_full,
persistence_queue: persistence.queue_depth,
indexed_documents: state.search.num_docs(),
verification_queue: verification.map_or(0, |stats| stats.queue_depth),
verification_accepted: verification.map_or(0, |stats| stats.accepted),
verification_deduplicated: verification.map_or(0, |stats| stats.deduplicated),
verification_rejected_full: verification.map_or(0, |stats| stats.rejected_full),
verification_started: verification.map_or(0, |stats| stats.started),
verification_succeeded: verification.map_or(0, |stats| stats.succeeded),
verification_failed: verification.map_or(0, |stats| stats.failed),
verification_peers_discovered: verification.map_or(0, |stats| stats.peers_discovered),
verification_handshakes_succeeded: verification
.map_or(0, |stats| stats.handshakes_succeeded),
})
}
pub(crate) async fn diagnostics_current(
State(state): State<ApiState>,
) -> Json<CurrentDiagnosticsResponse> {
Json(state.diagnostics.current())
}
pub(crate) async fn diagnostics_history(
State(state): State<ApiState>,
Query(request): Query<DiagnosticHistoryRequest>,
) -> Result<Json<DiagnosticHistory>, ApiError> {
const MAX_RANGE_SECS: u64 = 31 * 86_400;
if request.range_secs == 0 || request.range_secs > MAX_RANGE_SECS {
return Err(ApiError::bad_request("range_secs 必须在 1 到 2678400 之间"));
}
let resolution = match request.resolution.as_str() {
"raw" => HistoryResolution::Raw,
"minute" => HistoryResolution::Minute,
_ => return Err(ApiError::bad_request("resolution 只支持 raw 或 minute")),
};
let to = SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap_or_default()
.as_secs();
let from = to.saturating_sub(request.range_secs);
let diagnostics = state.diagnostics.clone();
let history = tokio::task::spawn_blocking(move || diagnostics.history(resolution, from, to))
.await
.map_err(|error| ApiError::internal(format!("诊断历史查询任务失败: {error}")))?
.map_err(|error| ApiError::internal(error.to_string()))?;
Ok(Json(history))
}
pub(crate) async fn config_snapshot(State(state): State<ApiState>) -> Json<ConfigSnapshot> {
Json(state.config.snapshot())
}
pub(crate) async fn config_update(
State(state): State<ApiState>,
Json(request): Json<ConfigUpdateRequest>,
) -> Result<Json<ConfigSnapshot>, ApiError> {
let config = state.config.clone();
let snapshot = tokio::task::spawn_blocking(move || config.update(request))
.await
.map_err(|error| ApiError::internal(format!("配置保存任务失败: {error}")))?
.map_err(|error| match error {
ConfigServiceError::Conflict => ApiError::conflict(error.to_string()),
ConfigServiceError::Validation(_) => ApiError::bad_request(error.to_string()),
ConfigServiceError::Persistence(_) => ApiError::internal(error.to_string()),
})?;
Ok(Json(snapshot))
}
pub(crate) async fn search(
State(state): State<ApiState>,
Query(request): Query<SearchRequest>,
) -> Result<Json<SearchPage>, ApiError> {
let verification = state.verification.clone();
if request.q.len() > 512 {
return Err(ApiError::bad_request("查询文本不能超过 512 字节"));
}
if (request.mode != SearchMode::Text || request.q.contains('*') || request.q.contains('?'))
&& request.q.len() > 256
{
return Err(ApiError::bad_request("通配符或正则表达式不能超过 256 字节"));
}
validate_range(request.min_size, request.max_size, "min_size", "max_size")?;
validate_range(
request.min_files,
request.max_files,
"min_files",
"max_files",
)?;
validate_range(
request.first_seen_after,
request.first_seen_before,
"first_seen_after",
"first_seen_before",
)?;
validate_range(
request.last_seen_after,
request.last_seen_before,
"last_seen_after",
"last_seen_before",
)?;
let mut query = request.q;
let content_key = if request.mode == SearchMode::Text {
InfoHash::from_str(query.trim()).ok()
} else {
None
};
let content_key = if let Some(info_hash) = content_key {
let repository = state.repository.clone();
let record = tokio::task::spawn_blocking(move || repository.get(info_hash))
.await
.map_err(|error| ApiError::internal(error.to_string()))?
.map_err(|error| ApiError::internal(error.to_string()))?;
let Some(record) = record else {
return Ok(Json(SearchPage {
total: 0,
offset: request.offset.min(10_000),
limit: request.limit.clamp(1, 100),
hits: Vec::new(),
sort: request.sort.unwrap_or(SearchSort::Relevance),
}));
};
query.clear();
Some(record.content_key)
} else {
None
};
let page = tokio::task::spawn_blocking(move || {
state.search.search_with(SearchOptions {
query,
mode: request.mode,
offset: request.offset,
limit: request.limit,
min_size: request.min_size,
max_size: request.max_size,
extension: request.extension,
min_files: request.min_files,
max_files: request.max_files,
first_seen_after: request.first_seen_after,
first_seen_before: request.first_seen_before,
last_seen_after: request.last_seen_after,
last_seen_before: request.last_seen_before,
availability: request.availability,
heat: request.heat,
sort: request.sort,
content_key,
})
})
.await
.map_err(|error| ApiError::internal(error.to_string()))?
.map_err(|error| ApiError::bad_request(error.to_string()))?;
if let Some(verification) = &verification {
let hashes = page
.hits
.iter()
.filter_map(|hit| InfoHash::from_str(&hit.info_hash).ok())
.collect();
verification
.enqueue(hashes, VerificationPriority::Normal)
.await;
}
Ok(Json(page))
}
pub(crate) async fn content_variants(
State(state): State<ApiState>,
Path(content_key): Path<String>,
Query(request): Query<ContentVariantsRequest>,
) -> Result<Json<ContentVariantsResponse>, ApiError> {
if request.limit == 0 || request.limit > 100 {
return Err(ApiError::bad_request("limit 必须在 1 到 100 之间"));
}
if request.offset > 10_000 {
return Err(ApiError::bad_request("offset 不能超过 10000"));
}
let decoded = hex::decode(&content_key)
.map_err(|_| ApiError::bad_request("content_key 必须是六十四位十六进制字符串"))?;
let content_key_bytes: [u8; 32] = decoded
.try_into()
.map_err(|_| ApiError::bad_request("content_key 必须是六十四位十六进制字符串"))?;
let repository = state.repository.clone();
let offset = request.offset;
let limit = request.limit;
let variants = tokio::task::spawn_blocking(move || {
repository.content_variants(&content_key_bytes, offset, limit)
})
.await
.map_err(|error| ApiError::internal(error.to_string()))?
.map_err(|error| ApiError::internal(error.to_string()))?;
if variants.total == 0 {
return Err(ApiError::not_found("没有找到该 content_key"));
}
Ok(Json(ContentVariantsResponse {
content_key: content_key.to_lowercase(),
total: variants.total,
offset,
limit,
variants: variants
.records
.into_iter()
.map(TorrentVariantResponse::from)
.collect(),
}))
}
pub(crate) async fn torrent(
State(state): State<ApiState>,
Path(info_hash): Path<String>,
Query(request): Query<TorrentRequest>,
) -> Result<Json<TorrentResponse>, ApiError> {
if request.file_limit == 0 || request.file_limit > 200 {
return Err(ApiError::bad_request("file_limit 必须在 1 到 200 之间"));
}
if request.file_offset > 1_000_000 {
return Err(ApiError::bad_request("file_offset 不能超过 1000000"));
}
let verification = state.verification.clone();
let info_hash =
InfoHash::from_str(&info_hash).map_err(|error| ApiError::bad_request(error.to_string()))?;
let record = tokio::task::spawn_blocking(move || state.repository.get_visible(info_hash))
.await
.map_err(|error| ApiError::internal(error.to_string()))?
.map_err(|error| ApiError::internal(error.to_string()))?
.ok_or_else(|| ApiError::not_found("没有找到该 infohash"))?;
if let Some(verification) = &verification {
verification
.enqueue(vec![info_hash], VerificationPriority::High)
.await;
}
Ok(Json(TorrentResponse::from_record(
record,
request.file_offset,
request.file_limit,
)))
}
pub(crate) struct ApiError {
status: StatusCode,
message: String,
}
impl ApiError {
fn bad_request(message: impl Into<String>) -> Self {
Self {
status: StatusCode::BAD_REQUEST,
message: message.into(),
}
}
fn not_found(message: impl Into<String>) -> Self {
Self {
status: StatusCode::NOT_FOUND,
message: message.into(),
}
}
fn conflict(message: impl Into<String>) -> Self {
Self {
status: StatusCode::CONFLICT,
message: message.into(),
}
}
fn internal(message: impl Into<String>) -> Self {
Self {
status: StatusCode::INTERNAL_SERVER_ERROR,
message: message.into(),
}
}
}
fn validate_range(
min: Option<u64>,
max: Option<u64>,
min_name: &str,
max_name: &str,
) -> Result<(), ApiError> {
if min.zip(max).is_some_and(|(min, max)| min > max) {
return Err(ApiError::bad_request(format!(
"{min_name} 不能大于 {max_name}"
)));
}
Ok(())
}
impl IntoResponse for ApiError {
fn into_response(self) -> Response {
(
self.status,
Json(ErrorResponse {
error: self.message,
}),
)
.into_response()
}
}
+487
View File
@@ -0,0 +1,487 @@
// 负责组合 HTTP 路由和共享接口状态但不直接访问数据库实现
mod handlers;
mod request;
mod response;
use std::{net::SocketAddr, path::PathBuf, sync::Arc};
use crate::{search::SearchEngine, storage::TorrentRepository};
use axum::{
Router,
extract::{Request, State},
middleware::{self, Next},
response::Response,
routing::get,
};
use dht_crawler::DhtRuntimeStats;
use tokio_util::sync::CancellationToken;
use tower_http::services::{ServeDir, ServeFile};
use crate::{
backup::BackupStats,
config::ConfigService,
crawler::pipeline::PersistenceIngress,
diagnostics::{DiagnosticsHandle, HttpStats},
disk_guard::DiskGuard,
verification::VerificationIngress,
};
#[derive(Clone)]
pub(crate) struct ApiState {
pub(crate) repository: Arc<dyn TorrentRepository>,
pub(crate) search: SearchEngine,
pub(crate) dht_stats: DhtRuntimeStats,
pub(crate) persistence: PersistenceIngress,
pub(crate) verification: Option<VerificationIngress>,
pub(crate) disk_guard: DiskGuard,
pub(crate) backup_stats: BackupStats,
pub(crate) diagnostics: DiagnosticsHandle,
pub(crate) config: ConfigService,
pub(crate) http_stats: HttpStats,
}
pub(crate) async fn serve(
listen: SocketAddr,
web_dir: PathBuf,
state: ApiState,
cancel: CancellationToken,
) -> std::io::Result<()> {
let router = router(state, web_dir);
let listener = tokio::net::TcpListener::bind(listen).await?;
tracing::info!(%listen, "HTTP 服务启动");
axum::serve(listener, router)
.with_graceful_shutdown(cancel.cancelled_owned())
.await
}
fn router(state: ApiState, web_dir: PathBuf) -> Router {
let index = web_dir.join("index.html");
let http_stats = state.http_stats.clone();
Router::new()
.route("/health", get(handlers::health))
.route("/ready", get(handlers::ready))
.route("/stats", get(handlers::stats))
.route("/diagnostics/current", get(handlers::diagnostics_current))
.route("/diagnostics/history", get(handlers::diagnostics_history))
.route(
"/config",
get(handlers::config_snapshot).put(handlers::config_update),
)
.route("/search", get(handlers::search))
.route("/contents/{content_key}", get(handlers::content_variants))
.route("/torrents/{info_hash}", get(handlers::torrent))
.fallback_service(ServeDir::new(web_dir).fallback(ServeFile::new(index)))
.layer(middleware::from_fn_with_state(http_stats, observe_http))
.with_state(state)
}
async fn observe_http(State(stats): State<HttpStats>, request: Request, next: Next) -> Response {
let timer = stats.begin();
let response = next.run(request).await;
timer.finish(response.status().as_u16());
response
}
#[cfg(test)]
mod tests {
use std::sync::Arc;
use crate::{
domain::{InfoHash, MetadataCandidate, MetadataLimits, TorrentFile, TorrentRecord},
search::SearchEngine,
storage::{RocksTorrentRepository, TorrentRepository},
};
use axum::{
body::{Body, to_bytes},
http::{Request, StatusCode},
};
use dht_crawler::DhtRuntimeStats;
use tempfile::TempDir;
use tower::ServiceExt;
use crate::{
config::{AppConfigDto, ConfigService, DiskGuardConfig, TomlConfigStore},
crawler::pipeline::PersistencePipeline,
diagnostics::DiagnosticsRuntime,
disk_guard::DiskGuard,
verification::VerificationIngress,
};
use super::*;
#[tokio::test]
async fn search_and_detail_return_user_fields_and_enqueue_verification() {
let directory = TempDir::new().unwrap();
let repository =
Arc::new(RocksTorrentRepository::open(directory.path().join("rocksdb")).unwrap());
let mut record = TorrentRecord::try_from(MetadataCandidate {
info_hash: "0101010101010101010101010101010101010101".into(),
name: "Example Movie".into(),
total_size: 205,
files: (0..205)
.map(|index| TorrentFile {
path: if index == 0 {
"movie.mkv".into()
} else {
format!("extras/{index}.txt")
},
size: 1,
})
.collect(),
piece_length: 16_384,
source_peers: vec!["127.0.0.1:6881".into()],
timestamp: 10,
})
.unwrap();
record.availability = crate::domain::Availability::default();
repository.upsert(record.clone()).unwrap();
let mut variant = record.clone();
variant.info_hash = InfoHash::from_bytes([2; 20]);
variant.name = "Example Movie Alternate".into();
repository.upsert(variant).unwrap();
let search = SearchEngine::open(directory.path().join("tantivy")).unwrap();
search.index_pending(repository.as_ref(), 10, 20).unwrap();
let repository_trait: Arc<dyn TorrentRepository> = repository.clone();
let disk_guard = DiskGuard::new(&DiskGuardConfig {
enabled: false,
..DiskGuardConfig::default()
});
let persistence = PersistencePipeline::start(
repository_trait.clone(),
4,
MetadataLimits::default(),
disk_guard.clone(),
);
let verification = VerificationIngress::for_test(repository.clone(), 10);
let web_dir = directory.path().join("web");
std::fs::create_dir_all(&web_dir).unwrap();
std::fs::write(web_dir.join("index.html"), "<main>DHT Search</main>").unwrap();
let app = router(
ApiState {
repository: repository_trait,
search,
dht_stats: DhtRuntimeStats::default(),
persistence: persistence.ingress.clone(),
verification: Some(verification),
disk_guard,
backup_stats: BackupStats::default(),
diagnostics: DiagnosticsRuntime::disabled().handle(),
config: ConfigService::new(
Arc::new(TomlConfigStore::new(directory.path().join("service.toml"))),
AppConfigDto::default(),
Vec::new(),
)
.unwrap(),
http_stats: HttpStats::default(),
},
web_dir,
);
for uri in ["/", "/system", "/settings"] {
let response = app
.clone()
.oneshot(Request::builder().uri(uri).body(Body::empty()).unwrap())
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
let body = to_bytes(response.into_body(), usize::MAX).await.unwrap();
assert_eq!(body.as_ref(), b"<main>DHT Search</main>");
}
let response = app
.clone()
.oneshot(
Request::builder()
.uri("/stats")
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
let json: serde_json::Value =
serde_json::from_slice(&to_bytes(response.into_body(), usize::MAX).await.unwrap())
.unwrap();
assert_eq!(json["metadata_filtered"], 0);
assert_eq!(json["metadata_filtered_too_many_files"], 0);
assert_eq!(json["disk_state"], "normal");
assert!(json["disk_available_bytes"].is_null());
assert_eq!(json["backup_created"], 0);
assert_eq!(json["http_active_requests"], 1);
assert!(
json["http_requests"]
.as_u64()
.is_some_and(|value| value >= 3)
);
let response = app
.clone()
.oneshot(
Request::builder()
.uri("/diagnostics/current")
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
let json: serde_json::Value =
serde_json::from_slice(&to_bytes(response.into_body(), usize::MAX).await.unwrap())
.unwrap();
assert_eq!(json["status"]["enabled"], false);
assert!(json["sample"].is_null());
let response = app
.clone()
.oneshot(
Request::builder()
.uri("/diagnostics/history?resolution=seconds")
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::BAD_REQUEST);
let response = app
.clone()
.oneshot(
Request::builder()
.uri("/config")
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
let mut config_snapshot: serde_json::Value =
serde_json::from_slice(&to_bytes(response.into_body(), usize::MAX).await.unwrap())
.unwrap();
let original_revision = config_snapshot["revision"].as_str().unwrap().to_owned();
config_snapshot["config"]["dht"]["port"] = serde_json::json!(22_313);
let update = serde_json::json!({
"revision": original_revision,
"config": config_snapshot["config"].clone(),
});
let response = app
.clone()
.oneshot(
Request::builder()
.method("PUT")
.uri("/config")
.header("content-type", "application/json")
.body(Body::from(serde_json::to_vec(&update).unwrap()))
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
let json: serde_json::Value =
serde_json::from_slice(&to_bytes(response.into_body(), usize::MAX).await.unwrap())
.unwrap();
assert_eq!(json["config"]["dht"]["port"], 22_313);
assert_eq!(json["restart_required"], true);
assert!(directory.path().join("service.toml").exists());
let response = app
.clone()
.oneshot(
Request::builder()
.method("PUT")
.uri("/config")
.header("content-type", "application/json")
.body(Body::from(serde_json::to_vec(&update).unwrap()))
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::CONFLICT);
let response = app
.clone()
.oneshot(
Request::builder()
.uri("/search?q=Example")
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
let json: serde_json::Value =
serde_json::from_slice(&to_bytes(response.into_body(), usize::MAX).await.unwrap())
.unwrap();
assert_eq!(json["hits"][0]["name"], "Example Movie");
assert!(json["hits"][0]["heat"]["score"].is_number());
assert_eq!(json["hits"][0]["availability"]["status"], "unknown");
assert_eq!(json["hits"][0]["variant_count"], 2);
assert_eq!(repository.verification_queue_len().unwrap(), 1);
let response = app
.clone()
.oneshot(
Request::builder()
.uri("/search?q=example.%2Amovie&mode=regex")
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
let json: serde_json::Value =
serde_json::from_slice(&to_bytes(response.into_body(), usize::MAX).await.unwrap())
.unwrap();
assert_eq!(json["total"], 1);
let response = app
.clone()
.oneshot(
Request::builder()
.uri("/search?q=%2A.mkv")
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
let json: serde_json::Value =
serde_json::from_slice(&to_bytes(response.into_body(), usize::MAX).await.unwrap())
.unwrap();
assert_eq!(json["total"], 1);
let response = app
.clone()
.oneshot(
Request::builder()
.uri("/search?q=%5B&mode=regex")
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::BAD_REQUEST);
let response = app
.clone()
.oneshot(
Request::builder()
.uri(format!("/search?q={}", record.info_hash))
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
let json: serde_json::Value =
serde_json::from_slice(&to_bytes(response.into_body(), usize::MAX).await.unwrap())
.unwrap();
assert_eq!(json["total"], 1);
let response = app
.clone()
.oneshot(
Request::builder()
.uri(format!("/contents/{}", hex::encode(record.content_key)))
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
let json: serde_json::Value =
serde_json::from_slice(&to_bytes(response.into_body(), usize::MAX).await.unwrap())
.unwrap();
assert_eq!(json["total"], 2);
assert_eq!(json["variants"].as_array().unwrap().len(), 2);
let response = app
.clone()
.oneshot(
Request::builder()
.uri("/search?min_files=3&max_files=1")
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::BAD_REQUEST);
let response = app
.clone()
.oneshot(
Request::builder()
.uri("/search?sort=not_a_sort")
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::BAD_REQUEST);
let response = app
.clone()
.oneshot(
Request::builder()
.uri(format!("/torrents/{}", InfoHash::from_bytes([1; 20])))
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
let json: serde_json::Value =
serde_json::from_slice(&to_bytes(response.into_body(), usize::MAX).await.unwrap())
.unwrap();
assert_eq!(json["name"], "Example Movie");
assert!(
json["magnet_link"]
.as_str()
.unwrap()
.starts_with("magnet:?xt=")
);
assert_eq!(json["files"][0]["path"], "movie.mkv");
assert_eq!(json["files"].as_array().unwrap().len(), 100);
assert_eq!(json["file_count"], 205);
assert_eq!(json["file_offset"], 0);
assert_eq!(json["file_limit"], 100);
assert_eq!(repository.verification_queue_len().unwrap(), 1);
let response = app
.clone()
.oneshot(
Request::builder()
.uri(format!(
"/torrents/{}?file_limit=0",
InfoHash::from_bytes([1; 20])
))
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::BAD_REQUEST);
let response = app
.clone()
.oneshot(
Request::builder()
.uri(format!(
"/torrents/{}?file_offset=200&file_limit=100",
InfoHash::from_bytes([1; 20])
))
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
let json: serde_json::Value =
serde_json::from_slice(&to_bytes(response.into_body(), usize::MAX).await.unwrap())
.unwrap();
assert_eq!(json["files"].as_array().unwrap().len(), 5);
assert_eq!(json["file_offset"], 200);
persistence.close_and_join().await.unwrap();
}
}
+71
View File
@@ -0,0 +1,71 @@
// 负责定义 HTTP 查询参数和输入校验模型
use crate::{
domain::{AvailabilityStatus, HeatLevel},
search::{SearchMode, SearchSort},
};
use serde::Deserialize;
fn default_limit() -> usize {
20
}
#[derive(Debug, Deserialize)]
pub(crate) struct SearchRequest {
#[serde(default)]
pub(crate) q: String,
#[serde(default)]
pub(crate) mode: SearchMode,
#[serde(default)]
pub(crate) offset: usize,
#[serde(default = "default_limit")]
pub(crate) limit: usize,
pub(crate) min_size: Option<u64>,
pub(crate) max_size: Option<u64>,
pub(crate) extension: Option<String>,
pub(crate) min_files: Option<u64>,
pub(crate) max_files: Option<u64>,
pub(crate) first_seen_after: Option<u64>,
pub(crate) first_seen_before: Option<u64>,
pub(crate) last_seen_after: Option<u64>,
pub(crate) last_seen_before: Option<u64>,
pub(crate) availability: Option<AvailabilityStatus>,
pub(crate) heat: Option<HeatLevel>,
pub(crate) sort: Option<SearchSort>,
}
#[derive(Debug, Deserialize)]
pub(crate) struct ContentVariantsRequest {
#[serde(default)]
pub(crate) offset: usize,
#[serde(default = "default_limit")]
pub(crate) limit: usize,
}
#[derive(Debug, Deserialize)]
pub(crate) struct TorrentRequest {
#[serde(default)]
pub(crate) file_offset: usize,
#[serde(default = "default_file_limit")]
pub(crate) file_limit: usize,
}
#[derive(Debug, Deserialize)]
pub(crate) struct DiagnosticHistoryRequest {
#[serde(default = "default_diagnostic_range")]
pub(crate) range_secs: u64,
#[serde(default = "default_diagnostic_resolution")]
pub(crate) resolution: String,
}
fn default_diagnostic_range() -> u64 {
3_600
}
fn default_diagnostic_resolution() -> String {
"raw".to_owned()
}
fn default_file_limit() -> usize {
100
}
+204
View File
@@ -0,0 +1,204 @@
// 负责定义稳定的 HTTP 响应模型和领域对象转换边界
use crate::domain::{Availability, Heat, TorrentFile, TorrentRecord};
use serde::Serialize;
use std::time::{SystemTime, UNIX_EPOCH};
#[derive(Debug, Serialize)]
pub(crate) struct StatusResponse {
pub(crate) status: &'static str,
}
#[derive(Debug, Serialize)]
pub(crate) struct ErrorResponse {
pub(crate) error: String,
}
#[derive(Debug, Serialize)]
pub(crate) struct StatsResponse {
pub(crate) http_active_requests: u64,
pub(crate) http_requests: u64,
pub(crate) http_client_errors: u64,
pub(crate) http_server_errors: u64,
pub(crate) http_latency_average_micros: Option<u64>,
pub(crate) http_latency_p95_millis: Option<u64>,
pub(crate) http_latency_max_millis: u64,
pub(crate) backup_created: u64,
pub(crate) backup_failed: u64,
pub(crate) backup_skipped: u64,
pub(crate) backup_pruned: u64,
pub(crate) backup_last_success_at: Option<u64>,
pub(crate) backup_last_duration_millis: u64,
pub(crate) backup_latest_records: u64,
pub(crate) disk_state: &'static str,
pub(crate) disk_available_bytes: Option<u64>,
pub(crate) disk_minimum_free_bytes: u64,
pub(crate) disk_resume_free_bytes: u64,
pub(crate) disk_active_writes: usize,
pub(crate) disk_probe_failed: bool,
pub(crate) disk_probe_failures: u64,
pub(crate) disk_transitions: u64,
pub(crate) disk_rejected_new_work: u64,
pub(crate) nodes: usize,
pub(crate) udp_tx_packets: u64,
pub(crate) find_node_queries: u64,
pub(crate) peer_lookup_queries: u64,
pub(crate) peer_lookup_preferred_succeeded: u64,
pub(crate) peer_lookup_fallbacks: u64,
pub(crate) sample_candidate_queue: usize,
pub(crate) sample_candidate_queue_capacity: usize,
pub(crate) sample_candidates_routed: u64,
pub(crate) sample_candidates_fallback: u64,
pub(crate) sample_queries: u64,
pub(crate) sample_direct_queries: u64,
pub(crate) sample_snapshot_queries: u64,
pub(crate) sample_responses: u64,
pub(crate) sample_direct_responses: u64,
pub(crate) sample_snapshot_responses: u64,
pub(crate) sample_timeouts: u64,
pub(crate) sampled_hashes: u64,
pub(crate) sampled_hashes_filtered: u64,
pub(crate) sampled_hashes_duplicate: u64,
pub(crate) sampled_hashes_dropped: u64,
pub(crate) metadata_peer_attempts: u64,
pub(crate) metadata_in_flight: usize,
pub(crate) metadata_ok: u64,
pub(crate) metadata_ok_from_announce: u64,
pub(crate) metadata_ok_from_sample_direct: u64,
pub(crate) metadata_ok_from_sample_snapshot: u64,
pub(crate) metadata_ok_from_active_lookup: u64,
pub(crate) metadata_failed: u64,
pub(crate) metadata_filtered: u64,
pub(crate) metadata_filtered_too_large: u64,
pub(crate) metadata_filtered_invalid_info_hash: u64,
pub(crate) metadata_filtered_empty_name: u64,
pub(crate) metadata_filtered_name_too_long: u64,
pub(crate) metadata_filtered_invalid_name: u64,
pub(crate) metadata_filtered_empty_file_list: u64,
pub(crate) metadata_filtered_too_many_files: u64,
pub(crate) metadata_filtered_empty_path: u64,
pub(crate) metadata_filtered_path_too_long: u64,
pub(crate) metadata_filtered_path_too_deep: u64,
pub(crate) metadata_filtered_invalid_path: u64,
pub(crate) metadata_filtered_size_overflow: u64,
pub(crate) metadata_filtered_total_size_mismatch: u64,
pub(crate) persistence_accepted: u64,
pub(crate) persistence_inserted: u64,
pub(crate) persistence_updated: u64,
pub(crate) persistence_rejected_full: u64,
pub(crate) persistence_queue: usize,
pub(crate) indexed_documents: u64,
pub(crate) verification_queue: u64,
pub(crate) verification_accepted: u64,
pub(crate) verification_deduplicated: u64,
pub(crate) verification_rejected_full: u64,
pub(crate) verification_started: u64,
pub(crate) verification_succeeded: u64,
pub(crate) verification_failed: u64,
pub(crate) verification_peers_discovered: u64,
pub(crate) verification_handshakes_succeeded: u64,
}
#[derive(Debug, Serialize)]
pub(crate) struct TorrentResponse {
pub(crate) info_hash: String,
pub(crate) magnet_link: String,
pub(crate) name: String,
pub(crate) total_size: u64,
pub(crate) file_count: usize,
pub(crate) file_offset: usize,
pub(crate) file_limit: usize,
pub(crate) files: Vec<TorrentFile>,
pub(crate) piece_length: u64,
pub(crate) content_key: String,
pub(crate) first_seen: u64,
pub(crate) last_seen: u64,
pub(crate) seen_count: u64,
pub(crate) heat: Heat,
pub(crate) availability: Availability,
}
#[derive(Debug, Serialize)]
pub(crate) struct ContentVariantsResponse {
pub(crate) content_key: String,
pub(crate) total: u64,
pub(crate) offset: usize,
pub(crate) limit: usize,
pub(crate) variants: Vec<TorrentVariantResponse>,
}
#[derive(Debug, Serialize)]
pub(crate) struct TorrentVariantResponse {
pub(crate) info_hash: String,
pub(crate) magnet_link: String,
pub(crate) name: String,
pub(crate) total_size: u64,
pub(crate) file_count: usize,
pub(crate) first_seen: u64,
pub(crate) last_seen: u64,
pub(crate) seen_count: u64,
pub(crate) heat: Heat,
pub(crate) availability: Availability,
}
impl From<TorrentRecord> for TorrentVariantResponse {
fn from(record: TorrentRecord) -> Self {
let info_hash = record.info_hash.to_string();
let heat = record.heat(unix_timestamp());
Self {
magnet_link: format!("magnet:?xt=urn:btih:{info_hash}"),
info_hash,
name: record.name,
total_size: record.total_size,
file_count: record.files.len(),
first_seen: record.first_seen,
last_seen: record.last_seen,
seen_count: record.seen_count,
heat,
availability: record.availability,
}
}
}
impl TorrentResponse {
pub(crate) fn from_record(
record: TorrentRecord,
file_offset: usize,
file_limit: usize,
) -> Self {
let info_hash = record.info_hash.to_string();
let heat = record.heat(unix_timestamp());
let file_count = record.files.len();
let file_offset = file_offset.min(file_count);
let files = record
.files
.into_iter()
.skip(file_offset)
.take(file_limit)
.collect();
Self {
magnet_link: format!("magnet:?xt=urn:btih:{info_hash}"),
info_hash,
name: record.name,
total_size: record.total_size,
file_count,
file_offset,
file_limit,
files,
piece_length: record.piece_length,
content_key: hex::encode(record.content_key),
first_seen: record.first_seen,
last_seen: record.last_seen,
seen_count: record.seen_count,
heat,
availability: record.availability,
}
}
}
fn unix_timestamp() -> u64 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap_or_default()
.as_secs()
}
+385
View File
@@ -0,0 +1,385 @@
// 负责连接采集存储索引和接口层并定义应用级启动顺序
use std::{
str::FromStr,
sync::Arc,
time::{Duration, SystemTime, UNIX_EPOCH},
};
use crate::{
domain::InfoHash,
search::SearchEngine,
storage::{RocksTorrentRepository, TorrentRepository},
};
use dht_crawler::DHTServer;
use tokio_util::sync::CancellationToken;
use crate::{
api::{self, ApiState},
backup::{self, BackupStats},
config::{AppConfig, ConfigService},
crawler::pipeline::PersistencePipeline,
diagnostics::{DiagnosticSources, DiagnosticsRuntime, HttpStats},
disk_guard::{self, DiskGuard},
error::AppError,
index_worker, monitor, shutdown, verification,
};
pub(crate) async fn run(config: AppConfig, config_service: ConfigService) -> Result<(), AppError> {
std::fs::create_dir_all(&config.data_dir)?;
let _data_lock = backup::acquire_data_lock(&config.data_dir)?;
let disk_guard = DiskGuard::new(&config.disk_guard);
let database_path = config.data_dir.join("rocksdb");
let metadata_limits = config.metadata_limits();
let content_filter = Arc::new(config.content_filter()?);
let repository = Arc::new(RocksTorrentRepository::open_with_rules(
&database_path,
metadata_limits.rule_id(),
content_filter,
)?);
let search_path = config.data_dir.join("tantivy");
let (search, search_created) = if repository.content_filter_changed() {
(SearchEngine::recreate(&search_path)?, true)
} else {
SearchEngine::open_with_status(&search_path)?
};
disk_guard.probe(&config.data_dir, 0);
let repository_api: Arc<dyn TorrentRepository> = repository.clone();
let mut persistence = PersistencePipeline::start(
repository_api,
config.persistence_queue_capacity,
metadata_limits,
disk_guard.clone(),
);
let ingress = persistence.ingress.clone();
let disk_cancel = CancellationToken::new();
let disk_task = tokio::spawn(disk_guard::run(
disk_guard.clone(),
config.data_dir.clone(),
config.disk_guard.clone(),
ingress.clone(),
disk_cancel.clone(),
));
let backup_cancel = CancellationToken::new();
let (backup_stats, backup_task) = if config.backup.enabled {
let (stats, task) = backup::start(
repository.clone(),
config.backup.clone(),
disk_guard.clone(),
backup_cancel.clone(),
);
(stats, Some(task))
} else {
(BackupStats::default(), None)
};
let options = config.dht_options();
let server = DHTServer::new(options.clone()).await?;
server.on_error(|error| tracing::error!(%error, "DHT 运行时错误"));
let sampled_repository = repository.clone();
let sampled_disk_guard = disk_guard.clone();
server.on_sampled_hashes(move |hashes| {
let repository = sampled_repository.clone();
let disk_guard = sampled_disk_guard.clone();
async move {
let Some(permit) = disk_guard.begin_admission() else {
return Vec::new();
};
let fallback = hashes.clone();
let info_hashes: Vec<_> = hashes.into_iter().map(InfoHash::from_bytes).collect();
let result = tokio::task::spawn_blocking(move || {
let _permit = permit;
repository.filter_unknown_and_observe(&info_hashes, unix_timestamp())
})
.await;
match result {
Ok(Ok(unknown)) => unknown
.into_iter()
.map(|info_hash| *info_hash.as_bytes())
.collect(),
Ok(Err(error)) => {
tracing::error!(%error, "采样 infohash 批量持久化去重失败");
fallback
}
Err(error) => {
tracing::error!(%error, "采样 infohash 批量去重任务异常");
fallback
}
}
}
});
let gate_repository = repository.clone();
let gate_disk_guard = disk_guard.clone();
server.on_metadata_fetch(move |hash| {
let repository = gate_repository.clone();
let disk_guard = gate_disk_guard.clone();
async move {
let Some(permit) = disk_guard.begin_admission() else {
return false;
};
let Ok(info_hash) = InfoHash::from_str(&hash) else {
tracing::warn!(%hash, "DHT 提供了无效 infohash");
return false;
};
let result = tokio::task::spawn_blocking(move || {
let _permit = permit;
repository.observe_existing(info_hash, unix_timestamp())
})
.await;
match result {
Ok(Ok(already_exists)) => !already_exists,
Ok(Err(error)) => {
tracing::error!(%error, %hash, "持久化去重查询失败");
true
}
Err(error) => {
tracing::error!(%error, %hash, "持久化去重任务异常");
true
}
}
}
});
let callback_ingress = ingress.clone();
server.on_torrent_with_ack(move |torrent| callback_ingress.try_enqueue(torrent));
server.on_metadata_fetch_complete(|completion| {
tracing::debug!(
info_hash = %completion.info_hash,
status = ?completion.status,
attempts = completion.attempts,
"Metadata 任务完成"
)
});
let verification_cancel = CancellationToken::new();
let (verification_fatal_tx, mut verification_fatal) = tokio::sync::oneshot::channel();
let mut verification_fatal_guard = None;
let (verification_ingress, verification_task) = if config.verification.enabled {
let (ingress, worker) = verification::start(
repository.clone(),
server.clone(),
config.verification.clone(),
disk_guard.clone(),
verification_cancel.clone(),
);
let cancel = verification_cancel.clone();
let task = tokio::spawn(async move {
let result = match worker.await {
Ok(result) => result,
Err(error) => Err(error.to_string()),
};
if !cancel.is_cancelled() {
let message = result
.as_ref()
.err()
.cloned()
.unwrap_or_else(|| "可用性验证 worker 意外停止".to_owned());
let _ = verification_fatal_tx.send(message);
}
result
});
(Some(ingress), Some(task))
} else {
verification_fatal_guard = Some(verification_fatal_tx);
(None, None)
};
tracing::info!(
dht_port = options.port,
data_dir = %config.data_dir.display(),
persistence_queue_capacity = config.persistence_queue_capacity,
"dht-search 启动"
);
let monitor_cancel = CancellationToken::new();
let monitor = tokio::spawn(monitor::run(
server.clone(),
ingress,
disk_guard.clone(),
config.stats_interval_secs,
monitor_cancel.clone(),
));
let index_cancel = CancellationToken::new();
let (index_fatal_tx, mut index_fatal) = tokio::sync::oneshot::channel();
let index_task = tokio::spawn(index_worker::run(
repository.clone(),
search.clone(),
index_worker::IndexWorkerOptions {
batch_size: config.index_batch_size,
interval: Duration::from_millis(config.index_interval_millis),
prepare_full_reindex: search_created,
},
disk_guard.clone(),
index_cancel.clone(),
index_fatal_tx,
));
let http_stats = HttpStats::default();
let diagnostics = if config.diagnostics.enabled {
match DiagnosticsRuntime::start(
config.diagnostics.clone(),
DiagnosticSources {
repository: repository.clone(),
search: search.clone(),
dht: server.runtime_stats(),
persistence: persistence.ingress.clone(),
disk_guard: disk_guard.clone(),
http: http_stats.clone(),
},
) {
Ok(runtime) => runtime,
Err(error) => {
tracing::warn!(%error, "运行诊断历史初始化失败 将继续提供核心服务");
DiagnosticsRuntime::disabled()
}
}
} else {
DiagnosticsRuntime::disabled()
};
let api_cancel = CancellationToken::new();
let mut api_task = tokio::spawn(api::serve(
config.http.listen,
config.http.web_dir.clone(),
ApiState {
repository: repository.clone(),
search,
dht_stats: server.runtime_stats(),
persistence: persistence.ingress.clone(),
verification: verification_ingress,
disk_guard: disk_guard.clone(),
backup_stats,
diagnostics: diagnostics.handle(),
config: config_service,
http_stats,
},
api_cancel.clone(),
));
let run_duration = async {
match config.run_duration_secs {
Some(seconds) => tokio::time::sleep(Duration::from_secs(seconds)).await,
None => std::future::pending().await,
}
};
tokio::pin!(run_duration);
let run_result = tokio::select! {
result = server.start() => result.map_err(AppError::from),
_ = shutdown::signal() => {
tracing::info!("收到退出信号");
Ok(())
}
_ = &mut run_duration => {
tracing::info!("达到配置的运行时长");
Ok(())
}
fatal = &mut persistence.fatal => {
let message = fatal.unwrap_or_else(|_| "持久化 worker 意外停止".to_owned());
Err(AppError::PersistenceWorker(message))
}
fatal = &mut index_fatal => {
let message = fatal.unwrap_or_else(|_| "索引 worker 意外停止".to_owned());
Err(AppError::IndexWorker(message))
}
fatal = &mut verification_fatal => {
let message = fatal.unwrap_or_else(|_| "可用性验证 worker 意外停止".to_owned());
Err(AppError::VerificationWorker(message))
}
result = &mut api_task => {
match result {
Ok(Ok(())) => Err(AppError::Config("HTTP 服务意外停止".to_owned())),
Ok(Err(error)) => Err(AppError::Io(error)),
Err(error) => Err(AppError::Config(format!("HTTP 服务任务异常: {error}"))),
}
}
};
let mut shutdown_error = None;
diagnostics.request_shutdown();
verification_cancel.cancel();
backup_cancel.cancel();
server.shutdown();
if let Some(task) = backup_task {
let _ = task.await;
}
drop(verification_fatal_guard);
if let Some(task) = verification_task {
match task.await {
Ok(Ok(())) => {}
Ok(Err(error)) => {
remember_shutdown_error(&mut shutdown_error, AppError::VerificationWorker(error));
}
Err(error) => {
remember_shutdown_error(
&mut shutdown_error,
AppError::VerificationWorker(error.to_string()),
);
}
}
}
disk_cancel.cancel();
let _ = disk_task.await;
monitor_cancel.cancel();
let _ = monitor.await;
if let Err(error) = persistence.close_and_join().await {
remember_shutdown_error(&mut shutdown_error, error);
}
index_cancel.cancel();
match index_task.await {
Ok(Ok(())) => {}
Ok(Err(error)) => {
remember_shutdown_error(&mut shutdown_error, AppError::IndexWorker(error));
}
Err(error) => {
remember_shutdown_error(
&mut shutdown_error,
AppError::IndexWorker(error.to_string()),
);
}
}
if let Err(error) = diagnostics.shutdown().await {
remember_shutdown_error(
&mut shutdown_error,
AppError::Diagnostics(error.to_string()),
);
}
api_cancel.cancel();
if !api_task.is_finished() {
match api_task.await {
Ok(Ok(())) => {}
Ok(Err(error)) => {
remember_shutdown_error(&mut shutdown_error, AppError::Io(error));
}
Err(error) => {
remember_shutdown_error(
&mut shutdown_error,
AppError::Config(format!("HTTP 服务任务异常: {error}")),
);
}
}
}
if let Some(error) = shutdown_error {
if run_result.is_ok() {
return Err(error);
}
tracing::error!(%error, "关闭阶段发生附加错误");
}
tracing::info!("dht-search 已安全停止");
run_result
}
fn remember_shutdown_error(slot: &mut Option<AppError>, error: AppError) {
if slot.is_none() {
*slot = Some(error);
} else {
tracing::error!(%error, "关闭阶段发生附加错误");
}
}
fn unix_timestamp() -> u64 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap_or_default()
.as_secs()
}
+500
View File
@@ -0,0 +1,500 @@
// 负责在线创建 RocksDB 检查点限制备份数量并执行带旧库保留的离线恢复
use std::{
fs::{self, File, OpenOptions},
path::{Path, PathBuf},
sync::{
Arc,
atomic::{AtomicU64, Ordering},
},
time::{Duration, Instant, SystemTime, UNIX_EPOCH},
};
use crate::storage::{CheckpointSummary, RocksTorrentRepository, StorageError};
use fs2::FileExt;
use tokio::task::JoinHandle;
use tokio_util::sync::CancellationToken;
use crate::{config::BackupConfig, disk_guard::DiskGuard};
const CHECKPOINT_PREFIX: &str = "checkpoint-";
const CHECKPOINT_DIGITS: usize = 20;
const DATA_LOCK_FILE: &str = ".dht-search.lock";
pub(crate) struct DataDirectoryLock {
_file: File,
}
#[derive(Clone, Default)]
pub(crate) struct BackupStats {
inner: Arc<BackupStatsInner>,
}
#[derive(Default)]
struct BackupStatsInner {
created: AtomicU64,
failed: AtomicU64,
skipped: AtomicU64,
pruned: AtomicU64,
last_success_at: AtomicU64,
last_duration_millis: AtomicU64,
latest_records: AtomicU64,
}
#[derive(Debug, Clone, Copy)]
pub(crate) struct BackupSnapshot {
pub(crate) created: u64,
pub(crate) failed: u64,
pub(crate) skipped: u64,
pub(crate) pruned: u64,
pub(crate) last_success_at: Option<u64>,
pub(crate) last_duration_millis: u64,
pub(crate) latest_records: u64,
}
#[derive(Debug)]
pub(crate) struct RestoreOutcome {
pub(crate) records: u64,
pub(crate) previous_database: Option<PathBuf>,
}
#[derive(Debug, thiserror::Error)]
pub(crate) enum BackupError {
#[error("备份文件操作失败: {0}")]
Io(#[from] std::io::Error),
#[error("RocksDB 检查点操作失败: {0}")]
Storage(#[from] StorageError),
#[error("数据目录正被另一个服务或恢复进程使用: {0}")]
DataDirectoryLocked(PathBuf),
#[error("检查点路径必须是独立目录且不能位于当前 RocksDB 内部")]
UnsafeCheckpointPath,
#[error("检查点包含不支持的符号链接或特殊文件: {0}")]
UnsafeCheckpointEntry(PathBuf),
#[error("无法生成唯一的检查点目录名")]
CheckpointNameExhausted,
#[error("恢复暂存目录已经存在: {0}")]
RestoreStagingExists(PathBuf),
#[error("数据库目录切换失败且旧数据库回滚失败: {0}")]
RestoreRollbackFailed(String),
}
pub(crate) fn acquire_data_lock(data_dir: &Path) -> Result<DataDirectoryLock, BackupError> {
fs::create_dir_all(data_dir)?;
let path = data_dir.join(DATA_LOCK_FILE);
let file = OpenOptions::new()
.create(true)
.truncate(false)
.read(true)
.write(true)
.open(&path)?;
file.try_lock_exclusive()
.map_err(|_| BackupError::DataDirectoryLocked(path))?;
Ok(DataDirectoryLock { _file: file })
}
pub(crate) fn start(
repository: Arc<RocksTorrentRepository>,
config: BackupConfig,
disk_guard: DiskGuard,
cancel: CancellationToken,
) -> (BackupStats, JoinHandle<()>) {
let stats = BackupStats::default();
let task_stats = stats.clone();
let task = tokio::spawn(async move {
run(repository, config, disk_guard, task_stats, cancel).await;
});
(stats, task)
}
async fn run(
repository: Arc<RocksTorrentRepository>,
config: BackupConfig,
disk_guard: DiskGuard,
stats: BackupStats,
cancel: CancellationToken,
) {
if config.create_on_start {
create_one(
repository.clone(),
config.clone(),
disk_guard.clone(),
stats.clone(),
)
.await;
}
let start = tokio::time::Instant::now() + Duration::from_secs(config.interval_secs);
let mut ticker = tokio::time::interval_at(start, Duration::from_secs(config.interval_secs));
ticker.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
loop {
tokio::select! {
_ = cancel.cancelled() => break,
_ = ticker.tick() => {
create_one(
repository.clone(),
config.clone(),
disk_guard.clone(),
stats.clone(),
).await;
}
}
}
}
async fn create_one(
repository: Arc<RocksTorrentRepository>,
config: BackupConfig,
disk_guard: DiskGuard,
stats: BackupStats,
) {
let Some(permit) = disk_guard.begin_new_write() else {
stats.inner.skipped.fetch_add(1, Ordering::Relaxed);
tracing::warn!("磁盘处于保护状态并跳过 RocksDB 检查点");
return;
};
if let Err(error) = fs::create_dir_all(&config.directory) {
stats.inner.failed.fetch_add(1, Ordering::Relaxed);
tracing::error!(%error, "无法创建检查点目录");
return;
}
let minimum_free_bytes = disk_guard.snapshot().minimum_free_bytes;
match fs2::available_space(&config.directory) {
Ok(available) if available < minimum_free_bytes => {
stats.inner.skipped.fetch_add(1, Ordering::Relaxed);
tracing::warn!(
available,
minimum_free_bytes,
"备份磁盘空间不足并跳过检查点"
);
return;
}
Err(error) => {
stats.inner.skipped.fetch_add(1, Ordering::Relaxed);
tracing::warn!(%error, "无法确认备份磁盘剩余空间并跳过检查点");
return;
}
Ok(_) => {}
}
let started = Instant::now();
let result = tokio::task::spawn_blocking(move || {
let _permit = permit;
create_checkpoint(&repository, &config, unix_timestamp_millis())
})
.await;
match result {
Ok(Ok((summary, pruned))) => {
let now = unix_timestamp();
stats.inner.created.fetch_add(1, Ordering::Relaxed);
stats.inner.pruned.fetch_add(pruned, Ordering::Relaxed);
stats.inner.last_success_at.store(now, Ordering::Relaxed);
stats.inner.last_duration_millis.store(
started.elapsed().as_millis().min(u128::from(u64::MAX)) as u64,
Ordering::Relaxed,
);
stats
.inner
.latest_records
.store(summary.records, Ordering::Relaxed);
tracing::info!(records = summary.records, pruned, "RocksDB 检查点创建完成");
}
Ok(Err(error)) => {
stats.inner.failed.fetch_add(1, Ordering::Relaxed);
tracing::error!(%error, "RocksDB 检查点创建失败");
}
Err(error) => {
stats.inner.failed.fetch_add(1, Ordering::Relaxed);
tracing::error!(%error, "RocksDB 检查点任务异常");
}
}
}
fn create_checkpoint(
repository: &RocksTorrentRepository,
config: &BackupConfig,
timestamp_millis: u128,
) -> Result<(CheckpointSummary, u64), BackupError> {
fs::create_dir_all(&config.directory)?;
let checkpoint_path = unique_checkpoint_path(&config.directory, timestamp_millis)?;
repository.create_checkpoint(&checkpoint_path)?;
let summary = RocksTorrentRepository::validate_checkpoint(&checkpoint_path)?;
let pruned = prune_checkpoints(&config.directory, config.retain_checkpoints)?;
Ok((summary, pruned))
}
fn unique_checkpoint_path(
directory: &Path,
timestamp_millis: u128,
) -> Result<PathBuf, BackupError> {
for offset in 0..1_000_u128 {
let value = timestamp_millis.saturating_add(offset);
let path = directory.join(format!("{CHECKPOINT_PREFIX}{value:0CHECKPOINT_DIGITS$}"));
if !path.exists() {
return Ok(path);
}
}
Err(BackupError::CheckpointNameExhausted)
}
fn prune_checkpoints(directory: &Path, retain: usize) -> Result<u64, BackupError> {
let mut checkpoints = checkpoint_directories(directory)?;
checkpoints.sort_unstable_by_key(|(timestamp, _)| *timestamp);
let remove_count = checkpoints.len().saturating_sub(retain);
for (_, path) in checkpoints.into_iter().take(remove_count) {
fs::remove_dir_all(path)?;
}
Ok(remove_count as u64)
}
fn checkpoint_directories(directory: &Path) -> Result<Vec<(u128, PathBuf)>, BackupError> {
let mut checkpoints = Vec::new();
for entry in fs::read_dir(directory)? {
let entry = entry?;
let file_type = entry.file_type()?;
if !file_type.is_dir() || file_type.is_symlink() {
continue;
}
let name = entry.file_name();
let Some(name) = name.to_str() else { continue };
let Some(timestamp) = parse_checkpoint_name(name) else {
continue;
};
checkpoints.push((timestamp, entry.path()));
}
Ok(checkpoints)
}
fn parse_checkpoint_name(name: &str) -> Option<u128> {
let digits = name.strip_prefix(CHECKPOINT_PREFIX)?;
if digits.len() != CHECKPOINT_DIGITS || !digits.bytes().all(|byte| byte.is_ascii_digit()) {
return None;
}
digits.parse().ok()
}
pub(crate) fn restore(data_dir: &Path, checkpoint: &Path) -> Result<RestoreOutcome, BackupError> {
let _lock = acquire_data_lock(data_dir)?;
let source = dunce::canonicalize(checkpoint)?;
let database_path = data_dir.join("rocksdb");
let database_absolute = absolute_path(&database_path)?;
if !source.is_dir() || source == database_absolute || source.starts_with(&database_absolute) {
return Err(BackupError::UnsafeCheckpointPath);
}
let source_summary = RocksTorrentRepository::validate_checkpoint(&source)?;
let timestamp = unix_timestamp_millis();
let staging = data_dir.join(format!("rocksdb.restore-staging-{timestamp:020}"));
if staging.exists() {
return Err(BackupError::RestoreStagingExists(staging));
}
copy_directory(&source, &staging)?;
let staged_summary = RocksTorrentRepository::validate_checkpoint(&staging)?;
if staged_summary != source_summary {
return Err(BackupError::Storage(StorageError::CorruptContentGroup));
}
let tantivy_path = data_dir.join("tantivy");
if tantivy_path.exists() {
fs::remove_dir_all(&tantivy_path)?;
}
let previous_database = if database_path.exists() {
let previous = data_dir.join(format!("rocksdb.pre-restore-{timestamp:020}"));
fs::rename(&database_path, &previous)?;
Some(previous)
} else {
None
};
if let Err(error) = fs::rename(&staging, &database_path) {
if let Some(previous) = &previous_database
&& let Err(rollback) = fs::rename(previous, &database_path)
{
return Err(BackupError::RestoreRollbackFailed(format!(
"切换错误 {error}; 回滚错误 {rollback}"
)));
}
return Err(BackupError::Io(error));
}
Ok(RestoreOutcome {
records: staged_summary.records,
previous_database,
})
}
fn copy_directory(source: &Path, destination: &Path) -> Result<(), BackupError> {
fs::create_dir(destination)?;
for entry in fs::read_dir(source)? {
let entry = entry?;
let source_path = entry.path();
let destination_path = destination.join(entry.file_name());
let file_type = entry.file_type()?;
if file_type.is_symlink() {
return Err(BackupError::UnsafeCheckpointEntry(source_path));
}
if file_type.is_dir() {
copy_directory(&source_path, &destination_path)?;
} else if file_type.is_file() {
fs::copy(&source_path, &destination_path)?;
} else {
return Err(BackupError::UnsafeCheckpointEntry(source_path));
}
}
Ok(())
}
fn absolute_path(path: &Path) -> Result<PathBuf, std::io::Error> {
if path.is_absolute() {
Ok(path.to_path_buf())
} else {
Ok(std::env::current_dir()?.join(path))
}
}
impl BackupStats {
pub(crate) fn snapshot(&self) -> BackupSnapshot {
let last_success_at = self.inner.last_success_at.load(Ordering::Relaxed);
BackupSnapshot {
created: self.inner.created.load(Ordering::Relaxed),
failed: self.inner.failed.load(Ordering::Relaxed),
skipped: self.inner.skipped.load(Ordering::Relaxed),
pruned: self.inner.pruned.load(Ordering::Relaxed),
last_success_at: (last_success_at != 0).then_some(last_success_at),
last_duration_millis: self.inner.last_duration_millis.load(Ordering::Relaxed),
latest_records: self.inner.latest_records.load(Ordering::Relaxed),
}
}
}
fn unix_timestamp() -> u64 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap_or_default()
.as_secs()
}
fn unix_timestamp_millis() -> u128 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap_or_default()
.as_millis()
}
#[cfg(test)]
mod tests {
use crate::{
domain::{MetadataCandidate, TorrentFile, TorrentRecord},
search::SearchEngine,
storage::TorrentRepository,
};
use tempfile::TempDir;
use super::*;
fn record(byte: u8, name: &str) -> TorrentRecord {
TorrentRecord::try_from(MetadataCandidate {
info_hash: format!("{byte:02x}").repeat(20),
name: name.into(),
total_size: 42,
files: vec![TorrentFile {
path: format!("{name}.bin"),
size: 42,
}],
piece_length: 16_384,
source_peers: Vec::new(),
timestamp: 10,
})
.unwrap()
}
#[test]
fn data_directory_lock_rejects_a_second_owner() {
let directory = TempDir::new().unwrap();
let first = acquire_data_lock(directory.path()).unwrap();
assert!(matches!(
acquire_data_lock(directory.path()),
Err(BackupError::DataDirectoryLocked(_))
));
drop(first);
acquire_data_lock(directory.path()).unwrap();
}
#[test]
fn retention_prunes_only_recognized_checkpoint_directories() {
let directory = TempDir::new().unwrap();
for timestamp in 1..=4_u128 {
fs::create_dir(directory.path().join(format!(
"{CHECKPOINT_PREFIX}{timestamp:0CHECKPOINT_DIGITS$}"
)))
.unwrap();
}
fs::create_dir(directory.path().join("checkpoint-manual")).unwrap();
fs::write(
directory
.path()
.join(format!("{CHECKPOINT_PREFIX}{:0CHECKPOINT_DIGITS$}", 5)),
"not a directory",
)
.unwrap();
assert_eq!(prune_checkpoints(directory.path(), 2).unwrap(), 2);
assert_eq!(checkpoint_directories(directory.path()).unwrap().len(), 2);
assert!(directory.path().join("checkpoint-manual").is_dir());
}
#[test]
fn repeated_checkpoints_keep_only_the_configured_latest_snapshots() {
let directory = TempDir::new().unwrap();
let repository = RocksTorrentRepository::open(directory.path().join("rocksdb")).unwrap();
let config = BackupConfig {
enabled: true,
directory: directory.path().join("backups"),
interval_secs: 1,
retain_checkpoints: 2,
create_on_start: true,
};
for index in 1..=3_u8 {
repository
.upsert(record(index, &format!("record-{index}")))
.unwrap();
let (summary, _) = create_checkpoint(&repository, &config, u128::from(index)).unwrap();
assert_eq!(summary.records, u64::from(index));
}
let checkpoints = checkpoint_directories(&config.directory).unwrap();
assert_eq!(checkpoints.len(), 2);
assert!(checkpoints.iter().all(|(timestamp, _)| *timestamp >= 2));
}
#[test]
fn restore_keeps_previous_database_and_forces_search_rebuild() {
let directory = TempDir::new().unwrap();
let data_dir = directory.path().join("data");
fs::create_dir_all(&data_dir).unwrap();
let database_path = data_dir.join("rocksdb");
let checkpoint_path = directory.path().join("checkpoint");
let first = record(1, "checkpoint-first");
let second = record(2, "newer-second");
{
let repository = RocksTorrentRepository::open(&database_path).unwrap();
repository.upsert(first.clone()).unwrap();
repository.create_checkpoint(&checkpoint_path).unwrap();
repository.upsert(second.clone()).unwrap();
}
fs::create_dir(data_dir.join("tantivy")).unwrap();
fs::write(data_dir.join("tantivy/old-index"), "derived").unwrap();
let outcome = restore(&data_dir, &checkpoint_path).unwrap();
assert_eq!(outcome.records, 1);
assert!(!data_dir.join("tantivy").exists());
let restored = RocksTorrentRepository::open(&database_path).unwrap();
assert!(restored.get(first.info_hash).unwrap().is_some());
assert!(restored.get(second.info_hash).unwrap().is_none());
let previous_path = outcome.previous_database.unwrap();
let previous = RocksTorrentRepository::open(previous_path).unwrap();
assert!(previous.get(first.info_hash).unwrap().is_some());
assert!(previous.get(second.info_hash).unwrap().is_some());
drop(previous);
let search = SearchEngine::open(data_dir.join("tantivy")).unwrap();
restored.prepare_full_reindex().unwrap();
while search.index_pending(&restored, 100, 20).unwrap() > 0 {}
assert_eq!(search.search("checkpoint-first", 0, 10).unwrap().total, 1);
assert_eq!(search.search("newer-second", 0, 10).unwrap().total, 0);
}
}
+22
View File
@@ -0,0 +1,22 @@
// 负责启动规模基准并把具体阶段委托给独立组件
use std::error::Error;
use clap::Parser;
#[path = "dht-benchmark/config.rs"]
mod config;
#[path = "dht-benchmark/dataset.rs"]
mod dataset;
#[path = "dht-benchmark/metrics.rs"]
mod metrics;
#[path = "dht-benchmark/report.rs"]
mod report;
#[path = "dht-benchmark/runner.rs"]
mod runner;
#[path = "dht-benchmark/workload.rs"]
mod workload;
fn main() -> Result<(), Box<dyn Error>> {
runner::run(config::Args::parse())
}
@@ -0,0 +1,60 @@
// 负责解析和校验规模基准命令行参数
use std::{error::Error, path::PathBuf};
use clap::Parser;
const MAX_RECORDS: usize = 10_000_000;
#[derive(Debug, Parser)]
#[command(name = "dht-benchmark", about = "RocksDB 和 Tantivy 端到端规模基准")]
pub(crate) struct Args {
#[arg(long, default_value_t = 10_000)]
pub(crate) records: usize,
#[arg(long, default_value_t = 1_000)]
pub(crate) generation_batch_size: usize,
#[arg(long, default_value_t = 1_000)]
pub(crate) index_batch_size: usize,
#[arg(long, default_value_t = 20)]
pub(crate) index_max_retries: usize,
#[arg(long, default_value_t = 50)]
pub(crate) query_iterations: usize,
#[arg(long, default_value_t = 5)]
pub(crate) query_warmup: usize,
#[arg(long, default_value_t = 10)]
pub(crate) duplicate_every: usize,
#[arg(long, default_value = "benchmark-data")]
pub(crate) output_dir: PathBuf,
#[arg(long)]
pub(crate) cleanup: bool,
}
impl Args {
pub(crate) fn validate(&self) -> Result<(), Box<dyn Error>> {
if !(100..=MAX_RECORDS).contains(&self.records) {
return Err(format!("records 必须在 100 到 {MAX_RECORDS} 之间").into());
}
if self.generation_batch_size == 0
|| self.index_batch_size == 0
|| self.index_max_retries == 0
|| self.query_iterations == 0
{
return Err("批量大小重试次数和查询次数必须大于零".into());
}
if self.duplicate_every == 1 {
return Err("duplicate-every 必须是零或至少为二".into());
}
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn rejects_ambiguous_duplicate_interval() {
let args = Args::parse_from(["benchmark", "--duplicate-every", "1"]);
assert!(args.validate().is_err());
}
}
@@ -0,0 +1,84 @@
// 负责确定性生成种子记录和可控比例的相同内容变体
use std::error::Error;
use dht_search::domain::{MetadataCandidate, MetadataLimits, TorrentFile, TorrentRecord};
pub(crate) const BASE_TIMESTAMP: u64 = 1_700_000_000;
pub(crate) fn generate_record(
index: usize,
duplicate_every: usize,
) -> Result<TorrentRecord, Box<dyn Error>> {
let content_id = content_id(index, duplicate_every);
let file_count = 1 + content_id % 4;
let mut files = Vec::with_capacity(file_count);
let main_size = 64 * 1024 * 1024 + (content_id as u64 % 8_192) * 1_048_576;
files.push(TorrentFile {
path: format!("media/category_{}/item_{content_id}.mkv", content_id % 100),
size: main_size,
});
for part in 1..file_count {
files.push(TorrentFile {
path: format!("docs/item_{content_id}/part_{part}.txt"),
size: 1_024 + (content_id as u64 + part as u64) % 65_536,
});
}
let total_size = files.iter().try_fold(0_u64, |total, file| {
total
.checked_add(file.size)
.ok_or("生成数据的文件总大小溢出")
})?;
let name = match content_id % 4 {
0 => format!("流浪地球 第{content_id}集 1080p"),
1 => format!("Ubuntu Linux Desktop Build {content_id}"),
2 => format!("Nature Documentary 4K Episode {content_id}"),
_ => format!("Open Source Archive Collection {content_id}"),
};
let digest = blake3::hash(&(index as u64).to_be_bytes());
let info_hash = hex::encode(&digest.as_bytes()[..20]);
TorrentRecord::try_from_with_limits(
MetadataCandidate {
info_hash,
name,
total_size,
files,
piece_length: 16_384,
source_peers: Vec::new(),
timestamp: BASE_TIMESTAMP.saturating_add(index as u64),
},
MetadataLimits::default(),
)
.map_err(Into::into)
}
pub(crate) fn expected_document_count(records: usize, duplicate_every: usize) -> usize {
if duplicate_every >= 2 {
records.saturating_sub(records / duplicate_every)
} else {
records
}
}
fn content_id(index: usize, duplicate_every: usize) -> usize {
if duplicate_every >= 2 && (index + 1).is_multiple_of(duplicate_every) {
index.saturating_sub(1)
} else {
index
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn generator_is_deterministic_and_creates_requested_duplicates() {
let first = generate_record(8, 10).unwrap();
let duplicate = generate_record(9, 10).unwrap();
assert_ne!(first.info_hash, duplicate.info_hash);
assert_eq!(first.content_key, duplicate.content_key);
assert_eq!(generate_record(8, 10).unwrap(), first);
assert_eq!(expected_document_count(100, 10), 90);
}
}
+146
View File
@@ -0,0 +1,146 @@
// 负责采样查询延迟计算分位数磁盘占用和进程峰值内存
use std::{
error::Error,
fs,
path::Path,
time::{Duration, Instant},
};
use dht_search::search::SearchEngine;
use serde::Serialize;
use super::workload::QueryCase;
#[derive(Debug, Serialize)]
pub(crate) struct QueryReport {
pub(crate) name: String,
pub(crate) iterations: usize,
pub(crate) result_count: usize,
pub(crate) mean_micros: u64,
pub(crate) p50_micros: u64,
pub(crate) p95_micros: u64,
pub(crate) p99_micros: u64,
}
pub(crate) fn benchmark_query(
search: &SearchEngine,
case: QueryCase,
warmup: usize,
iterations: usize,
) -> Result<QueryReport, Box<dyn Error>> {
for _ in 0..warmup {
search.search_with(case.options.clone())?;
}
let mut samples = Vec::with_capacity(iterations);
let mut result_count = 0;
for _ in 0..iterations {
let started = Instant::now();
let page = search.search_with(case.options.clone())?;
samples.push(duration_micros(started.elapsed()));
result_count = page.total;
}
samples.sort_unstable();
let mean = samples.iter().copied().sum::<u64>() / samples.len() as u64;
Ok(QueryReport {
name: case.name.to_owned(),
iterations,
result_count,
mean_micros: mean,
p50_micros: percentile(&samples, 50),
p95_micros: percentile(&samples, 95),
p99_micros: percentile(&samples, 99),
})
}
pub(crate) fn directory_size(path: &Path) -> Result<u64, std::io::Error> {
let mut total = 0_u64;
let mut pending = vec![path.to_path_buf()];
while let Some(directory) = pending.pop() {
for entry in fs::read_dir(directory)? {
let entry = entry?;
let metadata = entry.metadata()?;
if metadata.is_dir() {
pending.push(entry.path());
} else if metadata.is_file() {
total = total.saturating_add(metadata.len());
}
}
}
Ok(total)
}
pub(crate) fn rate(items: u64, duration: Duration) -> f64 {
if duration.is_zero() {
0.0
} else {
items as f64 / duration.as_secs_f64()
}
}
pub(crate) fn ratio(bytes: u64, items: u64) -> f64 {
if items == 0 {
0.0
} else {
bytes as f64 / items as f64
}
}
fn percentile(sorted: &[u64], percentile: usize) -> u64 {
let rank = sorted
.len()
.saturating_mul(percentile)
.div_ceil(100)
.saturating_sub(1)
.min(sorted.len().saturating_sub(1));
sorted[rank]
}
fn duration_micros(duration: Duration) -> u64 {
duration.as_micros().min(u128::from(u64::MAX)) as u64
}
#[cfg(windows)]
pub(crate) fn peak_memory_bytes() -> Option<u64> {
use std::mem::{size_of, zeroed};
use windows_sys::Win32::System::{
ProcessStatus::{GetProcessMemoryInfo, PROCESS_MEMORY_COUNTERS},
Threading::GetCurrentProcess,
};
let mut counters: PROCESS_MEMORY_COUNTERS = unsafe { zeroed() };
let result = unsafe {
GetProcessMemoryInfo(
GetCurrentProcess(),
&mut counters,
size_of::<PROCESS_MEMORY_COUNTERS>() as u32,
)
};
(result != 0).then_some(counters.PeakWorkingSetSize as u64)
}
#[cfg(target_os = "linux")]
pub(crate) fn peak_memory_bytes() -> Option<u64> {
let status = fs::read_to_string("/proc/self/status").ok()?;
let line = status.lines().find(|line| line.starts_with("VmHWM:"))?;
let kibibytes = line.split_whitespace().nth(1)?.parse::<u64>().ok()?;
kibibytes.checked_mul(1024)
}
#[cfg(not(any(windows, target_os = "linux")))]
pub(crate) fn peak_memory_bytes() -> Option<u64> {
None
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn percentiles_use_nearest_rank() {
let samples: Vec<_> = (1..=100).collect();
assert_eq!(percentile(&samples, 50), 50);
assert_eq!(percentile(&samples, 95), 95);
assert_eq!(percentile(&samples, 99), 99);
}
}
+109
View File
@@ -0,0 +1,109 @@
// 负责序列化并展示规模基准的稳定报告格式
use serde::Serialize;
use super::metrics::QueryReport;
#[derive(Debug, Serialize)]
pub(crate) struct BenchmarkReport {
pub(crate) generated_at: u64,
pub(crate) build_profile: String,
pub(crate) target: String,
pub(crate) logical_cpus: usize,
pub(crate) records: usize,
pub(crate) indexed_documents: u64,
pub(crate) duplicate_every: usize,
pub(crate) generation_seconds: f64,
pub(crate) rocksdb_write_seconds: f64,
pub(crate) rocksdb_records_per_second: f64,
pub(crate) index_seconds: f64,
pub(crate) index_documents_per_second: f64,
pub(crate) index_transient_retries: usize,
pub(crate) index_retry_wait_seconds: f64,
pub(crate) rocksdb_bytes: u64,
pub(crate) tantivy_bytes: u64,
pub(crate) total_bytes: u64,
pub(crate) rocksdb_bytes_per_record: f64,
pub(crate) tantivy_bytes_per_document: f64,
pub(crate) peak_memory_bytes: Option<u64>,
pub(crate) queries: Vec<QueryReport>,
}
pub(crate) fn print_query(report: &QueryReport) {
println!(
"查询 {:<16} 结果 {:>8} P50 {:>8} µs P95 {:>8} µs P99 {:>8} µs",
report.name,
format_integer(report.result_count as u64),
report.p50_micros,
report.p95_micros,
report.p99_micros
);
}
pub(crate) fn print_summary(report: &BenchmarkReport) {
println!();
println!("基准汇总");
println!("生成耗时: {:.3}", report.generation_seconds);
println!(
"RocksDB 写入: {:.3}{:.0} 条/秒",
report.rocksdb_write_seconds, report.rocksdb_records_per_second
);
println!(
"Tantivy 索引: {:.3}{:.0} 文档/秒",
report.index_seconds, report.index_documents_per_second
);
println!(
"Tantivy 临时 IO 重试: {} 次 等待 {:.3}",
report.index_transient_retries, report.index_retry_wait_seconds
);
println!(
"RocksDB: {} 平均 {:.1} 字节/记录",
format_bytes(report.rocksdb_bytes),
report.rocksdb_bytes_per_record
);
println!(
"Tantivy: {} 平均 {:.1} 字节/文档",
format_bytes(report.tantivy_bytes),
report.tantivy_bytes_per_document
);
println!("合计磁盘: {}", format_bytes(report.total_bytes));
if let Some(bytes) = report.peak_memory_bytes {
println!("进程峰值内存: {}", format_bytes(bytes));
} else {
println!("进程峰值内存: 当前平台暂不支持读取");
}
}
pub(crate) fn format_integer(value: u64) -> String {
let digits = value.to_string();
let mut formatted = String::with_capacity(digits.len() + digits.len() / 3);
for (index, character) in digits.chars().enumerate() {
if index > 0 && (digits.len() - index).is_multiple_of(3) {
formatted.push(',');
}
formatted.push(character);
}
formatted
}
fn format_bytes(bytes: u64) -> String {
const UNITS: &[&str] = &["B", "KiB", "MiB", "GiB", "TiB"];
let mut value = bytes as f64;
let mut unit = 0;
while value >= 1024.0 && unit + 1 < UNITS.len() {
value /= 1024.0;
unit += 1;
}
format!("{value:.2} {}", UNITS[unit])
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn byte_and_integer_formatting_are_stable() {
assert_eq!(format_integer(1_234_567), "1,234,567");
assert_eq!(format_bytes(1024), "1.00 KiB");
}
}
+227
View File
@@ -0,0 +1,227 @@
// 负责按生成写入索引查询报告顺序编排一次完整规模基准
use std::{
error::Error,
fs,
time::{Duration, Instant, SystemTime, UNIX_EPOCH},
};
use dht_search::{
search::SearchEngine,
storage::{RocksTorrentRepository, TorrentRepository},
};
use super::{
config::Args,
dataset::{BASE_TIMESTAMP, expected_document_count, generate_record},
metrics::{benchmark_query, directory_size, peak_memory_bytes, rate, ratio},
report::{BenchmarkReport, format_integer, print_query, print_summary},
workload::query_cases,
};
pub(crate) fn run(args: Args) -> Result<(), Box<dyn Error>> {
args.validate()?;
let generated_at = unix_timestamp();
let build_profile = if cfg!(debug_assertions) {
"debug"
} else {
"release"
};
let run_name = format!("records-{}-{generated_at}", args.records);
let runs_dir = args.output_dir.join("runs");
let reports_dir = args.output_dir.join("reports");
let run_dir = runs_dir.join(&run_name);
if run_dir.exists() {
return Err(format!("基准目录已经存在 {}", run_dir.display()).into());
}
fs::create_dir_all(&run_dir)?;
fs::create_dir_all(&reports_dir)?;
println!("DHT Search 规模基准");
println!("构建模式: {build_profile}");
if cfg!(debug_assertions) {
println!("警告: debug 模式仅用于流程验证 性能结论必须使用 --release");
}
println!("数据量: {}", format_integer(args.records as u64));
println!("运行目录: {}", run_dir.display());
let rocksdb_dir = run_dir.join("rocksdb");
let tantivy_dir = run_dir.join("tantivy");
let repository = RocksTorrentRepository::open(&rocksdb_dir)?;
let (generation_duration, write_duration) = populate(&repository, &args)?;
let search = SearchEngine::open(&tantivy_dir)?;
let index_result = build_index(&repository, &search, &args)?;
let mut query_reports = Vec::new();
for case in query_cases(args.records, args.duplicate_every)? {
let report = benchmark_query(&search, case, args.query_warmup, args.query_iterations)?;
print_query(&report);
query_reports.push(report);
}
let indexed_documents = search.num_docs();
let peak_memory_bytes = peak_memory_bytes();
drop(search);
drop(repository);
let rocksdb_bytes = directory_size(&rocksdb_dir)?;
let tantivy_bytes = directory_size(&tantivy_dir)?;
let report = BenchmarkReport {
generated_at,
build_profile: build_profile.to_owned(),
target: format!("{}-{}", std::env::consts::OS, std::env::consts::ARCH),
logical_cpus: std::thread::available_parallelism().map_or(1, usize::from),
records: args.records,
indexed_documents,
duplicate_every: args.duplicate_every,
generation_seconds: generation_duration.as_secs_f64(),
rocksdb_write_seconds: write_duration.as_secs_f64(),
rocksdb_records_per_second: rate(args.records as u64, write_duration),
index_seconds: index_result.duration.as_secs_f64(),
index_documents_per_second: rate(indexed_documents, index_result.duration),
index_transient_retries: index_result.transient_retries,
index_retry_wait_seconds: index_result.retry_wait.as_secs_f64(),
rocksdb_bytes,
tantivy_bytes,
total_bytes: rocksdb_bytes.saturating_add(tantivy_bytes),
rocksdb_bytes_per_record: ratio(rocksdb_bytes, args.records as u64),
tantivy_bytes_per_document: ratio(tantivy_bytes, indexed_documents),
peak_memory_bytes,
queries: query_reports,
};
print_summary(&report);
let report_path = reports_dir.join(format!("{run_name}.json"));
fs::write(&report_path, serde_json::to_vec_pretty(&report)?)?;
println!("报告: {}", report_path.display());
if args.cleanup {
fs::remove_dir_all(&run_dir)?;
println!("已清理本次基准数据: {}", run_dir.display());
} else {
println!("基准数据已保留 使用 --cleanup 可在完成后自动删除");
}
Ok(())
}
fn populate(
repository: &RocksTorrentRepository,
args: &Args,
) -> Result<(Duration, Duration), Box<dyn Error>> {
let mut generation_duration = Duration::ZERO;
let mut write_duration = Duration::ZERO;
let progress_step = (args.records / 20).max(1);
for batch_start in (0..args.records).step_by(args.generation_batch_size) {
let batch_end = (batch_start + args.generation_batch_size).min(args.records);
let generation_started = Instant::now();
let records: Vec<_> = (batch_start..batch_end)
.map(|index| generate_record(index, args.duplicate_every))
.collect::<Result<_, _>>()?;
generation_duration += generation_started.elapsed();
let write_started = Instant::now();
for record in records {
repository.upsert(record)?;
}
write_duration += write_started.elapsed();
if batch_end == args.records || batch_end / progress_step != batch_start / progress_step {
eprintln!(
"RocksDB 写入进度: {:>3}% ({}/{})",
batch_end.saturating_mul(100) / args.records,
format_integer(batch_end as u64),
format_integer(args.records as u64)
);
}
}
Ok((generation_duration, write_duration))
}
struct IndexResult {
duration: Duration,
transient_retries: usize,
retry_wait: Duration,
}
fn build_index(
repository: &RocksTorrentRepository,
search: &SearchEngine,
args: &Args,
) -> Result<IndexResult, Box<dyn Error>> {
let started = Instant::now();
let mut indexed_documents = 0_usize;
let mut transient_retries = 0_usize;
let mut retry_wait = Duration::ZERO;
let expected_documents = expected_document_count(args.records, args.duplicate_every);
let progress_step = (expected_documents / 20).max(1);
let mut next_progress = progress_step;
loop {
let mut consecutive_retries = 0_usize;
let indexed = loop {
match search.index_pending(
repository,
args.index_batch_size,
BASE_TIMESTAMP.saturating_add(args.records as u64),
) {
Ok(indexed) => break indexed,
Err(error) if error.is_retryable_io() => {
consecutive_retries = consecutive_retries.saturating_add(1);
transient_retries = transient_retries.saturating_add(1);
if consecutive_retries > args.index_max_retries {
return Err(format!(
"Tantivy 临时 IO 错误连续重试超过 {} 次: {error}",
args.index_max_retries
)
.into());
}
let delay = index_retry_delay(consecutive_retries);
retry_wait += delay;
eprintln!(
"Tantivy 临时 IO 错误 第 {consecutive_retries} 次重试 等待 {} ms: {error}",
delay.as_millis()
);
std::thread::sleep(delay);
}
Err(error) => return Err(Box::new(error)),
}
};
if indexed == 0 {
break;
}
indexed_documents = indexed_documents.saturating_add(indexed);
if indexed_documents >= next_progress || indexed_documents >= expected_documents {
eprintln!(
"Tantivy 索引进度: {:>3}% ({} 个内容文档)",
indexed_documents.saturating_mul(100) / expected_documents.max(1),
format_integer(indexed_documents as u64)
);
next_progress = next_progress.saturating_add(progress_step);
}
}
Ok(IndexResult {
duration: started.elapsed(),
transient_retries,
retry_wait,
})
}
fn index_retry_delay(retry: usize) -> Duration {
let exponent = retry.saturating_sub(1).min(5) as u32;
Duration::from_millis(250_u64.saturating_mul(2_u64.pow(exponent))).min(Duration::from_secs(10))
}
fn unix_timestamp() -> u64 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap_or_default()
.as_secs()
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn index_retry_delay_is_bounded() {
assert_eq!(index_retry_delay(1), Duration::from_millis(250));
assert_eq!(index_retry_delay(3), Duration::from_secs(1));
assert!(index_retry_delay(100) <= Duration::from_secs(10));
}
}
@@ -0,0 +1,83 @@
// 负责定义覆盖关键词路径哈希正则排序和过滤的固定查询工作负载
use std::error::Error;
use dht_search::search::{SearchMode, SearchOptions, SearchSort};
use super::dataset::generate_record;
pub(crate) struct QueryCase {
pub(crate) name: &'static str,
pub(crate) options: SearchOptions,
}
pub(crate) fn query_cases(
records: usize,
duplicate_every: usize,
) -> Result<Vec<QueryCase>, Box<dyn Error>> {
let exact_index = records.saturating_sub(1).min(42);
let exact_hash = generate_record(exact_index, duplicate_every)?
.info_hash
.to_string();
Ok(vec![
QueryCase {
name: "中文关键词",
options: SearchOptions {
query: "流浪地球".into(),
limit: 20,
..SearchOptions::default()
},
},
QueryCase {
name: "英文关键词",
options: SearchOptions {
query: "ubuntu desktop".into(),
limit: 20,
..SearchOptions::default()
},
},
QueryCase {
name: "文件路径片段",
options: SearchOptions {
query: "item_42".into(),
limit: 20,
..SearchOptions::default()
},
},
QueryCase {
name: "精确 infohash",
options: SearchOptions {
query: exact_hash,
limit: 20,
..SearchOptions::default()
},
},
QueryCase {
name: "有限状态正则",
options: SearchOptions {
query: "ubuntu.*desktop".into(),
mode: SearchMode::Regex,
limit: 20,
..SearchOptions::default()
},
},
QueryCase {
name: "最近收录排序",
options: SearchOptions {
limit: 20,
sort: Some(SearchSort::Latest),
..SearchOptions::default()
},
},
QueryCase {
name: "大小扩展名过滤",
options: SearchOptions {
min_size: Some(1024 * 1024 * 1024),
extension: Some("mkv".into()),
limit: 20,
sort: Some(SearchSort::SizeDesc),
..SearchOptions::default()
},
},
])
}
+222
View File
@@ -0,0 +1,222 @@
// 负责组合配置 DTO 持久化读取命令行覆盖和运行时解析
mod model;
mod runtime;
mod service;
mod store;
use std::{
path::{Path, PathBuf},
sync::Arc,
};
use clap::Parser;
use crate::error::AppError;
pub(crate) use model::{
AppConfigDto, BackupConfig, DiagnosticsConfig, DiskGuardConfig, LogRotation, LoggingConfig,
VerificationConfig,
};
pub(crate) use runtime::AppConfig;
pub(crate) use service::{ConfigService, ConfigServiceError, ConfigSnapshot, ConfigUpdateRequest};
pub(crate) use store::{ConfigStore, TomlConfigStore};
#[derive(Debug, Parser)]
#[command(name = "dht-search", version, about = "DHT 元数据采集和搜索服务")]
pub(crate) struct Cli {
#[arg(long, default_value = "dht-search.toml")]
config: PathBuf,
#[arg(long)]
data_dir: Option<PathBuf>,
#[arg(long)]
run_duration_secs: Option<u64>,
#[arg(long)]
restore_checkpoint: Option<PathBuf>,
}
#[derive(Debug)]
pub(crate) struct StartupConfig {
pub(crate) app: AppConfig,
pub(crate) config_service: ConfigService,
pub(crate) restore_checkpoint: Option<PathBuf>,
}
impl Cli {
pub(crate) fn load(self) -> Result<StartupConfig, AppError> {
let config_path = absolute_path(&self.config)?;
let store = Arc::new(TomlConfigStore::new(config_path.clone()));
let persisted = store.load()?.unwrap_or_default();
let mut dto = persisted.clone();
let mut command_line_overrides = Vec::new();
if let Some(data_dir) = self.data_dir {
dto.data_dir = data_dir;
command_line_overrides.push("data_dir".to_owned());
}
if self.run_duration_secs.is_some() {
dto.run_duration_secs = self.run_duration_secs;
command_line_overrides.push("run_duration_secs".to_owned());
}
let base = config_path
.parent()
.ok_or_else(|| AppError::Config("配置文件没有父目录".to_owned()))?;
let app = AppConfig::resolve(dto, base)?;
let config_service = ConfigService::new(store, persisted, command_line_overrides)?;
let restore_checkpoint = self
.restore_checkpoint
.map(|path| absolute_path(&path))
.transpose()?;
Ok(StartupConfig {
app,
config_service,
restore_checkpoint,
})
}
}
fn absolute_path(path: &Path) -> Result<PathBuf, AppError> {
if path.is_absolute() {
Ok(path.to_path_buf())
} else {
Ok(std::env::current_dir()?.join(path))
}
}
#[cfg(test)]
mod tests {
use std::fs;
use tempfile::TempDir;
use super::*;
fn resolve(dto: AppConfigDto) -> Result<AppConfig, AppError> {
AppConfig::resolve(dto, Path::new("C:/config-root"))
}
#[test]
fn defaults_produce_valid_dht_options() {
let config = resolve(AppConfigDto::default()).unwrap();
let options = config.dht_options();
assert_eq!(options.port, 12_313);
assert_eq!(options.metadata.max_worker_count, 400);
assert_eq!(options.metadata.max_connects_per_second, 400);
assert_eq!(options.metadata.max_metadata_size_bytes, 10 * 1024 * 1024);
assert_eq!(options.max_outbound_queries_per_second, 1_000);
assert_eq!(options.sample_infohashes.max_queries_per_second, 60);
assert_eq!(options.sample_infohashes.max_in_flight, 100);
assert_eq!(options.sample_infohashes.new_node_sample_percent, 50);
assert_eq!(options.sample_infohashes.candidate_queue_capacity, 8_192);
assert!(!options.sample_infohashes.fallback_to_iterative);
assert_eq!(options.peer_lookup.max_lookups_per_second, 200);
assert_eq!(options.peer_lookup.max_active_lookups, 200);
assert_eq!(options.crawl.rate_limit.max_find_node_rate_per_sec, 10);
assert_eq!(
options.crawl.rate_limit.max_new_destinations_per_minute,
12_000
);
}
#[test]
fn dto_round_trips_through_toml() {
let dto = AppConfigDto::default();
let encoded = toml::to_string(&dto).unwrap();
let decoded: AppConfigDto = toml::from_str(&encoded).unwrap();
assert_eq!(decoded.data_dir, dto.data_dir);
assert_eq!(decoded.dht.port, dto.dht.port);
assert_eq!(decoded.http.listen, dto.http.listen);
assert_eq!(decoded.logging.file_prefix, dto.logging.file_prefix);
assert_eq!(decoded.diagnostics.database, dto.diagnostics.database);
}
#[test]
fn relative_directories_are_resolved_from_config_file() {
let directory = TempDir::new().unwrap();
let config_path = directory.path().join("service.toml");
fs::write(&config_path, "data_dir = 'state'").unwrap();
let startup = Cli {
config: config_path,
data_dir: None,
run_duration_secs: None,
restore_checkpoint: None,
}
.load()
.unwrap();
let config = startup.app;
assert_eq!(config.data_dir, directory.path().join("state"));
assert_eq!(
config.content_filter_file,
directory.path().join("content-filters.toml")
);
assert_eq!(config.logging.directory, directory.path().join("data/logs"));
assert_eq!(
config.backup.directory,
directory.path().join("data/backups")
);
assert_eq!(
config.diagnostics.database,
directory.path().join("data/diagnostics.sqlite3")
);
assert_eq!(config.http.web_dir, directory.path().join("src/web/dist"));
}
#[test]
fn unknown_config_field_is_rejected() {
let directory = TempDir::new().unwrap();
let config_path = directory.path().join("service.toml");
fs::write(&config_path, "unknown = true").unwrap();
let error = Cli {
config: config_path,
data_dir: None,
run_duration_secs: None,
restore_checkpoint: None,
}
.load()
.unwrap_err();
assert!(matches!(error, AppError::Toml(_)));
}
#[test]
fn zero_metadata_limit_is_rejected() {
let mut dto = AppConfigDto::default();
dto.metadata_limits.max_files = 0;
assert!(matches!(resolve(dto), Err(AppError::Config(_))));
}
#[test]
fn invalid_new_node_sample_percent_is_rejected() {
let mut dto = AppConfigDto::default();
dto.dht.sample_new_node_percent = 101;
assert!(matches!(resolve(dto), Err(AppError::Config(_))));
}
#[test]
fn disk_resume_threshold_must_exceed_minimum() {
let mut dto = AppConfigDto::default();
dto.disk_guard.resume_free_bytes = dto.disk_guard.minimum_free_bytes;
assert!(matches!(resolve(dto), Err(AppError::Config(_))));
}
#[test]
fn logging_requires_at_least_one_output() {
let mut dto = AppConfigDto::default();
dto.logging.file_enabled = false;
dto.logging.console_enabled = false;
assert!(matches!(resolve(dto), Err(AppError::Config(_))));
}
#[test]
fn logging_prefix_cannot_escape_the_log_directory() {
let mut dto = AppConfigDto::default();
dto.logging.file_prefix = "../service".into();
assert!(matches!(resolve(dto), Err(AppError::Config(_))));
}
#[test]
fn backup_directory_cannot_be_inside_rocksdb() {
let mut dto = AppConfigDto::default();
dto.backup.directory = dto.data_dir.join("rocksdb/checkpoints");
assert!(matches!(resolve(dto), Err(AppError::Config(_))));
}
}
+268
View File
@@ -0,0 +1,268 @@
// 负责定义可序列化的用户配置 DTO 和默认值
use std::{net::SocketAddr, path::PathBuf};
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub(crate) struct AppConfigDto {
pub(crate) data_dir: PathBuf,
pub(crate) content_filter_file: PathBuf,
pub(crate) persistence_queue_capacity: usize,
pub(crate) stats_interval_secs: u64,
pub(crate) run_duration_secs: Option<u64>,
pub(crate) index_batch_size: usize,
pub(crate) index_interval_millis: u64,
pub(crate) metadata_limits: MetadataLimitsConfig,
pub(crate) dht: DhtConfig,
pub(crate) disk_guard: DiskGuardConfig,
pub(crate) backup: BackupConfig,
pub(crate) diagnostics: DiagnosticsConfig,
pub(crate) logging: LoggingConfig,
pub(crate) http: HttpConfig,
pub(crate) verification: VerificationConfig,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub(crate) struct DhtConfig {
pub(crate) port: u16,
pub(crate) netmode: NetworkMode,
pub(crate) hash_queue_capacity: usize,
pub(crate) max_outbound_queries_per_second: u32,
pub(crate) outbound_query_burst: u32,
pub(crate) metadata_timeout_secs: u64,
pub(crate) metadata_queue_capacity: usize,
pub(crate) metadata_workers: usize,
pub(crate) metadata_connects_per_second: u32,
pub(crate) sample_queries_per_second: u32,
pub(crate) sample_max_in_flight: usize,
pub(crate) sample_new_node_percent: u8,
pub(crate) sample_candidate_queue_capacity: usize,
pub(crate) sample_fallback_to_iterative: bool,
pub(crate) peer_lookups_per_second: u32,
pub(crate) peer_lookup_max_active: usize,
pub(crate) find_node_queries_per_second: u32,
pub(crate) find_node_max_in_flight: usize,
pub(crate) new_destinations_per_minute: u32,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub(crate) struct HttpConfig {
pub(crate) listen: SocketAddr,
pub(crate) web_dir: PathBuf,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub(crate) struct DiskGuardConfig {
pub(crate) enabled: bool,
pub(crate) check_interval_secs: u64,
pub(crate) minimum_free_bytes: u64,
pub(crate) resume_free_bytes: u64,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub(crate) struct LoggingConfig {
pub(crate) directory: PathBuf,
pub(crate) file_enabled: bool,
pub(crate) console_enabled: bool,
pub(crate) rotation: LogRotation,
pub(crate) retain_files: usize,
pub(crate) file_prefix: String,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub(crate) struct BackupConfig {
pub(crate) enabled: bool,
pub(crate) directory: PathBuf,
pub(crate) interval_secs: u64,
pub(crate) retain_checkpoints: usize,
pub(crate) create_on_start: bool,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub(crate) struct DiagnosticsConfig {
pub(crate) enabled: bool,
pub(crate) database: PathBuf,
pub(crate) sample_interval_secs: u64,
pub(crate) raw_retention_hours: u64,
pub(crate) minute_retention_days: u64,
pub(crate) queue_capacity: usize,
}
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "kebab-case")]
pub(crate) enum LogRotation {
Minutely,
Hourly,
#[default]
Daily,
Never,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub(crate) struct VerificationConfig {
pub(crate) enabled: bool,
pub(crate) queue_capacity: usize,
pub(crate) max_active: usize,
pub(crate) max_peer_attempts: usize,
pub(crate) lease_secs: u64,
pub(crate) poll_interval_millis: u64,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub(crate) struct MetadataLimitsConfig {
pub(crate) max_metadata_bytes: usize,
pub(crate) max_files: usize,
pub(crate) max_name_bytes: usize,
pub(crate) max_path_bytes: usize,
pub(crate) max_path_depth: usize,
}
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "kebab-case")]
pub(crate) enum NetworkMode {
#[default]
Ipv4Only,
Ipv6Only,
DualStack,
}
impl Default for AppConfigDto {
fn default() -> Self {
Self {
data_dir: PathBuf::from("data"),
content_filter_file: PathBuf::from("content-filters.toml"),
persistence_queue_capacity: 8_192,
stats_interval_secs: 10,
run_duration_secs: None,
index_batch_size: 1_024,
index_interval_millis: 5_000,
metadata_limits: MetadataLimitsConfig::default(),
dht: DhtConfig::default(),
disk_guard: DiskGuardConfig::default(),
backup: BackupConfig::default(),
diagnostics: DiagnosticsConfig::default(),
logging: LoggingConfig::default(),
http: HttpConfig::default(),
verification: VerificationConfig::default(),
}
}
}
impl Default for MetadataLimitsConfig {
fn default() -> Self {
Self {
max_metadata_bytes: 10 * 1024 * 1024,
max_files: 20_000,
max_name_bytes: 1_024,
max_path_bytes: 4_096,
max_path_depth: 64,
}
}
}
impl Default for DhtConfig {
fn default() -> Self {
Self {
port: 12_313,
netmode: NetworkMode::Ipv4Only,
hash_queue_capacity: 20_000,
max_outbound_queries_per_second: 1_000,
outbound_query_burst: 200,
metadata_timeout_secs: 6,
metadata_queue_capacity: 20_000,
metadata_workers: 400,
metadata_connects_per_second: 400,
sample_queries_per_second: 60,
sample_max_in_flight: 100,
sample_new_node_percent: 50,
sample_candidate_queue_capacity: 8_192,
sample_fallback_to_iterative: false,
peer_lookups_per_second: 200,
peer_lookup_max_active: 200,
find_node_queries_per_second: 10,
find_node_max_in_flight: 100,
new_destinations_per_minute: 12_000,
}
}
}
impl Default for HttpConfig {
fn default() -> Self {
Self {
listen: SocketAddr::from(([127, 0, 0, 1], 8080)),
web_dir: PathBuf::from("src/web/dist"),
}
}
}
impl Default for DiskGuardConfig {
fn default() -> Self {
Self {
enabled: true,
check_interval_secs: 10,
minimum_free_bytes: 5 * 1024 * 1024 * 1024,
resume_free_bytes: 6 * 1024 * 1024 * 1024,
}
}
}
impl Default for LoggingConfig {
fn default() -> Self {
Self {
directory: PathBuf::from("data/logs"),
file_enabled: true,
console_enabled: false,
rotation: LogRotation::Daily,
retain_files: 7,
file_prefix: "dht-search".to_owned(),
}
}
}
impl Default for BackupConfig {
fn default() -> Self {
Self {
enabled: true,
directory: PathBuf::from("data/backups"),
interval_secs: 6 * 60 * 60,
retain_checkpoints: 3,
create_on_start: true,
}
}
}
impl Default for DiagnosticsConfig {
fn default() -> Self {
Self {
enabled: true,
database: PathBuf::from("data/diagnostics.sqlite3"),
sample_interval_secs: 10,
raw_retention_hours: 24,
minute_retention_days: 30,
queue_capacity: 128,
}
}
}
impl Default for VerificationConfig {
fn default() -> Self {
Self {
enabled: true,
queue_capacity: 10_000,
max_active: 8,
max_peer_attempts: 3,
lease_secs: 60,
poll_interval_millis: 250,
}
}
}
+238
View File
@@ -0,0 +1,238 @@
// 负责解析校验用户配置并生成应用可直接使用的运行时配置
use std::{fs, ops::Deref, path::Path};
use crate::domain::{ContentFilter, ContentFilterConfig, MetadataLimits};
use dht_crawler::{
BootstrapOptions, CrawlOptions, DHTOptions, MetadataOptions, NetMode, PeerLookupOptions,
PoolOptions, RateLimitOptions, SampleInfohashesOptions, SchedulerOptions, TargetOptions,
};
use crate::error::AppError;
use super::model::{AppConfigDto, NetworkMode};
#[derive(Debug, Clone)]
pub(crate) struct AppConfig(AppConfigDto);
impl AppConfig {
pub(crate) fn resolve(mut dto: AppConfigDto, base: &Path) -> Result<Self, AppError> {
dto.data_dir = resolve_path(base, dto.data_dir);
dto.content_filter_file = resolve_path(base, dto.content_filter_file);
dto.logging.directory = resolve_path(base, dto.logging.directory);
dto.backup.directory = resolve_path(base, dto.backup.directory);
dto.diagnostics.database = resolve_path(base, dto.diagnostics.database);
dto.http.web_dir = resolve_path(base, dto.http.web_dir);
let config = Self(dto);
config.validate()?;
Ok(config)
}
pub(crate) fn content_filter(&self) -> Result<ContentFilter, AppError> {
let contents = fs::read_to_string(&self.content_filter_file)?;
let config = toml::from_str::<ContentFilterConfig>(&contents)?;
ContentFilter::compile(config).map_err(AppError::from)
}
pub(crate) fn dht_options(&self) -> DHTOptions {
let defaults = DHTOptions::default();
DHTOptions {
port: self.dht.port,
netmode: self.dht.netmode.into(),
hash_queue_capacity: self.dht.hash_queue_capacity,
max_outbound_queries_per_second: self.dht.max_outbound_queries_per_second,
outbound_query_burst: self.dht.outbound_query_burst,
metadata: MetadataOptions {
timeout_secs: self.dht.metadata_timeout_secs,
max_queue_size: self.dht.metadata_queue_capacity,
max_worker_count: self.dht.metadata_workers,
max_connects_per_second: self.dht.metadata_connects_per_second,
max_metadata_size_bytes: self.metadata_limits.max_metadata_bytes,
..defaults.metadata
},
peer_lookup: PeerLookupOptions {
max_lookups_per_second: self.dht.peer_lookups_per_second,
burst: self.dht.peer_lookups_per_second.max(1),
max_active_lookups: self.dht.peer_lookup_max_active,
},
sample_infohashes: SampleInfohashesOptions {
max_queries_per_second: self.dht.sample_queries_per_second,
burst: self.dht.sample_queries_per_second.max(1),
max_in_flight: self.dht.sample_max_in_flight,
new_node_sample_percent: self.dht.sample_new_node_percent,
candidate_queue_capacity: self.dht.sample_candidate_queue_capacity,
fallback_to_iterative: self.dht.sample_fallback_to_iterative,
..defaults.sample_infohashes
},
crawl: CrawlOptions {
pool: PoolOptions {
..defaults.crawl.pool
},
rate_limit: RateLimitOptions {
max_find_node_rate_per_sec: self.dht.find_node_queries_per_second,
burst: self.dht.outbound_query_burst,
max_in_flight: self.dht.find_node_max_in_flight,
max_new_destinations_per_minute: self.dht.new_destinations_per_minute,
..defaults.crawl.rate_limit
},
bootstrap: BootstrapOptions {
..defaults.crawl.bootstrap
},
target: TargetOptions {
..defaults.crawl.target
},
scheduler: SchedulerOptions {
..defaults.crawl.scheduler
},
},
}
}
pub(crate) fn metadata_limits(&self) -> MetadataLimits {
MetadataLimits {
max_files: self.metadata_limits.max_files,
max_name_bytes: self.metadata_limits.max_name_bytes,
max_path_bytes: self.metadata_limits.max_path_bytes,
max_path_depth: self.metadata_limits.max_path_depth,
}
}
fn validate(&self) -> Result<(), AppError> {
if self.persistence_queue_capacity == 0 {
return Err(AppError::Config(
"persistence_queue_capacity 必须大于零".to_owned(),
));
}
if self.stats_interval_secs == 0 {
return Err(AppError::Config(
"stats_interval_secs 必须大于零".to_owned(),
));
}
if self.run_duration_secs == Some(0) {
return Err(AppError::Config(
"run_duration_secs 必须大于零或不设置".to_owned(),
));
}
if self.index_batch_size == 0 || self.index_interval_millis == 0 {
return Err(AppError::Config(
"索引批量大小和执行间隔必须大于零".to_owned(),
));
}
if self.metadata_limits.max_metadata_bytes == 0
|| self.metadata_limits.max_files == 0
|| self.metadata_limits.max_name_bytes == 0
|| self.metadata_limits.max_path_bytes == 0
|| self.metadata_limits.max_path_depth == 0
{
return Err(AppError::Config(
"Metadata 大小文件数名称路径和目录层级上限必须大于零".to_owned(),
));
}
if self.dht.metadata_workers == 0 || self.dht.metadata_queue_capacity == 0 {
return Err(AppError::Config(
"Metadata worker 和队列容量必须大于零".to_owned(),
));
}
if self.verification.queue_capacity == 0
|| self.verification.max_active == 0
|| self.verification.max_peer_attempts == 0
|| self.verification.lease_secs == 0
|| self.verification.poll_interval_millis == 0
{
return Err(AppError::Config(
"验证队列容量并发尝试数租约和轮询间隔必须大于零".to_owned(),
));
}
if self.disk_guard.check_interval_secs == 0
|| self.disk_guard.minimum_free_bytes == 0
|| self.disk_guard.resume_free_bytes <= self.disk_guard.minimum_free_bytes
{
return Err(AppError::Config(
"磁盘检查间隔必须大于零且恢复阈值必须大于保护阈值".to_owned(),
));
}
if self.backup.interval_secs == 0 || self.backup.retain_checkpoints == 0 {
return Err(AppError::Config(
"备份间隔和检查点保留数量必须大于零".to_owned(),
));
}
if self.diagnostics.sample_interval_secs == 0
|| self.diagnostics.raw_retention_hours == 0
|| self.diagnostics.minute_retention_days == 0
|| self.diagnostics.queue_capacity == 0
{
return Err(AppError::Config(
"诊断采样间隔保留时间和队列容量必须大于零".to_owned(),
));
}
let database_path = self.data_dir.join("rocksdb");
if self.backup.directory == database_path
|| self.backup.directory.starts_with(&database_path)
{
return Err(AppError::Config(
"检查点目录不能位于 RocksDB 数据库目录内部".to_owned(),
));
}
if !self.logging.file_enabled && !self.logging.console_enabled {
return Err(AppError::Config(
"文件日志和终端日志不能同时关闭".to_owned(),
));
}
if self.logging.file_enabled && self.logging.retain_files == 0 {
return Err(AppError::Config("日志保留文件数量必须大于零".to_owned()));
}
if self.logging.file_prefix.trim().is_empty()
|| self
.logging
.file_prefix
.chars()
.any(|character| character.is_control() || matches!(character, '/' | '\\'))
{
return Err(AppError::Config(
"日志文件前缀不能为空且不能包含路径分隔符或控制字符".to_owned(),
));
}
if self.dht.max_outbound_queries_per_second == 0
|| self.dht.outbound_query_burst == 0
|| self.dht.metadata_connects_per_second == 0
|| self.dht.sample_max_in_flight == 0
|| self.dht.sample_candidate_queue_capacity == 0
|| self.dht.peer_lookup_max_active == 0
|| self.dht.find_node_max_in_flight == 0
{
return Err(AppError::Config("网络速率和并发上限必须大于零".to_owned()));
}
if self.dht.sample_new_node_percent > 100 {
return Err(AppError::Config(
"sample_new_node_percent 必须在 0 到 100 之间".to_owned(),
));
}
Ok(())
}
}
impl Deref for AppConfig {
type Target = AppConfigDto;
fn deref(&self) -> &Self::Target {
&self.0
}
}
impl From<NetworkMode> for NetMode {
fn from(value: NetworkMode) -> Self {
match value {
NetworkMode::Ipv4Only => Self::Ipv4Only,
NetworkMode::Ipv6Only => Self::Ipv6Only,
NetworkMode::DualStack => Self::DualStack,
}
}
}
fn resolve_path(base: &Path, path: std::path::PathBuf) -> std::path::PathBuf {
if path.is_absolute() {
path
} else {
base.join(path)
}
}
+221
View File
@@ -0,0 +1,221 @@
// 负责配置修订查询完整校验并发覆盖保护和持久化更新
use std::{
path::PathBuf,
sync::{Arc, Mutex},
};
use serde::{Deserialize, Serialize};
use crate::error::AppError;
use super::{AppConfig, AppConfigDto, ConfigStore};
#[derive(Debug, Clone, Serialize)]
pub(crate) struct ConfigSnapshot {
pub(crate) revision: String,
pub(crate) restart_required: bool,
pub(crate) source: String,
pub(crate) command_line_overrides: Vec<String>,
pub(crate) config: AppConfigDto,
}
#[derive(Debug, Deserialize)]
pub(crate) struct ConfigUpdateRequest {
pub(crate) revision: String,
pub(crate) config: AppConfigDto,
}
#[derive(Clone)]
pub(crate) struct ConfigService {
inner: Arc<ConfigServiceInner>,
}
impl std::fmt::Debug for ConfigService {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
formatter
.debug_struct("ConfigService")
.field("snapshot", &self.snapshot())
.finish()
}
}
struct ConfigServiceInner {
store: Arc<dyn ConfigStore>,
base: PathBuf,
startup: AppConfigDto,
command_line_overrides: Vec<String>,
state: Mutex<ConfigState>,
}
struct ConfigState {
revision: String,
config: AppConfigDto,
}
#[derive(Debug, thiserror::Error)]
pub(crate) enum ConfigServiceError {
#[error("配置已经被其他请求修改 请重新加载后再保存")]
Conflict,
#[error("配置校验失败: {0}")]
Validation(String),
#[error("配置保存失败: {0}")]
Persistence(String),
}
impl ConfigService {
pub(crate) fn new(
store: Arc<dyn ConfigStore>,
initial: AppConfigDto,
command_line_overrides: Vec<String>,
) -> Result<Self, AppError> {
let base = store
.path()
.parent()
.ok_or_else(|| AppError::Config("配置文件没有父目录".to_owned()))?
.to_path_buf();
let revision = revision(&initial)?;
Ok(Self {
inner: Arc::new(ConfigServiceInner {
store,
base,
startup: initial.clone(),
command_line_overrides,
state: Mutex::new(ConfigState {
revision,
config: initial,
}),
}),
})
}
pub(crate) fn snapshot(&self) -> ConfigSnapshot {
let state = self
.inner
.state
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
self.snapshot_from(&state)
}
pub(crate) fn update(
&self,
request: ConfigUpdateRequest,
) -> Result<ConfigSnapshot, ConfigServiceError> {
AppConfig::resolve(request.config.clone(), &self.inner.base)
.map_err(|error| ConfigServiceError::Validation(error.to_string()))?;
let candidate_revision = revision(&request.config)
.map_err(|error| ConfigServiceError::Persistence(error.to_string()))?;
let mut state = self
.inner
.state
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
if request.revision != state.revision {
return Err(ConfigServiceError::Conflict);
}
self.inner
.store
.save(&request.config)
.map_err(|error| ConfigServiceError::Persistence(error.to_string()))?;
state.config = request.config;
state.revision = candidate_revision;
Ok(self.snapshot_from(&state))
}
fn snapshot_from(&self, state: &ConfigState) -> ConfigSnapshot {
ConfigSnapshot {
revision: state.revision.clone(),
restart_required: state.config != self.inner.startup,
source: self.inner.store.path().to_string_lossy().into_owned(),
command_line_overrides: self.inner.command_line_overrides.clone(),
config: state.config.clone(),
}
}
}
fn revision(config: &AppConfigDto) -> Result<String, AppError> {
let canonical = toml::to_string(config)?;
Ok(blake3::hash(canonical.as_bytes()).to_hex().to_string())
}
#[cfg(test)]
mod tests {
use tempfile::TempDir;
use super::*;
use crate::config::TomlConfigStore;
fn service(directory: &TempDir) -> ConfigService {
ConfigService::new(
Arc::new(TomlConfigStore::new(directory.path().join("service.toml"))),
AppConfigDto::default(),
Vec::new(),
)
.unwrap()
}
#[test]
fn valid_update_is_persisted_and_requires_restart() {
let directory = TempDir::new().unwrap();
let service = service(&directory);
let current = service.snapshot();
let mut config = current.config;
config.dht.port = 22_313;
let updated = service
.update(ConfigUpdateRequest {
revision: current.revision,
config: config.clone(),
})
.unwrap();
assert!(updated.restart_required);
assert_eq!(updated.config, config);
let stored = TomlConfigStore::new(directory.path().join("service.toml"))
.load()
.unwrap();
assert_eq!(stored, Some(config));
}
#[test]
fn stale_revision_cannot_overwrite_a_newer_update() {
let directory = TempDir::new().unwrap();
let service = service(&directory);
let original = service.snapshot();
let mut first = original.config.clone();
first.dht.port = 22_313;
service
.update(ConfigUpdateRequest {
revision: original.revision.clone(),
config: first.clone(),
})
.unwrap();
let mut stale = original.config;
stale.dht.port = 32_313;
assert!(matches!(
service.update(ConfigUpdateRequest {
revision: original.revision,
config: stale,
}),
Err(ConfigServiceError::Conflict)
));
assert_eq!(service.snapshot().config, first);
}
#[test]
fn invalid_update_does_not_change_memory_or_disk() {
let directory = TempDir::new().unwrap();
let service = service(&directory);
let current = service.snapshot();
let mut invalid = current.config.clone();
invalid.persistence_queue_capacity = 0;
assert!(matches!(
service.update(ConfigUpdateRequest {
revision: current.revision,
config: invalid,
}),
Err(ConfigServiceError::Validation(_))
));
assert_eq!(service.snapshot().config, current.config);
assert!(!directory.path().join("service.toml").exists());
}
}
+155
View File
@@ -0,0 +1,155 @@
// 负责从持久化介质读取并原子保存用户配置 DTO
use std::{
fs::{self, OpenOptions},
io::Write,
path::{Path, PathBuf},
sync::atomic::{AtomicU64, Ordering},
};
use crate::error::AppError;
use super::AppConfigDto;
static TEMP_SEQUENCE: AtomicU64 = AtomicU64::new(0);
pub(crate) trait ConfigStore: Send + Sync {
fn load(&self) -> Result<Option<AppConfigDto>, AppError>;
fn save(&self, config: &AppConfigDto) -> Result<(), AppError>;
fn path(&self) -> &Path;
}
#[derive(Debug, Clone)]
pub(crate) struct TomlConfigStore {
path: PathBuf,
}
impl TomlConfigStore {
pub(crate) fn new(path: PathBuf) -> Self {
Self { path }
}
}
impl ConfigStore for TomlConfigStore {
fn load(&self) -> Result<Option<AppConfigDto>, AppError> {
if !self.path.exists() {
return Ok(None);
}
let contents = fs::read_to_string(&self.path)?;
toml::from_str(&contents).map(Some).map_err(AppError::from)
}
fn save(&self, config: &AppConfigDto) -> Result<(), AppError> {
let contents = toml::to_string_pretty(config)?;
let parent = self
.path
.parent()
.ok_or_else(|| AppError::Config("配置文件没有父目录".to_owned()))?;
fs::create_dir_all(parent)?;
let temporary = temporary_path(&self.path);
let result = (|| {
let mut file = OpenOptions::new()
.create_new(true)
.write(true)
.open(&temporary)?;
file.write_all(contents.as_bytes())?;
file.sync_all()?;
drop(file);
replace_file(&temporary, &self.path)?;
sync_parent(parent)?;
Ok::<(), std::io::Error>(())
})();
if result.is_err() {
let _ = fs::remove_file(&temporary);
}
result.map_err(AppError::from)
}
fn path(&self) -> &Path {
&self.path
}
}
fn temporary_path(destination: &Path) -> PathBuf {
let sequence = TEMP_SEQUENCE.fetch_add(1, Ordering::Relaxed);
let file_name = destination
.file_name()
.and_then(|name| name.to_str())
.unwrap_or("dht-search.toml");
destination.with_file_name(format!(
".{file_name}.tmp-{}-{sequence}",
std::process::id()
))
}
#[cfg(windows)]
fn replace_file(source: &Path, destination: &Path) -> std::io::Result<()> {
use std::os::windows::ffi::OsStrExt;
use windows_sys::Win32::Storage::FileSystem::{
MOVEFILE_REPLACE_EXISTING, MOVEFILE_WRITE_THROUGH, MoveFileExW,
};
let source: Vec<u16> = source.as_os_str().encode_wide().chain(Some(0)).collect();
let destination: Vec<u16> = destination
.as_os_str()
.encode_wide()
.chain(Some(0))
.collect();
if unsafe {
MoveFileExW(
source.as_ptr(),
destination.as_ptr(),
MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH,
)
} == 0
{
Err(std::io::Error::last_os_error())
} else {
Ok(())
}
}
#[cfg(not(windows))]
fn replace_file(source: &Path, destination: &Path) -> std::io::Result<()> {
fs::rename(source, destination)
}
#[cfg(unix)]
fn sync_parent(parent: &Path) -> std::io::Result<()> {
std::fs::File::open(parent)?.sync_all()
}
#[cfg(not(unix))]
fn sync_parent(_parent: &Path) -> std::io::Result<()> {
Ok(())
}
#[cfg(test)]
mod tests {
use tempfile::TempDir;
use super::*;
#[test]
fn save_replaces_complete_toml_and_leaves_no_temporary_file() {
let directory = TempDir::new().unwrap();
let path = directory.path().join("service.toml");
fs::write(&path, "data_dir = 'old'").unwrap();
let store = TomlConfigStore::new(path.clone());
let expected = AppConfigDto {
data_dir: PathBuf::from("new-data"),
..AppConfigDto::default()
};
store.save(&expected).unwrap();
assert_eq!(store.load().unwrap(), Some(expected));
assert_eq!(
fs::read_dir(directory.path())
.unwrap()
.filter_map(Result::ok)
.filter(|entry| entry.file_name().to_string_lossy().contains(".tmp-"))
.count(),
0
);
}
}
+23
View File
@@ -0,0 +1,23 @@
// 负责将 DHT 采集结果转换为不依赖传输层的领域输入
use crate::domain::{MetadataCandidate, TorrentFile};
use dht_crawler::TorrentInfo;
pub(crate) fn metadata_candidate(info: TorrentInfo) -> MetadataCandidate {
MetadataCandidate {
info_hash: info.info_hash,
name: info.name,
total_size: info.total_size,
files: info
.files
.into_iter()
.map(|file| TorrentFile {
path: file.path,
size: file.size,
})
.collect(),
piece_length: info.piece_length,
source_peers: info.peers,
timestamp: info.timestamp,
}
}
+4
View File
@@ -0,0 +1,4 @@
// 负责组合 DHT 发现 Metadata 下载和持久化提交管线
pub(crate) mod mapper;
pub(crate) mod pipeline;
+437
View File
@@ -0,0 +1,437 @@
// 负责定义采集阶段之间的有界队列背压和任务流转规则
use std::{
str::FromStr,
sync::{
Arc, Mutex,
atomic::{AtomicU64, AtomicUsize, Ordering},
mpsc::{self, SyncSender, TrySendError},
},
thread::{self, JoinHandle},
};
use crate::{
domain::{InfoHash, MetadataLimits, MetadataRejectionReason, RejectedMetadata, TorrentRecord},
storage::{StorageError, TorrentRepository, UpsertOutcome},
};
use dht_crawler::TorrentInfo;
use tokio::sync::oneshot;
use crate::crawler::mapper::metadata_candidate;
use crate::disk_guard::DiskGuard;
use crate::error::AppError;
#[derive(Clone)]
pub(crate) struct PersistenceIngress {
sender: Arc<Mutex<Option<SyncSender<PersistenceItem>>>>,
stats: Arc<PersistenceStats>,
limits: MetadataLimits,
disk_guard: DiskGuard,
}
pub(crate) struct PersistencePipeline {
pub(crate) ingress: PersistenceIngress,
pub(crate) fatal: oneshot::Receiver<String>,
worker: JoinHandle<Result<(), StorageError>>,
}
#[derive(Default)]
pub(crate) struct PersistenceStats {
accepted: AtomicU64,
inserted: AtomicU64,
updated: AtomicU64,
rejected_full: AtomicU64,
invalid: AtomicU64,
failed: AtomicU64,
queue_depth: AtomicUsize,
filtered: [AtomicU64; MetadataRejectionReason::COUNT],
}
#[derive(Debug, Clone, Copy)]
pub(crate) struct PersistenceSnapshot {
pub(crate) accepted: u64,
pub(crate) inserted: u64,
pub(crate) updated: u64,
pub(crate) rejected_full: u64,
pub(crate) invalid: u64,
pub(crate) failed: u64,
pub(crate) queue_depth: usize,
pub(crate) filtered: MetadataFilterSnapshot,
}
#[derive(Debug, Clone, Copy)]
pub(crate) struct MetadataFilterSnapshot {
counts: [u64; MetadataRejectionReason::COUNT],
}
enum PersistenceItem {
Record(TorrentRecord),
Rejection(RejectedMetadata),
}
impl PersistencePipeline {
pub(crate) fn start(
repository: Arc<dyn TorrentRepository>,
capacity: usize,
limits: MetadataLimits,
disk_guard: DiskGuard,
) -> Self {
let (sender, receiver) = mpsc::sync_channel::<PersistenceItem>(capacity);
let (fatal_tx, fatal) = oneshot::channel();
let stats = Arc::new(PersistenceStats::default());
let worker_stats = stats.clone();
let worker_disk_guard = disk_guard.clone();
let worker = thread::Builder::new()
.name("torrent-persistence".to_owned())
.spawn(move || {
while let Ok(item) = receiver.recv() {
let _permit = loop {
if let Some(permit) = worker_disk_guard.begin_drain_write() {
break permit;
}
thread::sleep(std::time::Duration::from_millis(250));
};
worker_stats.queue_depth.fetch_sub(1, Ordering::Relaxed);
let outcome = match item {
PersistenceItem::Record(record) => repository.upsert(record).map(Some),
PersistenceItem::Rejection(rejection) => {
repository.record_rejection(rejection).map(|()| None)
}
};
match outcome {
Ok(Some(UpsertOutcome::Inserted)) => {
worker_stats.inserted.fetch_add(1, Ordering::Relaxed);
}
Ok(Some(UpsertOutcome::Updated { .. })) => {
worker_stats.updated.fetch_add(1, Ordering::Relaxed);
}
Ok(None) => {}
Err(error) => {
worker_stats.failed.fetch_add(1, Ordering::Relaxed);
let _ = fatal_tx.send(error.to_string());
return Err(error);
}
}
}
Ok(())
})
.expect("persistence worker thread must spawn");
Self {
ingress: PersistenceIngress {
sender: Arc::new(Mutex::new(Some(sender))),
stats,
limits,
disk_guard,
},
fatal,
worker,
}
}
pub(crate) async fn close_and_join(self) -> Result<(), AppError> {
self.ingress.close();
let result = tokio::task::spawn_blocking(move || self.worker.join())
.await
.map_err(|_| AppError::PersistenceWorkerPanicked)?
.map_err(|_| AppError::PersistenceWorkerPanicked)?;
result.map_err(AppError::from)
}
}
impl PersistenceIngress {
pub(crate) fn try_enqueue(&self, torrent: TorrentInfo) -> bool {
let Some(_permit) = self.disk_guard.begin_admission() else {
return false;
};
let rejected_info_hash = InfoHash::from_str(&torrent.info_hash).ok();
let rejected_at = torrent.timestamp;
let record =
match TorrentRecord::try_from_with_limits(metadata_candidate(torrent), self.limits) {
Ok(record) => record,
Err(error) => {
self.stats.invalid.fetch_add(1, Ordering::Relaxed);
let reason = error.rejection_reason();
self.stats.filtered[reason.index()].fetch_add(1, Ordering::Relaxed);
tracing::warn!(%error, ?reason, "拒绝无效 Metadata");
if let Some(info_hash) = rejected_info_hash {
let rejection = RejectedMetadata::new(
info_hash,
reason,
self.limits.rule_id(),
rejected_at,
);
self.try_send(PersistenceItem::Rejection(rejection), false);
}
return false;
}
};
self.try_send(PersistenceItem::Record(record), true)
}
fn try_send(&self, item: PersistenceItem, accepted_record: bool) -> bool {
let sender = self
.sender
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
let Some(sender) = sender.as_ref() else {
return false;
};
match sender.try_send(item) {
Ok(()) => {
if accepted_record {
self.stats.accepted.fetch_add(1, Ordering::Relaxed);
}
self.stats.queue_depth.fetch_add(1, Ordering::Relaxed);
true
}
Err(TrySendError::Full(_)) => {
self.stats.rejected_full.fetch_add(1, Ordering::Relaxed);
false
}
Err(TrySendError::Disconnected(_)) => false,
}
}
pub(crate) fn close(&self) {
self.sender
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.take();
}
pub(crate) fn snapshot(&self) -> PersistenceSnapshot {
self.stats.snapshot()
}
}
impl PersistenceStats {
fn snapshot(&self) -> PersistenceSnapshot {
PersistenceSnapshot {
accepted: self.accepted.load(Ordering::Relaxed),
inserted: self.inserted.load(Ordering::Relaxed),
updated: self.updated.load(Ordering::Relaxed),
rejected_full: self.rejected_full.load(Ordering::Relaxed),
invalid: self.invalid.load(Ordering::Relaxed),
failed: self.failed.load(Ordering::Relaxed),
queue_depth: self.queue_depth.load(Ordering::Relaxed),
filtered: MetadataFilterSnapshot {
counts: std::array::from_fn(|index| self.filtered[index].load(Ordering::Relaxed)),
},
}
}
}
impl MetadataFilterSnapshot {
pub(crate) fn total(self) -> u64 {
self.counts.iter().copied().sum()
}
pub(crate) fn count(self, reason: MetadataRejectionReason) -> u64 {
self.counts[reason.index()]
}
}
#[cfg(test)]
mod tests {
use std::sync::Mutex;
use crate::{
domain::{ContentGroup, InfoHash, VerificationResult},
storage::{
ContentGroupTask, ContentVariants, UpsertOutcome, VerificationEnqueueOutcome,
VerificationPriority, VerificationRequest,
},
};
use dht_crawler::FileInfo;
use super::*;
use crate::config::DiskGuardConfig;
fn disk_guard() -> DiskGuard {
DiskGuard::new(&DiskGuardConfig {
enabled: false,
..DiskGuardConfig::default()
})
}
#[derive(Default)]
struct MemoryRepository {
records: Mutex<Vec<TorrentRecord>>,
rejections: Mutex<Vec<RejectedMetadata>>,
}
impl TorrentRepository for MemoryRepository {
fn get(&self, info_hash: InfoHash) -> Result<Option<TorrentRecord>, StorageError> {
Ok(self
.records
.lock()
.unwrap()
.iter()
.find(|record| record.info_hash == info_hash)
.cloned())
}
fn upsert(&self, record: TorrentRecord) -> Result<UpsertOutcome, StorageError> {
self.records.lock().unwrap().push(record);
Ok(UpsertOutcome::Inserted)
}
fn rejection(&self, _: InfoHash) -> Result<Option<RejectedMetadata>, StorageError> {
Ok(None)
}
fn record_rejection(&self, rejection: RejectedMetadata) -> Result<(), StorageError> {
self.rejections.lock().unwrap().push(rejection);
Ok(())
}
fn observe_existing(&self, _: InfoHash, _: u64) -> Result<bool, StorageError> {
Ok(false)
}
fn pending_index(&self, _: usize) -> Result<Vec<ContentGroupTask>, StorageError> {
Ok(Vec::new())
}
fn content_group(
&self,
_: &[u8; 32],
_: u64,
) -> Result<Option<ContentGroup>, StorageError> {
Ok(None)
}
fn mark_indexed(&self, _: &[u8; 32], _: u64) -> Result<bool, StorageError> {
Ok(true)
}
fn prepare_full_reindex(&self) -> Result<u64, StorageError> {
Ok(0)
}
fn content_variants(
&self,
_: &[u8; 32],
_: usize,
_: usize,
) -> Result<ContentVariants, StorageError> {
Ok(ContentVariants {
total: 0,
records: Vec::new(),
})
}
fn enqueue_verification(
&self,
_: &[InfoHash],
_: VerificationPriority,
_: u64,
_: usize,
) -> Result<VerificationEnqueueOutcome, StorageError> {
Ok(VerificationEnqueueOutcome::default())
}
fn claim_verification(
&self,
_: u64,
_: u64,
) -> Result<Option<VerificationRequest>, StorageError> {
Ok(None)
}
fn finish_verification(
&self,
_: InfoHash,
_: VerificationResult,
) -> Result<(), StorageError> {
Ok(())
}
fn verification_queue_len(&self) -> Result<usize, StorageError> {
Ok(0)
}
}
fn torrent() -> TorrentInfo {
TorrentInfo {
info_hash: "0101010101010101010101010101010101010101".into(),
magnet_link: String::new(),
name: "test".into(),
total_size: 1,
files: vec![FileInfo {
path: "test".into(),
size: 1,
}],
piece_length: 16_384,
peers: Vec::new(),
timestamp: 1,
}
}
#[tokio::test]
async fn accepted_record_is_drained_before_shutdown() {
let repository = Arc::new(MemoryRepository::default());
let pipeline = PersistencePipeline::start(
repository.clone(),
1,
MetadataLimits::default(),
disk_guard(),
);
assert!(pipeline.ingress.try_enqueue(torrent()));
pipeline.close_and_join().await.unwrap();
let records = repository.records.lock().unwrap();
assert_eq!(records.len(), 1);
}
#[tokio::test]
async fn invalid_record_is_classified_and_persists_only_a_rejection() {
let repository = Arc::new(MemoryRepository::default());
let limits = MetadataLimits {
max_files: 1,
..MetadataLimits::default()
};
let pipeline = PersistencePipeline::start(repository.clone(), 1, limits, disk_guard());
let mut invalid = torrent();
invalid.files.push(FileInfo {
path: "second".into(),
size: 1,
});
invalid.total_size = 2;
assert!(!pipeline.ingress.try_enqueue(invalid));
let snapshot = pipeline.ingress.snapshot();
assert_eq!(snapshot.accepted, 0);
assert_eq!(snapshot.filtered.total(), 1);
assert_eq!(
snapshot
.filtered
.count(MetadataRejectionReason::TooManyFiles),
1
);
pipeline.close_and_join().await.unwrap();
assert!(repository.records.lock().unwrap().is_empty());
let rejections = repository.rejections.lock().unwrap();
assert_eq!(rejections.len(), 1);
assert_eq!(rejections[0].reason, MetadataRejectionReason::TooManyFiles);
}
#[tokio::test]
async fn read_only_guard_rejects_new_metadata_without_blocking_shutdown() {
let repository = Arc::new(MemoryRepository::default());
let guard = DiskGuard::new(&DiskGuardConfig {
enabled: true,
check_interval_secs: 1,
minimum_free_bytes: 100,
resume_free_bytes: 200,
});
guard.observe_available(99, 0);
let pipeline =
PersistencePipeline::start(repository.clone(), 1, MetadataLimits::default(), guard);
assert!(!pipeline.ingress.try_enqueue(torrent()));
pipeline.close_and_join().await.unwrap();
assert!(repository.records.lock().unwrap().is_empty());
}
}
+166
View File
@@ -0,0 +1,166 @@
// 负责记录 HTTP 请求并发状态错误分类和固定桶延迟分布
use std::{
sync::{
Arc,
atomic::{AtomicU64, Ordering},
},
time::Instant,
};
use super::model::HttpDiagnostics;
const LATENCY_BUCKETS_MICROS: [u64; 11] = [
1_000,
5_000,
10_000,
25_000,
50_000,
100_000,
250_000,
500_000,
1_000_000,
5_000_000,
u64::MAX,
];
#[derive(Clone, Default)]
pub(crate) struct HttpStats {
inner: Arc<HttpStatsInner>,
}
struct HttpStatsInner {
active: AtomicU64,
requests: AtomicU64,
client_errors: AtomicU64,
server_errors: AtomicU64,
latency_total_micros: AtomicU64,
latency_max_micros: AtomicU64,
latency_buckets: [AtomicU64; LATENCY_BUCKETS_MICROS.len()],
}
impl Default for HttpStatsInner {
fn default() -> Self {
Self {
active: AtomicU64::new(0),
requests: AtomicU64::new(0),
client_errors: AtomicU64::new(0),
server_errors: AtomicU64::new(0),
latency_total_micros: AtomicU64::new(0),
latency_max_micros: AtomicU64::new(0),
latency_buckets: std::array::from_fn(|_| AtomicU64::new(0)),
}
}
}
pub(crate) struct HttpRequestTimer {
stats: HttpStats,
started: Instant,
}
impl HttpStats {
pub(crate) fn begin(&self) -> HttpRequestTimer {
self.inner.active.fetch_add(1, Ordering::Relaxed);
HttpRequestTimer {
stats: self.clone(),
started: Instant::now(),
}
}
pub(crate) fn snapshot(&self) -> HttpDiagnostics {
let requests = self.inner.requests.load(Ordering::Relaxed);
let total_micros = self.inner.latency_total_micros.load(Ordering::Relaxed);
HttpDiagnostics {
active_requests: self.inner.active.load(Ordering::Relaxed),
requests,
client_errors: self.inner.client_errors.load(Ordering::Relaxed),
server_errors: self.inner.server_errors.load(Ordering::Relaxed),
latency_average_micros: (requests != 0).then(|| total_micros / requests),
latency_p95_millis: percentile_millis(&self.inner.latency_buckets, requests, 95),
latency_max_millis: micros_to_millis(
self.inner.latency_max_micros.load(Ordering::Relaxed),
),
}
}
fn observe(&self, status: u16, elapsed_micros: u64) {
self.inner.requests.fetch_add(1, Ordering::Relaxed);
if (400..500).contains(&status) {
self.inner.client_errors.fetch_add(1, Ordering::Relaxed);
} else if status >= 500 {
self.inner.server_errors.fetch_add(1, Ordering::Relaxed);
}
let _ = self.inner.latency_total_micros.fetch_update(
Ordering::Relaxed,
Ordering::Relaxed,
|value| Some(value.saturating_add(elapsed_micros)),
);
self.inner
.latency_max_micros
.fetch_max(elapsed_micros, Ordering::Relaxed);
let index = LATENCY_BUCKETS_MICROS
.iter()
.position(|upper| elapsed_micros <= *upper)
.unwrap_or(LATENCY_BUCKETS_MICROS.len() - 1);
self.inner.latency_buckets[index].fetch_add(1, Ordering::Relaxed);
}
}
impl HttpRequestTimer {
pub(crate) fn finish(self, status: u16) {
let elapsed_micros = self.started.elapsed().as_micros().min(u128::from(u64::MAX)) as u64;
self.stats.observe(status, elapsed_micros);
}
}
impl Drop for HttpRequestTimer {
fn drop(&mut self) {
self.stats.inner.active.fetch_sub(1, Ordering::Relaxed);
}
}
fn percentile_millis(
buckets: &[AtomicU64; LATENCY_BUCKETS_MICROS.len()],
count: u64,
percentile: u64,
) -> Option<u64> {
if count == 0 {
return None;
}
let rank = count.saturating_mul(percentile).saturating_add(99) / 100;
let mut cumulative = 0_u64;
for (index, bucket) in buckets.iter().enumerate() {
cumulative = cumulative.saturating_add(bucket.load(Ordering::Relaxed));
if cumulative >= rank {
let upper = LATENCY_BUCKETS_MICROS[index];
return (upper != u64::MAX).then(|| micros_to_millis(upper));
}
}
None
}
fn micros_to_millis(micros: u64) -> u64 {
micros.saturating_add(999) / 1_000
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn snapshots_track_active_completed_and_error_requests() {
let stats = HttpStats::default();
let active = stats.begin();
assert_eq!(stats.snapshot().active_requests, 1);
active.finish(404);
let failed = stats.begin();
failed.finish(503);
let snapshot = stats.snapshot();
assert_eq!(snapshot.active_requests, 0);
assert_eq!(snapshot.requests, 2);
assert_eq!(snapshot.client_errors, 1);
assert_eq!(snapshot.server_errors, 1);
assert!(snapshot.latency_average_micros.is_some());
assert!(snapshot.latency_p95_millis.is_some());
}
}
+392
View File
@@ -0,0 +1,392 @@
// 负责采集运行资源快照并协调有界 SQLite 历史写入和查询
mod http;
mod model;
mod process;
mod store;
use std::{
path::PathBuf,
sync::{
Arc, Mutex, RwLock,
atomic::{AtomicU64, Ordering},
mpsc::{SyncSender, TrySendError, sync_channel},
},
thread::JoinHandle,
time::{Duration, SystemTime, UNIX_EPOCH},
};
use crate::{
search::SearchEngine,
storage::{RocksTorrentRepository, StorageDiagnostics as RocksDiagnostics},
};
use dht_crawler::DhtRuntimeStats;
use tokio_util::sync::CancellationToken;
use crate::{
config::DiagnosticsConfig, crawler::pipeline::PersistenceIngress, disk_guard::DiskGuard,
};
pub(crate) use http::HttpStats;
pub(crate) use model::{
CurrentDiagnosticsResponse, DiagnosticHistory, DiagnosticSample, HistoryResolution,
};
use model::{DiagnosticsStatus, RuntimeDiagnostics, SearchDiagnostics, StorageDiagnostics};
use store::{DiagnosticStore, DiagnosticStoreError};
#[derive(Clone)]
pub(crate) struct DiagnosticSources {
pub(crate) repository: Arc<RocksTorrentRepository>,
pub(crate) search: SearchEngine,
pub(crate) dht: DhtRuntimeStats,
pub(crate) persistence: PersistenceIngress,
pub(crate) disk_guard: DiskGuard,
pub(crate) http: HttpStats,
}
#[derive(Clone)]
pub(crate) struct DiagnosticsHandle {
state: Arc<DiagnosticsState>,
database: Option<PathBuf>,
}
pub(crate) struct DiagnosticsRuntime {
handle: DiagnosticsHandle,
cancel: CancellationToken,
collector: Option<tokio::task::JoinHandle<()>>,
writer: Option<JoinHandle<()>>,
}
#[derive(Default)]
struct DiagnosticsState {
enabled: bool,
current: RwLock<Option<DiagnosticSample>>,
persisted_samples: AtomicU64,
dropped_samples: AtomicU64,
skipped_samples: AtomicU64,
write_failures: AtomicU64,
last_persisted_at: AtomicU64,
last_error: Mutex<Option<String>>,
}
#[derive(Debug, thiserror::Error)]
pub(crate) enum DiagnosticsError {
#[error("运行诊断线程启动失败: {0}")]
Io(#[from] std::io::Error),
#[error(transparent)]
Store(#[from] DiagnosticStoreError),
#[error("运行诊断历史未启用")]
Disabled,
#[error("运行诊断 writer 异常退出")]
WriterPanicked,
}
impl DiagnosticsRuntime {
pub(crate) fn disabled() -> Self {
Self {
handle: DiagnosticsHandle {
state: Arc::new(DiagnosticsState::default()),
database: None,
},
cancel: CancellationToken::new(),
collector: None,
writer: None,
}
}
pub(crate) fn start(
config: DiagnosticsConfig,
sources: DiagnosticSources,
) -> Result<Self, DiagnosticsError> {
let store = DiagnosticStore::open(&config.database)?;
let state = Arc::new(DiagnosticsState {
enabled: true,
..DiagnosticsState::default()
});
let (sender, receiver) = sync_channel(config.queue_capacity);
let writer_state = state.clone();
let writer_config = config.clone();
let writer_disk_guard = sources.disk_guard.clone();
let writer = std::thread::Builder::new()
.name("diagnostics-sqlite".to_owned())
.spawn(move || {
let mut store = store;
while let Ok(sample) = receiver.recv() {
let Some(_permit) = writer_disk_guard.begin_new_write() else {
writer_state.skipped_samples.fetch_add(1, Ordering::Relaxed);
continue;
};
match store.record(&sample, &writer_config) {
Ok(()) => {
writer_state
.persisted_samples
.fetch_add(1, Ordering::Relaxed);
writer_state
.last_persisted_at
.store(sample.captured_at, Ordering::Relaxed);
}
Err(error) => {
writer_state.write_failures.fetch_add(1, Ordering::Relaxed);
set_last_error(&writer_state, error.to_string());
tracing::warn!(%error, "运行诊断采样写入失败");
}
}
}
})?;
let cancel = CancellationToken::new();
let collector = tokio::spawn(collect_loop(
sources,
sender,
state.clone(),
config.sample_interval_secs,
cancel.clone(),
));
Ok(Self {
handle: DiagnosticsHandle {
state,
database: Some(config.database),
},
cancel,
collector: Some(collector),
writer: Some(writer),
})
}
pub(crate) fn handle(&self) -> DiagnosticsHandle {
self.handle.clone()
}
pub(crate) fn request_shutdown(&self) {
self.cancel.cancel();
}
pub(crate) async fn shutdown(mut self) -> Result<(), DiagnosticsError> {
self.cancel.cancel();
if let Some(collector) = self.collector.take() {
let _ = collector.await;
}
if let Some(writer) = self.writer.take() {
tokio::task::spawn_blocking(move || writer.join())
.await
.map_err(|_| DiagnosticsError::WriterPanicked)?
.map_err(|_| DiagnosticsError::WriterPanicked)?;
}
Ok(())
}
}
impl DiagnosticsHandle {
pub(crate) fn current(&self) -> CurrentDiagnosticsResponse {
let last_persisted_at = self.state.last_persisted_at.load(Ordering::Relaxed);
CurrentDiagnosticsResponse {
status: DiagnosticsStatus {
enabled: self.state.enabled,
persisted_samples: self.state.persisted_samples.load(Ordering::Relaxed),
dropped_samples: self.state.dropped_samples.load(Ordering::Relaxed),
skipped_samples: self.state.skipped_samples.load(Ordering::Relaxed),
write_failures: self.state.write_failures.load(Ordering::Relaxed),
last_persisted_at: (last_persisted_at != 0).then_some(last_persisted_at),
last_error: self
.state
.last_error
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.clone(),
},
sample: self
.state
.current
.read()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.clone(),
}
}
pub(crate) fn history(
&self,
resolution: HistoryResolution,
from: u64,
to: u64,
) -> Result<DiagnosticHistory, DiagnosticsError> {
let database = self.database.as_ref().ok_or(DiagnosticsError::Disabled)?;
DiagnosticStore::history(database, resolution, from, to).map_err(Into::into)
}
}
async fn collect_loop(
sources: DiagnosticSources,
sender: SyncSender<DiagnosticSample>,
state: Arc<DiagnosticsState>,
interval_secs: u64,
cancel: CancellationToken,
) {
let session_started_at = unix_timestamp();
let mut ticker = tokio::time::interval(Duration::from_secs(interval_secs));
ticker.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
loop {
tokio::select! {
_ = cancel.cancelled() => break,
_ = ticker.tick() => {
let sources = sources.clone();
let sample = match tokio::task::spawn_blocking(move || {
collect_sample(&sources, session_started_at)
}).await {
Ok(sample) => sample,
Err(error) => {
state.dropped_samples.fetch_add(1, Ordering::Relaxed);
set_last_error(&state, error.to_string());
continue;
}
};
*state
.current
.write()
.unwrap_or_else(std::sync::PoisonError::into_inner) = Some(sample.clone());
match sender.try_send(sample) {
Ok(()) => {}
Err(TrySendError::Full(_)) => {
state.dropped_samples.fetch_add(1, Ordering::Relaxed);
}
Err(TrySendError::Disconnected(_)) => {
state.dropped_samples.fetch_add(1, Ordering::Relaxed);
set_last_error(&state, "运行诊断 writer 已停止".to_owned());
break;
}
}
}
}
}
}
fn collect_sample(sources: &DiagnosticSources, session_started_at: u64) -> DiagnosticSample {
let dht = sources.dht.snapshot();
let observability = sources.dht.observability_snapshot();
let persistence = sources.persistence.snapshot();
let disk = sources.disk_guard.snapshot();
let storage = sources.repository.diagnostics().unwrap_or_else(|error| {
tracing::warn!(%error, "读取 RocksDB 诊断属性失败");
RocksDiagnostics::default()
});
let search = sources.search.diagnostics();
DiagnosticSample {
captured_at: unix_timestamp(),
session_started_at,
process: process::snapshot(),
storage: StorageDiagnostics {
block_cache_bytes: storage.block_cache_bytes,
memtable_bytes: storage.memtable_bytes,
pending_compaction_bytes: storage.pending_compaction_bytes,
live_sst_bytes: storage.live_sst_bytes,
running_compactions: storage.running_compactions,
estimated_keys: storage.estimated_keys,
},
search: SearchDiagnostics {
documents: search.documents,
writer_memory_budget_bytes: search.writer_memory_budget_bytes,
commits: search.commits,
commit_failures: search.commit_failures,
last_commit_at: search.last_commit_at,
last_commit_duration_millis: search.last_commit_duration_millis,
last_commit_documents: search.last_commit_documents,
},
http: sources.http.snapshot(),
runtime: RuntimeDiagnostics {
nodes: saturating_u64(dht.node_pool_size),
udp_tx_packets: observability.udp_tx_packets,
metadata_in_flight: saturating_u64(dht.metadata_in_flight),
metadata_succeeded: dht.metadata_peer_succeeded,
metadata_failed: dht.metadata_peer_failed,
sample_queue_depth: saturating_u64(dht.sample_candidate_queue_depth),
sample_queue_capacity: saturating_u64(dht.sample_candidate_queue_capacity),
persistence_queue_depth: saturating_u64(persistence.queue_depth),
indexed_documents: search.documents,
disk_available_bytes: disk.available_bytes,
},
}
}
fn set_last_error(state: &DiagnosticsState, message: String) {
*state
.last_error
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner) = Some(message);
}
fn saturating_u64(value: usize) -> u64 {
value.min(u64::MAX as usize) as u64
}
fn unix_timestamp() -> u64 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap_or_default()
.as_secs()
}
#[cfg(test)]
mod tests {
use crate::{
search::SearchEngine,
storage::{RocksTorrentRepository, TorrentRepository},
};
use tempfile::TempDir;
use super::*;
use crate::{
config::DiskGuardConfig, crawler::pipeline::PersistencePipeline, disk_guard::DiskGuard,
};
#[tokio::test]
async fn runtime_persists_a_queryable_snapshot_and_stops_cleanly() {
let directory = TempDir::new().unwrap();
let repository =
Arc::new(RocksTorrentRepository::open(directory.path().join("rocksdb")).unwrap());
let repository_trait: Arc<dyn TorrentRepository> = repository.clone();
let disk_guard = DiskGuard::new(&DiskGuardConfig {
enabled: false,
..DiskGuardConfig::default()
});
let persistence = PersistencePipeline::start(
repository_trait,
4,
crate::domain::MetadataLimits::default(),
disk_guard.clone(),
);
let config = DiagnosticsConfig {
database: directory.path().join("diagnostics.sqlite3"),
sample_interval_secs: 1,
..DiagnosticsConfig::default()
};
let runtime = DiagnosticsRuntime::start(
config,
DiagnosticSources {
repository,
search: SearchEngine::open(directory.path().join("tantivy")).unwrap(),
dht: DhtRuntimeStats::default(),
persistence: persistence.ingress.clone(),
disk_guard,
http: HttpStats::default(),
},
)
.unwrap();
let handle = runtime.handle();
for _ in 0..40 {
if handle.current().status.persisted_samples > 0 {
break;
}
tokio::time::sleep(Duration::from_millis(25)).await;
}
let current = handle.current();
assert!(current.status.enabled);
assert_eq!(current.status.write_failures, 0);
assert!(current.sample.is_some());
let now = unix_timestamp();
let history = handle
.history(HistoryResolution::Raw, now.saturating_sub(5), now)
.unwrap();
assert!(!history.samples.is_empty());
runtime.shutdown().await.unwrap();
persistence.close_and_join().await.unwrap();
}
}
+117
View File
@@ -0,0 +1,117 @@
// 负责定义运行诊断采样状态和历史查询 DTO
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
pub(crate) struct DiagnosticSample {
pub(crate) captured_at: u64,
pub(crate) session_started_at: u64,
pub(crate) process: ProcessDiagnostics,
pub(crate) storage: StorageDiagnostics,
pub(crate) search: SearchDiagnostics,
#[serde(default)]
pub(crate) http: HttpDiagnostics,
pub(crate) runtime: RuntimeDiagnostics,
}
#[derive(Debug, Clone, Copy, Default, Serialize, Deserialize, PartialEq, Eq)]
pub(crate) struct ProcessDiagnostics {
pub(crate) resident_memory_bytes: Option<u64>,
pub(crate) private_memory_bytes: Option<u64>,
pub(crate) cpu_time_millis: Option<u64>,
pub(crate) thread_count: Option<u64>,
pub(crate) handle_count: Option<u64>,
}
#[derive(Debug, Clone, Copy, Default, Serialize, Deserialize, PartialEq, Eq)]
pub(crate) struct StorageDiagnostics {
pub(crate) block_cache_bytes: Option<u64>,
pub(crate) memtable_bytes: Option<u64>,
pub(crate) pending_compaction_bytes: Option<u64>,
pub(crate) live_sst_bytes: Option<u64>,
pub(crate) running_compactions: Option<u64>,
pub(crate) estimated_keys: Option<u64>,
}
#[derive(Debug, Clone, Copy, Default, Serialize, Deserialize, PartialEq, Eq)]
pub(crate) struct SearchDiagnostics {
pub(crate) documents: u64,
pub(crate) writer_memory_budget_bytes: u64,
pub(crate) commits: u64,
pub(crate) commit_failures: u64,
pub(crate) last_commit_at: Option<u64>,
pub(crate) last_commit_duration_millis: u64,
pub(crate) last_commit_documents: u64,
}
#[derive(Debug, Clone, Copy, Default, Serialize, Deserialize, PartialEq, Eq)]
pub(crate) struct HttpDiagnostics {
pub(crate) active_requests: u64,
pub(crate) requests: u64,
pub(crate) client_errors: u64,
pub(crate) server_errors: u64,
pub(crate) latency_average_micros: Option<u64>,
pub(crate) latency_p95_millis: Option<u64>,
pub(crate) latency_max_millis: u64,
}
#[derive(Debug, Clone, Copy, Default, Serialize, Deserialize, PartialEq, Eq)]
pub(crate) struct RuntimeDiagnostics {
pub(crate) nodes: u64,
pub(crate) udp_tx_packets: u64,
pub(crate) metadata_in_flight: u64,
pub(crate) metadata_succeeded: u64,
pub(crate) metadata_failed: u64,
pub(crate) sample_queue_depth: u64,
pub(crate) sample_queue_capacity: u64,
pub(crate) persistence_queue_depth: u64,
pub(crate) indexed_documents: u64,
pub(crate) disk_available_bytes: Option<u64>,
}
#[derive(Debug, Clone, Serialize, PartialEq, Eq)]
pub(crate) struct DiagnosticsStatus {
pub(crate) enabled: bool,
pub(crate) persisted_samples: u64,
pub(crate) dropped_samples: u64,
pub(crate) skipped_samples: u64,
pub(crate) write_failures: u64,
pub(crate) last_persisted_at: Option<u64>,
pub(crate) last_error: Option<String>,
}
#[derive(Debug, Clone, Serialize)]
pub(crate) struct CurrentDiagnosticsResponse {
pub(crate) status: DiagnosticsStatus,
pub(crate) sample: Option<DiagnosticSample>,
}
#[derive(Debug, Clone, Serialize)]
pub(crate) struct DiagnosticHistory {
pub(crate) resolution: &'static str,
pub(crate) from: u64,
pub(crate) to: u64,
pub(crate) samples: Vec<DiagnosticSample>,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum HistoryResolution {
Raw,
Minute,
}
impl HistoryResolution {
pub(crate) const fn as_str(self) -> &'static str {
match self {
Self::Raw => "raw",
Self::Minute => "minute",
}
}
pub(crate) const fn database_value(self) -> i64 {
match self {
Self::Raw => 0,
Self::Minute => 1,
}
}
}
+167
View File
@@ -0,0 +1,167 @@
// 负责采集当前进程内存 CPU 线程和句柄资源快照
use super::model::ProcessDiagnostics;
#[cfg(windows)]
pub(crate) fn snapshot() -> ProcessDiagnostics {
use std::mem::size_of;
use windows_sys::Win32::{
Foundation::{CloseHandle, FILETIME, INVALID_HANDLE_VALUE},
System::{
Diagnostics::ToolHelp::{
CreateToolhelp32Snapshot, TH32CS_SNAPTHREAD, THREADENTRY32, Thread32First,
Thread32Next,
},
ProcessStatus::{
K32GetProcessMemoryInfo, PROCESS_MEMORY_COUNTERS, PROCESS_MEMORY_COUNTERS_EX,
},
Threading::{
GetCurrentProcess, GetCurrentProcessId, GetProcessHandleCount, GetProcessTimes,
},
},
};
let process = unsafe { GetCurrentProcess() };
let mut memory = PROCESS_MEMORY_COUNTERS_EX {
cb: size_of::<PROCESS_MEMORY_COUNTERS_EX>() as u32,
..PROCESS_MEMORY_COUNTERS_EX::default()
};
let memory_ok = unsafe {
K32GetProcessMemoryInfo(
process,
(&raw mut memory).cast::<PROCESS_MEMORY_COUNTERS>(),
memory.cb,
) != 0
};
let mut handles = 0_u32;
let handles_ok = unsafe { GetProcessHandleCount(process, &raw mut handles) != 0 };
let mut creation = FILETIME::default();
let mut exit = FILETIME::default();
let mut kernel = FILETIME::default();
let mut user = FILETIME::default();
let cpu_ok = unsafe {
GetProcessTimes(
process,
&raw mut creation,
&raw mut exit,
&raw mut kernel,
&raw mut user,
) != 0
};
let result = ProcessDiagnostics {
resident_memory_bytes: memory_ok.then_some(memory.WorkingSetSize as u64),
private_memory_bytes: memory_ok.then_some(memory.PrivateUsage as u64),
cpu_time_millis: cpu_ok
.then(|| filetime_ticks(kernel).saturating_add(filetime_ticks(user)) / 10_000),
thread_count: windows_thread_count(),
handle_count: handles_ok.then_some(u64::from(handles)),
};
fn filetime_ticks(value: FILETIME) -> u64 {
(u64::from(value.dwHighDateTime) << 32) | u64::from(value.dwLowDateTime)
}
fn windows_thread_count() -> Option<u64> {
let snapshot = unsafe { CreateToolhelp32Snapshot(TH32CS_SNAPTHREAD, 0) };
if snapshot == INVALID_HANDLE_VALUE {
return None;
}
let process_id = unsafe { GetCurrentProcessId() };
let mut entry = THREADENTRY32 {
dwSize: size_of::<THREADENTRY32>() as u32,
..THREADENTRY32::default()
};
let mut count = 0_u64;
let mut has_entry = unsafe { Thread32First(snapshot, &raw mut entry) != 0 };
while has_entry {
if entry.th32OwnerProcessID == process_id {
count = count.saturating_add(1);
}
has_entry = unsafe { Thread32Next(snapshot, &raw mut entry) != 0 };
}
unsafe {
CloseHandle(snapshot);
}
Some(count)
}
result
}
#[cfg(target_os = "linux")]
pub(crate) fn snapshot() -> ProcessDiagnostics {
let status = std::fs::read_to_string("/proc/self/status").unwrap_or_default();
ProcessDiagnostics {
resident_memory_bytes: status_kib(&status, "VmRSS:"),
private_memory_bytes: status_kib(&status, "RssAnon:")
.or_else(|| status_kib(&status, "VmData:")),
cpu_time_millis: linux_cpu_time_millis(),
thread_count: status_value(&status, "Threads:"),
handle_count: std::fs::read_dir("/proc/self/fd")
.ok()
.map(|entries| entries.count().min(u64::MAX as usize) as u64),
}
}
#[cfg(target_os = "linux")]
fn status_kib(status: &str, key: &str) -> Option<u64> {
status_value(status, key).map(|value| value.saturating_mul(1_024))
}
#[cfg(target_os = "linux")]
fn status_value(status: &str, key: &str) -> Option<u64> {
status
.lines()
.find_map(|line| line.strip_prefix(key))?
.split_whitespace()
.next()?
.parse()
.ok()
}
#[cfg(target_os = "linux")]
fn linux_cpu_time_millis() -> Option<u64> {
let stat = std::fs::read_to_string("/proc/self/stat").ok()?;
let fields = stat.get(stat.rfind(')')?.saturating_add(2)..)?;
let mut fields = fields.split_whitespace();
let user_ticks: u64 = fields.nth(11)?.parse().ok()?;
let system_ticks: u64 = fields.next()?.parse().ok()?;
let ticks_per_second = unsafe { libc::sysconf(libc::_SC_CLK_TCK) };
if ticks_per_second <= 0 {
return None;
}
Some(
user_ticks
.saturating_add(system_ticks)
.saturating_mul(1_000)
/ ticks_per_second as u64,
)
}
#[cfg(not(any(windows, target_os = "linux")))]
pub(crate) fn snapshot() -> ProcessDiagnostics {
ProcessDiagnostics::default()
}
#[cfg(test)]
mod tests {
#[test]
fn current_process_reports_available_platform_resources() {
let snapshot = super::snapshot();
#[cfg(any(windows, target_os = "linux"))]
{
assert!(
snapshot
.resident_memory_bytes
.is_some_and(|value| value > 0)
);
assert!(snapshot.cpu_time_millis.is_some());
assert!(snapshot.thread_count.is_some_and(|value| value > 0));
assert!(snapshot.handle_count.is_some_and(|value| value > 0));
}
}
}
+209
View File
@@ -0,0 +1,209 @@
// 负责使用独立 SQLite 数据库持久化和查询有界诊断历史
use std::{fs, path::Path};
use rusqlite::{Connection, OpenFlags, params};
use crate::config::DiagnosticsConfig;
use super::model::{DiagnosticHistory, DiagnosticSample, HistoryResolution};
const DATABASE_VERSION: i64 = 1;
const MAX_HISTORY_SAMPLES: usize = 50_000;
pub(crate) struct DiagnosticStore {
connection: Connection,
}
#[derive(Debug, thiserror::Error)]
pub(crate) enum DiagnosticStoreError {
#[error("无法准备诊断数据库目录: {0}")]
Io(#[from] std::io::Error),
#[error("SQLite 诊断数据库操作失败: {0}")]
Sqlite(#[from] rusqlite::Error),
#[error("诊断采样序列化失败: {0}")]
Json(#[from] serde_json::Error),
#[error("诊断数据库格式版本不兼容")]
IncompatibleVersion,
}
impl DiagnosticStore {
pub(crate) fn open(path: &Path) -> Result<Self, DiagnosticStoreError> {
if let Some(parent) = path.parent() {
fs::create_dir_all(parent)?;
}
let connection = Connection::open(path)?;
configure(&connection)?;
initialize(&connection)?;
Ok(Self { connection })
}
pub(crate) fn record(
&mut self,
sample: &DiagnosticSample,
config: &DiagnosticsConfig,
) -> Result<(), DiagnosticStoreError> {
let payload = serde_json::to_string(sample)?;
let transaction = self.connection.transaction()?;
transaction.execute(
"INSERT OR REPLACE INTO diagnostic_samples (kind, captured_at, payload) VALUES (0, ?1, ?2)",
params![as_i64(sample.captured_at), payload],
)?;
let minute = sample.captured_at / 60 * 60;
transaction.execute(
"INSERT OR REPLACE INTO diagnostic_samples (kind, captured_at, payload) VALUES (1, ?1, ?2)",
params![as_i64(minute), payload],
)?;
let raw_cutoff = sample
.captured_at
.saturating_sub(config.raw_retention_hours.saturating_mul(3_600));
let minute_cutoff = sample
.captured_at
.saturating_sub(config.minute_retention_days.saturating_mul(86_400));
transaction.execute(
"DELETE FROM diagnostic_samples WHERE kind = 0 AND captured_at < ?1",
[as_i64(raw_cutoff)],
)?;
transaction.execute(
"DELETE FROM diagnostic_samples WHERE kind = 1 AND captured_at < ?1",
[as_i64(minute_cutoff)],
)?;
transaction.commit()?;
Ok(())
}
pub(crate) fn history(
path: &Path,
resolution: HistoryResolution,
from: u64,
to: u64,
) -> Result<DiagnosticHistory, DiagnosticStoreError> {
let connection = Connection::open_with_flags(
path,
OpenFlags::SQLITE_OPEN_READ_ONLY | OpenFlags::SQLITE_OPEN_NO_MUTEX,
)?;
let mut statement = connection.prepare(
"SELECT payload FROM diagnostic_samples
WHERE kind = ?1 AND captured_at >= ?2 AND captured_at <= ?3
ORDER BY captured_at ASC LIMIT ?4",
)?;
let rows = statement.query_map(
params![
resolution.database_value(),
as_i64(from),
as_i64(to),
MAX_HISTORY_SAMPLES as i64
],
|row| row.get::<_, String>(0),
)?;
let mut samples = Vec::new();
for payload in rows {
samples.push(serde_json::from_str(&payload?)?);
}
Ok(DiagnosticHistory {
resolution: resolution.as_str(),
from,
to,
samples,
})
}
#[cfg(test)]
fn count(&self, resolution: HistoryResolution) -> Result<u64, DiagnosticStoreError> {
let count: i64 = self
.connection
.query_row(
"SELECT COUNT(*) FROM diagnostic_samples WHERE kind = ?1",
[resolution.database_value()],
|row| row.get(0),
)
.map_err(DiagnosticStoreError::from)?;
Ok(count.max(0) as u64)
}
}
fn configure(connection: &Connection) -> Result<(), rusqlite::Error> {
connection.pragma_update(None, "journal_mode", "WAL")?;
connection.pragma_update(None, "synchronous", "NORMAL")?;
connection.busy_timeout(std::time::Duration::from_secs(5))
}
fn initialize(connection: &Connection) -> Result<(), DiagnosticStoreError> {
let version: i64 = connection.pragma_query_value(None, "user_version", |row| row.get(0))?;
if version != 0 && version != DATABASE_VERSION {
return Err(DiagnosticStoreError::IncompatibleVersion);
}
connection.execute_batch(
"CREATE TABLE IF NOT EXISTS diagnostic_samples (
kind INTEGER NOT NULL,
captured_at INTEGER NOT NULL,
payload TEXT NOT NULL,
PRIMARY KEY (kind, captured_at)
) WITHOUT ROWID;",
)?;
connection.pragma_update(None, "user_version", DATABASE_VERSION)?;
Ok(())
}
fn as_i64(value: u64) -> i64 {
value.min(i64::MAX as u64) as i64
}
#[cfg(test)]
mod tests {
use tempfile::TempDir;
use super::*;
use crate::diagnostics::model::{
HttpDiagnostics, ProcessDiagnostics, RuntimeDiagnostics, SearchDiagnostics,
StorageDiagnostics,
};
fn sample(captured_at: u64) -> DiagnosticSample {
DiagnosticSample {
captured_at,
session_started_at: 1,
process: ProcessDiagnostics::default(),
storage: StorageDiagnostics::default(),
search: SearchDiagnostics::default(),
http: HttpDiagnostics::default(),
runtime: RuntimeDiagnostics::default(),
}
}
#[test]
fn samples_survive_reopen_and_minute_rows_are_compacted() {
let directory = TempDir::new().unwrap();
let path = directory.path().join("diagnostics.sqlite3");
let config = DiagnosticsConfig {
raw_retention_hours: 1,
minute_retention_days: 1,
..DiagnosticsConfig::default()
};
let mut store = DiagnosticStore::open(&path).unwrap();
store.record(&sample(3_600), &config).unwrap();
store.record(&sample(3_610), &config).unwrap();
assert_eq!(store.count(HistoryResolution::Raw).unwrap(), 2);
assert_eq!(store.count(HistoryResolution::Minute).unwrap(), 1);
drop(store);
let history = DiagnosticStore::history(&path, HistoryResolution::Raw, 0, 4_000).unwrap();
assert_eq!(history.samples, vec![sample(3_600), sample(3_610)]);
}
#[test]
fn retention_deletes_only_expired_resolution_rows() {
let directory = TempDir::new().unwrap();
let path = directory.path().join("diagnostics.sqlite3");
let config = DiagnosticsConfig {
raw_retention_hours: 1,
minute_retention_days: 1,
..DiagnosticsConfig::default()
};
let mut store = DiagnosticStore::open(&path).unwrap();
store.record(&sample(1), &config).unwrap();
store.record(&sample(90_000), &config).unwrap();
assert_eq!(store.count(HistoryResolution::Raw).unwrap(), 1);
assert_eq!(store.count(HistoryResolution::Minute).unwrap(), 1);
}
}
+346
View File
@@ -0,0 +1,346 @@
// 负责监测数据盘剩余空间并协调所有后台写入进入可恢复的只读状态
use std::{
io,
path::{Path, PathBuf},
sync::{
Arc, Mutex,
atomic::{AtomicBool, AtomicU64, Ordering},
},
time::Duration,
};
use tokio_util::sync::CancellationToken;
use crate::{config::DiskGuardConfig, crawler::pipeline::PersistenceIngress};
const UNKNOWN_AVAILABLE_BYTES: u64 = u64::MAX;
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum DiskMode {
Normal,
Draining,
ReadOnly,
}
impl DiskMode {
pub(crate) const fn as_str(self) -> &'static str {
match self {
Self::Normal => "normal",
Self::Draining => "draining",
Self::ReadOnly => "read_only",
}
}
}
#[derive(Debug, Clone, Copy)]
pub(crate) struct DiskGuardSnapshot {
pub(crate) mode: DiskMode,
pub(crate) available_bytes: Option<u64>,
pub(crate) minimum_free_bytes: u64,
pub(crate) resume_free_bytes: u64,
pub(crate) active_writes: usize,
pub(crate) probe_failed: bool,
pub(crate) probe_failures: u64,
pub(crate) transitions: u64,
pub(crate) rejected_new_work: u64,
}
#[derive(Clone)]
pub(crate) struct DiskGuard {
inner: Arc<DiskGuardInner>,
}
struct DiskGuardInner {
enabled: bool,
minimum_free_bytes: u64,
resume_free_bytes: u64,
state: Mutex<GateState>,
available_bytes: AtomicU64,
probe_failed: AtomicBool,
probe_failures: AtomicU64,
transitions: AtomicU64,
rejected_new_work: AtomicU64,
}
struct GateState {
mode: DiskMode,
active_writes: usize,
}
pub(crate) struct DiskWritePermit {
inner: Arc<DiskGuardInner>,
}
pub(crate) async fn run(
guard: DiskGuard,
data_dir: PathBuf,
config: DiskGuardConfig,
persistence: PersistenceIngress,
cancel: CancellationToken,
) {
let mut ticker = tokio::time::interval(Duration::from_secs(config.check_interval_secs));
ticker.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
loop {
tokio::select! {
_ = cancel.cancelled() => break,
_ = ticker.tick() => {
guard.probe(&data_dir, persistence.snapshot().queue_depth);
}
}
}
}
impl DiskGuard {
pub(crate) fn new(config: &DiskGuardConfig) -> Self {
Self {
inner: Arc::new(DiskGuardInner {
enabled: config.enabled,
minimum_free_bytes: config.minimum_free_bytes,
resume_free_bytes: config.resume_free_bytes,
state: Mutex::new(GateState {
mode: DiskMode::Normal,
active_writes: 0,
}),
available_bytes: AtomicU64::new(UNKNOWN_AVAILABLE_BYTES),
probe_failed: AtomicBool::new(false),
probe_failures: AtomicU64::new(0),
transitions: AtomicU64::new(0),
rejected_new_work: AtomicU64::new(0),
}),
}
}
pub(crate) fn probe(&self, path: &Path, persistence_queue: usize) {
match fs2::available_space(path) {
Ok(available) => self.observe_available(available, persistence_queue),
Err(error) => self.observe_probe_error(&error, persistence_queue),
}
}
pub(crate) fn begin_admission(&self) -> Option<DiskWritePermit> {
let permit = self.begin_write(false);
if permit.is_none() {
self.inner.rejected_new_work.fetch_add(1, Ordering::Relaxed);
}
permit
}
pub(crate) fn begin_new_write(&self) -> Option<DiskWritePermit> {
self.begin_write(false)
}
pub(crate) fn begin_drain_write(&self) -> Option<DiskWritePermit> {
self.begin_write(true)
}
#[cfg(test)]
pub(crate) fn mode(&self) -> DiskMode {
self.inner
.state
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.mode
}
pub(crate) fn snapshot(&self) -> DiskGuardSnapshot {
let state = self
.inner
.state
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
let available = self.inner.available_bytes.load(Ordering::Relaxed);
DiskGuardSnapshot {
mode: state.mode,
available_bytes: (available != UNKNOWN_AVAILABLE_BYTES).then_some(available),
minimum_free_bytes: self.inner.minimum_free_bytes,
resume_free_bytes: self.inner.resume_free_bytes,
active_writes: state.active_writes,
probe_failed: self.inner.probe_failed.load(Ordering::Relaxed),
probe_failures: self.inner.probe_failures.load(Ordering::Relaxed),
transitions: self.inner.transitions.load(Ordering::Relaxed),
rejected_new_work: self.inner.rejected_new_work.load(Ordering::Relaxed),
}
}
fn begin_write(&self, allow_draining: bool) -> Option<DiskWritePermit> {
if !self.inner.enabled {
let mut state = self
.inner
.state
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
state.active_writes = state.active_writes.saturating_add(1);
return Some(DiskWritePermit {
inner: self.inner.clone(),
});
}
let mut state = self
.inner
.state
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
let allowed =
state.mode == DiskMode::Normal || (allow_draining && state.mode == DiskMode::Draining);
if !allowed {
return None;
}
state.active_writes = state.active_writes.saturating_add(1);
Some(DiskWritePermit {
inner: self.inner.clone(),
})
}
pub(crate) fn observe_available(&self, available: u64, persistence_queue: usize) {
self.inner
.available_bytes
.store(available, Ordering::Relaxed);
self.inner.probe_failed.store(false, Ordering::Relaxed);
if !self.inner.enabled {
return;
}
if available < self.inner.minimum_free_bytes {
self.transition_to(DiskMode::Draining, Some(available), None);
self.finish_draining(persistence_queue);
} else if available >= self.inner.resume_free_bytes {
self.transition_to(DiskMode::Normal, Some(available), None);
} else {
self.finish_draining(persistence_queue);
}
}
fn observe_probe_error(&self, error: &io::Error, persistence_queue: usize) {
self.inner
.available_bytes
.store(UNKNOWN_AVAILABLE_BYTES, Ordering::Relaxed);
self.inner.probe_failed.store(true, Ordering::Relaxed);
self.inner.probe_failures.fetch_add(1, Ordering::Relaxed);
if !self.inner.enabled {
return;
}
self.transition_to(DiskMode::Draining, None, Some(error));
self.finish_draining(persistence_queue);
}
fn finish_draining(&self, persistence_queue: usize) {
if persistence_queue != 0 {
return;
}
let mut state = self
.inner
.state
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
if state.mode == DiskMode::Draining && state.active_writes == 0 {
let old = state.mode;
state.mode = DiskMode::ReadOnly;
drop(state);
self.record_transition(old, DiskMode::ReadOnly, None, None);
}
}
fn transition_to(&self, target: DiskMode, available: Option<u64>, error: Option<&io::Error>) {
let mut state = self
.inner
.state
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
if state.mode == target
|| (state.mode == DiskMode::ReadOnly && target == DiskMode::Draining)
{
return;
}
let old = state.mode;
state.mode = target;
drop(state);
self.record_transition(old, target, available, error);
}
fn record_transition(
&self,
old: DiskMode,
new: DiskMode,
available: Option<u64>,
error: Option<&io::Error>,
) {
self.inner.transitions.fetch_add(1, Ordering::Relaxed);
match new {
DiskMode::Normal => tracing::info!(
previous = old.as_str(),
available_bytes = available,
"磁盘空间恢复并重新接受写入"
),
DiskMode::Draining => tracing::warn!(
previous = old.as_str(),
available_bytes = available,
error = error.map(ToString::to_string),
"磁盘空间不足并停止接收新任务"
),
DiskMode::ReadOnly => {
tracing::warn!(previous = old.as_str(), "待写入任务已排空并进入只读保护")
}
}
}
}
impl Drop for DiskWritePermit {
fn drop(&mut self) {
let mut state = self
.inner
.state
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
state.active_writes = state.active_writes.saturating_sub(1);
}
}
#[cfg(test)]
mod tests {
use super::*;
fn guard() -> DiskGuard {
DiskGuard::new(&DiskGuardConfig {
enabled: true,
check_interval_secs: 1,
minimum_free_bytes: 100,
resume_free_bytes: 200,
})
}
#[test]
fn low_space_drains_then_enters_read_only_and_recovers_with_hysteresis() {
let guard = guard();
let permit = guard.begin_new_write().unwrap();
guard.observe_available(99, 1);
assert_eq!(guard.mode(), DiskMode::Draining);
assert!(guard.begin_admission().is_none());
guard.observe_available(150, 0);
assert_eq!(guard.mode(), DiskMode::Draining);
drop(permit);
guard.observe_available(150, 0);
assert_eq!(guard.mode(), DiskMode::ReadOnly);
assert!(guard.begin_new_write().is_none());
guard.observe_available(199, 0);
assert_eq!(guard.mode(), DiskMode::ReadOnly);
guard.observe_available(200, 0);
assert_eq!(guard.mode(), DiskMode::Normal);
}
#[test]
fn probe_failure_conservatively_enters_read_only() {
let guard = guard();
guard.observe_probe_error(&io::Error::other("probe failed"), 0);
let snapshot = guard.snapshot();
assert_eq!(snapshot.mode, DiskMode::ReadOnly);
assert!(snapshot.probe_failed);
assert_eq!(snapshot.probe_failures, 1);
}
#[test]
fn draining_allows_only_existing_queue_writes() {
let guard = guard();
guard.observe_available(1, 1);
assert!(guard.begin_new_write().is_none());
assert!(guard.begin_drain_write().is_some());
}
}
+408
View File
@@ -0,0 +1,408 @@
// 负责定义可配置的无效文件识别规则和面向用户的有效内容视图
use std::collections::HashSet;
use regex::{Regex, RegexBuilder};
use serde::{Deserialize, Serialize};
use unicode_normalization::UnicodeNormalization;
use super::{TorrentFile, TorrentRecord, TorrentRecordError, content_key};
const FILTER_FORMAT_VERSION: u64 = 1;
const MAX_RULES: usize = 256;
const MAX_PATTERN_BYTES: usize = 1_024;
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct ContentFilterConfig {
pub version: u64,
#[serde(default)]
pub file_rules: Vec<FileFilterRule>,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct FileFilterRule {
pub id: String,
#[serde(default = "default_true")]
pub enabled: bool,
pub field: FileMatchField,
#[serde(rename = "match")]
pub match_kind: FileMatchKind,
pub value: String,
#[serde(default)]
pub case_sensitive: bool,
pub action: FileRuleAction,
pub reason: String,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "kebab-case")]
pub enum FileMatchField {
FileName,
FilePath,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "kebab-case")]
pub enum FileMatchKind {
Exact,
Prefix,
Suffix,
Contains,
Wildcard,
Regex,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "kebab-case")]
pub enum FileRuleAction {
Hide,
}
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub struct FilterOutcome {
pub hidden_files: usize,
pub searchable: bool,
}
#[derive(Debug, thiserror::Error)]
pub enum ContentFilterError {
#[error("内容过滤规则版本 {0} 不受支持")]
UnsupportedVersion(u64),
#[error("内容过滤规则数量 {actual} 超过上限 {limit}")]
TooManyRules { actual: usize, limit: usize },
#[error("内容过滤规则 ID 不能为空")]
EmptyRuleId,
#[error("内容过滤规则 ID 重复: {0}")]
DuplicateRuleId(String),
#[error("内容过滤规则 {0} 的匹配值不能为空")]
EmptyPattern(String),
#[error("内容过滤规则 {id} 的匹配值超过 {limit} 字节")]
PatternTooLong { id: String, limit: usize },
#[error("内容过滤规则 {id} 的正则表达式无效: {source}")]
InvalidRegex { id: String, source: regex::Error },
#[error("内容过滤规则无法生成稳定指纹: {0}")]
Fingerprint(serde_json::Error),
}
#[derive(Debug)]
pub struct ContentFilter {
fingerprint: [u8; 32],
rules: Vec<CompiledRule>,
}
#[derive(Debug)]
struct CompiledRule {
field: FileMatchField,
case_sensitive: bool,
matcher: CompiledMatcher,
}
#[derive(Debug)]
enum CompiledMatcher {
Exact(String),
Prefix(String),
Suffix(String),
Contains(String),
Pattern(Regex),
}
impl ContentFilter {
pub fn compile(config: ContentFilterConfig) -> Result<Self, ContentFilterError> {
if config.version != FILTER_FORMAT_VERSION {
return Err(ContentFilterError::UnsupportedVersion(config.version));
}
if config.file_rules.len() > MAX_RULES {
return Err(ContentFilterError::TooManyRules {
actual: config.file_rules.len(),
limit: MAX_RULES,
});
}
let fingerprint =
*blake3::hash(&serde_json::to_vec(&config).map_err(ContentFilterError::Fingerprint)?)
.as_bytes();
let mut ids = HashSet::with_capacity(config.file_rules.len());
let mut rules = Vec::new();
for rule in config.file_rules {
if rule.id.trim().is_empty() {
return Err(ContentFilterError::EmptyRuleId);
}
if !ids.insert(rule.id.clone()) {
return Err(ContentFilterError::DuplicateRuleId(rule.id));
}
if rule.value.is_empty() {
return Err(ContentFilterError::EmptyPattern(rule.id));
}
if rule.value.len() > MAX_PATTERN_BYTES {
return Err(ContentFilterError::PatternTooLong {
id: rule.id,
limit: MAX_PATTERN_BYTES,
});
}
if !rule.enabled {
continue;
}
let mut value = normalize(&rule.value, rule.case_sensitive);
if rule.field == FileMatchField::FilePath && rule.match_kind != FileMatchKind::Regex {
value = value.replace('\\', "/");
}
let matcher = match rule.match_kind {
FileMatchKind::Exact => CompiledMatcher::Exact(value),
FileMatchKind::Prefix => CompiledMatcher::Prefix(value),
FileMatchKind::Suffix => CompiledMatcher::Suffix(value),
FileMatchKind::Contains => CompiledMatcher::Contains(value),
FileMatchKind::Wildcard => CompiledMatcher::Pattern(
RegexBuilder::new(&glob_regex(&value))
.case_insensitive(false)
.build()
.map_err(|source| ContentFilterError::InvalidRegex {
id: rule.id.clone(),
source,
})?,
),
FileMatchKind::Regex => CompiledMatcher::Pattern(
RegexBuilder::new(&rule.value)
.case_insensitive(!rule.case_sensitive)
.build()
.map_err(|source| ContentFilterError::InvalidRegex {
id: rule.id.clone(),
source,
})?,
),
};
rules.push(CompiledRule {
field: rule.field,
case_sensitive: rule.case_sensitive || rule.match_kind == FileMatchKind::Regex,
matcher,
});
}
Ok(Self { fingerprint, rules })
}
pub fn fingerprint(&self) -> [u8; 32] {
self.fingerprint
}
pub fn apply_derivatives(
&self,
record: &mut TorrentRecord,
) -> Result<FilterOutcome, TorrentRecordError> {
let visible: Vec<_> = record
.files
.iter()
.filter(|file| !self.is_hidden(file))
.cloned()
.collect();
let hidden_files = record.files.len().saturating_sub(visible.len());
record.searchable = !visible.is_empty();
record.content_key = if record.searchable {
content_key(&visible)?
} else {
[0; 32]
};
Ok(FilterOutcome {
hidden_files,
searchable: record.searchable,
})
}
pub fn public_record(&self, record: &TorrentRecord) -> Option<TorrentRecord> {
if !record.searchable {
return None;
}
let mut public = record.clone();
public.files.retain(|file| !self.is_hidden(file));
if public.files.is_empty() {
return None;
}
public.total_size = public
.files
.iter()
.fold(0_u64, |total, file| total.saturating_add(file.size));
Some(public)
}
pub fn is_hidden(&self, file: &TorrentFile) -> bool {
self.rules.iter().any(|rule| rule.matches(file))
}
}
impl Default for ContentFilter {
fn default() -> Self {
Self::compile(ContentFilterConfig {
version: FILTER_FORMAT_VERSION,
file_rules: Vec::new(),
})
.expect("空内容过滤规则必须有效")
}
}
impl CompiledRule {
fn matches(&self, file: &TorrentFile) -> bool {
let target = match self.field {
FileMatchField::FileName => file
.path
.rsplit(['/', '\\'])
.next()
.unwrap_or(file.path.as_str()),
FileMatchField::FilePath => &file.path,
};
let mut target = normalize(target, self.case_sensitive);
if self.field == FileMatchField::FilePath {
target = target.replace('\\', "/");
}
match &self.matcher {
CompiledMatcher::Exact(value) => target == *value,
CompiledMatcher::Prefix(value) => target.starts_with(value),
CompiledMatcher::Suffix(value) => target.ends_with(value),
CompiledMatcher::Contains(value) => target.contains(value),
CompiledMatcher::Pattern(pattern) => pattern.is_match(&target),
}
}
}
fn normalize(value: &str, case_sensitive: bool) -> String {
let normalized: String = value.nfkc().collect();
if case_sensitive {
normalized
} else {
normalized.to_lowercase()
}
}
fn glob_regex(pattern: &str) -> String {
let mut output = String::from("^");
let mut escaped = false;
for character in pattern.chars() {
if escaped {
output.push_str(&regex::escape(&character.to_string()));
escaped = false;
} else {
match character {
'\\' => escaped = true,
'*' => output.push_str(".*"),
'?' => output.push('.'),
_ => output.push_str(&regex::escape(&character.to_string())),
}
}
}
if escaped {
output.push_str(r"\\");
}
output.push('$');
output
}
const fn default_true() -> bool {
true
}
#[cfg(test)]
mod tests {
use super::*;
use crate::domain::test_record;
fn filter(rules: Vec<FileFilterRule>) -> ContentFilter {
ContentFilter::compile(ContentFilterConfig {
version: 1,
file_rules: rules,
})
.unwrap()
}
fn rule(field: FileMatchField, match_kind: FileMatchKind, value: &str) -> FileFilterRule {
FileFilterRule {
id: format!("{field:?}-{match_kind:?}"),
enabled: true,
field,
match_kind,
value: value.into(),
case_sensitive: false,
action: FileRuleAction::Hide,
reason: "测试".into(),
}
}
#[test]
fn prefix_rule_hides_bitcomet_padding_case_insensitively() {
let filter = filter(vec![rule(
FileMatchField::FileName,
FileMatchKind::Prefix,
"_____padding_file_",
)]);
assert!(filter.is_hidden(&TorrentFile {
path: "目录/_____PADDING_FILE_1_请升级____".into(),
size: 16,
}));
}
#[test]
fn wildcard_matches_the_whole_selected_field() {
let filter = filter(vec![rule(
FileMatchField::FilePath,
FileMatchKind::Wildcard,
".pad/*",
)]);
assert!(filter.is_hidden(&TorrentFile {
path: ".pad/123".into(),
size: 1,
}));
assert!(!filter.is_hidden(&TorrentFile {
path: "movie.pad/123".into(),
size: 1,
}));
}
#[test]
fn regex_rule_matches_nested_libtorrent_padding_directory() {
let filter = filter(vec![rule(
FileMatchField::FilePath,
FileMatchKind::Regex,
r"(^|/)\.____padding_file/",
)]);
assert!(filter.is_hidden(&TorrentFile {
path: "release/.____padding_file/47".into(),
size: 1,
}));
assert!(!filter.is_hidden(&TorrentFile {
path: "release/real_padding_file.txt".into(),
size: 1,
}));
}
#[test]
fn public_record_keeps_raw_record_unchanged() {
let filter = filter(vec![rule(
FileMatchField::FileName,
FileMatchKind::Prefix,
"_____padding_file_",
)]);
let mut record = test_record(1, 1);
record.files.push(TorrentFile {
path: "_____padding_file_1_".into(),
size: 100,
});
record.total_size += 100;
filter.apply_derivatives(&mut record).unwrap();
let public = filter.public_record(&record).unwrap();
assert_eq!(record.files.len(), 2);
assert_eq!(record.total_size, 142);
assert_eq!(public.files.len(), 1);
assert_eq!(public.total_size, 42);
}
#[test]
fn record_with_only_hidden_files_is_not_searchable() {
let filter = filter(vec![rule(
FileMatchField::FileName,
FileMatchKind::Prefix,
"_____padding_file_",
)]);
let mut record = test_record(1, 1);
record.files[0].path = "_____padding_file_1_".into();
let outcome = filter.apply_derivatives(&mut record).unwrap();
assert!(!outcome.searchable);
assert!(filter.public_record(&record).is_none());
}
}
+195
View File
@@ -0,0 +1,195 @@
// 负责把相同内容的多个 infohash 聚合为稳定且可排序的搜索内容组
#[cfg(any(feature = "rocksdb-storage", test))]
use std::cmp::Ordering;
use super::{Availability, Heat, TorrentRecord};
#[cfg(any(feature = "rocksdb-storage", test))]
use super::{AvailabilityStatus, InfoHash};
#[derive(Debug, Clone, PartialEq)]
pub struct ContentGroup {
pub content_key: [u8; 32],
pub representative: TorrentRecord,
pub aliases: Vec<String>,
pub variant_count: u64,
pub first_seen: u64,
pub last_seen: u64,
pub seen_count: u64,
pub heat: Heat,
pub availability: Availability,
}
#[cfg(any(feature = "rocksdb-storage", test))]
pub(crate) struct ContentGroupBuilder {
content_key: [u8; 32],
now: u64,
representative: Option<TorrentRecord>,
aliases: Vec<(RepresentativeRank, String)>,
variant_count: u64,
first_seen: u64,
last_seen: u64,
seen_count: u64,
heat: Heat,
availability: Availability,
availability_initialized: bool,
}
#[cfg(any(feature = "rocksdb-storage", test))]
impl ContentGroupBuilder {
pub(crate) fn new(content_key: [u8; 32], now: u64) -> Self {
Self {
content_key,
now,
representative: None,
aliases: Vec::new(),
variant_count: 0,
first_seen: u64::MAX,
last_seen: 0,
seen_count: 0,
heat: Heat::from_score(0),
availability: Availability::default(),
availability_initialized: false,
}
}
pub(crate) fn push(&mut self, record: TorrentRecord) {
if record.content_key != self.content_key {
return;
}
self.variant_count = self.variant_count.saturating_add(1);
self.first_seen = self.first_seen.min(record.first_seen);
self.last_seen = self.last_seen.max(record.last_seen);
self.seen_count = self.seen_count.saturating_add(record.seen_count);
let heat = record.heat(self.now);
let rank = representative_rank(&record, heat);
if heat.score > self.heat.score {
self.heat = heat;
}
if self.availability_initialized {
merge_availability(&mut self.availability, &record.availability);
} else {
self.availability = record.availability.clone();
self.availability_initialized = true;
}
if let Some((existing_rank, _)) = self
.aliases
.iter_mut()
.find(|(_, name)| name == &record.name)
{
*existing_rank = (*existing_rank).max(rank);
} else {
self.aliases.push((rank, record.name.clone()));
}
self.aliases
.sort_unstable_by_key(|item| std::cmp::Reverse(item.0));
self.aliases.truncate(32);
let replace = self.representative.as_ref().is_none_or(|current| {
rank.cmp(&representative_rank(current, current.heat(self.now))) == Ordering::Greater
});
if replace {
self.representative = Some(record);
}
}
pub(crate) fn finish(mut self) -> Option<ContentGroup> {
let representative = self.representative?;
if !self
.aliases
.iter()
.any(|(_, name)| name == &representative.name)
{
if self.aliases.len() == 32 {
self.aliases.pop();
}
self.aliases.push((
representative_rank(&representative, representative.heat(self.now)),
representative.name.clone(),
));
}
Some(ContentGroup {
content_key: self.content_key,
representative,
aliases: self.aliases.into_iter().map(|(_, name)| name).collect(),
variant_count: self.variant_count,
first_seen: self.first_seen,
last_seen: self.last_seen,
seen_count: self.seen_count,
heat: self.heat,
availability: self.availability,
})
}
}
#[cfg(any(feature = "rocksdb-storage", test))]
type RepresentativeRank = (u8, u8, u32, u64, u64, std::cmp::Reverse<InfoHash>);
#[cfg(any(feature = "rocksdb-storage", test))]
fn representative_rank(record: &TorrentRecord, heat: Heat) -> RepresentativeRank {
let availability = match record.availability.status {
AvailabilityStatus::Active => 2,
AvailabilityStatus::Unknown => 1,
AvailabilityStatus::PossiblyStale => 0,
};
(
availability,
heat.score,
record.availability.reachable_peers,
record.last_seen,
record.seen_count,
std::cmp::Reverse(record.info_hash),
)
}
#[cfg(any(feature = "rocksdb-storage", test))]
fn merge_availability(target: &mut Availability, candidate: &Availability) {
if availability_rank(candidate.status) > availability_rank(target.status) {
target.status = candidate.status;
}
target.last_verified_at = target.last_verified_at.max(candidate.last_verified_at);
target.last_success_at = target.last_success_at.max(candidate.last_success_at);
target.discovered_peers = target.discovered_peers.max(candidate.discovered_peers);
target.reachable_peers = target.reachable_peers.max(candidate.reachable_peers);
target.consecutive_failures = target
.consecutive_failures
.min(candidate.consecutive_failures);
target.next_check_at = target.next_check_at.max(candidate.next_check_at);
}
#[cfg(any(feature = "rocksdb-storage", test))]
fn availability_rank(status: AvailabilityStatus) -> u8 {
match status {
AvailabilityStatus::Active => 2,
AvailabilityStatus::Unknown => 1,
AvailabilityStatus::PossiblyStale => 0,
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::domain::test_record;
#[test]
fn uses_best_variant_and_aggregates_activity() {
let mut stale = test_record(1, 10);
stale.name = "旧名称".into();
stale.availability.status = AvailabilityStatus::PossiblyStale;
let mut active = test_record(2, 20);
active.name = "流浪地球 S01E03".into();
active.availability.status = AvailabilityStatus::Active;
active.availability.reachable_peers = 2;
active.seen_count = 3;
let mut builder = ContentGroupBuilder::new(stale.content_key, 20);
builder.push(stale);
builder.push(active.clone());
let group = builder.finish().unwrap();
assert_eq!(group.representative.info_hash, active.info_hash);
assert_eq!(group.variant_count, 2);
assert_eq!(group.first_seen, 10);
assert_eq!(group.last_seen, 20);
assert_eq!(group.seen_count, 4);
assert_eq!(group.availability.status, AvailabilityStatus::Active);
}
}
+107
View File
@@ -0,0 +1,107 @@
// 负责规范化文件结构并生成用于内容聚合的稳定 BLAKE3 指纹
use unicode_normalization::UnicodeNormalization;
use super::{TorrentFile, TorrentRecordError};
const FINGERPRINT_DOMAIN: &[u8] = b"dht-search-content\0";
pub fn content_key(files: &[TorrentFile]) -> Result<[u8; 32], TorrentRecordError> {
let mut normalized = normalize_files(files)?;
normalized.sort_unstable();
let mut hasher = blake3::Hasher::new();
hasher.update(FINGERPRINT_DOMAIN);
hasher.update(&(normalized.len() as u64).to_be_bytes());
for (path, size) in normalized {
hasher.update(&(path.len() as u64).to_be_bytes());
hasher.update(path.as_bytes());
hasher.update(&size.to_be_bytes());
}
Ok(*hasher.finalize().as_bytes())
}
fn normalize_files(files: &[TorrentFile]) -> Result<Vec<(String, u64)>, TorrentRecordError> {
let mut paths = Vec::with_capacity(files.len());
for file in files {
let parts: Vec<String> = file
.path
.replace('\\', "/")
.split('/')
.filter(|part| !part.is_empty() && *part != ".")
.map(|part| part.nfc().collect::<String>().to_lowercase())
.collect();
if parts.is_empty() {
return Err(TorrentRecordError::EmptyNormalizedPath);
}
paths.push((parts, file.size));
}
paths
.into_iter()
.map(|(parts, size)| {
let path = parts.join("/");
if path.is_empty() {
Err(TorrentRecordError::EmptyNormalizedPath)
} else {
Ok((path, size))
}
})
.collect()
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn equivalent_layouts_have_the_same_content_key() {
let left = vec![
TorrentFile {
path: "Dir\\A.txt".into(),
size: 1,
},
TorrentFile {
path: "Dir/B.bin".into(),
size: 2,
},
];
let right = vec![
TorrentFile {
path: "dir/b.bin".into(),
size: 2,
},
TorrentFile {
path: "dir/a.TXT".into(),
size: 1,
},
];
assert_eq!(content_key(&left).unwrap(), content_key(&right).unwrap());
}
#[test]
fn real_subdirectory_is_part_of_the_content_key() {
let left = vec![TorrentFile {
path: "season1/a.mkv".into(),
size: 1,
}];
let right = vec![TorrentFile {
path: "season2/a.mkv".into(),
size: 1,
}];
assert_ne!(content_key(&left).unwrap(), content_key(&right).unwrap());
}
#[test]
fn file_size_is_part_of_the_content_key() {
let left = vec![TorrentFile {
path: "a.txt".into(),
size: 1,
}];
let right = vec![TorrentFile {
path: "a.txt".into(),
size: 2,
}];
assert_ne!(content_key(&left).unwrap(), content_key(&right).unwrap());
}
}
+51
View File
@@ -0,0 +1,51 @@
// 负责定义二十字节 infohash 的解析显示和稳定二进制表示
use std::{fmt, str::FromStr};
use serde::{Deserialize, Serialize};
use super::TorrentRecordError;
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)]
pub struct InfoHash([u8; 20]);
impl InfoHash {
pub const BYTE_LEN: usize = 20;
pub fn from_bytes(bytes: [u8; Self::BYTE_LEN]) -> Self {
Self(bytes)
}
pub fn as_bytes(&self) -> &[u8; Self::BYTE_LEN] {
&self.0
}
}
impl FromStr for InfoHash {
type Err = TorrentRecordError;
fn from_str(value: &str) -> Result<Self, Self::Err> {
let decoded = hex::decode(value).map_err(|_| TorrentRecordError::InvalidInfoHash)?;
let bytes = decoded
.try_into()
.map_err(|_| TorrentRecordError::InvalidInfoHash)?;
Ok(Self(bytes))
}
}
impl fmt::Display for InfoHash {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter.write_str(&hex::encode(self.0))
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn round_trips_as_lowercase_hex() {
let hash = InfoHash::from_str("ABABABABABABABABABABABABABABABABABABABAB").unwrap();
assert_eq!(hash.to_string(), "abababababababababababababababababababab");
}
}
+394
View File
@@ -0,0 +1,394 @@
// 负责校验外部 Metadata 并生成领域记录和可持久化拒绝原因
use std::str::FromStr;
use serde::{Deserialize, Serialize};
use super::{
Availability, AvailabilityStatus, InfoHash, TorrentFile, TorrentRecord, content_key,
torrent::NewTorrentRecord,
};
const MAX_STORED_PEERS: usize = 32;
const METADATA_VALIDATION_VERSION: u64 = 1;
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct MetadataCandidate {
pub info_hash: String,
pub name: String,
pub total_size: u64,
pub files: Vec<TorrentFile>,
pub piece_length: u64,
pub source_peers: Vec<String>,
pub timestamp: u64,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct MetadataLimits {
pub max_files: usize,
pub max_name_bytes: usize,
pub max_path_bytes: usize,
pub max_path_depth: usize,
}
impl MetadataLimits {
pub fn rule_id(self) -> [u8; 32] {
let mut hasher = blake3::Hasher::new();
hasher.update(b"dht-search-metadata-limits\0");
hasher.update(&METADATA_VALIDATION_VERSION.to_be_bytes());
hasher.update(&(self.max_files as u64).to_be_bytes());
hasher.update(&(self.max_name_bytes as u64).to_be_bytes());
hasher.update(&(self.max_path_bytes as u64).to_be_bytes());
hasher.update(&(self.max_path_depth as u64).to_be_bytes());
*hasher.finalize().as_bytes()
}
}
impl Default for MetadataLimits {
fn default() -> Self {
Self {
max_files: 20_000,
max_name_bytes: 1_024,
max_path_bytes: 4_096,
max_path_depth: 64,
}
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum MetadataRejectionReason {
InvalidInfoHash,
EmptyName,
NameTooLong,
InvalidName,
EmptyFileList,
TooManyFiles,
EmptyPath,
PathTooLong,
PathTooDeep,
InvalidPath,
SizeOverflow,
TotalSizeMismatch,
}
impl MetadataRejectionReason {
pub const COUNT: usize = 12;
pub const fn index(self) -> usize {
match self {
Self::InvalidInfoHash => 0,
Self::EmptyName => 1,
Self::NameTooLong => 2,
Self::InvalidName => 3,
Self::EmptyFileList => 4,
Self::TooManyFiles => 5,
Self::EmptyPath => 6,
Self::PathTooLong => 7,
Self::PathTooDeep => 8,
Self::InvalidPath => 9,
Self::SizeOverflow => 10,
Self::TotalSizeMismatch => 11,
}
}
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct RejectedMetadata {
pub info_hash: InfoHash,
pub reason: MetadataRejectionReason,
pub rule_id: [u8; 32],
pub first_rejected_at: u64,
pub last_seen: u64,
pub seen_count: u64,
}
impl RejectedMetadata {
pub fn new(
info_hash: InfoHash,
reason: MetadataRejectionReason,
rule_id: [u8; 32],
timestamp: u64,
) -> Self {
Self {
info_hash,
reason,
rule_id,
first_rejected_at: timestamp,
last_seen: timestamp,
seen_count: 1,
}
}
pub fn observe_again(&mut self, timestamp: u64) {
self.last_seen = self.last_seen.max(timestamp);
self.seen_count = self.seen_count.saturating_add(1);
}
}
#[derive(Debug, thiserror::Error, PartialEq, Eq)]
pub enum TorrentRecordError {
#[error("infohash 必须是二十字节的十六进制字符串")]
InvalidInfoHash,
#[error("种子名称不能为空")]
EmptyName,
#[error("种子名称长度 {actual} 字节超过上限 {limit}")]
NameTooLong { actual: usize, limit: usize },
#[error("种子名称包含控制字符")]
InvalidName,
#[error("文件列表不能为空")]
EmptyFileList,
#[error("文件数量 {actual} 超过上限 {limit}")]
TooManyFiles { actual: usize, limit: usize },
#[error("文件路径长度 {actual} 字节超过上限 {limit}")]
PathTooLong { actual: usize, limit: usize },
#[error("文件路径目录层级 {actual} 超过上限 {limit}")]
PathTooDeep { actual: usize, limit: usize },
#[error("文件路径包含空段上级目录当前目录或控制字符")]
InvalidPath,
#[error("文件总大小溢出")]
SizeOverflow,
#[error("声明大小 {declared} 与文件计算大小 {calculated} 不一致")]
TotalSizeMismatch { declared: u64, calculated: u64 },
#[error("文件路径规范化后为空")]
EmptyNormalizedPath,
}
impl TorrentRecordError {
pub fn rejection_reason(&self) -> MetadataRejectionReason {
match self {
Self::InvalidInfoHash => MetadataRejectionReason::InvalidInfoHash,
Self::EmptyName => MetadataRejectionReason::EmptyName,
Self::NameTooLong { .. } => MetadataRejectionReason::NameTooLong,
Self::InvalidName => MetadataRejectionReason::InvalidName,
Self::EmptyFileList => MetadataRejectionReason::EmptyFileList,
Self::TooManyFiles { .. } => MetadataRejectionReason::TooManyFiles,
Self::EmptyNormalizedPath => MetadataRejectionReason::EmptyPath,
Self::PathTooLong { .. } => MetadataRejectionReason::PathTooLong,
Self::PathTooDeep { .. } => MetadataRejectionReason::PathTooDeep,
Self::InvalidPath => MetadataRejectionReason::InvalidPath,
Self::SizeOverflow => MetadataRejectionReason::SizeOverflow,
Self::TotalSizeMismatch { .. } => MetadataRejectionReason::TotalSizeMismatch,
}
}
}
pub(crate) fn into_record(
info: MetadataCandidate,
limits: MetadataLimits,
) -> Result<TorrentRecord, TorrentRecordError> {
let info_hash = InfoHash::from_str(&info.info_hash)?;
validate_name(&info.name, limits)?;
if info.files.is_empty() {
return Err(TorrentRecordError::EmptyFileList);
}
if info.files.len() > limits.max_files {
return Err(TorrentRecordError::TooManyFiles {
actual: info.files.len(),
limit: limits.max_files,
});
}
for file in &info.files {
validate_path(&file.path, limits)?;
}
let files = info.files;
let calculated_size = files.iter().try_fold(0_u64, |total, file| {
total
.checked_add(file.size)
.ok_or(TorrentRecordError::SizeOverflow)
})?;
if calculated_size != info.total_size {
return Err(TorrentRecordError::TotalSizeMismatch {
declared: info.total_size,
calculated: calculated_size,
});
}
let content_key = content_key(&files)?;
let mut source_peers = info.source_peers;
source_peers.sort_unstable();
source_peers.dedup();
source_peers.truncate(MAX_STORED_PEERS);
let reachable_peers = source_peers.len().min(u32::MAX as usize) as u32;
let availability = if reachable_peers > 0 {
Availability {
status: AvailabilityStatus::Active,
last_verified_at: Some(info.timestamp),
last_success_at: Some(info.timestamp),
discovered_peers: reachable_peers,
reachable_peers,
consecutive_failures: 0,
next_check_at: info.timestamp.saturating_add(86_400),
}
} else {
Availability::default()
};
Ok(TorrentRecord::from_new(NewTorrentRecord {
info_hash,
name: info.name,
total_size: info.total_size,
files,
piece_length: info.piece_length,
source_peers,
content_key,
timestamp: info.timestamp,
availability,
}))
}
fn validate_name(name: &str, limits: MetadataLimits) -> Result<(), TorrentRecordError> {
if name.trim().is_empty() {
return Err(TorrentRecordError::EmptyName);
}
if name.len() > limits.max_name_bytes {
return Err(TorrentRecordError::NameTooLong {
actual: name.len(),
limit: limits.max_name_bytes,
});
}
if name.chars().any(char::is_control) {
return Err(TorrentRecordError::InvalidName);
}
Ok(())
}
fn validate_path(path: &str, limits: MetadataLimits) -> Result<(), TorrentRecordError> {
if path.is_empty() {
return Err(TorrentRecordError::EmptyNormalizedPath);
}
if path.len() > limits.max_path_bytes {
return Err(TorrentRecordError::PathTooLong {
actual: path.len(),
limit: limits.max_path_bytes,
});
}
if path.chars().any(char::is_control) {
return Err(TorrentRecordError::InvalidPath);
}
let mut depth = 0_usize;
for component in path.split(['/', '\\']) {
if component.is_empty() || matches!(component, "." | "..") {
return Err(TorrentRecordError::InvalidPath);
}
depth += 1;
}
if depth > limits.max_path_depth {
return Err(TorrentRecordError::PathTooDeep {
actual: depth,
limit: limits.max_path_depth,
});
}
Ok(())
}
impl TryFrom<MetadataCandidate> for TorrentRecord {
type Error = TorrentRecordError;
fn try_from(info: MetadataCandidate) -> Result<Self, Self::Error> {
into_record(info, MetadataLimits::default())
}
}
#[cfg(test)]
mod tests {
use super::*;
fn metadata_candidate(files: Vec<TorrentFile>) -> MetadataCandidate {
MetadataCandidate {
info_hash: "0101010101010101010101010101010101010101".into(),
name: "Example".into(),
total_size: files
.iter()
.fold(0_u64, |total, file| total.saturating_add(file.size)),
files,
piece_length: 16_384,
source_peers: Vec::new(),
timestamp: 1,
}
}
#[test]
fn limits_accept_boundary_and_reject_excess() {
let limits = MetadataLimits {
max_files: 1,
max_name_bytes: 7,
max_path_bytes: 8,
max_path_depth: 2,
};
let accepted = metadata_candidate(vec![TorrentFile {
path: "dir/a.rs".into(),
size: 1,
}]);
assert!(into_record(accepted, limits).is_ok());
let too_many = metadata_candidate(vec![
TorrentFile {
path: "a".into(),
size: 1,
},
TorrentFile {
path: "b".into(),
size: 1,
},
]);
assert_eq!(
into_record(too_many, limits)
.unwrap_err()
.rejection_reason(),
MetadataRejectionReason::TooManyFiles
);
}
#[test]
fn unsafe_and_deep_paths_are_rejected_by_reason() {
let limits = MetadataLimits {
max_path_depth: 2,
..MetadataLimits::default()
};
for (path, reason) in [
("dir/../file", MetadataRejectionReason::InvalidPath),
("dir//file", MetadataRejectionReason::InvalidPath),
("a/b/c", MetadataRejectionReason::PathTooDeep),
("a\0b", MetadataRejectionReason::InvalidPath),
] {
let info = metadata_candidate(vec![TorrentFile {
path: path.into(),
size: 1,
}]);
assert_eq!(
into_record(info, limits).unwrap_err().rejection_reason(),
reason
);
}
}
#[test]
fn size_overflow_is_rejected_without_panicking() {
let mut info = metadata_candidate(vec![
TorrentFile {
path: "a".into(),
size: u64::MAX,
},
TorrentFile {
path: "b".into(),
size: 1,
},
]);
info.total_size = u64::MAX;
assert_eq!(
into_record(info, MetadataLimits::default())
.unwrap_err()
.rejection_reason(),
MetadataRejectionReason::SizeOverflow
);
}
#[test]
fn rule_id_changes_when_a_limit_changes() {
let defaults = MetadataLimits::default();
let changed = MetadataLimits {
max_files: defaults.max_files - 1,
..defaults
};
assert_ne!(defaults.rule_id(), changed.rule_id());
}
}
+28
View File
@@ -0,0 +1,28 @@
// 负责导出不依赖存储搜索和传输实现的核心领域模型
mod content_filter;
mod content_group;
mod fingerprint;
mod info_hash;
mod metadata;
mod torrent;
pub use content_filter::{
ContentFilter, ContentFilterConfig, ContentFilterError, FileFilterRule, FileMatchField,
FileMatchKind, FileRuleAction, FilterOutcome,
};
pub use content_group::ContentGroup;
#[cfg(any(feature = "rocksdb-storage", test))]
pub(crate) use content_group::ContentGroupBuilder;
pub use fingerprint::content_key;
pub use info_hash::InfoHash;
pub use metadata::{
MetadataCandidate, MetadataLimits, MetadataRejectionReason, RejectedMetadata,
TorrentRecordError,
};
#[cfg(test)]
pub(crate) use torrent::test_record;
pub use torrent::{
Availability, AvailabilityStatus, Heat, HeatLevel, TorrentFile, TorrentRecord,
VerificationResult,
};
+314
View File
@@ -0,0 +1,314 @@
// 负责定义种子记录活跃度可用性和重复发现时的状态演进
use serde::{Deserialize, Serialize};
#[cfg(test)]
use super::content_key;
use super::{InfoHash, MetadataCandidate, MetadataLimits, TorrentRecordError};
const MAX_STORED_PEERS: usize = 32;
const ACTIVITY_SCALE: u64 = 1_000;
const ACTIVITY_HALF_LIFE_SECS: f64 = 86_400.0;
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct TorrentFile {
pub path: String,
pub size: u64,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)]
#[serde(rename_all = "snake_case")]
pub enum AvailabilityStatus {
#[default]
Unknown,
Active,
PossiblyStale,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, Default)]
pub struct Availability {
pub status: AvailabilityStatus,
pub last_verified_at: Option<u64>,
pub last_success_at: Option<u64>,
pub discovered_peers: u32,
pub reachable_peers: u32,
pub consecutive_failures: u32,
pub next_check_at: u64,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub struct VerificationResult {
pub verified_at: u64,
pub discovered_peers: u32,
pub reachable_peers: u32,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum HeatLevel {
Hot,
Active,
Normal,
Cold,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)]
pub struct Heat {
pub score: u8,
pub level: HeatLevel,
}
impl Heat {
pub fn from_score(score: u8) -> Self {
let level = match score {
75..=100 => HeatLevel::Hot,
50..=74 => HeatLevel::Active,
25..=49 => HeatLevel::Normal,
_ => HeatLevel::Cold,
};
Self { score, level }
}
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct TorrentRecord {
pub info_hash: InfoHash,
pub name: String,
pub total_size: u64,
pub files: Vec<TorrentFile>,
pub piece_length: u64,
pub source_peers: Vec<String>,
pub content_key: [u8; 32],
#[serde(default = "default_searchable")]
pub searchable: bool,
pub first_seen: u64,
pub last_seen: u64,
pub seen_count: u64,
#[serde(default)]
pub availability: Availability,
#[serde(default = "default_activity_score")]
pub activity_score_millis: u64,
#[serde(default)]
pub activity_updated_at: u64,
}
pub(crate) struct NewTorrentRecord {
pub(crate) info_hash: InfoHash,
pub(crate) name: String,
pub(crate) total_size: u64,
pub(crate) files: Vec<TorrentFile>,
pub(crate) piece_length: u64,
pub(crate) source_peers: Vec<String>,
pub(crate) content_key: [u8; 32],
pub(crate) timestamp: u64,
pub(crate) availability: Availability,
}
impl TorrentRecord {
pub(crate) fn from_new(new: NewTorrentRecord) -> Self {
Self {
info_hash: new.info_hash,
name: new.name,
total_size: new.total_size,
files: new.files,
piece_length: new.piece_length,
source_peers: new.source_peers,
content_key: new.content_key,
searchable: true,
first_seen: new.timestamp,
last_seen: new.timestamp,
seen_count: 1,
availability: new.availability,
activity_score_millis: ACTIVITY_SCALE,
activity_updated_at: new.timestamp,
}
}
pub fn try_from_with_limits(
info: MetadataCandidate,
limits: MetadataLimits,
) -> Result<Self, TorrentRecordError> {
super::metadata::into_record(info, limits)
}
pub fn observe_again(&mut self, timestamp: u64, peers: &[String]) {
self.last_seen = self.last_seen.max(timestamp);
self.seen_count = self.seen_count.saturating_add(1);
self.update_activity(timestamp);
for peer in peers {
if self.source_peers.len() >= MAX_STORED_PEERS {
break;
}
if !self.source_peers.contains(peer) {
self.source_peers.push(peer.clone());
}
}
}
pub fn apply_verification(&mut self, result: VerificationResult) {
self.availability.last_verified_at = Some(result.verified_at);
self.availability.discovered_peers = result.discovered_peers;
self.availability.reachable_peers = result.reachable_peers;
if result.reachable_peers > 0 {
self.availability.status = AvailabilityStatus::Active;
self.availability.last_success_at = Some(result.verified_at);
self.availability.consecutive_failures = 0;
self.availability.next_check_at = result.verified_at.saturating_add(86_400);
} else {
self.availability.status = AvailabilityStatus::PossiblyStale;
self.availability.consecutive_failures =
self.availability.consecutive_failures.saturating_add(1);
self.availability.next_check_at = result
.verified_at
.saturating_add(failure_retry_secs(self.availability.consecutive_failures));
}
}
pub fn heat(&self, now: u64) -> Heat {
let activity = decayed_activity(self.activity_score_millis, self.activity_updated_at, now)
as f64
/ ACTIVITY_SCALE as f64;
let discovery = (activity.ln_1p() / 101_f64.ln()).clamp(0.0, 1.0);
let age = now.saturating_sub(self.last_seen) as f64;
let freshness = 2_f64.powf(-age / (7.0 * 86_400.0));
let availability = match self.availability.status {
AvailabilityStatus::Unknown => 0.25,
AvailabilityStatus::PossiblyStale => 0.0,
AvailabilityStatus::Active => {
((f64::from(self.availability.reachable_peers) + 1.0).ln() / 4_f64.ln())
.clamp(0.0, 1.0)
}
};
let score = (100.0 * (0.60 * discovery + 0.25 * freshness + 0.15 * availability))
.round()
.clamp(0.0, 100.0) as u8;
Heat::from_score(score)
}
fn update_activity(&mut self, timestamp: u64) {
self.activity_score_millis = decayed_activity(
self.activity_score_millis,
self.activity_updated_at,
timestamp,
)
.saturating_add(ACTIVITY_SCALE);
self.activity_updated_at = self.activity_updated_at.max(timestamp);
}
}
const fn default_searchable() -> bool {
true
}
fn default_activity_score() -> u64 {
ACTIVITY_SCALE
}
fn decayed_activity(value: u64, updated_at: u64, now: u64) -> u64 {
if updated_at == 0 || now <= updated_at {
return value;
}
let elapsed = now - updated_at;
(value as f64 * 2_f64.powf(-(elapsed as f64) / ACTIVITY_HALF_LIFE_SECS)).round() as u64
}
fn failure_retry_secs(failures: u32) -> u64 {
match failures {
0 | 1 => 3_600,
2 => 6 * 3_600,
3 => 24 * 3_600,
4 => 72 * 3_600,
_ => 7 * 24 * 3_600,
}
}
#[cfg(test)]
pub(crate) fn test_record(hash_byte: u8, timestamp: u64) -> TorrentRecord {
let files = vec![TorrentFile {
path: "Example/file.txt".to_owned(),
size: 42,
}];
TorrentRecord {
info_hash: InfoHash::from_bytes([hash_byte; 20]),
name: "Example".to_owned(),
total_size: 42,
content_key: content_key(&files).expect("test file path is valid"),
searchable: true,
files,
piece_length: 16_384,
source_peers: vec!["127.0.0.1:6881".to_owned()],
first_seen: timestamp,
last_seen: timestamp,
seen_count: 1,
availability: Availability::default(),
activity_score_millis: ACTIVITY_SCALE,
activity_updated_at: timestamp,
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn repeated_observation_updates_time_count_and_unique_peers() {
let mut record = test_record(1, 10);
record.observe_again(20, &["127.0.0.1:6881".into(), "127.0.0.2:6881".into()]);
assert_eq!(record.first_seen, 10);
assert_eq!(record.last_seen, 20);
assert_eq!(record.seen_count, 2);
assert_eq!(record.source_peers.len(), 2);
}
#[test]
fn activity_decays_and_recent_observation_increases_heat() {
let mut record = test_record(1, 10);
let now = 10 + 30 * 86_400;
let old_heat = record.heat(now).score;
record.observe_again(now, &[]);
assert!(record.heat(now).score > old_heat);
}
#[test]
fn verification_success_and_failures_update_status_and_backoff() {
let mut record = test_record(1, 10);
record.apply_verification(VerificationResult {
verified_at: 100,
discovered_peers: 3,
reachable_peers: 2,
});
assert_eq!(record.availability.status, AvailabilityStatus::Active);
assert_eq!(record.availability.next_check_at, 86_500);
record.apply_verification(VerificationResult {
verified_at: 200,
discovered_peers: 0,
reachable_peers: 0,
});
assert_eq!(
record.availability.status,
AvailabilityStatus::PossiblyStale
);
assert_eq!(record.availability.next_check_at, 3_800);
}
#[test]
fn freshly_downloaded_metadata_is_immediately_active() {
let record = TorrentRecord::try_from(MetadataCandidate {
info_hash: "0101010101010101010101010101010101010101".into(),
name: "Example".into(),
total_size: 42,
files: vec![TorrentFile {
path: "example.bin".into(),
size: 42,
}],
piece_length: 16_384,
source_peers: vec!["127.0.0.1:6881".into()],
timestamp: 100,
})
.unwrap();
assert_eq!(record.availability.status, AvailabilityStatus::Active);
assert_eq!(record.availability.reachable_peers, 1);
assert_eq!(record.availability.last_verified_at, Some(100));
assert_eq!(record.availability.next_check_at, 86_500);
}
}
+52
View File
@@ -0,0 +1,52 @@
// 负责加载启动配置初始化日志并选择正常运行或离线恢复模式
use clap::Parser;
use crate::{backup, config, error, telemetry};
pub async fn run_cli() -> std::process::ExitCode {
let startup = match config::Cli::parse().load() {
Ok(config) => config,
Err(error) => {
eprintln!("无法加载配置: {error}");
return std::process::ExitCode::FAILURE;
}
};
let restore_requested = startup.restore_checkpoint.is_some();
let mut logging = startup.app.logging.clone();
if restore_requested {
logging.file_enabled = false;
logging.console_enabled = true;
}
let _telemetry = match telemetry::init(&logging) {
Ok(guard) => guard,
Err(error) => {
eprintln!("无法初始化日志: {error}");
return std::process::ExitCode::FAILURE;
}
};
let result = if let Some(checkpoint) = startup.restore_checkpoint {
backup::restore(&startup.app.data_dir, &checkpoint)
.map(|outcome| {
tracing::info!(
records = outcome.records,
previous_database = ?outcome.previous_database,
"RocksDB 检查点恢复完成 下次正常启动将重建搜索索引"
);
println!("恢复完成: {} 条记录", outcome.records);
if let Some(previous) = outcome.previous_database {
println!("原数据库保留在: {}", previous.display());
}
})
.map_err(error::AppError::from)
} else {
crate::app::run(startup.app, startup.config_service).await
};
match result {
Ok(()) => std::process::ExitCode::SUCCESS,
Err(error) => {
tracing::error!(%error, "dht-search 退出");
std::process::ExitCode::FAILURE
}
}
}
+33
View File
@@ -0,0 +1,33 @@
// 负责定义应用层统一错误类型和跨模块错误转换边界
#[derive(Debug, thiserror::Error)]
pub(crate) enum AppError {
#[error("I/O 操作失败: {0}")]
Io(#[from] std::io::Error),
#[error("配置解析失败: {0}")]
Toml(#[from] toml::de::Error),
#[error("配置序列化失败: {0}")]
TomlSerialize(#[from] toml::ser::Error),
#[error("内容过滤配置无效: {0}")]
ContentFilter(#[from] crate::domain::ContentFilterError),
#[error("备份或恢复失败: {0}")]
Backup(#[from] crate::backup::BackupError),
#[error("配置无效: {0}")]
Config(String),
#[error("DHT 服务失败: {0}")]
Dht(#[from] dht_crawler::DHTError),
#[error("存储失败: {0}")]
Storage(#[from] crate::storage::StorageError),
#[error("搜索失败: {0}")]
Search(#[from] crate::search::SearchError),
#[error("持久化 worker 异常退出")]
PersistenceWorkerPanicked,
#[error("持久化 worker 失败: {0}")]
PersistenceWorker(String),
#[error("索引 worker 失败: {0}")]
IndexWorker(String),
#[error("可用性验证 worker 失败: {0}")]
VerificationWorker(String),
#[error("运行诊断失败: {0}")]
Diagnostics(String),
}
+151
View File
@@ -0,0 +1,151 @@
// 负责批量提交搜索索引并处理临时错误和关闭前排空
use std::{sync::Arc, time::Duration};
use crate::{
search::SearchEngine,
storage::{RocksTorrentRepository, TorrentRepository},
};
use tokio_util::sync::CancellationToken;
use crate::disk_guard::DiskGuard;
pub(crate) struct IndexWorkerOptions {
pub(crate) batch_size: usize,
pub(crate) interval: Duration,
pub(crate) prepare_full_reindex: bool,
}
pub(crate) async fn run(
repository: Arc<RocksTorrentRepository>,
search: SearchEngine,
options: IndexWorkerOptions,
disk_guard: DiskGuard,
cancel: CancellationToken,
fatal: tokio::sync::oneshot::Sender<String>,
) -> Result<(), String> {
let mut prepare_full_reindex = options.prepare_full_reindex;
let mut ticker = tokio::time::interval(options.interval);
ticker.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
let mut consecutive_retries = 0_u32;
loop {
tokio::select! {
_ = cancel.cancelled() => break,
_ = ticker.tick() => {
let Some(_permit) = disk_guard.begin_new_write() else {
continue;
};
if prepare_full_reindex {
let rebuild_repository = repository.clone();
let records = tokio::task::spawn_blocking(move || {
rebuild_repository.prepare_full_reindex()
})
.await
.map_err(|error| error.to_string())?
.map_err(|error| error.to_string())?;
tracing::info!(records, "检测到新搜索索引并准备全量重建");
prepare_full_reindex = false;
}
match index_one_batch(repository.clone(), search.clone(), options.batch_size).await {
Ok(count) => {
consecutive_retries = 0;
if count > 0 {
tracing::debug!(count, "搜索索引已提交");
}
}
Err(IndexBatchError::Retryable(error)) => {
consecutive_retries = consecutive_retries.saturating_add(1);
let delay = retry_delay(consecutive_retries);
tracing::warn!(%error, retry = consecutive_retries, delay_ms = delay.as_millis(), "搜索索引遇到临时 I/O 错误");
tokio::select! {
_ = cancel.cancelled() => break,
_ = tokio::time::sleep(delay) => {}
}
}
Err(IndexBatchError::Fatal(error)) => {
let _ = fatal.send(error.clone());
return Err(error);
}
}
}
}
}
drain_before_shutdown(repository, search, options.batch_size, disk_guard).await
}
async fn drain_before_shutdown(
repository: Arc<RocksTorrentRepository>,
search: SearchEngine,
batch_size: usize,
disk_guard: DiskGuard,
) -> Result<(), String> {
let Some(_permit) = disk_guard.begin_new_write() else {
return Ok(());
};
let mut retries = 0_u32;
loop {
match index_one_batch(repository.clone(), search.clone(), batch_size).await {
Ok(0) => return Ok(()),
Ok(_) => retries = 0,
Err(IndexBatchError::Retryable(error)) if retries < 3 => {
retries += 1;
let delay = retry_delay(retries);
tracing::warn!(%error, retry = retries, delay_ms = delay.as_millis(), "关闭前提交搜索索引时遇到临时 I/O 错误");
tokio::time::sleep(delay).await;
}
Err(IndexBatchError::Retryable(error)) => {
tracing::warn!(%error, "关闭前搜索索引仍被占用 待索引状态将在下次启动恢复");
return Ok(());
}
Err(IndexBatchError::Fatal(error)) => return Err(error),
}
}
}
enum IndexBatchError {
Retryable(String),
Fatal(String),
}
async fn index_one_batch(
repository: Arc<RocksTorrentRepository>,
search: SearchEngine,
batch_size: usize,
) -> Result<usize, IndexBatchError> {
tokio::task::spawn_blocking(move || {
match search.index_pending(repository.as_ref(), batch_size, unix_timestamp()) {
Ok(count) => Ok(count),
Err(error) if error.is_retryable_io() => {
Err(IndexBatchError::Retryable(error.to_string()))
}
Err(error) => Err(IndexBatchError::Fatal(error.to_string())),
}
})
.await
.map_err(|error| IndexBatchError::Fatal(error.to_string()))?
}
fn retry_delay(attempt: u32) -> Duration {
let shift = attempt.saturating_sub(1).min(6);
Duration::from_millis((250_u64 << shift).min(10_000))
}
fn unix_timestamp() -> u64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap_or_default()
.as_secs()
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn retry_delay_is_exponential_and_bounded() {
assert_eq!(retry_delay(1), Duration::from_millis(250));
assert_eq!(retry_delay(3), Duration::from_secs(1));
assert_eq!(retry_delay(100), Duration::from_secs(10));
}
}
+36
View File
@@ -0,0 +1,36 @@
// 负责导出可测试核心能力并组合应用运行入口
#[cfg(feature = "rocksdb-storage")]
mod api;
#[cfg(feature = "rocksdb-storage")]
mod app;
#[cfg(feature = "rocksdb-storage")]
mod backup;
#[cfg(feature = "rocksdb-storage")]
mod config;
#[cfg(feature = "rocksdb-storage")]
mod crawler;
#[cfg(feature = "rocksdb-storage")]
mod diagnostics;
#[cfg(feature = "rocksdb-storage")]
mod disk_guard;
pub mod domain;
#[cfg(feature = "rocksdb-storage")]
mod entry;
#[cfg(feature = "rocksdb-storage")]
mod error;
#[cfg(feature = "rocksdb-storage")]
mod index_worker;
#[cfg(feature = "rocksdb-storage")]
mod monitor;
pub mod search;
#[cfg(feature = "rocksdb-storage")]
mod shutdown;
pub mod storage;
#[cfg(feature = "rocksdb-storage")]
mod telemetry;
#[cfg(feature = "rocksdb-storage")]
mod verification;
#[cfg(feature = "rocksdb-storage")]
pub use entry::run_cli;
+6
View File
@@ -0,0 +1,6 @@
// 负责进入异步运行时并把退出状态返回给操作系统
#[tokio::main]
async fn main() -> std::process::ExitCode {
dht_search::run_cli().await
}
+93
View File
@@ -0,0 +1,93 @@
// 负责定期汇总采集持久化和过滤指标并输出结构化运行状态
use std::time::Duration;
use crate::domain::MetadataRejectionReason;
use dht_crawler::DHTServer;
use tokio_util::sync::CancellationToken;
use crate::{crawler::pipeline::PersistenceIngress, disk_guard::DiskGuard};
pub(crate) async fn run(
server: DHTServer,
ingress: PersistenceIngress,
disk_guard: DiskGuard,
interval_secs: u64,
cancel: CancellationToken,
) {
let mut interval = tokio::time::interval(Duration::from_secs(interval_secs));
interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
let mut previous_udp_tx = 0;
let mut previous_metadata_attempts = 0;
loop {
tokio::select! {
_ = cancel.cancelled() => break,
_ = interval.tick() => {
let dht = server.runtime_stats().snapshot();
let observability = server.runtime_stats().observability_snapshot();
let storage = ingress.snapshot();
let disk = disk_guard.snapshot();
let udp_tx_per_second = observability
.udp_tx_packets
.saturating_sub(previous_udp_tx)
/ interval_secs;
let metadata_connects_per_second = dht
.metadata_peer_attempts
.saturating_sub(previous_metadata_attempts)
/ interval_secs;
previous_udp_tx = observability.udp_tx_packets;
previous_metadata_attempts = dht.metadata_peer_attempts;
tracing::info!(
nodes = dht.node_pool_size,
udp_tx = observability.udp_tx_packets,
udp_tx_per_second,
find_node_queries = dht.queries_new + dht.queries_revisit + dht.queries_bootstrap,
peer_lookup_queries = dht.peer_lookup_queries,
peer_lookup_preferred_succeeded = dht.peer_lookup_preferred_succeeded,
peer_lookup_fallbacks = dht.peer_lookup_fallbacks,
sample_queries = dht.sample_infohashes_queries,
sample_direct_queries = dht.sample_infohashes_direct_queries,
sample_direct_responses = dht.sample_infohashes_direct_responses,
sample_candidate_queue = dht.sample_candidate_queue_depth,
sample_candidates_routed = dht.sample_candidates_routed,
sample_candidates_fallback = dht.sample_candidates_fallback,
sampled_hashes = dht.sample_infohashes_hashes_discovered,
sampled_hashes_filtered = dht.sample_infohashes_hashes_filtered,
sampled_hashes_duplicate = dht.sample_infohashes_hashes_duplicate,
sampled_hashes_dropped = dht.sample_infohashes_hashes_dropped,
peers = dht.peer_lookup_peers_found,
metadata_connects_per_second,
metadata_in_flight = dht.metadata_in_flight,
metadata_ok = dht.metadata_peer_succeeded,
metadata_ok_from_announce = dht.metadata_success_from_announce,
metadata_ok_from_sample_direct = dht.metadata_success_from_sample_direct,
metadata_ok_from_sample_snapshot = dht.metadata_success_from_sample_snapshot,
metadata_ok_from_active_lookup = dht.metadata_success_from_active_lookup,
metadata_failed = dht.metadata_peer_failed,
metadata_filtered = observability
.metadata_failure_size_limit
.saturating_add(storage.filtered.total()),
metadata_filtered_too_many_files = storage
.filtered
.count(MetadataRejectionReason::TooManyFiles),
metadata_filtered_invalid_path = storage
.filtered
.count(MetadataRejectionReason::InvalidPath),
persistence_accepted = storage.accepted,
persistence_inserted = storage.inserted,
persistence_updated = storage.updated,
persistence_rejected_full = storage.rejected_full,
persistence_invalid = storage.invalid,
persistence_failed = storage.failed,
persistence_queue = storage.queue_depth,
disk_state = disk.mode.as_str(),
disk_available_bytes = disk.available_bytes,
disk_active_writes = disk.active_writes,
disk_probe_failed = disk.probe_failed,
disk_rejected_new_work = disk.rejected_new_work,
"运行状态"
)
}
}
}
}
+158
View File
@@ -0,0 +1,158 @@
// 负责在领域内容组和 Tantivy 文档之间执行双向字段映射
use std::{collections::BTreeSet, path::Path};
use tantivy::{TantivyDocument, schema::Value};
use unicode_normalization::UnicodeNormalization;
use crate::domain::{AvailabilityStatus, ContentGroup, Heat, TorrentRecord};
use super::{
SearchError,
query::{AvailabilitySummary, SearchHit},
schema::SearchFields,
};
pub(crate) fn from_group(group: &ContentGroup, fields: SearchFields) -> TantivyDocument {
const MAX_INDEXED_FILES: usize = 512;
const MAX_PATH_TEXT_BYTES: usize = 32 * 1024;
let record = &group.representative;
let mut document = TantivyDocument::default();
document.add_text(fields.info_hash, record.info_hash.to_string());
document.add_text(fields.name, normalize_bounded(&record.name, 512));
document.add_text(fields.regex_text, normalize_bounded(&record.name, 512));
document.add_text(fields.display_name, &record.name);
for alias in &group.aliases {
let alias = normalize_bounded(alias, 512);
document.add_text(fields.aliases, &alias);
document.add_text(fields.regex_text, alias);
}
let mut indexed_path_bytes = 0_usize;
for file in record.files.iter().take(MAX_INDEXED_FILES) {
let path = normalize_bounded(&file.path, 512);
if indexed_path_bytes.saturating_add(path.len()) > MAX_PATH_TEXT_BYTES {
break;
}
indexed_path_bytes += path.len();
document.add_text(fields.files_text, &path);
document.add_text(fields.regex_text, path);
}
for extension in extensions(record) {
document.add_text(fields.extensions, extension);
}
document.add_u64(fields.total_size, record.total_size);
document.add_u64(fields.file_count, record.files.len() as u64);
document.add_u64(fields.first_seen, group.first_seen);
document.add_u64(fields.last_seen, group.last_seen);
document.add_u64(fields.seen_count, group.seen_count);
document.add_text(fields.content_key, hex::encode(group.content_key));
document.add_u64(
fields.availability_status,
availability_number(group.availability.status),
);
document.add_u64(
fields.reachable_peers,
u64::from(group.availability.reachable_peers),
);
document.add_u64(
fields.last_verified_at,
group.availability.last_verified_at.unwrap_or(0),
);
document.add_u64(fields.heat_score, u64::from(group.heat.score));
document.add_u64(fields.variant_count, group.variant_count);
document
}
pub(crate) fn to_hit(
document: &TantivyDocument,
fields: SearchFields,
score: f32,
) -> Result<SearchHit, SearchError> {
Ok(SearchHit {
info_hash: text(document, fields.info_hash, "info_hash")?,
name: text(document, fields.display_name, "display_name")?,
total_size: number(document, fields.total_size, "total_size")?,
file_count: number(document, fields.file_count, "file_count")?,
first_seen: number(document, fields.first_seen, "first_seen")?,
last_seen: number(document, fields.last_seen, "last_seen")?,
seen_count: number(document, fields.seen_count, "seen_count")?,
content_key: text(document, fields.content_key, "content_key")?,
variant_count: number(document, fields.variant_count, "variant_count")?,
score,
heat: Heat::from_score(number(document, fields.heat_score, "heat_score")?.min(100) as u8),
availability: AvailabilitySummary {
status: availability_status(number(
document,
fields.availability_status,
"availability_status",
)?),
last_verified_at: match number(document, fields.last_verified_at, "last_verified_at")? {
0 => None,
value => Some(value),
},
reachable_peers: number(document, fields.reachable_peers, "reachable_peers")?
.min(u64::from(u32::MAX)) as u32,
},
})
}
pub(crate) fn availability_number(status: AvailabilityStatus) -> u64 {
match status {
AvailabilityStatus::Unknown => 0,
AvailabilityStatus::Active => 1,
AvailabilityStatus::PossiblyStale => 2,
}
}
fn availability_status(value: u64) -> AvailabilityStatus {
match value {
1 => AvailabilityStatus::Active,
2 => AvailabilityStatus::PossiblyStale,
_ => AvailabilityStatus::Unknown,
}
}
fn normalize_bounded(value: &str, max_chars: usize) -> String {
value
.chars()
.take(max_chars)
.collect::<String>()
.nfkc()
.collect::<String>()
.to_lowercase()
}
fn extensions(record: &TorrentRecord) -> BTreeSet<String> {
record
.files
.iter()
.take(512)
.filter_map(|file| Path::new(&file.path).extension())
.filter_map(|extension| extension.to_str())
.map(str::to_lowercase)
.collect()
}
fn text(
document: &TantivyDocument,
field: tantivy::schema::Field,
name: &'static str,
) -> Result<String, SearchError> {
document
.get_first(field)
.and_then(|value| value.as_str())
.map(str::to_owned)
.ok_or(SearchError::MissingField(name))
}
fn number(
document: &TantivyDocument,
field: tantivy::schema::Field,
name: &'static str,
) -> Result<u64, SearchError> {
document
.get_first(field)
.and_then(|value| value.as_u64())
.ok_or(SearchError::MissingField(name))
}
+285
View File
@@ -0,0 +1,285 @@
// 负责将搜索选项组合为 Tantivy 查询条件但不执行查询
use std::ops::Bound;
use tantivy::{
Term,
query::{AllQuery, BooleanQuery, BoostQuery, Occur, Query, RangeQuery, RegexQuery, TermQuery},
schema::IndexRecordOption,
};
use unicode_normalization::UnicodeNormalization;
use super::{
SearchError,
document::availability_number,
query::{SearchMode, SearchOptions, SearchSort},
schema::SearchFields,
};
pub(crate) struct PreparedQuery {
pub(crate) query: Box<dyn Query>,
pub(crate) sort: SearchSort,
}
pub(crate) fn prepare(
options: &SearchOptions,
fields: SearchFields,
) -> Result<PreparedQuery, SearchError> {
let mut clauses: Vec<Box<dyn Query>> = Vec::new();
let query_text = options.query.trim();
if query_text.is_empty() || query_text == "*" {
clauses.push(Box::new(AllQuery));
} else {
clauses.push(match options.mode {
SearchMode::Text if has_wildcard_syntax(query_text) => {
wildcard_query(query_text, fields)?
}
SearchMode::Text => text_query(query_text, fields),
SearchMode::Regex => regex_query(query_text, fields)?,
});
}
if let Some(content_key) = options.content_key {
clauses.push(Box::new(TermQuery::new(
Term::from_field_text(fields.content_key, &hex::encode(content_key)),
IndexRecordOption::Basic,
)));
}
add_range(
&mut clauses,
fields.total_size,
options.min_size,
options.max_size,
);
add_range(
&mut clauses,
fields.file_count,
options.min_files,
options.max_files,
);
add_range(
&mut clauses,
fields.first_seen,
options.first_seen_after,
options.first_seen_before,
);
add_range(
&mut clauses,
fields.last_seen,
options.last_seen_after,
options.last_seen_before,
);
if let Some(extension) = &options.extension {
let extension = extension.trim().trim_start_matches('.').to_lowercase();
if !extension.is_empty() {
clauses.push(Box::new(TermQuery::new(
Term::from_field_text(fields.extensions, &extension),
IndexRecordOption::Basic,
)));
}
}
if let Some(status) = options.availability {
let value = availability_number(status);
add_range(
&mut clauses,
fields.availability_status,
Some(value),
Some(value),
);
}
if let Some(level) = options.heat {
let (min, max) = match level {
crate::domain::HeatLevel::Hot => (75, 100),
crate::domain::HeatLevel::Active => (50, 74),
crate::domain::HeatLevel::Normal => (25, 49),
crate::domain::HeatLevel::Cold => (0, 24),
};
add_range(&mut clauses, fields.heat_score, Some(min), Some(max));
}
let query = if clauses.len() == 1 {
clauses.pop().expect("one query clause exists")
} else {
Box::new(BooleanQuery::intersection(clauses))
};
let sort = options.sort.unwrap_or_else(|| {
if query_text.is_empty() || query_text == "*" {
SearchSort::Latest
} else {
SearchSort::Relevance
}
});
Ok(PreparedQuery { query, sort })
}
fn regex_query(pattern: &str, fields: SearchFields) -> Result<Box<dyn Query>, SearchError> {
let mut pattern = pattern.to_lowercase();
let anchored_start = pattern.starts_with('^');
if anchored_start {
pattern.remove(0);
}
let anchored_end = pattern.ends_with('$')
&& pattern[..pattern.len() - 1]
.chars()
.rev()
.take_while(|character| *character == '\\')
.count()
.is_multiple_of(2);
if anchored_end {
pattern.pop();
}
let prefix = if anchored_start { "" } else { ".*" };
let suffix = if anchored_end { "" } else { ".*" };
let contains_pattern = format!("{prefix}({pattern}){suffix}");
Ok(Box::new(RegexQuery::from_pattern(
&contains_pattern,
fields.regex_text,
)?))
}
fn wildcard_query(pattern: &str, fields: SearchFields) -> Result<Box<dyn Query>, SearchError> {
Ok(Box::new(RegexQuery::from_pattern(
&wildcard_pattern(pattern),
fields.regex_text,
)?))
}
fn has_wildcard_syntax(pattern: &str) -> bool {
pattern.contains('*') || pattern.contains('?')
}
fn wildcard_pattern(pattern: &str) -> String {
let pattern = normalize_text(pattern);
let mut regex = String::with_capacity(pattern.len());
let mut escaped = false;
for character in pattern.chars() {
if escaped {
push_regex_literal(&mut regex, character);
escaped = false;
continue;
}
match character {
'\\' => escaped = true,
'*' => regex.push_str(".*"),
'?' => regex.push('.'),
literal => push_regex_literal(&mut regex, literal),
}
}
if escaped {
push_regex_literal(&mut regex, '\\');
}
regex
}
fn push_regex_literal(regex: &mut String, character: char) {
if matches!(
character,
'\\' | '.' | '+' | '*' | '?' | '(' | ')' | '|' | '[' | ']' | '{' | '}' | '^' | '$'
) {
regex.push('\\');
}
regex.push(character);
}
fn text_query(query: &str, fields: SearchFields) -> Box<dyn Query> {
let normalized = normalize_text(query.trim());
if normalized.len() == 40 && normalized.bytes().all(|byte| byte.is_ascii_hexdigit()) {
return Box::new(TermQuery::new(
Term::from_field_text(fields.info_hash, &normalized),
IndexRecordOption::Basic,
));
}
let terms = query_terms(query);
if terms.is_empty() {
return Box::new(AllQuery);
}
let mut required = Vec::with_capacity(terms.len());
for term in terms {
let alternatives: Vec<(Occur, Box<dyn Query>)> = vec![
(
Occur::Should,
Box::new(BoostQuery::new(
Box::new(TermQuery::new(
Term::from_field_text(fields.name, &term),
IndexRecordOption::WithFreqs,
)),
3.0,
)),
),
(
Occur::Should,
Box::new(BoostQuery::new(
Box::new(TermQuery::new(
Term::from_field_text(fields.aliases, &term),
IndexRecordOption::WithFreqs,
)),
2.0,
)),
),
(
Occur::Should,
Box::new(TermQuery::new(
Term::from_field_text(fields.files_text, &term),
IndexRecordOption::WithFreqs,
)),
),
];
required.push((
Occur::Must,
Box::new(BooleanQuery::new(alternatives)) as Box<dyn Query>,
));
}
Box::new(BooleanQuery::new(required))
}
fn query_terms(query: &str) -> Vec<String> {
normalize_text(query)
.split_whitespace()
.flat_map(|part| {
let chars: Vec<_> = part.chars().collect();
if chars.len() <= 20 {
vec![part.to_owned()]
} else {
chars
.windows(20)
.map(|window| window.iter().collect())
.collect()
}
})
.collect()
}
fn normalize_text(value: &str) -> String {
value.nfkc().collect::<String>().to_lowercase()
}
fn add_range(
clauses: &mut Vec<Box<dyn Query>>,
field: tantivy::schema::Field,
min: Option<u64>,
max: Option<u64>,
) {
if min.is_none() && max.is_none() {
return;
}
let lower = min
.map(|value| Bound::Included(Term::from_field_u64(field, value)))
.unwrap_or(Bound::Unbounded);
let upper = max
.map(|value| Bound::Included(Term::from_field_u64(field, value)))
.unwrap_or(Bound::Unbounded);
clauses.push(Box::new(RangeQuery::new(lower, upper)));
}
#[cfg(test)]
mod tests {
use super::{has_wildcard_syntax, wildcard_pattern};
#[test]
fn wildcard_conversion_is_case_insensitive_and_escapes_regex_syntax() {
assert_eq!(wildcard_pattern("*.ISO"), r".*\.iso");
assert_eq!(wildcard_pattern("file?.[ch]"), r"file.\.\[ch\]");
assert_eq!(wildcard_pattern(r"literal\*name"), r"literal\*name");
assert!(has_wildcard_syntax("*.iso"));
assert!(has_wildcard_syntax("file?.mkv"));
assert!(has_wildcard_syntax(r"literal\*name"));
}
}
+626
View File
@@ -0,0 +1,626 @@
// 负责批量写入删除提交和从权威存储重建 Tantivy 索引
use std::{
path::Path,
sync::{
Arc, Mutex,
atomic::{AtomicU64, Ordering as AtomicOrdering},
},
time::Instant,
};
use crate::domain::ContentGroup;
use crate::storage::TorrentRepository;
use tantivy::{
DocAddress, Index, IndexReader, IndexWriter, Order, ReloadPolicy, Searcher, TantivyDocument,
Term,
collector::{Count, TopDocs},
directory::MmapDirectory,
query::Query,
tokenizer::{LowerCaser, NgramTokenizer, TextAnalyzer},
};
use super::{
IndexingError, SearchError,
query::{SearchOptions, SearchPage, SearchSort},
schema::{MIXED_NGRAM_TOKENIZER, SearchFields, build_schema},
};
const INDEX_WRITER_MEMORY_BYTES: usize = 64 * 1024 * 1024;
const MAX_PAGE_SIZE: usize = 100;
const MAX_OFFSET: usize = 10_000;
#[derive(Clone)]
pub struct SearchEngine {
inner: Arc<SearchInner>,
}
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub struct SearchDiagnostics {
pub documents: u64,
pub writer_memory_budget_bytes: u64,
pub commits: u64,
pub commit_failures: u64,
pub last_commit_at: Option<u64>,
pub last_commit_duration_millis: u64,
pub last_commit_documents: u64,
}
struct SearchInner {
reader: IndexReader,
writer: Mutex<IndexWriter>,
fields: SearchFields,
commits: AtomicU64,
commit_failures: AtomicU64,
last_commit_at: AtomicU64,
last_commit_duration_millis: AtomicU64,
last_commit_documents: AtomicU64,
}
impl SearchEngine {
pub fn open(path: impl AsRef<Path>) -> Result<Self, SearchError> {
Self::open_with_status(path).map(|(engine, _)| engine)
}
pub fn recreate(path: impl AsRef<Path>) -> Result<Self, SearchError> {
let path = path.as_ref();
if path.exists() {
std::fs::remove_dir_all(path)
.map_err(|error| SearchError::Directory(error.to_string()))?;
}
Self::open(path)
}
pub fn open_with_status(path: impl AsRef<Path>) -> Result<(Self, bool), SearchError> {
let path = path.as_ref().to_path_buf();
std::fs::create_dir_all(&path)
.map_err(|error| SearchError::Directory(error.to_string()))?;
let mut directory = MmapDirectory::open(&path)
.map_err(|error| SearchError::Directory(error.to_string()))?;
let (expected_schema, fields) = build_schema();
let exists =
Index::exists(&directory).map_err(|error| SearchError::Directory(error.to_string()))?;
let mut created = !exists;
let index = if exists {
let index = Index::open(directory)?;
if index.schema() != expected_schema {
drop(index);
std::fs::remove_dir_all(&path)
.map_err(|error| SearchError::Directory(error.to_string()))?;
std::fs::create_dir_all(&path)
.map_err(|error| SearchError::Directory(error.to_string()))?;
directory = MmapDirectory::open(&path)
.map_err(|error| SearchError::Directory(error.to_string()))?;
created = true;
Index::open_or_create(directory, expected_schema)?
} else {
index
}
} else {
Index::open_or_create(directory, expected_schema)?
};
let analyzer = TextAnalyzer::builder(NgramTokenizer::all_ngrams(1, 20)?)
.filter(LowerCaser)
.build();
index.tokenizers().register(MIXED_NGRAM_TOKENIZER, analyzer);
let reader = index
.reader_builder()
.reload_policy(ReloadPolicy::OnCommitWithDelay)
.try_into()?;
let writer = index.writer_with_num_threads(1, INDEX_WRITER_MEMORY_BYTES)?;
Ok((
Self {
inner: Arc::new(SearchInner {
reader,
writer: Mutex::new(writer),
fields,
commits: AtomicU64::new(0),
commit_failures: AtomicU64::new(0),
last_commit_at: AtomicU64::new(0),
last_commit_duration_millis: AtomicU64::new(0),
last_commit_documents: AtomicU64::new(0),
}),
},
created,
))
}
fn index_groups(&self, groups: &[ContentGroup]) -> Result<(), SearchError> {
if groups.is_empty() {
return Ok(());
}
let started = Instant::now();
let fields = self.inner.fields;
let result = (|| {
let mut writer = self
.inner
.writer
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
for group in groups {
writer.delete_term(Term::from_field_text(
fields.content_key,
&hex::encode(group.content_key),
));
writer.add_document(super::document::from_group(group, fields))?;
}
writer.commit()?;
self.inner.reader.reload()?;
Ok(())
})();
self.inner.last_commit_duration_millis.store(
started.elapsed().as_millis().min(u128::from(u64::MAX)) as u64,
AtomicOrdering::Relaxed,
);
if result.is_ok() {
self.inner.commits.fetch_add(1, AtomicOrdering::Relaxed);
self.inner
.last_commit_at
.store(unix_timestamp(), AtomicOrdering::Relaxed);
self.inner.last_commit_documents.store(
groups.len().min(u64::MAX as usize) as u64,
AtomicOrdering::Relaxed,
);
} else {
self.inner
.commit_failures
.fetch_add(1, AtomicOrdering::Relaxed);
}
result
}
pub fn num_docs(&self) -> u64 {
self.inner.reader.searcher().num_docs()
}
pub fn diagnostics(&self) -> SearchDiagnostics {
let last_commit_at = self.inner.last_commit_at.load(AtomicOrdering::Relaxed);
SearchDiagnostics {
documents: self.num_docs(),
writer_memory_budget_bytes: INDEX_WRITER_MEMORY_BYTES as u64,
commits: self.inner.commits.load(AtomicOrdering::Relaxed),
commit_failures: self.inner.commit_failures.load(AtomicOrdering::Relaxed),
last_commit_at: (last_commit_at != 0).then_some(last_commit_at),
last_commit_duration_millis: self
.inner
.last_commit_duration_millis
.load(AtomicOrdering::Relaxed),
last_commit_documents: self
.inner
.last_commit_documents
.load(AtomicOrdering::Relaxed),
}
}
pub fn index_pending(
&self,
repository: &dyn TorrentRepository,
limit: usize,
now: u64,
) -> Result<usize, IndexingError> {
let tasks = repository.pending_index(limit)?;
let mut groups = Vec::with_capacity(tasks.len());
for task in &tasks {
if let Some(group) = repository.content_group(&task.content_key, now)? {
groups.push(group);
}
}
self.index_groups(&groups)?;
for task in &tasks {
repository.mark_indexed(&task.content_key, task.revision)?;
}
Ok(groups.len())
}
pub fn search(
&self,
query_text: &str,
offset: usize,
limit: usize,
) -> Result<SearchPage, SearchError> {
self.search_with(SearchOptions {
query: query_text.to_owned(),
offset,
limit,
..SearchOptions::default()
})
}
pub fn search_with(&self, options: SearchOptions) -> Result<SearchPage, SearchError> {
let offset = options.offset.min(MAX_OFFSET);
let limit = options.limit.clamp(1, MAX_PAGE_SIZE);
let fields = self.inner.fields;
let prepared = super::filter::prepare(&options, fields)?;
let query = prepared.query;
let searcher = self.inner.reader.searcher();
let sort = prepared.sort;
let total = searcher.search(query.as_ref(), &Count)?;
let documents = match sort {
SearchSort::Relevance => searcher
.search(
query.as_ref(),
&TopDocs::with_limit(limit)
.and_offset(offset)
.order_by_score(),
)?
.into_iter()
.collect(),
SearchSort::Latest => sorted_documents(
&searcher,
query.as_ref(),
limit,
offset,
"last_seen",
Order::Desc,
)?,
SearchSort::Oldest => sorted_documents(
&searcher,
query.as_ref(),
limit,
offset,
"first_seen",
Order::Asc,
)?,
SearchSort::Heat => sorted_documents(
&searcher,
query.as_ref(),
limit,
offset,
"heat_score",
Order::Desc,
)?,
SearchSort::SizeDesc => sorted_documents(
&searcher,
query.as_ref(),
limit,
offset,
"total_size",
Order::Desc,
)?,
SearchSort::SizeAsc => sorted_documents(
&searcher,
query.as_ref(),
limit,
offset,
"total_size",
Order::Asc,
)?,
SearchSort::Discoveries => sorted_documents(
&searcher,
query.as_ref(),
limit,
offset,
"seen_count",
Order::Desc,
)?,
};
let mut hits = Vec::with_capacity(documents.len());
for (score, address) in documents {
let document: TantivyDocument = searcher.doc(address)?;
hits.push(super::document::to_hit(&document, fields, score)?);
}
Ok(SearchPage {
total,
offset,
limit,
hits,
sort,
})
}
}
fn sorted_documents(
searcher: &Searcher,
query: &dyn Query,
limit: usize,
offset: usize,
field: &str,
order: Order,
) -> Result<Vec<(f32, DocAddress)>, SearchError> {
let documents: Vec<(Option<u64>, DocAddress)> = searcher.search(
query,
&TopDocs::with_limit(limit)
.and_offset(offset)
.order_by_fast_field::<u64>(field, order),
)?;
Ok(documents
.into_iter()
.map(|(_, address)| (0.0, address))
.collect())
}
fn unix_timestamp() -> u64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap_or_default()
.as_secs()
}
#[cfg(test)]
mod tests {
use std::collections::BTreeMap;
use tempfile::TempDir;
use crate::domain::{
AvailabilityStatus, ContentGroupBuilder, InfoHash, TorrentFile, TorrentRecord,
};
use crate::search::SearchMode;
use crate::storage::{RocksTorrentRepository, TorrentRepository};
use super::*;
fn record() -> TorrentRecord {
TorrentRecord {
info_hash: InfoHash::from_bytes([1; 20]),
name: "Ubuntu Linux 24.04".into(),
total_size: 42,
files: vec![TorrentFile {
path: "ubuntu.iso".into(),
size: 42,
}],
piece_length: 16_384,
source_peers: Vec::new(),
content_key: [2; 32],
searchable: true,
first_seen: 10,
last_seen: 20,
seen_count: 3,
availability: crate::domain::Availability::default(),
activity_score_millis: 1_000,
activity_updated_at: 20,
}
}
fn index_records(engine: &SearchEngine, records: &[TorrentRecord]) {
let mut grouped: BTreeMap<[u8; 32], Vec<TorrentRecord>> = BTreeMap::new();
for record in records {
grouped
.entry(record.content_key)
.or_default()
.push(record.clone());
}
let groups: Vec<_> = grouped
.into_iter()
.filter_map(|(content_key, records)| {
let mut builder = ContentGroupBuilder::new(content_key, unix_timestamp());
for record in records {
builder.push(record);
}
builder.finish()
})
.collect();
engine.index_groups(&groups).unwrap();
}
#[test]
fn record_is_searchable_and_update_is_idempotent() {
let directory = TempDir::new().unwrap();
let engine = SearchEngine::open(directory.path()).unwrap();
let mut record = record();
index_records(&engine, &[record.clone()]);
record.seen_count = 4;
index_records(&engine, &[record]);
let page = engine.search("ubuntu", 0, 10).unwrap();
assert_eq!(page.total, 1);
assert_eq!(page.hits[0].name, "Ubuntu Linux 24.04");
assert_eq!(page.hits[0].seen_count, 4);
}
#[test]
fn file_path_is_searchable() {
let directory = TempDir::new().unwrap();
let engine = SearchEngine::open(directory.path()).unwrap();
index_records(&engine, &[record()]);
assert_eq!(engine.search("ubuntu.iso", 0, 10).unwrap().total, 1);
}
#[test]
fn mixed_substrings_match_chinese_and_release_names() {
let directory = TempDir::new().unwrap();
let engine = SearchEngine::open(directory.path()).unwrap();
let mut record = record();
record.name = "流浪地球 S01E03 1080P".into();
record.files[0].path = "影片/流浪地球.第三集.mkv".into();
index_records(&engine, &[record]);
assert_eq!(engine.search("浪地", 0, 10).unwrap().total, 1);
assert_eq!(engine.search("01E0", 0, 10).unwrap().total, 1);
assert_eq!(engine.search("第三集", 0, 10).unwrap().total, 1);
assert_eq!(engine.search("1080p", 0, 10).unwrap().total, 1);
}
#[test]
fn regex_matches_names_and_file_paths_case_insensitively() {
let directory = TempDir::new().unwrap();
let engine = SearchEngine::open(directory.path()).unwrap();
index_records(&engine, &[record()]);
let name = engine
.search_with(SearchOptions {
query: r"ubuntu\s+linux\s+24\.0[0-9]".into(),
mode: SearchMode::Regex,
limit: 10,
..SearchOptions::default()
})
.unwrap();
assert_eq!(name.total, 1);
let path = engine
.search_with(SearchOptions {
query: r"ubuntu\.(iso|img)$".into(),
mode: SearchMode::Regex,
limit: 10,
..SearchOptions::default()
})
.unwrap();
assert_eq!(path.total, 1);
}
#[test]
fn wildcard_matches_names_and_file_extensions() {
let directory = TempDir::new().unwrap();
let engine = SearchEngine::open(directory.path()).unwrap();
index_records(&engine, &[record()]);
for pattern in ["*.iso", "Ubuntu*", "ubuntu.?so"] {
let page = engine
.search_with(SearchOptions {
query: pattern.into(),
limit: 10,
..SearchOptions::default()
})
.unwrap();
assert_eq!(page.total, 1, "pattern {pattern}");
}
let missing = engine
.search_with(SearchOptions {
query: "*.img".into(),
limit: 10,
..SearchOptions::default()
})
.unwrap();
assert_eq!(missing.total, 0);
}
#[test]
fn invalid_regex_is_rejected() {
let directory = TempDir::new().unwrap();
let engine = SearchEngine::open(directory.path()).unwrap();
index_records(&engine, &[record()]);
let result = engine.search_with(SearchOptions {
query: "[".into(),
mode: SearchMode::Regex,
limit: 10,
..SearchOptions::default()
});
assert!(result.is_err());
}
#[test]
fn equal_content_is_collapsed_and_aliases_remain_searchable() {
let directory = TempDir::new().unwrap();
let engine = SearchEngine::open(directory.path()).unwrap();
let first = record();
let mut second = first.clone();
second.info_hash = InfoHash::from_bytes([2; 20]);
second.name = "Ubuntu Alternate Name".into();
second.availability.status = AvailabilityStatus::Active;
second.availability.reachable_peers = 1;
index_records(&engine, &[first, second.clone()]);
let page = engine.search("alternate", 0, 10).unwrap();
assert_eq!(page.total, 1);
assert_eq!(page.hits[0].variant_count, 2);
assert_eq!(page.hits[0].info_hash, second.info_hash.to_string());
}
#[test]
fn size_and_extension_filters_use_the_index() {
let directory = TempDir::new().unwrap();
let engine = SearchEngine::open(directory.path()).unwrap();
index_records(&engine, &[record()]);
let matching = engine
.search_with(SearchOptions {
query: String::new(),
min_size: Some(40),
max_size: Some(50),
extension: Some("ISO".into()),
limit: 10,
..SearchOptions::default()
})
.unwrap();
assert_eq!(matching.total, 1);
let excluded = engine
.search_with(SearchOptions {
query: String::new(),
min_size: Some(100),
limit: 10,
..SearchOptions::default()
})
.unwrap();
assert_eq!(excluded.total, 0);
}
#[test]
fn exact_infohash_is_searchable_without_ngram_splitting() {
let directory = TempDir::new().unwrap();
let engine = SearchEngine::open(directory.path()).unwrap();
let record = record();
index_records(&engine, std::slice::from_ref(&record));
let page = engine.search(&record.info_hash.to_string(), 0, 10).unwrap();
assert_eq!(page.total, 1);
assert_eq!(page.hits[0].info_hash, record.info_hash.to_string());
}
#[test]
fn filters_and_sorts_use_group_fast_fields() {
let directory = TempDir::new().unwrap();
let engine = SearchEngine::open(directory.path()).unwrap();
let first = record();
let mut second = record();
second.info_hash = InfoHash::from_bytes([3; 20]);
second.content_key = [3; 32];
second.name = "New Release".into();
second.first_seen = 30;
second.last_seen = 40;
second.seen_count = 10;
second.total_size = 100;
second.files.push(TorrentFile {
path: "extra.mkv".into(),
size: 58,
});
second.availability.status = AvailabilityStatus::Active;
index_records(&engine, &[first, second.clone()]);
let page = engine
.search_with(SearchOptions {
query: String::new(),
min_files: Some(2),
last_seen_after: Some(30),
availability: Some(AvailabilityStatus::Active),
sort: Some(SearchSort::SizeDesc),
limit: 10,
..SearchOptions::default()
})
.unwrap();
assert_eq!(page.total, 1);
assert_eq!(page.sort, SearchSort::SizeDesc);
assert_eq!(page.hits[0].info_hash, second.info_hash.to_string());
}
#[test]
fn incompatible_schema_is_deleted_and_recreated() {
let directory = TempDir::new().unwrap();
let index_path = directory.path().join("tantivy");
std::fs::create_dir(&index_path).unwrap();
let mut old_schema = tantivy::schema::Schema::builder();
old_schema.add_text_field("old_name", tantivy::schema::TEXT);
Index::create_in_dir(&index_path, old_schema.build()).unwrap();
let (engine, created) = SearchEngine::open_with_status(&index_path).unwrap();
assert!(created);
assert_eq!(engine.num_docs(), 0);
assert!(index_path.exists());
}
#[test]
fn deleted_index_is_rebuilt_from_rocksdb_content_groups() {
let directory = TempDir::new().unwrap();
let repository = RocksTorrentRepository::open(directory.path().join("rocks")).unwrap();
repository.upsert(record()).unwrap();
let index_path = directory.path().join("tantivy");
let engine = SearchEngine::open(&index_path).unwrap();
engine.index_pending(&repository, 10, 30).unwrap();
assert_eq!(engine.search("ubuntu", 0, 10).unwrap().total, 1);
drop(engine);
std::fs::remove_dir_all(&index_path).unwrap();
let (rebuilt, created) = SearchEngine::open_with_status(&index_path).unwrap();
assert!(created);
assert_eq!(repository.prepare_full_reindex().unwrap(), 1);
rebuilt.index_pending(&repository, 10, 40).unwrap();
assert_eq!(rebuilt.search("ubuntu", 0, 10).unwrap().total, 1);
}
}
+75
View File
@@ -0,0 +1,75 @@
// 负责暴露全文搜索抽象并隐藏 Tantivy 的具体实现细节
mod document;
mod filter;
mod indexer;
mod query;
mod schema;
pub use indexer::{SearchDiagnostics, SearchEngine};
pub use query::{
AvailabilitySummary, SearchHit, SearchMode, SearchOptions, SearchPage, SearchSort,
};
#[derive(Debug, thiserror::Error)]
pub enum SearchError {
#[error("搜索索引操作失败: {0}")]
Tantivy(#[from] tantivy::TantivyError),
#[error("无法打开搜索索引目录: {0}")]
Directory(String),
#[error("搜索文档缺少字段 {0}")]
MissingField(&'static str),
}
impl SearchError {
pub fn is_retryable_io(&self) -> bool {
matches!(
self,
Self::Tantivy(tantivy::TantivyError::IoError(error))
if matches!(
error.kind(),
std::io::ErrorKind::PermissionDenied
| std::io::ErrorKind::WouldBlock
| std::io::ErrorKind::Interrupted
| std::io::ErrorKind::TimedOut
)
)
}
}
#[derive(Debug, thiserror::Error)]
pub enum IndexingError {
#[error(transparent)]
Search(#[from] SearchError),
#[error(transparent)]
Storage(#[from] crate::storage::StorageError),
}
impl IndexingError {
pub fn is_retryable_io(&self) -> bool {
matches!(self, Self::Search(error) if error.is_retryable_io())
}
}
#[cfg(test)]
mod tests {
use std::{io, sync::Arc};
use super::{IndexingError, SearchError};
#[test]
fn permission_denied_index_error_is_retryable() {
let error = IndexingError::Search(SearchError::Tantivy(tantivy::TantivyError::IoError(
Arc::new(io::Error::from(io::ErrorKind::PermissionDenied)),
)));
assert!(error.is_retryable_io());
}
#[test]
fn invalid_query_is_not_retryable() {
let error = IndexingError::Search(SearchError::Tantivy(
tantivy::TantivyError::InvalidArgument("invalid".into()),
));
assert!(!error.is_retryable_io());
}
}
+79
View File
@@ -0,0 +1,79 @@
// 负责构建全文查询过滤排序分页和内容聚合逻辑
use serde::{Deserialize, Serialize};
use crate::domain::{AvailabilityStatus, Heat, HeatLevel};
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Deserialize, Serialize)]
#[serde(rename_all = "snake_case")]
pub enum SearchSort {
#[default]
Relevance,
Latest,
Oldest,
Heat,
SizeDesc,
SizeAsc,
Discoveries,
}
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Deserialize, Serialize)]
#[serde(rename_all = "snake_case")]
pub enum SearchMode {
#[default]
Text,
Regex,
}
#[derive(Debug, Clone, Default)]
pub struct SearchOptions {
pub query: String,
pub mode: SearchMode,
pub offset: usize,
pub limit: usize,
pub min_size: Option<u64>,
pub max_size: Option<u64>,
pub extension: Option<String>,
pub min_files: Option<u64>,
pub max_files: Option<u64>,
pub first_seen_after: Option<u64>,
pub first_seen_before: Option<u64>,
pub last_seen_after: Option<u64>,
pub last_seen_before: Option<u64>,
pub availability: Option<AvailabilityStatus>,
pub heat: Option<HeatLevel>,
pub sort: Option<SearchSort>,
pub content_key: Option<[u8; 32]>,
}
#[derive(Debug, Clone, PartialEq, Serialize)]
pub struct SearchHit {
pub info_hash: String,
pub name: String,
pub total_size: u64,
pub file_count: u64,
pub first_seen: u64,
pub last_seen: u64,
pub seen_count: u64,
pub content_key: String,
pub variant_count: u64,
pub score: f32,
pub heat: Heat,
pub availability: AvailabilitySummary,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
pub struct AvailabilitySummary {
pub status: AvailabilityStatus,
pub last_verified_at: Option<u64>,
pub reachable_peers: u32,
}
#[derive(Debug, Clone, PartialEq, Serialize)]
pub struct SearchPage {
pub total: usize,
pub offset: usize,
pub limit: usize,
pub hits: Vec<SearchHit>,
pub sort: SearchSort,
}
+80
View File
@@ -0,0 +1,80 @@
// 负责定义 Tantivy 字段分词索引存储和快速字段策略
use tantivy::schema::{
FAST, Field, IndexRecordOption, STORED, STRING, Schema, TextFieldIndexing, TextOptions,
};
pub(crate) const MIXED_NGRAM_TOKENIZER: &str = "dht_mixed_ngram";
#[derive(Debug, Clone, Copy)]
pub(crate) struct SearchFields {
pub(crate) info_hash: Field,
pub(crate) name: Field,
pub(crate) display_name: Field,
pub(crate) aliases: Field,
pub(crate) files_text: Field,
pub(crate) regex_text: Field,
pub(crate) extensions: Field,
pub(crate) total_size: Field,
pub(crate) file_count: Field,
pub(crate) first_seen: Field,
pub(crate) last_seen: Field,
pub(crate) seen_count: Field,
pub(crate) content_key: Field,
pub(crate) availability_status: Field,
pub(crate) reachable_peers: Field,
pub(crate) last_verified_at: Field,
pub(crate) heat_score: Field,
pub(crate) variant_count: Field,
}
pub(crate) fn build_schema() -> (Schema, SearchFields) {
let mut builder = Schema::builder();
let info_hash = builder.add_text_field("info_hash", STRING | STORED);
let indexed_text = TextOptions::default().set_indexing_options(
TextFieldIndexing::default()
.set_tokenizer(MIXED_NGRAM_TOKENIZER)
.set_index_option(IndexRecordOption::WithFreqsAndPositions),
);
let name = builder.add_text_field("name", indexed_text.clone());
let display_name = builder.add_text_field("display_name", STORED);
let aliases = builder.add_text_field("aliases", indexed_text.clone());
let files_text = builder.add_text_field("files_text", indexed_text);
let regex_text = builder.add_text_field("regex_text", STRING);
let extensions = builder.add_text_field("extensions", STRING);
let total_size = builder.add_u64_field("total_size", FAST | STORED);
let file_count = builder.add_u64_field("file_count", FAST | STORED);
let first_seen = builder.add_u64_field("first_seen", FAST | STORED);
let last_seen = builder.add_u64_field("last_seen", FAST | STORED);
let seen_count = builder.add_u64_field("seen_count", FAST | STORED);
let content_key = builder.add_text_field("content_key", STRING | STORED);
let availability_status = builder.add_u64_field("availability_status", FAST | STORED);
let reachable_peers = builder.add_u64_field("reachable_peers", FAST | STORED);
let last_verified_at = builder.add_u64_field("last_verified_at", FAST | STORED);
let heat_score = builder.add_u64_field("heat_score", FAST | STORED);
let variant_count = builder.add_u64_field("variant_count", FAST | STORED);
let schema = builder.build();
(
schema,
SearchFields {
info_hash,
name,
display_name,
aliases,
files_text,
regex_text,
extensions,
total_size,
file_count,
first_seen,
last_seen,
seen_count,
content_key,
availability_status,
reachable_peers,
last_verified_at,
heat_score,
variant_count,
},
)
}
+23
View File
@@ -0,0 +1,23 @@
// 负责监听退出信号并协调有界队列排空提交和资源关闭
pub(crate) async fn signal() {
#[cfg(unix)]
{
use tokio::signal::unix::{SignalKind, signal};
let mut terminate = signal(SignalKind::terminate()).expect("SIGTERM handler must install");
tokio::select! {
result = tokio::signal::ctrl_c() => {
if let Err(error) = result {
tracing::error!(%error, "无法监听 Ctrl+C")
}
}
_ = terminate.recv() => {}
}
}
#[cfg(not(unix))]
if let Err(error) = tokio::signal::ctrl_c().await {
tracing::error!(%error, "无法监听 Ctrl+C")
}
}
+190
View File
@@ -0,0 +1,190 @@
// 负责定义稳定的 RocksDB 键空间编码和版本边界
use crate::domain::InfoHash;
pub(crate) const DATABASE_FORMAT_KEY: &[u8] = b"\x00database-format";
pub(crate) const DATABASE_FORMAT_VALUE: &[u8] = b"dht-search";
pub(crate) const VERIFICATION_QUEUE_COUNT_KEY: &[u8] = b"\x00verification-queue-count";
pub(crate) const CONTENT_FILTER_FINGERPRINT_KEY: &[u8] = b"\x00content-filter-fingerprint";
pub(crate) const CONTENT_FILTER_MIGRATION_KEY: &[u8] = b"\x00content-filter-migration";
const TORRENT_PREFIX: u8 = b't';
const REJECTED_METADATA_PREFIX: u8 = b'r';
const CONTENT_PREFIX: u8 = b'c';
const CONTENT_GROUP_PREFIX: u8 = b'g';
const PENDING_INDEX_PREFIX: u8 = b'p';
const VERIFICATION_HIGH_PREFIX: u8 = b'h';
const VERIFICATION_NORMAL_PREFIX: u8 = b'n';
const VERIFICATION_LEASE_PREFIX: u8 = b'l';
const VERIFICATION_LOCATOR_PREFIX: u8 = b'v';
pub(crate) fn torrent_key(info_hash: InfoHash) -> [u8; 1 + InfoHash::BYTE_LEN] {
prefixed_info_hash(TORRENT_PREFIX, info_hash)
}
pub(crate) fn torrent_prefix() -> [u8; 1] {
[TORRENT_PREFIX]
}
pub(crate) fn rejected_metadata_key(info_hash: InfoHash) -> [u8; 1 + InfoHash::BYTE_LEN] {
prefixed_info_hash(REJECTED_METADATA_PREFIX, info_hash)
}
pub(crate) fn content_group_key(content_key: &[u8; 32]) -> [u8; 1 + 32] {
prefixed_content_key(CONTENT_GROUP_PREFIX, content_key)
}
pub(crate) fn content_group_prefix() -> [u8; 1] {
[CONTENT_GROUP_PREFIX]
}
pub(crate) fn pending_index_key(content_key: &[u8; 32]) -> [u8; 1 + 32] {
prefixed_content_key(PENDING_INDEX_PREFIX, content_key)
}
pub(crate) fn pending_index_prefix() -> [u8; 1] {
[PENDING_INDEX_PREFIX]
}
pub(crate) fn verification_task_key(
high_priority: bool,
requested_at: u64,
info_hash: InfoHash,
) -> [u8; 1 + 8 + InfoHash::BYTE_LEN] {
timed_info_hash_key(
if high_priority {
VERIFICATION_HIGH_PREFIX
} else {
VERIFICATION_NORMAL_PREFIX
},
requested_at,
info_hash,
)
}
pub(crate) fn verification_task_prefix(high_priority: bool) -> [u8; 1] {
[if high_priority {
VERIFICATION_HIGH_PREFIX
} else {
VERIFICATION_NORMAL_PREFIX
}]
}
pub(crate) fn verification_lease_key(
lease_until: u64,
info_hash: InfoHash,
) -> [u8; 1 + 8 + InfoHash::BYTE_LEN] {
timed_info_hash_key(VERIFICATION_LEASE_PREFIX, lease_until, info_hash)
}
pub(crate) fn verification_lease_prefix() -> [u8; 1] {
[VERIFICATION_LEASE_PREFIX]
}
pub(crate) fn verification_locator_key(info_hash: InfoHash) -> [u8; 1 + InfoHash::BYTE_LEN] {
prefixed_info_hash(VERIFICATION_LOCATOR_PREFIX, info_hash)
}
pub(crate) fn decode_timed_info_hash(key: &[u8], prefix: u8) -> Option<(u64, InfoHash)> {
if key.len() != 1 + 8 + InfoHash::BYTE_LEN || key.first().copied() != Some(prefix) {
return None;
}
let timestamp = u64::from_be_bytes(key[1..9].try_into().ok()?);
let bytes = key[9..].try_into().ok()?;
Some((timestamp, InfoHash::from_bytes(bytes)))
}
pub(crate) fn decode_verification_task(key: &[u8]) -> Option<(bool, u64, InfoHash)> {
match key.first().copied()? {
VERIFICATION_HIGH_PREFIX => {
decode_timed_info_hash(key, VERIFICATION_HIGH_PREFIX).map(|(at, hash)| (true, at, hash))
}
VERIFICATION_NORMAL_PREFIX => decode_timed_info_hash(key, VERIFICATION_NORMAL_PREFIX)
.map(|(at, hash)| (false, at, hash)),
_ => None,
}
}
pub(crate) fn decode_verification_lease(key: &[u8]) -> Option<(u64, InfoHash)> {
decode_timed_info_hash(key, VERIFICATION_LEASE_PREFIX)
}
pub(crate) fn content_member_key(
content_key: &[u8; 32],
info_hash: InfoHash,
) -> [u8; 1 + 32 + InfoHash::BYTE_LEN] {
let mut key = [0_u8; 1 + 32 + InfoHash::BYTE_LEN];
key[0] = CONTENT_PREFIX;
key[1..33].copy_from_slice(content_key);
key[33..].copy_from_slice(info_hash.as_bytes());
key
}
pub(crate) fn content_members_prefix(content_key: &[u8; 32]) -> [u8; 1 + 32] {
let mut key = [0_u8; 1 + 32];
key[0] = CONTENT_PREFIX;
key[1..].copy_from_slice(content_key);
key
}
pub(crate) fn content_member_prefix() -> [u8; 1] {
[CONTENT_PREFIX]
}
pub(crate) fn decode_content_member_info_hash(
key: &[u8],
content_key: &[u8; 32],
) -> Option<InfoHash> {
let expected_prefix = content_members_prefix(content_key);
if key.len() != 1 + 32 + InfoHash::BYTE_LEN || !key.starts_with(&expected_prefix) {
return None;
}
let bytes: [u8; InfoHash::BYTE_LEN] = key[33..].try_into().ok()?;
Some(InfoHash::from_bytes(bytes))
}
pub(crate) fn decode_pending_content_key(key: &[u8]) -> Option<[u8; 32]> {
if key.len() != 1 + 32 || key.first().copied() != Some(PENDING_INDEX_PREFIX) {
return None;
}
key[1..].try_into().ok()
}
fn prefixed_content_key(prefix: u8, content_key: &[u8; 32]) -> [u8; 1 + 32] {
let mut key = [0_u8; 1 + 32];
key[0] = prefix;
key[1..].copy_from_slice(content_key);
key
}
fn prefixed_info_hash(prefix: u8, info_hash: InfoHash) -> [u8; 1 + InfoHash::BYTE_LEN] {
let mut key = [0_u8; 1 + InfoHash::BYTE_LEN];
key[0] = prefix;
key[1..].copy_from_slice(info_hash.as_bytes());
key
}
fn timed_info_hash_key(
prefix: u8,
timestamp: u64,
info_hash: InfoHash,
) -> [u8; 1 + 8 + InfoHash::BYTE_LEN] {
let mut key = [0_u8; 1 + 8 + InfoHash::BYTE_LEN];
key[0] = prefix;
key[1..9].copy_from_slice(&timestamp.to_be_bytes());
key[9..].copy_from_slice(info_hash.as_bytes());
key
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn pending_key_round_trips_content_key() {
let content_key = [7; 32];
assert_eq!(
decode_pending_content_key(&pending_index_key(&content_key)),
Some(content_key)
);
}
}
+15
View File
@@ -0,0 +1,15 @@
// 负责暴露持久化抽象并隐藏 RocksDB 的具体实现细节
#[cfg(feature = "rocksdb-storage")]
mod keys;
mod repository;
#[cfg(feature = "rocksdb-storage")]
mod rocks;
pub use repository::{
CheckpointSummary, ContentGroupTask, ContentVariants, StorageDiagnostics, StorageError,
TorrentRepository, UpsertOutcome, VerificationEnqueueOutcome, VerificationPriority,
VerificationRequest,
};
#[cfg(feature = "rocksdb-storage")]
pub use rocks::RocksTorrentRepository;
+155
View File
@@ -0,0 +1,155 @@
// 负责定义元数据去重状态恢复和索引任务所需的存储接口
use crate::domain::{
ContentGroup, InfoHash, RejectedMetadata, TorrentRecord, TorrentRecordError, VerificationResult,
};
pub trait TorrentRepository: Send + Sync {
fn get(&self, info_hash: InfoHash) -> Result<Option<TorrentRecord>, StorageError>;
fn get_visible(&self, info_hash: InfoHash) -> Result<Option<TorrentRecord>, StorageError> {
self.get(info_hash)
}
fn contains(&self, info_hash: InfoHash) -> Result<bool, StorageError> {
self.get(info_hash).map(|record| record.is_some())
}
fn upsert(&self, observation: TorrentRecord) -> Result<UpsertOutcome, StorageError>;
fn rejection(&self, info_hash: InfoHash) -> Result<Option<RejectedMetadata>, StorageError>;
fn record_rejection(&self, rejection: RejectedMetadata) -> Result<(), StorageError>;
fn observe_existing(&self, info_hash: InfoHash, observed_at: u64)
-> Result<bool, StorageError>;
fn filter_unknown_and_observe(
&self,
info_hashes: &[InfoHash],
observed_at: u64,
) -> Result<Vec<InfoHash>, StorageError> {
let mut unknown = Vec::with_capacity(info_hashes.len());
for info_hash in info_hashes {
if !self.observe_existing(*info_hash, observed_at)? {
unknown.push(*info_hash);
}
}
Ok(unknown)
}
fn pending_index(&self, limit: usize) -> Result<Vec<ContentGroupTask>, StorageError>;
fn content_group(
&self,
content_key: &[u8; 32],
now: u64,
) -> Result<Option<ContentGroup>, StorageError>;
fn mark_indexed(&self, content_key: &[u8; 32], revision: u64) -> Result<bool, StorageError>;
fn prepare_full_reindex(&self) -> Result<u64, StorageError>;
fn content_variants(
&self,
content_key: &[u8; 32],
offset: usize,
limit: usize,
) -> Result<ContentVariants, StorageError>;
fn enqueue_verification(
&self,
info_hashes: &[InfoHash],
priority: VerificationPriority,
requested_at: u64,
capacity: usize,
) -> Result<VerificationEnqueueOutcome, StorageError>;
fn claim_verification(
&self,
now: u64,
lease_secs: u64,
) -> Result<Option<VerificationRequest>, StorageError>;
fn finish_verification(
&self,
info_hash: InfoHash,
result: VerificationResult,
) -> Result<(), StorageError>;
fn verification_queue_len(&self) -> Result<usize, StorageError>;
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct ContentGroupTask {
pub content_key: [u8; 32],
pub revision: u64,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ContentVariants {
pub total: u64,
pub records: Vec<TorrentRecord>,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct CheckpointSummary {
pub records: u64,
}
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub struct StorageDiagnostics {
pub block_cache_bytes: Option<u64>,
pub memtable_bytes: Option<u64>,
pub pending_compaction_bytes: Option<u64>,
pub live_sst_bytes: Option<u64>,
pub running_compactions: Option<u64>,
pub estimated_keys: Option<u64>,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum VerificationPriority {
Normal,
High,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct VerificationRequest {
pub info_hash: InfoHash,
}
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub struct VerificationEnqueueOutcome {
pub accepted: usize,
pub deduplicated: usize,
pub rejected_full: usize,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum UpsertOutcome {
Inserted,
Updated { seen_count: u64 },
}
#[derive(Debug, thiserror::Error)]
pub enum StorageError {
#[cfg(feature = "rocksdb-storage")]
#[error("RocksDB 操作失败: {0}")]
RocksDb(#[from] rocksdb::Error),
#[error("记录编码失败: {0}")]
Encode(#[from] rmp_serde::encode::Error),
#[error("记录解码失败: {0}")]
Decode(#[from] rmp_serde::decode::Error),
#[error("无法计算过滤后的内容结构: {0}")]
DerivedContent(#[from] TorrentRecordError),
#[error("数据库格式与当前程序不兼容 请清理开发数据目录后重新启动")]
IncompatibleDatabaseFormat,
#[error("待索引内容组不存在 content_key={0}")]
MissingContentGroup(String),
#[error("种子记录不存在 infohash={0}")]
MissingRecord(InfoHash),
#[error("内容组状态数据损坏")]
CorruptContentGroup,
#[error("验证队列计数数据损坏")]
CorruptVerificationQueueCount,
}
File diff suppressed because it is too large Load Diff
+143
View File
@@ -0,0 +1,143 @@
// 负责初始化终端与滚动文件日志并持有异步日志刷新守卫
use tracing_appender::{
non_blocking::WorkerGuard,
rolling::{RollingFileAppender, Rotation},
};
use tracing_subscriber::{EnvFilter, layer::SubscriberExt, util::SubscriberInitExt};
use crate::config::{LogRotation, LoggingConfig};
pub(crate) struct TelemetryGuard {
_file_guard: Option<WorkerGuard>,
}
pub(crate) fn init(config: &LoggingConfig) -> Result<TelemetryGuard, String> {
let filter = EnvFilter::try_from_default_env()
.unwrap_or_else(|_| EnvFilter::new("warn,dht_search=info,dht_crawler=info"));
let console_layer = config.console_enabled.then(|| {
tracing_subscriber::fmt::layer()
.with_target(true)
.with_ansi(std::io::IsTerminal::is_terminal(&std::io::stderr()))
.with_writer(std::io::stderr)
});
let (file_layer, file_guard) = if config.file_enabled {
let appender = build_file_appender(config)?;
let (writer, guard) = tracing_appender::non_blocking(appender);
(
Some(
tracing_subscriber::fmt::layer()
.with_target(true)
.with_ansi(false)
.with_writer(writer),
),
Some(guard),
)
} else {
(None, None)
};
tracing_subscriber::registry()
.with(filter)
.with(console_layer)
.with(file_layer)
.try_init()
.map_err(|error| error.to_string())?;
Ok(TelemetryGuard {
_file_guard: file_guard,
})
}
fn build_file_appender(config: &LoggingConfig) -> Result<RollingFileAppender, String> {
std::fs::create_dir_all(&config.directory).map_err(|error| error.to_string())?;
RollingFileAppender::builder()
.rotation(rotation(config.rotation))
.filename_prefix(&config.file_prefix)
.filename_suffix("log")
.max_log_files(config.retain_files)
.build(&config.directory)
.map_err(|error| error.to_string())
}
const fn rotation(value: LogRotation) -> Rotation {
match value {
LogRotation::Minutely => Rotation::MINUTELY,
LogRotation::Hourly => Rotation::HOURLY,
LogRotation::Daily => Rotation::DAILY,
LogRotation::Never => Rotation::NEVER,
}
}
#[cfg(test)]
mod tests {
use std::io::Write;
use tempfile::TempDir;
use super::*;
#[test]
fn rolling_appender_writes_inside_configured_directory() {
let directory = TempDir::new().unwrap();
let config = LoggingConfig {
directory: directory.path().to_path_buf(),
rotation: LogRotation::Never,
retain_files: 2,
file_prefix: "service-test".into(),
..LoggingConfig::default()
};
let appender = build_file_appender(&config).unwrap();
let (mut writer, guard) = tracing_appender::non_blocking(appender);
writeln!(writer, "hello rolling log").unwrap();
drop(writer);
drop(guard);
let files: Vec<_> = std::fs::read_dir(directory.path())
.unwrap()
.map(|entry| entry.unwrap().file_name())
.collect();
assert_eq!(files.len(), 1);
assert!(files[0].to_string_lossy().starts_with("service-test"));
let contents = std::fs::read_to_string(directory.path().join(&files[0])).unwrap();
assert!(contents.contains("hello rolling log"));
}
#[test]
fn retention_limit_prunes_only_matching_log_files() {
let directory = TempDir::new().unwrap();
for day in 1..=3 {
std::fs::write(
directory
.path()
.join(format!("service-test.2020-01-0{day}.log")),
day.to_string(),
)
.unwrap();
std::thread::sleep(std::time::Duration::from_millis(10));
}
std::fs::write(directory.path().join("unrelated.log"), "keep").unwrap();
std::fs::create_dir(directory.path().join("service-test.directory.log")).unwrap();
let config = LoggingConfig {
directory: directory.path().to_path_buf(),
rotation: LogRotation::Daily,
retain_files: 2,
file_prefix: "service-test".into(),
..LoggingConfig::default()
};
drop(build_file_appender(&config).unwrap());
let matching_files = std::fs::read_dir(directory.path())
.unwrap()
.filter_map(Result::ok)
.filter(|entry| entry.file_type().is_ok_and(|kind| kind.is_file()))
.filter(|entry| {
let name = entry.file_name();
let name = name.to_string_lossy();
name.starts_with("service-test") && name.ends_with(".log")
})
.count();
assert!(matching_files <= 2);
assert!(directory.path().join("unrelated.log").exists());
assert!(directory.path().join("service-test.directory.log").is_dir());
}
}
+306
View File
@@ -0,0 +1,306 @@
// 负责按需调度种子可用性验证并以有界并发持久化验证结果
use std::sync::{
Arc,
atomic::{AtomicU64, Ordering},
};
use std::time::{Duration, SystemTime, UNIX_EPOCH};
use std::{collections::HashSet, net::SocketAddr};
use crate::{
domain::{InfoHash, VerificationResult},
storage::{RocksTorrentRepository, TorrentRepository, VerificationPriority},
};
use dht_crawler::DHTServer;
use tokio::task::{JoinHandle, JoinSet};
use tokio_util::sync::CancellationToken;
#[cfg(test)]
use crate::config::DiskGuardConfig;
use crate::{
config::VerificationConfig,
disk_guard::{DiskGuard, DiskWritePermit},
};
#[derive(Clone)]
pub(crate) struct VerificationIngress {
repository: Arc<RocksTorrentRepository>,
capacity: usize,
stats: VerificationStats,
disk_guard: DiskGuard,
}
#[derive(Clone, Default)]
pub(crate) struct VerificationStats {
inner: Arc<VerificationStatsInner>,
}
#[derive(Default)]
struct VerificationStatsInner {
accepted: AtomicU64,
deduplicated: AtomicU64,
rejected_full: AtomicU64,
started: AtomicU64,
succeeded: AtomicU64,
failed: AtomicU64,
peers_discovered: AtomicU64,
handshakes_succeeded: AtomicU64,
queue_depth: AtomicU64,
}
#[derive(Debug, Clone, Copy)]
pub(crate) struct VerificationSnapshot {
pub(crate) accepted: u64,
pub(crate) deduplicated: u64,
pub(crate) rejected_full: u64,
pub(crate) started: u64,
pub(crate) succeeded: u64,
pub(crate) failed: u64,
pub(crate) peers_discovered: u64,
pub(crate) handshakes_succeeded: u64,
pub(crate) queue_depth: u64,
}
pub(crate) fn start(
repository: Arc<RocksTorrentRepository>,
server: DHTServer,
config: VerificationConfig,
disk_guard: DiskGuard,
cancel: CancellationToken,
) -> (VerificationIngress, JoinHandle<Result<(), String>>) {
let stats = VerificationStats::default();
stats.inner.queue_depth.store(
repository.verification_queue_len().unwrap_or_default() as u64,
Ordering::Relaxed,
);
let ingress = VerificationIngress::new(
repository.clone(),
config.queue_capacity,
stats.clone(),
disk_guard.clone(),
);
let task = tokio::spawn(run(repository, server, config, stats, disk_guard, cancel));
(ingress, task)
}
impl VerificationIngress {
fn new(
repository: Arc<RocksTorrentRepository>,
capacity: usize,
stats: VerificationStats,
disk_guard: DiskGuard,
) -> Self {
Self {
repository,
capacity,
stats,
disk_guard,
}
}
#[cfg(test)]
pub(crate) fn for_test(repository: Arc<RocksTorrentRepository>, capacity: usize) -> Self {
Self::new(
repository,
capacity,
VerificationStats::default(),
DiskGuard::new(&DiskGuardConfig {
enabled: false,
..DiskGuardConfig::default()
}),
)
}
pub(crate) async fn enqueue(&self, hashes: Vec<InfoHash>, priority: VerificationPriority) {
if hashes.is_empty() {
return;
}
let Some(permit) = self.disk_guard.begin_admission() else {
return;
};
let repository = self.repository.clone();
let capacity = self.capacity;
let result = tokio::task::spawn_blocking(move || {
let _permit = permit;
let outcome =
repository.enqueue_verification(&hashes, priority, unix_timestamp(), capacity)?;
let queue_len = repository.verification_queue_len()?;
Ok::<_, crate::storage::StorageError>((outcome, queue_len))
})
.await;
match result {
Ok(Ok((outcome, queue_len))) => {
self.stats
.inner
.accepted
.fetch_add(outcome.accepted as u64, Ordering::Relaxed);
self.stats
.inner
.deduplicated
.fetch_add(outcome.deduplicated as u64, Ordering::Relaxed);
self.stats
.inner
.rejected_full
.fetch_add(outcome.rejected_full as u64, Ordering::Relaxed);
self.stats
.inner
.queue_depth
.store(queue_len as u64, Ordering::Relaxed);
}
Ok(Err(error)) => tracing::error!(%error, "可用性验证任务持久化失败"),
Err(error) => tracing::error!(%error, "可用性验证入队任务异常"),
}
}
pub(crate) fn stats(&self) -> VerificationStats {
self.stats.clone()
}
}
impl VerificationStats {
pub(crate) fn snapshot(&self) -> VerificationSnapshot {
VerificationSnapshot {
accepted: self.inner.accepted.load(Ordering::Relaxed),
deduplicated: self.inner.deduplicated.load(Ordering::Relaxed),
rejected_full: self.inner.rejected_full.load(Ordering::Relaxed),
started: self.inner.started.load(Ordering::Relaxed),
succeeded: self.inner.succeeded.load(Ordering::Relaxed),
failed: self.inner.failed.load(Ordering::Relaxed),
peers_discovered: self.inner.peers_discovered.load(Ordering::Relaxed),
handshakes_succeeded: self.inner.handshakes_succeeded.load(Ordering::Relaxed),
queue_depth: self.inner.queue_depth.load(Ordering::Relaxed),
}
}
}
async fn run(
repository: Arc<RocksTorrentRepository>,
server: DHTServer,
config: VerificationConfig,
stats: VerificationStats,
disk_guard: DiskGuard,
cancel: CancellationToken,
) -> Result<(), String> {
let mut active = JoinSet::new();
let mut ticker = tokio::time::interval(Duration::from_millis(config.poll_interval_millis));
ticker.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
loop {
tokio::select! {
_ = cancel.cancelled() => break,
result = active.join_next(), if !active.is_empty() => {
if let Some(result) = result {
result.map_err(|error| error.to_string())??;
}
}
_ = ticker.tick() => {
while active.len() < config.max_active {
let Some(permit) = disk_guard.begin_new_write() else { break };
let claim_repository = repository.clone();
let lease_secs = config.lease_secs;
let request = tokio::task::spawn_blocking(move || {
claim_repository
.claim_verification(unix_timestamp(), lease_secs)
.map(|request| (request, permit))
})
.await
.map_err(|error| error.to_string())?
.map_err(|error| error.to_string())?;
let (request, permit) = request;
let Some(request) = request else { break };
stats.inner.started.fetch_add(1, Ordering::Relaxed);
active.spawn(verify_one(
repository.clone(),
server.clone(),
request.info_hash,
config.max_peer_attempts,
stats.clone(),
permit,
));
}
}
}
}
while let Some(result) = active.join_next().await {
result.map_err(|error| error.to_string())??;
}
Ok(())
}
async fn verify_one(
repository: Arc<RocksTorrentRepository>,
server: DHTServer,
info_hash: InfoHash,
max_peer_attempts: usize,
stats: VerificationStats,
_permit: DiskWritePermit,
) -> Result<(), String> {
let peer_repository = repository.clone();
let stored_peers = tokio::task::spawn_blocking(move || peer_repository.get(info_hash))
.await
.map_err(|error| error.to_string())?
.map_err(|error| error.to_string())?
.map(|record| record.source_peers)
.unwrap_or_default();
let lookup = server.lookup_peers(*info_hash.as_bytes()).await;
let dht_peers = lookup.map(|result| result.peers).unwrap_or_default();
let mut unique = HashSet::with_capacity(dht_peers.len() + stored_peers.len());
let mut peers = Vec::with_capacity(dht_peers.len() + stored_peers.len());
for peer in dht_peers.into_iter().chain(
stored_peers
.iter()
.filter_map(|peer| peer.parse::<SocketAddr>().ok()),
) {
if unique.insert(peer) {
peers.push(peer);
}
}
stats
.inner
.peers_discovered
.fetch_add(peers.len() as u64, Ordering::Relaxed);
let mut handshakes = JoinSet::new();
for peer in peers.iter().copied().take(max_peer_attempts) {
let server = server.clone();
let hash = *info_hash.as_bytes();
handshakes.spawn(async move { server.verify_peer_handshake(hash, peer).await });
}
let mut reachable = 0_u32;
while let Some(result) = handshakes.join_next().await {
if result.map_err(|error| error.to_string())? {
reachable = reachable.saturating_add(1);
}
}
stats
.inner
.handshakes_succeeded
.fetch_add(u64::from(reachable), Ordering::Relaxed);
if reachable > 0 {
stats.inner.succeeded.fetch_add(1, Ordering::Relaxed);
} else {
stats.inner.failed.fetch_add(1, Ordering::Relaxed);
}
let result = VerificationResult {
verified_at: unix_timestamp(),
discovered_peers: peers.len().min(u32::MAX as usize) as u32,
reachable_peers: reachable,
};
let queue_len = tokio::task::spawn_blocking(move || {
repository.finish_verification(info_hash, result)?;
repository.verification_queue_len()
})
.await
.map_err(|error| error.to_string())?
.map_err(|error| error.to_string())?;
stats
.inner
.queue_depth
.store(queue_len as u64, Ordering::Relaxed);
Ok(())
}
fn unix_timestamp() -> u64 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap_or_default()
.as_secs()
}
+105
View File
@@ -0,0 +1,105 @@
// 负责从 crate 外部验证持久化索引过滤搜索和重启恢复的公开组合契约
#![cfg(feature = "rocksdb-storage")]
use dht_search::{
domain::{
ContentFilter, ContentFilterConfig, FileFilterRule, FileMatchField, FileMatchKind,
FileRuleAction, MetadataCandidate, TorrentFile, TorrentRecord,
},
search::SearchEngine,
storage::{RocksTorrentRepository, TorrentRepository, UpsertOutcome},
};
use std::sync::Arc;
use tempfile::TempDir;
#[test]
fn public_components_compose_into_a_restart_safe_search_flow() {
let directory = TempDir::new().unwrap();
let rocksdb = directory.path().join("rocksdb");
let tantivy = directory.path().join("tantivy");
let record = TorrentRecord::try_from(MetadataCandidate {
info_hash: "1212121212121212121212121212121212121212".into(),
name: "Public API 测试资源".into(),
total_size: 42,
files: vec![TorrentFile {
path: "docs/public-api.txt".into(),
size: 42,
}],
piece_length: 16_384,
source_peers: Vec::new(),
timestamp: 100,
})
.unwrap();
{
let repository = RocksTorrentRepository::open(&rocksdb).unwrap();
assert_eq!(
repository.upsert(record.clone()).unwrap(),
UpsertOutcome::Inserted
);
let search = SearchEngine::open(&tantivy).unwrap();
assert_eq!(search.index_pending(&repository, 100, 100).unwrap(), 1);
let page = search.search("public-api", 0, 10).unwrap();
assert_eq!(page.total, 1);
assert_eq!(page.hits[0].info_hash, record.info_hash.to_string());
}
let reopened = RocksTorrentRepository::open(&rocksdb).unwrap();
assert_eq!(reopened.get(record.info_hash).unwrap(), Some(record));
}
#[test]
fn content_filter_excludes_padding_from_search_and_visible_details() {
let directory = TempDir::new().unwrap();
let filter = Arc::new(
ContentFilter::compile(ContentFilterConfig {
version: 1,
file_rules: vec![FileFilterRule {
id: "bitcomet-padding".into(),
enabled: true,
field: FileMatchField::FileName,
match_kind: FileMatchKind::Prefix,
value: "_____padding_file_".into(),
case_sensitive: false,
action: FileRuleAction::Hide,
reason: "测试".into(),
}],
})
.unwrap(),
);
let repository = RocksTorrentRepository::open_with_rules(
directory.path().join("rocksdb"),
dht_search::domain::MetadataLimits::default().rule_id(),
filter,
)
.unwrap();
let record = TorrentRecord::try_from(MetadataCandidate {
info_hash: "3434343434343434343434343434343434343434".into(),
name: "Filtered Movie".into(),
total_size: 142,
files: vec![
TorrentFile {
path: "movie.mkv".into(),
size: 42,
},
TorrentFile {
path: "_____padding_file_1_请升级____".into(),
size: 100,
},
],
piece_length: 16_384,
source_peers: Vec::new(),
timestamp: 100,
})
.unwrap();
repository.upsert(record.clone()).unwrap();
let search = SearchEngine::open(directory.path().join("tantivy")).unwrap();
assert_eq!(search.index_pending(&repository, 100, 100).unwrap(), 1);
assert_eq!(search.search("padding_file", 0, 10).unwrap().total, 0);
assert_eq!(search.search("*.mkv", 0, 10).unwrap().total, 1);
let visible = repository.get_visible(record.info_hash).unwrap().unwrap();
assert_eq!(visible.files.len(), 1);
assert_eq!(visible.total_size, 42);
}
+24
View File
@@ -0,0 +1,24 @@
# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
pnpm-debug.log*
lerna-debug.log*
node_modules
dist
dist-ssr
*.local
# Editor directories and files
.vscode/*
!.vscode/extensions.json
.idea
.DS_Store
*.suo
*.ntvs*
*.njsproj
*.sln
*.sw?
+75
View File
@@ -0,0 +1,75 @@
# DHT Search Web
基于 Bun Vue 3 TypeScript Vite Tailwind CSS 和 shadcn-vue 的本地搜索界面
## 开发
先在仓库根目录启动后端服务
```powershell
$env:LIBCLANG_PATH = "D:\tools\dht\.tools\libclang\clang\native"
cargo run -p dht-search --bin dht-search -- --config dht-search.toml
```
然后打开另一个终端启动前端开发服务
```powershell
cd src/web
bun install
bun run dev
```
打开 `http://127.0.0.1:5173`
Vite 会将搜索详情状态诊断和配置接口代理到 `http://127.0.0.1:8080` 可以通过 `DHT_API_TARGET` 环境变量覆盖目标
## 检查和构建
```powershell
bun run check
```
生产静态资源输出到 `src/web/dist`
## 生产运行
在启动 Rust 服务前构建一次 Web 资源
```powershell
cd src/web
bun install --frozen-lockfile
bun run build
cd ../..
cargo run --release -p dht-search --bin dht-search -- --config dht-search.toml
```
打开 `http://127.0.0.1:8080`
Axum 根据配置中的 `http.web_dir` 提供静态资源和单页回退 不需要单独运行 Vite
## 已实现功能
- 关键词 文件名片段和精确 infohash 搜索
- 自动识别通配符的普通关键词搜索和名称 文件路径正则搜索
- 相关度 时间 热度 大小和发现次数排序
- 有上限的结果分页和 URL 查询恢复
- 基于 reka-ui 的搜索结果数字分页和浏览器持久化每页数量选择
- 文件列表 热度 可用性和收录时间详情
- 基于 reka-ui 的数字分页文件列表和浏览器持久化每页数量选择
- 相同内容的不同 infohash 变体展示
- 磁力链接打开和复制
- DHT 采集 索引 持久化和验证运行状态
- 每秒自动刷新的运行状态和点击外部关闭
- 独立的进程 RocksDB Tantivy DHT 和队列诊断页面
- 使用懒加载 ECharts 展示内存存储压力 Metadata 吞吐以及 HTTP 请求错误趋势
- 按职责分组的完整配置查看编辑校验保存和重启提示
- 搜索诊断配置使用互不冲突的单页路由
- 加载 空结果 接口错误 重试和移动端适配
## 添加组件
当前 shadcn-vue registry 与最新版 CLI 的样式列表暂时不一致 项目使用已验证的 CLI 版本
```powershell
bunx shadcn-vue@2.4.3 add button -y
```
+322
View File
@@ -0,0 +1,322 @@
{
"lockfileVersion": 1,
"configVersion": 1,
"workspaces": {
"": {
"name": "dht-search-web",
"dependencies": {
"@lucide/vue": "^1.29.0",
"class-variance-authority": "^0.7.1",
"clsx": "^2.1.1",
"echarts": "^6.1.0",
"reka-ui": "^2.10.1",
"tailwind-merge": "^3.6.0",
"vue": "^3.5.40",
"vue-echarts": "^8.1.0",
"vue-router": "^4.6.4",
},
"devDependencies": {
"@tailwindcss/vite": "^4.3.3",
"@types/node": "^24.13.3",
"@vitejs/plugin-vue": "^6.0.8",
"@vue/tsconfig": "^0.9.1",
"tailwindcss": "^4.3.3",
"tw-animate-css": "^1.4.0",
"typescript": "~6.0.2",
"vite": "^8.2.0",
"vue-tsc": "^3.3.8",
},
},
},
"packages": {
"@babel/helper-string-parser": ["@babel/helper-string-parser@7.29.7", "https://registry.npmmirror.com/@babel/helper-string-parser/-/helper-string-parser-7.29.7.tgz", {}, "sha512-Pb5ijPrZ89GDH8223L4UP8i6QApWxs04RbPQJTeWDV0/keR2E36MeKnyr6LYmUUvqRRI+Iv87SuF1W6ErINzYw=="],
"@babel/helper-validator-identifier": ["@babel/helper-validator-identifier@7.29.7", "https://registry.npmmirror.com/@babel/helper-validator-identifier/-/helper-validator-identifier-7.29.7.tgz", {}, "sha512-qehxGkRj55h/ff8EMaJ+cYhyaKlHIxqYDn682wQD7RNp9UujOQsHog2uS0r2vzr4pW+sXf90NeeayjcNaX3fFg=="],
"@babel/parser": ["@babel/parser@7.29.8", "https://registry.npmmirror.com/@babel/parser/-/parser-7.29.8.tgz", { "dependencies": { "@babel/types": "^7.29.8" }, "bin": "./bin/babel-parser.js" }, "sha512-E8lTAYNB1KW+FH+VGJuZM1ioAx2E6oVlvQFRrf5P8ZZmsiJXYAD9vTFV7yyEURNzgh1dFqMZuO6tUwcARbqFCA=="],
"@babel/types": ["@babel/types@7.29.8", "https://registry.npmmirror.com/@babel/types/-/types-7.29.8.tgz", { "dependencies": { "@babel/helper-string-parser": "^7.29.7", "@babel/helper-validator-identifier": "^7.29.7" } }, "sha512-Vj1jF3cPfxg7OAfoI7QnVKLoILlm2JF9pnVHrX8qx7AHMiYWT+NDAA7jChlNgRS4WTLc/fD1lXLmPixluj+3Gg=="],
"@floating-ui/core": ["@floating-ui/core@1.8.0", "https://registry.npmmirror.com/@floating-ui/core/-/core-1.8.0.tgz", { "dependencies": { "@floating-ui/utils": "^0.2.12" } }, "sha512-0CIZ5itps/8x7BG8dEIhs53BvCUH2PCoogtakwRTut+Arm58sJooJ0AuZhLw2HJYIR5cMLNPBSS728sPho2khQ=="],
"@floating-ui/dom": ["@floating-ui/dom@1.8.0", "https://registry.npmmirror.com/@floating-ui/dom/-/dom-1.8.0.tgz", { "dependencies": { "@floating-ui/core": "^1.8.0", "@floating-ui/utils": "^0.2.12" } }, "sha512-yXSrzeHZBTZadLOlfyhCkJHNeLJnHRnRInwdZ40L7ZiaAtrBwoYlsDrX3v5zB1Utk7CLfzcOVnVVWoXEky7Ceg=="],
"@floating-ui/utils": ["@floating-ui/utils@0.2.12", "https://registry.npmmirror.com/@floating-ui/utils/-/utils-0.2.12.tgz", {}, "sha512-HpCo8tmWzLVad5s2d19EhAz5zqrrQ6s69qd6moPMQvkOuSwDT1YgRfWSVuc4ennqrgv3OHppiOGMQ7oC13yIww=="],
"@floating-ui/vue": ["@floating-ui/vue@1.1.11", "https://registry.npmmirror.com/@floating-ui/vue/-/vue-1.1.11.tgz", { "dependencies": { "@floating-ui/dom": "^1.7.6", "@floating-ui/utils": "^0.2.11", "vue-demi": ">=0.13.0" } }, "sha512-HzHKCNVxnGS35r9fCHBc3+uCnjw9IWIlCPL683cGgM9Kgj2BiAl8x1mS7vtvP6F9S/e/q4O6MApwSHj8hNLGfw=="],
"@internationalized/date": ["@internationalized/date@3.12.3", "https://registry.npmmirror.com/@internationalized/date/-/date-3.12.3.tgz", { "dependencies": { "@swc/helpers": "^0.5.0" } }, "sha512-fuLX+3ZKLsxI73y8b01EG/WjHb6gE6weCqlfawPO27kBWGMh9G1yH6Csv1uU7/cac9H2GHmOMt6CjmuQ1aia4Q=="],
"@internationalized/number": ["@internationalized/number@3.6.7", "https://registry.npmmirror.com/@internationalized/number/-/number-3.6.7.tgz", { "dependencies": { "@swc/helpers": "^0.5.0" } }, "sha512-3ji1fcrT+FPAK86UqEhB/psHixYo6niWPJtt7+qRaYFynt/BaJG8GhAPimtWUpEiVSTq8ZM8L5psMxGquiB/Vg=="],
"@jridgewell/gen-mapping": ["@jridgewell/gen-mapping@0.3.13", "https://registry.npmmirror.com/@jridgewell/gen-mapping/-/gen-mapping-0.3.13.tgz", { "dependencies": { "@jridgewell/sourcemap-codec": "^1.5.0", "@jridgewell/trace-mapping": "^0.3.24" } }, "sha512-2kkt/7niJ6MgEPxF0bYdQ6etZaA+fQvDcLKckhy1yIQOzaoKjBBjSj63/aLVjYE3qhRt5dvM+uUyfCg6UKCBbA=="],
"@jridgewell/remapping": ["@jridgewell/remapping@2.3.5", "https://registry.npmmirror.com/@jridgewell/remapping/-/remapping-2.3.5.tgz", { "dependencies": { "@jridgewell/gen-mapping": "^0.3.5", "@jridgewell/trace-mapping": "^0.3.24" } }, "sha512-LI9u/+laYG4Ds1TDKSJW2YPrIlcVYOwi2fUC6xB43lueCjgxV4lffOCZCtYFiH6TNOX+tQKXx97T4IKHbhyHEQ=="],
"@jridgewell/resolve-uri": ["@jridgewell/resolve-uri@3.1.2", "https://registry.npmmirror.com/@jridgewell/resolve-uri/-/resolve-uri-3.1.2.tgz", {}, "sha512-bRISgCIjP20/tbWSPWMEi54QVPRZExkuD9lJL+UIxUKtwVJA8wW1Trb1jMs1RFXo1CBTNZ/5hpC9QvmKWdopKw=="],
"@jridgewell/sourcemap-codec": ["@jridgewell/sourcemap-codec@1.5.5", "https://registry.npmmirror.com/@jridgewell/sourcemap-codec/-/sourcemap-codec-1.5.5.tgz", {}, "sha512-cYQ9310grqxueWbl+WuIUIaiUaDcj7WOq5fVhEljNVgRfOUhY9fy2zTvfoqWsnebh8Sl70VScFbICvJnLKB0Og=="],
"@jridgewell/trace-mapping": ["@jridgewell/trace-mapping@0.3.31", "https://registry.npmmirror.com/@jridgewell/trace-mapping/-/trace-mapping-0.3.31.tgz", { "dependencies": { "@jridgewell/resolve-uri": "^3.1.0", "@jridgewell/sourcemap-codec": "^1.4.14" } }, "sha512-zzNR+SdQSDJzc8joaeP8QQoCQr8NuYx2dIIytl1QeBEZHJ9uW6hebsrYgbz8hJwUQao3TWCMtmfV8Nu1twOLAw=="],
"@lucide/vue": ["@lucide/vue@1.29.0", "https://registry.npmmirror.com/@lucide/vue/-/vue-1.29.0.tgz", { "peerDependencies": { "vue": ">=3.0.1" } }, "sha512-UnbPPAdxWGqABrjcQJZFqawSgZTBmeKOrNFPI9sqKaDkgLoEbYyx+ivIdiYRR1YoYhWWg7dCdvjtCwK6zBxWDQ=="],
"@oxc-project/types": ["@oxc-project/types@0.143.0", "https://registry.npmmirror.com/@oxc-project/types/-/types-0.143.0.tgz", {}, "sha512-u6JZdLBTLotrNC9Vd6vPssINdzcCzleKAH6EJKImQb7GtYvX5keN2dxkoK44stCc4tffE6QQRtZTXVSzsLUlWA=="],
"@rolldown/binding-android-arm64": ["@rolldown/binding-android-arm64@1.2.3", "https://registry.npmmirror.com/@rolldown/binding-android-arm64/-/binding-android-arm64-1.2.3.tgz", { "os": "android", "cpu": "arm64" }, "sha512-zrJtHDcaZJ1Fp7xf4hNl+7seH9Cn/N5TwLYkhgXREtBwAd/jaqW3uqeHxpDugJLVICWg4eW44kOQEGJ1r6jCGw=="],
"@rolldown/binding-darwin-arm64": ["@rolldown/binding-darwin-arm64@1.2.3", "https://registry.npmmirror.com/@rolldown/binding-darwin-arm64/-/binding-darwin-arm64-1.2.3.tgz", { "os": "darwin", "cpu": "arm64" }, "sha512-ieIiibVCp0tX7TLu2cafoNPv8wJyYi01ekXpbf8q2j7F4rGAhhXb/eQh7ge9DRBY78GwmRQtvjZDux7EDbA8kA=="],
"@rolldown/binding-darwin-x64": ["@rolldown/binding-darwin-x64@1.2.3", "https://registry.npmmirror.com/@rolldown/binding-darwin-x64/-/binding-darwin-x64-1.2.3.tgz", { "os": "darwin", "cpu": "x64" }, "sha512-Zh9tCon19eDXJoihx0rqKhMUlMYqzwj3aPsSuHmI4RWZh62dWUL+DJN4C5YQya5TcQBJU/Fe8+rY0jhXTQITqA=="],
"@rolldown/binding-freebsd-x64": ["@rolldown/binding-freebsd-x64@1.2.3", "https://registry.npmmirror.com/@rolldown/binding-freebsd-x64/-/binding-freebsd-x64-1.2.3.tgz", { "os": "freebsd", "cpu": "x64" }, "sha512-nGbJWewA1wrXXZiQhjAT5rhibGfns5ZNkDVqxsO6zJ3f3YvpoDNNmGMSbbhLuXKjNScaBJVOAboztAWVespQMg=="],
"@rolldown/binding-linux-arm-gnueabihf": ["@rolldown/binding-linux-arm-gnueabihf@1.2.3", "https://registry.npmmirror.com/@rolldown/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-1.2.3.tgz", { "os": "linux", "cpu": "arm" }, "sha512-QNniJr5Kml0kDEB98jiDOJjXNroxIIi0IXIbdYzY26Xt1pVbeP62+KnoIZLwirOymX/0jDk/2gI/bNUv7A7OIw=="],
"@rolldown/binding-linux-arm64-gnu": ["@rolldown/binding-linux-arm64-gnu@1.2.3", "https://registry.npmmirror.com/@rolldown/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-1.2.3.tgz", { "os": "linux", "cpu": "arm64" }, "sha512-TkqEAcmmvH3I/q4114NB4RVt6241Dao48pF45uLcFGrwAaIn0iITgTAKP/dLjbN0R4buJjGb91+UHSoFmpgIWw=="],
"@rolldown/binding-linux-arm64-musl": ["@rolldown/binding-linux-arm64-musl@1.2.3", "https://registry.npmmirror.com/@rolldown/binding-linux-arm64-musl/-/binding-linux-arm64-musl-1.2.3.tgz", { "os": "linux", "cpu": "arm64" }, "sha512-NHqjnxpsndf4MPymxteFAWHHfkTL8HjWh1KB7z23ofZ6QO2euONuxDXjat69dKZRALnGypg8k8SsK8vZJoXv1Q=="],
"@rolldown/binding-linux-ppc64-gnu": ["@rolldown/binding-linux-ppc64-gnu@1.2.3", "https://registry.npmmirror.com/@rolldown/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-1.2.3.tgz", { "os": "linux", "cpu": "ppc64" }, "sha512-6tbrbwfz5GB9DQ4Jwo6hy9v+vR31xZlvzZ6n5Xut6Hhx5PvrA9q/HsK8KMaYQp063iqZGXwNvZtYNLD7EM/x0w=="],
"@rolldown/binding-linux-s390x-gnu": ["@rolldown/binding-linux-s390x-gnu@1.2.3", "https://registry.npmmirror.com/@rolldown/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-1.2.3.tgz", { "os": "linux", "cpu": "s390x" }, "sha512-oyuXxXmoZHjXC917IAPFAAv4wWAa0cM9afk8nx1+9/jNNOX1uPf8yDA6p7G0RypOfw/X0PQt5IfoquY1um+zSg=="],
"@rolldown/binding-linux-x64-gnu": ["@rolldown/binding-linux-x64-gnu@1.2.3", "https://registry.npmmirror.com/@rolldown/binding-linux-x64-gnu/-/binding-linux-x64-gnu-1.2.3.tgz", { "os": "linux", "cpu": "x64" }, "sha512-TytMwF2KVGqP2tgd0I1OY0PAv78dZRAYcF5ssDzjM34SUXCED3uXvSd5+lHoC0bTD6eEdFz7LdQNCO1y0oVk9w=="],
"@rolldown/binding-linux-x64-musl": ["@rolldown/binding-linux-x64-musl@1.2.3", "https://registry.npmmirror.com/@rolldown/binding-linux-x64-musl/-/binding-linux-x64-musl-1.2.3.tgz", { "os": "linux", "cpu": "x64" }, "sha512-/E9m3qstrJFVPoULV25mVQblSNExY2+kBsYe4sy0Tn0yOOgJ8wZbZt3KnRbF/XeU2Gl1STKUQnDNTqhIE5MD4A=="],
"@rolldown/binding-openharmony-arm64": ["@rolldown/binding-openharmony-arm64@1.2.3", "https://registry.npmmirror.com/@rolldown/binding-openharmony-arm64/-/binding-openharmony-arm64-1.2.3.tgz", { "os": "none", "cpu": "arm64" }, "sha512-Kr0OcsoQI816i6HOl3vFHpd1K0eZyh76zgfj4c1nTyaTsd5r2Mj1lwM4R90y/qaCfmTn9eHy0SKwi98eitRxug=="],
"@rolldown/binding-win32-arm64-msvc": ["@rolldown/binding-win32-arm64-msvc@1.2.3", "https://registry.npmmirror.com/@rolldown/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-1.2.3.tgz", { "os": "win32", "cpu": "arm64" }, "sha512-hOtMwTqnME+/gJcH/PCZ0wn0zPUjiWOgkHpxbSJpfGKMezHltx1S7/k1SitzVa7Ww2cqrDDaFbZEhcJZO8o+Jw=="],
"@rolldown/binding-win32-x64-msvc": ["@rolldown/binding-win32-x64-msvc@1.2.3", "https://registry.npmmirror.com/@rolldown/binding-win32-x64-msvc/-/binding-win32-x64-msvc-1.2.3.tgz", { "os": "win32", "cpu": "x64" }, "sha512-ekcqMMkI2PlhYnfzQnB/cEdYUVVJViWvoUyLrbzgDoi3Snfc1mVBwdnc306ufA5ejy8JSPjT2RlW1nQSjW7efg=="],
"@rolldown/pluginutils": ["@rolldown/pluginutils@1.0.1", "https://registry.npmmirror.com/@rolldown/pluginutils/-/pluginutils-1.0.1.tgz", {}, "sha512-2j9bGt5Jh8hj+vPtgzPtl72j0yRxHAyumoo6TNfAjsLB04UtpSvPbPcDcBMxz7n+9CYB0c1GxQFxYRg2jimqGw=="],
"@swc/helpers": ["@swc/helpers@0.5.23", "https://registry.npmmirror.com/@swc/helpers/-/helpers-0.5.23.tgz", { "dependencies": { "tslib": "^2.8.0" } }, "sha512-5lSsMOTXURePglDfvuAQUqkGek9Hg2kksOYay2m0+XR++b2NWYL/4sWyuvVBIs8oKnJaxkdi9whaL/sqN13afw=="],
"@tailwindcss/node": ["@tailwindcss/node@4.3.3", "https://registry.npmmirror.com/@tailwindcss/node/-/node-4.3.3.tgz", { "dependencies": { "@jridgewell/remapping": "^2.3.5", "enhanced-resolve": "^5.24.1", "jiti": "^2.7.0", "lightningcss": "1.32.0", "magic-string": "^0.30.21", "source-map-js": "^1.2.1", "tailwindcss": "4.3.3" } }, "sha512-/T8IKEsf9VTU6tLjgC7+sv2mOPtQxzE2jMw7u4Tt40Tx+QSZxpzh95/H6cMKoja9XuW7iMdLJYBB0o9G1CaAgg=="],
"@tailwindcss/oxide": ["@tailwindcss/oxide@4.3.3", "https://registry.npmmirror.com/@tailwindcss/oxide/-/oxide-4.3.3.tgz", { "optionalDependencies": { "@tailwindcss/oxide-android-arm64": "4.3.3", "@tailwindcss/oxide-darwin-arm64": "4.3.3", "@tailwindcss/oxide-darwin-x64": "4.3.3", "@tailwindcss/oxide-freebsd-x64": "4.3.3", "@tailwindcss/oxide-linux-arm-gnueabihf": "4.3.3", "@tailwindcss/oxide-linux-arm64-gnu": "4.3.3", "@tailwindcss/oxide-linux-arm64-musl": "4.3.3", "@tailwindcss/oxide-linux-x64-gnu": "4.3.3", "@tailwindcss/oxide-linux-x64-musl": "4.3.3", "@tailwindcss/oxide-wasm32-wasi": "4.3.3", "@tailwindcss/oxide-win32-arm64-msvc": "4.3.3", "@tailwindcss/oxide-win32-x64-msvc": "4.3.3" } }, "sha512-krXjAikiaFSPaK/FkAQT5UTx3VormQaiZ5hBFlJZ9UFQGB/rwg1MZIhHAG9smMQRTdyJxP6Qt5MwMtdyU5FWrA=="],
"@tailwindcss/oxide-android-arm64": ["@tailwindcss/oxide-android-arm64@4.3.3", "https://registry.npmmirror.com/@tailwindcss/oxide-android-arm64/-/oxide-android-arm64-4.3.3.tgz", { "os": "android", "cpu": "arm64" }, "sha512-Y85A2gmPSkl5Ve5qR86GL4HT509cFqQh1aes9p3sSkyTPwt0Pppf3GkwGe4JPACcRYjgJIEhQgM6dBClnr0NYw=="],
"@tailwindcss/oxide-darwin-arm64": ["@tailwindcss/oxide-darwin-arm64@4.3.3", "https://registry.npmmirror.com/@tailwindcss/oxide-darwin-arm64/-/oxide-darwin-arm64-4.3.3.tgz", { "os": "darwin", "cpu": "arm64" }, "sha512-BiaWatpBcERQFDlOjRDpIVXuFK5PJez5SA4JMg6VYZdBYU+qKfV/vqjcIs+IYmtitf1xYQZTwXvU/8y4lfZUGw=="],
"@tailwindcss/oxide-darwin-x64": ["@tailwindcss/oxide-darwin-x64@4.3.3", "https://registry.npmmirror.com/@tailwindcss/oxide-darwin-x64/-/oxide-darwin-x64-4.3.3.tgz", { "os": "darwin", "cpu": "x64" }, "sha512-fAeUqfV5ndhxRwai8cXGzdLvul9utWOmeTkv69unv4ZXixjn61Z+p9lCWdwOwA3TYboG3BwdVuN/RDjhBRl0mw=="],
"@tailwindcss/oxide-freebsd-x64": ["@tailwindcss/oxide-freebsd-x64@4.3.3", "https://registry.npmmirror.com/@tailwindcss/oxide-freebsd-x64/-/oxide-freebsd-x64-4.3.3.tgz", { "os": "freebsd", "cpu": "x64" }, "sha512-iyf5bV6+wnAlflVeEy7R25dupxTNECZN5QMI0qNT6eT+EgaGdZcKhGkr5SdoaWiLJ3spLqIY9VCeSGrwmtg4kw=="],
"@tailwindcss/oxide-linux-arm-gnueabihf": ["@tailwindcss/oxide-linux-arm-gnueabihf@4.3.3", "https://registry.npmmirror.com/@tailwindcss/oxide-linux-arm-gnueabihf/-/oxide-linux-arm-gnueabihf-4.3.3.tgz", { "os": "linux", "cpu": "arm" }, "sha512-aAYUprJAJQWWbRrPvtjdroZ56Md+JM8pMiopS6xGEwDfLhqj+2ver2p4nU4Mb3CRqcMmNBjo8KkUgcxhkzVQGQ=="],
"@tailwindcss/oxide-linux-arm64-gnu": ["@tailwindcss/oxide-linux-arm64-gnu@4.3.3", "https://registry.npmmirror.com/@tailwindcss/oxide-linux-arm64-gnu/-/oxide-linux-arm64-gnu-4.3.3.tgz", { "os": "linux", "cpu": "arm64" }, "sha512-nDxldcEENOxZRzC2uu9jrutZdAAQtb+8WWDCSnWL1zvBk1+FN+x6MtDViPB5AJMfttVCUhehGWus3XBPgatM/w=="],
"@tailwindcss/oxide-linux-arm64-musl": ["@tailwindcss/oxide-linux-arm64-musl@4.3.3", "https://registry.npmmirror.com/@tailwindcss/oxide-linux-arm64-musl/-/oxide-linux-arm64-musl-4.3.3.tgz", { "os": "linux", "cpu": "arm64" }, "sha512-Md44bD6veX/PC5iyF8cDVnw4HBIANZepRZZ7a8DQOvkfo5WUBwcp6iAuCUz23u+4SUkhJlD3eL7hNdW8ezd/kA=="],
"@tailwindcss/oxide-linux-x64-gnu": ["@tailwindcss/oxide-linux-x64-gnu@4.3.3", "https://registry.npmmirror.com/@tailwindcss/oxide-linux-x64-gnu/-/oxide-linux-x64-gnu-4.3.3.tgz", { "os": "linux", "cpu": "x64" }, "sha512-tx7us1muwOKAKWao2v/GaafFeQboE6aj88vC6ziN2NCGcRm8gWUhwjzg+YdVB1e4boAtdtma4L43onunI6NS4w=="],
"@tailwindcss/oxide-linux-x64-musl": ["@tailwindcss/oxide-linux-x64-musl@4.3.3", "https://registry.npmmirror.com/@tailwindcss/oxide-linux-x64-musl/-/oxide-linux-x64-musl-4.3.3.tgz", { "os": "linux", "cpu": "x64" }, "sha512-SJxX60smvHgasZoBy11dX6YRjXJFovwWBoedhbQPOBzgFWBHGB+TVPWB9BxzR7TTxU8FQZAI2AyiNCMzFm8Img=="],
"@tailwindcss/oxide-wasm32-wasi": ["@tailwindcss/oxide-wasm32-wasi@4.3.3", "https://registry.npmmirror.com/@tailwindcss/oxide-wasm32-wasi/-/oxide-wasm32-wasi-4.3.3.tgz", { "dependencies": { "@emnapi/core": "^1.11.1", "@emnapi/runtime": "^1.11.1", "@emnapi/wasi-threads": "^1.2.2", "@napi-rs/wasm-runtime": "^1.1.4", "@tybys/wasm-util": "^0.10.2", "tslib": "^2.8.1" }, "cpu": "none" }, "sha512-jx1+rPhY/5Ympkktd656HBWEBLxP7dH06losBLjjf5vgCODXvi9KhtftWcMIwTFIDqBr7cRnQkdLnAG+IOlGvQ=="],
"@tailwindcss/oxide-win32-arm64-msvc": ["@tailwindcss/oxide-win32-arm64-msvc@4.3.3", "https://registry.npmmirror.com/@tailwindcss/oxide-win32-arm64-msvc/-/oxide-win32-arm64-msvc-4.3.3.tgz", { "os": "win32", "cpu": "arm64" }, "sha512-3rc292Ca2ceK6Ulcc/bAVnTs/3nDtoPhyEKlgPv+yQJQi/JS/AMJlqzxvlDacL1nekbrcf6bTqp/jV4qgnPxNQ=="],
"@tailwindcss/oxide-win32-x64-msvc": ["@tailwindcss/oxide-win32-x64-msvc@4.3.3", "https://registry.npmmirror.com/@tailwindcss/oxide-win32-x64-msvc/-/oxide-win32-x64-msvc-4.3.3.tgz", { "os": "win32", "cpu": "x64" }, "sha512-yJ0pwIVc/nYeGoV02WtsN8KYyLQv7kyI2wDnkezyJlGGjkd4QLwDGAwl47YpPJeuI0M0ObaXGSPjvWDPeTPggw=="],
"@tailwindcss/vite": ["@tailwindcss/vite@4.3.3", "https://registry.npmmirror.com/@tailwindcss/vite/-/vite-4.3.3.tgz", { "dependencies": { "@tailwindcss/node": "4.3.3", "@tailwindcss/oxide": "4.3.3", "tailwindcss": "4.3.3" }, "peerDependencies": { "vite": "^5.2.0 || ^6 || ^7 || ^8" } }, "sha512-yYU8cogLeSh/ms2jh8Fj7jaba/EWa7Ja6GoUqYZaraEuCI5YS6ms6ObZgjjedm+jm6XZjdNRWBpPP6Z86oOxcw=="],
"@tanstack/virtual-core": ["@tanstack/virtual-core@3.17.7", "https://registry.npmmirror.com/@tanstack/virtual-core/-/virtual-core-3.17.7.tgz", {}, "sha512-bp+v10y65sp2H7WpWfIMyxTNfl8ZVfxFTLRjPIFRryi6FV/J33z4IS53WO4pTk36KlvJ4iLiQz+oaydDC1xbcA=="],
"@tanstack/vue-virtual": ["@tanstack/vue-virtual@3.13.35", "https://registry.npmmirror.com/@tanstack/vue-virtual/-/vue-virtual-3.13.35.tgz", { "dependencies": { "@tanstack/virtual-core": "3.17.7" }, "peerDependencies": { "vue": "^2.7.0 || ^3.0.0" } }, "sha512-lOfSPvgPdlaH6Qy+CyIc3XpycitaSQ9GECndGpTuDiu+uDA1am+90yWXwzDSd/20ZM196ggWJLS+Qb6WjVd/OA=="],
"@types/node": ["@types/node@24.13.3", "https://registry.npmmirror.com/@types/node/-/node-24.13.3.tgz", { "dependencies": { "undici-types": "~7.18.0" } }, "sha512-Dh8vAsV36ig5wa9OX4pXvMc9D3Veibfw2wix0CUwYODLD8nkj9UsLjASr49nPg+2eKzxhBV+v7L8pXvT4e639Q=="],
"@types/web-bluetooth": ["@types/web-bluetooth@0.0.21", "https://registry.npmmirror.com/@types/web-bluetooth/-/web-bluetooth-0.0.21.tgz", {}, "sha512-oIQLCGWtcFZy2JW77j9k8nHzAOpqMHLQejDA48XXMWH6tjCQHz5RCFz1bzsmROyL6PUm+LLnUiI4BCn221inxA=="],
"@vitejs/plugin-vue": ["@vitejs/plugin-vue@6.0.8", "https://registry.npmmirror.com/@vitejs/plugin-vue/-/plugin-vue-6.0.8.tgz", { "dependencies": { "@rolldown/pluginutils": "^1.0.1" }, "peerDependencies": { "vite": "^5.0.0 || ^6.0.0 || ^7.0.0 || ^8.0.0", "vue": "^3.2.25" } }, "sha512-0ZjgOg7oO6farnNGup7yvoM/YXZV84OZxHAwtflItNa/6zzQyVb5LNxyea3FEKEX2XlagIKzrlH7wwxkKgtiew=="],
"@volar/language-core": ["@volar/language-core@2.4.28", "https://registry.npmmirror.com/@volar/language-core/-/language-core-2.4.28.tgz", { "dependencies": { "@volar/source-map": "2.4.28" } }, "sha512-w4qhIJ8ZSitgLAkVay6AbcnC7gP3glYM3fYwKV3srj8m494E3xtrCv6E+bWviiK/8hs6e6t1ij1s2Endql7vzQ=="],
"@volar/source-map": ["@volar/source-map@2.4.28", "https://registry.npmmirror.com/@volar/source-map/-/source-map-2.4.28.tgz", {}, "sha512-yX2BDBqJkRXfKw8my8VarTyjv48QwxdJtvRgUpNE5erCsgEUdI2DsLbpa+rOQVAJYshY99szEcRDmyHbF10ggQ=="],
"@volar/typescript": ["@volar/typescript@2.4.28", "https://registry.npmmirror.com/@volar/typescript/-/typescript-2.4.28.tgz", { "dependencies": { "@volar/language-core": "2.4.28", "path-browserify": "^1.0.1", "vscode-uri": "^3.0.8" } }, "sha512-Ja6yvWrbis2QtN4ClAKreeUZPVYMARDYZl9LMEv1iQ1QdepB6wn0jTRxA9MftYmYa4DQ4k/DaSZpFPUfxl8giw=="],
"@vue/compiler-core": ["@vue/compiler-core@3.5.41", "https://registry.npmmirror.com/@vue/compiler-core/-/compiler-core-3.5.41.tgz", { "dependencies": { "@babel/parser": "^7.29.8", "@vue/shared": "3.5.41", "entities": "^7.0.1", "estree-walker": "^2.0.2", "source-map-js": "^1.2.1" } }, "sha512-q0Xtv/F9w2YO/7htQhtiL+Ev2WCJbe5N2hc+XfgyKkEKqWpSxknmT8QOuGdEKNdjPq0c3F7rNpFkTo3Kfrm7pg=="],
"@vue/compiler-dom": ["@vue/compiler-dom@3.5.41", "https://registry.npmmirror.com/@vue/compiler-dom/-/compiler-dom-3.5.41.tgz", { "dependencies": { "@vue/compiler-core": "3.5.41", "@vue/shared": "3.5.41" } }, "sha512-oKacVfNglLvGjnS6BXOlGL7EyG2h8X03pqXCjzotRZUaXGjbrTJUnVAQjrCqUnS+lyu31nwQjZY/d817GmCnfw=="],
"@vue/compiler-sfc": ["@vue/compiler-sfc@3.5.41", "https://registry.npmmirror.com/@vue/compiler-sfc/-/compiler-sfc-3.5.41.tgz", { "dependencies": { "@babel/parser": "^7.29.8", "@vue/compiler-core": "3.5.41", "@vue/compiler-dom": "3.5.41", "@vue/compiler-ssr": "3.5.41", "@vue/shared": "3.5.41", "estree-walker": "^2.0.2", "magic-string": "^0.30.21", "postcss": "^8.5.19", "source-map-js": "^1.2.1" } }, "sha512-XJhip7R2wy6vX3knCxdZN4KracFaZUef58s1KYewqluedHIJaPIVfXoYT7MF1F8nCvv6k8bWWxDC8opMkg1VTQ=="],
"@vue/compiler-ssr": ["@vue/compiler-ssr@3.5.41", "https://registry.npmmirror.com/@vue/compiler-ssr/-/compiler-ssr-3.5.41.tgz", { "dependencies": { "@vue/compiler-dom": "3.5.41", "@vue/shared": "3.5.41" } }, "sha512-U3v5OejKEGqOI0Wy0+Sz7hGuIFZHA4LSXzrNM3IMIeDyJEBBfTpX26n3SDgToRpP2bLc9FfI2j/kSgcJ8Emq5A=="],
"@vue/devtools-api": ["@vue/devtools-api@6.6.4", "https://registry.npmmirror.com/@vue/devtools-api/-/devtools-api-6.6.4.tgz", {}, "sha512-sGhTPMuXqZ1rVOk32RylztWkfXTRhuS7vgAKv0zjqk8gbsHkJ7xfFf+jbySxt7tWObEJwyKaHMikV/WGDiQm8g=="],
"@vue/language-core": ["@vue/language-core@3.3.9", "https://registry.npmmirror.com/@vue/language-core/-/language-core-3.3.9.tgz", { "dependencies": { "@volar/language-core": "2.4.28", "@vue/compiler-dom": "^3.5.0", "@vue/shared": "^3.5.0", "alien-signals": "^3.2.1", "muggle-string": "^0.4.1", "path-browserify": "^1.0.1", "picomatch": "^4.0.4" } }, "sha512-in/68oAa4BCtVY6n/nkuhLIkV8DHYd2UivedJ6cMZ6UYtlq9jaoaSNUBHYCVO44z3nKg7MdE5OBoHKt5SxeBKQ=="],
"@vue/reactivity": ["@vue/reactivity@3.5.41", "https://registry.npmmirror.com/@vue/reactivity/-/reactivity-3.5.41.tgz", { "dependencies": { "@vue/shared": "3.5.41" } }, "sha512-rznsqKM0np0x18EjzF8x88MpEhdNsffbvFbckLL5+oUKz1BxAImEmO7J1ArRYSyo6aQaVoBDp7jEkT91OOxydA=="],
"@vue/runtime-core": ["@vue/runtime-core@3.5.41", "https://registry.npmmirror.com/@vue/runtime-core/-/runtime-core-3.5.41.tgz", { "dependencies": { "@vue/reactivity": "3.5.41", "@vue/shared": "3.5.41" } }, "sha512-Vcry58hiAKwGen9Z1jUZE0feFsNArPCMOImYI8el48A9Idf6DuQYD0U05zZIF2Iad1hGhPSvcbBbAOhNr55fhg=="],
"@vue/runtime-dom": ["@vue/runtime-dom@3.5.41", "https://registry.npmmirror.com/@vue/runtime-dom/-/runtime-dom-3.5.41.tgz", { "dependencies": { "@vue/reactivity": "3.5.41", "@vue/runtime-core": "3.5.41", "@vue/shared": "3.5.41", "csstype": "^3.2.3" } }, "sha512-3vVBahVBS9+U6cmXBLyb8nE6/yYo4J/CGI9eVFs3KiMc0YHuudwKyShTD65jtJy/L9PUUxNAFu4cj4LiJ0UFbw=="],
"@vue/server-renderer": ["@vue/server-renderer@3.5.41", "https://registry.npmmirror.com/@vue/server-renderer/-/server-renderer-3.5.41.tgz", { "dependencies": { "@vue/compiler-ssr": "3.5.41", "@vue/runtime-dom": "3.5.41", "@vue/shared": "3.5.41" } }, "sha512-n6hx/pNFfbD6SuyeuMVkvqox8bwf/ET9JlA/kAz/imw8sw++wkqKe2mHX5KutjPpbKE4Z56yTHszoOjGMI9igQ=="],
"@vue/shared": ["@vue/shared@3.5.41", "https://registry.npmmirror.com/@vue/shared/-/shared-3.5.41.tgz", {}, "sha512-IOnwSCma8j+9xJT6b8H0dEYidC80NsYmNMlZxRsukYcSoGaDBohog5hDxzeUXdFeGWFA++vWvxqOmrr96VlqMA=="],
"@vue/tsconfig": ["@vue/tsconfig@0.9.1", "https://registry.npmmirror.com/@vue/tsconfig/-/tsconfig-0.9.1.tgz", { "peerDependencies": { "typescript": ">= 5.8", "vue": "^3.4.0" }, "optionalPeers": ["typescript", "vue"] }, "sha512-buvjm+9NzLCJL29KY1j1991YYJ5e6275OiK+G4jtmfIb+z4POywbdm0wXusT9adVWqe0xqg70TbI7+mRx4uU9w=="],
"@vueuse/core": ["@vueuse/core@14.4.0", "https://registry.npmmirror.com/@vueuse/core/-/core-14.4.0.tgz", { "dependencies": { "@types/web-bluetooth": "^0.0.21", "@vueuse/metadata": "14.4.0", "@vueuse/shared": "14.4.0" }, "peerDependencies": { "vue": "^3.5.0" } }, "sha512-X4WHz1HlCzCBoYXesUkifzzWBAcZgXG8Fi5iNPQg/epdzOB3gu8Fawj3hvuwYR1nGcXGnvxwYYcUC/71++svtQ=="],
"@vueuse/metadata": ["@vueuse/metadata@14.4.0", "https://registry.npmmirror.com/@vueuse/metadata/-/metadata-14.4.0.tgz", {}, "sha512-swx/255R6JyHZFJhx845iz5CRWDZdCfvkZOpACWc5+c5WHcG24mv8gUT1WIdFQaHt6dq79rvILd9QnCWiyVm9g=="],
"@vueuse/shared": ["@vueuse/shared@14.4.0", "https://registry.npmmirror.com/@vueuse/shared/-/shared-14.4.0.tgz", { "peerDependencies": { "vue": "^3.5.0" } }, "sha512-JRgY90Sz8DDtPMsaDflvPMp9xYk69JZAmbuDvAquUVXKr2gEjqtzGNTTthLfckH0BzBqvnu31gb4a8TGLRe79g=="],
"alien-signals": ["alien-signals@3.2.1", "https://registry.npmmirror.com/alien-signals/-/alien-signals-3.2.1.tgz", {}, "sha512-I8FjmltrfnDFoZedi5CG8DghVYNhzb/Ijluz7tCSJH0xpd0484Kowhbb1XDYOxfJpU1p5wnM2X54dA+IfGyD1g=="],
"aria-hidden": ["aria-hidden@1.2.6", "https://registry.npmmirror.com/aria-hidden/-/aria-hidden-1.2.6.tgz", { "dependencies": { "tslib": "^2.0.0" } }, "sha512-ik3ZgC9dY/lYVVM++OISsaYDeg1tb0VtP5uL3ouh1koGOaUMDPpbFIei4JkFimWUFPn90sbMNMXQAIVOlnYKJA=="],
"class-variance-authority": ["class-variance-authority@0.7.1", "https://registry.npmmirror.com/class-variance-authority/-/class-variance-authority-0.7.1.tgz", { "dependencies": { "clsx": "^2.1.1" } }, "sha512-Ka+9Trutv7G8M6WT6SeiRWz792K5qEqIGEGzXKhAE6xOWAY6pPH8U+9IY3oCMv6kqTmLsv7Xh/2w2RigkePMsg=="],
"clsx": ["clsx@2.1.1", "https://registry.npmmirror.com/clsx/-/clsx-2.1.1.tgz", {}, "sha512-eYm0QWBtUrBWZWG0d386OGAw16Z995PiOVo2B7bjWSbHedGl5e0ZWaq65kOGgUSNesEIDkB9ISbTg/JK9dhCZA=="],
"csstype": ["csstype@3.2.3", "https://registry.npmmirror.com/csstype/-/csstype-3.2.3.tgz", {}, "sha512-z1HGKcYy2xA8AGQfwrn0PAy+PB7X/GSj3UVJW9qKyn43xWa+gl5nXmU4qqLMRzWVLFC8KusUX8T/0kCiOYpAIQ=="],
"defu": ["defu@6.1.7", "https://registry.npmmirror.com/defu/-/defu-6.1.7.tgz", {}, "sha512-7z22QmUWiQ/2d0KkdYmANbRUVABpZ9SNYyH5vx6PZ+nE5bcC0l7uFvEfHlyld/HcGBFTL536ClDt3DEcSlEJAQ=="],
"detect-libc": ["detect-libc@2.1.2", "https://registry.npmmirror.com/detect-libc/-/detect-libc-2.1.2.tgz", {}, "sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ=="],
"echarts": ["echarts@6.1.0", "https://registry.npmmirror.com/echarts/-/echarts-6.1.0.tgz", { "dependencies": { "tslib": "2.3.0", "zrender": "6.1.0" } }, "sha512-q0yaFPggC9FUdsWH4blavRWFmxdrIodbkoKNAjJudAI6CA9gNPxHtV2RcZNEepZVlk4yvBYkOkbk6HIVpIyHZA=="],
"enhanced-resolve": ["enhanced-resolve@5.24.5", "https://registry.npmmirror.com/enhanced-resolve/-/enhanced-resolve-5.24.5.tgz", { "dependencies": { "graceful-fs": "^4.2.4", "tapable": "^2.3.3" } }, "sha512-L1l8TNvomm6UVW5B253AGxQagSQr+vGwhMlrrfRS2qmhx46AMpMVJKQYLvWYbysTMY8VoicOvzHzoHMbyzB+4A=="],
"entities": ["entities@7.0.1", "https://registry.npmmirror.com/entities/-/entities-7.0.1.tgz", {}, "sha512-TWrgLOFUQTH994YUyl1yT4uyavY5nNB5muff+RtWaqNVCAK408b5ZnnbNAUEWLTCpum9w6arT70i1XdQ4UeOPA=="],
"estree-walker": ["estree-walker@2.0.2", "https://registry.npmmirror.com/estree-walker/-/estree-walker-2.0.2.tgz", {}, "sha512-Rfkk/Mp/DL7JVje3u18FxFujQlTNR2q6QfMSMB7AvCBx91NGj/ba3kCfza0f6dVDbw7YlRf/nDrn7pQrCCyQ/w=="],
"fdir": ["fdir@6.5.0", "https://registry.npmmirror.com/fdir/-/fdir-6.5.0.tgz", { "peerDependencies": { "picomatch": "^3 || ^4" }, "optionalPeers": ["picomatch"] }, "sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg=="],
"fsevents": ["fsevents@2.3.3", "https://registry.npmmirror.com/fsevents/-/fsevents-2.3.3.tgz", { "os": "darwin" }, "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw=="],
"graceful-fs": ["graceful-fs@4.2.11", "https://registry.npmmirror.com/graceful-fs/-/graceful-fs-4.2.11.tgz", {}, "sha512-RbJ5/jmFcNNCcDV5o9eTnBLJ/HszWV0P73bc+Ff4nS/rJj+YaS6IGyiOL0VoBYX+l1Wrl3k63h/KrH+nhJ0XvQ=="],
"jiti": ["jiti@2.7.0", "https://registry.npmmirror.com/jiti/-/jiti-2.7.0.tgz", { "bin": { "jiti": "lib/jiti-cli.mjs" } }, "sha512-AC/7JofJvZGrrneWNaEnJeOLUx+JlGt7tNa0wZiRPT4MY1wmfKjt2+6O2p2uz2+skll8OZZmJMNqeke7kKbNgQ=="],
"lightningcss": ["lightningcss@1.33.0", "https://registry.npmmirror.com/lightningcss/-/lightningcss-1.33.0.tgz", { "dependencies": { "detect-libc": "^2.0.3" }, "optionalDependencies": { "lightningcss-android-arm64": "1.33.0", "lightningcss-darwin-arm64": "1.33.0", "lightningcss-darwin-x64": "1.33.0", "lightningcss-freebsd-x64": "1.33.0", "lightningcss-linux-arm-gnueabihf": "1.33.0", "lightningcss-linux-arm64-gnu": "1.33.0", "lightningcss-linux-arm64-musl": "1.33.0", "lightningcss-linux-x64-gnu": "1.33.0", "lightningcss-linux-x64-musl": "1.33.0", "lightningcss-win32-arm64-msvc": "1.33.0", "lightningcss-win32-x64-msvc": "1.33.0" } }, "sha512-WkUDrojuJs0xkgGf2udWxa3yGBRxPtxUkB79i6aCZLRgc7PM8fZe9TosfPDcvEpQZbuFASnHYmRLBLUbmLOIIA=="],
"lightningcss-android-arm64": ["lightningcss-android-arm64@1.33.0", "https://registry.npmmirror.com/lightningcss-android-arm64/-/lightningcss-android-arm64-1.33.0.tgz", { "os": "android", "cpu": "arm64" }, "sha512-gEpRTalKdosp4Bb8qWtc2iOgE5SeIHlpS1up9bFq2wAyYhl1UdTObYiHe98zEM9SQvSoqQZ1IQD0JNpg3Ml5pg=="],
"lightningcss-darwin-arm64": ["lightningcss-darwin-arm64@1.33.0", "https://registry.npmmirror.com/lightningcss-darwin-arm64/-/lightningcss-darwin-arm64-1.33.0.tgz", { "os": "darwin", "cpu": "arm64" }, "sha512-Sciaz8eenNTKn9b3t7+xr0ipTp9YxKQY4npwQ3mrRuL0BAVHBLyZxofhaKBAVtzmtRZ/zTyo0/to4B1uWG/Djg=="],
"lightningcss-darwin-x64": ["lightningcss-darwin-x64@1.33.0", "https://registry.npmmirror.com/lightningcss-darwin-x64/-/lightningcss-darwin-x64-1.33.0.tgz", { "os": "darwin", "cpu": "x64" }, "sha512-Z5UPAxzrjlWNNyGy6i65cJzzvgJ5D3T6wMvs+gWpY9d7qRhANrxqAp6LhxIgZhWEw18RfJTGcRxjuLIBr+m8XQ=="],
"lightningcss-freebsd-x64": ["lightningcss-freebsd-x64@1.33.0", "https://registry.npmmirror.com/lightningcss-freebsd-x64/-/lightningcss-freebsd-x64-1.33.0.tgz", { "os": "freebsd", "cpu": "x64" }, "sha512-QQM/Ti/hQajJwCY+RiWuCZ9sdtI/XQk7nDK5vC8kkdwixezOlDgvDx7+RT+QjK6FcFT4MpsuoBnHIo/O3StRRg=="],
"lightningcss-linux-arm-gnueabihf": ["lightningcss-linux-arm-gnueabihf@1.33.0", "https://registry.npmmirror.com/lightningcss-linux-arm-gnueabihf/-/lightningcss-linux-arm-gnueabihf-1.33.0.tgz", { "os": "linux", "cpu": "arm" }, "sha512-N7FVBe6iS24MlM6R/4RBTxGhQheZGs7tiQ9U32UtF75NzP5Q7xWPRqLBCKxlRQRk3rY1jCIPLzx7WzOhuUIRLQ=="],
"lightningcss-linux-arm64-gnu": ["lightningcss-linux-arm64-gnu@1.33.0", "https://registry.npmmirror.com/lightningcss-linux-arm64-gnu/-/lightningcss-linux-arm64-gnu-1.33.0.tgz", { "os": "linux", "cpu": "arm64" }, "sha512-j2v/itmy4HlNxlc6voKXYgBqNi0Ng2LShg4z7GufpEgs05P+2suBVyi9I6YHq5uoVFx9ETin3eCEhLVyXGQnKg=="],
"lightningcss-linux-arm64-musl": ["lightningcss-linux-arm64-musl@1.33.0", "https://registry.npmmirror.com/lightningcss-linux-arm64-musl/-/lightningcss-linux-arm64-musl-1.33.0.tgz", { "os": "linux", "cpu": "arm64" }, "sha512-yiO5ROMuYQgXbC60yjZU5CYSFZGKXL0HFATXt9mHJn1+zW55oCtMI9NfcVhYLMFDL7gV7oBPon/EmMMGg2OvtQ=="],
"lightningcss-linux-x64-gnu": ["lightningcss-linux-x64-gnu@1.33.0", "https://registry.npmmirror.com/lightningcss-linux-x64-gnu/-/lightningcss-linux-x64-gnu-1.33.0.tgz", { "os": "linux", "cpu": "x64" }, "sha512-ar+Ju7LmcN0Jo4FpL4hpFybwNG9/3A/Br5KW2n2jyODg3MEZXaDYADdemoNS+BDNfMgKvylJLj4S5tyRActuAg=="],
"lightningcss-linux-x64-musl": ["lightningcss-linux-x64-musl@1.33.0", "https://registry.npmmirror.com/lightningcss-linux-x64-musl/-/lightningcss-linux-x64-musl-1.33.0.tgz", { "os": "linux", "cpu": "x64" }, "sha512-RYiYbkokw0trfKqqzfF55lginwEPrD3OJDfTuJzFs1MK6iFnDenaz1fqLLtX4ITG3OktJQXOeTaw1awrBAlZPw=="],
"lightningcss-win32-arm64-msvc": ["lightningcss-win32-arm64-msvc@1.33.0", "https://registry.npmmirror.com/lightningcss-win32-arm64-msvc/-/lightningcss-win32-arm64-msvc-1.33.0.tgz", { "os": "win32", "cpu": "arm64" }, "sha512-1K+MPfLSFVpphzpdbfkhlWk6wBrTObBzS2T6db10PNOZgR9GoVsAWzwNyuhUYYbTp23j+4RrncfujZ4uAzXvwA=="],
"lightningcss-win32-x64-msvc": ["lightningcss-win32-x64-msvc@1.33.0", "https://registry.npmmirror.com/lightningcss-win32-x64-msvc/-/lightningcss-win32-x64-msvc-1.33.0.tgz", { "os": "win32", "cpu": "x64" }, "sha512-OlEICDx/Xl0FqSp4bry8zFnCvGpig3Gl4gCquvYwHuqJKEC1+n9NgDniFvqHGmMv1ZkqDJrDqKKSykTDX+ehuA=="],
"magic-string": ["magic-string@0.30.21", "https://registry.npmmirror.com/magic-string/-/magic-string-0.30.21.tgz", { "dependencies": { "@jridgewell/sourcemap-codec": "^1.5.5" } }, "sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ=="],
"muggle-string": ["muggle-string@0.4.1", "https://registry.npmmirror.com/muggle-string/-/muggle-string-0.4.1.tgz", {}, "sha512-VNTrAak/KhO2i8dqqnqnAHOa3cYBwXEZe9h+D5h/1ZqFSTEFHdM65lR7RoIqq3tBBYavsOXV84NoHXZ0AkPyqQ=="],
"nanoid": ["nanoid@3.3.17", "https://registry.npmmirror.com/nanoid/-/nanoid-3.3.17.tgz", { "bin": { "nanoid": "bin/nanoid.cjs" } }, "sha512-xQLf0A3HOMlgHq0n247/LRuAOYmB7dXJ/DvAxGvsSBij45XtBSmQycu+F8ODbHwns/XyFZagyL1+J0Offw1E0g=="],
"ohash": ["ohash@2.0.11", "https://registry.npmmirror.com/ohash/-/ohash-2.0.11.tgz", {}, "sha512-RdR9FQrFwNBNXAr4GixM8YaRZRJ5PUWbKYbE5eOsrwAjJW0q2REGcf79oYPsLyskQCZG1PLN+S/K1V00joZAoQ=="],
"path-browserify": ["path-browserify@1.0.1", "https://registry.npmmirror.com/path-browserify/-/path-browserify-1.0.1.tgz", {}, "sha512-b7uo2UCUOYZcnF/3ID0lulOJi/bafxa1xPe7ZPsammBSpjSWQkjNxlt635YGS2MiR9GjvuXCtz2emr3jbsz98g=="],
"picocolors": ["picocolors@1.1.1", "https://registry.npmmirror.com/picocolors/-/picocolors-1.1.1.tgz", {}, "sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA=="],
"picomatch": ["picomatch@4.0.5", "https://registry.npmmirror.com/picomatch/-/picomatch-4.0.5.tgz", {}, "sha512-RvwwcruNjI1ncT5xRakeyS9Lf8lcItv34KD+aif+VH9kduAyfYBipGh12274xtenIPZ119/R9BdTBa8gAwSh0A=="],
"postcss": ["postcss@8.5.26", "https://registry.npmmirror.com/postcss/-/postcss-8.5.26.tgz", { "dependencies": { "nanoid": "^3.3.17", "picocolors": "^1.1.1", "source-map-js": "^1.2.1" } }, "sha512-u82N74LFzG8ca+dD8puPnplTXoGH4fTPpVGuIbt36G3qvNlkvfD0lEAZSxaly3KX8TS/L1A1gsCEmvKmBcVbkQ=="],
"reka-ui": ["reka-ui@2.10.1", "https://registry.npmmirror.com/reka-ui/-/reka-ui-2.10.1.tgz", { "dependencies": { "@floating-ui/dom": "^1.6.13", "@floating-ui/vue": "^1.1.6", "@internationalized/date": "^3.5.0", "@internationalized/number": "^3.5.0", "@tanstack/vue-virtual": "^3.12.0", "@vueuse/core": "^14.1.0", "@vueuse/shared": "^14.1.0", "aria-hidden": "^1.2.4", "defu": "^6.1.5", "ohash": "^2.0.11" }, "peerDependencies": { "vue": ">= 3.4.0" } }, "sha512-drcOQ4rQtDYAcGCsyQBqQg8QQ+H3B+zDaMJU0h8KPEPMa7g9BHu3zcOi4OB39XJSWizceFoNO0Z9tctSGLOXqg=="],
"rolldown": ["rolldown@1.2.3", "https://registry.npmmirror.com/rolldown/-/rolldown-1.2.3.tgz", { "dependencies": { "@oxc-project/types": "=0.143.0", "@rolldown/pluginutils": "^1.0.0" }, "optionalDependencies": { "@rolldown/binding-android-arm64": "1.2.3", "@rolldown/binding-darwin-arm64": "1.2.3", "@rolldown/binding-darwin-x64": "1.2.3", "@rolldown/binding-freebsd-x64": "1.2.3", "@rolldown/binding-linux-arm-gnueabihf": "1.2.3", "@rolldown/binding-linux-arm64-gnu": "1.2.3", "@rolldown/binding-linux-arm64-musl": "1.2.3", "@rolldown/binding-linux-ppc64-gnu": "1.2.3", "@rolldown/binding-linux-s390x-gnu": "1.2.3", "@rolldown/binding-linux-x64-gnu": "1.2.3", "@rolldown/binding-linux-x64-musl": "1.2.3", "@rolldown/binding-openharmony-arm64": "1.2.3", "@rolldown/binding-win32-arm64-msvc": "1.2.3", "@rolldown/binding-win32-x64-msvc": "1.2.3" }, "bin": { "rolldown": "./bin/cli.mjs" } }, "sha512-rn9wpmxplLf7NLNyCk9FyWh3FM43DbY8jOzCdEPzH7uflhTftRbCEpqi6Ly2osgoU8OwObtmavMbWLaWy4LX7A=="],
"source-map-js": ["source-map-js@1.2.1", "https://registry.npmmirror.com/source-map-js/-/source-map-js-1.2.1.tgz", {}, "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA=="],
"tailwind-merge": ["tailwind-merge@3.6.0", "https://registry.npmmirror.com/tailwind-merge/-/tailwind-merge-3.6.0.tgz", {}, "sha512-uxL7qAVQriqRQPAyK3pj66VqskWqoZ37PW94jwOTwNfq/z9oyu1V+eqrZqtR2+fCiXdYOZe/Modt8GtvqNzu+w=="],
"tailwindcss": ["tailwindcss@4.3.3", "https://registry.npmmirror.com/tailwindcss/-/tailwindcss-4.3.3.tgz", {}, "sha512-gOhV3P7ufE62QDGg1zVaTgCR+EtPv92k2nIhVcVKcLmxT1sUBsQGhnZj175j+MqRt4zLF7ic+sCYjfhxMxj7YQ=="],
"tapable": ["tapable@2.3.3", "https://registry.npmmirror.com/tapable/-/tapable-2.3.3.tgz", {}, "sha512-uxc/zpqFg6x7C8vOE7lh6Lbda8eEL9zmVm/PLeTPBRhh1xCgdWaQ+J1CUieGpIfm2HdtsUpRv+HshiasBMcc6A=="],
"tinyglobby": ["tinyglobby@0.2.17", "https://registry.npmmirror.com/tinyglobby/-/tinyglobby-0.2.17.tgz", { "dependencies": { "fdir": "^6.5.0", "picomatch": "^4.0.4" } }, "sha512-wXR/dYpcqKmfWpEdZjiKJOwCNFndD0DMnrW/cYjVGttEkBfVgcLFHoNrlj47mjOVic9yyNu65alsgF4NQyTa2g=="],
"tslib": ["tslib@2.3.0", "https://registry.npmmirror.com/tslib/-/tslib-2.3.0.tgz", {}, "sha512-N82ooyxVNm6h1riLCoyS9e3fuJ3AMG2zIZs2Gd1ATcSFjSA23Q0fzjjZeh0jbJvWVDZ0cJT8yaNNaaXHzueNjg=="],
"tw-animate-css": ["tw-animate-css@1.4.0", "https://registry.npmmirror.com/tw-animate-css/-/tw-animate-css-1.4.0.tgz", {}, "sha512-7bziOlRqH0hJx80h/3mbicLW7o8qLsH5+RaLR2t+OHM3D0JlWGODQKQ4cxbK7WlvmUxpcj6Kgu6EKqjrGFe3QQ=="],
"typescript": ["typescript@6.0.3", "https://registry.npmmirror.com/typescript/-/typescript-6.0.3.tgz", { "bin": { "tsc": "bin/tsc", "tsserver": "bin/tsserver" } }, "sha512-y2TvuxSZPDyQakkFRPZHKFm+KKVqIisdg9/CZwm9ftvKXLP8NRWj38/ODjNbr43SsoXqNuAisEf1GdCxqWcdBw=="],
"undici-types": ["undici-types@7.18.2", "https://registry.npmmirror.com/undici-types/-/undici-types-7.18.2.tgz", {}, "sha512-AsuCzffGHJybSaRrmr5eHr81mwJU3kjw6M+uprWvCXiNeN9SOGwQ3Jn8jb8m3Z6izVgknn1R0FTCEAP2QrLY/w=="],
"vite": ["vite@8.2.1", "https://registry.npmmirror.com/vite/-/vite-8.2.1.tgz", { "dependencies": { "lightningcss": "^1.33.0", "picomatch": "^4.0.5", "postcss": "^8.5.25", "rolldown": "~1.2.1", "tinyglobby": "^0.2.17" }, "optionalDependencies": { "fsevents": "~2.3.3" }, "peerDependencies": { "@types/node": "^20.19.0 || >=22.12.0", "@vitejs/devtools": "^0.4.0", "esbuild": "^0.27.0 || ^0.28.0", "jiti": ">=1.21.0", "less": "^4.0.0", "sass": "^1.70.0", "sass-embedded": "^1.70.0", "stylus": ">=0.54.8", "sugarss": "^5.0.0", "terser": "^5.16.0", "tsx": "^4.8.1", "yaml": "^2.4.2" }, "optionalPeers": ["@types/node", "@vitejs/devtools", "esbuild", "jiti", "less", "sass", "sass-embedded", "stylus", "sugarss", "terser", "tsx", "yaml"], "bin": { "vite": "bin/vite.js" } }, "sha512-EU/eS7BH3XROHh2YnBefjM6DBKA6ZeMZEYQbj7NLWg5wHYlhB8B/Mayd5XsgWq+NFYccDOTemRpdETWR6Ka/lw=="],
"vscode-uri": ["vscode-uri@3.1.0", "https://registry.npmmirror.com/vscode-uri/-/vscode-uri-3.1.0.tgz", {}, "sha512-/BpdSx+yCQGnCvecbyXdxHDkuk55/G3xwnC0GqY4gmQ3j+A+g8kzzgB4Nk/SINjqn6+waqw3EgbVF2QKExkRxQ=="],
"vue": ["vue@3.5.41", "https://registry.npmmirror.com/vue/-/vue-3.5.41.tgz", { "dependencies": { "@vue/compiler-dom": "3.5.41", "@vue/compiler-sfc": "3.5.41", "@vue/runtime-dom": "3.5.41", "@vue/server-renderer": "3.5.41", "@vue/shared": "3.5.41" }, "peerDependencies": { "typescript": "*" }, "optionalPeers": ["typescript"] }, "sha512-2laE0p+aK+/AOPG/XL/WepOs/GlK755LJ1XECi9kDUrz1FKNw8rb2Xzlw9JS1rqEV55nb0ttsKxVlTCcd+R5cg=="],
"vue-demi": ["vue-demi@0.14.10", "https://registry.npmmirror.com/vue-demi/-/vue-demi-0.14.10.tgz", { "peerDependencies": { "@vue/composition-api": "^1.0.0-rc.1", "vue": "^3.0.0-0 || ^2.6.0" }, "optionalPeers": ["@vue/composition-api"], "bin": { "vue-demi-fix": "bin/vue-demi-fix.js", "vue-demi-switch": "bin/vue-demi-switch.js" } }, "sha512-nMZBOwuzabUO0nLgIcc6rycZEebF6eeUfaiQx9+WSk8e29IbLvPU9feI6tqW4kTo3hvoYAJkMh8n8D0fuISphg=="],
"vue-echarts": ["vue-echarts@8.1.0", "https://registry.npmmirror.com/vue-echarts/-/vue-echarts-8.1.0.tgz", { "peerDependencies": { "echarts": "^6.0.0", "vue": "^3.3.0" } }, "sha512-/uJVwijy3M2vIZ0NcPDgdZVqgboc6zZtC/vERfESau0eFpkHOIujj0sIiMiLP4kztQrckMFnDYWrid+gLI+IOg=="],
"vue-router": ["vue-router@4.6.4", "https://registry.npmmirror.com/vue-router/-/vue-router-4.6.4.tgz", { "dependencies": { "@vue/devtools-api": "^6.6.4" }, "peerDependencies": { "vue": "^3.5.0" } }, "sha512-Hz9q5sa33Yhduglwz6g9skT8OBPii+4bFn88w6J+J4MfEo4KRRpmiNG/hHHkdbRFlLBOqxN8y8gf2Fb0MTUgVg=="],
"vue-tsc": ["vue-tsc@3.3.9", "https://registry.npmmirror.com/vue-tsc/-/vue-tsc-3.3.9.tgz", { "dependencies": { "@volar/typescript": "2.4.28", "@vue/language-core": "3.3.9" }, "peerDependencies": { "typescript": ">=5.0.0" }, "bin": { "vue-tsc": "bin/vue-tsc.js" } }, "sha512-TS3Y1ux/IRoE8OCP2PpACAeOseuIs0UvWrcr7u+w3PmfY+SlCfEf8zjrBgnQksHUgLpthi5vHlffcQTQTdPBZA=="],
"zrender": ["zrender@6.1.0", "https://registry.npmmirror.com/zrender/-/zrender-6.1.0.tgz", { "dependencies": { "tslib": "2.3.0" } }, "sha512-oEGMDB6pOP2S6OwRR4PdVv610zrjnA3Bh+JnSG12fYJlBKjtNAoEb5fSUoCOOINlH96I2fU38/A2UpRKs67xYQ=="],
"@swc/helpers/tslib": ["tslib@2.8.1", "https://registry.npmmirror.com/tslib/-/tslib-2.8.1.tgz", {}, "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w=="],
"@tailwindcss/node/lightningcss": ["lightningcss@1.32.0", "https://registry.npmmirror.com/lightningcss/-/lightningcss-1.32.0.tgz", { "dependencies": { "detect-libc": "^2.0.3" }, "optionalDependencies": { "lightningcss-android-arm64": "1.32.0", "lightningcss-darwin-arm64": "1.32.0", "lightningcss-darwin-x64": "1.32.0", "lightningcss-freebsd-x64": "1.32.0", "lightningcss-linux-arm-gnueabihf": "1.32.0", "lightningcss-linux-arm64-gnu": "1.32.0", "lightningcss-linux-arm64-musl": "1.32.0", "lightningcss-linux-x64-gnu": "1.32.0", "lightningcss-linux-x64-musl": "1.32.0", "lightningcss-win32-arm64-msvc": "1.32.0", "lightningcss-win32-x64-msvc": "1.32.0" } }, "sha512-NXYBzinNrblfraPGyrbPoD19C1h9lfI/1mzgWYvXUTe414Gz/X1FD2XBZSZM7rRTrMA8JL3OtAaGifrIKhQ5yQ=="],
"@tailwindcss/oxide-wasm32-wasi/@emnapi/core": ["@emnapi/core@1.11.3", "https://registry.npmmirror.com/@emnapi/core/-/core-1.11.3.tgz", { "dependencies": { "@emnapi/wasi-threads": "1.2.3", "tslib": "^2.4.0" }, "bundled": true }, "sha512-zLpS5asjEb7lq8jYLq37N6XKaE41DIexlY1rF/z4/tIl3wo13Sqm28fRyfIsKZD+NZ8mM5RoKkpW/rBcuoSZSg=="],
"@tailwindcss/oxide-wasm32-wasi/@emnapi/runtime": ["@emnapi/runtime@1.11.3", "https://registry.npmmirror.com/@emnapi/runtime/-/runtime-1.11.3.tgz", { "dependencies": { "tslib": "^2.4.0" }, "bundled": true }, "sha512-Xz4Tpyki7XyrpbUK1jR1AhdAdaXyhhY4lZ3neLodmhpuWfy2PAQN5B46sAiU4liOXGLkHypn/qU+jvfWSCYYLA=="],
"@tailwindcss/oxide-wasm32-wasi/@emnapi/wasi-threads": ["@emnapi/wasi-threads@1.2.3", "https://registry.npmmirror.com/@emnapi/wasi-threads/-/wasi-threads-1.2.3.tgz", { "dependencies": { "tslib": "^2.4.0" }, "bundled": true }, "sha512-ELEBe8PsLvvJ6QMr0zLt8ffvOHW/dc1m3CEzNMg7aJUv3bMaoDtw2TXyDAwkYBuroxxuHEwhRTLJSe5sya547g=="],
"@tailwindcss/oxide-wasm32-wasi/@napi-rs/wasm-runtime": ["@napi-rs/wasm-runtime@1.2.2", "https://registry.npmmirror.com/@napi-rs/wasm-runtime/-/wasm-runtime-1.2.2.tgz", { "dependencies": { "@tybys/wasm-util": "^0.10.3" }, "peerDependencies": { "@emnapi/core": "^1.7.1 || ^2.0.0-alpha.3", "@emnapi/runtime": "^1.7.1 || ^2.0.0-alpha.3" }, "bundled": true }, "sha512-JfB4kuJQjaoHuCTseIINHtHWeJnvgEcxjwA5t/Y00ZgaOO1Crz3fjT/p8kT28zA/Caz7oiUMn3d6H2yOVCVwuw=="],
"@tailwindcss/oxide-wasm32-wasi/@tybys/wasm-util": ["@tybys/wasm-util@0.10.3", "https://registry.npmmirror.com/@tybys/wasm-util/-/wasm-util-0.10.3.tgz", { "dependencies": { "tslib": "^2.4.0" }, "bundled": true }, "sha512-F3fo1MYrRJYL3zER0OUOmkutjr1Vp23m7OsSgp7nq4SP6OqX6C/56XFIPAl5bt3zaBRjmW7SGz3u/6LwFpYcOg=="],
"@tailwindcss/oxide-wasm32-wasi/tslib": ["tslib@2.8.1", "https://registry.npmmirror.com/tslib/-/tslib-2.8.1.tgz", { "bundled": true }, "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w=="],
"aria-hidden/tslib": ["tslib@2.8.1", "https://registry.npmmirror.com/tslib/-/tslib-2.8.1.tgz", {}, "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w=="],
"@tailwindcss/node/lightningcss/lightningcss-android-arm64": ["lightningcss-android-arm64@1.32.0", "https://registry.npmmirror.com/lightningcss-android-arm64/-/lightningcss-android-arm64-1.32.0.tgz", { "os": "android", "cpu": "arm64" }, "sha512-YK7/ClTt4kAK0vo6w3X+Pnm0D2cf2vPHbhOXdoNti1Ga0al1P4TBZhwjATvjNwLEBCnKvjJc2jQgHXH0NEwlAg=="],
"@tailwindcss/node/lightningcss/lightningcss-darwin-arm64": ["lightningcss-darwin-arm64@1.32.0", "https://registry.npmmirror.com/lightningcss-darwin-arm64/-/lightningcss-darwin-arm64-1.32.0.tgz", { "os": "darwin", "cpu": "arm64" }, "sha512-RzeG9Ju5bag2Bv1/lwlVJvBE3q6TtXskdZLLCyfg5pt+HLz9BqlICO7LZM7VHNTTn/5PRhHFBSjk5lc4cmscPQ=="],
"@tailwindcss/node/lightningcss/lightningcss-darwin-x64": ["lightningcss-darwin-x64@1.32.0", "https://registry.npmmirror.com/lightningcss-darwin-x64/-/lightningcss-darwin-x64-1.32.0.tgz", { "os": "darwin", "cpu": "x64" }, "sha512-U+QsBp2m/s2wqpUYT/6wnlagdZbtZdndSmut/NJqlCcMLTWp5muCrID+K5UJ6jqD2BFshejCYXniPDbNh73V8w=="],
"@tailwindcss/node/lightningcss/lightningcss-freebsd-x64": ["lightningcss-freebsd-x64@1.32.0", "https://registry.npmmirror.com/lightningcss-freebsd-x64/-/lightningcss-freebsd-x64-1.32.0.tgz", { "os": "freebsd", "cpu": "x64" }, "sha512-JCTigedEksZk3tHTTthnMdVfGf61Fky8Ji2E4YjUTEQX14xiy/lTzXnu1vwiZe3bYe0q+SpsSH/CTeDXK6WHig=="],
"@tailwindcss/node/lightningcss/lightningcss-linux-arm-gnueabihf": ["lightningcss-linux-arm-gnueabihf@1.32.0", "https://registry.npmmirror.com/lightningcss-linux-arm-gnueabihf/-/lightningcss-linux-arm-gnueabihf-1.32.0.tgz", { "os": "linux", "cpu": "arm" }, "sha512-x6rnnpRa2GL0zQOkt6rts3YDPzduLpWvwAF6EMhXFVZXD4tPrBkEFqzGowzCsIWsPjqSK+tyNEODUBXeeVHSkw=="],
"@tailwindcss/node/lightningcss/lightningcss-linux-arm64-gnu": ["lightningcss-linux-arm64-gnu@1.32.0", "https://registry.npmmirror.com/lightningcss-linux-arm64-gnu/-/lightningcss-linux-arm64-gnu-1.32.0.tgz", { "os": "linux", "cpu": "arm64" }, "sha512-0nnMyoyOLRJXfbMOilaSRcLH3Jw5z9HDNGfT/gwCPgaDjnx0i8w7vBzFLFR1f6CMLKF8gVbebmkUN3fa/kQJpQ=="],
"@tailwindcss/node/lightningcss/lightningcss-linux-arm64-musl": ["lightningcss-linux-arm64-musl@1.32.0", "https://registry.npmmirror.com/lightningcss-linux-arm64-musl/-/lightningcss-linux-arm64-musl-1.32.0.tgz", { "os": "linux", "cpu": "arm64" }, "sha512-UpQkoenr4UJEzgVIYpI80lDFvRmPVg6oqboNHfoH4CQIfNA+HOrZ7Mo7KZP02dC6LjghPQJeBsvXhJod/wnIBg=="],
"@tailwindcss/node/lightningcss/lightningcss-linux-x64-gnu": ["lightningcss-linux-x64-gnu@1.32.0", "https://registry.npmmirror.com/lightningcss-linux-x64-gnu/-/lightningcss-linux-x64-gnu-1.32.0.tgz", { "os": "linux", "cpu": "x64" }, "sha512-V7Qr52IhZmdKPVr+Vtw8o+WLsQJYCTd8loIfpDaMRWGUZfBOYEJeyJIkqGIDMZPwPx24pUMfwSxxI8phr/MbOA=="],
"@tailwindcss/node/lightningcss/lightningcss-linux-x64-musl": ["lightningcss-linux-x64-musl@1.32.0", "https://registry.npmmirror.com/lightningcss-linux-x64-musl/-/lightningcss-linux-x64-musl-1.32.0.tgz", { "os": "linux", "cpu": "x64" }, "sha512-bYcLp+Vb0awsiXg/80uCRezCYHNg1/l3mt0gzHnWV9XP1W5sKa5/TCdGWaR/zBM2PeF/HbsQv/j2URNOiVuxWg=="],
"@tailwindcss/node/lightningcss/lightningcss-win32-arm64-msvc": ["lightningcss-win32-arm64-msvc@1.32.0", "https://registry.npmmirror.com/lightningcss-win32-arm64-msvc/-/lightningcss-win32-arm64-msvc-1.32.0.tgz", { "os": "win32", "cpu": "arm64" }, "sha512-8SbC8BR40pS6baCM8sbtYDSwEVQd4JlFTOlaD3gWGHfThTcABnNDBda6eTZeqbofalIJhFx0qKzgHJmcPTnGdw=="],
"@tailwindcss/node/lightningcss/lightningcss-win32-x64-msvc": ["lightningcss-win32-x64-msvc@1.32.0", "https://registry.npmmirror.com/lightningcss-win32-x64-msvc/-/lightningcss-win32-x64-msvc-1.32.0.tgz", { "os": "win32", "cpu": "x64" }, "sha512-Amq9B/SoZYdDi1kFrojnoqPLxYhQ4Wo5XiL8EVJrVsB8ARoC1PWW6VGtT0WKCemjy8aC+louJnjS7U18x3b06Q=="],
}
}
+21
View File
@@ -0,0 +1,21 @@
{
"$schema": "https://shadcn-vue.com/schema.json",
"style": "new-york",
"typescript": true,
"tailwind": {
"config": "",
"css": "src/style.css",
"baseColor": "neutral",
"cssVariables": true,
"prefix": ""
},
"iconLibrary": "lucide",
"aliases": {
"components": "@/components",
"utils": "@/lib/utils",
"ui": "@/components/ui",
"lib": "@/lib",
"composables": "@/composables"
},
"registries": {}
}
+21
View File
@@ -0,0 +1,21 @@
<!doctype html>
<html lang="zh-CN">
<head>
<meta charset="UTF-8" />
<link rel="icon" type="image/svg+xml" href="/favicon.svg" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="本地 DHT 元数据搜索服务" />
<script>
(() => {
const saved = localStorage.getItem('dht-search-theme')
const dark = saved === 'dark' || (!saved && matchMedia('(prefers-color-scheme: dark)').matches)
document.documentElement.classList.toggle('dark', dark)
})()
</script>
<title>DHT Search</title>
</head>
<body>
<div id="app"></div>
<script type="module" src="/src/main.ts"></script>
</body>
</html>
+36
View File
@@ -0,0 +1,36 @@
{
"name": "dht-search-web",
"private": true,
"version": "0.1.0",
"packageManager": "bun@1.3.14",
"type": "module",
"scripts": {
"dev": "vite",
"build": "vue-tsc -b && vite build",
"check": "bun run typecheck && bun run build",
"typecheck": "vue-tsc -b",
"preview": "vite preview"
},
"dependencies": {
"@lucide/vue": "^1.29.0",
"class-variance-authority": "^0.7.1",
"clsx": "^2.1.1",
"echarts": "^6.1.0",
"reka-ui": "^2.10.1",
"tailwind-merge": "^3.6.0",
"vue": "^3.5.40",
"vue-echarts": "^8.1.0",
"vue-router": "^4.6.4"
},
"devDependencies": {
"@tailwindcss/vite": "^4.3.3",
"@types/node": "^24.13.3",
"@vitejs/plugin-vue": "^6.0.8",
"@vue/tsconfig": "^0.9.1",
"tailwindcss": "^4.3.3",
"tw-animate-css": "^1.4.0",
"typescript": "~6.0.2",
"vite": "^8.2.0",
"vue-tsc": "^3.3.8"
}
}
File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 9.3 KiB

+24
View File
@@ -0,0 +1,24 @@
<svg xmlns="http://www.w3.org/2000/svg">
<symbol id="bluesky-icon" viewBox="0 0 16 17">
<g clip-path="url(#bluesky-clip)"><path fill="#08060d" d="M7.75 7.735c-.693-1.348-2.58-3.86-4.334-5.097-1.68-1.187-2.32-.981-2.74-.79C.188 2.065.1 2.812.1 3.251s.241 3.602.398 4.13c.52 1.744 2.367 2.333 4.07 2.145-2.495.37-4.71 1.278-1.805 4.512 3.196 3.309 4.38-.71 4.987-2.746.608 2.036 1.307 5.91 4.93 2.746 2.72-2.746.747-4.143-1.747-4.512 1.702.189 3.55-.4 4.07-2.145.156-.528.397-3.691.397-4.13s-.088-1.186-.575-1.406c-.42-.19-1.06-.395-2.741.79-1.755 1.24-3.64 3.752-4.334 5.099"/></g>
<defs><clipPath id="bluesky-clip"><path fill="#fff" d="M.1.85h15.3v15.3H.1z"/></clipPath></defs>
</symbol>
<symbol id="discord-icon" viewBox="0 0 20 19">
<path fill="#08060d" d="M16.224 3.768a14.5 14.5 0 0 0-3.67-1.153c-.158.286-.343.67-.47.976a13.5 13.5 0 0 0-4.067 0c-.128-.306-.317-.69-.476-.976A14.4 14.4 0 0 0 3.868 3.77C1.546 7.28.916 10.703 1.231 14.077a14.7 14.7 0 0 0 4.5 2.306q.545-.748.965-1.587a9.5 9.5 0 0 1-1.518-.74q.191-.14.372-.293c2.927 1.369 6.107 1.369 8.999 0q.183.152.372.294-.723.437-1.52.74.418.838.963 1.588a14.6 14.6 0 0 0 4.504-2.308c.37-3.911-.63-7.302-2.644-10.309m-9.13 8.234c-.878 0-1.599-.82-1.599-1.82 0-.998.705-1.82 1.6-1.82.894 0 1.614.82 1.599 1.82.001 1-.705 1.82-1.6 1.82m5.91 0c-.878 0-1.599-.82-1.599-1.82 0-.998.705-1.82 1.6-1.82.893 0 1.614.82 1.599 1.82 0 1-.706 1.82-1.6 1.82"/>
</symbol>
<symbol id="documentation-icon" viewBox="0 0 21 20">
<path fill="none" stroke="#aa3bff" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.35" d="m15.5 13.333 1.533 1.322c.645.555.967.833.967 1.178s-.322.623-.967 1.179L15.5 18.333m-3.333-5-1.534 1.322c-.644.555-.966.833-.966 1.178s.322.623.966 1.179l1.534 1.321"/>
<path fill="none" stroke="#aa3bff" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.35" d="M17.167 10.836v-4.32c0-1.41 0-2.117-.224-2.68-.359-.906-1.118-1.621-2.08-1.96-.599-.21-1.349-.21-2.848-.21-2.623 0-3.935 0-4.983.369-1.684.591-3.013 1.842-3.641 3.428C3 6.449 3 7.684 3 10.154v2.122c0 2.558 0 3.838.706 4.726q.306.383.713.671c.76.536 1.79.64 3.581.66"/>
<path fill="none" stroke="#aa3bff" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.35" d="M3 10a2.78 2.78 0 0 1 2.778-2.778c.555 0 1.209.097 1.748-.047.48-.129.854-.503.982-.982.145-.54.048-1.194.048-1.749a2.78 2.78 0 0 1 2.777-2.777"/>
</symbol>
<symbol id="github-icon" viewBox="0 0 19 19">
<path fill="#08060d" fill-rule="evenodd" d="M9.356 1.85C5.05 1.85 1.57 5.356 1.57 9.694a7.84 7.84 0 0 0 5.324 7.44c.387.079.528-.168.528-.376 0-.182-.013-.805-.013-1.454-2.165.467-2.616-.935-2.616-.935-.349-.91-.864-1.143-.864-1.143-.71-.48.051-.48.051-.48.787.051 1.2.805 1.2.805.695 1.194 1.817.857 2.268.649.064-.507.27-.857.49-1.052-1.728-.182-3.545-.857-3.545-3.87 0-.857.31-1.558.8-2.104-.078-.195-.349-1 .077-2.078 0 0 .657-.208 2.14.805a7.5 7.5 0 0 1 1.946-.26c.657 0 1.328.092 1.946.26 1.483-1.013 2.14-.805 2.14-.805.426 1.078.155 1.883.078 2.078.502.546.799 1.247.799 2.104 0 3.013-1.818 3.675-3.558 3.87.284.247.528.714.528 1.454 0 1.052-.012 1.896-.012 2.156 0 .208.142.455.528.377a7.84 7.84 0 0 0 5.324-7.441c.013-4.338-3.48-7.844-7.773-7.844" clip-rule="evenodd"/>
</symbol>
<symbol id="social-icon" viewBox="0 0 20 20">
<path fill="none" stroke="#aa3bff" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.35" d="M12.5 6.667a4.167 4.167 0 1 0-8.334 0 4.167 4.167 0 0 0 8.334 0"/>
<path fill="none" stroke="#aa3bff" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.35" d="M2.5 16.667a5.833 5.833 0 0 1 8.75-5.053m3.837.474.513 1.035c.07.144.257.282.414.309l.93.155c.596.1.736.536.307.965l-.723.73a.64.64 0 0 0-.152.531l.207.903c.164.715-.213.991-.84.618l-.872-.52a.63.63 0 0 0-.577 0l-.872.52c-.624.373-1.003.094-.84-.618l.207-.903a.64.64 0 0 0-.152-.532l-.723-.729c-.426-.43-.289-.864.306-.964l.93-.156a.64.64 0 0 0 .412-.31l.513-1.034c.28-.562.735-.562 1.012 0"/>
</symbol>
<symbol id="x-icon" viewBox="0 0 19 19">
<path fill="#08060d" fill-rule="evenodd" d="M1.893 1.98c.052.072 1.245 1.769 2.653 3.77l2.892 4.114c.183.261.333.48.333.486s-.068.089-.152.183l-.522.593-.765.867-3.597 4.087c-.375.426-.734.834-.798.905a1 1 0 0 0-.118.148c0 .01.236.017.664.017h.663l.729-.83c.4-.457.796-.906.879-.999a692 692 0 0 0 1.794-2.038c.034-.037.301-.34.594-.675l.551-.624.345-.392a7 7 0 0 1 .34-.374c.006 0 .93 1.306 2.052 2.903l2.084 2.965.045.063h2.275c1.87 0 2.273-.003 2.266-.021-.008-.02-1.098-1.572-3.894-5.547-2.013-2.862-2.28-3.246-2.273-3.266.008-.019.282-.332 2.085-2.38l2-2.274 1.567-1.782c.022-.028-.016-.03-.65-.03h-.674l-.3.342a871 871 0 0 1-1.782 2.025c-.067.075-.405.458-.75.852a100 100 0 0 1-.803.91c-.148.172-.299.344-.99 1.127-.304.343-.32.358-.345.327-.015-.019-.904-1.282-1.976-2.808L6.365 1.85H1.8zm1.782.91 8.078 11.294c.772 1.08 1.413 1.973 1.425 1.984.016.017.241.02 1.05.017l1.03-.004-2.694-3.766L7.796 5.75 5.722 2.852l-1.039-.004-1.039-.004z" clip-rule="evenodd"/>
</symbol>
</svg>

After

Width:  |  Height:  |  Size: 4.9 KiB

+101
View File
@@ -0,0 +1,101 @@
<script setup lang="ts">
import { onBeforeUnmount, onMounted, ref, watch } from 'vue'
import { Activity, ChartLine, Database, Moon, Search, Settings, Sun, X } from '@lucide/vue'
import { RouterLink, RouterView } from 'vue-router'
import { Button } from '@/components/ui/button'
import { getStats } from '@/lib/api'
import { formatBytes } from '@/lib/format'
import type { ServiceStats } from '@/types/api'
const darkMode = ref(false)
const stats = ref<ServiceStats | null>(null)
const statsOpen = ref(false)
let statsTimer: number | null = null
let statsRequestActive = false
const navigation = [
{ to: '/', label: '搜索', icon: Search },
{ to: '/system', label: '诊断', icon: ChartLine },
{ to: '/settings', label: '配置', icon: Settings },
]
function toggleTheme() {
darkMode.value = !darkMode.value
document.documentElement.classList.toggle('dark', darkMode.value)
localStorage.setItem('dht-search-theme', darkMode.value ? 'dark' : 'light')
}
function diskStateLabel(state: ServiceStats['disk_state']): string {
if (state === 'normal') return '正常'
if (state === 'draining') return '正在排空'
return '只读保护'
}
function closeStatsOnOutsideClick(event: PointerEvent) {
if (statsOpen.value && event.target instanceof Element && !event.target.closest('[data-stats-panel]')) {
statsOpen.value = false
}
}
async function loadStats() {
if (statsRequestActive) return
statsRequestActive = true
try { stats.value = await getStats() } catch { stats.value = null } finally { statsRequestActive = false }
}
watch(statsOpen, (open) => {
if (statsTimer !== null) window.clearInterval(statsTimer)
statsTimer = null
if (open) {
void loadStats()
statsTimer = window.setInterval(() => void loadStats(), 1_000)
}
})
onMounted(() => {
darkMode.value = document.documentElement.classList.contains('dark')
document.addEventListener('pointerdown', closeStatsOnOutsideClick)
})
onBeforeUnmount(() => {
document.removeEventListener('pointerdown', closeStatsOnOutsideClick)
if (statsTimer !== null) window.clearInterval(statsTimer)
})
</script>
<template>
<div class="min-h-svh bg-background text-foreground">
<header class="sticky top-0 z-30 border-b bg-background/90 backdrop-blur-xl">
<div class="mx-auto flex h-15 max-w-7xl items-center gap-2 px-4 sm:px-6">
<RouterLink class="mr-1 flex items-center gap-2" to="/">
<span class="flex size-8 items-center justify-center rounded-lg bg-primary text-primary-foreground"><Database class="size-4" aria-hidden="true" /></span>
<span class="hidden text-sm font-semibold tracking-tight sm:inline">DHT Search</span>
</RouterLink>
<nav class="flex items-center gap-1" aria-label="主导航">
<RouterLink v-for="item in navigation" :key="item.to" v-slot="{ isActive }" :to="item.to">
<span class="flex h-9 items-center gap-2 rounded-lg px-2.5 text-sm transition-colors" :class="isActive ? 'bg-secondary font-medium text-secondary-foreground' : 'text-muted-foreground hover:bg-accent hover:text-accent-foreground'">
<component :is="item.icon" class="size-4" aria-hidden="true" />
<span class="hidden md:inline">{{ item.label }}</span>
</span>
</RouterLink>
</nav>
<div class="ml-auto flex items-center gap-1">
<Button size="icon" variant="ghost" :aria-label="darkMode ? '切换到浅色主题' : '切换到深色主题'" :title="darkMode ? '浅色主题' : '深色主题'" @click="toggleTheme"><Sun v-if="darkMode" /><Moon v-else /></Button>
<div class="relative" data-stats-panel>
<Button size="icon" variant="ghost" aria-label="运行状态" title="运行状态" @click="statsOpen = !statsOpen"><span class="relative"><Activity class="size-4" /><i class="absolute -right-0.5 -top-0.5 size-1.5 rounded-full" :class="!stats ? 'bg-muted-foreground' : stats.disk_state === 'normal' ? 'bg-emerald-500' : 'bg-amber-500'" /></span></Button>
<div v-if="statsOpen" class="absolute right-0 top-11 w-72 rounded-xl border bg-popover p-4 text-popover-foreground shadow-xl">
<div class="mb-3 flex items-center"><p class="text-sm font-semibold">服务运行状态</p><button class="ml-auto" aria-label="关闭状态面板" @click="statsOpen = false"><X class="size-4" /></button></div>
<div v-if="stats" class="grid grid-cols-2 gap-3 text-xs"><div class="status-cell"><span>磁盘状态</span><b>{{ diskStateLabel(stats.disk_state) }}</b></div><div class="status-cell"><span>磁盘剩余</span><b>{{ stats.disk_available_bytes === null ? '未知' : formatBytes(stats.disk_available_bytes) }}</b></div><div class="status-cell"><span>DHT 节点</span><b>{{ stats.nodes.toLocaleString() }}</b></div><div class="status-cell"><span>已索引内容</span><b>{{ stats.indexed_documents.toLocaleString() }}</b></div><div class="status-cell"><span>获取成功</span><b>{{ stats.metadata_ok.toLocaleString() }}</b></div><div class="status-cell"><span>下载中</span><b>{{ stats.metadata_in_flight }}</b></div><div class="status-cell"><span>新收录</span><b>{{ stats.persistence_inserted.toLocaleString() }}</b></div><div class="status-cell"><span>已过滤</span><b>{{ stats.metadata_filtered.toLocaleString() }}</b></div><div class="status-cell"><span>验证成功</span><b>{{ stats.verification_succeeded.toLocaleString() }}</b></div></div>
<p v-else class="py-4 text-center text-xs text-muted-foreground">无法获取服务状态</p>
</div>
</div>
</div>
</div>
</header>
<RouterView />
</div>
</template>
@@ -0,0 +1,29 @@
<script setup lang="ts">
import { ChevronRight, Files, Flame, HardDrive, Radio, Repeat2 } from '@lucide/vue'
import { availabilityLabel, formatBytes, heatLabel, relativeDate } from '@/lib/format'
import type { SearchHit } from '@/types/api'
defineProps<{ hit: SearchHit }>()
defineEmits<{ select: [hit: SearchHit] }>()
</script>
<template>
<button class="group w-full rounded-xl border bg-card p-5 text-left shadow-xs transition hover:-translate-y-0.5 hover:border-foreground/20 hover:shadow-md" type="button" @click="$emit('select', hit)">
<div class="flex items-start gap-4">
<div class="min-w-0 flex-1">
<h2 class="truncate text-base font-semibold tracking-tight">{{ hit.name }}</h2>
<p class="mt-1.5 truncate font-mono text-[11px] text-muted-foreground">{{ hit.info_hash }}</p>
</div>
<ChevronRight class="mt-1 size-4 shrink-0 text-muted-foreground transition group-hover:translate-x-0.5 group-hover:text-foreground" />
</div>
<div class="mt-4 flex flex-wrap gap-x-5 gap-y-2 text-xs text-muted-foreground">
<span class="result-meta"><HardDrive />{{ formatBytes(hit.total_size) }}</span>
<span class="result-meta"><Files />{{ hit.file_count.toLocaleString() }} 个文件</span>
<span class="result-meta"><Repeat2 />{{ hit.variant_count }} 个版本</span>
<span class="result-meta"><Flame />{{ heatLabel[hit.heat.level] }} {{ hit.heat.score }}</span>
<span class="result-meta" :class="hit.availability.status === 'active' && 'text-emerald-600'"><Radio />{{ availabilityLabel[hit.availability.status] }}</span>
<span class="ml-auto">最近发现 {{ relativeDate(hit.last_seen) }}</span>
</div>
</button>
</template>
@@ -0,0 +1,147 @@
<script setup lang="ts">
import { computed, onBeforeUnmount, onMounted, ref, watch } from 'vue'
import { Check, Copy, Download, LoaderCircle, X } from '@lucide/vue'
import { Button } from '@/components/ui/button'
import { AppPagination } from '@/components/ui/pagination'
import { AppSelect } from '@/components/ui/select'
import { availabilityLabel, formatBytes, formatDate, heatLabel } from '@/lib/format'
import type { ContentVariants, TorrentDetail } from '@/types/api'
const props = defineProps<{
open: boolean
detail: TorrentDetail | null
variants: ContentVariants | null
loading: boolean
filesLoading: boolean
filesError: string
error: string
}>()
const emit = defineEmits<{ close: []; retry: []; 'file-page': [offset: number]; 'file-page-size': [size: number] }>()
const copied = ref('')
const filePage = computed(() => props.detail ? Math.floor(props.detail.file_offset / props.detail.file_limit) + 1 : 1)
const filePageSizeOptions = [
{ value: '25', label: '25 条/页' },
{ value: '50', label: '50 条/页' },
{ value: '100', label: '100 条/页' },
{ value: '200', label: '200 条/页' },
]
function fileName(path: string) {
const normalized = path.replaceAll('\\', '/')
return normalized.slice(normalized.lastIndexOf('/') + 1) || normalized
}
function fileDirectory(path: string) {
const normalized = path.replaceAll('\\', '/')
const separator = normalized.lastIndexOf('/')
return separator > 0 ? normalized.slice(0, separator) : '根目录'
}
async function copy(value: string, type: string) {
if (navigator.clipboard && window.isSecureContext) {
await navigator.clipboard.writeText(value)
} else {
const input = document.createElement('textarea')
input.value = value
input.style.position = 'fixed'
input.style.opacity = '0'
document.body.appendChild(input)
input.select()
document.execCommand('copy')
input.remove()
}
copied.value = type
window.setTimeout(() => { if (copied.value === type) copied.value = '' }, 1600)
}
function changeFilePage(page: number) {
if (props.detail) emit('file-page', (page - 1) * props.detail.file_limit)
}
function changeFilePageSize(value: string) {
emit('file-page-size', Number(value))
}
function onKeydown(event: KeyboardEvent) {
if (event.key === 'Escape' && props.open) emit('close')
}
watch(() => props.open, (open) => { document.body.style.overflow = open ? 'hidden' : '' })
onMounted(() => window.addEventListener('keydown', onKeydown))
onBeforeUnmount(() => {
window.removeEventListener('keydown', onKeydown)
document.body.style.overflow = ''
})
</script>
<template>
<Teleport to="body">
<Transition name="fade">
<div v-if="open" class="fixed inset-0 z-50 bg-black/45 backdrop-blur-[2px]" @click.self="$emit('close')">
<Transition appear name="slide">
<section aria-label="种子详情" aria-modal="true" class="absolute inset-y-0 right-0 flex w-full max-w-2xl flex-col border-l bg-background shadow-2xl" role="dialog">
<header class="flex h-16 shrink-0 items-center border-b px-5">
<span class="font-semibold">种子详情</span>
<Button class="ml-auto" size="icon" variant="ghost" aria-label="关闭详情" @click="$emit('close')"><X /></Button>
</header>
<div v-if="loading" class="flex flex-1 items-center justify-center text-sm text-muted-foreground"><LoaderCircle class="mr-2 size-4 animate-spin" />正在加载详情</div>
<div v-else-if="error" class="flex flex-1 flex-col items-center justify-center gap-4 px-6 text-center"><p class="text-sm text-destructive">{{ error }}</p><Button variant="outline" @click="$emit('retry')">重新加载</Button></div>
<div v-else-if="detail" class="flex-1 overflow-y-auto">
<div class="space-y-5 border-b p-5 sm:p-7">
<div><h2 class="break-words text-xl font-semibold tracking-tight">{{ detail.name }}</h2><button class="mt-2 flex max-w-full items-center gap-1.5 text-left font-mono text-[11px] leading-5 text-muted-foreground hover:text-foreground" title="复制 infohash" @click="copy(detail.info_hash, 'hash')"><span class="truncate">{{ detail.info_hash }}</span><Check v-if="copied === 'hash'" class="size-3 shrink-0 text-emerald-600" /><Copy v-else class="size-3 shrink-0" /></button></div>
<div class="flex flex-wrap gap-2">
<Button as-child><a :href="detail.magnet_link"><Download />打开磁力链接</a></Button>
<Button variant="outline" @click="copy(detail.magnet_link, 'magnet')"><Check v-if="copied === 'magnet'" /><Copy v-else />{{ copied === 'magnet' ? '已复制' : '复制磁力链接' }}</Button>
</div>
<dl class="grid grid-cols-2 gap-x-5 gap-y-4 rounded-xl bg-muted/55 p-4 text-sm sm:grid-cols-3">
<div><dt>总大小</dt><dd>{{ formatBytes(detail.total_size) }}</dd></div>
<div><dt>文件数量</dt><dd>{{ detail.file_count.toLocaleString() }}</dd></div>
<div><dt>分片大小</dt><dd>{{ formatBytes(detail.piece_length) }}</dd></div>
<div><dt>热度</dt><dd>{{ heatLabel[detail.heat.level] }} · {{ detail.heat.score }}</dd></div>
<div><dt>可用性</dt><dd>{{ availabilityLabel[detail.availability.status] }}<template v-if="detail.availability.reachable_peers"> · {{ detail.availability.reachable_peers }} Peer</template></dd></div>
<div><dt>发现次数</dt><dd>{{ detail.seen_count.toLocaleString() }}</dd></div>
<div><dt>首次收录</dt><dd>{{ formatDate(detail.first_seen) }}</dd></div>
<div><dt>最近发现</dt><dd>{{ formatDate(detail.last_seen) }}</dd></div>
<div><dt>最近验证</dt><dd>{{ formatDate(detail.availability.last_verified_at) }}</dd></div>
</dl>
</div>
<div class="border-b p-5 sm:p-7">
<div class="mb-4 flex flex-wrap items-center gap-3">
<div><h3 class="font-semibold">文件详情</h3><p class="mt-1 text-xs text-muted-foreground"> {{ detail.file_count.toLocaleString() }} 个文件</p></div>
<div class="ml-auto flex flex-wrap items-center justify-end gap-2">
<AppSelect :model-value="String(detail.file_limit)" :options="filePageSizeOptions" label="每页文件数量" @update:model-value="changeFilePageSize" />
<AppPagination v-if="detail.file_count > detail.file_limit" :disabled="filesLoading" :page="filePage" :page-size="detail.file_limit" :total="detail.file_count" @update:page="changeFilePage" />
</div>
</div>
<div class="relative overflow-hidden rounded-xl border bg-card">
<div v-if="filesLoading" class="absolute inset-0 z-10 flex items-center justify-center bg-background/75 backdrop-blur-sm"><LoaderCircle class="size-5 animate-spin text-muted-foreground" /></div>
<div v-if="filesError" class="border-b bg-destructive/10 px-4 py-2 text-xs text-destructive">{{ filesError }}</div>
<div class="divide-y">
<div v-for="(file, index) in detail.files" :key="`${detail.file_offset}-${file.path}-${file.size}`" class="group flex items-center gap-3 px-3 py-2 transition-colors hover:bg-muted/45 sm:px-4">
<span class="w-7 shrink-0 text-left font-mono text-[10px] tabular-nums text-muted-foreground/70">{{ detail.file_offset + index + 1 }}</span>
<div class="min-w-0 flex-1"><p class="truncate text-sm font-medium leading-4" :title="file.path">{{ fileName(file.path) }}</p><p class="truncate text-[10px] leading-4 text-muted-foreground" :title="fileDirectory(file.path)">{{ fileDirectory(file.path) }}</p></div>
<span class="w-20 shrink-0 text-right text-xs tabular-nums text-muted-foreground">{{ formatBytes(file.size) }}</span>
</div>
<p v-if="!detail.files.length" class="px-4 py-8 text-center text-sm text-muted-foreground">没有文件信息</p>
</div>
</div>
</div>
<div v-if="variants && variants.total > 1" class="p-5 sm:p-7">
<h3 class="mb-1 font-semibold">相同内容的其他版本</h3><p class="mb-3 text-xs text-muted-foreground">文件结构一致 infohash 不同</p>
<div class="space-y-2">
<div v-for="variant in variants.variants" :key="variant.info_hash" class="rounded-lg border p-3" :class="variant.info_hash === detail.info_hash && 'bg-muted/50'">
<div class="flex items-center gap-3"><div class="min-w-0 flex-1"><p class="truncate text-sm font-medium">{{ variant.name }}</p><p class="truncate font-mono text-[10px] text-muted-foreground">{{ variant.info_hash }}</p></div><Button size="icon" variant="ghost" aria-label="复制该版本磁力链接" @click="copy(variant.magnet_link, variant.info_hash)"><Check v-if="copied === variant.info_hash" /><Copy v-else /></Button></div>
</div>
</div>
</div>
</div>
</section>
</Transition>
</div>
</Transition>
</Teleport>
</template>
@@ -0,0 +1,31 @@
<script setup lang="ts">
import type { PrimitiveProps } from "reka-ui"
import type { HTMLAttributes } from "vue"
import type { ButtonVariants } from "."
import { Primitive } from "reka-ui"
import { cn } from "@/lib/utils"
import { buttonVariants } from "."
interface Props extends PrimitiveProps {
variant?: ButtonVariants["variant"]
size?: ButtonVariants["size"]
class?: HTMLAttributes["class"]
}
const props = withDefaults(defineProps<Props>(), {
as: "button",
})
</script>
<template>
<Primitive
data-slot="button"
:data-variant="variant"
:data-size="size"
:as="as"
:as-child="asChild"
:class="cn(buttonVariants({ variant, size }), props.class)"
>
<slot />
</Primitive>
</template>
+40
View File
@@ -0,0 +1,40 @@
import type { VariantProps } from "class-variance-authority"
import { cva } from "class-variance-authority"
export { default as Button } from "./Button.vue"
export const buttonVariants = cva(
"inline-flex items-center justify-center gap-2 whitespace-nowrap rounded-md text-sm font-medium transition-all disabled:pointer-events-none disabled:opacity-50 [&_svg]:pointer-events-none [&_svg:not([class*='size-'])]:size-4 shrink-0 [&_svg]:shrink-0 outline-none focus-visible:border-ring focus-visible:ring-ring/50 focus-visible:ring-3 aria-invalid:ring-destructive/20 dark:aria-invalid:ring-destructive/40 aria-invalid:border-destructive",
{
variants: {
variant: {
default:
"bg-primary text-primary-foreground hover:bg-primary/90",
destructive:
"bg-destructive text-white hover:bg-destructive/90 focus-visible:ring-destructive/20 dark:focus-visible:ring-destructive/40 dark:bg-destructive/60",
outline:
"border bg-background shadow-xs hover:bg-accent hover:text-accent-foreground dark:bg-input/30 dark:border-input dark:hover:bg-input/50",
secondary:
"bg-secondary text-secondary-foreground hover:bg-secondary/80",
ghost:
"hover:bg-accent hover:text-accent-foreground dark:hover:bg-accent/50",
link: "text-primary underline-offset-4 hover:underline",
},
size: {
"default": "h-9 px-4 py-2 has-[>svg]:px-3",
"xs": "h-6 gap-1 rounded-md px-2 text-xs has-[>svg]:px-1.5 [&_svg:not([class*='size-'])]:size-3",
"sm": "h-8 rounded-md gap-1.5 px-3 has-[>svg]:px-2.5",
"lg": "h-10 rounded-md px-6 has-[>svg]:px-4",
"icon": "size-9",
"icon-xs": "size-6 rounded-md [&_svg:not([class*='size-'])]:size-3",
"icon-sm": "size-8",
"icon-lg": "size-10",
},
},
defaultVariants: {
variant: "default",
size: "default",
},
},
)
export type ButtonVariants = VariantProps<typeof buttonVariants>
@@ -0,0 +1,33 @@
<script setup lang="ts">
import { ChevronLeft, ChevronRight, MoreHorizontal } from '@lucide/vue'
import {
PaginationEllipsis,
PaginationList,
PaginationListItem,
PaginationNext,
PaginationPrev,
PaginationRoot,
} from 'reka-ui'
defineProps<{
page: number
pageSize: number
total: number
disabled?: boolean
}>()
const emit = defineEmits<{ 'update:page': [page: number] }>()
</script>
<template>
<PaginationRoot :page="page" :items-per-page="pageSize" :total="total" :disabled="disabled" :sibling-count="1" show-edges @update:page="emit('update:page', $event)">
<PaginationList v-slot="{ items }" class="flex items-center gap-1">
<PaginationPrev class="pagination-button" title="上一页"><ChevronLeft class="size-4" /></PaginationPrev>
<template v-for="(item, index) in items" :key="item.type === 'page' ? item.value : `ellipsis-${index}`">
<PaginationListItem v-if="item.type === 'page'" class="pagination-button data-[selected]:bg-primary data-[selected]:text-primary-foreground" :value="item.value">{{ item.value }}</PaginationListItem>
<PaginationEllipsis v-else class="flex size-8 items-center justify-center text-muted-foreground"><MoreHorizontal class="size-4" /></PaginationEllipsis>
</template>
<PaginationNext class="pagination-button" title="下一页"><ChevronRight class="size-4" /></PaginationNext>
</PaginationList>
</PaginationRoot>
</template>
@@ -0,0 +1 @@
export { default as AppPagination } from './AppPagination.vue'
@@ -0,0 +1,41 @@
<script setup lang="ts">
import { Check, ChevronDown } from '@lucide/vue'
import {
SelectContent,
SelectItem,
SelectItemIndicator,
SelectItemText,
SelectPortal,
SelectRoot,
SelectTrigger,
SelectValue,
SelectViewport,
} from 'reka-ui'
defineProps<{
modelValue: string
options: ReadonlyArray<{ value: string; label: string }>
label: string
}>()
const emit = defineEmits<{ 'update:modelValue': [value: string] }>()
</script>
<template>
<SelectRoot :model-value="modelValue" @update:model-value="(value) => typeof value === 'string' && emit('update:modelValue', value)">
<SelectTrigger :aria-label="label" class="flex h-9 min-w-28 items-center justify-between gap-2 rounded-md border bg-background px-3 text-sm outline-none transition hover:bg-accent focus:ring-3 focus:ring-ring/20 data-[placeholder]:text-muted-foreground">
<SelectValue />
<ChevronDown class="size-4 text-muted-foreground" />
</SelectTrigger>
<SelectPortal>
<SelectContent class="z-50 min-w-[var(--reka-select-trigger-width)] overflow-hidden rounded-lg border bg-popover text-popover-foreground shadow-lg" position="popper" :side-offset="5">
<SelectViewport class="p-1">
<SelectItem v-for="option in options" :key="option.value" :value="option.value" class="relative flex cursor-default select-none items-center rounded-md py-2 pl-8 pr-3 text-sm outline-none data-[highlighted]:bg-accent data-[highlighted]:text-accent-foreground data-[disabled]:pointer-events-none data-[disabled]:opacity-50">
<SelectItemIndicator class="absolute left-2 flex size-4 items-center justify-center"><Check class="size-4" /></SelectItemIndicator>
<SelectItemText>{{ option.label }}</SelectItemText>
</SelectItem>
</SelectViewport>
</SelectContent>
</SelectPortal>
</SelectRoot>
</template>
@@ -0,0 +1 @@
export { default as AppSelect } from './AppSelect.vue'

Some files were not shown because too many files have changed in this diff Show More