mirror of
https://github.com/Egonex-AI/Understand-Anything.git
synced 2026-06-22 10:58:03 +08:00
Tirth KananiandClaude Opus 4.7
eea73b656d
The /understand pipeline reads every code file twice during analysis: once in compute-batches (`extractExports` for the cross-batch neighbour map) and once again in extract-import-map (per-language config loaders). Both sites used sequential `readFileSync` loops, so on the iOS repo in issue #226 (~15k files) the disk-read time was effectively serialised behind a single libuv thread while the rest of the pool sat idle. ## Changes - `extractExports` now batches files into `IO_PARALLELISM = 64` slices and issues all `readFile` calls in each slice through `Promise.all`, letting libuv's worker-thread pool overlap disk reads. The tree-sitter parse stays on the main thread because `web-tree-sitter` is single-threaded WASM — pipelining the I/O while parses run is where the wall-time savings come from. - `loadTsConfigs`, `loadGoModules`, `loadPhpAutoloads` and `buildResolutionContext` switch to async / `Promise.all` for the same reason. `buildResolutionContext` also runs the three loader passes concurrently (`Promise.all([...])`) since they're independent. - A small `readFilesParallel(paths)` helper is added at the top of `extract-import-map.mjs` so the three loaders share the same error-preserving shape. ## Why behavior stays identical - Each loader collects its candidate paths in `files[]` order *before* issuing reads, then iterates `reads` in the same order to emit warnings + populate output maps. So stderr order and the final map contents are byte-identical to the previous sequential loops. - `extractExports` collects per-file errors in-place in the `Promise.all` callbacks and emits warnings during the post-read serial loop, again in chunk order — so warning text and order match the previous implementation. - Tree-sitter parsing is unchanged: parses still run serially on the main thread, just with reads pipelined alongside. ## What's NOT in this PR - `buildFingerprintStore` and `analyzeChanges` in `core/fingerprint.ts` have the same sequential pattern. They're left alone here because they're part of the public `@understand-anything/core` API; making them async would be a breaking change worth its own discussion. Internal-only `.mjs` scripts are safe to refactor without API churn. - No change to scan-project: most of its sync I/O is `statSync` (metadata, not content) plus a handful of small `.gitignore` / `.understandignore` reads. The parallelism win is marginal there. ## Verification - `pnpm lint` clean - `pnpm --filter @understand-anything/core build` clean - `pnpm --filter @understand-anything/skill build` clean - `pnpm test`: 196/196 — including `test_compute_batches.test.mjs` (19 tests) and `test_extract_import_map.test.mjs` (40 tests), which exercise both changed pipelines end-to-end with fixture projects. No output diff vs main. Refs #76 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>