feat(pipeline): bound extraction retention with opt-in two-pass batches

[zhiyuzhang001-a11y]

Fixes #2081. Standalone batching-only follow-up to #1925; addresses the extraction-retention part of #1997.

Large full indexes currently retain every file's complete extraction result through resolution. CBM_STREAMING_BATCH_FILES=128 opts into two passes: register definitions and retain compact definition/import metadata, then re-extract, resolve and release one batch at a time. The graph and compact cross-file surface remain global; this is a bound on the live extraction set, not a process-wide memory budget.

The split preserves Java's shared cross registry and the existing presence-based CBM_DISABLE_LSP_CROSS opt-out. Cross-file definitions are collected only after the full registry and IMPORTS graph exist (important for Python/TS base QNs). Macros/pkgmap use the whole repository; replay avoids duplicate nodes and coverage diagnostics; final serial graph scans run once with the current ID watermark.

No scheduler, worker-scoping patch, TSNodeStack/result-array optimization or complexity-order rewrite is included. Main's #2013 is retained; #2076/#2079 remain the maintainer-owned fixes. #1925 remains open as the discussion entry point.

Reviewable commits

1. Batch-safe primitives and compact surface, with a lifetime regression. Builds independently in a separate worktree.
2. Opt-in pipeline and graph/surface/diagnostic parity regression, including one/four workers and disabled cross-LSP.
3. Reproducible benchmark harness and measurements. The larger evidence diff is separate from the C implementation.

Evidence

macOS arm64, Apple clang 21, 24 GiB RAM, four workers, three fresh worker runs per variant against aa44c28e. Whole-worker peak RSS comes from wait4, separately from phase logs:

• 2,049 generated TS files: median RSS about 649 → 301 MiB; worker wall about 2.64 → 3.23 s. Feature-off matches baseline.
• Repomix e3b15a4: median RSS about 272 → 194 MiB; worker wall about 2.06 → 2.42 s.
• All 15 runs match their corpus's baseline exactly on normalized persisted nodes, node properties, edges including properties, and LSP surface JSON; integrity checks return ok. Parse-partial counts remain 0 / 4 respectively.

Commands, corpus generator, raw per-run measurements and limitations: docs/benchmarks/bounded-two-pass.md, results, harness.

Validation status

• ASan/UBSan pipeline: 275 passed, including the compact-surface lifetime and batch parity regressions. The full canonical run also reports pipeline 275/275, including batch=1 with one worker.
• Baseline, standalone primitives-commit, and final-head production builds: passed. The final head also ran the committed benchmark harness successfully with exact graph/surface parity.
• git diff --check, project DCO check (all three commits), and project lint-format: passed.
• Full canonical scripts/test.sh: passed — 7,952 passed, 0 failed, 7 Windows-only skips across all 141 C suites; all entry-point contract checks and production parent/worker watchdog, worker-error transport, watcher kill-switch and security-string guards passed too.
• Canonical scripts/lint.sh --ci passed (exit 0), including no-skips policy, cppcheck 2.21.0, clang-format and NOLINT checks. Task-local clang-tidy 22.1.8 also passed scripts/ci/lint-mem.sh. The additional full clang-tidy run is not clean: it reports readability/include diagnostics across the tree, including magic-number and cognitive-complexity diagnostics on added lines. The project CI lint entry excludes full clang-tidy; no lint rules or workflows were changed.
• GitHub CodeQL analysis/gate, static security, DCO, memory analysis, acknowledgement and change detection have passed at head 16a739d3. License and lint gates plus native Linux/macOS/Windows smoke checks remain running or queued; those platform results remain pending.

The additional native Apple Clang analyzer run on the three changed C translation units reported only two errno-check diagnostics in the unchanged macro-file reader; it is supplementary evidence, not a substitute for the project's pinned clang-tidy/cppcheck CI lanes.

Windows test follow-up

Commit d8dbf01c reserves a 512-byte temporary-path buffer in the new streaming parity test. The Windows cbm_mkdtemp API expands /tmp/ to %TEMP% in place; the original template-sized array could overflow. The previous Windows shard 2 stopped in the pipeline suite with an ASan abort; its retained job log contains only the diagnostic tail. Local ASan/UBSan pipeline tests passed 275/275 after the fix, with formatting and diff checks passing. Fresh-head CI is pending; earlier CI results above refer to 16a739d3.

Fixed-head Windows validation: both full Windows shards passed on d8dbf01c. Shard 2 reports pipeline rc=0 pass=275 fail=0 skip=0 and finishes successfully, confirming the temporary-buffer fix. Remaining Unix shards are still running. The independent Windows guards cold-start failure still needs the maintainer rerun requested above; this PR is not yet all-green.

[github-actions]

Thanks for opening this — it has been seen, and it is queued.

This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence.

Current review status: working through a backlog. 0.9.1-rc.1 is out, so the release freeze that held reviews is over — but it left a large queue of open pull requests behind it, and we are reading through them oldest-first. The background is in discussion #1144.

What that means for this PR, concretely:

• It will not be closed for inactivity. No stale bot touches pull requests here.
• It may still sit a while before a human reads it. That is on us, not on you.
• Older PRs are read first, so a recent one is not being skipped — it is behind a queue.

Things that will genuinely speed it up whenever review does happen:

• Keep it rebased on main — the tree is moving quickly right now, and a conflicting branch cannot be reviewed as the diff you intended.
• Get CI green, or say which failures you believe are pre-existing.
• Keep the change to one claim. Bundled features and refactors get split before they get merged, which costs you a round trip.
• Every commit needs a sign-off (git commit -s) — CI enforces DCO.

If this fixes a bug, a reproduction we can run is worth more than a description of the symptom.

Thanks for contributing, and sorry in advance for the wait.

[zhiyuzhang001-a11y]

At head d8dbf01c, the Windows guards job failed in test_daemon_stability.py::section_cold_storm: one of six concurrent cli list_projects {} clients reported secure CLI coordination could not be created (endpoint). Failing job.

This occurs during endpoint initialization before indexing; both that startup code and the guard are unchanged by this PR. The same guard passed on the preceding head; the only intervening change enlarges the new pipeline test temporary-path buffer. Upstream main is still aa44c28e, so there is no newer base fix to bring in. This appears to be an intermittent startup race, though a rerun is needed to confirm.

Could a maintainer rerun this job? My targeted rerun request was denied with HTTP 403 (Must have admin rights to Repository). I have not changed or weakened the guard. The separate Windows full pipeline test still needs to validate the buffer fix.