Prediction reports: training-set membership, novelty and conformal intervals (v4.4.0)

[RobbinBouwmeester]

Stacked on #117 (the report reuses MultiHeadRidgeCalibration and the uses_all_heads flag); the diff shown here is the report feature only. Retarget to main once #117 is merged.

What

prediction_report(psm_list, psm_list_reference=..., coverage=0.90, training_index=...) returns predictions together with what a bare number cannot say, one row per PSM:

column	meaning
predicted_rt	calibrated prediction, as predict_and_calibrate
ci_lower, ci_upper	conformal interval at the requested coverage
in_reference	exact peptidoform match in the calibration reference
dist_to_reference	Levenshtein distance to the closest reference sequence
in_training *	exact match anywhere in the 10,105,640-peptidoform corpus behind the bundled multitask model
in_selected_heads_training *	match within the training sets of the setups the calibration selected (all 80 for the multi-head calibration, the single best head otherwise)
dist_to_training *	distance to the closest training sequence, exact to 10 edits, capped beyond

* with a TrainingIndex, a ~400 MB memory-mapped artifact (sorted key hashes, per-setup membership CSR, 6.16 M unique sequences) built offline from the training cache by build_training_index.py in the research repo. The report works without it. Hosting to be decided (HF hub?).

The interval

Cross-fitted split-conformal on the reference: five folds, each predicted by a calibration fitted on the other four, half-width = finite-sample quantile of those honest |residuals| per predicted-RT bin (peak width varies along a gradient). No retraining, works for any model and calibration, finite-sample marginal coverage guarantee. Chosen over quantile regression (pinball) for exactly those reasons.

Validated on the eight held-out PRIDE setups (Figure 1b corpus), nominal 90 %:

• coverage per setup 0.88-0.97 (median 0.91); at nominal 80 %: median 0.80; at 95 %: median 0.96;
• width 4-10 % of the gradient on well-behaved setups, honestly wide where predictions are bad (35 % on the wheat-gluten out-of-domain setup, 79 % on the run that pools fractions);
• distance-scaled widths were tested and rejected: error is flat in edit distance (Figure 2), so they only overcover.

End-to-end through this API on three held-out setups: coverage@90 = 0.914 / 0.898 / 0.974, and the membership columns behave (PXD079927: 60 % of test peptidoforms in the corpus but only 1.2 % in the 80 selected setups; PXD081880: 100 % / 95 %).

Per-peptide widths (added after review of the first version)

A per-RT-bin quantile gives a setup five distinct widths, so two peptides predicted at the same retention time always got the same interval — a fair complaint about a "confidence interval". A multi-head calibration combines heads that each estimate the same retention time, and their spread differs per peptide, so Calibration.disagreement now exposes it and the interval divides the honest residuals by it before taking the per-bin quantile (the quantile is multiplied back at prediction time). Ratios are clipped to 0.2-5x the median reference peptide; when the spread carries no signal the ratios collapse to 1 and the method degenerates to the previous behaviour.

Measured through the public API on the six included held-out setups (per_peptide_width True vs False), nominal 90 %:

	per RT bin	per peptide
worst conditional slice	0.851	0.882
Spearman(width, |error|)	0.151	0.248
coverage	0.909	0.919
relative width	0.0436	0.0478
distinct widths	5	877

The gains are largest where the RT-only width was weakest (PXD080826 0.818 → 0.888, PXD081924 0.814 → 0.845); the cost is 10 % wider intervals and mild overcoverage. Edit distance to the reference was tested as the scale instead and rejected again: three times the width, coverage 0.977, no correlation with the error. per_peptide_width=False restores the previous behaviour, which also stays the behaviour of single-head calibrations.

Canonical keys

canonical_peptidoform_key reproduces the corpus key format (peprec positions, U:<unimod id>, lowercased unmapped names, position-sorted, charge ignored) — verified against real corpus keys, including modified ones.

Verification

• 167 tests pass; 20 new across tests/test_report.py and tests/test_multihead_calibration.py (key format, interval coverage / adaptivity / thin-bin fallback on synthetic data, per-input widths and their coverage, an interval refusing to drop the score it was fitted with, head disagreement per input and its collapse when heads agree, index membership global and per-setup, capped distances, non-index directory refused, report end-to-end with and without index, per-peptide against per-bin widths, pre-fitted calibration refused, multi-head head listing).
• ruff check / format clean on the new files.
• New dependency: rapidfuzz (pure wheel).
• Version 4.4.0, CHANGELOG and usage docs updated.

🤖 Generated with Claude Code

[RobbinBouwmeester]

Index size addressed in 39b621e: the artifact is now a single 105 MB .dlcidx file instead of a 400 MB directory. Stdlib LZMA zip; the interesting part is the hashes — random bits compress 1.4x at best, so they are truncated to 40 bits in a bucketed layout instead (2^24 bucket counts + 16-bit remainders, ~26 MB), costing a false positive about once per 100,000 membership queries, which a provenance flag can afford. Setup lists 2.8x, sorted sequences 3.1x, CSR pointers stored as row lengths 5.3x. Loads in about a second, answers bit-identical to the raw form, and every index test now runs against both formats.