Add chunk-wise BitPackedV2 compression with PatchesV2 - #9677
Add chunk-wise BitPackedV2 compression with PatchesV2#9677joseph-isaacs wants to merge 1 commit into
Conversation
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Merging this PR will regress 2 benchmarks
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | mul_i32_nonnull_avx512 |
7.2 µs | 8.1 µs | -10.29% |
| ❌ | Simulation | compress_fsst[(500, 64, 8)] |
482.6 µs | 537.8 µs | -10.27% |
| ⚡ | WallTime | arrow_checked_add_u32_neon[16384] |
20.5 µs | 13.5 µs | +51.29% |
| ⚡ | WallTime | arrow_checked_add_u32_avx2[16384] |
21.3 µs | 17.7 µs | +20.37% |
| ⚡ | Simulation | take[duplicates/repeated/primitive/nonnull/chunks=16/indices=1000] |
234.2 µs | 201.2 µs | +16.37% |
| ⚡ | WallTime | words_gather_scalar_avx2[65536] |
9.4 µs | 8.3 µs | +13.63% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ji/bitpacked-v2-patches (f4d3659) with develop (68e2aee)
Footnotes
-
106 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
4 benchmarks were run, but are now archived. If they were deleted in another branch, consider rebasing to remove them from the report. Instead if they were added back, click here to restore them. ↩
Summary
BitPackedV2with a bit width per 1,024-value chunkPatchesV2with chunk-local indices and compact slot metadataBitPackedV2for unstable integer bitpacking and FoR residualsThis is the base of a two-PR stack. It intentionally excludes BlockedFoR, FloatQuantV2, and temporary benchmark/scatter experiments.
Validation
cargo +nightly fmt --all -- --checkcargo check -p vortex-btrblockscargo check -p vortex-btrblocks --features unstable_encodingscargo test -p vortex-btrblocks --features unstable_encodings test_bitpacking_v2_compresses_attached_patchescargo clippy -p vortex-btrblocks -p vortex-bench --all-targets --features unstable_encodings -- -D warningsCompression note
The prior combined branch reduced controlled TPC-H SF10 output from 2,455,096,884 to 2,416,147,124 bytes (-38,949,760, -1.5865%). That number is stack-level evidence, not an isolated claim for this PR; a clean branch-by-branch recompression should be used for final attribution.