Conditionally slice elements to selected index bounds before filtering wide list arrays - #9679
Conditionally slice elements to selected index bounds before filtering wide list arrays#9679mhk197 wants to merge 8 commits into
Conversation
Merging this PR will degrade performance by 0.12%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | take_map[(0.1, 1.0)] |
226.8 µs | 262.8 µs | -13.73% |
| ❌ | Simulation | take_map[(0.1, 0.5)] |
151 µs | 173.8 µs | -13.12% |
| ❌ | Simulation | compact_sliced[(1024, 90)] |
1.4 µs | 1.5 µs | -10.74% |
| ❌ | Simulation | compact_sliced[(2048, 90)] |
1.4 µs | 1.6 µs | -10.34% |
| ⚡ | WallTime | arrow_checked_add_u32_avx512[16384] |
21.4 µs | 17.7 µs | +20.84% |
| ⚡ | WallTime | arrow_checked_add_u32_avx2[16384] |
21.4 µs | 17.7 µs | +20.83% |
| ⚡ | WallTime | words_gather_scalar_avx2[65536] |
9.4 µs | 8.3 µs | +13.19% |
| 🆕 | Simulation | filter_list_primitive_short[AllSelected] |
N/A | 135.8 µs | N/A |
| 🆕 | Simulation | filter_list_primitive_short[Clustered] |
N/A | 145.6 µs | N/A |
| 🆕 | Simulation | filter_list_primitive_short[EdgeSpanning] |
N/A | 160.3 µs | N/A |
| 🆕 | Simulation | filter_list_primitive_short[Prefix] |
N/A | 147.9 µs | N/A |
| 🆕 | Simulation | filter_list_primitive_short[Sparse] |
N/A | 142.5 µs | N/A |
| 🆕 | Simulation | filter_list_primitive_wide[AllSelected] |
N/A | 134.8 µs | N/A |
| 🆕 | Simulation | filter_list_primitive_wide[Clustered] |
N/A | 149.1 µs | N/A |
| 🆕 | Simulation | filter_list_primitive_wide[EdgeSpanning] |
N/A | 898.7 µs | N/A |
| 🆕 | Simulation | filter_list_primitive_wide[Prefix] |
N/A | 149.9 µs | N/A |
| 🆕 | Simulation | filter_list_primitive_wide[Sparse] |
N/A | 143.5 µs | N/A |
| 🆕 | Simulation | filter_list_primitive_short[Repeated] |
N/A | 246.3 µs | N/A |
| 🆕 | Simulation | filter_list_primitive_wide[Repeated] |
N/A | 2 ms | N/A |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing mk/crop-list-filter-elements (784c0b3) with develop (d1a4ad6)3
Footnotes
-
106 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
4 benchmarks were run, but are now archived. If they were deleted in another branch, consider rebasing to remove them from the report. Instead if they were added back, click here to restore them. ↩
-
No successful run was found on
develop(68d6573) during the generation of this report, so d1a4ad6 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
Polar Signals Profiling ResultsLatest Run
Previous Runs (10)
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.981x ➖, 2↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.009x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.983x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.982x ➖, 1↑ 1↓)
duckdb / parquet / ns (0.996x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.005x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.003x ➖, 1↑ 1↓)
duckdb / vortex-file-compressed / ns (0.987x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.929x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.034x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.962x ➖, 2↑ 0↓)
duckdb / parquet / ns (1.009x ➖, 0↑ 0↓)
File Size Changes (100 files changed, +0.0% overall, 61↑ 39↓)
Totals:
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.002x ➖, 2↑ 0↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 1↑ 3↓)
duckdb / parquet / ns (0.992x ➖, 9↑ 4↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.995x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 1↓)
No file size changes detected. |
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.059x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.912x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.977x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.103x ➖, 3↑ 4↓)
|
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
duckdb / vortex-file-compressed / ns (0.946x ➖, 3↑ 1↓)
duckdb / parquet / ns (0.987x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.006x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.007x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.988x ➖, 1↑ 1↓)
datafusion / parquet / ns (0.955x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.010x ➖, 0↑ 0↓)
|
0df99f1 to
c0fbe2a
Compare
dba7820 to
ff9eddd
Compare
connortsui20
left a comment
There was a problem hiding this comment.
could you add a comment that explains the algorithm here at a higher level, literally just "if the lists are very long then we should do a cheap slice first before performing a potentially very expensive filter with a large mask" or something
ff9eddd to
6ac2e0c
Compare
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
3a40b1e to
6873b0d
Compare
Summary
Avoid expanding ListArray element masks over large unselected prefixes and suffixes. For lists averaging more than 1,024 elements per sublist, filter the child slice spanning the first through last selected list instead of the complete element range.
Example
Suppose a ListArray contains eight sublists with 2,000 elements each, and the outer filter selects rows 3 and 4. The existing path expands the mask across all 16,000 child elements. This change slices elements 6,000..10,000 and expands the mask across only those 4,000 elements.
If the selected rows span the complete element range, the optimization keeps the existing full-range path.