fix: prune row groups when file statistics collapse the predicate to a constant

[jensholdgaard]

Note

This fix was investigated and written with AI assistance (Claude Code), posted with the account owner's review and consent.

Which issue does this PR close?

• Closes Regression: DataFusion 55 no longer prunes row groups for col = <literal> when statistics show the column is entirely NULL #24769.

Rationale for this change

A query like SELECT ... WHERE col = 'x' over a Parquet file whose statistics say col is entirely NULL cannot match any row, and DataFusion 54 pruned such row groups from the footer without reading them. On 55 the same query reads them.

The cause is an interaction rather than a bug in pruning itself. constant_columns_from_stats substitutes columns that file statistics prove constant, and its all-NULL branch folds such a column to a NULL literal. Once substituted, the predicate simplifies to a bare constant — NULL here — at which point build_pruning_predicates returns None (there are no column references left to build a pruning predicate over), and prune_row_groups falls through with no pruning at all.

So for exactly the files where the statistics carry the most information, the substitution is strictly counterproductive: before it, the pruning predicate's own col_null_count != row_count conjunct proved the row group empty and skipped it.

This surfaced as a regression when bisecting a real workload (a Parquet log store where a body column is NULL for the vast majority of rows) from 54 → 55. The bisect lands on #22969, which removed ListingOptions::collect_stat in favour of the session's execution.collect_statistics — default true. That change is correct in itself; it simply began feeding per-file statistics to the substitution on paths that previously had none, exposing the gap. Setting execution.collect_statistics = false restores pruning on 55, which is a useful confirmation but obviously not a fix.

Results were never wrong — a filter drops NULL and false rows alike — but the scan work is real: row groups that used to be skipped from the footer are now decoded in full.

What changes are included in this PR?

prune_row_groups now recognises the collapsed-to-constant case: if the (post-substitution, post-simplification) predicate is a false or NULL literal, every remaining row group is skipped and the skip is credited to row_groups_pruned_statistics. A small RowGroupAccessPlanFilter::skip_all helper is added alongside the existing prune_by_* methods.

The check is deliberately narrow — a downcast to Literal plus a NULL/false value test — so it cannot affect predicates that still reference columns; those take the existing path unchanged.

Are these changes tested?

Yes: test_prune_all_null_column_equality_from_file_statistics in opener/mod.rs, modelled on the neighbouring test_prune_on_partition_values_and_file_statistics. It fails on current main (3 rows scanned, 0 pruned) and passes with this change.

One note on how it asserts, in case it saves a reviewer time: it checks the row_groups_pruned_statistics metric rather than the returned row count. A row-count assertion cannot distinguish "pruned" from "scanned, then row-filtered" — both give zero rows — and an earlier draft of this test passed against the unfixed code for exactly that reason.

Verified cargo test -p datafusion-datasource-parquet --lib opener:: — 47 passed. The 16 pre-existing bloom_filter::tests failures on this crate are unrelated and reproduce identically on an untouched checkout.

Are there any user-facing changes?

No API or result changes. Queries that were already correct stay correct; affected scans read less data. Plans may show more row groups pruned, and row_groups_pruned_statistics increases correspondingly.

[jensholdgaard]

The cargo test (amd64) failure was real and pointed at a scoping problem in my first push — fixed in the update.

The nine failing evolved_schema* / test_pushdown_with_missing_* tests exercise the other path that substitutes NULL for a column: files missing the column entirely (schema evolution). My original check keyed only on "the simplified predicate is a constant NULL/false literal", so it also fired there, pruning row groups those tests expect to see in the raw scan output and shifting their pushdown metric counts. Semantically that pruning would be sound too (a file missing c2 cannot satisfy c2 = 2), but it is a behaviour change well beyond the regression this PR fixes, and partition-value-driven collapse is similarly already FilePruner's territory.

The update narrows the skip with a stats_constants_in_predicate guard: the collapse only counts as statistics-proven when the original predicate referenced a column that constant_columns_from_stats proved constant for this file. With that:

• the nine previously failing tests pass again unchanged,
• the new regression test still fails without the fix and passes with it,
• cargo test -p datafusion-datasource-parquet --lib opener:: — 47 passed,
• cargo test -p datafusion --lib datasource::physical_plan::parquet::tests — 35 passed (the 4 parquet_exec_with_* failures on my machine reproduce on an untouched checkout and pass in CI, so they are environmental).

If maintainers would rather generalise the skip to the missing-column collapse as well, I'm happy to do that in a follow-up with the corresponding test updates — it just seemed wrong to smuggle a second behaviour change into a targeted regression fix.