introduce optional rle reads from parquet - #24227
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #24227 +/- ##
==========================================
+ Coverage 81.58% 81.59% +0.01%
==========================================
Files 1123 1123
Lines 406610 407018 +408
Branches 406610 407018 +408
==========================================
+ Hits 331719 332099 +380
- Misses 55453 55476 +23
- Partials 19438 19443 +5 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
@adriangb pinging you since you seem interested in parquet related speed ups 👍 |
|
If I understand correctly the goal is to evaluate filters during filter pushdown against dictionary / RLE encoded columns? We can't propagate these dynamic type changes to the rest of the query plan / scan. Is that right? |
no not exactly. The goal of this PR is to keep RLE parquet columns in their compacted form by materializing them as dictionary arrays instead of regular strings.
exactly! This is why it needs to be done as early in the plan as possible. we inspect |
0479f9c to
6e1cbc1
Compare
Why only RLE and not dictionaries as well? How does this compare to / relate to the It also looks like this goes through I'd be more interested in seeing something at the parquet scan level that was able to e.g. optimize how row filters are applied by applying them to the dictionary instead of expanding into |
6a4e89d to
d1273bf
Compare
my bad when I say RLE i'm referring to
I agree, ill update the PR to target all parquet scans. |
|
ideally we surface columns that are physically RLE_DICTIONARY-encoded in the parquet file as Arrow To know whether a specific column is RLE_DICTIONARY-encoded you need to read the parquet file footer. For the Downstream physical operators ( So when the flag is enabled we promote all string/binary columns to dict at planning time, not just the ones that are actually RLE-encoded, because that's the only way to guarantee schema consistency across all parquet scan paths without introducing file I/O into the planning stage. I feel like i'm missing something here. if we could take a peak at the parquets metadata before physical planning and change the schema for all operators from the point forward that would be perfect. Im not sure this is currently possible |
717d31a to
fc360e8
Compare
|
@adriangb When the flag is on, DataFusion promotes string and binary columns that are physically RLE_DICTIONARY encoded in the parquet file to Dictionary(Int32, Utf8) or Dictionary(Int32, Binary) at schema inference time. This applies to all parquet scans regardless of how the table was registered. the PR is ready for review |
|
bc5011f to
6899579
Compare
4865205 to
c6a627a
Compare
c6a627a to
1cbb472
Compare
|
run benchmarks |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/introduce-rle-parquet-flag (b0b0c0e) to db8796d (merge-base) diff Run configurationrun benchmark tpcdsResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/introduce-rle-parquet-flag (b0b0c0e) to db8796d (merge-base) diff Run configurationrun benchmark tpchResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/introduce-rle-parquet-flag (b0b0c0e) to db8796d (merge-base) diff Run configurationrun benchmark clickbench_partitionedResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-T-kid/introduce-rle-parquet-flag (b0b0c0e) to db8796d (merge-base) diff Run configurationrun benchmark tpchCPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-T-kid/introduce-rle-parquet-flag (b0b0c0e) to db8796d (merge-base) diff Run configurationrun benchmark tpcdsCPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-T-kid/introduce-rle-parquet-flag (b0b0c0e) to db8796d (merge-base) diff Run configurationrun benchmark clickbench_partitionedCPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
72342ef to
b74533a
Compare
|
run benchmarks |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/introduce-rle-parquet-flag (35dd8d8) to 26b40dd (merge-base) diff Run configurationrun benchmark clickbench_partitionedResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/introduce-rle-parquet-flag (35dd8d8) to 26b40dd (merge-base) diff Run configurationrun benchmark tpchResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/introduce-rle-parquet-flag (35dd8d8) to 26b40dd (merge-base) diff Run configurationrun benchmark tpcdsResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-T-kid/introduce-rle-parquet-flag (35dd8d8) to 26b40dd (merge-base) diff Run configurationrun benchmark tpchCPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-T-kid/introduce-rle-parquet-flag (35dd8d8) to 26b40dd (merge-base) diff Run configurationrun benchmark tpcdsCPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-T-kid/introduce-rle-parquet-flag (35dd8d8) to 26b40dd (merge-base) diff Run configurationrun benchmark clickbench_partitionedCPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
35dd8d8 to
28458b7
Compare
3690c2d to
d0003c9
Compare
d0003c9 to
6b75dcf
Compare
|
thank you for the review @kumarUjjawal, I implemented all of you suggestions in 6b75dcf. everything related to the per-column allowlist has been removed |
…ssion in schema coercion - Drop rle_column_allowlist from ParquetFormat and DFParquetMetadata: no callers existed and new public API is hard to remove post-release - Remove Dictionary arm from transform_schema_to_view and transform_binary_to_string: those arms fired unconditionally regardless of enable_rle_to_dictionary, causing pre-existing dict columns to get wrong value types when the flag was off - Remove the unit test that covered the now-deleted transform_binary_to_string dict arm - Drop Utf8View/BinaryView arms added to common_dictionary_value_type: the parquet reader never produces view types in the physical file schema so they were dead code - Clean up parquet_rle_to_dictionary.slt: remove redundant SET and spurious RESET
6b75dcf to
b7c4ea7
Compare
a large portion of this PR is test!
Which issue does this PR close?
Rationale for this change
When DataFusion reads a parquet file with dictionary-encoded string or binary columns, it currently decodes the dictionary and returns plain Utf8/Binary arrays, discarding the encoding. For low-cardinality columns (status, country, category, etc.) this doesn't take full advantage of the compacted format parquet gives the engine Preserving the dictionary encoding reduces memory usage and can improve aggregation performance on these columns.
What changes are included in this PR?
Adds
datafusion.execution.parquet.enable_rle_to_dictionarywith a default offalse.When enabled for inferred-schema Parquet tables, DataFusion inspects Parquet footer metadata and promotes top-level string/binary columns with dictionary pages to Arrow dictionary types. Mixed dictionary/plain files are normalized before schema merge when the value types are compatible. At scan time, the parquet opener passes the promoted schema to arrow-rs so those columns can be read as dictionary arrays directly.
Tables with a user-supplied schema are not promoted because DataFusion does not use footer metadata to infer their schema.
Are these changes tested?
yes.
datafusion/sqllogictest/test_files/parquet_rle_to_dictionary.sltdatafusion/datasource-parquet/src/schema_coercion.rsuniform_dict_schemas_respects_value_type_familieschecks that mixed file schemas are normalized only across compatible string/binary families.rle_schema_coercion_respects_dictionary_value_typechecks scan-time coercion into dictionary types, including incompatible cases and the flag-off path.datafusion/datasource-parquet/src/opener/mod.rstest_rle_binary_column_promotionverifies the opener passes a promotedDictionary(Int32, Binary)schema to arrow-rs so binary columns can be read as dictionary arrays directly.Are there any user-facing changes?
New session config option:
SET datafusion.execution.parquet.enable_rle_to_dictionary = true. Default is false so existing behavior is unchanged.