Skip to content

feat: add narrow group-contiguous source property - #24698

Draft
xavlee wants to merge 3 commits into
apache:mainfrom
xavlee:feat/issue-24438-group-contiguous-property
Draft

feat: add narrow group-contiguous source property#24698
xavlee wants to merge 3 commits into
apache:mainfrom
xavlee:feat/issue-24438-group-contiguous-property

Conversation

@xavlee

@xavlee xavlee commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR relate to?

Rationale for this change

GroupCompletionMode separates an aggregate's runtime group-completion mechanism from InputOrderMode, but aggregate planning still needs independent evidence for completion on unsorted input. An input may be Linear because its group values are not ordered while still making every group complete before the next one begins.

The relevant guarantee is that every distinct value of a complete expression tuple appears in a single contiguous run within each output partition. It does not require the tuple values themselves to be sorted:

AAABBBCCCC -> sorted and group-contiguous
CCCAAAABBB -> unsorted and group-contiguous
AAABBBAACC -> not group-contiguous (A reappears after being introduced)

This PR introduces group_contiguous_exprs as the physical-plan assertion for that guarantee. A source advertises the complete tuple it knows to be contiguous, DataSourceExec caches the assertion in PlanProperties, and operators propagate it when they preserve the relevant row sequence and expression mapping. ProjectionExec retains the assertion only when it can map every expression in the tuple.

The final PR in the stack uses an exact match between this assertion and the aggregate grouping tuple to select GroupCompletionMode::Full for a Linear input.

Property contract

group_contiguous_exprs represents one complete composite tuple. Within each output partition, all rows for every distinct tuple occupy at most one contiguous range. Tuple equality follows GROUP BY semantics, and tuple values may occur in arbitrary order.

This is a correctness assertion: a source declaration must describe every output stream accurately because a consumer may emit a completed group before end of input.

What changes are included in this PR?

The assertion follows this lifecycle:

  1. DataSource::group_contiguous_exprs declares the complete tuple; the empty vector represents no assertion.
  2. DataSourceExec stores the declaration in its PlanProperties.
  3. PlanProperties owns the cached tuple and exposes a builder and accessor through ExecutionPlanProperties.
  4. ProjectionExec uses all-or-nothing tuple mapping: a complete mapping stores the projected tuple, while every incomplete mapping stores the empty assertion.
  5. CooperativeExec, BufferExec, and ScalarSubqueryExec preserve the assertion by copying the complete property cache.
  6. Operators that construct fresh PlanProperties begin with the empty assertion.
  7. Child replacement uses ordinary property-cache identity to select Keep or Recompute.

Stack

  1. #24737 — test: cover unsorted contiguous groups in one partition
  2. #24697 — refactor: separate aggregate group completion from input ordering
  3. #24698 — feat: add narrow group-contiguous source propertythis PR
  4. #24497 — feat: stream exact group-contiguous aggregates

Are these changes tested?

Tests cover:

  • source declaration and storage in DataSourceExec;
  • complete and incomplete projection mapping;
  • property-cache identity during child replacement;
  • preservation through CooperativeExec, BufferExec, and ScalarSubqueryExec;
  • clearing the assertion at a row-changing operator; and
  • the FFI plan-property snapshot.

Are there any user-facing changes?

DataSource implementations can declare one complete group-contiguous tuple through group_contiguous_exprs. Custom physical plans can attach the same assertion with PlanProperties::with_group_contiguous_exprs.

Review this layer

View only this PR layer

@codecov-commenter

codecov-commenter commented Aug 26, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 70.75472% with 93 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.47%. Comparing base (ee59f62) to head (72d66b3).
⚠️ Report is 7 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/datasource/src/source.rs 40.98% 36 Missing ⚠️
datafusion/physical-plan/src/projection.rs 79.41% 5 Missing and 9 partials ⚠️
datafusion/physical-plan/src/test.rs 62.16% 13 Missing and 1 partial ⚠️
datafusion/physical-plan/src/aggregates/mod.rs 82.00% 2 Missing and 7 partials ⚠️
datafusion/physical-plan/src/execution_plan.rs 70.00% 6 Missing ⚠️
datafusion/physical-plan/src/buffer.rs 72.72% 0 Missing and 3 partials ⚠️
datafusion/physical-plan/src/coop.rs 75.00% 0 Missing and 3 partials ⚠️
datafusion/physical-plan/src/scalar_subquery.rs 76.92% 0 Missing and 3 partials ⚠️
...tafusion/physical-plan/src/aggregates/order/mod.rs 83.33% 2 Missing ⚠️
.../aggregates/aggregate_hash_table/common_ordered.rs 50.00% 0 Missing and 1 partial ⚠️
... and 2 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #24698      +/-   ##
==========================================
+ Coverage   81.42%   81.47%   +0.04%     
==========================================
  Files        1121     1122       +1     
  Lines      402142   404524    +2382     
  Branches   402142   404524    +2382     
==========================================
+ Hits       327460   329578    +2118     
- Misses      55484    55608     +124     
- Partials    19198    19338     +140     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@xavlee
xavlee force-pushed the feat/issue-24438-group-contiguous-property branch from b2ceb1a to 6bd697f Compare August 26, 2026 19:20
@github-actions github-actions Bot added datasource Changes to the datasource crate physical-plan Changes to the physical-plan crate labels Aug 26, 2026
@xavlee
xavlee force-pushed the feat/issue-24438-group-contiguous-property branch 2 times, most recently from 4dbd0df to 8626d7d Compare August 27, 2026 19:17
@xavlee
xavlee force-pushed the feat/issue-24438-group-contiguous-property branch 2 times, most recently from 0399935 to 1e7d83a Compare August 28, 2026 17:26
@xavlee
xavlee force-pushed the feat/issue-24438-group-contiguous-property branch from 1e7d83a to 72d66b3 Compare August 28, 2026 19:45
@github-actions github-actions Bot added the ffi Changes to the ffi crate label Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

datasource Changes to the datasource crate ffi Changes to the ffi crate physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants