[fix](fe) Avoid reusing dropped column unique ids - #67239
Draft
Yukang-Lian wants to merge 1 commit into
Draft
Conversation
### What problem does this PR solve? Issue Number: None Related PR: apache#63173 Problem Summary: Light schema change uses column unique IDs to match FE slots with columns stored in existing segments. Some supported legacy metadata can contain a maxColUniqueId lower than an ID that is still present in the current schema. The DROP COLUMN path replaced the index schema before preserving its unique-ID high-water mark. Dropping the highest-ID column could therefore lose that historical ID, and a later ADD COLUMN could assign it again. Existing segments could then be interpreted as containing values for the new column. This change normalizes maxColUniqueId when metadata is deserialized and preserves the maximum from both the old and new schemas during light schema changes. It prevents future drops from losing a high-water mark that is still available in current metadata. It cannot reconstruct an ID that was already absent from both the schema and maxColUniqueId before upgrade. ### Release note Prevent future light schema changes from reusing dropped column unique IDs when a legacy high-water mark is stale. Metadata that had already lost its historical high-water mark before upgrade is not repaired by this change. ### Check List (For Author) - Test: Regression test / Unit Test - `./run-fe-ut.sh --run 'org.apache.doris.alter.SchemaChangeHandlerTest#testReaddDroppedValueColumnUsesNewUniqueId,org.apache.doris.catalog.MaterializedIndexMetaTest'` - `LOCAL_DORIS_PATH=$PWD/.doris-compose ./run-regression-test.sh --run -d fault_injection_p0 -s test_readd_dropped_column_unique_id -image doris-pr63173-test:latest -runMode not_cloud` - `NODE_OPTIONS=--openssl-legacy-provider ./build.sh --fe` - Behavior changed: Yes. Light schema changes preserve the column unique-ID high-water mark so newly added columns do not reuse IDs still known to the metadata. - Does this need documentation: No Co-authored-by: Siyang Tang <tangsiyang@selectdb.com>
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Collaborator
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 16610 ms |
Contributor
TPC-DS: Total hot run time: 81548 ms |
Contributor
ClickBench: Total hot run time: 14.63 s |
Yukang-Lian
marked this pull request as draft
September 10, 2026 03:18
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: #63173
Problem Summary:
Light schema change uses column unique IDs to match FE slots with columns stored
in existing segments. Some supported legacy metadata can contain a
maxColUniqueIdlower than an ID that is still present in the current schema.The
DROP COLUMNpath replaced the index schema before preserving its unique-IDhigh-water mark. Dropping the highest-ID column could therefore lose that
historical ID, and a later
ADD COLUMNcould assign it again. Existing segmentscould then be interpreted as containing values for the new column.
This change normalizes
maxColUniqueIdwhen metadata is deserialized andpreserves the maximum from both the old and new schemas during light schema
changes. It prevents future drops from losing a high-water mark that is still
available in current metadata.
This is a preventive fix for metadata whose high-water mark is still inferable.
It cannot reconstruct an ID that was already absent from both the schema and
maxColUniqueIdbefore upgrade.Release note
Prevent future light schema changes from reusing dropped column unique IDs when
a legacy high-water mark is stale. Metadata that had already lost its historical
high-water mark before upgrade is not repaired by this change.
Check List (For Author)
Test
and re-adds a column, verifies the old row returns
NULL, and verifiesa new row returns the newly written value.
LOCAL_DORIS_PATH=$PWD/.doris-compose ./run-regression-test.sh --run -d fault_injection_p0 -s test_readd_dropped_column_unique_id -image doris-pr63173-test:latest -runMode not_cloud.coverage for repairing a stale maximum while preserving a higher one.
./run-fe-ut.sh --run 'org.apache.doris.alter.SchemaChangeHandlerTest#testReaddDroppedValueColumnUsesNewUniqueId,org.apache.doris.catalog.MaterializedIndexMetaTest'.NODE_OPTIONS=--openssl-legacy-provider ./build.sh --fe.Behavior changed:
mark so newly added columns do not reuse IDs still known to the metadata.
Does this need documentation?
Check List (For Reviewer who merge this PR)