Skip to content

[fix](fe) Reject MOW key additions with sync MVs - #67245

Open
Yukang-Lian wants to merge 1 commit into
apache:masterfrom
Yukang-Lian:codex/fix-mow-sync-mv-key-column
Open

[fix](fe) Reject MOW key additions with sync MVs#67245
Yukang-Lian wants to merge 1 commit into
apache:masterfrom
Yukang-Lian:codex/fix-mow-sync-mv-key-column

Conversation

@Yukang-Lian

Copy link
Copy Markdown
Collaborator

What problem does this PR solve?

Issue Number: None

Related PR: #62875

Problem Summary: Adding a key column to a merge-on-write unique table updates every visible materialized index schema without updating its synchronous materialized view definition. Queries rewritten to that index can then decode stored rows with the wrong schema and return incorrect values.

Reject this unsupported schema change while a visible synchronous materialized view or rollup exists. Users can drop the materialized index, add the key column, and recreate it. Value-column additions remain unchanged.

Release note

Reject adding key columns to merge-on-write unique tables while synchronous materialized views or rollups exist.

Check List (For Author)

  • Test: Unit Test
    • ./run-fe-ut.sh --run org.apache.doris.alter.SchemaChangeHandlerTest#testUniqAddKeyColumnWithSyncMv+testUniqAddKeyColumnWithRollup (2 tests passed)
  • Behavior changed: Yes (unsafe key-column additions are rejected until synchronous materialized views or rollups are dropped)
  • Does this need documentation: No

Co-authored-by: Siyang Tang tangsiyang@selectdb.com

### What problem does this PR solve?

Issue Number: None

Related PR: apache#62875

Problem Summary: Adding a key column to a merge-on-write unique table updates every visible materialized index schema without updating its synchronous materialized view definition. Queries rewritten to that index can then decode stored rows with the wrong schema and return incorrect values.

Reject this unsupported schema change while a visible synchronous materialized view or rollup exists. Users can drop the materialized index, add the key column, and recreate it. Value-column additions remain unchanged.

### Release note

Reject adding key columns to merge-on-write unique tables while synchronous materialized views or rollups exist.

### Check List (For Author)

- Test: Unit Test
    - ./run-fe-ut.sh --run org.apache.doris.alter.SchemaChangeHandlerTest#testUniqAddKeyColumnWithSyncMv+testUniqAddKeyColumnWithRollup (2 tests passed)
- Behavior changed: Yes (unsafe key-column additions are rejected until synchronous materialized views or rollups are dropped)
- Does this need documentation: No

Co-authored-by: Siyang Tang <tangsiyang@selectdb.com>
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Yukang-Lian

Copy link
Copy Markdown
Collaborator Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16997 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2b1a16374882d6d1215fe854bc5a7dff8fbb7e89, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17567	3042	3022	3022
q2	2117	268	238	238
q3	10192	843	501	501
q4	4668	249	205	205
q5	7676	555	385	385
q6	135	114	98	98
q7	524	507	388	388
q8	9256	938	946	938
q9	3485	2419	2410	2410
q10	6491	855	727	727
q11	397	197	182	182
q12	615	256	197	197
q13	18137	1527	1183	1183
q14	161	155	141	141
q15	q16	438	400	374	374
q17	1331	909	852	852
q18	3147	2330	2329	2329
q19	1120	901	755	755
q20	357	279	198	198
q21	4837	1645	1835	1645
q22	332	270	229	229
Total cold run time: 92983 ms
Total hot run time: 16997 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3370	3286	3307	3286
q2	517	393	371	371
q3	2250	2329	2173	2173
q4	1219	1192	903	903
q5	2220	2167	2163	2163
q6	169	124	88	88
q7	1051	935	894	894
q8	1617	1422	1411	1411
q9	3233	3234	3206	3206
q10	1911	1837	1666	1666
q11	362	273	261	261
q12	455	437	352	352
q13	1496	1560	1174	1174
q14	178	178	155	155
q15	q16	396	409	353	353
q17	3667	3369	3293	3293
q18	4964	4569	4913	4569
q19	960	869	857	857
q20	1052	985	824	824
q21	3872	3294	3171	3171
q22	404	352	323	323
Total cold run time: 35363 ms
Total hot run time: 31493 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 83615 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2b1a16374882d6d1215fe854bc5a7dff8fbb7e89, data reload: false

query5	4269	419	333	333
query6	388	146	131	131
query7	4945	436	228	228
query8	302	142	118	118
query9	8690	2899	2910	2899
query10	404	247	187	187
query11	5386	1033	930	930
query12	138	73	70	70
query13	1196	476	353	353
query14	6118	2255	2150	2150
query14_1	2034	2030	2003	2003
query15	171	128	115	115
query16	906	368	346	346
query17	818	453	374	374
query18	2343	335	244	244
query19	159	140	111	111
query20	73	71	72	71
query21	202	103	90	90
query22	5612	5389	5353	5353
query23	6860	6349	6393	6349
query23_1	6308	6254	6249	6249
query24	7320	1131	796	796
query24_1	796	772	803	772
query25	444	309	260	260
query26	1230	237	132	132
query27	2774	417	259	259
query28	4683	1498	1508	1498
query29	939	447	358	358
query30	244	160	133	133
query31	825	413	344	344
query32	132	75	72	72
query33	469	236	188	188
query34	988	840	491	491
query35	427	444	337	337
query36	570	559	529	529
query37	118	77	67	67
query38	1020	864	834	834
query39	504	517	485	485
query39_1	461	471	472	471
query40	200	90	74	74
query41	56	50	53	50
query42	78	72	73	72
query43	238	239	209	209
query44	1006	541	567	541
query45	110	108	97	97
query46	760	848	533	533
query47	765	769	725	725
query48	287	305	229	229
query49	539	244	192	192
query50	699	273	188	188
query51	8078	8050	8427	8050
query52	69	67	58	58
query53	191	198	150	150
query54	225	178	176	176
query55	85	67	58	58
query56	190	159	162	159
query57	685	694	682	682
query58	183	166	180	166
query59	1258	1254	1143	1143
query60	250	193	190	190
query61	114	114	117	114
query62	342	213	184	184
query63	172	154	139	139
query64	2771	720	569	569
query65	1693	1655	1585	1585
query66	1778	253	207	207
query67	10156	10002	10046	10002
query68	3046	1259	763	763
query69	345	219	195	195
query70	673	620	620	620
query71	253	183	161	161
query72	2362	1735	1609	1609
query73	653	585	345	345
query74	2015	1251	1166	1166
query75	1193	1116	985	985
query76	2371	763	532	532
query77	259	260	207	207
query78	3984	3821	3303	3303
query79	2282	886	588	588
query80	1555	338	285	285
query81	478	161	135	135
query82	627	121	102	102
query83	282	207	194	194
query84	293	113	89	89
query85	827	392	291	291
query86	387	171	163	163
query87	1023	991	921	921
query88	2768	2106	2123	2106
query89	289	198	173	173
query90	1991	133	133	133
query91	130	124	95	95
query92	83	68	66	66
query93	1422	1037	710	710
query94	630	269	223	223
query95	544	330	232	232
query96	840	594	279	279
query97	1074	1139	1038	1038
query98	146	144	133	133
query99	423	355	315	315
Total cold run time: 178931 ms
Total hot run time: 83615 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.88 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2b1a16374882d6d1215fe854bc5a7dff8fbb7e89, data reload: false

query1	0.01	0.01	0.00
query2	0.08	0.04	0.04
query3	0.25	0.11	0.11
query4	1.60	0.10	0.10
query5	0.16	0.16	0.18
query6	1.25	0.68	0.71
query7	0.03	0.00	0.00
query8	0.04	0.04	0.03
query9	0.29	0.22	0.21
query10	0.35	0.36	0.36
query11	0.16	0.12	0.12
query12	0.14	0.12	0.11
query13	0.30	0.30	0.31
query14	0.48	0.46	0.46
query15	0.37	0.36	0.35
query16	0.22	0.24	0.21
query17	0.70	0.74	0.71
query18	0.18	0.17	0.16
query19	1.24	1.16	1.14
query20	0.01	0.01	0.01
query21	15.44	0.15	0.11
query22	5.05	0.05	0.04
query23	16.19	0.25	0.12
query24	3.02	0.31	0.24
query25	0.12	0.04	0.04
query26	0.76	0.16	0.13
query27	0.04	0.04	0.03
query28	3.66	0.60	0.27
query29	12.44	3.19	2.55
query30	0.25	0.11	0.12
query31	2.76	0.34	0.17
query32	3.55	0.32	0.24
query33	1.41	1.56	1.51
query34	15.37	2.28	1.81
query35	1.77	1.74	1.75
query36	0.48	0.29	0.28
query37	0.06	0.04	0.04
query38	0.05	0.03	0.03
query39	0.03	0.03	0.02
query40	0.12	0.08	0.08
query41	0.07	0.02	0.02
query42	0.04	0.02	0.03
query43	0.03	0.02	0.03
Total cold run time: 90.57 s
Total hot run time: 14.88 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 1.10% (5/453) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants