Skip to content

[fix](function) Preserve invalid URL decoded bytes - #67841

Merged
morrySnow merged 1 commit into
apache:masterfrom
morrySnow:fix/skip-invalid-url-decode-folding
Sep 14, 2026
Merged

morrySnow merged 1 commit into
apache:masterfrom
morrySnow:fix/skip-invalid-url-decode-folding

Conversation

@morrySnow

Copy link
Copy Markdown
Contributor

Problem

URL_DECODE returned different bytes depending on whether the expression was folded by FE. For example:

SELECT HEX(URL_DECODE('%C0%AF'));

The folded path returned EFBFBDEFBFBD, while evaluation in BE returned C0AF.

Cause

The FE executable function delegates to Java URLDecoder, whose UTF-8 conversion silently replaces malformed byte sequences with U+FFFD. The BE URL decoder has byte-level semantics and preserves each decoded %HH byte. A Java string literal therefore cannot represent the BE result losslessly when the decoded bytes are invalid UTF-8.

Fix

  • Reconstruct the byte stream produced by URL decoding and validate it with a strict UTF-8 decoder before FE constant evaluation.
  • If the decoded bytes are invalid UTF-8, abort FE evaluation so the original function remains in the plan and BE preserves the bytes.
  • Keep valid UTF-8 folding unchanged, including a legitimately encoded U+FFFD value.
  • Preserve the existing handling of malformed percent escapes.

Tests

  • Added executable-function unit coverage for %80, %C0%AF, %E0%80%80, %ED%A0%80, %FF, valid multibyte text, and encoded U+FFFD.
  • Added regression coverage comparing constant and dynamic execution paths by their hexadecimal result.
  • StringArithmeticTest: 7 tests passed.
  • Full FE build and checkstyle passed.
  • Targeted regression suite: 1/1 passed.

Problem:
URL_DECODE returned different bytes for a constant expression and the runtime path when percent escapes decoded to invalid UTF-8.

Cause:
The FE executable function used Java URLDecoder, which silently replaced malformed UTF-8 with replacement characters before constructing a string literal. The BE decoder preserves the decoded bytes.

Solution:
Reconstruct and strictly validate the decoded UTF-8 byte stream before FE evaluation. If it is invalid, leave the function expression unfolded so BE retains its byte-level behavior. Keep folding valid input and preserve malformed-percent handling.

Tests:
Add unit coverage for invalid UTF-8, valid multibyte text, and a legitimate replacement character. Add regression coverage comparing constant and runtime decoding across invalid byte sequences.
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morrySnow

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 17045 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 79ad9a2a4a8c14c885b1de80199172dc4c897fce, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17589	3075	3051	3051
q2	2122	269	222	222
q3	10213	919	515	515
q4	4670	250	202	202
q5	7679	565	391	391
q6	140	117	97	97
q7	576	499	387	387
q8	9229	955	926	926
q9	3536	2416	2503	2416
q10	6506	866	747	747
q11	403	208	182	182
q12	611	266	196	196
q13	18133	1527	1177	1177
q14	158	148	141	141
q15	q16	455	403	376	376
q17	1319	900	786	786
q18	3080	2267	2303	2267
q19	1138	960	774	774
q20	391	287	207	207
q21	5388	1756	1789	1756
q22	321	270	229	229
Total cold run time: 93657 ms
Total hot run time: 17045 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3447	3350	3348	3348
q2	503	392	386	386
q3	2264	2376	2266	2266
q4	1206	1182	912	912
q5	2208	2138	2159	2138
q6	169	120	89	89
q7	1048	930	858	858
q8	1600	1421	1408	1408
q9	3178	3166	3153	3153
q10	1925	1857	1669	1669
q11	357	276	252	252
q12	461	437	343	343
q13	1489	1568	1181	1181
q14	187	169	170	169
q15	q16	400	401	378	378
q17	3684	3400	3276	3276
q18	4933	4521	4978	4521
q19	953	881	868	868
q20	1017	996	834	834
q21	3878	3252	3198	3198
q22	383	355	333	333
Total cold run time: 35290 ms
Total hot run time: 31580 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 83330 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 79ad9a2a4a8c14c885b1de80199172dc4c897fce, data reload: false

query5	4277	434	373	373
query6	391	145	136	136
query7	4869	424	237	237
query8	290	130	123	123
query9	8694	2921	2930	2921
query10	398	236	184	184
query11	5374	1051	933	933
query12	125	77	75	75
query13	1196	433	311	311
query14	6110	2258	2137	2137
query14_1	2056	1996	1996	1996
query15	175	119	114	114
query16	935	372	359	359
query17	1056	477	372	372
query18	2361	332	236	236
query19	167	136	108	108
query20	86	71	74	71
query21	206	103	88	88
query22	5445	5420	5358	5358
query23	6789	6303	6164	6164
query23_1	6324	6399	6345	6345
query24	7358	1087	788	788
query24_1	778	789	770	770
query25	432	319	252	252
query26	1238	251	131	131
query27	2762	384	244	244
query28	4759	1501	1463	1463
query29	934	445	357	357
query30	260	157	133	133
query31	835	428	344	344
query32	129	79	75	75
query33	464	244	173	173
query34	978	795	483	483
query35	393	410	342	342
query36	575	559	541	541
query37	125	86	72	72
query38	1021	865	820	820
query39	518	503	492	492
query39_1	460	462	466	462
query40	200	88	77	77
query41	55	54	54	54
query42	78	73	78	73
query43	242	243	207	207
query44	1000	535	535	535
query45	110	106	100	100
query46	760	868	549	549
query47	786	766	711	711
query48	315	305	241	241
query49	547	244	196	196
query50	711	265	200	200
query51	8378	8301	8150	8150
query52	71	73	59	59
query53	190	198	141	141
query54	203	212	200	200
query55	72	59	64	59
query56	212	163	148	148
query57	702	681	657	657
query58	187	177	161	161
query59	1214	1263	1109	1109
query60	235	205	163	163
query61	124	129	116	116
query62	356	217	187	187
query63	170	146	145	145
query64	2763	734	600	600
query65	1637	1622	1615	1615
query66	1929	251	221	221
query67	10034	10008	9708	9708
query68	2790	1136	744	744
query69	343	226	199	199
query70	680	604	616	604
query71	254	181	165	165
query72	2257	1716	1647	1647
query73	651	602	323	323
query74	1570	1276	1153	1153
query75	1193	1107	971	971
query76	2291	753	508	508
query77	256	257	235	235
query78	4103	3790	3302	3302
query79	2369	845	587	587
query80	1561	312	271	271
query81	488	160	136	136
query82	593	120	93	93
query83	270	210	196	196
query84	256	112	93	93
query85	755	347	290	290
query86	382	186	177	177
query87	1045	992	913	913
query88	2785	2113	2111	2111
query89	273	202	172	172
query90	1987	131	133	131
query91	131	126	102	102
query92	81	73	71	71
query93	1413	1084	680	680
query94	630	250	221	221
query95	525	263	303	263
query96	799	600	293	293
query97	1128	1103	1010	1010
query98	158	135	135	135
query99	431	350	319	319
Total cold run time: 178467 ms
Total hot run time: 83330 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.83 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 79ad9a2a4a8c14c885b1de80199172dc4c897fce, data reload: false

query1	0.00	0.00	0.00
query2	0.08	0.04	0.04
query3	0.25	0.08	0.10
query4	1.60	0.09	0.10
query5	0.18	0.16	0.17
query6	1.24	0.72	0.70
query7	0.03	0.00	0.01
query8	0.04	0.03	0.03
query9	0.29	0.22	0.22
query10	0.35	0.35	0.34
query11	0.16	0.12	0.12
query12	0.14	0.12	0.12
query13	0.30	0.31	0.32
query14	0.48	0.46	0.46
query15	0.37	0.35	0.34
query16	0.22	0.21	0.22
query17	0.70	0.73	0.74
query18	0.19	0.17	0.17
query19	1.21	1.22	1.19
query20	0.01	0.01	0.01
query21	15.45	0.17	0.11
query22	5.07	0.04	0.04
query23	16.18	0.25	0.11
query24	2.99	0.30	0.24
query25	0.11	0.04	0.04
query26	0.77	0.17	0.13
query27	0.03	0.03	0.02
query28	3.60	0.56	0.25
query29	12.47	3.24	2.57
query30	0.26	0.11	0.12
query31	2.76	0.38	0.18
query32	3.53	0.32	0.24
query33	1.41	1.42	1.45
query34	15.37	2.24	1.80
query35	1.79	1.75	1.77
query36	0.46	0.28	0.28
query37	0.06	0.04	0.04
query38	0.04	0.03	0.03
query39	0.03	0.03	0.02
query40	0.12	0.08	0.07
query41	0.07	0.02	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 90.48 s
Total hot run time: 14.83 s

@morrySnow
morrySnow merged commit 366fd37 into apache:master Sep 14, 2026
34 of 36 checks passed
@morrySnow
morrySnow deleted the fix/skip-invalid-url-decode-folding branch September 14, 2026 02:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants