Skip to content

[fix](exec) Return Thrift DATETIME as naive Arrow timestamp - #67232

Open
Mryange wants to merge 1 commit into
apache:masterfrom
Mryange:fix-thrift-datetime-naive
Open

[fix](exec) Return Thrift DATETIME as naive Arrow timestamp#67232
Mryange wants to merge 1 commit into
apache:masterfrom
Mryange:fix-thrift-datetime-naive

Conversation

@Mryange

@Mryange Mryange commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Thrift external scanner responses encoded Doris DATETIME values with the BE session timezone in the Arrow schema. This made clients interpret wall-clock values as instants and could introduce an eight-hour offset. The MemoryScratch sink now uses the existing naive-timestamp conversion for DATETIMEV2, while preserving timezone-aware TIMESTAMPTZ behavior.

Release note

DATETIME values returned through the Thrift external scanner are represented as timezone-naive Arrow timestamps.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Mryange

Mryange commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 17052 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a3b92a3f4b1f7d4df4cad571b82f4fcdaae66a24, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17612	3038	3002	3002
q2	2080	254	229	229
q3	10250	890	525	525
q4	4665	246	207	207
q5	7672	552	388	388
q6	134	111	92	92
q7	526	503	380	380
q8	9243	974	941	941
q9	3479	2428	2398	2398
q10	6516	841	713	713
q11	399	196	177	177
q12	607	254	197	197
q13	18134	1514	1141	1141
q14	155	149	141	141
q15	q16	432	393	369	369
q17	1348	913	843	843
q18	2994	2236	2246	2236
q19	1103	874	793	793
q20	380	281	198	198
q21	4826	1848	1880	1848
q22	326	267	234	234
Total cold run time: 92881 ms
Total hot run time: 17052 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3368	3348	3331	3331
q2	502	383	372	372
q3	2182	2293	2214	2214
q4	1185	1158	880	880
q5	2159	2129	2112	2112
q6	175	116	89	89
q7	1018	941	850	850
q8	1581	1391	1407	1391
q9	3140	3084	3060	3060
q10	1844	1792	1598	1598
q11	352	266	248	248
q12	448	427	343	343
q13	1506	1540	1164	1164
q14	184	177	152	152
q15	q16	386	401	349	349
q17	3589	3254	3244	3244
q18	4824	4424	4787	4424
q19	999	805	903	805
q20	1001	1021	818	818
q21	3837	3106	3274	3106
q22	390	340	320	320
Total cold run time: 34670 ms
Total hot run time: 30870 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 81695 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a3b92a3f4b1f7d4df4cad571b82f4fcdaae66a24, data reload: false

query5	4285	420	343	343
query6	395	134	119	119
query7	4936	433	236	236
query8	292	121	116	116
query9	8670	2883	2875	2875
query10	386	217	184	184
query11	5384	1033	910	910
query12	124	72	71	71
query13	1195	447	329	329
query14	6145	2188	2059	2059
query14_1	1954	1938	1943	1938
query15	175	118	109	109
query16	921	369	353	353
query17	796	472	390	390
query18	2326	323	253	253
query19	170	142	111	111
query20	69	66	69	66
query21	198	100	90	90
query22	5302	5291	5263	5263
query23	6530	6141	5990	5990
query23_1	6210	6126	6128	6126
query24	7548	1142	810	810
query24_1	812	807	798	798
query25	456	326	273	273
query26	1269	237	132	132
query27	2935	398	292	292
query28	4894	1529	1531	1529
query29	981	450	351	351
query30	245	148	125	125
query31	1086	412	323	323
query32	121	70	69	69
query33	450	212	191	191
query34	1224	839	473	473
query35	390	410	355	355
query36	567	547	526	526
query37	122	80	68	68
query38	999	846	821	821
query39	510	517	494	494
query39_1	472	483	485	483
query40	218	88	73	73
query41	50	49	52	49
query42	72	69	69	69
query43	238	243	206	206
query44	1021	560	559	559
query45	113	105	98	98
query46	804	820	538	538
query47	764	761	698	698
query48	315	307	233	233
query49	541	223	176	176
query50	726	259	196	196
query51	7952	7954	7972	7954
query52	64	64	60	60
query53	186	195	140	140
query54	209	166	166	166
query55	75	58	52	52
query56	173	153	161	153
query57	688	623	601	601
query58	195	154	164	154
query59	1134	1125	1028	1028
query60	238	182	169	169
query61	111	112	106	106
query62	339	201	175	175
query63	171	135	137	135
query64	2591	673	530	530
query65	1569	1565	1532	1532
query66	1762	252	193	193
query67	10101	9813	9607	9607
query68	3031	1166	737	737
query69	357	225	185	185
query70	655	599	610	599
query71	240	175	171	171
query72	2329	1690	1597	1597
query73	643	565	333	333
query74	1979	1224	1119	1119
query75	1199	1114	951	951
query76	2367	745	542	542
query77	250	261	206	206
query78	3855	3630	3230	3230
query79	2785	824	589	589
query80	1620	349	285	285
query81	512	153	137	137
query82	646	121	99	99
query83	276	211	190	190
query84	294	109	87	87
query85	856	354	296	296
query86	478	169	171	169
query87	1004	973	889	889
query88	2886	2093	2085	2085
query89	283	193	171	171
query90	1967	128	125	125
query91	131	120	96	96
query92	87	69	67	67
query93	2026	1116	706	706
query94	626	251	227	227
query95	518	320	231	231
query96	816	574	274	274
query97	1041	1068	1008	1008
query98	163	133	142	133
query99	418	340	309	309
Total cold run time: 179706 ms
Total hot run time: 81695 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.45 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a3b92a3f4b1f7d4df4cad571b82f4fcdaae66a24, data reload: false

query1	0.01	0.01	0.01
query2	0.08	0.03	0.04
query3	0.24	0.11	0.11
query4	1.60	0.10	0.10
query5	0.18	0.16	0.17
query6	1.24	0.70	0.66
query7	0.04	0.01	0.01
query8	0.05	0.03	0.03
query9	0.28	0.21	0.20
query10	0.35	0.37	0.36
query11	0.16	0.11	0.12
query12	0.14	0.12	0.12
query13	0.31	0.30	0.29
query14	0.44	0.44	0.44
query15	0.37	0.34	0.35
query16	0.23	0.20	0.21
query17	0.71	0.66	0.69
query18	0.18	0.17	0.17
query19	1.22	1.14	1.15
query20	0.01	0.01	0.01
query21	15.47	0.16	0.11
query22	5.09	0.05	0.05
query23	16.18	0.24	0.11
query24	3.01	0.32	0.24
query25	0.08	0.04	0.03
query26	0.78	0.17	0.13
query27	0.03	0.04	0.03
query28	3.62	0.52	0.27
query29	12.47	3.18	2.55
query30	0.25	0.10	0.11
query31	2.75	0.38	0.17
query32	3.54	0.32	0.23
query33	1.46	1.33	1.37
query34	15.38	2.12	1.77
query35	1.72	1.70	1.69
query36	0.46	0.30	0.29
query37	0.06	0.04	0.03
query38	0.04	0.02	0.02
query39	0.03	0.03	0.02
query40	0.12	0.08	0.07
query41	0.07	0.02	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.02
Total cold run time: 90.51 s
Total hot run time: 14.45 s

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 100.00% (3/3) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 62.70% (29321/46765)
Line Coverage 47.68% (306698/643270)
Region Coverage 43.34% (247825/571847)
Branch Coverage 44.88% (115311/256958)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (3/3) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.00% (34421/45293)
Line Coverage 60.96% (387623/635907)
Region Coverage 57.11% (325359/569664)
Branch Coverage 57.89% (148321/256225)

@Mryange

Mryange commented Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

run nonConcurrent

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants