Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](load) return DataQualityError when filtered rows exceeds limit #47617

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

[fix](load) return DataQualityError when filtered rows exceeds limit #47617

wants to merge 6 commits into from

Conversation

kaijchen
Copy link
Contributor

@kaijchen kaijchen commented Feb 8, 2025 •
edited
Loading

What problem does this PR solve?

Issue Number: DORIS-17408

Problem Summary:

When the load is canceled due to an excessive number of filtered rows in CsvReader, the end of stream flag is set, but an error is not returned. As a result, the following confusing message is shown to the user:

"Message": "[CANCELLED]cancelled: closed",

This PR addresses the issue by returning a DataQualityError in such cases. The updated message will be:

"Message": "[CANCELLED]cancelled: [DATA_QUALITY_ERROR]Encountered unqualified data, stop processing. cur path: .",

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Feb 8, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@kaijchen
Copy link
Contributor Author

kaijchen commented Feb 8, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31865 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f6ed8a9f3fa9264d7556cb40f036b898277103ad, data reload: false

------ Round 1 ----------------------------------
q1	17580	5212	5115	5115
q2	2049	322	186	186
q3	10371	1272	776	776
q4	10213	1018	550	550
q5	7539	2359	2362	2359
q6	178	165	132	132
q7	896	796	610	610
q8	9301	1284	1084	1084
q9	4878	4697	4659	4659
q10	6818	2319	1908	1908
q11	473	282	256	256
q12	358	351	221	221
q13	17767	3712	3128	3128
q14	221	225	211	211
q15	533	480	479	479
q16	625	643	580	580
q17	556	850	347	347
q18	6642	6240	6205	6205
q19	1219	973	557	557
q20	319	323	194	194
q21	2814	2233	2009	2009
q22	376	329	299	299
Total cold run time: 101726 ms
Total hot run time: 31865 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5233	5168	5124	5124
q2	249	331	233	233
q3	2219	2747	2342	2342
q4	1459	1862	1393	1393
q5	4264	4123	4199	4123
q6	207	167	129	129
q7	1842	1817	1671	1671
q8	2616	2674	2578	2578
q9	7262	7201	7128	7128
q10	3016	3225	2797	2797
q11	582	518	483	483
q12	693	773	647	647
q13	3379	3992	3252	3252
q14	292	290	265	265
q15	524	461	447	447
q16	678	690	654	654
q17	1111	1547	1401	1401
q18	7481	7436	7364	7364
q19	776	849	1021	849
q20	1966	2052	1935	1935
q21	5453	4919	4795	4795
q22	623	599	557	557
Total cold run time: 51925 ms
Total hot run time: 50167 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192159 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f6ed8a9f3fa9264d7556cb40f036b898277103ad, data reload: false

query1	1326	962	956	956
query2	6221	1858	1835	1835
query3	10958	4544	4542	4542
query4	53770	26137	23596	23596
query5	5292	577	503	503
query6	338	201	200	200
query7	4917	512	296	296
query8	310	253	243	243
query9	5754	2581	2596	2581
query10	433	315	260	260
query11	15415	15269	15102	15102
query12	165	115	109	109
query13	1056	530	388	388
query14	10664	6906	6404	6404
query15	217	215	181	181
query16	7002	686	475	475
query17	1074	740	562	562
query18	1446	423	318	318
query19	217	187	154	154
query20	135	132	133	132
query21	208	136	112	112
query22	4358	4392	4363	4363
query23	34447	33653	33552	33552
query24	5704	2443	2471	2443
query25	458	462	402	402
query26	688	285	156	156
query27	1723	517	336	336
query28	2729	2450	2411	2411
query29	572	586	426	426
query30	230	206	167	167
query31	908	879	830	830
query32	71	62	56	56
query33	452	362	311	311
query34	773	891	520	520
query35	822	839	783	783
query36	955	1021	910	910
query37	126	105	86	86
query38	4374	4433	4360	4360
query39	1502	1517	1520	1517
query40	237	121	107	107
query41	51	51	48	48
query42	124	114	107	107
query43	501	528	469	469
query44	1357	803	811	803
query45	191	177	177	177
query46	877	1068	678	678
query47	1844	1931	1764	1764
query48	400	422	317	317
query49	718	567	459	459
query50	714	779	432	432
query51	4403	4369	4301	4301
query52	120	116	100	100
query53	249	287	205	205
query54	495	540	423	423
query55	91	81	80	80
query56	267	293	270	270
query57	1192	1188	1089	1089
query58	252	248	268	248
query59	2757	2703	2626	2626
query60	303	281	272	272
query61	127	122	124	122
query62	744	734	682	682
query63	236	201	199	199
query64	1651	1063	703	703
query65	3535	3368	3336	3336
query66	772	398	296	296
query67	16007	15463	15401	15401
query68	4976	791	520	520
query69	541	314	273	273
query70	1208	1150	1147	1147
query71	446	303	276	276
query72	5992	3672	3824	3672
query73	1151	726	344	344
query74	8971	8974	8975	8974
query75	3210	3177	2729	2729
query76	3745	1218	762	762
query77	566	374	280	280
query78	10446	10455	9717	9717
query79	1781	802	595	595
query80	657	575	459	459
query81	520	281	254	254
query82	222	156	131	131
query83	174	172	159	159
query84	291	91	77	77
query85	744	346	307	307
query86	348	309	286	286
query87	4554	4711	4370	4370
query88	3134	2321	2200	2200
query89	402	316	284	284
query90	1802	196	199	196
query91	135	140	109	109
query92	78	61	57	57
query93	2328	1019	574	574
query94	677	404	291	291
query95	351	267	256	256
query96	473	568	265	265
query97	2803	2892	2790	2790
query98	254	203	198	198
query99	1315	1419	1315	1315
Total cold run time: 292320 ms
Total hot run time: 192159 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.22 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f6ed8a9f3fa9264d7556cb40f036b898277103ad, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.05	0.04
query3	0.24	0.06	0.07
query4	1.62	0.10	0.10
query5	0.41	0.40	0.42
query6	1.15	0.65	0.66
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.60	0.51	0.54
query10	0.59	0.58	0.58
query11	0.16	0.10	0.11
query12	0.15	0.11	0.12
query13	0.62	0.59	0.61
query14	2.77	2.82	2.69
query15	0.92	0.86	0.86
query16	0.37	0.37	0.38
query17	1.01	1.05	1.03
query18	0.21	0.20	0.19
query19	1.95	1.80	1.98
query20	0.01	0.01	0.02
query21	15.37	0.92	0.55
query22	0.75	1.25	0.64
query23	14.94	1.41	0.60
query24	6.87	2.21	0.55
query25	0.49	0.23	0.12
query26	0.61	0.17	0.14
query27	0.05	0.05	0.05
query28	8.83	0.89	0.44
query29	12.61	3.95	3.26
query30	0.25	0.09	0.06
query31	2.82	0.60	0.39
query32	3.23	0.54	0.48
query33	2.99	3.06	3.04
query34	15.84	5.17	4.47
query35	4.55	4.52	4.49
query36	0.66	0.49	0.48
query37	0.09	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.14
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 104.3 s
Total hot run time: 30.22 s

@kaijchen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31535 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bab15dc8371f4b232255846bed2333316c82a995, data reload: false

------ Round 1 ----------------------------------
q1	17598	5168	5044	5044
q2	2051	324	170	170
q3	10368	1355	701	701
q4	10210	994	552	552
q5	7493	2418	2418	2418
q6	186	165	134	134
q7	892	755	618	618
q8	9312	1308	1092	1092
q9	4831	4543	4556	4543
q10	6818	2303	1922	1922
q11	481	276	252	252
q12	347	351	217	217
q13	17766	3662	3063	3063
q14	225	222	207	207
q15	493	468	458	458
q16	615	608	576	576
q17	567	883	336	336
q18	6641	6277	6320	6277
q19	1201	944	548	548
q20	309	329	192	192
q21	2841	2161	1911	1911
q22	362	328	304	304
Total cold run time: 101607 ms
Total hot run time: 31535 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5127	5071	5110	5071
q2	230	333	242	242
q3	2166	2707	2265	2265
q4	1419	1811	1387	1387
q5	4260	4143	4111	4111
q6	205	161	125	125
q7	1879	1796	1630	1630
q8	2591	2736	2590	2590
q9	7314	7149	7219	7149
q10	3041	3131	2749	2749
q11	575	490	494	490
q12	648	809	628	628
q13	3460	3890	3266	3266
q14	272	304	279	279
q15	544	478	451	451
q16	630	682	633	633
q17	1135	1564	1358	1358
q18	7493	7398	7184	7184
q19	777	808	858	808
q20	1923	2019	1852	1852
q21	5365	4899	4807	4807
q22	640	599	571	571
Total cold run time: 51694 ms
Total hot run time: 49646 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 183577 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bab15dc8371f4b232255846bed2333316c82a995, data reload: false

query1	983	367	388	367
query2	6531	1986	1936	1936
query3	6792	206	206	206
query4	26834	23446	23499	23446
query5	4329	666	478	478
query6	293	200	181	181
query7	4598	506	307	307
query8	306	235	230	230
query9	8599	2496	2496	2496
query10	479	301	246	246
query11	15282	15268	14954	14954
query12	153	106	101	101
query13	1662	510	375	375
query14	9084	6082	6486	6082
query15	207	192	179	179
query16	7150	623	475	475
query17	963	717	568	568
query18	1941	408	322	322
query19	193	190	161	161
query20	124	117	116	116
query21	209	137	105	105
query22	4359	4336	4549	4336
query23	34590	33775	32819	32819
query24	7718	2369	2373	2369
query25	565	486	390	390
query26	1231	274	153	153
query27	2105	495	326	326
query28	3947	2395	2351	2351
query29	772	531	415	415
query30	231	195	155	155
query31	928	857	781	781
query32	72	61	60	60
query33	552	349	287	287
query34	774	844	513	513
query35	807	806	742	742
query36	977	989	885	885
query37	113	100	76	76
query38	4184	4224	4033	4033
query39	1431	1390	1417	1390
query40	206	118	99	99
query41	51	48	49	48
query42	123	100	101	100
query43	503	504	491	491
query44	1278	796	786	786
query45	176	168	158	158
query46	860	1033	640	640
query47	1769	1796	1726	1726
query48	375	403	304	304
query49	766	491	392	392
query50	686	740	408	408
query51	4197	4178	4095	4095
query52	112	110	94	94
query53	230	254	187	187
query54	474	498	403	403
query55	79	77	78	77
query56	271	282	253	253
query57	1101	1088	1059	1059
query58	243	230	237	230
query59	2801	2866	2684	2684
query60	291	262	265	262
query61	116	140	140	140
query62	815	743	653	653
query63	222	191	193	191
query64	4603	1081	772	772
query65	3244	3100	3145	3100
query66	1156	396	301	301
query67	15889	15586	15502	15502
query68	8145	767	511	511
query69	512	295	271	271
query70	1231	1144	1092	1092
query71	416	286	267	267
query72	5743	3517	3696	3517
query73	725	707	348	348
query74	9043	9122	8962	8962
query75	3336	3199	2706	2706
query76	3328	1181	734	734
query77	602	359	340	340
query78	9958	10088	9387	9387
query79	2518	821	595	595
query80	630	525	446	446
query81	498	282	234	234
query82	669	126	95	95
query83	167	173	148	148
query84	280	94	139	94
query85	772	338	292	292
query86	339	313	274	274
query87	4361	4522	4292	4292
query88	3652	2201	2178	2178
query89	391	312	283	283
query90	1941	196	191	191
query91	139	139	107	107
query92	75	63	57	57
query93	1701	994	577	577
query94	713	398	277	277
query95	357	268	254	254
query96	476	578	275	275
query97	2797	2852	2751	2751
query98	233	205	200	200
query99	1367	1381	1255	1255
Total cold run time: 271820 ms
Total hot run time: 183577 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.84 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bab15dc8371f4b232255846bed2333316c82a995, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.03	0.04
query3	0.24	0.07	0.06
query4	1.62	0.11	0.11
query5	0.43	0.43	0.41
query6	1.16	0.66	0.66
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.59	0.53	0.51
query10	0.56	0.58	0.57
query11	0.15	0.10	0.10
query12	0.14	0.11	0.11
query13	0.63	0.59	0.61
query14	2.70	2.68	2.72
query15	0.93	0.85	0.84
query16	0.39	0.40	0.38
query17	1.04	1.02	1.02
query18	0.22	0.19	0.19
query19	1.90	1.81	2.00
query20	0.01	0.01	0.01
query21	15.36	0.91	0.55
query22	0.74	1.09	0.66
query23	15.05	1.39	0.59
query24	7.05	2.23	1.19
query25	0.47	0.27	0.06
query26	0.53	0.18	0.14
query27	0.05	0.05	0.05
query28	9.80	0.82	0.43
query29	12.56	3.95	3.29
query30	0.24	0.11	0.06
query31	2.81	0.58	0.38
query32	3.22	0.53	0.46
query33	2.98	3.02	3.06
query34	16.17	5.10	4.52
query35	4.49	4.49	4.55
query36	0.68	0.50	0.48
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.13	0.13
query41	0.09	0.02	0.02
query42	0.04	0.02	0.03
query43	0.03	0.04	0.03
Total cold run time: 105.6 s
Total hot run time: 30.84 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 42.95% (11301/26312)
Line Coverage: 32.88% (94894/288588)
Region Coverage: 32.05% (48667/151830)
Branch Coverage: 27.91% (24546/87938)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bab15dc8371f4b232255846bed2333316c82a995_bab15dc8371f4b232255846bed2333316c82a995/report/index.html

@kaijchen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31806 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 38446efd79eea61f0a81c364638e1646e427d151, data reload: false

------ Round 1 ----------------------------------
q1	17604	5279	5089	5089
q2	2051	304	163	163
q3	10408	1226	763	763
q4	10206	1002	536	536
q5	7519	2401	2324	2324
q6	198	162	132	132
q7	904	737	603	603
q8	9309	1268	1109	1109
q9	4857	4741	4626	4626
q10	6825	2307	1861	1861
q11	466	284	244	244
q12	354	349	224	224
q13	17802	3694	3311	3311
q14	227	238	218	218
q15	507	466	450	450
q16	657	608	591	591
q17	582	877	357	357
q18	7282	6184	6175	6175
q19	1224	949	550	550
q20	316	321	205	205
q21	2903	2256	1970	1970
q22	354	329	305	305
Total cold run time: 102555 ms
Total hot run time: 31806 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5174	5163	5151	5151
q2	242	326	232	232
q3	2145	2691	2277	2277
q4	1412	1814	1389	1389
q5	4242	4115	4089	4089
q6	214	165	123	123
q7	1866	1815	1642	1642
q8	2602	2662	2584	2584
q9	7199	7053	7099	7053
q10	3039	3221	2793	2793
q11	573	507	519	507
q12	710	779	616	616
q13	3509	3850	3321	3321
q14	273	320	283	283
q15	494	479	476	476
q16	639	675	661	661
q17	1137	1585	1364	1364
q18	7583	7438	7312	7312
q19	813	836	876	836
q20	1977	2021	1892	1892
q21	5485	5056	4578	4578
q22	654	569	546	546
Total cold run time: 51982 ms
Total hot run time: 49725 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190925 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 38446efd79eea61f0a81c364638e1646e427d151, data reload: false

query1	1332	944	949	944
query2	6232	1913	1873	1873
query3	10937	4573	4499	4499
query4	53763	24100	23421	23421
query5	4978	550	477	477
query6	337	201	200	200
query7	4917	535	291	291
query8	311	256	231	231
query9	5206	2518	2501	2501
query10	424	316	254	254
query11	15226	15152	15016	15016
query12	157	109	108	108
query13	1044	531	391	391
query14	10490	6450	6520	6450
query15	204	202	180	180
query16	6805	679	484	484
query17	1044	727	563	563
query18	1494	417	318	318
query19	201	198	162	162
query20	123	120	120	120
query21	210	134	108	108
query22	4404	4475	4319	4319
query23	33841	33342	33316	33316
query24	5611	2485	2465	2465
query25	485	499	436	436
query26	616	284	163	163
query27	1545	502	334	334
query28	3060	2424	2400	2400
query29	627	606	459	459
query30	215	196	166	166
query31	903	903	818	818
query32	80	72	69	69
query33	464	369	331	331
query34	810	901	508	508
query35	800	842	749	749
query36	950	1001	942	942
query37	133	104	76	76
query38	4264	4441	4430	4430
query39	1520	1437	1428	1428
query40	218	119	108	108
query41	57	52	49	49
query42	123	103	102	102
query43	503	540	501	501
query44	1328	856	808	808
query45	190	172	171	171
query46	880	1084	662	662
query47	1835	1865	1789	1789
query48	398	429	310	310
query49	691	516	429	429
query50	740	748	418	418
query51	4321	4248	4203	4203
query52	108	104	98	98
query53	232	270	193	193
query54	474	505	439	439
query55	80	77	80	77
query56	259	279	263	263
query57	1167	1173	1112	1112
query58	245	237	248	237
query59	2854	2876	2791	2791
query60	288	275	263	263
query61	124	122	116	116
query62	760	734	674	674
query63	235	197	194	194
query64	1376	1021	701	701
query65	3371	3279	3194	3194
query66	760	388	321	321
query67	15708	15568	15158	15158
query68	6211	781	506	506
query69	554	305	262	262
query70	1227	1150	1147	1147
query71	457	299	271	271
query72	6349	3737	3714	3714
query73	1134	763	360	360
query74	9143	9211	9217	9211
query75	3339	3183	2668	2668
query76	3956	1204	767	767
query77	532	386	292	292
query78	10116	10105	9275	9275
query79	2805	806	598	598
query80	665	539	443	443
query81	494	275	240	240
query82	470	126	98	98
query83	170	169	159	159
query84	289	93	73	73
query85	755	377	305	305
query86	376	320	302	302
query87	4432	4653	4453	4453
query88	3517	2246	2200	2200
query89	400	328	293	293
query90	1804	189	193	189
query91	134	135	108	108
query92	73	59	55	55
query93	2438	1039	590	590
query94	677	401	282	282
query95	353	265	264	264
query96	483	577	267	267
query97	2796	2846	2699	2699
query98	239	214	203	203
query99	1328	1480	1250	1250
Total cold run time: 292804 ms
Total hot run time: 190925 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.67 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 38446efd79eea61f0a81c364638e1646e427d151, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.04
query3	0.24	0.07	0.07
query4	1.62	0.10	0.10
query5	0.41	0.41	0.38
query6	1.19	0.67	0.65
query7	0.02	0.02	0.02
query8	0.04	0.02	0.03
query9	0.60	0.51	0.53
query10	0.57	0.56	0.57
query11	0.16	0.11	0.11
query12	0.15	0.12	0.11
query13	0.63	0.61	0.61
query14	2.70	2.78	2.70
query15	0.92	0.86	0.85
query16	0.39	0.39	0.36
query17	1.04	1.02	1.07
query18	0.22	0.18	0.19
query19	1.91	1.75	2.01
query20	0.02	0.01	0.01
query21	15.36	0.92	0.57
query22	0.76	1.16	0.63
query23	14.99	1.42	0.65
query24	7.66	1.07	1.29
query25	0.52	0.31	0.07
query26	0.48	0.16	0.15
query27	0.05	0.04	0.04
query28	9.69	0.81	0.42
query29	12.54	3.93	3.26
query30	0.25	0.09	0.06
query31	2.82	0.59	0.38
query32	3.22	0.54	0.46
query33	2.96	3.05	3.04
query34	15.75	5.15	4.49
query35	4.53	4.51	4.57
query36	0.66	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.13
query41	0.08	0.02	0.03
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 105.68 s
Total hot run time: 30.67 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 42.94% (11299/26312)
Line Coverage: 32.88% (94880/288573)
Region Coverage: 32.05% (48662/151816)
Branch Coverage: 27.91% (24540/87936)
Coverage Report: http://coverage.selectdb-in.cc/coverage/38446efd79eea61f0a81c364638e1646e427d151_38446efd79eea61f0a81c364638e1646e427d151/report/index.html

@kaijchen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31282 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b349d45c9ddb3b14b52754aa45d6261acc644b70, data reload: false

------ Round 1 ----------------------------------
q1	17580	5231	5089	5089
q2	2076	311	170	170
q3	10986	1260	752	752
q4	10207	1005	531	531
q5	7571	2384	2303	2303
q6	182	163	131	131
q7	902	737	585	585
q8	9293	1292	1050	1050
q9	4867	4653	4733	4653
q10	6821	2294	1872	1872
q11	471	267	249	249
q12	344	361	211	211
q13	17774	3659	3033	3033
q14	223	232	208	208
q15	514	456	440	440
q16	620	610	587	587
q17	577	872	331	331
q18	6543	6191	6183	6183
q19	1204	946	547	547
q20	314	316	181	181
q21	2746	2153	1877	1877
q22	366	334	299	299
Total cold run time: 102181 ms
Total hot run time: 31282 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5107	5119	5104	5104
q2	233	329	229	229
q3	2160	2672	2270	2270
q4	1424	1807	1325	1325
q5	4214	4114	4149	4114
q6	206	162	122	122
q7	1864	1805	1671	1671
q8	2568	2593	2592	2592
q9	7127	7177	7093	7093
q10	2992	3235	2746	2746
q11	570	509	519	509
q12	681	773	641	641
q13	3542	3787	3263	3263
q14	288	294	270	270
q15	505	485	457	457
q16	648	663	635	635
q17	1121	1559	1358	1358
q18	7405	7257	7316	7257
q19	793	790	844	790
q20	1993	2001	1861	1861
q21	5460	4894	4740	4740
q22	600	596	533	533
Total cold run time: 51501 ms
Total hot run time: 49580 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189660 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b349d45c9ddb3b14b52754aa45d6261acc644b70, data reload: false

query1	1286	956	934	934
query2	6236	1843	1831	1831
query3	11108	4392	4393	4392
query4	54425	25136	22975	22975
query5	4895	610	494	494
query6	332	194	179	179
query7	4912	517	293	293
query8	312	239	236	236
query9	5353	2541	2528	2528
query10	465	316	246	246
query11	15150	15100	14854	14854
query12	155	110	108	108
query13	1017	524	406	406
query14	10846	6864	7103	6864
query15	217	201	192	192
query16	7204	673	495	495
query17	1096	722	611	611
query18	1540	417	320	320
query19	199	193	172	172
query20	131	133	130	130
query21	215	123	103	103
query22	4382	4612	4440	4440
query23	34062	33339	33541	33339
query24	5762	2462	2426	2426
query25	469	466	425	425
query26	739	270	153	153
query27	2089	503	327	327
query28	2772	2450	2435	2435
query29	597	559	415	415
query30	218	184	156	156
query31	903	870	815	815
query32	71	65	56	56
query33	485	348	311	311
query34	812	873	525	525
query35	802	834	750	750
query36	957	982	899	899
query37	124	103	77	77
query38	4166	4351	4158	4158
query39	1487	1468	1426	1426
query40	207	112	103	103
query41	51	47	48	47
query42	123	110	111	110
query43	503	512	470	470
query44	1363	814	800	800
query45	181	181	166	166
query46	896	1076	672	672
query47	1868	1891	1789	1789
query48	384	422	315	315
query49	712	517	400	400
query50	708	755	445	445
query51	4362	4283	4227	4227
query52	105	107	97	97
query53	236	272	193	193
query54	485	509	418	418
query55	88	83	86	83
query56	279	278	267	267
query57	1151	1178	1087	1087
query58	253	232	247	232
query59	2601	2921	2639	2639
query60	291	275	269	269
query61	118	119	122	119
query62	755	753	681	681
query63	239	195	192	192
query64	2333	1050	679	679
query65	3280	3138	3132	3132
query66	762	389	292	292
query67	15765	15770	15383	15383
query68	4514	797	520	520
query69	497	309	262	262
query70	1184	1115	1107	1107
query71	448	286	264	264
query72	6267	3633	3831	3633
query73	854	746	340	340
query74	8950	9071	8963	8963
query75	3170	3148	2689	2689
query76	3696	1188	743	743
query77	543	373	279	279
query78	10107	10258	9224	9224
query79	2439	813	578	578
query80	1131	509	432	432
query81	548	267	233	233
query82	404	129	90	90
query83	215	166	148	148
query84	285	90	71	71
query85	785	341	291	291
query86	387	327	293	293
query87	4480	4560	4321	4321
query88	2845	2203	2159	2159
query89	390	326	287	287
query90	1593	188	192	188
query91	140	139	109	109
query92	63	61	61	61
query93	1542	1002	571	571
query94	654	397	293	293
query95	341	268	255	255
query96	500	542	263	263
query97	2823	2830	2743	2743
query98	216	201	204	201
query99	1308	1394	1246	1246
Total cold run time: 291409 ms
Total hot run time: 189660 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.88 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b349d45c9ddb3b14b52754aa45d6261acc644b70, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.63	0.11	0.10
query5	0.41	0.41	0.39
query6	1.17	0.66	0.65
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.59	0.52	0.51
query10	0.59	0.58	0.58
query11	0.15	0.10	0.11
query12	0.14	0.10	0.11
query13	0.62	0.60	0.60
query14	2.73	2.76	2.71
query15	0.93	0.84	0.83
query16	0.37	0.38	0.36
query17	0.99	1.02	1.03
query18	0.21	0.19	0.19
query19	1.91	1.75	2.03
query20	0.01	0.01	0.01
query21	15.38	0.88	0.54
query22	0.75	1.17	0.68
query23	14.98	1.38	0.63
query24	6.80	1.70	0.44
query25	0.52	0.21	0.07
query26	0.61	0.17	0.13
query27	0.06	0.05	0.05
query28	9.47	0.84	0.42
query29	12.58	3.88	3.21
query30	0.25	0.08	0.06
query31	2.83	0.57	0.38
query32	3.22	0.54	0.46
query33	2.95	2.99	3.04
query34	15.88	5.12	4.51
query35	4.56	4.45	4.54
query36	0.65	0.49	0.47
query37	0.09	0.06	0.06
query38	0.06	0.05	0.04
query39	0.03	0.02	0.02
query40	0.16	0.14	0.13
query41	0.09	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 104.83 s
Total hot run time: 29.88 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 42.95% (11300/26312)
Line Coverage: 32.88% (94901/288594)
Region Coverage: 32.05% (48662/151829)
Branch Coverage: 27.91% (24544/87944)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b349d45c9ddb3b14b52754aa45d6261acc644b70_b349d45c9ddb3b14b52754aa45d6261acc644b70/report/index.html

@kaijchen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31545 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 74534e7b85541288724242223dbc0de884c817f7, data reload: false

------ Round 1 ----------------------------------
q1	17603	5307	5068	5068
q2	2048	286	162	162
q3	10617	1204	745	745
q4	10204	1005	530	530
q5	7533	2362	2324	2324
q6	185	171	134	134
q7	898	740	609	609
q8	9291	1278	1049	1049
q9	4854	4796	4681	4681
q10	6839	2303	1878	1878
q11	479	297	266	266
q12	367	378	236	236
q13	17754	3661	3074	3074
q14	235	231	204	204
q15	520	459	482	459
q16	640	620	579	579
q17	583	859	333	333
q18	7055	6247	6305	6247
q19	1202	933	562	562
q20	307	316	189	189
q21	2778	2123	1914	1914
q22	356	324	302	302
Total cold run time: 102348 ms
Total hot run time: 31545 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5150	5170	5137	5137
q2	235	323	237	237
q3	2152	2706	2316	2316
q4	1469	1821	1432	1432
q5	4214	4173	4150	4150
q6	222	163	124	124
q7	1850	1850	1680	1680
q8	2606	2585	2592	2585
q9	7120	7160	7131	7131
q10	3006	3226	2769	2769
q11	601	517	494	494
q12	677	771	610	610
q13	3564	3879	3348	3348
q14	280	296	282	282
q15	519	476	456	456
q16	638	704	647	647
q17	1144	1608	1319	1319
q18	7483	7285	7417	7285
q19	800	819	913	819
q20	1944	2020	1877	1877
q21	5460	4977	4719	4719
q22	627	580	524	524
Total cold run time: 51761 ms
Total hot run time: 49941 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189477 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 74534e7b85541288724242223dbc0de884c817f7, data reload: false

query1	1332	947	941	941
query2	6277	1853	1811	1811
query3	10978	4240	4351	4240
query4	53815	24420	22864	22864
query5	5064	549	470	470
query6	345	212	181	181
query7	4914	514	297	297
query8	307	247	236	236
query9	5598	2554	2545	2545
query10	411	303	246	246
query11	15195	15088	14864	14864
query12	161	112	111	111
query13	1099	528	404	404
query14	10450	6387	6823	6387
query15	208	204	188	188
query16	7101	654	472	472
query17	1090	731	587	587
query18	1513	436	304	304
query19	208	196	172	172
query20	135	126	125	125
query21	214	129	112	112
query22	4297	4466	4510	4466
query23	33931	33405	33534	33405
query24	5720	2413	2430	2413
query25	465	457	389	389
query26	730	267	155	155
query27	1908	492	331	331
query28	2807	2459	2436	2436
query29	584	551	447	447
query30	220	191	151	151
query31	897	903	836	836
query32	76	61	58	58
query33	440	365	332	332
query34	767	857	501	501
query35	806	839	766	766
query36	952	991	931	931
query37	125	103	78	78
query38	4293	4353	4213	4213
query39	1470	1448	1413	1413
query40	202	119	115	115
query41	54	49	47	47
query42	120	101	104	101
query43	504	512	502	502
query44	1312	812	813	812
query45	185	175	176	175
query46	877	1051	666	666
query47	1893	1920	1841	1841
query48	393	431	332	332
query49	685	487	424	424
query50	693	745	420	420
query51	4258	4275	4203	4203
query52	107	101	96	96
query53	244	264	191	191
query54	490	488	417	417
query55	80	81	81	81
query56	259	270	252	252
query57	1194	1204	1118	1118
query58	258	233	229	229
query59	2768	2843	2684	2684
query60	284	308	271	271
query61	122	118	113	113
query62	744	769	678	678
query63	231	189	188	188
query64	1710	1037	680	680
query65	3275	3201	3172	3172
query66	731	388	297	297
query67	15709	15573	15332	15332
query68	4881	764	514	514
query69	519	296	270	270
query70	1193	1119	1137	1119
query71	445	288	251	251
query72	5860	3587	3748	3587
query73	1197	746	351	351
query74	8983	9106	9009	9009
query75	3191	3184	2686	2686
query76	3761	1177	721	721
query77	538	370	279	279
query78	10168	10323	9327	9327
query79	1543	846	591	591
query80	847	529	447	447
query81	515	280	240	240
query82	403	125	95	95
query83	191	169	154	154
query84	283	90	72	72
query85	736	350	373	350
query86	348	304	282	282
query87	4451	4525	4402	4402
query88	2791	2218	2188	2188
query89	393	315	275	275
query90	1639	191	190	190
query91	130	141	116	116
query92	65	54	55	54
query93	1395	1042	578	578
query94	628	375	300	300
query95	345	263	252	252
query96	490	553	265	265
query97	2797	2834	2739	2739
query98	227	206	203	203
query99	1389	1373	1255	1255
Total cold run time: 288627 ms
Total hot run time: 189477 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.26 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 74534e7b85541288724242223dbc0de884c817f7, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.04	0.02
query3	0.24	0.07	0.07
query4	1.62	0.10	0.11
query5	0.43	0.43	0.40
query6	1.18	0.64	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.60	0.52	0.52
query10	0.57	0.58	0.58
query11	0.15	0.10	0.10
query12	0.14	0.11	0.11
query13	0.63	0.59	0.61
query14	2.68	2.82	2.70
query15	0.92	0.85	0.86
query16	0.38	0.38	0.38
query17	1.03	1.03	1.05
query18	0.22	0.20	0.20
query19	1.98	1.79	1.98
query20	0.02	0.02	0.01
query21	15.38	0.90	0.55
query22	0.76	1.26	0.67
query23	14.86	1.39	0.64
query24	11.41	1.12	0.43
query25	0.34	0.14	0.08
query26	1.07	0.20	0.15
query27	0.05	0.06	0.05
query28	6.09	0.80	0.42
query29	12.56	4.04	3.31
query30	0.25	0.09	0.06
query31	2.82	0.61	0.38
query32	3.22	0.55	0.47
query33	2.96	3.03	3.07
query34	15.75	5.16	4.51
query35	4.53	4.52	4.58
query36	0.66	0.51	0.49
query37	0.09	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.14	0.14
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.13 s
Total hot run time: 30.26 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 42.97% (11308/26314)
Line Coverage: 32.90% (94936/288570)
Region Coverage: 32.06% (48672/151831)
Branch Coverage: 27.92% (24554/87948)
Coverage Report: http://coverage.selectdb-in.cc/coverage/74534e7b85541288724242223dbc0de884c817f7_74534e7b85541288724242223dbc0de884c817f7/report/index.html

dataroaring
dataroaring reviewed Feb 18, 2025
View reviewed changes
be/src/vec/sink/vtablet_block_convertor.cpp
_num_filtered_rows += filtered_rows;
has_filtered_rows = filtered_rows > 0;
if (stop_processing) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we remove stop_processing?

@kaijchen
Copy link
Contributor Author

run buildall

@kaijchen
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the lfs-detected! Warning Label for use when LFS is detected in the commits of a Pull Request label Mar 10, 2025
@doris-robot
Copy link

TPC-H: Total hot run time: 34230 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ccbb76b74cd1caf214e8ddd292e962332727f1bc, data reload: false

------ Round 1 ----------------------------------
q1	17607	5492	5402	5402
q2	2041	300	172	172
q3	10403	1680	756	756
q4	10228	1139	578	578
q5	7555	2822	2807	2807
q6	220	182	136	136
q7	1025	833	637	637
q8	9306	1536	1265	1265
q9	5191	4829	4868	4829
q10	6938	2419	1917	1917
q11	499	291	256	256
q12	375	401	228	228
q13	17788	3876	3197	3197
q14	251	240	218	218
q15	561	497	516	497
q16	628	639	585	585
q17	607	933	363	363
q18	7346	6595	6485	6485
q19	1213	1120	621	621
q20	348	348	213	213
q21	3274	2451	2094	2094
q22	1072	1034	974	974
Total cold run time: 104476 ms
Total hot run time: 34230 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5598	5586	5539	5539
q2	248	356	238	238
q3	2347	2864	2349	2349
q4	1588	2022	1484	1484
q5	4406	4391	4649	4391
q6	261	195	136	136
q7	2229	2090	1854	1854
q8	2980	3023	2862	2862
q9	7150	7192	7114	7114
q10	3233	3386	2976	2976
q11	654	533	497	497
q12	725	794	621	621
q13	3693	4174	3342	3342
q14	296	321	268	268
q15	574	489	476	476
q16	677	732	655	655
q17	1249	1797	1458	1458
q18	7943	7739	7638	7638
q19	914	865	1175	865
q20	2062	2163	1897	1897
q21	5949	5233	5033	5033
q22	1106	1061	1017	1017
Total cold run time: 55882 ms
Total hot run time: 52710 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185939 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ccbb76b74cd1caf214e8ddd292e962332727f1bc, data reload: false

query1	1012	406	387	387
query2	6535	1926	1894	1894
query3	6812	218	221	218
query4	26324	23359	23779	23359
query5	4374	681	494	494
query6	306	214	201	201
query7	4601	509	300	300
query8	310	256	238	238
query9	8615	2524	2522	2522
query10	460	310	260	260
query11	15607	15191	15102	15102
query12	164	112	106	106
query13	1663	527	407	407
query14	8928	6273	6319	6273
query15	218	186	175	175
query16	7270	653	461	461
query17	1206	727	567	567
query18	1980	421	314	314
query19	193	193	161	161
query20	123	119	119	119
query21	217	125	108	108
query22	4144	4285	4246	4246
query23	33980	33117	33012	33012
query24	8279	2409	2430	2409
query25	547	508	410	410
query26	1227	272	157	157
query27	2704	492	323	323
query28	4381	2394	2367	2367
query29	788	549	433	433
query30	285	224	186	186
query31	924	881	769	769
query32	73	66	64	64
query33	575	396	303	303
query34	804	852	508	508
query35	797	827	739	739
query36	959	997	894	894
query37	116	108	76	76
query38	4255	4204	4047	4047
query39	1494	1417	1375	1375
query40	217	118	105	105
query41	55	52	51	51
query42	118	104	104	104
query43	506	491	495	491
query44	1335	792	788	788
query45	181	168	165	165
query46	829	1024	625	625
query47	1750	1793	1697	1697
query48	399	430	308	308
query49	802	501	423	423
query50	688	735	412	412
query51	4208	4220	4201	4201
query52	107	113	98	98
query53	234	274	196	196
query54	496	522	416	416
query55	85	86	82	82
query56	270	266	256	256
query57	1120	1156	1071	1071
query58	247	251	236	236
query59	2752	2800	2700	2700
query60	315	285	262	262
query61	126	123	120	120
query62	792	717	650	650
query63	280	195	194	194
query64	4431	1005	669	669
query65	4543	4389	4416	4389
query66	1128	425	300	300
query67	15686	15654	15324	15324
query68	8617	859	502	502
query69	469	303	267	267
query70	1214	1137	1123	1123
query71	458	308	274	274
query72	5552	3543	3776	3543
query73	792	730	353	353
query74	8952	9343	8751	8751
query75	3823	3194	2722	2722
query76	3651	1192	745	745
query77	789	384	287	287
query78	10097	10151	9243	9243
query79	2828	840	599	599
query80	653	537	450	450
query81	477	258	223	223
query82	517	126	97	97
query83	214	174	155	155
query84	292	100	70	70
query85	787	354	404	354
query86	353	293	296	293
query87	4514	4553	4468	4468
query88	3247	2230	2291	2230
query89	411	318	285	285
query90	1937	220	224	220
query91	159	145	113	113
query92	75	61	58	58
query93	1557	1069	582	582
query94	685	497	307	307
query95	360	275	268	268
query96	485	571	276	276
query97	3428	3396	3309	3309
query98	228	210	205	205
query99	1437	1377	1250	1250
Total cold run time: 276008 ms
Total hot run time: 185939 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.92 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ccbb76b74cd1caf214e8ddd292e962332727f1bc, data reload: false

query1	0.03	0.04	0.04
query2	0.08	0.03	0.04
query3	0.24	0.07	0.07
query4	1.62	0.10	0.10
query5	0.56	0.55	0.56
query6	1.18	0.73	0.71
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.60	0.52	0.53
query10	0.57	0.62	0.58
query11	0.16	0.11	0.10
query12	0.14	0.11	0.11
query13	0.64	0.62	0.60
query14	2.67	2.72	2.66
query15	0.94	0.86	0.86
query16	0.39	0.39	0.39
query17	1.07	1.03	1.03
query18	0.21	0.20	0.19
query19	1.97	2.03	1.80
query20	0.01	0.01	0.02
query21	15.35	0.90	0.58
query22	0.76	1.23	0.77
query23	14.75	1.39	0.63
query24	7.73	1.48	0.69
query25	0.51	0.24	0.14
query26	0.54	0.16	0.14
query27	0.05	0.05	0.04
query28	9.31	0.82	0.44
query29	12.56	3.95	3.29
query30	0.25	0.09	0.07
query31	2.82	0.58	0.40
query32	3.23	0.55	0.47
query33	3.03	3.11	3.05
query34	15.81	5.10	4.48
query35	4.52	4.51	4.58
query36	0.69	0.49	0.51
query37	0.10	0.07	0.06
query38	0.06	0.04	0.04
query39	0.03	0.03	0.03
query40	0.19	0.14	0.13
query41	0.08	0.03	0.03
query42	0.04	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.58 s
Total hot run time: 30.92 s

@kaijchen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32500 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e3bc6e0d96fda5184a30dfc22ba884078ad2e69b, data reload: false

------ Round 1 ----------------------------------
q1	17633	5223	5065	5065
q2	2048	289	161	161
q3	10512	1307	774	774
q4	10204	1045	554	554
q5	7521	2436	2347	2347
q6	187	162	130	130
q7	939	733	611	611
q8	9299	1320	1071	1071
q9	4930	4614	4906	4614
q10	6812	2310	1900	1900
q11	491	275	259	259
q12	349	350	221	221
q13	17753	3717	3148	3148
q14	228	233	217	217
q15	532	478	475	475
q16	630	604	594	594
q17	601	888	337	337
q18	6982	6491	6362	6362
q19	1841	953	555	555
q20	318	321	193	193
q21	2818	2133	1937	1937
q22	1040	1032	975	975
Total cold run time: 103668 ms
Total hot run time: 32500 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5255	5141	5159	5141
q2	242	332	227	227
q3	2238	2723	2307	2307
q4	1445	1810	1378	1378
q5	4261	4170	4197	4170
q6	205	164	129	129
q7	1954	1920	1810	1810
q8	2617	2581	2580	2580
q9	7391	7189	7213	7189
q10	3043	3245	2801	2801
q11	592	497	497	497
q12	651	719	625	625
q13	3484	3957	3239	3239
q14	303	312	268	268
q15	532	478	483	478
q16	649	667	645	645
q17	1157	1619	1344	1344
q18	7652	7620	7524	7524
q19	832	855	918	855
q20	2051	2040	1856	1856
q21	5499	5089	4830	4830
q22	1088	1076	1051	1051
Total cold run time: 53141 ms
Total hot run time: 50944 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193463 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e3bc6e0d96fda5184a30dfc22ba884078ad2e69b, data reload: false

query1	1412	1024	994	994
query2	6131	1883	1852	1852
query3	11024	4627	4645	4627
query4	53402	24596	23426	23426
query5	5184	558	498	498
query6	360	218	186	186
query7	4917	512	306	306
query8	339	260	234	234
query9	5824	2658	2633	2633
query10	431	312	266	266
query11	15259	15063	14937	14937
query12	155	108	101	101
query13	1062	528	414	414
query14	10736	7136	6831	6831
query15	208	195	180	180
query16	7155	642	517	517
query17	1069	709	546	546
query18	1527	415	327	327
query19	201	221	166	166
query20	134	118	122	118
query21	208	141	109	109
query22	4545	4542	4413	4413
query23	34004	33191	33493	33191
query24	5802	2492	2472	2472
query25	518	491	456	456
query26	686	293	167	167
query27	1753	510	346	346
query28	2874	2506	2506	2506
query29	601	599	473	473
query30	275	223	191	191
query31	917	904	846	846
query32	85	65	71	65
query33	497	372	376	372
query34	772	868	511	511
query35	794	837	766	766
query36	954	1014	914	914
query37	121	102	77	77
query38	4183	4232	4447	4232
query39	1483	1421	1481	1421
query40	231	116	104	104
query41	50	51	50	50
query42	126	104	110	104
query43	527	526	490	490
query44	1360	822	822	822
query45	180	173	165	165
query46	863	1042	652	652
query47	1828	1893	1754	1754
query48	386	434	344	344
query49	688	556	432	432
query50	728	765	426	426
query51	4314	4331	4326	4326
query52	105	109	106	106
query53	242	264	187	187
query54	488	495	428	428
query55	83	81	80	80
query56	281	267	270	267
query57	1175	1175	1102	1102
query58	247	241	241	241
query59	2850	2884	2883	2883
query60	302	288	289	288
query61	131	119	157	119
query62	761	754	706	706
query63	239	200	195	195
query64	1475	1043	717	717
query65	4636	4442	4361	4361
query66	784	389	310	310
query67	15793	15695	15266	15266
query68	7267	903	535	535
query69	552	348	264	264
query70	1266	1146	1085	1085
query71	486	297	271	271
query72	5529	3597	3809	3597
query73	1420	786	352	352
query74	9126	9199	8797	8797
query75	3762	3173	2697	2697
query76	4218	1220	764	764
query77	633	383	298	298
query78	10069	10120	9305	9305
query79	2438	855	605	605
query80	649	536	473	473
query81	498	262	224	224
query82	702	122	97	97
query83	173	168	154	154
query84	294	94	81	81
query85	789	367	410	367
query86	382	275	299	275
query87	4393	4453	4451	4451
query88	3726	2290	2276	2276
query89	429	315	289	289
query90	1849	219	221	219
query91	141	142	107	107
query92	70	59	59	59
query93	1816	1092	586	586
query94	660	397	311	311
query95	354	348	290	290
query96	486	571	284	284
query97	3381	3375	3270	3270
query98	230	203	211	203
query99	1726	1358	1265	1265
Total cold run time: 297989 ms
Total hot run time: 193463 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.1 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e3bc6e0d96fda5184a30dfc22ba884078ad2e69b, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.23	0.06	0.06
query4	1.62	0.10	0.11
query5	0.56	0.54	0.55
query6	1.21	0.72	0.72
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.61	0.52	0.53
query10	0.60	0.60	0.59
query11	0.16	0.11	0.10
query12	0.14	0.11	0.11
query13	0.63	0.61	0.60
query14	2.70	2.66	2.83
query15	0.94	0.87	0.85
query16	0.38	0.38	0.38
query17	1.04	1.02	1.03
query18	0.22	0.20	0.19
query19	1.91	1.86	1.99
query20	0.01	0.01	0.01
query21	15.37	0.92	0.56
query22	0.75	1.07	0.62
query23	15.08	1.37	0.64
query24	6.72	1.79	1.03
query25	0.53	0.23	0.09
query26	0.56	0.16	0.14
query27	0.06	0.05	0.05
query28	9.80	0.88	0.43
query29	12.61	4.01	3.30
query30	0.25	0.09	0.06
query31	2.82	0.60	0.39
query32	3.23	0.55	0.47
query33	3.00	3.10	3.17
query34	15.88	5.13	4.52
query35	4.55	4.49	4.55
query36	0.66	0.50	0.48
query37	0.09	0.06	0.07
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.15	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.03
query43	0.04	0.03	0.04
Total cold run time: 105.48 s
Total hot run time: 31.1 s

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 1.04% (1/96) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 46.81% (12514/26733)
Line Coverage 36.45% (106694/292726)
Region Coverage 35.47% (54499/153634)
Branch Coverage 30.86% (27422/88868)

@kaijchen kaijchen closed this Mar 11, 2025
@kaijchen kaijchen reopened this Mar 11, 2025
@kaijchen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32530 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9cdbfaf6f31de60fe8dd6f4fb96b5a01193ff69f, data reload: false

------ Round 1 ----------------------------------
q1	17599	5248	5142	5142
q2	2052	283	169	169
q3	10497	1321	716	716
q4	10249	1044	550	550
q5	8333	2446	2371	2371
q6	194	173	137	137
q7	914	760	605	605
q8	9306	1353	1124	1124
q9	4967	4694	4922	4694
q10	6816	2309	1889	1889
q11	492	268	265	265
q12	350	352	219	219
q13	17800	3732	3070	3070
q14	225	241	209	209
q15	557	506	497	497
q16	632	610	583	583
q17	566	893	353	353
q18	7209	6401	6320	6320
q19	1652	964	552	552
q20	317	324	189	189
q21	2799	2192	1907	1907
q22	1008	1013	969	969
Total cold run time: 104534 ms
Total hot run time: 32530 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5237	5134	5141	5134
q2	241	323	233	233
q3	2187	2667	2280	2280
q4	1444	1863	1399	1399
q5	4223	4139	4211	4139
q6	215	167	124	124
q7	1998	1979	1793	1793
q8	2646	2612	2645	2612
q9	7219	7194	7079	7079
q10	3034	3173	2735	2735
q11	590	510	494	494
q12	702	773	622	622
q13	3508	3877	3260	3260
q14	286	296	278	278
q15	554	493	497	493
q16	643	685	640	640
q17	1157	1618	1367	1367
q18	7726	7708	7563	7563
q19	823	796	838	796
q20	1987	2031	1883	1883
q21	5428	4925	4818	4818
q22	1030	1002	996	996
Total cold run time: 52878 ms
Total hot run time: 50738 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185517 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9cdbfaf6f31de60fe8dd6f4fb96b5a01193ff69f, data reload: false

query1	989	415	382	382
query2	6539	1897	1815	1815
query3	6791	220	218	218
query4	26422	23845	23425	23425
query5	4357	668	502	502
query6	289	193	201	193
query7	4601	500	293	293
query8	289	233	233	233
query9	8602	2658	2658	2658
query10	477	311	265	265
query11	15823	15095	14967	14967
query12	161	106	102	102
query13	1661	532	405	405
query14	9635	7005	6345	6345
query15	215	194	176	176
query16	7353	630	483	483
query17	1200	720	575	575
query18	1971	414	320	320
query19	194	187	161	161
query20	124	120	116	116
query21	217	125	104	104
query22	4102	4236	4124	4124
query23	34151	33037	33100	33037
query24	7744	2384	2408	2384
query25	557	458	392	392
query26	1238	267	151	151
query27	2140	493	340	340
query28	3879	2455	2401	2401
query29	765	556	425	425
query30	291	215	195	195
query31	938	854	746	746
query32	79	64	70	64
query33	562	360	370	360
query34	798	836	506	506
query35	788	877	757	757
query36	954	973	891	891
query37	119	99	74	74
query38	4137	4260	4093	4093
query39	1459	1390	1573	1390
query40	212	113	108	108
query41	56	51	53	51
query42	116	104	102	102
query43	503	508	479	479
query44	1287	776	781	776
query45	172	169	166	166
query46	828	1013	619	619
query47	1726	1791	1719	1719
query48	374	406	306	306
query49	783	518	428	428
query50	679	756	401	401
query51	4191	4165	4307	4165
query52	107	102	99	99
query53	234	260	183	183
query54	480	487	412	412
query55	81	76	82	76
query56	275	268	252	252
query57	1120	1158	1051	1051
query58	244	245	250	245
query59	2616	2812	2413	2413
query60	279	277	259	259
query61	125	119	118	118
query62	797	737	671	671
query63	234	191	185	185
query64	4374	1044	686	686
query65	4438	4368	4355	4355
query66	1147	407	295	295
query67	15798	15561	15156	15156
query68	8453	913	502	502
query69	465	298	259	259
query70	1155	1113	1116	1113
query71	473	298	269	269
query72	5676	3615	3727	3615
query73	787	752	351	351
query74	8957	9165	8897	8897
query75	3823	3192	2721	2721
query76	3700	1191	743	743
query77	785	373	283	283
query78	10095	10050	9244	9244
query79	3671	829	572	572
query80	705	524	447	447
query81	464	262	222	222
query82	687	130	97	97
query83	212	173	158	158
query84	279	92	74	74
query85	774	350	309	309
query86	348	306	391	306
query87	4501	4602	4349	4349
query88	3200	2238	2231	2231
query89	425	320	280	280
query90	1964	207	208	207
query91	142	142	109	109
query92	79	62	58	58
query93	2161	1070	587	587
query94	664	424	307	307
query95	352	270	264	264
query96	483	568	275	275
query97	3418	3408	3312	3312
query98	233	213	204	204
query99	1446	1418	1308	1308
Total cold run time: 276464 ms
Total hot run time: 185517 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.85 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9cdbfaf6f31de60fe8dd6f4fb96b5a01193ff69f, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.04	0.03
query3	0.24	0.06	0.06
query4	1.61	0.10	0.10
query5	0.62	0.56	0.55
query6	1.18	0.73	0.71
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.58	0.52	0.52
query10	0.59	0.59	0.59
query11	0.15	0.11	0.11
query12	0.15	0.12	0.11
query13	0.61	0.61	0.60
query14	2.82	2.69	2.81
query15	0.93	0.85	0.85
query16	0.39	0.38	0.39
query17	0.99	1.05	1.02
query18	0.21	0.20	0.20
query19	1.91	1.84	1.98
query20	0.01	0.01	0.01
query21	15.38	0.92	0.54
query22	0.73	1.10	0.65
query23	15.04	1.38	0.57
query24	6.72	2.01	0.98
query25	0.53	0.29	0.07
query26	0.61	0.15	0.13
query27	0.06	0.05	0.05
query28	9.35	0.90	0.44
query29	12.78	4.01	3.29
query30	0.25	0.09	0.06
query31	2.83	0.56	0.39
query32	3.22	0.55	0.46
query33	3.05	3.02	3.16
query34	15.63	5.12	4.45
query35	4.55	4.58	4.50
query36	0.67	0.50	0.49
query37	0.09	0.07	0.07
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.14	0.13
query41	0.09	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.06 s
Total hot run time: 30.85 s

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 1.04% (1/96) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 46.88% (12538/26747)
Line Coverage 36.59% (107156/292821)
Region Coverage 35.66% (54795/153678)
Branch Coverage 31.06% (27610/88892)

@kaijchen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32428 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d21fea2e97346ca40c95ca8ceff3410d7957d773, data reload: false

------ Round 1 ----------------------------------
q1	17649	5213	5058	5058
q2	2046	285	162	162
q3	10434	1343	716	716
q4	10225	1061	518	518
q5	7556	2446	2373	2373
q6	190	165	132	132
q7	917	754	609	609
q8	9315	1315	1111	1111
q9	5113	4646	4837	4646
q10	6812	2319	1905	1905
q11	465	273	264	264
q12	344	355	214	214
q13	17797	3747	3074	3074
q14	239	220	208	208
q15	548	484	483	483
q16	614	620	582	582
q17	611	871	338	338
q18	7594	6625	6377	6377
q19	1226	958	568	568
q20	330	330	198	198
q21	3005	2147	1921	1921
q22	1025	1048	971	971
Total cold run time: 104055 ms
Total hot run time: 32428 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5189	5126	5157	5126
q2	242	336	234	234
q3	2157	2676	2290	2290
q4	1428	1812	1415	1415
q5	4232	4157	4166	4157
q6	205	165	125	125
q7	1961	1934	1814	1814
q8	2664	2586	2605	2586
q9	7290	7467	7036	7036
q10	3049	3211	2750	2750
q11	604	507	488	488
q12	714	786	595	595
q13	3516	3985	3288	3288
q14	275	301	278	278
q15	536	482	480	480
q16	672	697	669	669
q17	1183	1550	1417	1417
q18	7803	7693	7537	7537
q19	853	915	965	915
q20	1976	2021	1788	1788
q21	5368	4946	4972	4946
q22	1128	1075	1034	1034
Total cold run time: 53045 ms
Total hot run time: 50968 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192986 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d21fea2e97346ca40c95ca8ceff3410d7957d773, data reload: false

query1	1395	1016	987	987
query2	6380	1927	1952	1927
query3	11150	4732	4685	4685
query4	25855	23940	23374	23374
query5	5126	681	467	467
query6	295	192	185	185
query7	3983	503	290	290
query8	294	231	245	231
query9	8496	2624	2586	2586
query10	483	312	257	257
query11	15635	15190	14963	14963
query12	160	115	119	115
query13	1569	528	406	406
query14	8873	6784	7106	6784
query15	213	185	176	176
query16	7682	705	519	519
query17	1087	731	567	567
query18	2021	424	316	316
query19	196	193	163	163
query20	126	124	128	124
query21	213	129	114	114
query22	4433	4712	4339	4339
query23	34512	33561	33450	33450
query24	7729	2466	2431	2431
query25	504	507	412	412
query26	1182	278	154	154
query27	2558	519	336	336
query28	4328	2476	2430	2430
query29	703	583	442	442
query30	286	232	193	193
query31	947	881	786	786
query32	79	63	70	63
query33	555	354	322	322
query34	844	894	514	514
query35	816	828	787	787
query36	982	1019	913	913
query37	123	108	76	76
query38	4193	4442	4121	4121
query39	1518	1433	1452	1433
query40	215	129	111	111
query41	58	59	61	59
query42	118	105	105	105
query43	521	522	496	496
query44	1400	823	804	804
query45	185	180	175	175
query46	844	1020	651	651
query47	1849	1903	1773	1773
query48	414	415	311	311
query49	762	515	425	425
query50	716	769	429	429
query51	4273	4333	4257	4257
query52	105	108	106	106
query53	250	280	199	199
query54	526	518	421	421
query55	81	79	85	79
query56	295	271	267	267
query57	1141	1161	1111	1111
query58	244	240	245	240
query59	2723	2880	2712	2712
query60	305	289	275	275
query61	127	116	122	116
query62	813	755	671	671
query63	223	182	190	182
query64	3943	999	659	659
query65	4472	4335	4396	4335
query66	926	397	286	286
query67	15711	15731	15441	15441
query68	8497	873	498	498
query69	469	305	275	275
query70	1225	1063	1097	1063
query71	464	296	260	260
query72	5646	3625	3723	3625
query73	785	732	351	351
query74	8949	9117	8919	8919
query75	3762	3156	2728	2728
query76	3687	1189	777	777
query77	783	387	277	277
query78	10051	10182	9320	9320
query79	2815	846	585	585
query80	781	534	447	447
query81	499	254	226	226
query82	713	126	95	95
query83	173	237	166	166
query84	255	93	70	70
query85	754	354	309	309
query86	377	301	303	301
query87	4477	4427	4429	4427
query88	3816	2275	2260	2260
query89	426	314	290	290
query90	1858	225	293	225
query91	140	136	106	106
query92	73	63	56	56
query93	2056	1102	582	582
query94	667	422	269	269
query95	355	278	257	257
query96	486	579	276	276
query97	3320	3381	3324	3324
query98	227	203	207	203
query99	1494	1415	1285	1285
Total cold run time: 280844 ms
Total hot run time: 192986 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.81 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d21fea2e97346ca40c95ca8ceff3410d7957d773, data reload: false

query1	0.04	0.03	0.04
query2	0.07	0.03	0.04
query3	0.23	0.07	0.07
query4	1.62	0.11	0.10
query5	0.55	0.54	0.55
query6	1.21	0.72	0.72
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.52	0.51
query10	0.58	0.60	0.59
query11	0.15	0.11	0.10
query12	0.14	0.11	0.11
query13	0.61	0.61	0.60
query14	2.67	2.70	2.73
query15	0.94	0.86	0.86
query16	0.38	0.38	0.37
query17	1.05	1.04	1.05
query18	0.21	0.20	0.19
query19	1.89	1.80	1.94
query20	0.01	0.01	0.01
query21	15.38	0.89	0.54
query22	0.76	1.14	0.71
query23	14.93	1.39	0.61
query24	7.23	2.38	0.73
query25	0.49	0.26	0.08
query26	0.61	0.16	0.13
query27	0.06	0.05	0.04
query28	9.57	0.83	0.44
query29	12.60	4.00	3.30
query30	0.25	0.09	0.07
query31	2.82	0.57	0.38
query32	3.24	0.55	0.48
query33	2.98	3.08	3.01
query34	15.73	5.13	4.56
query35	4.59	4.54	4.55
query36	0.65	0.50	0.47
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.03	0.02
query40	0.17	0.14	0.13
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.02	0.03
Total cold run time: 105.36 s
Total hot run time: 30.81 s

liaoxin01
liaoxin01 approved these changes Mar 13, 2025
View reviewed changes
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 13, 2025
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@dataroaring dataroaring added the usercase Important user case type label label Mar 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@liaoxin01 liaoxin01 liaoxin01 approved these changes

@dataroaring dataroaring dataroaring left review comments

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.x dev/3.0.x reviewed usercase Important user case type label
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@kaijchen @Thearas @doris-robot @liaoxin01 @hello-stephen @dataroaring