Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](inverted index) in_list support inverted index #38738

Merged
merged 1 commit into from
Aug 2, 2024
Merged

[fix](inverted index) in_list support inverted index #38738

merged 1 commit into from
Aug 2, 2024

Conversation

zzzxl1993
Copy link
Contributor

@zzzxl1993 zzzxl1993 commented Aug 1, 2024 •
edited
Loading

Proposed changes

#37921
#37842

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zzzxl1993
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 1, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 50024 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 220527e220a5f234f21aff042d5c45807b9c2870, data reload: false

------ Round 1 ----------------------------------
q1	17814	4392	4381	4381
q2	2061	153	145	145
q3	10268	1875	1939	1875
q4	10301	1213	1324	1213
q5	8728	3949	3939	3939
q6	235	126	146	126
q7	2011	1590	1601	1590
q8	9336	2726	2703	2703
q9	11060	10444	10389	10389
q10	8624	3504	3493	3493
q11	421	247	256	247
q12	466	310	304	304
q13	18319	3976	4039	3976
q14	359	332	331	331
q15	498	457	465	457
q16	673	580	566	566
q17	1134	946	964	946
q18	7325	6857	6881	6857
q19	1789	1642	1621	1621
q20	501	320	322	320
q21	4432	4121	4116	4116
q22	523	450	429	429
Total cold run time: 116878 ms
Total hot run time: 50024 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4399	4352	4379	4352
q2	320	220	230	220
q3	4196	4148	4177	4148
q4	2745	2760	2749	2749
q5	7315	7252	7180	7180
q6	238	120	118	118
q7	3286	2846	2888	2846
q8	4388	4519	4530	4519
q9	17570	17235	17434	17235
q10	4441	4444	4379	4379
q11	757	700	688	688
q12	1039	874	876	874
q13	7680	3845	3784	3784
q14	466	434	415	415
q15	504	461	459	459
q16	752	709	700	700
q17	3834	3812	3894	3812
q18	8827	8850	8789	8789
q19	1759	1696	1666	1666
q20	2387	2160	2117	2117
q21	8593	8503	8556	8503
q22	1102	975	1009	975
Total cold run time: 86598 ms
Total hot run time: 80528 ms

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 1, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.75% (8104/21468)
Line Coverage: 29.39% (66366/225841)
Region Coverage: 28.90% (34248/118510)
Branch Coverage: 24.79% (17604/71008)
Coverage Report: http://coverage.selectdb-in.cc/coverage/220527e220a5f234f21aff042d5c45807b9c2870_220527e220a5f234f21aff042d5c45807b9c2870/report/index.html

@zzzxl1993
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 1, 2024

clang-tidy review says "All clean, LGTM! 👍"

@zzzxl1993
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 1, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 49930 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f400fd3ac1493244eb061745848db718b18ebd51, data reload: false

------ Round 1 ----------------------------------
q1	17920	4408	4350	4350
q2	2076	159	153	153
q3	10260	1837	1901	1837
q4	10141	1270	1344	1270
q5	8541	3936	3913	3913
q6	236	127	126	126
q7	2076	1647	1579	1579
q8	9551	2727	2715	2715
q9	14433	10494	10334	10334
q10	8585	3497	3525	3497
q11	406	240	250	240
q12	467	316	307	307
q13	18356	3947	4052	3947
q14	358	336	337	336
q15	511	457	461	457
q16	664	574	583	574
q17	1133	929	904	904
q18	7256	6966	6930	6930
q19	1775	1636	1607	1607
q20	546	314	320	314
q21	4383	4132	4093	4093
q22	534	447	449	447
Total cold run time: 120208 ms
Total hot run time: 49930 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4339	4342	4332	4332
q2	323	224	225	224
q3	4179	4149	4138	4138
q4	2757	2771	2768	2768
q5	7229	7175	7133	7133
q6	241	123	120	120
q7	3210	2868	2859	2859
q8	4404	4486	4455	4455
q9	17467	17112	17219	17112
q10	4261	4282	4282	4282
q11	766	694	692	692
q12	1038	873	849	849
q13	6949	3790	3723	3723
q14	468	432	432	432
q15	510	464	465	464
q16	733	688	675	675
q17	3835	3884	3855	3855
q18	8831	8785	8807	8785
q19	1727	1642	1703	1642
q20	2356	2120	2151	2120
q21	8546	8560	8463	8463
q22	1070	945	1026	945
Total cold run time: 85239 ms
Total hot run time: 80068 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.74% (8102/21468)
Line Coverage: 29.38% (66352/225836)
Region Coverage: 28.89% (34237/118507)
Branch Coverage: 24.78% (17595/71006)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f400fd3ac1493244eb061745848db718b18ebd51_f400fd3ac1493244eb061745848db718b18ebd51/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 205184 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f400fd3ac1493244eb061745848db718b18ebd51, data reload: false

query1	942	398	419	398
query2	6545	2816	2598	2598
query3	6928	211	203	203
query4	21350	18047	18190	18047
query5	19735	6518	6564	6518
query6	308	225	244	225
query7	4163	313	317	313
query8	425	411	390	390
query9	3236	2779	2707	2707
query10	423	315	301	301
query11	11296	10869	10778	10778
query12	127	87	76	76
query13	5595	717	700	700
query14	18024	13941	13428	13428
query15	361	241	252	241
query16	6437	302	267	267
query17	1696	1426	878	878
query18	2318	419	431	419
query19	211	162	155	155
query20	81	82	82	82
query21	188	105	96	96
query22	5176	5000	4914	4914
query23	32546	32197	31963	31963
query24	6876	6577	6618	6577
query25	539	437	426	426
query26	532	175	166	166
query27	1871	307	305	305
query28	6198	2378	2354	2354
query29	2944	2837	2807	2807
query30	240	170	166	166
query31	914	735	757	735
query32	71	68	61	61
query33	400	259	271	259
query34	852	477	490	477
query35	1130	963	878	878
query36	1494	1200	1196	1196
query37	90	62	61	61
query38	3063	2960	2945	2945
query39	1370	1333	1312	1312
query40	213	100	100	100
query41	48	45	46	45
query42	93	87	83	83
query43	678	667	696	667
query44	1161	731	737	731
query45	251	240	241	240
query46	1253	953	983	953
query47	1827	1749	1732	1732
query48	1030	710	728	710
query49	627	389	388	388
query50	884	644	584	584
query51	4790	4653	4670	4653
query52	98	95	90	90
query53	456	332	325	325
query54	2684	2476	2478	2476
query55	98	91	90	90
query56	251	219	219	219
query57	1095	1199	1125	1125
query58	234	205	214	205
query59	4087	4309	4158	4158
query60	226	233	212	212
query61	101	97	101	97
query62	794	461	491	461
query63	489	362	352	352
query64	2488	1526	1546	1526
query65	3670	3568	3582	3568
query66	819	388	391	388
query67	15787	15388	15760	15388
query68	10076	648	652	648
query69	587	347	370	347
query70	1757	1458	1445	1445
query71	424	320	321	320
query72	6549	3548	3569	3548
query73	738	334	326	326
query74	6314	5805	5896	5805
query75	5339	3805	3685	3685
query76	6396	1176	1229	1176
query77	1061	295	271	271
query78	13010	11793	11737	11737
query79	6475	639	647	639
query80	1081	412	411	411
query81	490	239	242	239
query82	1504	109	99	99
query83	169	138	135	135
query84	268	75	75	75
query85	900	346	350	346
query86	330	305	319	305
query87	3266	3036	3003	3003
query88	4763	2324	2339	2324
query89	380	297	316	297
query90	1935	204	203	203
query91	172	141	161	141
query92	59	57	58	57
query93	4983	619	586	586
query94	710	216	216	216
query95	1114	1068	1053	1053
query96	650	336	323	323
query97	6558	6405	6445	6405
query98	188	181	176	176
query99	2958	898	855	855
Total cold run time: 314131 ms
Total hot run time: 205184 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.26 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f400fd3ac1493244eb061745848db718b18ebd51, data reload: false

query1	0.03	0.02	0.02
query2	0.07	0.03	0.02
query3	0.24	0.05	0.05
query4	1.79	0.06	0.07
query5	0.55	0.53	0.52
query6	1.31	0.61	0.61
query7	0.01	0.02	0.01
query8	0.03	0.02	0.03
query9	0.53	0.48	0.48
query10	0.55	0.54	0.55
query11	0.13	0.08	0.09
query12	0.11	0.09	0.09
query13	0.63	0.62	0.61
query14	0.78	0.78	0.80
query15	0.79	0.77	0.78
query16	0.37	0.37	0.38
query17	1.02	1.01	1.02
query18	0.23	0.25	0.25
query19	1.87	1.85	1.86
query20	0.01	0.02	0.01
query21	15.46	0.56	0.56
query22	2.06	1.92	1.78
query23	17.09	1.12	0.94
query24	3.87	3.78	2.14
query25	0.37	0.15	0.04
query26	0.76	0.16	0.15
query27	0.05	0.03	0.03
query28	5.57	0.75	0.73
query29	12.76	2.35	2.41
query30	0.59	0.54	0.53
query31	2.82	0.39	0.37
query32	3.35	0.51	0.50
query33	3.09	3.07	3.05
query34	15.27	4.85	4.82
query35	4.88	4.87	4.85
query36	1.14	1.01	1.02
query37	0.06	0.05	0.05
query38	0.04	0.02	0.03
query39	0.02	0.01	0.02
query40	0.16	0.14	0.15
query41	0.06	0.02	0.02
query42	0.02	0.01	0.02
query43	0.02	0.01	0.02
Total cold run time: 100.56 s
Total hot run time: 32.26 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit f400fd3ac1493244eb061745848db718b18ebd51 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.6 seconds inserted 10000000 Rows, about 462K ops/s

@zzzxl1993
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 2, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.75% (8104/21466)
Line Coverage: 29.38% (66369/225871)
Region Coverage: 28.90% (34255/118544)
Branch Coverage: 24.78% (17602/71030)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4112111e1347ae7b9fbd6256d9973bbe8801681b_4112111e1347ae7b9fbd6256d9973bbe8801681b/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 49622 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4112111e1347ae7b9fbd6256d9973bbe8801681b, data reload: false

------ Round 1 ----------------------------------
q1	17718	4414	4344	4344
q2	2075	156	150	150
q3	10473	1882	1955	1882
q4	10413	1353	1325	1325
q5	8555	3828	3888	3828
q6	238	153	128	128
q7	2063	1658	1607	1607
q8	9544	2720	2718	2718
q9	14027	10387	10198	10198
q10	8622	3499	3515	3499
q11	413	250	247	247
q12	473	305	302	302
q13	18342	3947	3972	3947
q14	366	320	324	320
q15	507	461	458	458
q16	674	569	579	569
q17	1143	937	901	901
q18	7317	6980	6820	6820
q19	1774	1642	1597	1597
q20	545	314	299	299
q21	4400	4144	4044	4044
q22	522	459	439	439
Total cold run time: 120204 ms
Total hot run time: 49622 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4339	4307	4324	4307
q2	325	229	223	223
q3	4115	4145	4157	4145
q4	2752	2733	2733	2733
q5	7137	7091	7049	7049
q6	243	121	122	121
q7	3248	2820	2891	2820
q8	4334	4437	4492	4437
q9	16802	16876	16680	16680
q10	4263	4254	4244	4244
q11	750	685	679	679
q12	1024	874	857	857
q13	7335	3725	3751	3725
q14	449	428	422	422
q15	501	455	451	451
q16	735	700	671	671
q17	3753	3838	3754	3754
q18	8752	8688	8934	8688
q19	1735	1713	1716	1713
q20	2341	2108	2140	2108
q21	8430	8406	8363	8363
q22	1019	976	973	973
Total cold run time: 84382 ms
Total hot run time: 79163 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 203869 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4112111e1347ae7b9fbd6256d9973bbe8801681b, data reload: false

query1	929	421	375	375
query2	6546	2916	2707	2707
query3	6920	203	202	202
query4	20257	18037	17942	17942
query5	19740	6507	6498	6498
query6	291	216	230	216
query7	4164	291	309	291
query8	440	449	406	406
query9	3113	2662	2588	2588
query10	411	308	305	305
query11	11291	10712	10752	10712
query12	125	79	73	73
query13	5607	706	686	686
query14	17275	13465	13811	13465
query15	367	245	251	245
query16	6485	280	266	266
query17	1702	1465	885	885
query18	2304	416	407	407
query19	222	154	147	147
query20	81	76	79	76
query21	190	98	93	93
query22	5115	5011	5024	5011
query23	32703	31942	32067	31942
query24	7082	6651	6564	6564
query25	527	442	423	423
query26	626	168	161	161
query27	2035	296	295	295
query28	6138	2351	2324	2324
query29	2927	2750	2723	2723
query30	249	170	168	168
query31	932	751	780	751
query32	67	67	61	61
query33	407	265	263	263
query34	852	470	475	470
query35	1125	962	952	952
query36	1186	1263	1183	1183
query37	89	62	58	58
query38	3084	2963	2927	2927
query39	1374	1316	1315	1315
query40	250	92	97	92
query41	47	45	44	44
query42	80	80	90	80
query43	727	706	680	680
query44	1133	709	718	709
query45	247	236	235	235
query46	1232	958	960	958
query47	1959	1748	1758	1748
query48	1018	721	688	688
query49	630	366	368	366
query50	868	613	604	604
query51	4733	4675	4621	4621
query52	95	79	80	79
query53	454	318	329	318
query54	2676	2490	2416	2416
query55	97	73	86	73
query56	244	217	220	217
query57	1112	1166	1046	1046
query58	229	213	199	199
query59	4203	4137	4140	4137
query60	218	208	199	199
query61	103	98	101	98
query62	837	444	473	444
query63	490	347	340	340
query64	2668	1492	1502	1492
query65	3622	3563	3535	3535
query66	803	388	385	385
query67	15670	15357	16826	15357
query68	9624	649	654	649
query69	582	364	358	358
query70	1667	1223	1323	1223
query71	415	311	317	311
query72	6585	3490	3500	3490
query73	736	310	315	310
query74	6303	5802	5808	5802
query75	5398	3741	3685	3685
query76	6171	1140	1220	1140
query77	1046	248	250	248
query78	12786	11854	11577	11577
query79	9647	651	656	651
query80	1281	396	409	396
query81	484	238	236	236
query82	1632	94	100	94
query83	173	133	130	130
query84	267	70	70	70
query85	895	338	344	338
query86	334	310	302	302
query87	3236	3013	2990	2990
query88	4762	2310	2299	2299
query89	480	293	287	287
query90	1943	206	212	206
query91	177	153	142	142
query92	58	49	56	49
query93	7025	597	567	567
query94	686	209	209	209
query95	1100	1056	1044	1044
query96	639	329	320	320
query97	6551	6327	6356	6327
query98	182	176	171	171
query99	2992	922	912	912
Total cold run time: 317280 ms
Total hot run time: 203869 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.98 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4112111e1347ae7b9fbd6256d9973bbe8801681b, data reload: false

query1	0.03	0.02	0.02
query2	0.07	0.02	0.02
query3	0.24	0.05	0.05
query4	1.77	0.08	0.08
query5	0.53	0.52	0.51
query6	1.28	0.61	0.61
query7	0.01	0.01	0.01
query8	0.03	0.02	0.02
query9	0.53	0.50	0.47
query10	0.52	0.54	0.52
query11	0.12	0.09	0.09
query12	0.12	0.09	0.09
query13	0.62	0.62	0.62
query14	0.80	0.79	0.79
query15	0.77	0.76	0.77
query16	0.39	0.38	0.36
query17	1.02	1.03	0.99
query18	0.22	0.25	0.24
query19	1.90	1.86	1.87
query20	0.01	0.01	0.01
query21	15.44	0.55	0.55
query22	1.96	2.20	1.91
query23	17.28	1.02	0.97
query24	5.95	1.22	0.87
query25	0.39	0.09	0.05
query26	0.68	0.18	0.16
query27	0.04	0.04	0.03
query28	7.03	0.76	0.70
query29	12.61	2.34	2.34
query30	0.58	0.50	0.55
query31	2.81	0.38	0.38
query32	3.38	0.50	0.49
query33	3.10	3.05	3.05
query34	15.26	4.78	4.77
query35	4.86	4.86	4.83
query36	1.07	1.01	1.01
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.14	0.14
query41	0.07	0.02	0.01
query42	0.02	0.02	0.02
query43	0.02	0.02	0.01
Total cold run time: 103.81 s
Total hot run time: 30.98 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 4112111e1347ae7b9fbd6256d9973bbe8801681b with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       22.0 seconds inserted 10000000 Rows, about 454K ops/s

@xiaokang xiaokang merged commit 373aa35 into apache:branch-2.0 Aug 2, 2024
21 of 24 checks passed
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@zzzxl1993 @Baymine
[fix](inverted index) in_list support inverted index apache#37921 apa…
471af2a 
…che#37842 (apache#38738)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
kind/test
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@zzzxl1993 @doris-robot @xiaokang