Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[enhencement](hive-catalog) add hive.recursive_directories_table and hive.ignore_absent_partitions configs for HiveCatalog #39494

Merged
merged 2 commits into from
Aug 26, 2024
Merged

[enhencement](hive-catalog) add hive.recursive_directories_table and hive.ignore_absent_partitions configs for HiveCatalog #39494

merged 2 commits into from
Aug 26, 2024

Conversation

BePPPower
Copy link
Contributor

@BePPPower BePPPower commented Aug 16, 2024 •
edited
Loading

Add 2 configs for Hive Catalog:

  1. hive.recursive_directories_table: Enable reading data from subdirectories of table or partition locations. If disabled, subdirectories are ignored. Default value is false.

  2. hive.ignore_absent_partitions: Ignore partitions when the file system location does not exist rather than failing the query. This skips data that may be expected to be part of the table. Default value is true.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@BePPPower
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38220 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ad376079c3e83972905a0ea2c13ea3948ffdf281, data reload: false

------ Round 1 ----------------------------------
q1	17863	4350	4272	4272
q2	2056	219	225	219
q3	11771	975	1139	975
q4	10523	814	745	745
q5	7776	2827	2832	2827
q6	267	159	159	159
q7	1035	666	653	653
q8	9606	2079	2138	2079
q9	8795	6568	6589	6568
q10	7072	2207	2216	2207
q11	493	270	277	270
q12	428	258	268	258
q13	17794	3030	2988	2988
q14	300	265	265	265
q15	561	515	524	515
q16	518	428	415	415
q17	994	736	789	736
q18	7503	6895	6684	6684
q19	6762	1159	1124	1124
q20	720	359	371	359
q21	3924	2926	2866	2866
q22	1112	1036	1044	1036
Total cold run time: 117873 ms
Total hot run time: 38220 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4509	4328	4320	4320
q2	411	309	311	309
q3	2890	2699	2707	2699
q4	2056	1822	1665	1665
q5	5657	5763	5611	5611
q6	250	148	149	148
q7	2164	1775	1745	1745
q8	3285	3491	3466	3466
q9	8820	8795	8853	8795
q10	3596	3365	3307	3307
q11	622	518	516	516
q12	865	682	676	676
q13	16528	3256	3116	3116
q14	317	287	288	287
q15	552	520	507	507
q16	520	464	463	463
q17	1844	1576	1534	1534
q18	8618	8018	7756	7756
q19	8098	1488	1674	1488
q20	2194	1890	1874	1874
q21	14444	5560	5406	5406
q22	1164	1101	1080	1080
Total cold run time: 89404 ms
Total hot run time: 56768 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196184 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ad376079c3e83972905a0ea2c13ea3948ffdf281, data reload: false

query1	1341	889	897	889
query2	6634	1985	2074	1985
query3	10689	3815	3786	3786
query4	58232	27632	23259	23259
query5	6149	644	723	644
query6	545	212	205	205
query7	6435	335	332	332
query8	541	449	444	444
query9	9230	2563	2527	2527
query10	619	344	333	333
query11	17652	14994	15687	14994
query12	193	140	143	140
query13	1709	440	443	440
query14	12175	7145	7137	7137
query15	271	198	193	193
query16	7790	533	532	532
query17	1254	610	643	610
query18	2113	354	347	347
query19	313	172	170	170
query20	156	143	141	141
query21	251	148	145	145
query22	4797	4505	4424	4424
query23	34730	33852	34187	33852
query24	5652	2970	2996	2970
query25	588	446	438	438
query26	733	192	201	192
query27	1839	312	320	312
query28	3914	2216	2213	2213
query29	765	442	440	440
query30	215	178	186	178
query31	997	831	788	788
query32	121	75	76	75
query33	560	345	344	344
query34	896	502	502	502
query35	850	764	768	764
query36	1107	958	994	958
query37	158	104	101	101
query38	4078	3883	3931	3883
query39	1563	1499	1469	1469
query40	237	156	155	155
query41	141	136	138	136
query42	134	112	115	112
query43	554	521	530	521
query44	1103	788	795	788
query45	228	197	197	197
query46	1124	791	772	772
query47	1923	1845	1849	1845
query48	411	342	346	342
query49	919	579	576	576
query50	862	474	461	461
query51	6865	6727	6743	6727
query52	115	106	111	106
query53	297	227	230	227
query54	640	509	501	501
query55	91	90	86	86
query56	326	303	318	303
query57	1251	1126	1126	1126
query58	315	303	306	303
query59	3172	2828	3004	2828
query60	349	326	323	323
query61	147	148	148	148
query62	777	693	713	693
query63	259	221	225	221
query64	4396	2397	1858	1858
query65	3237	3186	3173	3173
query66	1027	685	728	685
query67	15303	15138	15005	15005
query68	7740	606	585	585
query69	656	431	337	337
query70	1276	1128	1166	1128
query71	522	318	313	313
query72	6670	2332	2101	2101
query73	1886	362	359	359
query74	9369	8816	8759	8759
query75	3825	2678	2759	2678
query76	4485	1021	1005	1005
query77	804	437	499	437
query78	9950	9175	9056	9056
query79	6812	555	557	555
query80	2261	621	608	608
query81	613	265	257	257
query82	641	163	159	159
query83	350	218	213	213
query84	299	97	98	97
query85	877	352	358	352
query86	351	323	328	323
query87	4424	4172	4337	4172
query88	4102	2525	2550	2525
query89	439	322	329	322
query90	1953	224	222	222
query91	153	173	127	127
query92	86	77	77	77
query93	5489	553	551	551
query94	763	324	333	324
query95	398	295	285	285
query96	636	285	283	283
query97	3257	3122	3085	3085
query98	236	229	223	223
query99	1572	1292	1324	1292
Total cold run time: 336953 ms
Total hot run time: 196184 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.06 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ad376079c3e83972905a0ea2c13ea3948ffdf281, data reload: false

query1	0.05	0.05	0.04
query2	0.08	0.04	0.04
query3	0.23	0.06	0.05
query4	1.67	0.08	0.08
query5	0.52	0.51	0.49
query6	1.14	0.72	0.74
query7	0.03	0.02	0.02
query8	0.06	0.05	0.05
query9	0.55	0.50	0.48
query10	0.55	0.55	0.55
query11	0.15	0.12	0.12
query12	0.15	0.13	0.13
query13	0.61	0.60	0.60
query14	0.78	0.79	0.79
query15	0.86	0.82	0.83
query16	0.37	0.38	0.38
query17	1.06	1.05	1.07
query18	0.22	0.23	0.22
query19	1.86	1.85	1.73
query20	0.01	0.01	0.02
query21	15.42	0.68	0.68
query22	4.00	6.81	2.60
query23	18.27	1.51	1.23
query24	2.09	0.23	0.23
query25	0.17	0.08	0.08
query26	0.30	0.22	0.22
query27	0.46	0.23	0.23
query28	13.30	1.04	1.03
query29	12.69	3.34	3.36
query30	0.42	0.24	0.24
query31	2.82	0.40	0.40
query32	3.24	0.49	0.49
query33	2.94	2.97	2.99
query34	17.32	4.34	4.35
query35	4.43	4.44	4.40
query36	0.68	0.48	0.48
query37	0.21	0.18	0.17
query38	0.19	0.17	0.16
query39	0.07	0.06	0.06
query40	0.18	0.15	0.16
query41	0.11	0.06	0.07
query42	0.07	0.07	0.07
query43	0.07	0.06	0.07
Total cold run time: 110.4 s
Total hot run time: 32.06 s

@BePPPower
Copy link
Contributor Author

run buildall

@BePPPower BePPPower changed the title [enhencement](hive-catalog) add hive.recursive-directories-table and hive.ignore-absent-partitions configs for HiveCatalog [enhencement](hive-catalog) add hive.recursive_directories_table and hive.ignore_absent_partitions configs for HiveCatalog Aug 21, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 38801 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dc55fc54dee628d66255101f476b889ff609c556, data reload: false

------ Round 1 ----------------------------------
q1	18020	4332	4330	4330
q2	2059	207	212	207
q3	10725	1133	1095	1095
q4	10169	680	797	680
q5	7798	2843	2833	2833
q6	265	160	158	158
q7	1011	667	669	667
q8	9374	2074	2076	2074
q9	7064	6609	6584	6584
q10	7104	2241	2282	2241
q11	485	270	270	270
q12	446	262	258	258
q13	18973	3041	3016	3016
q14	307	266	261	261
q15	589	521	546	521
q16	554	441	426	426
q17	1030	714	713	713
q18	8197	7173	7222	7173
q19	9541	1087	1168	1087
q20	1183	360	363	360
q21	3878	3095	2830	2830
q22	1128	1063	1017	1017
Total cold run time: 119900 ms
Total hot run time: 38801 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5422	4379	4293	4293
q2	583	319	306	306
q3	2909	2778	2732	2732
q4	2031	1727	1759	1727
q5	5618	5646	5595	5595
q6	246	151	153	151
q7	2177	1773	1755	1755
q8	3281	3363	3351	3351
q9	8752	8703	8742	8703
q10	3496	3316	3288	3288
q11	630	536	509	509
q12	847	659	654	654
q13	17916	3154	3178	3154
q14	346	318	313	313
q15	555	528	528	528
q16	513	437	477	437
q17	1821	1523	1571	1523
q18	8157	7744	7677	7677
q19	8831	1628	1351	1351
q20	2819	1923	1868	1868
q21	5393	5307	5296	5296
q22	1201	1050	1071	1050
Total cold run time: 83544 ms
Total hot run time: 56261 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192020 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dc55fc54dee628d66255101f476b889ff609c556, data reload: false

query1	962	399	396	396
query2	6759	2079	1984	1984
query3	6420	223	232	223
query4	30546	23302	23273	23273
query5	4358	706	698	698
query6	303	220	214	214
query7	4450	326	323	323
query8	484	433	436	433
query9	8572	2524	2507	2507
query10	492	349	341	341
query11	16134	15095	15038	15038
query12	180	139	139	139
query13	1647	462	454	454
query14	9934	7184	6888	6888
query15	262	200	193	193
query16	7701	545	533	533
query17	1537	612	597	597
query18	1950	347	336	336
query19	236	170	168	168
query20	145	139	139	139
query21	260	144	145	144
query22	4391	4255	4200	4200
query23	34028	33094	33370	33094
query24	11502	2954	2883	2883
query25	662	423	418	418
query26	1398	179	186	179
query27	2746	308	307	307
query28	7215	2106	2092	2092
query29	927	454	448	448
query30	331	195	186	186
query31	1032	852	843	843
query32	124	80	81	80
query33	815	349	343	343
query34	911	510	510	510
query35	875	775	775	775
query36	1102	961	960	960
query37	164	103	105	103
query38	3987	3866	3883	3866
query39	1534	1458	1465	1458
query40	313	154	156	154
query41	143	138	138	138
query42	133	119	121	119
query43	544	478	517	478
query44	1296	787	802	787
query45	224	196	199	196
query46	1138	792	819	792
query47	1934	1855	1834	1834
query48	412	338	348	338
query49	1158	591	583	583
query50	866	485	470	470
query51	7294	7255	7081	7081
query52	120	111	129	111
query53	301	228	225	225
query54	982	512	501	501
query55	90	89	89	89
query56	335	307	307	307
query57	1240	1126	1135	1126
query58	314	327	315	315
query59	2957	2782	2876	2782
query60	356	326	331	326
query61	148	145	149	145
query62	867	691	715	691
query63	263	227	227	227
query64	5463	2357	3241	2357
query65	3225	3175	3167	3167
query66	1271	691	687	687
query67	15706	15064	15139	15064
query68	6197	587	602	587
query69	764	402	354	354
query70	1200	1185	1163	1163
query71	534	309	325	309
query72	7684	2302	2000	2000
query73	823	359	359	359
query74	9374	8879	8803	8803
query75	4670	2740	2761	2740
query76	4296	1025	1056	1025
query77	879	438	444	438
query78	9829	10165	9261	9261
query79	7489	564	557	557
query80	1187	616	605	605
query81	633	262	257	257
query82	814	159	160	159
query83	430	224	215	215
query84	289	160	99	99
query85	1115	353	351	351
query86	470	312	344	312
query87	4350	4245	4359	4245
query88	4895	2531	2538	2531
query89	483	334	325	325
query90	2001	232	234	232
query91	163	125	128	125
query92	92	76	77	76
query93	4495	576	553	553
query94	1017	320	327	320
query95	388	293	292	292
query96	623	291	287	287
query97	3273	3051	3080	3051
query98	244	231	221	221
query99	1618	1325	1312	1312
Total cold run time: 308488 ms
Total hot run time: 192020 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.33 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dc55fc54dee628d66255101f476b889ff609c556, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.24	0.06	0.06
query4	1.65	0.07	0.08
query5	0.49	0.52	0.49
query6	1.13	0.73	0.74
query7	0.02	0.02	0.02
query8	0.06	0.05	0.05
query9	0.54	0.48	0.50
query10	0.55	0.55	0.56
query11	0.16	0.12	0.12
query12	0.16	0.12	0.13
query13	0.63	0.61	0.60
query14	0.76	0.80	0.79
query15	0.85	0.82	0.83
query16	0.37	0.38	0.36
query17	0.96	0.98	1.04
query18	0.21	0.21	0.21
query19	1.76	1.69	1.85
query20	0.01	0.01	0.02
query21	15.40	0.67	0.65
query22	4.27	7.34	2.16
query23	18.33	1.33	1.33
query24	2.24	0.23	0.22
query25	0.15	0.09	0.09
query26	0.28	0.19	0.18
query27	0.09	0.08	0.08
query28	13.19	1.04	1.03
query29	12.69	3.34	3.36
query30	0.43	0.24	0.25
query31	2.82	0.40	0.40
query32	3.24	0.49	0.50
query33	2.99	2.97	2.97
query34	17.02	4.32	4.37
query35	4.41	4.38	4.43
query36	0.67	0.49	0.49
query37	0.20	0.18	0.17
query38	0.19	0.17	0.17
query39	0.07	0.05	0.05
query40	0.19	0.15	0.15
query41	0.12	0.07	0.07
query42	0.08	0.06	0.07
query43	0.07	0.07	0.06
Total cold run time: 109.82 s
Total hot run time: 31.33 s

@BePPPower
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38464 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dc55fc54dee628d66255101f476b889ff609c556, data reload: false

------ Round 1 ----------------------------------
q1	18208	4476	4349	4349
q2	2092	213	212	212
q3	11156	979	1200	979
q4	10524	822	775	775
q5	7806	2845	2781	2781
q6	264	159	156	156
q7	1046	672	656	656
q8	9402	2152	2134	2134
q9	7093	6589	6541	6541
q10	7052	2281	2230	2230
q11	489	277	286	277
q12	428	243	244	243
q13	18169	2984	2991	2984
q14	306	245	244	244
q15	562	520	518	518
q16	546	407	407	407
q17	1008	700	787	700
q18	7501	6963	6917	6917
q19	6351	1169	1149	1149
q20	722	348	343	343
q21	3918	2919	2833	2833
q22	1134	1067	1036	1036
Total cold run time: 115777 ms
Total hot run time: 38464 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4559	4301	4311	4301
q2	406	302	304	302
q3	2884	2638	2656	2638
q4	1989	1766	1690	1690
q5	5689	5649	5739	5649
q6	270	162	163	162
q7	2160	1813	1806	1806
q8	3291	3505	3578	3505
q9	8832	8881	8810	8810
q10	3567	3424	3379	3379
q11	627	546	538	538
q12	827	651	681	651
q13	17249	3175	3165	3165
q14	313	290	287	287
q15	551	507	514	507
q16	515	444	450	444
q17	1851	1550	1533	1533
q18	8346	7828	7702	7702
q19	7355	1518	1598	1518
q20	2183	1916	1904	1904
q21	14071	5272	5413	5272
q22	1210	1151	1111	1111
Total cold run time: 88745 ms
Total hot run time: 56874 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197399 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dc55fc54dee628d66255101f476b889ff609c556, data reload: false

query1	1342	904	888	888
query2	6577	1917	1857	1857
query3	10733	3963	3815	3815
query4	58379	27475	23497	23497
query5	5878	730	725	725
query6	545	216	212	212
query7	6389	345	329	329
query8	542	452	449	449
query9	9286	2560	2533	2533
query10	586	353	330	330
query11	18653	15118	15627	15118
query12	218	128	127	127
query13	1655	444	440	440
query14	11958	7363	7540	7363
query15	289	194	192	192
query16	7608	498	529	498
query17	1143	630	633	630
query18	2123	361	348	348
query19	301	173	189	173
query20	149	137	136	136
query21	245	145	151	145
query22	4513	4279	4335	4279
query23	34397	34079	34055	34055
query24	5617	3003	3007	3003
query25	610	428	441	428
query26	709	187	188	187
query27	1768	311	306	306
query28	3916	2229	2212	2212
query29	752	468	461	461
query30	235	204	193	193
query31	1049	873	873	873
query32	107	80	81	80
query33	525	351	358	351
query34	907	517	511	511
query35	871	758	754	754
query36	1105	968	976	968
query37	160	104	102	102
query38	4010	3956	3923	3923
query39	1520	1491	1478	1478
query40	229	155	156	155
query41	140	139	137	137
query42	137	117	115	115
query43	543	503	505	503
query44	1108	794	801	794
query45	229	200	196	196
query46	1140	766	756	756
query47	1911	1822	1859	1822
query48	423	342	343	342
query49	903	604	585	585
query50	859	464	472	464
query51	7173	7141	7118	7118
query52	123	110	109	109
query53	301	226	227	226
query54	611	511	495	495
query55	90	88	87	87
query56	336	312	315	312
query57	1205	1129	1132	1129
query58	303	299	300	299
query59	2844	2825	2759	2759
query60	362	325	332	325
query61	148	148	151	148
query62	775	706	697	697
query63	262	226	256	226
query64	3412	1901	1835	1835
query65	3277	3211	3157	3157
query66	974	674	674	674
query67	15549	15231	14840	14840
query68	7588	593	592	592
query69	668	451	353	353
query70	1234	1179	1096	1096
query71	565	311	313	311
query72	6736	2365	2130	2130
query73	832	359	357	357
query74	9213	8871	8925	8871
query75	3940	2752	2771	2752
query76	4414	1065	1068	1065
query77	874	438	442	438
query78	9740	9147	9080	9080
query79	6421	552	548	548
query80	1777	601	636	601
query81	616	272	256	256
query82	795	164	161	161
query83	359	215	216	215
query84	302	97	98	97
query85	1006	365	357	357
query86	375	320	327	320
query87	4537	4241	4269	4241
query88	4032	2466	2490	2466
query89	453	322	319	319
query90	2295	229	223	223
query91	155	128	128	128
query92	84	76	77	76
query93	5384	556	557	556
query94	1060	319	324	319
query95	393	296	300	296
query96	626	289	283	283
query97	3279	3105	3091	3091
query98	243	233	238	233
query99	1597	1308	1294	1294
Total cold run time: 334330 ms
Total hot run time: 197399 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.76 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dc55fc54dee628d66255101f476b889ff609c556, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.07	0.08
query5	0.52	0.48	0.49
query6	1.14	0.73	0.72
query7	0.02	0.02	0.01
query8	0.06	0.04	0.05
query9	0.55	0.48	0.47
query10	0.55	0.54	0.52
query11	0.16	0.13	0.12
query12	0.15	0.12	0.13
query13	0.62	0.60	0.58
query14	0.77	0.79	0.80
query15	0.85	0.82	0.83
query16	0.38	0.38	0.39
query17	0.95	1.03	0.96
query18	0.21	0.21	0.21
query19	1.86	1.74	1.80
query20	0.02	0.01	0.01
query21	15.39	0.69	0.67
query22	4.46	7.22	1.64
query23	18.26	1.41	1.32
query24	2.10	0.24	0.22
query25	0.16	0.09	0.09
query26	0.27	0.18	0.18
query27	0.08	0.09	0.08
query28	13.26	1.03	1.02
query29	12.69	3.35	3.33
query30	0.44	0.20	0.19
query31	2.80	0.41	0.40
query32	3.25	0.49	0.49
query33	2.94	3.03	2.93
query34	16.95	4.38	4.40
query35	4.41	4.42	4.44
query36	0.65	0.49	0.49
query37	0.22	0.18	0.17
query38	0.18	0.17	0.17
query39	0.06	0.05	0.05
query40	0.19	0.15	0.15
query41	0.12	0.07	0.07
query42	0.08	0.07	0.07
query43	0.07	0.06	0.06
Total cold run time: 109.86 s
Total hot run time: 30.76 s

@morningman morningman force-pushed the addTwoConfigsForHiveCatalog branch from dc55fc5 to 1cf2633 Compare August 22, 2024 08:32
morningman
morningman approved these changes Aug 22, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Aug 22, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@morningman
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38140 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1cf2633f20b66326c45c500397abf0e5b6e85b2e, data reload: false

------ Round 1 ----------------------------------
q1	17607	4446	4301	4301
q2	2026	182	169	169
q3	10479	1176	1129	1129
q4	10136	775	632	632
q5	7756	2961	2833	2833
q6	227	139	134	134
q7	966	626	603	603
q8	9523	2082	2092	2082
q9	8855	6549	6536	6536
q10	7072	2150	2169	2150
q11	467	244	246	244
q12	553	222	223	222
q13	17769	3045	3050	3045
q14	274	243	222	222
q15	524	494	493	493
q16	490	407	382	382
q17	979	717	755	717
q18	7559	6838	6916	6838
q19	1383	1153	1097	1097
q20	698	334	338	334
q21	3937	3035	2976	2976
q22	1118	1031	1001	1001
Total cold run time: 110398 ms
Total hot run time: 38140 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4317	4368	4249	4249
q2	363	277	260	260
q3	2951	2662	2654	2654
q4	1951	1656	1613	1613
q5	5384	5405	5420	5405
q6	219	127	126	126
q7	2086	1766	1729	1729
q8	3237	3358	3381	3358
q9	8431	8457	8405	8405
q10	3454	3215	3195	3195
q11	588	501	500	500
q12	787	635	624	624
q13	14059	3030	2999	2999
q14	311	278	282	278
q15	531	485	466	466
q16	481	415	440	415
q17	1802	1498	1454	1454
q18	7754	7481	7435	7435
q19	1673	1593	1638	1593
q20	2050	1819	1822	1819
q21	5348	5243	5333	5243
q22	1123	1039	1004	1004
Total cold run time: 68900 ms
Total hot run time: 54824 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186569 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1cf2633f20b66326c45c500397abf0e5b6e85b2e, data reload: false

query1	871	371	364	364
query2	5393	1868	1889	1868
query3	3525	209	217	209
query4	33183	23167	23256	23167
query5	2612	511	497	497
query6	210	160	166	160
query7	3556	297	288	288
query8	242	202	207	202
query9	6371	2491	2480	2480
query10	395	281	257	257
query11	17408	15039	15034	15034
query12	155	106	101	101
query13	1257	365	369	365
query14	9157	6920	6910	6910
query15	240	171	169	169
query16	7153	459	479	459
query17	1063	542	558	542
query18	2001	304	319	304
query19	183	140	138	138
query20	117	112	110	110
query21	205	105	101	101
query22	4381	4290	4326	4290
query23	33965	33507	33319	33319
query24	11776	2837	2863	2837
query25	692	377	404	377
query26	1774	155	155	155
query27	2375	278	274	274
query28	5944	2065	2047	2047
query29	818	403	398	398
query30	285	158	151	151
query31	956	763	759	759
query32	74	56	59	56
query33	577	320	288	288
query34	866	477	477	477
query35	850	734	723	723
query36	1066	942	920	920
query37	237	83	79	79
query38	3973	3812	3869	3812
query39	1427	1394	1381	1381
query40	250	116	116	116
query41	52	48	47	47
query42	118	96	106	96
query43	502	473	461	461
query44	1199	743	764	743
query45	196	168	167	167
query46	1105	746	733	733
query47	1921	1802	1837	1802
query48	385	288	289	288
query49	990	416	422	416
query50	816	419	400	400
query51	7287	7076	7085	7076
query52	99	89	86	86
query53	247	178	175	175
query54	574	455	447	447
query55	79	73	80	73
query56	275	250	250	250
query57	1185	1088	1105	1088
query58	221	222	223	222
query59	2976	2827	2722	2722
query60	286	258	259	258
query61	104	96	99	96
query62	807	649	653	649
query63	211	188	181	181
query64	4804	2254	1738	1738
query65	3265	3167	3150	3150
query66	940	340	336	336
query67	15755	15216	15297	15216
query68	2845	591	587	587
query69	398	267	284	267
query70	1137	1129	1018	1018
query71	330	278	269	269
query72	2688	2091	2110	2091
query73	699	318	319	318
query74	9069	8711	8778	8711
query75	3338	2722	2643	2643
query76	1499	976	990	976
query77	460	326	322	322
query78	9537	9123	9649	9123
query79	1016	530	533	530
query80	693	511	493	493
query81	451	223	230	223
query82	287	138	142	138
query83	176	150	150	150
query84	234	81	76	76
query85	666	278	326	278
query86	298	304	303	303
query87	4293	4192	4274	4192
query88	2874	2356	2284	2284
query89	370	286	288	286
query90	1835	195	198	195
query91	126	102	102	102
query92	60	51	51	51
query93	1043	540	536	536
query94	681	291	292	291
query95	331	262	269	262
query96	580	264	269	264
query97	3165	3088	3047	3047
query98	214	202	196	196
query99	1459	1247	1295	1247
Total cold run time: 266834 ms
Total hot run time: 186569 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.25 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1cf2633f20b66326c45c500397abf0e5b6e85b2e, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.06
query4	1.68	0.09	0.08
query5	0.52	0.48	0.51
query6	1.14	0.72	0.72
query7	0.02	0.02	0.02
query8	0.05	0.05	0.06
query9	0.54	0.48	0.47
query10	0.53	0.54	0.53
query11	0.16	0.12	0.11
query12	0.16	0.12	0.13
query13	0.61	0.58	0.58
query14	0.76	0.78	0.80
query15	0.84	0.81	0.81
query16	0.37	0.37	0.37
query17	1.06	1.04	0.98
query18	0.21	0.20	0.20
query19	1.83	1.80	1.78
query20	0.01	0.02	0.01
query21	15.39	0.66	0.65
query22	4.03	6.25	2.59
query23	18.34	1.37	1.32
query24	2.20	0.21	0.22
query25	0.14	0.10	0.07
query26	0.27	0.19	0.18
query27	0.07	0.07	0.08
query28	13.24	1.01	1.00
query29	12.66	3.32	3.28
query30	0.24	0.05	0.05
query31	2.88	0.39	0.39
query32	3.29	0.47	0.49
query33	2.98	2.94	3.02
query34	17.01	4.38	4.43
query35	4.47	4.48	4.36
query36	0.65	0.47	0.48
query37	0.19	0.16	0.16
query38	0.15	0.16	0.15
query39	0.04	0.04	0.04
query40	0.15	0.13	0.13
query41	0.09	0.04	0.05
query42	0.06	0.04	0.05
query43	0.04	0.04	0.05
Total cold run time: 109.42 s
Total hot run time: 31.25 s

@morningman morningman merged commit 2f84873 into apache:master Aug 26, 2024
28 of 30 checks passed
dataroaring pushed a commit that referenced this pull request Aug 26, 2024
@BePPPower @dataroaring
[enhencement](hive-catalog) add hive.recursive_directories_table an…
219dc56 
…d `hive.ignore_absent_partitions` configs for HiveCatalog (#39494)

Add 2 configs for Hive Catalog:

1. `hive.recursive_directories_table`: Enable reading data from
subdirectories of table or partition locations. If disabled,
subdirectories are ignored. Default value is `false`.

2. `hive.ignore_absent_partitions`: Ignore partitions when the file
system location does not exist rather than failing the query. This skips
data that may be expected to be part of the table. Default value is
`true`.
@gavinchou gavinchou mentioned this pull request Oct 13, 2024
Open
morningman pushed a commit that referenced this pull request Feb 17, 2025
@BePPPower @morningman
[enhencement](hive-catalog) add hive.recursive_directories_table an…
2e5fe2e 
…d `hive.ignore_absent_partitions` configs for HiveCatalog (#39494)

Add 2 configs for Hive Catalog:

1. `hive.recursive_directories_table`: Enable reading data from
subdirectories of table or partition locations. If disabled,
subdirectories are ignored. Default value is `false`.

2. `hive.ignore_absent_partitions`: Ignore partitions when the file
system location does not exist rather than failing the query. This skips
data that may be expected to be part of the table. Default value is
`true`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morningman morningman morningman approved these changes

@wuwenchi wuwenchi wuwenchi approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.2-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@BePPPower @doris-robot @morningman @wuwenchi @dataroaring