Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improvement](file cache) Try to read from remote storage when opening segment with CachedRemoteFileReader #38645

Merged
merged 4 commits into from
Aug 6, 2024
Merged

[improvement](file cache) Try to read from remote storage when opening segment with CachedRemoteFileReader #38645

merged 4 commits into from
Aug 6, 2024

Conversation

gavinchou
Copy link
Contributor

We may encounter the error "Bad segment" file in some rare cases where the file cache may not hold the correct segment files. We should read the remote original segment files to increase robustness.

@gavinchou
[improvement](file cache) Try to read from remote storage when openin…
5811a85 
…g segment with CachedRemoteFileReader

We may encounter the error "Bad segment" file in some rare cases that
the file cache may not hold the correct segment files.
We should try to read the remote original segment files to increase robustness.
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@gavinchou
Copy link
Contributor Author

run buildall

@github-actions github-actions bot added the doing label Jul 31, 2024
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 41256 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5811a85ea4db92288f637ca951d9be100fb86eb9, data reload: false

------ Round 1 ----------------------------------
q1	17608	4045	4068	4045
q2	2023	205	219	205
q3	10445	1277	1375	1277
q4	10187	811	995	811
q5	7643	2892	2957	2892
q6	225	139	140	139
q7	1052	626	627	626
q8	9449	1742	1953	1742
q9	8769	6608	6586	6586
q10	8733	3835	3820	3820
q11	429	249	249	249
q12	453	241	233	233
q13	17773	2935	2942	2935
q14	276	249	244	244
q15	527	490	497	490
q16	534	406	393	393
q17	953	931	900	900
q18	7922	7273	7143	7143
q19	1593	1213	1210	1210
q20	564	321	339	321
q21	5288	4706	4778	4706
q22	360	289	289	289
Total cold run time: 112806 ms
Total hot run time: 41256 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4119	4023	4063	4023
q2	327	230	221	221
q3	3021	3018	3108	3018
q4	1955	1999	1963	1963
q5	5593	5482	5426	5426
q6	222	132	130	130
q7	2098	1737	1812	1737
q8	3307	3366	3321	3321
q9	8636	8666	8695	8666
q10	3986	4034	4050	4034
q11	564	443	451	443
q12	738	552	577	552
q13	16416	3136	3116	3116
q14	301	283	267	267
q15	523	490	495	490
q16	478	433	443	433
q17	1739	1742	1736	1736
q18	8188	7677	7711	7677
q19	1742	1722	1694	1694
q20	2065	1814	1872	1814
q21	5865	5461	5507	5461
q22	516	452	459	452
Total cold run time: 72399 ms
Total hot run time: 56674 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 170071 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5811a85ea4db92288f637ca951d9be100fb86eb9, data reload: false

query1	918	380	364	364
query2	6458	1708	1796	1708
query3	6654	230	231	230
query4	19020	17619	17363	17363
query5	3636	535	531	531
query6	280	194	173	173
query7	4604	307	304	304
query8	260	203	209	203
query9	8520	2404	2380	2380
query10	433	289	279	279
query11	10370	9896	9870	9870
query12	127	91	90	90
query13	1638	370	376	370
query14	10351	6996	8580	6996
query15	222	165	162	162
query16	6814	438	424	424
query17	937	570	540	540
query18	1891	282	281	281
query19	185	148	139	139
query20	92	89	83	83
query21	209	104	99	99
query22	4118	4108	4026	4026
query23	33648	33543	33605	33543
query24	9366	3133	3086	3086
query25	635	432	405	405
query26	736	155	165	155
query27	2375	287	283	283
query28	5796	2041	2015	2015
query29	917	424	458	424
query30	240	167	161	161
query31	946	758	779	758
query32	107	59	67	59
query33	678	323	342	323
query34	920	518	536	518
query35	919	775	775	775
query36	1023	913	883	883
query37	146	86	82	82
query38	2940	2900	2751	2751
query39	893	809	825	809
query40	211	117	114	114
query41	53	45	44	44
query42	123	102	102	102
query43	471	419	450	419
query44	1198	748	735	735
query45	213	182	193	182
query46	1105	798	792	792
query47	1760	1710	1720	1710
query48	372	295	291	291
query49	831	433	422	422
query50	912	439	438	438
query51	6770	6765	6592	6592
query52	109	99	96	96
query53	252	182	185	182
query54	627	465	465	465
query55	81	75	75	75
query56	283	256	255	255
query57	1128	1044	1016	1016
query58	277	303	305	303
query59	2648	2442	2584	2442
query60	283	280	279	279
query61	98	92	93	92
query62	891	669	660	660
query63	220	189	192	189
query64	4670	1905	1871	1871
query65	3229	3127	3116	3116
query66	884	349	348	348
query67	15212	14864	14888	14864
query68	4314	572	581	572
query69	448	307	307	307
query70	1124	1014	1076	1014
query71	398	291	287	287
query72	7276	2767	2489	2489
query73	765	329	325	325
query74	6006	5685	5686	5685
query75	3363	2736	2735	2735
query76	2202	1347	1413	1347
query77	435	313	319	313
query78	9401	8964	8841	8841
query79	2365	543	542	542
query80	1269	569	525	525
query81	561	228	230	228
query82	780	136	129	129
query83	237	178	171	171
query84	265	86	78	78
query85	1760	319	312	312
query86	474	295	313	295
query87	3280	3116	3126	3116
query88	3890	2427	2419	2419
query89	392	289	284	284
query90	1756	197	195	195
query91	123	101	99	99
query92	66	51	53	51
query93	2423	624	614	614
query94	776	299	294	294
query95	387	269	272	269
query96	673	285	278	278
query97	3269	3048	3045	3045
query98	240	201	196	196
query99	1604	1296	1277	1277
Total cold run time: 257255 ms
Total hot run time: 170071 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.75 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5811a85ea4db92288f637ca951d9be100fb86eb9, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.03
query3	0.22	0.05	0.05
query4	1.70	0.07	0.06
query5	0.48	0.48	0.49
query6	1.15	0.72	0.71
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.56	0.52	0.51
query10	0.57	0.55	0.56
query11	0.16	0.12	0.12
query12	0.14	0.12	0.13
query13	0.62	0.61	0.60
query14	0.78	0.81	0.78
query15	0.89	0.85	0.88
query16	0.34	0.36	0.36
query17	0.97	1.00	0.99
query18	0.22	0.21	0.21
query19	1.84	1.72	1.72
query20	0.01	0.00	0.01
query21	15.39	0.78	0.65
query22	4.27	7.74	1.13
query23	17.83	1.29	1.19
query24	2.25	0.22	0.22
query25	0.18	0.08	0.09
query26	0.32	0.22	0.21
query27	0.46	0.24	0.23
query28	13.17	1.00	0.99
query29	12.57	3.32	3.31
query30	0.26	0.06	0.05
query31	2.88	0.41	0.42
query32	3.23	0.50	0.49
query33	2.96	2.95	2.99
query34	15.45	4.25	4.26
query35	4.30	4.29	4.32
query36	0.68	0.47	0.49
query37	0.19	0.17	0.16
query38	0.15	0.15	0.15
query39	0.04	0.04	0.04
query40	0.16	0.12	0.13
query41	0.10	0.05	0.05
query42	0.05	0.05	0.06
query43	0.05	0.04	0.05
Total cold run time: 107.77 s
Total hot run time: 29.75 s

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 1, 2024

clang-tidy review says "All clean, LGTM! 👍"

platoneko
platoneko previously approved these changes Aug 1, 2024
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 1, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 1, 2024

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 1, 2024

PR approved by anyone and no changes requested.

@gavinchou
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41580 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 77a2d9313aadcdd71132fd06506a817a8fece0dd, data reload: false

------ Round 1 ----------------------------------
q1	18846	4163	4143	4143
q2	3196	219	201	201
q3	11540	1390	1405	1390
q4	10558	820	923	820
q5	8099	2943	2956	2943
q6	219	137	135	135
q7	1032	622	620	620
q8	9427	1812	1925	1812
q9	8461	6599	6552	6552
q10	8702	3860	3817	3817
q11	427	249	262	249
q12	414	235	222	222
q13	17765	2910	2924	2910
q14	269	249	245	245
q15	525	483	502	483
q16	500	408	402	402
q17	953	917	904	904
q18	7980	7291	7202	7202
q19	1387	1215	1215	1215
q20	578	319	344	319
q21	5295	4717	4779	4717
q22	351	286	279	279
Total cold run time: 116524 ms
Total hot run time: 41580 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4036	3988	3986	3986
q2	338	225	217	217
q3	2985	2990	3002	2990
q4	1864	1846	1840	1840
q5	5239	5250	5191	5191
q6	215	130	130	130
q7	2048	1672	1711	1672
q8	3165	3226	3217	3217
q9	8306	8247	8298	8247
q10	3794	3826	3824	3824
q11	547	467	435	435
q12	732	577	535	535
q13	10352	2946	2948	2946
q14	295	269	259	259
q15	513	472	476	472
q16	463	412	402	402
q17	1719	1714	1679	1679
q18	7754	7359	7290	7290
q19	1690	1654	1660	1654
q20	1974	1733	1758	1733
q21	5395	5161	5174	5161
q22	510	456	464	456
Total cold run time: 63934 ms
Total hot run time: 54336 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 169508 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 77a2d9313aadcdd71132fd06506a817a8fece0dd, data reload: false

query1	912	367	371	367
query2	6470	1702	1681	1681
query3	6677	215	224	215
query4	20271	17661	17452	17452
query5	4299	510	507	507
query6	279	171	177	171
query7	4606	296	290	290
query8	256	208	192	192
query9	8529	2369	2348	2348
query10	425	271	274	271
query11	10668	10219	10197	10197
query12	138	90	86	86
query13	1626	397	385	385
query14	9190	6897	7001	6897
query15	208	164	163	163
query16	7116	487	437	437
query17	960	579	567	567
query18	1922	290	295	290
query19	203	152	152	152
query20	94	87	88	87
query21	214	103	100	100
query22	4466	4297	4241	4241
query23	34007	33292	33175	33175
query24	10390	3104	3022	3022
query25	695	407	402	402
query26	1769	155	159	155
query27	2955	281	280	280
query28	6971	1965	1960	1960
query29	1361	437	433	433
query30	286	151	157	151
query31	941	744	759	744
query32	106	55	56	55
query33	721	314	317	314
query34	913	489	510	489
query35	874	739	716	716
query36	1004	856	881	856
query37	284	81	78	78
query38	2867	2774	2756	2756
query39	848	817	798	798
query40	297	114	122	114
query41	48	46	47	46
query42	125	100	99	99
query43	487	407	434	407
query44	1206	727	724	724
query45	213	179	181	179
query46	1088	830	789	789
query47	1805	1732	1712	1712
query48	377	299	301	299
query49	1208	437	444	437
query50	918	434	449	434
query51	6779	6743	6689	6689
query52	98	91	92	91
query53	259	189	188	188
query54	673	563	453	453
query55	78	73	75	73
query56	276	270	274	270
query57	1173	1057	1033	1033
query58	260	258	269	258
query59	2572	2252	2372	2252
query60	307	276	281	276
query61	98	95	95	95
query62	913	656	680	656
query63	219	188	182	182
query64	5876	1934	1902	1902
query65	3189	3119	3103	3103
query66	1452	341	341	341
query67	15285	15068	14787	14787
query68	5095	597	603	597
query69	703	364	322	322
query70	1144	1033	1095	1033
query71	481	276	280	276
query72	8038	2688	2481	2481
query73	938	340	340	340
query74	6109	5663	5604	5604
query75	4063	2713	2707	2707
query76	3513	1266	1285	1266
query77	633	307	311	307
query78	9474	9025	8800	8800
query79	2652	544	544	544
query80	1197	553	510	510
query81	567	225	228	225
query82	779	131	131	131
query83	286	170	177	170
query84	271	77	79	77
query85	1510	323	298	298
query86	470	321	308	308
query87	3310	3129	3107	3107
query88	3782	2458	2448	2448
query89	401	283	280	280
query90	1939	194	198	194
query91	127	100	102	100
query92	63	51	50	50
query93	3460	616	621	616
query94	899	304	288	288
query95	378	273	267	267
query96	616	277	280	277
query97	3253	3094	3046	3046
query98	214	200	195	195
query99	1629	1275	1291	1275
Total cold run time: 271794 ms
Total hot run time: 169508 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.81 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 77a2d9313aadcdd71132fd06506a817a8fece0dd, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.23	0.06	0.05
query4	1.67	0.07	0.07
query5	0.49	0.49	0.48
query6	1.14	0.71	0.71
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.57	0.52	0.50
query10	0.57	0.58	0.56
query11	0.16	0.12	0.11
query12	0.15	0.13	0.12
query13	0.61	0.61	0.59
query14	0.77	0.78	0.77
query15	0.89	0.87	0.87
query16	0.35	0.36	0.36
query17	1.03	0.97	0.99
query18	0.23	0.21	0.22
query19	1.86	1.77	1.81
query20	0.02	0.01	0.01
query21	15.40	0.79	0.68
query22	4.46	8.05	1.11
query23	17.69	1.26	1.31
query24	2.26	0.22	0.22
query25	0.19	0.08	0.08
query26	0.32	0.21	0.21
query27	0.45	0.24	0.23
query28	13.18	1.01	0.97
query29	12.55	3.28	3.28
query30	0.26	0.05	0.07
query31	2.85	0.41	0.41
query32	3.25	0.50	0.48
query33	2.93	3.00	2.93
query34	15.45	4.25	4.28
query35	4.28	4.30	4.33
query36	0.67	0.50	0.48
query37	0.20	0.17	0.16
query38	0.16	0.15	0.15
query39	0.04	0.03	0.04
query40	0.16	0.13	0.12
query41	0.10	0.04	0.05
query42	0.06	0.05	0.05
query43	0.04	0.05	0.04
Total cold run time: 107.87 s
Total hot run time: 29.81 s

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Aug 1, 2024
@gavinchou gavinchou force-pushed the gavin-try-remote-if-bad-segment branch from 282a513 to c8a5823 Compare August 1, 2024 10:31
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 1, 2024

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 1, 2024

clang-tidy review says "All clean, LGTM! 👍"

@gavinchou
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41598 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c8a5823a36b9eae92413d97dea1fe04d7875e9f8, data reload: false

------ Round 1 ----------------------------------
q1	17637	4092	4063	4063
q2	2021	196	196	196
q3	10478	1338	1353	1338
q4	10175	831	885	831
q5	7643	3001	2973	2973
q6	216	137	140	137
q7	1043	613	608	608
q8	9455	1919	1910	1910
q9	8511	6579	6564	6564
q10	8796	3858	3839	3839
q11	431	243	254	243
q12	444	233	228	228
q13	17782	2938	2930	2930
q14	275	249	242	242
q15	510	486	494	486
q16	523	402	386	386
q17	972	895	909	895
q18	7957	7266	7153	7153
q19	2103	1216	1209	1209
q20	561	322	339	322
q21	5262	4813	4771	4771
q22	344	279	274	274
Total cold run time: 113139 ms
Total hot run time: 41598 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4130	4011	4028	4011
q2	323	217	217	217
q3	3004	3008	3137	3008
q4	2012	2011	1991	1991
q5	5706	5471	5460	5460
q6	220	132	131	131
q7	2164	1829	1809	1809
q8	3321	3365	3384	3365
q9	8692	8655	8774	8655
q10	3956	4039	3936	3936
q11	579	450	478	450
q12	794	597	625	597
q13	16515	3151	3135	3135
q14	315	284	271	271
q15	537	480	505	480
q16	466	429	421	421
q17	1778	1734	1718	1718
q18	8264	7721	7616	7616
q19	1747	1724	1740	1724
q20	2060	1838	1835	1835
q21	5728	5476	5410	5410
q22	539	475	475	475
Total cold run time: 72850 ms
Total hot run time: 56715 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 168954 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c8a5823a36b9eae92413d97dea1fe04d7875e9f8, data reload: false

query1	927	377	376	376
query2	6480	1670	1683	1670
query3	6664	228	224	224
query4	19514	17368	17258	17258
query5	3655	508	488	488
query6	261	159	166	159
query7	4623	292	298	292
query8	246	190	197	190
query9	8502	2324	2325	2324
query10	441	290	268	268
query11	10655	9929	9924	9924
query12	120	91	88	88
query13	1630	375	383	375
query14	9900	6863	7244	6863
query15	218	163	166	163
query16	6919	482	445	445
query17	973	595	562	562
query18	1789	292	294	292
query19	203	149	146	146
query20	92	85	89	85
query21	206	101	97	97
query22	4224	4122	3939	3939
query23	33646	33697	33328	33328
query24	9288	3162	3097	3097
query25	622	411	426	411
query26	717	166	181	166
query27	2154	292	288	288
query28	5813	1995	1986	1986
query29	878	442	416	416
query30	239	157	158	157
query31	959	758	770	758
query32	105	56	60	56
query33	677	311	337	311
query34	929	498	505	498
query35	858	761	760	760
query36	1060	891	904	891
query37	138	78	82	78
query38	2958	2912	2803	2803
query39	872	798	845	798
query40	210	111	112	111
query41	47	43	44	43
query42	120	100	99	99
query43	474	417	422	417
query44	1177	715	731	715
query45	212	179	181	179
query46	1104	804	798	798
query47	1778	1720	1719	1719
query48	364	285	289	285
query49	839	439	426	426
query50	897	440	436	436
query51	6807	6667	6622	6622
query52	109	90	88	88
query53	253	182	175	175
query54	601	444	444	444
query55	75	73	77	73
query56	294	244	246	244
query57	1122	1045	1056	1045
query58	254	264	256	256
query59	2440	2472	2376	2376
query60	319	262	264	262
query61	95	104	96	96
query62	864	660	649	649
query63	210	210	180	180
query64	4177	1905	1861	1861
query65	3166	3098	3098	3098
query66	828	348	321	321
query67	15111	14678	15214	14678
query68	4416	569	585	569
query69	641	370	291	291
query70	1126	1039	1055	1039
query71	413	282	275	275
query72	7170	2669	2519	2519
query73	781	333	331	331
query74	6005	5627	5669	5627
query75	3379	2738	2755	2738
query76	2290	1197	1282	1197
query77	573	306	303	303
query78	9403	8887	8863	8863
query79	2398	530	540	530
query80	981	511	503	503
query81	555	227	229	227
query82	847	131	129	129
query83	256	171	169	169
query84	260	80	78	78
query85	1048	368	337	337
query86	451	313	323	313
query87	3251	3120	3124	3120
query88	3077	2384	2398	2384
query89	375	304	296	296
query90	1662	192	191	191
query91	127	103	106	103
query92	56	51	50	50
query93	1896	623	613	613
query94	776	299	302	299
query95	379	264	259	259
query96	605	281	285	281
query97	3243	3092	3097	3092
query98	213	197	187	187
query99	1597	1315	1317	1315
Total cold run time: 254352 ms
Total hot run time: 168954 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.99 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c8a5823a36b9eae92413d97dea1fe04d7875e9f8, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.04
query3	0.22	0.05	0.05
query4	1.69	0.06	0.06
query5	0.49	0.47	0.48
query6	1.13	0.72	0.71
query7	0.02	0.01	0.02
query8	0.05	0.04	0.05
query9	0.59	0.53	0.52
query10	0.56	0.57	0.56
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.62	0.61	0.59
query14	0.77	0.80	0.80
query15	0.91	0.87	0.87
query16	0.35	0.36	0.35
query17	1.02	0.99	0.99
query18	0.21	0.21	0.21
query19	1.86	1.70	1.70
query20	0.01	0.01	0.01
query21	15.42	0.80	0.66
query22	3.90	7.26	1.35
query23	17.90	1.35	1.26
query24	2.19	0.23	0.21
query25	0.18	0.09	0.08
query26	0.31	0.21	0.21
query27	0.46	0.23	0.23
query28	13.19	1.02	0.98
query29	12.93	3.32	3.28
query30	0.25	0.06	0.05
query31	2.87	0.42	0.41
query32	3.23	0.48	0.47
query33	2.93	2.96	2.96
query34	15.47	4.25	4.25
query35	4.31	4.29	4.32
query36	0.69	0.49	0.48
query37	0.19	0.17	0.16
query38	0.15	0.14	0.14
query39	0.05	0.03	0.04
query40	0.16	0.12	0.13
query41	0.10	0.05	0.05
query42	0.05	0.05	0.06
query43	0.05	0.04	0.05
Total cold run time: 107.89 s
Total hot run time: 29.99 s

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 1, 2024

clang-tidy review says "All clean, LGTM! 👍"

@dataroaring
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41402 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit de0085805592b41052d6ec95bc2f4864d0d8807c, data reload: false

------ Round 1 ----------------------------------
q1	17613	4106	4106	4106
q2	2015	197	196	196
q3	10464	1356	1340	1340
q4	10144	812	905	812
q5	7658	2989	2945	2945
q6	223	141	139	139
q7	1048	602	609	602
q8	9432	1702	1917	1702
q9	8579	6601	6574	6574
q10	8749	3864	3853	3853
q11	438	250	247	247
q12	451	221	222	221
q13	17772	2927	2945	2927
q14	277	245	243	243
q15	526	491	494	491
q16	536	385	385	385
q17	960	937	892	892
q18	7981	7226	7269	7226
q19	1868	1226	1219	1219
q20	553	325	349	325
q21	5286	4678	4801	4678
q22	355	279	287	279
Total cold run time: 112928 ms
Total hot run time: 41402 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4154	4041	4032	4032
q2	331	221	217	217
q3	2961	2999	3113	2999
q4	2017	2012	1980	1980
q5	5609	5469	5427	5427
q6	227	131	128	128
q7	2126	1788	1868	1788
q8	3324	3353	3316	3316
q9	8593	8666	8789	8666
q10	3943	4093	3899	3899
q11	585	467	466	466
q12	791	584	587	584
q13	16438	3124	3090	3090
q14	312	262	281	262
q15	526	478	499	478
q16	462	435	408	408
q17	1758	1741	1729	1729
q18	8140	7647	7707	7647
q19	1727	1730	1714	1714
q20	2086	1838	1835	1835
q21	5693	5550	5203	5203
q22	517	480	464	464
Total cold run time: 72320 ms
Total hot run time: 56332 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 168959 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit de0085805592b41052d6ec95bc2f4864d0d8807c, data reload: false

query1	920	373	364	364
query2	6455	1762	1741	1741
query3	6689	209	234	209
query4	19689	17412	17233	17233
query5	3654	512	533	512
query6	293	173	173	173
query7	4591	296	300	296
query8	257	206	206	206
query9	8522	2322	2312	2312
query10	449	279	278	278
query11	10550	9957	10028	9957
query12	119	92	90	90
query13	1641	379	377	377
query14	9862	6564	7522	6564
query15	204	167	159	159
query16	6971	466	483	466
query17	955	554	535	535
query18	1855	281	282	281
query19	189	141	138	138
query20	91	87	84	84
query21	205	118	103	103
query22	4215	4032	4061	4032
query23	33808	33833	33242	33242
query24	9518	3202	3066	3066
query25	663	409	417	409
query26	1151	152	156	152
query27	2880	279	278	278
query28	7538	1988	1975	1975
query29	997	452	432	432
query30	248	155	163	155
query31	941	744	773	744
query32	100	57	60	57
query33	696	324	309	309
query34	909	500	510	500
query35	891	762	721	721
query36	1011	898	881	881
query37	169	78	76	76
query38	2899	2760	2767	2760
query39	865	811	812	811
query40	217	111	116	111
query41	44	44	44	44
query42	119	97	97	97
query43	485	450	424	424
query44	1172	730	727	727
query45	211	178	177	177
query46	1082	830	791	791
query47	1811	1747	1742	1742
query48	360	291	291	291
query49	863	433	423	423
query50	893	434	422	422
query51	6799	6724	6757	6724
query52	102	90	91	90
query53	249	189	208	189
query54	599	453	451	451
query55	75	72	74	72
query56	272	248	274	248
query57	1111	1047	1031	1031
query58	266	265	266	265
query59	2669	2479	2411	2411
query60	304	272	269	269
query61	102	93	93	93
query62	892	669	644	644
query63	225	177	181	177
query64	4435	1898	1870	1870
query65	3180	3071	3106	3071
query66	834	381	326	326
query67	15480	14926	14966	14926
query68	6000	566	587	566
query69	762	375	308	308
query70	1136	1056	1089	1056
query71	448	280	264	264
query72	7553	2692	2547	2547
query73	921	333	330	330
query74	6073	5679	5583	5583
query75	3660	2737	2710	2710
query76	3615	1274	1276	1274
query77	684	307	300	300
query78	9306	8870	8874	8870
query79	2198	535	529	529
query80	921	497	530	497
query81	535	222	228	222
query82	766	128	128	128
query83	236	171	167	167
query84	255	79	78	78
query85	1300	305	302	302
query86	464	288	315	288
query87	3304	3129	3105	3105
query88	3588	2485	2469	2469
query89	486	299	289	289
query90	1762	198	189	189
query91	123	100	104	100
query92	58	49	48	48
query93	2049	608	616	608
query94	746	280	291	280
query95	379	261	266	261
query96	603	293	289	289
query97	3257	3069	3067	3067
query98	216	197	192	192
query99	1599	1259	1279	1259
Total cold run time: 263514 ms
Total hot run time: 168959 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.85 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit de0085805592b41052d6ec95bc2f4864d0d8807c, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.06
query5	0.51	0.49	0.49
query6	1.14	0.71	0.71
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.56	0.50	0.50
query10	0.56	0.58	0.57
query11	0.15	0.12	0.12
query12	0.14	0.13	0.12
query13	0.60	0.61	0.60
query14	0.77	0.80	0.79
query15	0.91	0.87	0.87
query16	0.35	0.35	0.36
query17	1.01	1.04	0.98
query18	0.21	0.21	0.21
query19	1.83	1.81	1.71
query20	0.01	0.01	0.00
query21	15.40	0.74	0.65
query22	4.49	7.96	1.13
query23	17.84	1.45	1.37
query24	2.27	0.22	0.22
query25	0.18	0.08	0.09
query26	0.32	0.21	0.22
query27	0.47	0.23	0.23
query28	13.15	0.99	0.96
query29	12.51	3.34	3.38
query30	0.26	0.06	0.05
query31	2.86	0.40	0.40
query32	3.27	0.50	0.48
query33	2.92	2.89	2.96
query34	15.44	4.27	4.26
query35	4.31	4.26	4.28
query36	0.69	0.47	0.49
query37	0.19	0.16	0.17
query38	0.17	0.15	0.15
query39	0.04	0.04	0.03
query40	0.16	0.13	0.13
query41	0.10	0.05	0.04
query42	0.05	0.05	0.05
query43	0.04	0.04	0.03
Total cold run time: 107.96 s
Total hot run time: 29.85 s

dataroaring
dataroaring approved these changes Aug 6, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 6, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 6, 2024
TangSiyang2001
TangSiyang2001 approved these changes Aug 6, 2024
View reviewed changes
Copy link
Collaborator

@TangSiyang2001 TangSiyang2001 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gavinchou gavinchou merged commit 01b9e62 into apache:master Aug 6, 2024
28 of 30 checks passed
dataroaring pushed a commit that referenced this pull request Aug 7, 2024
@gavinchou @dataroaring
[improvement](file cache) Try to read from remote storage when openin…
04e559b 
…g segment with CachedRemoteFileReader (#38645)

We may encounter the error "Bad segment" file in some rare cases where
the file cache may not hold the correct segment files. We should read
the remote original segment files to increase robustness.
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@freemandealer freemandealer freemandealer left review comments

@TangSiyang2001 TangSiyang2001 TangSiyang2001 approved these changes

@dataroaring dataroaring dataroaring approved these changes

@platoneko platoneko platoneko left review comments

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.1-merged doing p0_w reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@gavinchou @doris-robot @dataroaring @freemandealer @platoneko @TangSiyang2001