Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[branch-2.0](load) fix commit txn timeout when loading to table with many tablet (#40031) #40140

Merged
merged 1 commit into from
Sep 11, 2024
Merged

[branch-2.0](load) fix commit txn timeout when loading to table with many tablet (#40031) #40140

merged 1 commit into from
Sep 11, 2024
+43 −24

Conversation

zclllyybb
Copy link
Contributor

pick #40031

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zclllyybb
Copy link
Contributor Author

run buildall

@github-actions github-actions bot added the area/load Issues or PRs related to all kinds of load label Aug 29, 2024
@zclllyybb
[Fix](load) fix commit txn timeout when loading to table with many ta…
260eb65 
…blet (apache#40031)

fix commit txn timeout when loading to table with many tablet
@zclllyybb
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49998 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 260eb6514975810fee5545aad5081362f99ee9bc, data reload: false

------ Round 1 ----------------------------------
q1	17915	4345	4367	4345
q2	2072	157	148	148
q3	10536	1916	1899	1899
q4	10281	1235	1296	1235
q5	8939	3897	3960	3897
q6	235	121	125	121
q7	2038	1600	1609	1600
q8	9304	2734	2712	2712
q9	10885	10537	10552	10537
q10	8636	3461	3454	3454
q11	428	242	253	242
q12	460	303	309	303
q13	18367	3979	4024	3979
q14	352	328	326	326
q15	523	459	455	455
q16	680	583	567	567
q17	1119	956	971	956
q18	7277	6930	6871	6871
q19	1712	1571	1546	1546
q20	547	319	316	316
q21	4433	4164	4092	4092
q22	498	397	399	397
Total cold run time: 117237 ms
Total hot run time: 49998 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4306	4254	4326	4254
q2	323	235	229	229
q3	4179	4133	4144	4133
q4	2735	2732	2753	2732
q5	7249	7160	7076	7076
q6	236	116	120	116
q7	3256	2865	2856	2856
q8	4353	4462	4492	4462
q9	17327	16992	17166	16992
q10	4226	4258	4272	4258
q11	743	684	684	684
q12	1042	826	861	826
q13	7040	3738	3735	3735
q14	444	428	421	421
q15	517	458	467	458
q16	723	685	663	663
q17	3833	3815	3868	3815
q18	8831	8660	8682	8660
q19	1694	1699	1628	1628
q20	2345	2127	2149	2127
q21	8500	8499	8454	8454
q22	1015	907	957	907
Total cold run time: 84917 ms
Total hot run time: 79486 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 203007 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 260eb6514975810fee5545aad5081362f99ee9bc, data reload: false

query1	937	387	437	387
query2	6545	2200	2252	2200
query3	6917	208	202	202
query4	20257	18035	17915	17915
query5	19736	6507	6434	6434
query6	281	218	248	218
query7	4156	309	318	309
query8	261	243	294	243
query9	3134	2687	2613	2613
query10	412	324	285	285
query11	11305	10720	10708	10708
query12	132	78	73	73
query13	5598	664	663	663
query14	17751	13077	13089	13077
query15	374	218	227	218
query16	6484	294	267	267
query17	1725	1469	879	879
query18	2316	433	423	423
query19	219	149	156	149
query20	80	80	80	80
query21	188	97	93	93
query22	5294	5013	5056	5013
query23	32396	31721	31825	31721
query24	6972	6590	6497	6497
query25	520	425	424	424
query26	535	165	162	162
query27	1863	292	299	292
query28	6116	2289	2255	2255
query29	2868	2805	2877	2805
query30	236	166	165	165
query31	878	728	752	728
query32	67	63	60	60
query33	395	262	256	256
query34	830	477	470	470
query35	1156	935	900	900
query36	1340	1372	1083	1083
query37	92	62	64	62
query38	3024	2909	2925	2909
query39	1377	1319	1331	1319
query40	207	98	92	92
query41	39	37	37	37
query42	96	88	83	83
query43	651	626	677	626
query44	1131	718	715	715
query45	238	233	227	227
query46	1242	975	984	975
query47	1942	1790	1944	1790
query48	967	665	670	665
query49	620	378	372	372
query50	875	644	630	630
query51	4780	4683	4648	4648
query52	94	82	75	75
query53	450	322	322	322
query54	2618	2460	2468	2460
query55	86	77	87	77
query56	245	247	214	214
query57	1236	1099	1204	1099
query58	218	202	208	202
query59	3540	3296	3244	3244
query60	210	206	212	206
query61	98	102	96	96
query62	876	474	518	474
query63	477	341	347	341
query64	2592	1540	1507	1507
query65	3618	3555	3546	3546
query66	806	381	380	380
query67	15494	16518	16679	16518
query68	9753	672	657	657
query69	580	337	374	337
query70	1723	1426	1360	1360
query71	411	313	314	313
query72	6574	3468	3484	3468
query73	727	321	330	321
query74	6224	5843	5935	5843
query75	5364	3751	3609	3609
query76	6266	1138	1165	1138
query77	1059	261	258	258
query78	12559	11777	11935	11777
query79	7904	658	643	643
query80	1345	399	411	399
query81	492	235	238	235
query82	1640	97	100	97
query83	177	132	132	132
query84	258	71	73	71
query85	894	326	324	324
query86	323	282	289	282
query87	3229	3086	3003	3003
query88	4804	2310	2306	2306
query89	381	302	291	291
query90	1973	209	196	196
query91	162	133	126	126
query92	63	54	52	52
query93	6538	564	585	564
query94	706	218	207	207
query95	1126	1067	1069	1067
query96	645	339	329	329
query97	6457	6440	6434	6434
query98	188	170	169	169
query99	2951	821	982	821
Total cold run time: 313709 ms
Total hot run time: 203007 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.91 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 260eb6514975810fee5545aad5081362f99ee9bc, data reload: false

query1	0.03	0.03	0.02
query2	0.06	0.02	0.02
query3	0.24	0.04	0.04
query4	1.80	0.08	0.06
query5	0.54	0.53	0.53
query6	1.38	0.63	0.66
query7	0.02	0.00	0.01
query8	0.03	0.02	0.02
query9	0.52	0.49	0.49
query10	0.53	0.54	0.52
query11	0.12	0.08	0.09
query12	0.11	0.10	0.09
query13	0.62	0.62	0.62
query14	0.78	0.79	0.80
query15	0.79	0.78	0.78
query16	0.39	0.37	0.36
query17	0.96	1.04	1.03
query18	0.21	0.27	0.25
query19	1.90	1.87	1.84
query20	0.02	0.01	0.01
query21	15.49	0.55	0.55
query22	2.41	2.31	1.52
query23	17.29	1.09	0.94
query24	7.10	0.86	0.84
query25	0.42	0.10	0.06
query26	0.72	0.15	0.15
query27	0.05	0.05	0.03
query28	6.32	0.75	0.78
query29	12.68	2.41	2.31
query30	0.59	0.53	0.54
query31	2.82	0.40	0.39
query32	3.34	0.51	0.51
query33	3.18	3.16	3.17
query34	15.25	4.91	4.84
query35	4.93	4.89	4.89
query36	1.04	1.02	1.02
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.14
query41	0.06	0.02	0.01
query42	0.02	0.02	0.01
query43	0.02	0.01	0.02
Total cold run time: 105.06 s
Total hot run time: 30.91 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 260eb6514975810fee5545aad5081362f99ee9bc with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.5 seconds inserted 10000000 Rows, about 465K ops/s

@xiaokang xiaokang merged commit 1617866 into apache:branch-2.0 Sep 11, 2024
21 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
area/load Issues or PRs related to all kinds of load
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@zclllyybb @doris-robot @xiaokang