Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fix](load) Lower lock granularity to reduce loading CPU pressure #40134

Merged
merged 1 commit into from
Aug 29, 2024
Merged

[Fix](load) Lower lock granularity to reduce loading CPU pressure #40134

merged 1 commit into from
Aug 29, 2024

Conversation

zclllyybb
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

before when writing block, we have a lock to protect the whole process. now reduced it

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zclllyybb
Copy link
Contributor Author

run buildall

@zclllyybb zclllyybb mentioned this pull request Aug 29, 2024
Merged
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 38083 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2186201fe54cec1db910507dec302a4ad0c20dd5, data reload: false

------ Round 1 ----------------------------------
q1	17627	4883	4265	4265
q2	2029	180	178	178
q3	11717	952	1214	952
q4	10528	725	754	725
q5	7740	2821	2806	2806
q6	220	136	136	136
q7	949	618	585	585
q8	9326	2023	2090	2023
q9	6996	6530	6529	6529
q10	7006	2200	2232	2200
q11	460	245	240	240
q12	394	229	223	223
q13	18157	3082	3050	3050
q14	277	233	247	233
q15	510	481	482	481
q16	590	513	510	510
q17	973	719	809	719
q18	7312	6938	6884	6884
q19	1395	1008	1027	1008
q20	694	331	325	325
q21	3916	3091	2977	2977
q22	1134	1034	1043	1034
Total cold run time: 109950 ms
Total hot run time: 38083 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4352	4303	4225	4225
q2	379	280	270	270
q3	2904	2661	2666	2661
q4	1954	1681	1642	1642
q5	5614	5651	5720	5651
q6	232	138	140	138
q7	2210	1796	1822	1796
q8	3313	3424	3425	3424
q9	8840	8813	8819	8813
q10	3564	3359	3353	3353
q11	606	521	526	521
q12	839	626	670	626
q13	14987	3160	3175	3160
q14	321	304	295	295
q15	536	490	492	490
q16	607	571	582	571
q17	1843	1555	1536	1536
q18	8112	7677	7863	7677
q19	1741	1602	1497	1497
q20	2167	1907	1896	1896
q21	5743	5409	5410	5409
q22	1203	1053	1058	1053
Total cold run time: 72067 ms
Total hot run time: 56704 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193274 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2186201fe54cec1db910507dec302a4ad0c20dd5, data reload: false

query1	1257	875	879	875
query2	6327	1930	1864	1864
query3	10599	4074	3814	3814
query4	59771	24330	23371	23371
query5	5463	502	506	502
query6	411	163	169	163
query7	5763	300	302	300
query8	280	202	208	202
query9	9070	2496	2483	2483
query10	490	262	262	262
query11	18155	15059	15243	15059
query12	145	103	106	103
query13	1544	399	372	372
query14	11206	8583	7363	7363
query15	259	174	188	174
query16	7333	452	496	452
query17	1093	584	581	581
query18	1577	279	301	279
query19	285	145	149	145
query20	131	112	110	110
query21	209	101	101	101
query22	4678	4413	4393	4393
query23	34084	33557	33552	33552
query24	5985	2912	2862	2862
query25	550	399	408	399
query26	684	159	159	159
query27	1782	284	282	282
query28	3903	2114	2096	2096
query29	677	429	418	418
query30	237	158	161	158
query31	936	757	801	757
query32	78	55	59	55
query33	424	290	283	283
query34	864	488	485	485
query35	869	723	747	723
query36	1053	924	895	895
query37	147	97	94	94
query38	3995	3903	3892	3892
query39	1465	1386	1473	1386
query40	196	121	116	116
query41	49	47	46	46
query42	114	99	99	99
query43	503	452	454	452
query44	1085	750	776	750
query45	203	168	172	168
query46	1109	760	755	755
query47	1885	1828	1836	1828
query48	379	300	293	293
query49	768	445	438	438
query50	833	424	414	414
query51	7148	7087	7011	7011
query52	98	91	89	89
query53	257	182	178	178
query54	561	455	459	455
query55	84	76	77	76
query56	285	264	264	264
query57	1183	1062	1084	1062
query58	237	235	239	235
query59	3051	2763	2841	2763
query60	299	275	281	275
query61	123	119	116	116
query62	740	634	653	634
query63	225	186	188	186
query64	2875	745	731	731
query65	3194	3160	3134	3134
query66	617	348	457	348
query67	15516	15237	15197	15197
query68	4395	577	567	567
query69	425	276	277	276
query70	1189	1099	1076	1076
query71	340	273	273	273
query72	6521	4005	3978	3978
query73	749	334	332	332
query74	9169	8875	8904	8875
query75	3336	2650	2707	2650
query76	1835	1084	998	998
query77	552	329	329	329
query78	9870	9063	9071	9063
query79	1162	539	525	525
query80	1029	531	509	509
query81	558	232	236	232
query82	287	143	147	143
query83	251	151	151	151
query84	259	74	77	74
query85	947	305	274	274
query86	367	269	286	269
query87	4573	4422	4292	4292
query88	3305	2333	2438	2333
query89	383	289	286	286
query90	1944	193	191	191
query91	124	101	100	100
query92	64	54	52	52
query93	1984	558	547	547
query94	784	298	285	285
query95	353	274	263	263
query96	610	270	271	270
query97	3268	3041	3084	3041
query98	212	202	205	202
query99	1575	1241	1281	1241
Total cold run time: 309816 ms
Total hot run time: 193274 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.08 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2186201fe54cec1db910507dec302a4ad0c20dd5, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.03
query3	0.22	0.05	0.05
query4	1.69	0.06	0.07
query5	0.49	0.49	0.50
query6	1.12	0.72	0.73
query7	0.02	0.01	0.02
query8	0.06	0.04	0.04
query9	0.55	0.48	0.50
query10	0.55	0.55	0.53
query11	0.17	0.12	0.11
query12	0.16	0.12	0.12
query13	0.60	0.59	0.59
query14	2.07	2.10	2.07
query15	0.84	0.81	0.83
query16	0.38	0.37	0.35
query17	0.99	0.97	1.03
query18	0.21	0.20	0.22
query19	1.79	1.70	1.72
query20	0.01	0.01	0.01
query21	15.39	0.67	0.67
query22	4.09	7.49	2.27
query23	18.28	1.35	1.17
query24	2.17	0.22	0.22
query25	0.16	0.08	0.07
query26	0.27	0.18	0.17
query27	0.07	0.07	0.08
query28	13.24	1.03	1.00
query29	12.67	3.35	3.34
query30	0.25	0.06	0.05
query31	2.88	0.40	0.39
query32	3.26	0.48	0.47
query33	2.92	2.98	3.06
query34	16.92	4.40	4.33
query35	4.41	4.43	4.43
query36	0.66	0.47	0.47
query37	0.19	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.03	0.04
query40	0.15	0.12	0.13
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.42 s
Total hot run time: 32.08 s

yiguolei pushed a commit that referenced this pull request Aug 29, 2024
@zclllyybb
[branch-2.1](load) Lower lock granularity to reduce loading CPU press…
090474a 
…ure (#40135)

pick #40134
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 29, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@yiguolei yiguolei merged commit eae7cf0 into apache:master Aug 29, 2024
25 of 29 checks passed
@zclllyybb zclllyybb deleted the fix_lock branch September 2, 2024 09:45
dataroaring pushed a commit that referenced this pull request Sep 3, 2024
@zclllyybb @dataroaring
[Fix](load) Lower lock granularity to reduce loading CPU pressure (#4…
6d4eb9b 
…0134)

## Proposed changes

Issue Number: close #xxx

before when writing block, we have a lock to protect the whole process.
now reduced it
@gavinchou gavinchou mentioned this pull request Oct 13, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yiguolei yiguolei yiguolei approved these changes

@jacktengg jacktengg jacktengg approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.2-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@zclllyybb @doris-robot @yiguolei @jacktengg @dataroaring