Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](group commit) Add retry when delete bitmap lock expired on group commit #37600

Merged
merged 2 commits into from
Jul 10, 2024
Merged

[fix](group commit) Add retry when delete bitmap lock expired on group commit #37600

merged 2 commits into from
Jul 10, 2024

Conversation

hust-hhb
Copy link
Contributor

group commit writing mow table may encouter delete bitmap lock expired, add retry mechanism to avoid group commit fail.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@hust-hhb hust-hhb changed the title (selectdb-cloud) Add retry when delete bitmap lock expired on group commit [fix](group commit) Add retry when delete bitmap lock expired on group commit Jul 10, 2024
@hust-hhb hust-hhb force-pushed the add-retry-group-commit branch from 952f0c9 to 9ad5849 Compare July 10, 2024 07:56
@hust-hhb
Copy link
Contributor Author

run buildall

@hust-hhb
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39436 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c5bf8f9f30b433c33aac4a1ebdc05a55d8ec5edf, data reload: false

------ Round 1 ----------------------------------
q1	17632	4259	4243	4243
q2	2013	195	189	189
q3	10446	1182	1085	1085
q4	10182	881	779	779
q5	7539	2623	2606	2606
q6	227	139	141	139
q7	950	597	608	597
q8	9216	2061	2102	2061
q9	8964	6487	6486	6486
q10	8941	3648	3720	3648
q11	470	230	225	225
q12	443	227	223	223
q13	17755	3001	3000	3000
q14	264	229	217	217
q15	516	481	470	470
q16	480	367	367	367
q17	981	690	700	690
q18	8038	7362	7316	7316
q19	8164	1434	1330	1330
q20	679	315	314	314
q21	4877	3116	3204	3116
q22	397	338	335	335
Total cold run time: 119174 ms
Total hot run time: 39436 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4452	4212	4287	4212
q2	381	274	259	259
q3	2957	2920	2918	2918
q4	1968	1673	1740	1673
q5	5533	5491	5419	5419
q6	223	141	134	134
q7	2244	1874	1830	1830
q8	3242	3427	3477	3427
q9	8671	8738	8740	8738
q10	4080	3661	3833	3661
q11	596	494	512	494
q12	804	664	615	615
q13	16110	3183	3204	3183
q14	294	296	297	296
q15	518	503	485	485
q16	507	449	460	449
q17	1811	1514	1502	1502
q18	7989	7743	7826	7743
q19	1853	1594	1499	1499
q20	2584	1890	1869	1869
q21	5136	4881	4862	4862
q22	645	576	576	576
Total cold run time: 72598 ms
Total hot run time: 55844 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173826 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c5bf8f9f30b433c33aac4a1ebdc05a55d8ec5edf, data reload: false

query1	918	381	367	367
query2	6452	2290	2264	2264
query3	6640	208	217	208
query4	25736	17425	17652	17425
query5	3667	494	479	479
query6	277	174	173	173
query7	4573	289	284	284
query8	301	302	286	286
query9	8513	2377	2351	2351
query10	434	292	302	292
query11	10729	9918	10116	9918
query12	125	85	78	78
query13	1642	372	368	368
query14	9950	8383	7522	7522
query15	223	183	195	183
query16	7657	317	307	307
query17	1604	560	518	518
query18	1794	284	271	271
query19	204	157	163	157
query20	93	87	85	85
query21	216	133	123	123
query22	4266	4047	3935	3935
query23	34073	33625	33773	33625
query24	11302	2854	2923	2854
query25	610	410	389	389
query26	1169	152	144	144
query27	2808	272	279	272
query28	7274	2145	2123	2123
query29	878	647	639	639
query30	259	157	157	157
query31	939	761	763	761
query32	96	55	53	53
query33	750	291	299	291
query34	1033	501	495	495
query35	692	589	622	589
query36	1127	986	973	973
query37	140	86	86	86
query38	2941	2851	2881	2851
query39	908	826	831	826
query40	203	125	118	118
query41	52	50	52	50
query42	117	104	97	97
query43	578	555	558	555
query44	1236	723	750	723
query45	202	164	158	158
query46	1069	726	713	713
query47	1851	1747	1768	1747
query48	374	307	300	300
query49	862	425	414	414
query50	771	391	391	391
query51	6941	6706	6797	6706
query52	106	96	89	89
query53	361	279	279	279
query54	873	452	453	452
query55	107	75	74	74
query56	282	266	262	262
query57	1095	1031	1052	1031
query58	253	240	249	240
query59	3474	3349	3224	3224
query60	298	276	267	267
query61	111	92	93	92
query62	784	648	656	648
query63	319	289	290	289
query64	9452	2195	1642	1642
query65	3170	3106	3098	3098
query66	755	329	327	327
query67	15424	15150	14961	14961
query68	6080	542	547	542
query69	689	435	365	365
query70	1183	1140	1107	1107
query71	464	287	277	277
query72	7488	5153	5407	5153
query73	770	321	321	321
query74	6006	5427	5498	5427
query75	3569	2676	2679	2676
query76	3208	936	952	936
query77	659	305	301	301
query78	9464	8851	10585	8851
query79	2528	513	505	505
query80	1345	472	467	467
query81	567	221	215	215
query82	537	130	134	130
query83	194	169	165	165
query84	275	88	87	87
query85	1430	315	301	301
query86	454	311	338	311
query87	3235	3080	3058	3058
query88	3801	2463	2444	2444
query89	466	388	389	388
query90	1760	187	184	184
query91	126	101	103	101
query92	56	50	48	48
query93	1709	507	503	503
query94	1078	212	205	205
query95	412	320	309	309
query96	588	281	269	269
query97	3190	3064	3080	3064
query98	230	202	197	197
query99	1663	1283	1250	1250
Total cold run time: 279357 ms
Total hot run time: 173826 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.19 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c5bf8f9f30b433c33aac4a1ebdc05a55d8ec5edf, data reload: false

query1	0.04	0.04	0.02
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.69	0.06	0.07
query5	0.50	0.47	0.49
query6	1.13	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.49	0.48
query10	0.55	0.54	0.54
query11	0.16	0.11	0.11
query12	0.15	0.13	0.12
query13	0.60	0.58	0.59
query14	0.75	0.79	0.77
query15	0.84	0.82	0.82
query16	0.38	0.36	0.36
query17	1.02	1.03	1.04
query18	0.23	0.21	0.22
query19	1.76	1.73	1.72
query20	0.02	0.01	0.01
query21	15.41	0.76	0.65
query22	4.00	7.31	1.55
query23	18.68	1.39	1.24
query24	2.12	0.22	0.22
query25	0.16	0.09	0.09
query26	0.29	0.21	0.21
query27	0.46	0.23	0.23
query28	13.30	1.02	1.00
query29	12.65	3.31	3.34
query30	0.25	0.06	0.05
query31	2.89	0.39	0.39
query32	3.27	0.49	0.48
query33	2.91	2.88	2.88
query34	17.03	4.41	4.30
query35	4.41	4.40	4.41
query36	0.65	0.47	0.49
query37	0.19	0.15	0.16
query38	0.16	0.14	0.14
query39	0.04	0.03	0.03
query40	0.14	0.13	0.13
query41	0.09	0.04	0.05
query42	0.06	0.04	0.05
query43	0.04	0.03	0.04
Total cold run time: 109.95 s
Total hot run time: 30.19 s

dataroaring
dataroaring approved these changes Jul 10, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 10, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit a793988 into apache:master Jul 10, 2024
24 of 29 checks passed
seawinde pushed a commit to seawinde/doris that referenced this pull request Jul 17, 2024
@hust-hhb @seawinde
[fix](group commit) Add retry when delete bitmap lock expired on grou…
04be89c 
…p commit (apache#37600)

group commit writing mow table may encouter delete bitmap lock expired,
add retry mechanism to avoid group commit fail.
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@hust-hhb @dataroaring
[fix](group commit) Add retry when delete bitmap lock expired on grou…
107d9e3 
…p commit (#37600)

group commit writing mow table may encouter delete bitmap lock expired,
add retry mechanism to avoid group commit fail.
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mymeiyi mymeiyi mymeiyi approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.1-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@hust-hhb @doris-robot @mymeiyi @dataroaring