Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](group commit) fix group commit insert rpc may stuck #39391

Merged
merged 1 commit into from
Aug 15, 2024

Conversation

mymeiyi
Copy link
Contributor

@mymeiyi mymeiyi commented Aug 15, 2024

  1. when exec_plan_fragment failed and PipelineFragmentContext is not constructed, the group_commit_insert rpc will stuck
  2. the LoadBlockQueue.add_block.back_pressure_time_out debug point is not work

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@mymeiyi mymeiyi changed the title [fix](group commit) fix group commit insert rpc [fix](group commit) fix group commit insert rpc may stuck Aug 15, 2024
@mymeiyi
Copy link
Contributor Author

mymeiyi commented Aug 15, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37707 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 979e5bf001674542ca482e15f221b111826351a1, data reload: false

------ Round 1 ----------------------------------
q1	17625	4320	4284	4284
q2	2018	177	175	175
q3	10531	1249	1122	1122
q4	10706	700	785	700
q5	8267	2822	2820	2820
q6	229	144	142	142
q7	979	611	610	610
q8	9394	2057	2059	2057
q9	7392	6586	6514	6514
q10	7098	2220	2174	2174
q11	452	241	239	239
q12	393	224	230	224
q13	17808	2986	2964	2964
q14	290	236	242	236
q15	523	480	476	476
q16	514	399	381	381
q17	973	637	747	637
q18	7511	6864	6868	6864
q19	5388	1036	1074	1036
q20	703	312	325	312
q21	3798	2986	2749	2749
q22	1121	1014	991	991
Total cold run time: 113713 ms
Total hot run time: 37707 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4436	4293	4257	4257
q2	390	257	256	256
q3	2878	2633	2628	2628
q4	1864	1575	1605	1575
q5	5316	5380	5344	5344
q6	221	129	130	129
q7	2055	1694	1682	1682
q8	3171	3369	3314	3314
q9	8396	8400	8348	8348
q10	3378	3146	3160	3146
q11	592	491	480	480
q12	780	663	605	605
q13	17333	3018	2954	2954
q14	302	285	265	265
q15	513	481	489	481
q16	486	417	406	406
q17	1788	1462	1477	1462
q18	7720	7594	7470	7470
q19	1663	1494	1539	1494
q20	1976	1779	1807	1779
q21	5275	5158	5153	5153
q22	1099	1027	1013	1013
Total cold run time: 71632 ms
Total hot run time: 54241 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184691 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 979e5bf001674542ca482e15f221b111826351a1, data reload: false

query1	920	405	363	363
query2	6452	1909	1837	1837
query3	6650	223	213	213
query4	28065	23265	23060	23060
query5	4203	490	496	490
query6	262	161	160	160
query7	4600	303	294	294
query8	245	200	199	199
query9	8674	2449	2441	2441
query10	429	268	254	254
query11	17723	14996	15007	14996
query12	153	99	104	99
query13	1635	373	366	366
query14	9076	6896	6742	6742
query15	213	164	169	164
query16	7706	490	461	461
query17	1590	579	550	550
query18	1923	286	286	286
query19	195	148	143	143
query20	113	106	105	105
query21	209	100	98	98
query22	4429	4081	4011	4011
query23	33875	33363	33112	33112
query24	12366	2904	2898	2898
query25	684	392	394	392
query26	1804	158	159	158
query27	2892	269	275	269
query28	7677	2063	2054	2054
query29	1150	419	414	414
query30	307	152	151	151
query31	966	755	774	755
query32	99	54	55	54
query33	741	290	282	282
query34	953	459	461	459
query35	867	728	731	728
query36	1063	926	911	911
query37	273	80	83	80
query38	3924	3764	3778	3764
query39	1429	1359	1395	1359
query40	270	119	114	114
query41	50	46	50	46
query42	117	98	101	98
query43	508	469	455	455
query44	1213	738	740	738
query45	196	166	169	166
query46	1109	724	757	724
query47	1871	1807	1781	1781
query48	357	302	303	302
query49	1224	443	440	440
query50	803	403	408	403
query51	6867	6684	6670	6670
query52	106	93	94	93
query53	254	182	187	182
query54	971	452	442	442
query55	74	74	76	74
query56	274	259	257	257
query57	1179	1073	1079	1073
query58	234	224	232	224
query59	3018	2924	2713	2713
query60	296	280	271	271
query61	118	224	95	95
query62	826	649	655	649
query63	213	186	178	178
query64	6258	2270	1687	1687
query65	3215	3142	3175	3142
query66	1382	326	325	325
query67	15536	14755	14895	14755
query68	9039	553	556	553
query69	714	381	297	297
query70	1335	1140	1089	1089
query71	553	278	268	268
query72	7613	2215	2018	2018
query73	2191	321	323	321
query74	9433	8747	8755	8747
query75	5368	2682	2680	2680
query76	5438	1080	1011	1011
query77	763	313	305	305
query78	9810	9095	9192	9095
query79	10163	534	530	530
query80	1152	488	482	482
query81	577	224	228	224
query82	593	132	130	130
query83	324	141	142	141
query84	273	75	73	73
query85	1068	268	260	260
query86	320	296	292	292
query87	4350	4221	4169	4169
query88	4479	2301	2307	2301
query89	513	298	287	287
query90	1860	235	183	183
query91	118	95	92	92
query92	58	51	49	49
query93	7012	539	524	524
query94	716	294	279	279
query95	346	253	257	253
query96	620	269	265	265
query97	3191	3049	3079	3049
query98	229	201	199	199
query99	1757	1256	1265	1256
Total cold run time: 317050 ms
Total hot run time: 184691 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.17 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 979e5bf001674542ca482e15f221b111826351a1, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.66	0.09	0.09
query5	0.50	0.49	0.50
query6	1.14	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.54	0.49	0.50
query10	0.53	0.53	0.53
query11	0.15	0.11	0.12
query12	0.16	0.12	0.12
query13	0.62	0.60	0.58
query14	0.77	0.77	0.79
query15	0.84	0.81	0.82
query16	0.38	0.38	0.37
query17	0.99	1.05	1.02
query18	0.23	0.21	0.21
query19	1.94	1.71	1.75
query20	0.01	0.01	0.00
query21	15.41	0.74	0.64
query22	3.61	7.30	2.24
query23	18.25	1.39	1.35
query24	2.09	0.23	0.22
query25	0.15	0.09	0.08
query26	0.31	0.22	0.21
query27	0.46	0.22	0.24
query28	13.32	1.03	0.99
query29	12.58	3.41	3.37
query30	0.24	0.04	0.05
query31	2.86	0.40	0.39
query32	3.26	0.49	0.48
query33	2.96	2.99	2.95
query34	17.03	4.30	4.39
query35	4.45	4.44	4.42
query36	0.67	0.47	0.47
query37	0.19	0.17	0.17
query38	0.15	0.15	0.15
query39	0.04	0.03	0.04
query40	0.16	0.12	0.13
query41	0.10	0.05	0.04
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.28 s
Total hot run time: 31.17 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 15, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit 90d92e9 into apache:master Aug 15, 2024
27 of 31 checks passed
mymeiyi added a commit to mymeiyi/doris that referenced this pull request Aug 16, 2024
dataroaring pushed a commit that referenced this pull request Aug 17, 2024
1. when exec_plan_fragment failed and PipelineFragmentContext is not
constructed, the group_commit_insert rpc will stuck
2. the `LoadBlockQueue.add_block.back_pressure_time_out` debug point is
not work
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.2-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants