Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](routine load) fix invalid EOF tasks scheduling delay #44949

Merged
merged 1 commit into from
Dec 6, 2024

Conversation

sollhui
Copy link
Contributor

@sollhui sollhui commented Dec 3, 2024

What problem does this PR solve?

Related PR: #39975

Problem Summary:

In the above PR, we introduce delay schedule EOF tasks to avoid too many small transactions, but it is invalid, the pr fix the issue.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@sollhui
Copy link
Contributor Author

sollhui commented Dec 3, 2024

run buildall

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

github-actions bot commented Dec 3, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Dec 3, 2024
Copy link
Contributor

github-actions bot commented Dec 3, 2024

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 40142 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 83af8d1a8567ef542916a9c499b5a33f6b54f6d5, data reload: false

------ Round 1 ----------------------------------
q1	17578	7440	7290	7290
q2	2047	185	174	174
q3	10571	1083	1239	1083
q4	10495	769	699	699
q5	7608	2723	2717	2717
q6	237	155	151	151
q7	993	638	622	622
q8	9248	1839	1916	1839
q9	6690	6478	6433	6433
q10	7000	2279	2326	2279
q11	471	267	263	263
q12	432	226	224	224
q13	17809	3027	3031	3027
q14	237	219	218	218
q15	567	536	518	518
q16	685	594	575	575
q17	1003	629	526	526
q18	7372	6661	6700	6661
q19	1339	977	968	968
q20	466	183	182	182
q21	4081	3447	3383	3383
q22	372	310	319	310
Total cold run time: 107301 ms
Total hot run time: 40142 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7223	7277	7192	7192
q2	332	231	232	231
q3	2914	2866	2994	2866
q4	2082	1874	1901	1874
q5	5710	5669	5685	5669
q6	229	143	144	143
q7	2258	1825	1841	1825
q8	3406	3570	3503	3503
q9	8848	9035	8953	8953
q10	3622	3597	3548	3548
q11	604	520	502	502
q12	843	597	617	597
q13	12590	3265	3235	3235
q14	304	277	288	277
q15	581	529	524	524
q16	689	643	631	631
q17	1857	1652	1620	1620
q18	8298	7749	7497	7497
q19	1703	1646	1581	1581
q20	2131	1876	1892	1876
q21	5622	5545	5505	5505
q22	662	594	583	583
Total cold run time: 72508 ms
Total hot run time: 60232 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197754 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 83af8d1a8567ef542916a9c499b5a33f6b54f6d5, data reload: false

query1	1247	965	937	937
query2	6233	2096	2091	2091
query3	10978	4336	4426	4336
query4	67446	27863	23676	23676
query5	4999	480	445	445
query6	410	193	190	190
query7	5528	298	292	292
query8	305	227	222	222
query9	8564	2697	2688	2688
query10	416	257	242	242
query11	16918	15253	15889	15253
query12	152	106	111	106
query13	1472	437	429	429
query14	10541	7347	7611	7347
query15	211	200	203	200
query16	7169	514	489	489
query17	1097	585	590	585
query18	1931	318	318	318
query19	201	175	153	153
query20	122	117	115	115
query21	210	105	108	105
query22	4846	4620	4608	4608
query23	34885	34537	34630	34537
query24	5452	2524	2490	2490
query25	471	388	398	388
query26	640	156	151	151
query27	1928	280	278	278
query28	4616	2470	2496	2470
query29	669	400	406	400
query30	211	157	147	147
query31	1001	833	863	833
query32	65	60	52	52
query33	400	291	290	290
query34	954	527	568	527
query35	890	761	760	760
query36	1084	958	990	958
query37	116	69	77	69
query38	4530	4375	4479	4375
query39	1529	1512	1461	1461
query40	208	99	98	98
query41	45	44	41	41
query42	109	98	98	98
query43	521	494	501	494
query44	1216	858	815	815
query45	188	173	169	169
query46	1169	744	730	730
query47	2035	1938	1925	1925
query48	407	315	319	315
query49	740	391	378	378
query50	849	397	387	387
query51	7378	7234	7108	7108
query52	102	88	87	87
query53	259	185	181	181
query54	517	395	392	392
query55	81	76	74	74
query56	243	246	232	232
query57	1255	1144	1123	1123
query58	219	203	220	203
query59	3271	3009	3052	3009
query60	273	236	240	236
query61	105	107	116	107
query62	777	696	667	667
query63	213	188	198	188
query64	1376	660	701	660
query65	3280	3221	3215	3215
query66	695	301	304	301
query67	16136	15512	15720	15512
query68	3811	564	557	557
query69	446	249	256	249
query70	1202	1149	1045	1045
query71	352	250	242	242
query72	6399	4085	4041	4041
query73	755	358	356	356
query74	10201	9076	9080	9076
query75	3421	2668	2684	2668
query76	1919	1071	1059	1059
query77	476	267	281	267
query78	10424	9417	9368	9368
query79	1510	599	598	598
query80	902	473	431	431
query81	480	242	224	224
query82	1281	120	116	116
query83	167	148	138	138
query84	280	72	73	72
query85	837	297	297	297
query86	335	295	282	282
query87	4617	4561	4684	4561
query88	3552	2189	2173	2173
query89	419	299	289	289
query90	2013	190	188	188
query91	138	100	172	100
query92	60	50	47	47
query93	1966	543	544	543
query94	766	284	293	284
query95	335	242	249	242
query96	611	280	279	279
query97	2873	2713	2704	2704
query98	229	193	191	191
query99	1595	1309	1321	1309
Total cold run time: 318476 ms
Total hot run time: 197754 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.81 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 83af8d1a8567ef542916a9c499b5a33f6b54f6d5, data reload: false

query1	0.03	0.03	0.04
query2	0.06	0.04	0.03
query3	0.22	0.07	0.07
query4	1.64	0.10	0.11
query5	0.41	0.42	0.44
query6	1.13	0.66	0.66
query7	0.02	0.02	0.02
query8	0.05	0.03	0.04
query9	0.59	0.51	0.51
query10	0.56	0.57	0.55
query11	0.15	0.10	0.11
query12	0.15	0.12	0.12
query13	0.62	0.60	0.60
query14	2.76	2.69	2.81
query15	0.92	0.83	0.83
query16	0.39	0.38	0.37
query17	1.00	1.07	1.06
query18	0.22	0.21	0.21
query19	1.91	1.86	1.98
query20	0.01	0.02	0.02
query21	15.38	0.60	0.60
query22	2.54	2.08	1.20
query23	16.94	1.11	0.84
query24	2.94	1.86	1.66
query25	0.20	0.14	0.10
query26	0.63	0.15	0.15
query27	0.04	0.04	0.04
query28	9.61	1.11	1.08
query29	12.55	3.23	3.22
query30	0.25	0.06	0.07
query31	2.86	0.40	0.38
query32	3.27	0.46	0.48
query33	3.01	3.14	3.03
query34	17.10	4.42	4.42
query35	4.50	4.54	4.50
query36	0.68	0.48	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.12	0.12
query41	0.07	0.02	0.03
query42	0.03	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 105.82 s
Total hot run time: 32.81 s

Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@liaoxin01 liaoxin01 merged commit 682be10 into apache:master Dec 6, 2024
28 of 30 checks passed
github-actions bot pushed a commit that referenced this pull request Dec 6, 2024
Related PR: #39975
In the above PR, we introduce delay schedule EOF tasks to avoid too many
small transactions, but it is invalid, the pr fix the issue.
github-actions bot pushed a commit that referenced this pull request Dec 6, 2024
Related PR: #39975
In the above PR, we introduce delay schedule EOF tasks to avoid too many
small transactions, but it is invalid, the pr fix the issue.
yiguolei pushed a commit that referenced this pull request Dec 6, 2024
dataroaring pushed a commit that referenced this pull request Dec 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.8-merged dev/3.0.4-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants