Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](resource) Fix register ResourceContext to RuntimeQueryStatisticsMgr #48004

Merged
merged 1 commit into from
Feb 19, 2025
Merged

[fix](resource) Fix register ResourceContext to RuntimeQueryStatisticsMgr #48004

merged 1 commit into from
Feb 19, 2025

Conversation

xinyiZzz
Copy link
Contributor

@xinyiZzz xinyiZzz commented Feb 18, 2025 •
edited
Loading

What problem does this PR solve?

Wait for QueryTaskController to be initialized before register ResourceContext

fix

/root/doris/be/src/runtime/workload_management/task_controller.h:59:44: runtime error: load of value 3200171710, which is not a valid value for type 'TQueryType::type'
    #0 0x555ff9336162 in doris::TaskController::query_type() /root/doris/be/src/runtime/workload_management/task_controller.h:59:44
    #1 0x555ff9336162 in doris::RuntimeQueryStatisticsMgr::report_runtime_query_statistics() /root/doris/be/src/runtime/runtime_query_statistics_mgr.cpp:346:50
    #2 0x555ff50d789d in doris::Daemon::report_runtime_query_statistics_thread() /root/doris/be/src/common/daemon.cpp:487:65
    #3 0x555ff9a3efae in std::function<void ()>::operator()() const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9
    #4 0x555ff9a3efae in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5
    #5 0x7f4935d00608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8
    #6 0x7f4935fad132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Feb 18, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@xinyiZzz
Copy link
Contributor Author

run buildall

@xinyiZzz xinyiZzz changed the title [fix](resource) Fix register ResourceContextto RuntimeQueryStatisticsMgr [fix](resource) Fix register ResourceContext to RuntimeQueryStatisticsMgr Feb 18, 2025
@doris-robot
Copy link

TPC-H: Total hot run time: 31222 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f71166b766df2d9928629e0c024b7d1c54bf3af5, data reload: false

------ Round 1 ----------------------------------
q1	17599	5134	5030	5030
q2	2051	280	168	168
q3	10419	1220	731	731
q4	10201	997	508	508
q5	7518	2345	2292	2292
q6	188	165	133	133
q7	912	741	598	598
q8	9332	1286	1047	1047
q9	4836	4783	4749	4749
q10	6820	2292	1871	1871
q11	470	275	256	256
q12	345	344	211	211
q13	17755	3729	3033	3033
q14	230	220	215	215
q15	495	461	453	453
q16	603	620	577	577
q17	552	849	334	334
q18	6544	6411	6109	6109
q19	1214	942	519	519
q20	302	312	186	186
q21	2639	2127	1901	1901
q22	365	331	301	301
Total cold run time: 101390 ms
Total hot run time: 31222 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5169	5147	5097	5097
q2	236	331	238	238
q3	2125	2650	2324	2324
q4	1435	1790	1358	1358
q5	4219	4151	4155	4151
q6	206	161	123	123
q7	1853	1812	1673	1673
q8	2563	2638	2553	2553
q9	7145	7162	7092	7092
q10	3021	3168	2752	2752
q11	576	522	494	494
q12	666	754	622	622
q13	3542	3852	3289	3289
q14	278	290	269	269
q15	510	478	468	468
q16	633	673	645	645
q17	1128	1627	1304	1304
q18	7398	7323	7264	7264
q19	758	764	792	764
q20	1934	2062	1896	1896
q21	5375	4943	4843	4843
q22	615	567	532	532
Total cold run time: 51385 ms
Total hot run time: 49751 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189350 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f71166b766df2d9928629e0c024b7d1c54bf3af5, data reload: false

query1	1285	953	950	950
query2	6233	1843	1781	1781
query3	10960	4393	4483	4393
query4	56012	25449	22922	22922
query5	5189	541	479	479
query6	360	183	185	183
query7	5009	487	295	295
query8	319	245	236	236
query9	6247	2465	2451	2451
query10	400	305	247	247
query11	15415	15067	14988	14988
query12	153	105	103	103
query13	1122	494	372	372
query14	10200	7173	6405	6405
query15	195	204	175	175
query16	7064	647	468	468
query17	1062	721	544	544
query18	1526	403	306	306
query19	195	199	149	149
query20	119	119	124	119
query21	206	128	104	104
query22	4665	4440	4444	4440
query23	33831	33391	33304	33304
query24	5719	2464	2407	2407
query25	470	489	387	387
query26	664	271	155	155
query27	1659	507	336	336
query28	2844	2414	2383	2383
query29	560	579	421	421
query30	209	188	155	155
query31	874	896	841	841
query32	78	59	63	59
query33	428	352	295	295
query34	750	866	504	504
query35	831	853	745	745
query36	950	996	914	914
query37	115	101	77	77
query38	4276	4200	4228	4200
query39	1506	1448	1410	1410
query40	208	122	107	107
query41	56	56	54	54
query42	128	105	109	105
query43	498	505	489	489
query44	1270	782	783	782
query45	177	172	165	165
query46	876	1060	661	661
query47	1822	1867	1800	1800
query48	380	427	312	312
query49	722	575	405	405
query50	693	756	423	423
query51	4236	4271	4312	4271
query52	105	103	99	99
query53	223	260	185	185
query54	485	488	411	411
query55	85	81	78	78
query56	286	279	288	279
query57	1152	1178	1137	1137
query58	234	241	258	241
query59	2541	2776	2723	2723
query60	283	272	267	267
query61	133	115	121	115
query62	744	740	682	682
query63	236	187	188	187
query64	1444	1034	716	716
query65	3350	3168	3159	3159
query66	763	393	291	291
query67	15784	15480	15640	15480
query68	5400	760	501	501
query69	527	295	258	258
query70	1194	1061	1091	1061
query71	430	295	276	276
query72	6027	3670	3747	3670
query73	1236	743	334	334
query74	9028	9499	8852	8852
query75	3288	3145	2653	2653
query76	3903	1190	725	725
query77	526	363	277	277
query78	9941	9933	9372	9372
query79	2105	829	590	590
query80	748	527	454	454
query81	489	278	238	238
query82	279	131	103	103
query83	177	174	149	149
query84	287	94	71	71
query85	739	362	306	306
query86	337	300	262	262
query87	4499	4665	4439	4439
query88	2954	2154	2194	2154
query89	395	310	285	285
query90	1872	196	194	194
query91	131	135	108	108
query92	71	59	54	54
query93	2624	1011	570	570
query94	667	411	302	302
query95	341	268	255	255
query96	475	547	266	266
query97	2749	2810	2725	2725
query98	225	235	201	201
query99	1320	1424	1318	1318
Total cold run time: 293598 ms
Total hot run time: 189350 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.64 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f71166b766df2d9928629e0c024b7d1c54bf3af5, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.24	0.06	0.06
query4	1.63	0.10	0.10
query5	0.44	0.41	0.40
query6	1.16	0.67	0.67
query7	0.02	0.02	0.02
query8	0.05	0.03	0.03
query9	0.59	0.52	0.51
query10	0.58	0.58	0.56
query11	0.16	0.10	0.11
query12	0.15	0.11	0.11
query13	0.62	0.60	0.59
query14	2.70	2.70	2.67
query15	0.91	0.85	0.86
query16	0.38	0.37	0.37
query17	1.01	1.03	1.04
query18	0.21	0.21	0.20
query19	1.93	1.84	1.99
query20	0.01	0.02	0.01
query21	15.36	0.92	0.55
query22	0.77	1.18	0.72
query23	14.85	1.41	0.59
query24	10.71	1.43	0.84
query25	0.57	0.24	0.08
query26	0.93	0.19	0.15
query27	0.04	0.05	0.05
query28	6.52	0.77	0.44
query29	12.53	3.93	3.27
query30	0.25	0.08	0.05
query31	2.83	0.58	0.39
query32	3.22	0.56	0.47
query33	3.00	3.00	3.04
query34	15.60	5.15	4.54
query35	4.55	4.52	4.54
query36	0.68	0.49	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.14	0.13
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.8 s
Total hot run time: 30.64 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 43.81% (11441/26114)
Line Coverage: 33.82% (96350/284896)
Region Coverage: 32.53% (49294/151557)
Branch Coverage: 28.20% (24771/87856)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f71166b766df2d9928629e0c024b7d1c54bf3af5_f71166b766df2d9928629e0c024b7d1c54bf3af5/report/index.html

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 18, 2025
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

wangbo
wangbo approved these changes Feb 19, 2025
View reviewed changes
Copy link
Contributor

@wangbo wangbo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xinyiZzz xinyiZzz merged commit e7c966c into apache:master Feb 19, 2025
27 of 29 checks passed
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
@xinyiZzz @lzyy2024
[fix](resource) Fix register ResourceContext to RuntimeQueryStatistic…
2265362 
…sMgr (apache#48004)

### What problem does this PR solve?

Wait for QueryTaskController to be initialized before register
ResourceContext

fix
```
/root/doris/be/src/runtime/workload_management/task_controller.h:59:44: runtime error: load of value 3200171710, which is not a valid value for type 'TQueryType::type'
    #0 0x555ff9336162 in doris::TaskController::query_type() /root/doris/be/src/runtime/workload_management/task_controller.h:59:44
    apache#1 0x555ff9336162 in doris::RuntimeQueryStatisticsMgr::report_runtime_query_statistics() /root/doris/be/src/runtime/runtime_query_statistics_mgr.cpp:346:50
    apache#2 0x555ff50d789d in doris::Daemon::report_runtime_query_statistics_thread() /root/doris/be/src/common/daemon.cpp:487:65
    apache#3 0x555ff9a3efae in std::function<void ()>::operator()() const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9
    apache#4 0x555ff9a3efae in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5
    apache#5 0x7f4935d00608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8
    apache#6 0x7f4935fad132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yiguolei yiguolei yiguolei approved these changes

@wangbo wangbo wangbo approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@xinyiZzz @Thearas @doris-robot @yiguolei @wangbo