Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feat](binlog) filter the async mv binlogs #49028

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

w41ter
Copy link
Contributor

@w41ter w41ter commented Mar 13, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: selectdb/ccr-syncer#509

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Mar 13, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@w41ter
Copy link
Contributor Author

w41ter commented Mar 13, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32673 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 61a81a97af80f5c48d034e80602686af076b151e, data reload: false

------ Round 1 ----------------------------------
q1	17643	5261	5103	5103
q2	2075	296	163	163
q3	10421	1364	715	715
q4	10212	1067	558	558
q5	7545	2464	2369	2369
q6	188	170	134	134
q7	922	742	633	633
q8	9322	1326	1128	1128
q9	5037	4826	4632	4632
q10	6830	2308	1897	1897
q11	502	282	261	261
q12	347	344	220	220
q13	17770	3708	3132	3132
q14	223	229	211	211
q15	540	480	493	480
q16	643	636	609	609
q17	575	890	342	342
q18	6855	6399	6362	6362
q19	1073	957	550	550
q20	322	340	202	202
q21	2957	2239	2008	2008
q22	1017	1007	964	964
Total cold run time: 103019 ms
Total hot run time: 32673 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5153	5155	5142	5142
q2	243	326	231	231
q3	2185	2670	2281	2281
q4	1486	1837	1373	1373
q5	4224	4143	4194	4143
q6	202	172	124	124
q7	1879	2043	1798	1798
q8	2643	2645	2535	2535
q9	7184	7173	7111	7111
q10	3025	3210	2765	2765
q11	585	522	496	496
q12	718	804	620	620
q13	3512	3958	3273	3273
q14	281	292	274	274
q15	524	484	471	471
q16	646	696	631	631
q17	1164	1587	1374	1374
q18	7801	7652	7509	7509
q19	835	817	933	817
q20	1981	2105	1913	1913
q21	5541	5036	4872	4872
q22	1119	1036	1014	1014
Total cold run time: 52931 ms
Total hot run time: 50767 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185505 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 61a81a97af80f5c48d034e80602686af076b151e, data reload: false

query1	996	386	382	382
query2	6535	1917	1919	1917
query3	6789	216	215	215
query4	26260	24044	23445	23445
query5	4912	629	472	472
query6	320	213	193	193
query7	4607	500	301	301
query8	305	258	245	245
query9	8642	2615	2617	2615
query10	503	313	268	268
query11	15976	15148	15003	15003
query12	171	117	106	106
query13	1652	533	413	413
query14	10071	6820	6328	6328
query15	210	195	177	177
query16	7657	630	458	458
query17	1557	722	568	568
query18	1996	413	310	310
query19	196	186	158	158
query20	117	114	117	114
query21	212	125	105	105
query22	4113	4336	3950	3950
query23	34044	32837	32911	32837
query24	7622	2377	2394	2377
query25	498	476	386	386
query26	1223	276	153	153
query27	2104	485	326	326
query28	3949	2420	2414	2414
query29	732	565	428	428
query30	289	220	210	210
query31	946	854	768	768
query32	84	66	62	62
query33	562	362	299	299
query34	769	836	506	506
query35	786	821	744	744
query36	990	1003	891	891
query37	126	102	77	77
query38	4173	4297	4265	4265
query39	1456	1411	1428	1411
query40	207	119	104	104
query41	56	51	56	51
query42	118	101	103	101
query43	493	498	484	484
query44	1292	780	781	780
query45	179	170	169	169
query46	813	1024	627	627
query47	1728	1780	1710	1710
query48	379	412	294	294
query49	785	549	445	445
query50	672	717	401	401
query51	4205	4245	4141	4141
query52	108	107	95	95
query53	240	260	200	200
query54	503	501	412	412
query55	77	82	80	80
query56	284	284	248	248
query57	1126	1136	1076	1076
query58	250	240	237	237
query59	2555	2676	2473	2473
query60	286	282	256	256
query61	121	148	122	122
query62	778	749	672	672
query63	225	188	189	188
query64	4252	1021	686	686
query65	4423	4339	4366	4339
query66	1087	420	308	308
query67	15498	15735	15255	15255
query68	8209	890	520	520
query69	461	306	263	263
query70	1227	1150	1130	1130
query71	467	283	262	262
query72	5722	3587	3825	3587
query73	811	733	345	345
query74	9141	9167	8836	8836
query75	3760	3132	2715	2715
query76	3739	1200	764	764
query77	791	360	284	284
query78	9889	10233	9297	9297
query79	2790	826	578	578
query80	747	526	453	453
query81	467	262	218	218
query82	678	133	98	98
query83	199	174	159	159
query84	290	98	74	74
query85	876	358	309	309
query86	357	310	310	310
query87	4447	4637	4335	4335
query88	2950	2282	2261	2261
query89	409	314	287	287
query90	1915	219	210	210
query91	141	143	115	115
query92	79	63	56	56
query93	1203	1038	577	577
query94	686	418	310	310
query95	363	268	259	259
query96	490	558	278	278
query97	3406	3383	3312	3312
query98	229	207	207	207
query99	1438	1402	1245	1245
Total cold run time: 275267 ms
Total hot run time: 185505 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.42 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 61a81a97af80f5c48d034e80602686af076b151e, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.04	0.03
query3	0.23	0.06	0.07
query4	1.62	0.10	0.11
query5	0.57	0.55	0.54
query6	1.18	0.72	0.72
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.59	0.52	0.54
query10	0.58	0.61	0.57
query11	0.15	0.11	0.11
query12	0.14	0.12	0.11
query13	0.62	0.61	0.60
query14	2.69	2.67	2.70
query15	0.92	0.85	0.86
query16	0.38	0.38	0.38
query17	1.01	1.04	1.06
query18	0.20	0.20	0.19
query19	1.97	1.86	2.01
query20	0.02	0.01	0.01
query21	15.36	0.92	0.54
query22	0.77	1.23	0.66
query23	14.86	1.41	0.65
query24	7.48	1.29	1.35
query25	0.51	0.20	0.25
query26	0.57	0.16	0.13
query27	0.05	0.05	0.04
query28	9.86	0.88	0.44
query29	12.57	4.05	3.36
query30	0.25	0.09	0.06
query31	2.81	0.57	0.40
query32	3.22	0.54	0.47
query33	3.07	3.01	3.00
query34	15.67	5.12	4.46
query35	4.52	4.49	4.51
query36	0.66	0.50	0.49
query37	0.09	0.07	0.06
query38	0.06	0.04	0.03
query39	0.04	0.03	0.02
query40	0.17	0.13	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 105.77 s
Total hot run time: 31.42 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 13, 2025
Copy link
Contributor

PR approved by anyone and no changes requested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.x dev/3.0.x reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants