Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix](log) Fix some delete bitmap log #49009

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

bobhan1
Copy link
Contributor

@bobhan1 bobhan1 commented Mar 13, 2025

What problem does this PR solve?

  1. fix wrong delete bitmap logs in [improve](mow) merge and remove old version of delete bitmap when cumulative compaction is done on local mode #41636 and [Opt](log) Add more delete bitmap log for merge-on-write table #48930
  2. add count and cardinality log when calculating between segments

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@bobhan1
Copy link
Contributor Author

bobhan1 commented Mar 13, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32334 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cd27f9aac0366a496e2531bfac972aab6671b209, data reload: false

------ Round 1 ----------------------------------
q1	17630	5198	5094	5094
q2	2038	287	166	166
q3	10420	1313	685	685
q4	10222	1012	548	548
q5	7496	2420	2367	2367
q6	196	165	132	132
q7	937	737	596	596
q8	9305	1260	1116	1116
q9	4863	4663	4803	4663
q10	6828	2299	1895	1895
q11	478	276	254	254
q12	352	352	220	220
q13	17774	3646	3111	3111
q14	233	237	220	220
q15	533	483	481	481
q16	650	626	603	603
q17	572	855	359	359
q18	7032	6459	6176	6176
q19	1633	949	566	566
q20	319	321	195	195
q21	2947	2102	1915	1915
q22	1023	1023	972	972
Total cold run time: 103481 ms
Total hot run time: 32334 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5221	5171	5170	5170
q2	237	333	231	231
q3	2155	2672	2279	2279
q4	1459	1841	1387	1387
q5	4247	4164	4150	4150
q6	204	164	125	125
q7	1885	1956	1794	1794
q8	2601	2638	2645	2638
q9	7304	7183	7104	7104
q10	3032	3184	2810	2810
q11	585	499	488	488
q12	673	779	609	609
q13	3550	3879	3327	3327
q14	274	281	286	281
q15	526	474	484	474
q16	639	694	662	662
q17	1150	1581	1356	1356
q18	7752	7512	7362	7362
q19	829	845	857	845
q20	1967	2022	1881	1881
q21	5473	5071	4935	4935
q22	1113	1134	1016	1016
Total cold run time: 52876 ms
Total hot run time: 50924 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184863 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cd27f9aac0366a496e2531bfac972aab6671b209, data reload: false

query1	980	389	386	386
query2	6536	1833	1889	1833
query3	6795	225	225	225
query4	26624	23653	22932	22932
query5	4395	681	488	488
query6	297	202	200	200
query7	4608	503	293	293
query8	300	263	238	238
query9	8604	2512	2523	2512
query10	495	322	252	252
query11	15779	15199	14953	14953
query12	165	119	109	109
query13	1663	521	388	388
query14	9817	6473	7020	6473
query15	208	193	176	176
query16	7617	670	499	499
query17	1201	707	566	566
query18	1988	419	309	309
query19	192	184	171	171
query20	128	113	116	113
query21	214	123	108	108
query22	4247	4051	3944	3944
query23	34064	32783	32944	32783
query24	7666	2397	2375	2375
query25	520	471	388	388
query26	1208	275	152	152
query27	2157	490	322	322
query28	3911	2381	2363	2363
query29	723	575	422	422
query30	285	224	198	198
query31	957	850	779	779
query32	74	67	64	64
query33	577	358	308	308
query34	782	835	504	504
query35	791	833	771	771
query36	952	993	901	901
query37	114	98	76	76
query38	4105	4203	4166	4166
query39	1449	1384	1421	1384
query40	212	117	107	107
query41	55	52	51	51
query42	116	104	107	104
query43	508	528	467	467
query44	1285	774	795	774
query45	175	170	169	169
query46	844	1051	621	621
query47	1740	1793	1688	1688
query48	387	404	298	298
query49	788	518	421	421
query50	690	723	407	407
query51	4129	4194	4138	4138
query52	101	106	98	98
query53	226	252	188	188
query54	500	492	406	406
query55	79	81	83	81
query56	300	277	244	244
query57	1102	1116	1062	1062
query58	245	236	229	229
query59	2582	2691	2554	2554
query60	280	267	267	267
query61	126	118	118	118
query62	820	717	682	682
query63	234	189	192	189
query64	4254	1012	720	720
query65	4457	4361	4369	4361
query66	1092	407	296	296
query67	15775	15475	15279	15279
query68	8135	871	497	497
query69	460	325	259	259
query70	1201	1131	1149	1131
query71	463	309	274	274
query72	5507	3544	3718	3544
query73	801	704	340	340
query74	9295	9100	8800	8800
query75	3799	3155	2698	2698
query76	3728	1165	764	764
query77	786	361	304	304
query78	10042	10076	9296	9296
query79	2606	814	594	594
query80	657	522	458	458
query81	496	257	293	257
query82	697	130	99	99
query83	177	182	152	152
query84	247	92	73	73
query85	804	361	314	314
query86	396	311	302	302
query87	4466	4442	4432	4432
query88	3660	2327	2259	2259
query89	396	328	291	291
query90	1851	223	224	223
query91	151	137	112	112
query92	78	57	55	55
query93	1702	1074	587	587
query94	654	425	299	299
query95	358	277	269	269
query96	491	565	284	284
query97	3287	3372	3264	3264
query98	237	213	206	206
query99	1439	1388	1283	1283
Total cold run time: 275357 ms
Total hot run time: 184863 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.06 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cd27f9aac0366a496e2531bfac972aab6671b209, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.04
query3	0.24	0.07	0.06
query4	1.62	0.11	0.10
query5	0.56	0.53	0.54
query6	1.17	0.73	0.71
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.53	0.54
query10	0.58	0.61	0.58
query11	0.16	0.11	0.10
query12	0.15	0.11	0.10
query13	0.63	0.60	0.61
query14	2.64	2.71	2.80
query15	0.92	0.86	0.83
query16	0.41	0.38	0.37
query17	1.02	1.02	1.00
query18	0.21	0.19	0.19
query19	1.87	1.98	1.78
query20	0.01	0.01	0.01
query21	15.35	0.92	0.56
query22	0.75	1.23	0.92
query23	14.69	1.38	0.63
query24	6.61	2.11	0.67
query25	0.59	0.15	0.12
query26	0.60	0.16	0.15
query27	0.06	0.04	0.04
query28	10.17	0.91	0.45
query29	12.56	4.08	3.35
query30	0.25	0.10	0.07
query31	2.81	0.60	0.40
query32	3.22	0.54	0.46
query33	2.99	3.02	3.04
query34	15.87	5.18	4.54
query35	4.56	4.56	4.56
query36	0.67	0.50	0.50
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.04	0.03	0.02
query40	0.16	0.12	0.12
query41	0.07	0.02	0.03
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.2 s
Total hot run time: 31.06 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 13, 2025
Copy link
Contributor

PR approved by anyone and no changes requested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.x reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants