Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](inverted index) Writing to the inverted index also writes to the file cache. #39076

Merged
merged 1 commit into from
Aug 13, 2024
Merged

[fix](inverted index) Writing to the inverted index also writes to the file cache. #39076

merged 1 commit into from
Aug 13, 2024

Conversation

zzzxl1993
Copy link
Contributor

@zzzxl1993 zzzxl1993 commented Aug 8, 2024 •
edited
Loading

Proposed changes

  1. When write_file_cache is true, writing to the inverted index also writes to the file cache.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions github-actions bot added the doing label Aug 8, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 8, 2024

clang-tidy review says "All clean, LGTM! 👍"

@zzzxl1993
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 8, 2024

clang-tidy review says "All clean, LGTM! 👍"

2 similar comments
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 8, 2024

clang-tidy review says "All clean, LGTM! 👍"

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 8, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39692 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7956e1678a5be2bde57b45c40d1954736274c173, data reload: false

------ Round 1 ----------------------------------
q1	17606	4326	4356	4326
q2	2026	182	184	182
q3	10463	1253	1086	1086
q4	10128	708	802	708
q5	7551	2551	2541	2541
q6	225	139	140	139
q7	965	594	615	594
q8	9409	1911	1917	1911
q9	10212	6580	6598	6580
q10	7063	2153	2243	2153
q11	447	251	256	251
q12	500	223	232	223
q13	19011	2964	2959	2959
q14	284	232	237	232
q15	541	483	499	483
q16	505	397	390	390
q17	984	732	668	668
q18	8061	7482	7480	7480
q19	1385	962	1006	962
q20	649	335	339	335
q21	5307	4497	4628	4497
q22	1104	1024	992	992
Total cold run time: 114426 ms
Total hot run time: 39692 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4311	4292	4274	4274
q2	367	285	276	276
q3	2844	2658	2598	2598
q4	1857	1558	1555	1555
q5	5207	5260	5295	5260
q6	216	137	132	132
q7	2023	1670	1675	1670
q8	3151	3343	3330	3330
q9	8423	8328	8368	8328
q10	3386	3121	3186	3121
q11	590	482	483	482
q12	761	606	629	606
q13	16772	2959	2978	2959
q14	299	260	274	260
q15	518	473	480	473
q16	470	407	412	407
q17	1778	1493	1461	1461
q18	7603	7592	7296	7296
q19	1634	1571	1526	1526
q20	1979	1788	1813	1788
q21	5324	5078	5088	5078
q22	1080	1011	998	998
Total cold run time: 70593 ms
Total hot run time: 53878 ms

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 9, 2024

clang-tidy review says "All clean, LGTM! 👍"

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 9, 2024

clang-tidy review says "All clean, LGTM! 👍"

@zzzxl1993
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@zzzxl1993 zzzxl1993 changed the title [fix](inverted index) The inverted index is written into the file cache [fix](inverted index) while writing the inverted index, it is simultaneously written to the file cache. Aug 13, 2024
@zzzxl1993 zzzxl1993 changed the title [fix](inverted index) while writing the inverted index, it is simultaneously written to the file cache. [fix](inverted index) Writing to the inverted index also writes to the file cache. Aug 13, 2024
@zzzxl1993
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40079 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 84e550c41bb32a8fdfc9aec7d43583c10fc237a7, data reload: false

------ Round 1 ----------------------------------
q1	17628	4425	4294	4294
q2	2012	184	179	179
q3	11818	1012	1121	1012
q4	10511	739	759	739
q5	7768	2853	2857	2853
q6	229	143	140	140
q7	965	597	605	597
q8	9550	2073	2061	2061
q9	9990	6590	6623	6590
q10	7083	2218	2210	2210
q11	457	248	249	248
q12	411	241	229	229
q13	17752	3005	3026	3005
q14	283	232	228	228
q15	525	472	489	472
q16	503	403	387	387
q17	990	667	648	648
q18	8114	7548	7396	7396
q19	7552	1055	977	977
q20	668	345	337	337
q21	5359	4653	4471	4471
q22	1103	1006	1022	1006
Total cold run time: 121271 ms
Total hot run time: 40079 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4522	4307	4285	4285
q2	379	279	278	278
q3	3013	2833	2804	2804
q4	2006	1763	1701	1701
q5	5611	5584	5685	5584
q6	232	141	144	141
q7	2194	1777	1836	1777
q8	3339	3472	3428	3428
q9	8901	8940	8862	8862
q10	3439	3325	3294	3294
q11	612	525	529	525
q12	805	616	631	616
q13	16925	3218	3187	3187
q14	344	301	292	292
q15	562	499	482	482
q16	515	440	457	440
q17	1793	1532	1536	1532
q18	8261	8007	8068	8007
q19	1847	1696	1708	1696
q20	3067	1888	1878	1878
q21	8889	5210	5423	5210
q22	1132	1034	1054	1034
Total cold run time: 78388 ms
Total hot run time: 57053 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190847 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 84e550c41bb32a8fdfc9aec7d43583c10fc237a7, data reload: false

query1	1260	882	863	863
query2	6314	1888	1862	1862
query3	10614	4009	3528	3528
query4	59706	24802	23288	23288
query5	5487	521	483	483
query6	492	160	166	160
query7	6421	294	308	294
query8	312	205	213	205
query9	8794	2458	2452	2452
query10	503	273	266	266
query11	16604	15056	15066	15056
query12	161	101	97	97
query13	1581	394	385	385
query14	12640	8031	8129	8031
query15	247	172	182	172
query16	7278	484	483	483
query17	1162	626	606	606
query18	1969	323	304	304
query19	302	166	157	157
query20	127	115	114	114
query21	212	108	104	104
query22	4426	4258	4382	4258
query23	34294	33383	33171	33171
query24	5573	2893	2879	2879
query25	556	408	434	408
query26	695	164	163	163
query27	1793	278	278	278
query28	3874	2057	2048	2048
query29	704	436	440	436
query30	240	157	152	152
query31	909	758	764	758
query32	81	55	57	55
query33	493	304	301	301
query34	851	475	488	475
query35	804	715	753	715
query36	1054	940	925	925
query37	139	81	82	81
query38	3877	3936	3787	3787
query39	1425	1400	1407	1400
query40	205	126	119	119
query41	49	46	47	46
query42	114	99	103	99
query43	497	461	486	461
query44	1070	739	748	739
query45	195	166	164	164
query46	1100	757	742	742
query47	1896	1732	1736	1732
query48	366	303	293	293
query49	775	436	448	436
query50	812	418	422	418
query51	6833	6797	6688	6688
query52	107	92	91	91
query53	256	188	185	185
query54	568	466	449	449
query55	76	75	76	75
query56	265	254	271	254
query57	1154	1039	1039	1039
query58	223	224	232	224
query59	2882	2718	2761	2718
query60	299	273	263	263
query61	112	96	111	96
query62	736	639	646	639
query63	228	187	185	185
query64	3341	1782	1758	1758
query65	3189	3128	3171	3128
query66	682	335	329	329
query67	15437	14979	14728	14728
query68	4920	546	558	546
query69	578	297	272	272
query70	1183	1137	1120	1120
query71	511	280	280	280
query72	7092	2283	2040	2040
query73	800	330	340	330
query74	9105	8778	8767	8767
query75	4795	2693	2717	2693
query76	3344	1063	983	983
query77	769	316	320	316
query78	9773	9138	8949	8949
query79	3038	532	524	524
query80	2537	493	495	493
query81	565	224	224	224
query82	775	137	134	134
query83	303	142	143	142
query84	279	84	77	77
query85	1491	318	276	276
query86	461	306	304	304
query87	4399	4210	4154	4154
query88	4381	2525	2414	2414
query89	402	284	285	284
query90	1992	202	195	195
query91	125	114	100	100
query92	67	50	53	50
query93	4227	542	529	529
query94	990	301	289	289
query95	353	269	273	269
query96	626	280	277	277
query97	3237	3052	3052	3052
query98	231	203	204	203
query99	1678	1272	1263	1263
Total cold run time: 322468 ms
Total hot run time: 190847 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.51 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 84e550c41bb32a8fdfc9aec7d43583c10fc237a7, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.04	0.05
query4	1.69	0.07	0.08
query5	0.50	0.52	0.48
query6	1.13	0.71	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.47	0.48
query10	0.54	0.55	0.54
query11	0.15	0.12	0.11
query12	0.16	0.12	0.12
query13	0.59	0.59	0.58
query14	0.77	0.77	0.79
query15	0.87	0.81	0.81
query16	0.38	0.36	0.35
query17	0.98	0.97	1.01
query18	0.22	0.21	0.22
query19	1.89	1.75	1.76
query20	0.02	0.01	0.01
query21	15.39	0.76	0.67
query22	4.34	7.11	1.76
query23	18.25	1.34	1.29
query24	2.12	0.22	0.22
query25	0.14	0.08	0.10
query26	0.31	0.22	0.22
query27	0.45	0.22	0.22
query28	13.37	1.03	1.00
query29	12.62	3.33	3.29
query30	0.24	0.05	0.05
query31	2.90	0.39	0.39
query32	3.27	0.48	0.47
query33	2.94	3.03	2.96
query34	17.10	4.34	4.39
query35	4.38	4.39	4.40
query36	0.66	0.47	0.46
query37	0.19	0.16	0.15
query38	0.16	0.15	0.14
query39	0.05	0.04	0.04
query40	0.15	0.12	0.13
query41	0.09	0.05	0.05
query42	0.06	0.04	0.05
query43	0.05	0.05	0.04
Total cold run time: 110.07 s
Total hot run time: 30.51 s

airborne12
airborne12 approved these changes Aug 13, 2024
View reviewed changes
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 13, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

qidaye
qidaye approved these changes Aug 13, 2024
View reviewed changes
Copy link
Contributor

@qidaye qidaye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@qidaye qidaye merged commit b0519d2 into apache:master Aug 13, 2024
27 of 30 checks passed
wyxxxcat pushed a commit to wyxxxcat/doris that referenced this pull request Aug 14, 2024
@zzzxl1993 @wyxxxcat
[fix](inverted index) Writing to the inverted index also writes to th…
9381700 
…e file cache. (apache#39076)

1. When write_file_cache is true, writing to the inverted index also
writes to the file cache.
@xiaokang xiaokang removed the doing label Aug 14, 2024
dataroaring pushed a commit that referenced this pull request Aug 17, 2024
@zzzxl1993 @dataroaring
[fix](inverted index) Writing to the inverted index also writes to th…
f30529a 
…e file cache. (#39076)

1. When write_file_cache is true, writing to the inverted index also
writes to the file cache.
@qidaye qidaye mentioned this pull request Sep 2, 2024
Merged
qidaye added a commit that referenced this pull request Sep 3, 2024
@qidaye
[fix](index compaction)Fix core when having multiple dest segments (#…
ee3be4f 
…40254)

introduced by: #39076
qidaye added a commit to qidaye/incubator-doris that referenced this pull request Sep 3, 2024
@qidaye
[fix](index compaction)Fix core when having multiple dest segments (a…
58484a5 
…pache#40254)

introduced by: apache#39076
@gavinchou gavinchou mentioned this pull request Oct 13, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@airborne12 airborne12 airborne12 approved these changes

@qidaye qidaye qidaye approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.2-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@zzzxl1993 @doris-robot @airborne12 @qidaye @xiaokang @dataroaring