Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](Nereids) tablet prune wrong when decimal value scale is nagtive #37889

Merged
merged 1 commit into from
Jul 17, 2024
Merged

[fix](Nereids) tablet prune wrong when decimal value scale is nagtive #37889

merged 1 commit into from
Jul 17, 2024

Conversation

morrySnow
Copy link
Contributor

we use unscaled value of BigDecimal in tablet prune. So we need to ensure BigDecimal's precision and scale is same with the literal contains it.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@morrySnow
Copy link
Contributor Author

run buildall

@morrySnow morrySnow added usercase Important user case type label p0_w dev/2.0.x dev/2.1.x labels Jul 16, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 39744 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0cd67e7afca414ca0628570e28d5c7f97f1be72c, data reload: false

------ Round 1 ----------------------------------
q1	17637	4292	4293	4292
q2	2022	194	186	186
q3	10438	1214	1105	1105
q4	10183	781	832	781
q5	7549	2680	2644	2644
q6	224	140	140	140
q7	958	602	596	596
q8	9214	2070	2094	2070
q9	8791	6554	6531	6531
q10	8762	3802	3800	3800
q11	454	234	234	234
q12	453	229	232	229
q13	17777	2997	2969	2969
q14	279	232	233	232
q15	520	492	497	492
q16	487	397	376	376
q17	975	703	673	673
q18	7952	7455	7306	7306
q19	7099	1396	1300	1300
q20	707	331	332	331
q21	4884	3175	3264	3175
q22	349	282	288	282
Total cold run time: 117714 ms
Total hot run time: 39744 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4363	4238	4229	4229
q2	372	277	274	274
q3	2998	2838	2925	2838
q4	2029	1702	1750	1702
q5	5614	5531	5492	5492
q6	227	138	129	129
q7	2192	1861	1821	1821
q8	3244	3402	3430	3402
q9	8706	8821	8851	8821
q10	4153	3814	3821	3814
q11	607	508	502	502
q12	801	626	613	613
q13	15915	3147	3178	3147
q14	311	274	305	274
q15	529	484	481	481
q16	483	432	428	428
q17	1801	1513	1521	1513
q18	8191	7899	7839	7839
q19	1797	1516	1510	1510
q20	2118	1858	1892	1858
q21	5130	5034	4716	4716
q22	582	547	519	519
Total cold run time: 72163 ms
Total hot run time: 55922 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172727 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0cd67e7afca414ca0628570e28d5c7f97f1be72c, data reload: false

query1	913	357	365	357
query2	6798	1872	1831	1831
query3	6637	210	222	210
query4	28319	17438	17325	17325
query5	3648	475	465	465
query6	270	170	168	168
query7	4582	298	288	288
query8	252	192	207	192
query9	8771	2474	2450	2450
query10	439	297	281	281
query11	10566	10003	9980	9980
query12	124	82	84	82
query13	1650	371	365	365
query14	9301	7676	7315	7315
query15	239	166	167	166
query16	7770	316	305	305
query17	1817	536	541	536
query18	1999	284	272	272
query19	190	148	150	148
query20	88	80	77	77
query21	199	123	127	123
query22	4250	4223	3919	3919
query23	34231	33405	33697	33405
query24	11040	2905	2872	2872
query25	601	401	385	385
query26	719	148	154	148
query27	2298	274	285	274
query28	6673	2074	2081	2074
query29	867	667	619	619
query30	249	160	156	156
query31	983	793	744	744
query32	95	53	54	53
query33	761	296	305	296
query34	1295	520	509	509
query35	694	609	599	599
query36	1156	998	1009	998
query37	148	87	88	87
query38	3089	2991	2846	2846
query39	944	865	885	865
query40	212	123	120	120
query41	49	46	43	43
query42	112	105	95	95
query43	510	482	465	465
query44	1272	714	733	714
query45	195	158	168	158
query46	1098	722	723	722
query47	1873	1751	1779	1751
query48	370	287	296	287
query49	839	393	402	393
query50	768	398	396	396
query51	6890	6781	6680	6680
query52	109	92	98	92
query53	357	284	291	284
query54	945	448	442	442
query55	74	71	72	71
query56	286	263	266	263
query57	1144	1048	1059	1048
query58	233	240	252	240
query59	2837	2704	2755	2704
query60	315	269	279	269
query61	95	93	96	93
query62	779	630	621	621
query63	324	286	285	285
query64	9291	2209	1673	1673
query65	3148	3134	3106	3106
query66	740	339	319	319
query67	15588	15202	14915	14915
query68	4460	544	540	540
query69	595	422	349	349
query70	1164	1165	1106	1106
query71	432	279	268	268
query72	7136	5315	5603	5315
query73	749	329	325	325
query74	6050	5741	5636	5636
query75	3463	2646	2709	2646
query76	2895	934	1008	934
query77	627	299	301	299
query78	9629	9289	9100	9100
query79	2171	529	525	525
query80	1769	463	467	463
query81	592	214	226	214
query82	762	132	132	132
query83	320	160	160	160
query84	272	84	84	84
query85	1727	317	311	311
query86	458	297	294	294
query87	3269	3081	3124	3081
query88	3763	2451	2448	2448
query89	474	391	376	376
query90	1773	192	187	187
query91	126	97	100	97
query92	57	51	46	46
query93	2234	505	494	494
query94	988	214	213	213
query95	407	314	307	307
query96	607	280	274	274
query97	3263	3016	3009	3009
query98	216	193	191	191
query99	1600	1308	1269	1269
Total cold run time: 279166 ms
Total hot run time: 172727 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.48 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0cd67e7afca414ca0628570e28d5c7f97f1be72c, data reload: false

query1	0.05	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.07
query5	0.49	0.49	0.47
query6	1.13	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.05	0.05
query9	0.56	0.46	0.49
query10	0.55	0.55	0.53
query11	0.15	0.12	0.11
query12	0.14	0.12	0.13
query13	0.58	0.58	0.58
query14	0.76	0.78	0.78
query15	0.84	0.80	0.80
query16	0.38	0.37	0.35
query17	1.00	1.03	1.02
query18	0.23	0.22	0.22
query19	1.89	1.81	1.73
query20	0.01	0.01	0.01
query21	15.40	0.75	0.66
query22	3.89	6.68	2.89
query23	18.29	1.46	1.20
query24	2.09	0.24	0.21
query25	0.17	0.08	0.08
query26	0.30	0.21	0.21
query27	0.45	0.24	0.23
query28	13.24	1.02	0.99
query29	12.61	3.39	3.33
query30	0.25	0.05	0.05
query31	2.87	0.39	0.39
query32	3.27	0.47	0.47
query33	2.88	2.92	2.90
query34	17.19	4.31	4.39
query35	4.42	4.46	4.38
query36	0.64	0.46	0.48
query37	0.18	0.16	0.16
query38	0.15	0.15	0.14
query39	0.05	0.03	0.04
query40	0.15	0.12	0.12
query41	0.09	0.05	0.04
query42	0.06	0.04	0.05
query43	0.04	0.04	0.03
Total cold run time: 109.49 s
Total hot run time: 31.48 s

@morrySnow morrySnow force-pushed the fix_decimal_type_tablet_prune branch from 0cd67e7 to ccb53bd Compare July 16, 2024 12:51
@zclllyybb
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39473 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ccb53bd1d9cc360391e6a4564b7ba33174cb4a8b, data reload: false

------ Round 1 ----------------------------------
q1	17626	4411	4289	4289
q2	2015	185	181	181
q3	10457	1158	1099	1099
q4	10188	766	696	696
q5	7535	2663	2578	2578
q6	217	138	133	133
q7	938	596	605	596
q8	9221	2041	2071	2041
q9	8684	6521	6560	6521
q10	8859	3772	3764	3764
q11	444	232	253	232
q12	456	219	231	219
q13	17774	2989	2973	2973
q14	276	227	234	227
q15	538	479	480	479
q16	501	384	385	384
q17	964	673	712	673
q18	8171	7504	7347	7347
q19	7739	1321	1364	1321
q20	657	305	322	305
q21	5035	3135	3254	3135
q22	347	287	280	280
Total cold run time: 118642 ms
Total hot run time: 39473 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4465	4281	4242	4242
q2	372	273	266	266
q3	3037	2914	2949	2914
q4	1998	1698	1769	1698
q5	5645	5601	5482	5482
q6	235	140	135	135
q7	2262	1948	1948	1948
q8	3311	3474	3433	3433
q9	8744	8791	8840	8791
q10	4085	3700	3844	3700
q11	600	495	511	495
q12	803	629	626	626
q13	15935	3157	3186	3157
q14	322	291	288	288
q15	541	488	508	488
q16	479	451	427	427
q17	1800	1543	1521	1521
q18	8192	7903	7856	7856
q19	3213	1655	1557	1557
q20	2180	1880	1867	1867
q21	5046	5022	4813	4813
q22	616	498	520	498
Total cold run time: 73881 ms
Total hot run time: 56202 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173398 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ccb53bd1d9cc360391e6a4564b7ba33174cb4a8b, data reload: false

query1	898	371	360	360
query2	6462	1905	1825	1825
query3	6656	210	222	210
query4	24879	17635	17272	17272
query5	3604	480	480	480
query6	273	171	175	171
query7	4575	288	280	280
query8	241	192	196	192
query9	8491	2408	2375	2375
query10	441	283	271	271
query11	12026	10069	10067	10067
query12	119	88	81	81
query13	1644	367	357	357
query14	10178	7873	7749	7749
query15	230	170	169	169
query16	7715	321	316	316
query17	1791	584	536	536
query18	1839	289	278	278
query19	202	152	154	152
query20	90	82	83	82
query21	210	129	124	124
query22	4381	4224	4065	4065
query23	33971	33822	33633	33633
query24	10661	2937	2875	2875
query25	617	431	433	431
query26	726	151	154	151
query27	2262	278	281	278
query28	6092	2080	2072	2072
query29	914	654	668	654
query30	257	149	151	149
query31	948	837	758	758
query32	92	56	56	56
query33	644	286	293	286
query34	869	508	499	499
query35	670	584	586	584
query36	1166	975	997	975
query37	149	87	79	79
query38	3018	2834	2772	2772
query39	884	810	836	810
query40	202	122	116	116
query41	48	43	45	43
query42	116	96	101	96
query43	501	469	468	468
query44	1110	715	728	715
query45	196	162	160	160
query46	1082	738	715	715
query47	1851	1783	1772	1772
query48	361	289	299	289
query49	838	401	427	401
query50	767	381	376	376
query51	6906	6798	6796	6796
query52	111	92	92	92
query53	349	281	286	281
query54	876	445	442	442
query55	74	72	72	72
query56	303	259	275	259
query57	1124	1056	1044	1044
query58	260	266	262	262
query59	2894	2700	2568	2568
query60	301	276	264	264
query61	94	91	93	91
query62	799	656	649	649
query63	323	284	285	284
query64	9192	2265	1659	1659
query65	3143	3095	3111	3095
query66	751	325	325	325
query67	15596	15275	14989	14989
query68	4594	530	539	530
query69	684	412	339	339
query70	1207	1058	1117	1058
query71	452	285	277	277
query72	8413	5610	6033	5610
query73	750	318	325	318
query74	6089	5696	5649	5649
query75	3766	2689	2724	2689
query76	2735	885	876	876
query77	662	313	289	289
query78	10598	9123	8859	8859
query79	8232	529	522	522
query80	1444	480	473	473
query81	590	219	214	214
query82	1454	142	132	132
query83	337	164	163	163
query84	271	83	88	83
query85	1397	326	350	326
query86	442	323	318	318
query87	3307	3076	3086	3076
query88	4853	2356	2362	2356
query89	499	390	374	374
query90	1973	194	191	191
query91	130	102	100	100
query92	60	51	53	51
query93	6409	506	504	504
query94	972	208	207	207
query95	405	315	314	314
query96	613	290	278	278
query97	3207	3023	3016	3016
query98	220	202	196	196
query99	1540	1267	1213	1213
Total cold run time: 289358 ms
Total hot run time: 173398 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.11 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ccb53bd1d9cc360391e6a4564b7ba33174cb4a8b, data reload: false

query1	0.04	0.03	0.02
query2	0.08	0.03	0.04
query3	0.22	0.06	0.05
query4	1.66	0.08	0.08
query5	0.49	0.49	0.48
query6	1.12	0.72	0.73
query7	0.03	0.01	0.01
query8	0.05	0.04	0.04
query9	0.57	0.48	0.47
query10	0.53	0.54	0.54
query11	0.15	0.12	0.11
query12	0.16	0.12	0.13
query13	0.60	0.59	0.58
query14	0.77	0.77	0.79
query15	0.87	0.82	0.81
query16	0.37	0.37	0.36
query17	1.00	1.04	1.04
query18	0.24	0.22	0.21
query19	1.76	1.68	1.70
query20	0.01	0.02	0.02
query21	15.38	0.75	0.66
query22	4.51	6.42	2.49
query23	18.31	1.34	1.25
query24	2.16	0.22	0.22
query25	0.15	0.10	0.09
query26	0.32	0.21	0.21
query27	0.46	0.23	0.23
query28	13.26	1.02	1.00
query29	12.61	3.32	3.23
query30	0.25	0.07	0.06
query31	2.89	0.40	0.40
query32	3.22	0.48	0.47
query33	2.90	2.91	2.95
query34	17.00	4.32	4.39
query35	4.38	4.39	4.38
query36	0.64	0.47	0.46
query37	0.19	0.16	0.15
query38	0.15	0.16	0.15
query39	0.04	0.04	0.04
query40	0.16	0.12	0.12
query41	0.10	0.05	0.04
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.91 s
Total hot run time: 31.11 s

@morrySnow
Copy link
Contributor Author

run cloud_p1

@morrySnow
Copy link
Contributor Author

run could_p1

@morrySnow morrySnow requested a review from starocean999 July 17, 2024 09:58
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 17, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@morrySnow morrySnow merged commit 8bb3ff9 into apache:master Jul 17, 2024
27 of 29 checks passed
@morrySnow morrySnow deleted the fix_decimal_type_tablet_prune branch July 17, 2024 10:14
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request Jul 17, 2024
@morrySnow
[fix](Nereids) tablet prune wrong when decimal value scale is nagtive (…
b09d16b 
…apache#37889)

pick from master apache#37889

we use unscaled value of BigDecimal in tablet prune. So we need to
ensure BigDecimal's precision and scale is same with the literal
contains it.
@morrySnow morrySnow mentioned this pull request Jul 17, 2024
Merged
morrySnow added a commit that referenced this pull request Jul 17, 2024
@morrySnow
[fix](Nereids) tablet prune wrong when decimal value scale is nagtive (…
78d51ca 
…#37889) (#38013)

pick from master #37889

we use unscaled value of BigDecimal in tablet prune. So we need to
ensure BigDecimal's precision and scale is same with the literal
contains it.
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@morrySnow @dataroaring
[fix](Nereids) tablet prune wrong when decimal value scale is nagtive (…
46f5e00 
…#37889)

we use unscaled value of BigDecimal in tablet prune. So we need to
ensure BigDecimal's precision and scale is same with the literal
contains it.
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request Jul 24, 2024
@morrySnow
[fix](Nereids) tablet prune wrong when decimal value scale is nagtive (…
0e02907 
…apache#37889)

pick from master apache#37889

we use unscaled value of BigDecimal in tablet prune. So we need to
ensure BigDecimal's precision and scale is same with the literal
contains it.
@morrySnow morrySnow mentioned this pull request Jul 24, 2024
Merged
morrySnow added a commit that referenced this pull request Jul 24, 2024
@morrySnow
[fix](Nereids) tablet prune wrong when decimal value scale is nagtive (…
97d2d82 
…#37889) (#38280)

pick from master #37889
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@morrySnow
[fix](Nereids) tablet prune wrong when decimal value scale is nagtive (…
b0dbb0d 
…apache#37889) (apache#38280)

pick from master apache#37889
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@924060929 924060929 924060929 approved these changes

@starocean999 starocean999 starocean999 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.14-merged dev/2.1.5-merged dev/3.0.1-merged p0_w reviewed usercase Important user case type label
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@morrySnow @doris-robot @zclllyybb @924060929 @starocean999 @dataroaring