Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](ES Catalog)Add array types support in esquery function #36936

Merged
merged 3 commits into from
Jul 1, 2024
Merged

[fix](ES Catalog)Add array types support in esquery function #36936

merged 3 commits into from
Jul 1, 2024

Conversation

qidaye
Copy link
Contributor

@qidaye qidaye commented Jun 27, 2024

Proposed changes

Support array types in esquery function, and add some tests.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@qidaye
Copy link
Contributor Author

qidaye commented Jun 27, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39934 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1c6ef37b34aaaaa394d6a38ac87a56cdbc071b5e, data reload: false

------ Round 1 ----------------------------------
q1	18115	5076	4524	4524
q2	2801	197	193	193
q3	10970	1248	1131	1131
q4	10697	835	777	777
q5	7728	2661	2649	2649
q6	227	138	135	135
q7	966	608	624	608
q8	9428	2099	2083	2083
q9	8830	6539	6462	6462
q10	8884	3737	3689	3689
q11	486	238	237	237
q12	397	233	223	223
q13	18693	2956	2949	2949
q14	266	235	224	224
q15	522	478	482	478
q16	514	374	370	370
q17	960	684	700	684
q18	8015	7439	7396	7396
q19	5492	1520	1414	1414
q20	674	323	328	323
q21	4903	3049	4013	3049
q22	411	336	339	336
Total cold run time: 119979 ms
Total hot run time: 39934 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4417	4213	4200	4200
q2	368	287	269	269
q3	2967	2760	2773	2760
q4	1889	1593	1614	1593
q5	5233	5266	5256	5256
q6	215	128	125	125
q7	2114	1737	1759	1737
q8	3164	3328	3302	3302
q9	8364	8259	8296	8259
q10	3889	3674	3716	3674
q11	578	485	492	485
q12	807	604	610	604
q13	16338	2987	2975	2975
q14	287	259	265	259
q15	521	470	478	470
q16	464	408	433	408
q17	1762	1481	1455	1455
q18	7720	7431	7514	7431
q19	1671	1615	1557	1557
q20	1968	1795	1773	1773
q21	9749	4854	4766	4766
q22	606	545	520	520
Total cold run time: 75091 ms
Total hot run time: 53878 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 169537 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1c6ef37b34aaaaa394d6a38ac87a56cdbc071b5e, data reload: false

query1	930	381	379	379
query2	6453	2439	2318	2318
query3	6652	208	214	208
query4	19091	17400	17328	17328
query5	4238	483	485	483
query6	261	181	158	158
query7	4600	305	296	296
query8	321	269	280	269
query9	8417	2395	2404	2395
query10	612	303	284	284
query11	10518	10031	9976	9976
query12	127	89	87	87
query13	1646	378	383	378
query14	10263	6791	6335	6335
query15	223	182	187	182
query16	7848	274	281	274
query17	1910	559	543	543
query18	1958	284	285	284
query19	205	153	158	153
query20	87	83	80	80
query21	232	127	131	127
query22	4344	4110	4056	4056
query23	33737	33069	32905	32905
query24	12103	2901	2842	2842
query25	679	377	382	377
query26	1805	155	156	155
query27	3002	307	311	307
query28	7692	2086	2078	2078
query29	1163	633	614	614
query30	289	153	152	152
query31	975	783	746	746
query32	90	54	55	54
query33	805	304	297	297
query34	927	465	466	465
query35	727	631	628	628
query36	1105	919	945	919
query37	300	71	70	70
query38	2869	2761	2762	2761
query39	858	795	814	795
query40	284	128	132	128
query41	57	54	53	53
query42	121	98	98	98
query43	549	497	528	497
query44	1220	730	729	729
query45	196	168	168	168
query46	1079	747	710	710
query47	1869	1763	1787	1763
query48	375	295	294	294
query49	1212	428	418	418
query50	759	378	380	378
query51	6750	6785	6865	6785
query52	110	93	99	93
query53	354	290	287	287
query54	939	451	445	445
query55	77	76	75	75
query56	301	276	276	276
query57	1147	1050	1053	1050
query58	382	237	238	237
query59	3495	3008	3073	3008
query60	316	279	270	270
query61	90	92	91	91
query62	660	456	450	450
query63	310	284	289	284
query64	9843	2252	1758	1758
query65	3200	3082	3112	3082
query66	1370	332	325	325
query67	16071	14975	14965	14965
query68	8878	558	567	558
query69	708	505	409	409
query70	1393	1160	1084	1084
query71	537	273	267	267
query72	8875	2789	2553	2553
query73	2217	331	324	324
query74	5996	5501	5460	5460
query75	5064	2642	2638	2638
query76	5026	959	968	959
query77	800	359	295	295
query78	10311	9992	9862	9862
query79	7848	506	509	506
query80	1001	476	466	466
query81	563	219	214	214
query82	737	104	102	102
query83	353	170	171	170
query84	274	88	87	87
query85	1369	276	266	266
query86	404	309	305	305
query87	3259	3135	3095	3095
query88	4170	2393	2371	2371
query89	539	381	389	381
query90	1950	187	187	187
query91	128	105	99	99
query92	60	49	49	49
query93	5935	520	510	510
query94	1201	190	185	185
query95	407	323	318	318
query96	613	267	272	267
query97	3222	3031	3075	3031
query98	217	201	191	191
query99	1211	853	857	853
Total cold run time: 298464 ms
Total hot run time: 169537 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.91 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1c6ef37b34aaaaa394d6a38ac87a56cdbc071b5e, data reload: false

query1	0.04	0.04	0.02
query2	0.09	0.04	0.04
query3	0.22	0.05	0.06
query4	1.64	0.11	0.10
query5	0.50	0.48	0.47
query6	1.14	0.72	0.73
query7	0.02	0.01	0.01
query8	0.05	0.05	0.05
query9	0.55	0.50	0.50
query10	0.54	0.54	0.55
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.59	0.59	0.60
query14	0.79	0.79	0.76
query15	0.84	0.81	0.82
query16	0.38	0.35	0.36
query17	0.96	0.99	0.99
query18	0.21	0.27	0.24
query19	1.75	1.73	1.74
query20	0.01	0.01	0.01
query21	15.44	0.76	0.66
query22	4.19	6.91	2.26
query23	18.28	1.33	1.25
query24	2.08	0.21	0.24
query25	0.16	0.08	0.09
query26	0.27	0.17	0.18
query27	0.08	0.08	0.08
query28	13.25	1.01	0.99
query29	12.63	3.32	3.29
query30	0.25	0.06	0.07
query31	2.84	0.40	0.39
query32	3.27	0.47	0.48
query33	2.90	2.91	2.91
query34	17.11	4.39	4.39
query35	4.52	4.56	4.46
query36	0.66	0.45	0.47
query37	0.18	0.16	0.14
query38	0.15	0.14	0.15
query39	0.04	0.03	0.04
query40	0.17	0.15	0.15
query41	0.09	0.05	0.05
query42	0.05	0.05	0.04
query43	0.04	0.03	0.04
Total cold run time: 109.28 s
Total hot run time: 30.91 s

@qidaye
Copy link
Contributor Author

qidaye commented Jun 28, 2024

run external

@qidaye
Copy link
Contributor Author

qidaye commented Jun 28, 2024

run feut

@qidaye
Copy link
Contributor Author

qidaye commented Jun 28, 2024

run p0

@qidaye
Copy link
Contributor Author

qidaye commented Jun 28, 2024

run feut

@qidaye
Copy link
Contributor Author

qidaye commented Jun 28, 2024

run p0

1 similar comment
@qidaye
Copy link
Contributor Author

qidaye commented Jun 28, 2024

run p0

@qidaye
Copy link
Contributor Author

qidaye commented Jun 28, 2024

run buildall

xiaokang
xiaokang approved these changes Jun 28, 2024
View reviewed changes
Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jun 28, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 40245 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 054d273ecc59170cfe0c6b490a13814c37d169cd, data reload: false

------ Round 1 ----------------------------------
q1	17629	4335	4277	4277
q2	2028	190	197	190
q3	10589	1180	1084	1084
q4	10213	891	860	860
q5	7512	2702	2651	2651
q6	217	135	136	135
q7	941	600	616	600
q8	9214	2087	2080	2080
q9	8808	6592	6528	6528
q10	9090	3735	3769	3735
q11	488	246	243	243
q12	428	244	235	235
q13	17958	3030	3060	3030
q14	286	218	226	218
q15	538	495	486	486
q16	526	403	394	394
q17	947	677	677	677
q18	7948	7588	7430	7430
q19	6976	1384	1597	1384
q20	713	369	345	345
q21	5065	3320	4010	3320
q22	404	346	343	343
Total cold run time: 118518 ms
Total hot run time: 40245 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4552	4434	4363	4363
q2	384	278	285	278
q3	3292	3113	3153	3113
q4	2064	1810	1801	1801
q5	5550	5557	5515	5515
q6	231	135	135	135
q7	2252	1858	1831	1831
q8	3296	3441	3434	3434
q9	8629	8805	8778	8778
q10	4105	3781	3835	3781
q11	598	491	518	491
q12	844	640	603	603
q13	16302	3111	3159	3111
q14	302	283	257	257
q15	535	481	481	481
q16	504	414	451	414
q17	1796	1514	1514	1514
q18	8235	7834	7777	7777
q19	1818	1639	1602	1602
q20	3331	1851	1867	1851
q21	9886	4989	4815	4815
q22	620	552	553	552
Total cold run time: 79126 ms
Total hot run time: 56497 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174284 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 054d273ecc59170cfe0c6b490a13814c37d169cd, data reload: false

query1	929	380	373	373
query2	6445	2397	2286	2286
query3	6634	207	212	207
query4	18698	17697	17331	17331
query5	3621	489	480	480
query6	258	166	172	166
query7	4579	293	296	293
query8	297	266	283	266
query9	8749	2459	2443	2443
query10	572	320	278	278
query11	10535	9920	10019	9920
query12	119	92	80	80
query13	1637	368	379	368
query14	10226	6593	7546	6593
query15	223	181	183	181
query16	7603	265	265	265
query17	1470	551	523	523
query18	1861	267	267	267
query19	192	145	154	145
query20	86	82	85	82
query21	204	129	121	121
query22	4436	4023	4071	4023
query23	33657	33500	33770	33500
query24	11337	2896	2894	2894
query25	612	392	368	368
query26	1078	159	165	159
query27	2416	323	325	323
query28	6999	2180	2174	2174
query29	883	629	652	629
query30	260	168	157	157
query31	961	790	776	776
query32	93	58	62	58
query33	756	290	288	288
query34	1054	474	501	474
query35	752	621	610	610
query36	1128	995	989	989
query37	149	72	75	72
query38	3003	2812	2910	2812
query39	879	826	823	823
query40	220	135	132	132
query41	53	51	53	51
query42	109	107	99	99
query43	596	545	544	544
query44	1265	749	747	747
query45	192	167	167	167
query46	1080	721	717	717
query47	1819	1767	1776	1767
query48	370	299	306	299
query49	847	404	406	404
query50	772	382	391	382
query51	6843	6891	6870	6870
query52	102	95	97	95
query53	366	302	292	292
query54	915	450	454	450
query55	74	74	77	74
query56	302	274	288	274
query57	1101	1052	1056	1052
query58	251	251	243	243
query59	3474	3003	3111	3003
query60	324	306	300	300
query61	113	113	111	111
query62	614	436	458	436
query63	318	291	290	290
query64	8752	2392	1862	1862
query65	3267	3125	3115	3115
query66	792	334	417	334
query67	15627	14948	14930	14930
query68	5484	548	555	548
query69	658	447	407	407
query70	1154	1068	1100	1068
query71	480	276	280	276
query72	9226	5894	5880	5880
query73	796	327	323	323
query74	5814	5524	5532	5524
query75	4636	2668	2669	2668
query76	3421	934	889	889
query77	692	319	314	314
query78	10524	9903	9879	9879
query79	4284	527	537	527
query80	1036	476	488	476
query81	545	232	225	225
query82	762	107	104	104
query83	366	181	173	173
query84	274	89	83	83
query85	1320	293	272	272
query86	439	318	308	308
query87	3352	3112	3089	3089
query88	4616	2370	2366	2366
query89	493	393	390	390
query90	1831	190	190	190
query91	129	99	103	99
query92	62	55	50	50
query93	4953	528	526	526
query94	1152	186	190	186
query95	415	314	323	314
query96	619	276	270	270
query97	3232	3086	3015	3015
query98	217	202	188	188
query99	1145	820	834	820
Total cold run time: 280076 ms
Total hot run time: 174284 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.37 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 054d273ecc59170cfe0c6b490a13814c37d169cd, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.08	0.08
query5	0.50	0.50	0.49
query6	1.16	0.73	0.72
query7	0.02	0.02	0.01
query8	0.05	0.05	0.04
query9	0.55	0.51	0.50
query10	0.54	0.54	0.54
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.59	0.58	0.60
query14	0.77	0.79	0.78
query15	0.83	0.81	0.80
query16	0.36	0.37	0.37
query17	1.03	1.04	0.94
query18	0.22	0.23	0.24
query19	1.75	1.68	1.76
query20	0.01	0.01	0.00
query21	15.68	0.78	0.67
query22	4.18	6.61	2.61
query23	18.24	1.35	1.29
query24	2.16	0.22	0.21
query25	0.16	0.08	0.08
query26	0.26	0.17	0.18
query27	0.08	0.08	0.08
query28	13.23	1.01	0.99
query29	12.66	3.34	3.29
query30	0.26	0.07	0.05
query31	2.86	0.39	0.39
query32	3.27	0.47	0.47
query33	2.90	2.96	2.97
query34	16.82	4.44	4.42
query35	4.52	4.49	4.53
query36	0.66	0.48	0.49
query37	0.18	0.15	0.16
query38	0.16	0.15	0.15
query39	0.04	0.03	0.04
query40	0.17	0.14	0.14
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.39 s
Total hot run time: 31.37 s

@qidaye qidaye merged commit 64a78a0 into apache:master Jul 1, 2024
26 of 29 checks passed
@qidaye qidaye deleted the support_array_in_esquery branch July 1, 2024 02:47
qidaye added a commit to qidaye/incubator-doris that referenced this pull request Jul 1, 2024
@qidaye
[fix](ES Catalog)Add array types support in esquery function (apache#…
e4be0db 
…36936)

Support array types in `esquery` function, and add some tests.
@qidaye qidaye mentioned this pull request Jul 1, 2024
Merged
qidaye added a commit to qidaye/incubator-doris that referenced this pull request Jul 1, 2024
@qidaye
[fix](ES Catalog)Add array types support in esquery function (apache#…
bf132ed 
…36936)

Support array types in `esquery` function, and add some tests.
@qidaye qidaye mentioned this pull request Jul 1, 2024
Merged
dataroaring pushed a commit that referenced this pull request Jul 2, 2024
@qidaye @dataroaring
[fix](ES Catalog)Add array types support in esquery function (#36936)
ec1422a 
Support array types in `esquery` function, and add some tests.
morningman pushed a commit that referenced this pull request Jul 2, 2024
@qidaye
[fix](ES Catalog)Add array types support in esquery function (#36936) (…
b63e6ac 
…#37054)

backport #36936
morningman pushed a commit that referenced this pull request Jul 2, 2024
@qidaye
[fix](ES Catalog)Add array types support in esquery function (#36936) (…
fd754d7 
…#37056)

backport #36936
@yiguolei yiguolei mentioned this pull request Jul 19, 2024
Open
1 task
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@qidaye
[fix](ES Catalog)Add array types support in esquery function (apache#…
3968549 
…36936) (apache#37056)

backport apache#36936
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xiaokang xiaokang xiaokang approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.13-merged dev/2.1.5-merged dev/3.0.0-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@qidaye @doris-robot @xiaokang @morrySnow @morningman @dataroaring