[Improve](array-func)improve array union support multi params #24327

amorynan · 2023-09-13T13:34:28Z

Proposed changes

Issue Number: close #xxx
Now we just support array union with two arrays

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

amorynan · 2023-09-13T13:39:01Z

run buildall

github-actions · 2023-09-13T13:42:08Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-13T14:07:43Z

TeamCity be ut coverage result:
Function Coverage: 36.96% (7968/21556)
Line Coverage: 28.93% (63837/220662)
Region Coverage: 27.87% (33147/118919)
Branch Coverage: 24.43% (16999/69592)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7d4380de09b436d92f05c0e45bd7ed3e3617387a_7d4380de09b436d92f05c0e45bd7ed3e3617387a/report/index.html

doris-robot · 2023-09-13T14:19:58Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.7 seconds
stream load tsv: 594 seconds loaded 74807831229 Bytes, about 120 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162290689 Bytes

github-actions · 2023-09-14T03:53:53Z

clang-tidy review says "All clean, LGTM! 👍"

amorynan · 2023-09-14T03:54:39Z

run buildall

doris-robot · 2023-09-14T04:12:13Z

TeamCity be ut coverage result:
Function Coverage: 36.96% (7968/21558)
Line Coverage: 28.93% (63848/220721)
Region Coverage: 27.87% (33154/118958)
Branch Coverage: 24.43% (17006/69614)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5caff873b333586c520cb25d9e03c1539bbe9518_5caff873b333586c520cb25d9e03c1539bbe9518/report/index.html

doris-robot · 2023-09-14T04:33:55Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.08 seconds
stream load tsv: 612 seconds loaded 74807831229 Bytes, about 116 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162555795 Bytes

morrySnow · 2023-09-14T06:30:48Z

gensrc/script/doris_builtins_functions.py

-        [['array_union'], 'ARRAY_VARCHAR',   ['ARRAY_VARCHAR', 'ARRAY_VARCHAR'], ''],
-        [['array_union'], 'ARRAY_STRING',    ['ARRAY_STRING', 'ARRAY_STRING'], ''],
+        [['array_union'], 'ARRAY_BOOLEAN',  ['ARRAY_BOOLEAN', 'ARRAY_BOOLEAN', '...'],  ''],
+        [['array_union'], 'ARRAY_TINYINT',  ['ARRAY_TINYINT', 'ARRAY_TINYINT', '...'],  ''],


should update org.apache.doris.nereids.trees.expressions.functions.scalar.ArrayUnion to let it support multi param in Nereids too

cambyzju · 2023-09-14T06:36:26Z

be/src/vec/functions/array/function_array_map.h

-                if (entry.get_mapped() == params.size()) {
+                if constexpr (operation == MapOperation::INTERSECT) {
+                    if (entry.get_mapped() == params.size()) {
+                        ++dst_off;


same code as line 99~103

I will make micro define

be/src/vec/functions/array/function_array_map.h

cambyzju · 2023-09-14T06:40:11Z

be/src/vec/functions/array/function_array_union.cpp

-        if (!set.find(elem)) {
-            set.insert(elem);
-            return true;
+    void apply(Map& map, const int arg_idx, const int row_idx,


row_idx better use 64bit instead int.

const int == int?

maybe size_t is better ?

xy720

Why calculate multi array union need a map structure? Set is not enough?

amorynan · 2023-09-14T09:52:34Z

Why calculate multi array union need a map structure? Set is not enough?

because MapImpl can apply const ColumnArrayExecutionDatas params, but SetImpl just apply one ColumnArrayExecutionData

github-actions · 2023-09-14T10:00:38Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-09-16T00:37:23Z

clang-tidy review says "All clean, LGTM! 👍"

amorynan · 2023-09-16T01:10:50Z

run buildall

doris-robot · 2023-09-16T01:34:08Z

TeamCity be ut coverage result:
Function Coverage: 36.84% (7978/21653)
Line Coverage: 28.86% (64003/221770)
Region Coverage: 27.77% (33194/119515)
Branch Coverage: 24.38% (17040/69900)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4b774ea9e7d3fea3bf616e43c728129500a66705_4b774ea9e7d3fea3bf616e43c728129500a66705/report/index.html

doris-robot · 2023-09-16T01:53:21Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.27 seconds
stream load tsv: 600 seconds loaded 74807831229 Bytes, about 118 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162510316 Bytes

github-actions · 2023-09-18T02:41:03Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-09-18T02:41:05Z

PR approved by anyone and no changes requested.

github-actions · 2023-09-18T09:52:12Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-18T10:04:09Z

TeamCity be ut coverage result:
Function Coverage: 36.82% (7979/21672)
Line Coverage: 28.84% (64018/221948)
Region Coverage: 27.73% (33224/119799)
Branch Coverage: 24.36% (17029/69916)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2234a7a31488e4c32dff3f1923a4102c20d26d3f_2234a7a31488e4c32dff3f1923a4102c20d26d3f/report/index.html

doris-robot · 2023-09-18T10:17:06Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.19 seconds
stream load tsv: 596 seconds loaded 74807831229 Bytes, about 119 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162185221 Bytes

xiaokang · 2023-09-18T12:10:41Z

run buildall

github-actions · 2023-09-18T12:17:21Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-18T12:27:44Z

TeamCity be ut coverage result:
Function Coverage: 36.83% (7981/21672)
Line Coverage: 28.86% (64056/221948)
Region Coverage: 27.75% (33247/119799)
Branch Coverage: 24.38% (17043/69916)
Coverage Report: http://coverage.selectdb-in.cc/coverage/867f06795d2ca7ddb186352839f4ebad7c330ca4_867f06795d2ca7ddb186352839f4ebad7c330ca4/report/index.html

doris-robot · 2023-09-18T12:45:02Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.71 seconds
stream load tsv: 599 seconds loaded 74807831229 Bytes, about 119 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17162360025 Bytes

amorynan · 2023-09-19T09:34:20Z

run p0

amorynan · 2023-09-19T09:34:32Z

run p1

amorynan · 2023-09-20T06:04:59Z

@shuke987

github-actions · 2023-09-20T06:20:38Z

PR approved by at least one committer and no changes requested.

xiaokang

LGTM

…#24327)

…#24327)" This reverts commit 6090063.

improve array union support multi params

7d4380d

fix null init:

5caff87

xiaokang added dev/2.0.2 usercase Important user case type label labels Sep 14, 2023

morrySnow reviewed Sep 14, 2023

View reviewed changes

cambyzju reviewed Sep 14, 2023

View reviewed changes

be/src/vec/functions/array/function_array_map.h Outdated Show resolved Hide resolved

cambyzju reviewed Sep 14, 2023

View reviewed changes

xy720 reviewed Sep 14, 2023

View reviewed changes

Merge branch 'master' into improve-array-union-with-multi-param

99570af

update

4b774ea

morrySnow previously approved these changes Sep 18, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 18, 2023

github-actions bot added the reviewed label Sep 18, 2023

amorynan requested a review from morrySnow September 18, 2023 10:01

Merge branch 'master' into improve-array-union-with-multi-param

867f067

morrySnow approved these changes Sep 20, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 20, 2023

xiaokang approved these changes Sep 20, 2023

View reviewed changes

xiaokang merged commit e9435c1 into apache:master Sep 20, 2023

JingDas pushed a commit to JingDas/doris that referenced this pull request Sep 20, 2023

[Improve](array-func)improve array union support multi params (apache…

bf81ad3

…#24327)

xiaokang added merge_conflict dev/2.0.2-merged and removed dev/2.0.2 labels Sep 20, 2023

xiaokang pushed a commit that referenced this pull request Sep 20, 2023

[Improve](array-func)improve array union support multi params (#24327)

6090063

xiaokang added dev/2.0.2 dev/2.0.2-merged and removed dev/2.0.2-merged dev/2.0.2 labels Sep 20, 2023

xiaokang added a commit that referenced this pull request Sep 20, 2023

Revert "[Improve](array-func)improve array union support multi params (…

cf376a5

…#24327)" This reverts commit 6090063.

xiaokang pushed a commit that referenced this pull request Sep 20, 2023

[Improve](array-func)improve array union support multi params (#24327)

a7715ac

xiaokang mentioned this pull request Sep 30, 2023

Release Note 2.0.2 #25011

Open

xiaokang mentioned this pull request Dec 3, 2023

Release Note 2.0.3 #27909

Open

[Improve](array-func)improve array union support multi params #24327

[Improve](array-func)improve array union support multi params #24327

Conversation

amorynan commented Sep 13, 2023

Proposed changes

Further comments

amorynan commented Sep 13, 2023

github-actions bot commented Sep 13, 2023

doris-robot commented Sep 13, 2023

doris-robot commented Sep 13, 2023

github-actions bot commented Sep 14, 2023

amorynan commented Sep 14, 2023

doris-robot commented Sep 14, 2023

doris-robot commented Sep 14, 2023

morrySnow Sep 14, 2023

Choose a reason for hiding this comment

cambyzju Sep 14, 2023

Choose a reason for hiding this comment

amorynan Sep 15, 2023

Choose a reason for hiding this comment

cambyzju Sep 14, 2023

Choose a reason for hiding this comment

amorynan Sep 15, 2023

Choose a reason for hiding this comment

amorynan Sep 16, 2023

Choose a reason for hiding this comment

xy720 left a comment

Choose a reason for hiding this comment

amorynan commented Sep 14, 2023

github-actions bot commented Sep 14, 2023

github-actions bot commented Sep 16, 2023

amorynan commented Sep 16, 2023

doris-robot commented Sep 16, 2023

doris-robot commented Sep 16, 2023

github-actions bot commented Sep 18, 2023

github-actions bot commented Sep 18, 2023

github-actions bot commented Sep 18, 2023

doris-robot commented Sep 18, 2023

doris-robot commented Sep 18, 2023

xiaokang commented Sep 18, 2023

github-actions bot commented Sep 18, 2023

doris-robot commented Sep 18, 2023

doris-robot commented Sep 18, 2023

amorynan commented Sep 19, 2023

amorynan commented Sep 19, 2023

amorynan commented Sep 20, 2023

github-actions bot commented Sep 20, 2023

xiaokang left a comment

Choose a reason for hiding this comment