[enhancement](fix)change ordinary type null value is \N,complex type null value is null #24207

hubgeter · 2023-09-11T17:04:02Z

Proposed changes

For null values in ordinary types, we use \N to represent them;
for null values in nested types, we use null to represent them, just like the json format.

example:
If you have three nullable columns
a : int, b : string, c : map<string,int>
data:

\N,hello world,\N
1,\N,{"cmake":2,"null":11}
9,"\N",{"\N":null,null:0}
\N,"null",{null:null}
null,null,null

if you set trim_double_quotes = true
you will get :

NULL,hello world,NULL
1,NULL,{"cmake":2,"null":11}
9,\N,{"\N":NULL,NULL:0}
NULL,null,{NULL:NULL}
NULL,null,NULL

if you set trim_double_quotes = false
you will get :

NULL,hello world,NULL
1,\N,{"cmake":2,"null":11}
9,"\N",{"\N":NULL,NULL:0}
NULL,"null",{NULL:NULL}
NULL,null,NULL

in csv(text) for normal type: we only recognize \N for null , so
for not char family type, like int, if we put null literal ,
it will parse fail, and make result null，not just because it equals \N.
for char family type, like string, if we put null literal, it will parse success,
and "null" literal will be stored in doris.

For strings in the json complex type, we remove double quotes by default.

Because when querying complex types, such as selecting complexColumn from table,
we will add double quotes to the strings in the complex type.

For the map<string,int> column, insert { "abc" : 1, "hello",2 }.
If you do not remove the double quotes, it will display {""abc"":1,""hello"": 2 },
remove the double quotes to display { "abc" : 1, "hello",2 }.

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

hubgeter · 2023-09-11T17:04:18Z

run buildall

github-actions · 2023-09-11T17:10:32Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-11T18:27:23Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.55 seconds
stream load tsv: 583 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162358405 Bytes

hubgeter · 2023-09-12T03:22:16Z

run buildall

github-actions · 2023-09-12T03:27:39Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-12T04:11:36Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.25 seconds
stream load tsv: 583 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162249491 Bytes

doris-robot · 2023-09-12T04:15:39Z

TeamCity be ut coverage result:
Function Coverage: 36.92% (7917/21443)
Line Coverage: 28.96% (63647/219753)
Region Coverage: 27.88% (33021/118438)
Branch Coverage: 24.46% (16961/69334)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9039fff5c5de53c6f32de6f0ec7f5dc1cd4c1962_9039fff5c5de53c6f32de6f0ec7f5dc1cd4c1962/report/index.html

morningman

LGTM
But please describe more in your PR comment about how to handle null in text

github-actions · 2023-09-12T16:28:54Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-09-12T16:28:57Z

PR approved by anyone and no changes requested.

morningman · 2023-09-13T03:52:47Z

be/src/vec/data_types/serde/data_type_nullable_serde.cpp

    auto& null_column = assert_cast<ColumnNullable&>(column);
    // TODO(Amory) make null literal configurable

    if (!(options.converted_from_string && slice.trim_quote())) {
        //for map<string,string> type : {"abc","NULL"} , the NULL is string , instead of null values
-        if (slice.size == 4 && slice[0] == 'N' && slice[1] == 'U' && slice[2] == 'L' &&
-            slice[3] == 'L') {
+        if (nesting_level >= 2 && slice.size == 4 && slice[0] == 'n' && slice[1] == 'u' &&


Add comment to describe the logic here. Better give some example

…null value is null

github-actions · 2023-09-14T08:11:07Z

clang-tidy review says "All clean, LGTM! 👍"

hubgeter · 2023-09-14T09:48:54Z

run buildall

github-actions · 2023-09-14T09:54:39Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-14T10:40:53Z

TeamCity be ut coverage result:
Function Coverage: 36.94% (7970/21578)
Line Coverage: 28.90% (63879/221008)
Region Coverage: 27.83% (33152/119111)
Branch Coverage: 24.39% (17000/69706)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8bbd14c6636ff01dca07d4f396006b646d6ea6dd_8bbd14c6636ff01dca07d4f396006b646d6ea6dd/report/index.html

doris-robot · 2023-09-14T11:10:27Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.96 seconds
stream load tsv: 599 seconds loaded 74807831229 Bytes, about 119 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162423649 Bytes

amorynan · 2023-09-15T06:26:44Z

be/src/vec/data_types/serde/data_type_nullable_serde.cpp

+         * for null values in nested types, we use null to represent them, just like the json format.
+         *
+         * example:
+         * If you have three nullable columns


make a note: null -> int -> NULL | null -> char family -> "null"
in csv(text) for normal type: we only recognize \N for null
so
for not char family type, like int, if we put null literal , it will parse fail, and make result null，not just because it equals \N
for char family type, like string, if we put null literal, it will parse success, and "null" literal will be stored in doris

hubgeter · 2023-09-15T17:27:13Z

run buildall

github-actions · 2023-09-15T17:33:53Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-15T18:18:52Z

TeamCity be ut coverage result:
Function Coverage: 36.84% (7977/21654)
Line Coverage: 28.86% (64003/221769)
Region Coverage: 27.78% (33196/119502)
Branch Coverage: 24.37% (17031/69888)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c6d7df96fce2b1079cee8d892530c5f45c1f80d8_c6d7df96fce2b1079cee8d892530c5f45c1f80d8/report/index.html

doris-robot · 2023-09-15T18:40:21Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.75 seconds
stream load tsv: 598 seconds loaded 74807831229 Bytes, about 119 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162265867 Bytes

hubgeter · 2023-09-16T02:16:18Z

run buildall

github-actions · 2023-09-16T02:22:26Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-16T02:44:23Z

TeamCity be ut coverage result:
Function Coverage: 36.85% (7979/21654)
Line Coverage: 28.87% (64023/221769)
Region Coverage: 27.79% (33208/119502)
Branch Coverage: 24.37% (17031/69888)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e28841e1421bce02d3bc3ae8cfe3c2b59d0d9891_e28841e1421bce02d3bc3ae8cfe3c2b59d0d9891/report/index.html

doris-robot · 2023-09-16T02:59:43Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 51.84 seconds
stream load tsv: 599 seconds loaded 74807831229 Bytes, about 119 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162200224 Bytes

amorynan

LGTM

morningman

LGTM

github-actions · 2023-09-16T11:07:18Z

PR approved by at least one committer and no changes requested.

…null value is null (#24207)

hubgeter force-pushed the fix_csv_null_value branch from 90d64b9 to 9039fff Compare September 12, 2023 03:20

morningman previously approved these changes Sep 12, 2023

View reviewed changes

morningman added the kind/behavior-changed label Sep 12, 2023

github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Sep 12, 2023

morningman reviewed Sep 13, 2023

View reviewed changes

hubgeter added 2 commits September 14, 2023 16:03

[enhancement](fix)change ordinary type null value is \N,complex type …

4395a67

…null value is null

fix data_type_decimal_serde.h.orig

0a5ba7e

hubgeter dismissed morningman’s stale review via 0a5ba7e September 14, 2023 08:03

hubgeter force-pushed the fix_csv_null_value branch from 9039fff to 0a5ba7e Compare September 14, 2023 08:03

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Sep 14, 2023

fix format

8bbd14c

amorynan reviewed Sep 15, 2023

View reviewed changes

fix complex string double quote

c6d7df9

fix regression

e28841e

amorynan approved these changes Sep 16, 2023

View reviewed changes

morningman approved these changes Sep 16, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 16, 2023

xiaokang merged commit 4b5cea1 into apache:master Sep 16, 2023

xiaokang added dev/2.0.2 p0_b merge_conflict dev/2.0.2-merged and removed dev/2.0.2 labels Sep 16, 2023

xiaokang pushed a commit that referenced this pull request Sep 16, 2023

[enhancement](fix)change ordinary type null value is \N,complex type …

9f1e13b

…null value is null (#24207)

xiaokang mentioned this pull request Sep 28, 2023

Release Note 2.0.2 #25011

Open

xiaokang mentioned this pull request Dec 3, 2023

Release Note 2.0.3 #27909

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[enhancement](fix)change ordinary type null value is \N,complex type null value is null #24207

[enhancement](fix)change ordinary type null value is \N,complex type null value is null #24207

hubgeter commented Sep 11, 2023 •

edited

Loading

hubgeter commented Sep 11, 2023

github-actions bot commented Sep 11, 2023

doris-robot commented Sep 11, 2023

hubgeter commented Sep 12, 2023

github-actions bot commented Sep 12, 2023

doris-robot commented Sep 12, 2023

doris-robot commented Sep 12, 2023

morningman left a comment •

edited

Loading

github-actions bot commented Sep 12, 2023

github-actions bot commented Sep 12, 2023

morningman Sep 13, 2023

github-actions bot commented Sep 14, 2023

hubgeter commented Sep 14, 2023

github-actions bot commented Sep 14, 2023

doris-robot commented Sep 14, 2023

doris-robot commented Sep 14, 2023

amorynan Sep 15, 2023 •

edited

Loading

hubgeter commented Sep 15, 2023

github-actions bot commented Sep 15, 2023

doris-robot commented Sep 15, 2023

doris-robot commented Sep 15, 2023

hubgeter commented Sep 16, 2023

github-actions bot commented Sep 16, 2023

doris-robot commented Sep 16, 2023

doris-robot commented Sep 16, 2023

amorynan left a comment

morningman left a comment

github-actions bot commented Sep 16, 2023

[enhancement](fix)change ordinary type null value is \N,complex type null value is null #24207

[enhancement](fix)change ordinary type null value is \N,complex type null value is null #24207

Conversation

hubgeter commented Sep 11, 2023 • edited Loading

Proposed changes

Further comments

hubgeter commented Sep 11, 2023

github-actions bot commented Sep 11, 2023

doris-robot commented Sep 11, 2023

hubgeter commented Sep 12, 2023

github-actions bot commented Sep 12, 2023

doris-robot commented Sep 12, 2023

doris-robot commented Sep 12, 2023

morningman left a comment • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Sep 12, 2023

github-actions bot commented Sep 12, 2023

morningman Sep 13, 2023

Choose a reason for hiding this comment

github-actions bot commented Sep 14, 2023

hubgeter commented Sep 14, 2023

github-actions bot commented Sep 14, 2023

doris-robot commented Sep 14, 2023

doris-robot commented Sep 14, 2023

amorynan Sep 15, 2023 • edited Loading

Choose a reason for hiding this comment

hubgeter commented Sep 15, 2023

github-actions bot commented Sep 15, 2023

doris-robot commented Sep 15, 2023

doris-robot commented Sep 15, 2023

hubgeter commented Sep 16, 2023

github-actions bot commented Sep 16, 2023

doris-robot commented Sep 16, 2023

doris-robot commented Sep 16, 2023

amorynan left a comment

Choose a reason for hiding this comment

morningman left a comment

Choose a reason for hiding this comment

github-actions bot commented Sep 16, 2023

hubgeter commented Sep 11, 2023 •

edited

Loading

morningman left a comment •

edited

Loading

amorynan Sep 15, 2023 •

edited

Loading