-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[feature](information_schema)add metadata_name_ids for quickly get catlogs,db,table and add profiling table in order to Compatible with mysql #22702
Conversation
clang-tidy review says "All clean, LGTM! 👍" |
1 similar comment
clang-tidy review says "All clean, LGTM! 👍" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please add regression test cases
@@ -424,6 +429,10 @@ public class SessionVariable implements Serializable, Writable { | |||
@VariableMgr.VarAttr(name = INSERT_TIMEOUT) | |||
public int insertTimeoutS = 14400; | |||
|
|||
//query internal.information_schema.simple_tables time | |||
@VariableMgr.VarAttr(name = QUERY_SIMPLE_TABLES_TIMEOUT) | |||
public int querySimpleTablesTimeout = 3; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it can be a FE config, not session variable.
Because we always set it globally
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok good idea
gensrc/thrift/Descriptors.thrift
Outdated
@@ -117,7 +117,9 @@ enum TSchemaTableType { | |||
SCH_ROWSETS, | |||
SCH_BACKENDS, | |||
SCH_COLUMN_STATISTICS, | |||
SCH_PARAMETERS; | |||
SCH_PARAMETERS, | |||
SCH_SIMPLE_TABLES, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
SCH_SIMPLE_TABLES, | |
SCH_SIMPLE_TABLES, |
gensrc/thrift/Descriptors.thrift
Outdated
SCH_PARAMETERS; | ||
SCH_PARAMETERS, | ||
SCH_SIMPLE_TABLES, | ||
SCH_PROFILING; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
SCH_PROFILING; | |
SCH_PROFILING; |
gensrc/thrift/FrontendService.thrift
Outdated
@@ -351,6 +353,14 @@ struct TListTableStatusResult { | |||
1: required list<TTableStatus> tables | |||
} | |||
|
|||
struct TSimpleTableStatus { | |||
1: required string name |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
use 4 spaces instead of tab.
and use optional
instead of required
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok good idea
gensrc/thrift/FrontendService.thrift
Outdated
2: required i64 id | ||
} | ||
struct TListSimpleTableStatusResult { | ||
1: required list<TSimpleTableStatus> tables |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ditto
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok
.column("ROUTINE_TYPE", ScalarType.createVarchar(64)) | ||
.column("DATA_TYPEDTD_IDENDS", ScalarType.createVarchar(64)) | ||
.build())) | ||
.put("simple_tables", new SchemaTable(SystemIdGenerator.getNextId(), "simple_tables", TableType.SCHEMA, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
how about just name it as metadata_name_ids
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok good idea
clang-tidy review says "All clean, LGTM! 👍" |
1 similar comment
clang-tidy review says "All clean, LGTM! 👍" |
run buildall |
clang-tidy review says "All clean, LGTM! 👍" |
run buildall |
clang-tidy review says "All clean, LGTM! 👍" |
(From new machine)TeamCity pipeline, clickbench performance test result: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
PR approved by at least one committer and no changes requested. |
PR approved by anyone and no changes requested. |
clang-tidy review says "All clean, LGTM! 👍" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
run buildall |
clang-tidy review says "All clean, LGTM! 👍" |
(From new machine)TeamCity pipeline, clickbench performance test result: |
run buildall |
clang-tidy review says "All clean, LGTM! 👍" |
(From new machine)TeamCity pipeline, clickbench performance test result: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
PR approved by at least one committer and no changes requested. |
…tlogs,db,table and add profiling table in order to Compatible with mysql (apache#22702) add information_schema.metadata_name_idsfor quickly get catlogs,db,table. 1. table struct : ```mysql mysql> desc internal.information_schema.metadata_name_ids; +---------------+--------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +---------------+--------------+------+-------+---------+-------+ | CATALOG_ID | BIGINT | Yes | false | NULL | | | CATALOG_NAME | VARCHAR(512) | Yes | false | NULL | | | DATABASE_ID | BIGINT | Yes | false | NULL | | | DATABASE_NAME | VARCHAR(64) | Yes | false | NULL | | | TABLE_ID | BIGINT | Yes | false | NULL | | | TABLE_NAME | VARCHAR(64) | Yes | false | NULL | | +---------------+--------------+------+-------+---------+-------+ 6 rows in set (0.00 sec) mysql> select * from internal.information_schema.metadata_name_ids where CATALOG_NAME="hive1" limit 1 \G; *************************** 1. row *************************** CATALOG_ID: 113008 CATALOG_NAME: hive1 DATABASE_ID: 113042 DATABASE_NAME: ssb1_parquet TABLE_ID: 114009 TABLE_NAME: dates 1 row in set (0.07 sec) ``` 2. when you create / drop catalog , need not refresh catalog . ```mysql mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 21301 1 row in set (0.34 sec) mysql> drop catalog hive2; Query OK, 0 rows affected (0.01 sec) mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 10665 1 row in set (0.04 sec) mysql> create catalog hive3 ... mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 21301 1 row in set (0.32 sec) ``` 3. create / drop table , need not refresh catalog . ```mysql mysql> CREATE TABLE IF NOT EXISTS demo.example_tbl ... ; mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 10666 1 row in set (0.04 sec) mysql> drop table demo.example_tbl; Query OK, 0 rows affected (0.01 sec) mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 10665 1 row in set (0.04 sec) ``` 4. you can set query time , prevent queries from taking too long . ``` fe.conf : query_metadata_name_ids_timeout the time used to obtain all tables in one database ``` 5. add information_schema.profiling in order to Compatible with mysql ```mysql mysql> select * from information_schema.profiling; Empty set (0.07 sec) mysql> set profiling=1; Query OK, 0 rows affected (0.01 sec) ```
…tlogs,db,table and add profiling table in order to Compatible with mysql (apache#22702) add information_schema.metadata_name_idsfor quickly get catlogs,db,table. 1. table struct : ```mysql mysql> desc internal.information_schema.metadata_name_ids; +---------------+--------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +---------------+--------------+------+-------+---------+-------+ | CATALOG_ID | BIGINT | Yes | false | NULL | | | CATALOG_NAME | VARCHAR(512) | Yes | false | NULL | | | DATABASE_ID | BIGINT | Yes | false | NULL | | | DATABASE_NAME | VARCHAR(64) | Yes | false | NULL | | | TABLE_ID | BIGINT | Yes | false | NULL | | | TABLE_NAME | VARCHAR(64) | Yes | false | NULL | | +---------------+--------------+------+-------+---------+-------+ 6 rows in set (0.00 sec) mysql> select * from internal.information_schema.metadata_name_ids where CATALOG_NAME="hive1" limit 1 \G; *************************** 1. row *************************** CATALOG_ID: 113008 CATALOG_NAME: hive1 DATABASE_ID: 113042 DATABASE_NAME: ssb1_parquet TABLE_ID: 114009 TABLE_NAME: dates 1 row in set (0.07 sec) ``` 2. when you create / drop catalog , need not refresh catalog . ```mysql mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 21301 1 row in set (0.34 sec) mysql> drop catalog hive2; Query OK, 0 rows affected (0.01 sec) mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 10665 1 row in set (0.04 sec) mysql> create catalog hive3 ... mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 21301 1 row in set (0.32 sec) ``` 3. create / drop table , need not refresh catalog . ```mysql mysql> CREATE TABLE IF NOT EXISTS demo.example_tbl ... ; mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 10666 1 row in set (0.04 sec) mysql> drop table demo.example_tbl; Query OK, 0 rows affected (0.01 sec) mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 10665 1 row in set (0.04 sec) ``` 4. you can set query time , prevent queries from taking too long . ``` fe.conf : query_metadata_name_ids_timeout the time used to obtain all tables in one database ``` 5. add information_schema.profiling in order to Compatible with mysql ```mysql mysql> select * from information_schema.profiling; Empty set (0.07 sec) mysql> set profiling=1; Query OK, 0 rows affected (0.01 sec) ```
…tlogs,db,table and add profiling table in order to Compatible with mysql (apache#22702) add information_schema.metadata_name_idsfor quickly get catlogs,db,table. 1. table struct : ```mysql mysql> desc internal.information_schema.metadata_name_ids; +---------------+--------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +---------------+--------------+------+-------+---------+-------+ | CATALOG_ID | BIGINT | Yes | false | NULL | | | CATALOG_NAME | VARCHAR(512) | Yes | false | NULL | | | DATABASE_ID | BIGINT | Yes | false | NULL | | | DATABASE_NAME | VARCHAR(64) | Yes | false | NULL | | | TABLE_ID | BIGINT | Yes | false | NULL | | | TABLE_NAME | VARCHAR(64) | Yes | false | NULL | | +---------------+--------------+------+-------+---------+-------+ 6 rows in set (0.00 sec) mysql> select * from internal.information_schema.metadata_name_ids where CATALOG_NAME="hive1" limit 1 \G; *************************** 1. row *************************** CATALOG_ID: 113008 CATALOG_NAME: hive1 DATABASE_ID: 113042 DATABASE_NAME: ssb1_parquet TABLE_ID: 114009 TABLE_NAME: dates 1 row in set (0.07 sec) ``` 2. when you create / drop catalog , need not refresh catalog . ```mysql mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 21301 1 row in set (0.34 sec) mysql> drop catalog hive2; Query OK, 0 rows affected (0.01 sec) mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 10665 1 row in set (0.04 sec) mysql> create catalog hive3 ... mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 21301 1 row in set (0.32 sec) ``` 3. create / drop table , need not refresh catalog . ```mysql mysql> CREATE TABLE IF NOT EXISTS demo.example_tbl ... ; mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 10666 1 row in set (0.04 sec) mysql> drop table demo.example_tbl; Query OK, 0 rows affected (0.01 sec) mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 10665 1 row in set (0.04 sec) ``` 4. you can set query time , prevent queries from taking too long . ``` fe.conf : query_metadata_name_ids_timeout the time used to obtain all tables in one database ``` 5. add information_schema.profiling in order to Compatible with mysql ```mysql mysql> select * from information_schema.profiling; Empty set (0.07 sec) mysql> set profiling=1; Query OK, 0 rows affected (0.01 sec) ```
…tlogs,db,table and add profiling table in order to Compatible with mysql (apache#22702) add information_schema.metadata_name_idsfor quickly get catlogs,db,table. 1. table struct : ```mysql mysql> desc internal.information_schema.metadata_name_ids; +---------------+--------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +---------------+--------------+------+-------+---------+-------+ | CATALOG_ID | BIGINT | Yes | false | NULL | | | CATALOG_NAME | VARCHAR(512) | Yes | false | NULL | | | DATABASE_ID | BIGINT | Yes | false | NULL | | | DATABASE_NAME | VARCHAR(64) | Yes | false | NULL | | | TABLE_ID | BIGINT | Yes | false | NULL | | | TABLE_NAME | VARCHAR(64) | Yes | false | NULL | | +---------------+--------------+------+-------+---------+-------+ 6 rows in set (0.00 sec) mysql> select * from internal.information_schema.metadata_name_ids where CATALOG_NAME="hive1" limit 1 \G; *************************** 1. row *************************** CATALOG_ID: 113008 CATALOG_NAME: hive1 DATABASE_ID: 113042 DATABASE_NAME: ssb1_parquet TABLE_ID: 114009 TABLE_NAME: dates 1 row in set (0.07 sec) ``` 2. when you create / drop catalog , need not refresh catalog . ```mysql mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 21301 1 row in set (0.34 sec) mysql> drop catalog hive2; Query OK, 0 rows affected (0.01 sec) mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 10665 1 row in set (0.04 sec) mysql> create catalog hive3 ... mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 21301 1 row in set (0.32 sec) ``` 3. create / drop table , need not refresh catalog . ```mysql mysql> CREATE TABLE IF NOT EXISTS demo.example_tbl ... ; mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 10666 1 row in set (0.04 sec) mysql> drop table demo.example_tbl; Query OK, 0 rows affected (0.01 sec) mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 10665 1 row in set (0.04 sec) ``` 4. you can set query time , prevent queries from taking too long . ``` fe.conf : query_metadata_name_ids_timeout the time used to obtain all tables in one database ``` 5. add information_schema.profiling in order to Compatible with mysql ```mysql mysql> select * from information_schema.profiling; Empty set (0.07 sec) mysql> set profiling=1; Query OK, 0 rows affected (0.01 sec) ```
Proposed changes
add information_schema.metadata_name_idsfor quickly get catlogs,db,table.
Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...