Code

Convert to Qbeast#152

Code

Merged

Jiaweihu08 merged 79 commits intoQbeast-io:mainQbeast-io/qbeast-spark:mainfrom Jiaweihu08:read-staging-dataJiaweihu08/qbeast-spark:read-staging-dataCopy head branch name to clipboardJan 27, 2023

Commits

Commits on May 10, 2022

• First naive implementation of convert to qbeast

  osopardo1

  committed
  8a61514

  Copy full SHA for 8a61514

Commits on May 11, 2022

• Add headers

  osopardo1

  committed
  99a823b

  Copy full SHA for 99a823b

Commits on Sep 8, 2022

• Merge branch 'main' into convert-to-qbeast

  osopardo1

  committed
  f8bd3d0

  Copy full SHA for f8bd3d0

• Change RunnableCommand to LeafRunnableCommand

  osopardo1

  committed
  feb0754

  Copy full SHA for feb0754

Commits on Sep 9, 2022

• Access deltaLog/snapshot and modify log

  Jiaweihu08

  committed
  5d4e0bc

  Copy full SHA for 5d4e0bc

Commits on Sep 12, 2022

• Add transformers to revision

  Jiaweihu08

  committed
  51c5bb6

  Copy full SHA for 51c5bb6

• Converted tables should be readable using both formates

  Jiaweihu08

  committed
  9a2236b

  Copy full SHA for 9a2236b

Commits on Sep 20, 2022

• Add AddFiles with qbeast metadata using append write mode

  Jiaweihu08

  committed
  76219b3

  Copy full SHA for 76219b3

• Check input format

  Jiaweihu08

  committed
  d3d472d

  Copy full SHA for d3d472d

• Simplify test

  Jiaweihu08

  committed
  8f5ffe2

  Copy full SHA for 8f5ffe2

• Refactor code

  Jiaweihu08

  committed
  ac90d08

  Copy full SHA for ac90d08

• Add method for parquet to delta conversion

  Jiaweihu08

  committed
  b4d1466

  Copy full SHA for b4d1466

• Reformat tests

  Jiaweihu08

  committed
  471cb06

  Copy full SHA for 471cb06

• Separate metadata tag computation

  Jiaweihu08

  committed
  b22afc0

  Copy full SHA for b22afc0

Commits on Sep 21, 2022

• Extract record count from parquet file metadata

  Jiaweihu08

  committed
  1de47a8

  Copy full SHA for 1de47a8

• Add test for parquet to qbeast conversion, test index metrics resulted from conversion

  Jiaweihu08

  committed
  d435fc7

  Copy full SHA for d435fc7

• Test String data type

  Jiaweihu08

  committed
  0d26406

  Copy full SHA for 0d26406

• Make command idempotent

  Jiaweihu08

  committed
  6f92dcc

  Copy full SHA for 6f92dcc

• Test command idempotence

  Jiaweihu08

  committed
  0208678

  Copy full SHA for 0208678

• Test converting a partitioned delta table

  Jiaweihu08

  committed
  909bc11

  Copy full SHA for 909bc11

• Use global path parameter

  Jiaweihu08

  committed
  c30c6e8

  Copy full SHA for c30c6e8

Commits on Sep 22, 2022

• Consider cases where the provided fileFormat doesn't match with real format of the table

  Jiaweihu08

  committed
  304c991

  Copy full SHA for 304c991

• Add format inferrence, conversion for partitioned parquet files, inferr schema for partitioned parquet files

  Jiaweihu08

  committed
  72e40ae

  Copy full SHA for 72e40ae

• Add comment

  Jiaweihu08

  committed
  f009a54

  Copy full SHA for f009a54

• Test extraction of numRecords when AddFile stats is corrupted

  Jiaweihu08

  committed
  daf4454

  Copy full SHA for daf4454

• Remove unnecessary assertions

  Jiaweihu08

  committed
  c8efff5

  Copy full SHA for c8efff5

Commits on Sep 23, 2022

• Spark data type name extraction for only supported partition data types

  Jiaweihu08

  committed
  389401d

  Copy full SHA for 389401d

• Test supported partition data types

  Jiaweihu08

  committed
  1a4e5d5

  Copy full SHA for 1a4e5d5

• Secure return type

  Jiaweihu08

  committed
  f5858f9

  Copy full SHA for f5858f9

• Use wider default column min max to avoid out-of-scope points

  Jiaweihu08

  committed
  02e9100

  Copy full SHA for 02e9100

• Add tests for Analyze, and Compaction on a converted qbeast table

  Jiaweihu08

  committed
  c1c16d5

  Copy full SHA for c1c16d5

• Use table path from tablID

  Jiaweihu08

  committed
  25a3956

  Copy full SHA for 25a3956

• Add test for optimization on a converted table

  Jiaweihu08

  committed
  56baac9

  Copy full SHA for 56baac9

Commits on Sep 28, 2022

• Remove command

  Jiaweihu08

  committed
  6171063

  Copy full SHA for 6171063

Commits on Jan 16, 2023

• Add staging files during read

  Jiaweihu08

  committed
  486711f

  Copy full SHA for 486711f

Commits on Jan 18, 2023

• WIP, Qbeast format compatibility with delta lake

  Jiaweihu08

  committed
  116ae77

  Copy full SHA for 116ae77

Commits on Jan 19, 2023

• WIP, Use all columns for staging sampling

  Jiaweihu08

  committed
  e48732a

  Copy full SHA for e48732a

Commits on Jan 20, 2023

• Merge branch 'main' into convert-to-qbeast

  Jiaweihu08

  committed
  3338c90

  Copy full SHA for 3338c90

• Rely on user input for source format, direct metadata update

  Jiaweihu08

  committed
  fbb84c0

  Copy full SHA for fbb84c0

Commits on Jan 23, 2023

• Add comment

  Jiaweihu08

  committed
  1da9bff

  Copy full SHA for 1da9bff

• Merge to main

  Jiaweihu08

  committed
  e518465

  Copy full SHA for e518465

• Use the indexing columns from the latest revision for online sampling

  Jiaweihu08

  committed
  6e00d91

  Copy full SHA for 6e00d91

• Use EmptyTransformation/ers for the conversion revision

  Jiaweihu08

  committed
  df27cd3

  Copy full SHA for df27cd3

• Test Conversion command. Analyze command should not change ANNOUNCED set when index is empty

  Jiaweihu08

  committed
  a4c8919

  Copy full SHA for a4c8919

• Proper merge method for EmptyTransformation

  Jiaweihu08

  committed
  9a36b1c

  Copy full SHA for 9a36b1c

• Is a qbeast table if is converted or written using qbeast format

  Jiaweihu08

  committed
  951b68e

  Copy full SHA for 951b68e

• Use conversion revisionID(0) as staging revisionID

  Jiaweihu08

  committed
  9f4df90

  Copy full SHA for 9f4df90

• ConvertToQbeast can add conversion/staging revision to a qbeat table

  Jiaweihu08

  committed
  d3b1cfd

  Copy full SHA for d3b1cfd

Commits on Jan 24, 2023

• Discard staging revision via the creation of EmptySpace

  Jiaweihu08

  committed
  770b49f

  Copy full SHA for 770b49f

• Separate staging cube statuses computation

  Jiaweihu08

  committed
  9d7bb4f

  Copy full SHA for 9d7bb4f

• Add staging revision during first writes

  Jiaweihu08

  committed
  371b060

  Copy full SHA for 371b060

• Add staging revision during first write, idempotent conversion

  Jiaweihu08

  committed
  a2218ff

  Copy full SHA for a2218ff

• Remove staging revision from loadRevisionAt

  Jiaweihu08

  committed
  08c4aaf

  Copy full SHA for 08c4aaf

• Allow the creation of empty cubes for the staging revision

  Jiaweihu08

  committed
  1ff0dd2

  Copy full SHA for 1ff0dd2

• Compaction with no dataChange

  Jiaweihu08

  committed
  08395d1

  Copy full SHA for 08395d1

• Include staging revision and use initial revisionID = 1 for test

  Jiaweihu08

  committed
  b009c37

  Copy full SHA for b009c37

• Check for staging revision via Transformer type

  Jiaweihu08

  committed
  7b976a2

  Copy full SHA for 7b976a2

• Avoid appending on the staging revision

  Jiaweihu08

  committed
  85a6fcd

  Copy full SHA for 85a6fcd

• Complete conversion test

  Jiaweihu08

  committed
  abc6ea3

  Copy full SHA for abc6ea3

• Complete hybrid table test

  Jiaweihu08

  committed
  a353645

  Copy full SHA for a353645

Commits on Jan 25, 2023

• Code comment

  Jiaweihu08

  committed
  4c4ae59

  Copy full SHA for 4c4ae59

• Update documentation to include the staging revision and the ConvertToQbeatCommand

  Jiaweihu08

  committed
  44245df

  Copy full SHA for 44245df

• Optimize imports

  Jiaweihu08

  committed
  5895f97

  Copy full SHA for 5895f97

• Check total num revisions after append on a converted table

  Jiaweihu08

  committed
  2c6711d

  Copy full SHA for 2c6711d

• Place staging revision utils under RevisionUtils

  Jiaweihu08

  committed
  ab91ff0

  Copy full SHA for ab91ff0

• Test EmptyTransformer/ation

  Jiaweihu08

  committed
  d550497

  Copy full SHA for d550497

• Add comment and remove unnecessary changes

  Jiaweihu08

  committed
  6bc7ceb

  Copy full SHA for 6bc7ceb

• Move params and methods to RevisionUtils

  Jiaweihu08

  committed
  a7cbe6e

  Copy full SHA for a7cbe6e

• isStagingFile is specific of QbeastSnapshot

  Jiaweihu08

  committed
  b01e806

  Copy full SHA for b01e806

• Correct typo

  Jiaweihu08

  committed
  ba00ec4

  Copy full SHA for ba00ec4

• Fix loadRevisionAt

  Jiaweihu08

  committed
  8969983

  Copy full SHA for 8969983

Commits on Jan 26, 2023

• 'Remove' partitioned table conversion support

  Jiaweihu08

  committed
  d61538f

  Copy full SHA for d61538f

• Create exception message object

  Jiaweihu08

  committed
  5620d25

  Copy full SHA for 5620d25

• Update documentation

  Jiaweihu08

  committed
  c64fb5c

  Copy full SHA for c64fb5c

• Test table identifier format

  Jiaweihu08

  committed
  f089c50

  Copy full SHA for f089c50

• Test loadRevisionAt with invalid timestamp

  Jiaweihu08

  committed
  b69320f

  Copy full SHA for b69320f

• Skip Analyze and Optimize for the staging RevisionID

  Jiaweihu08

  committed
  60d36aa

  Copy full SHA for 60d36aa

• Use AnalysisException, more test for EmptyTransformer

  Jiaweihu08

  committed
  57cacd4

  Copy full SHA for 57cacd4

Commits on Jan 27, 2023

• Update metadata in MetadataManager

  Jiaweihu08

  committed
  1332bc9

  Copy full SHA for 1332bc9