Flink 1.20: Support Avro and Parquet timestamp(9), unknown, and defaults #12470
+462
−348
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I'll also follow up with a PR for Parquet readers, but that depends on changes in #12463. |
rdblue
force-pushed
the
flink-readers-writers
branch
from
5f505e7 to
0af3f01
Compare
|
#12463 was merged and the changes for Parquet were small, so I included them here. |
rdblue
changed the title
rdblue
commented
Mar 6, 2025
| } else { | ||
| return Optional.of(new MicrosToTimestampReader(desc)); | ||
| } | ||
| return Optional.of(new MicrosToTimestampReader(desc)); |
There was a problem hiding this comment.
Previously, the readers were converting values to LocalDateTime or OffsetDateTime and then Flink would convert those values back to a (millis, nanosOfMilli) pair. This involved a lot of unnecessary date/time logic in both Iceberg and Flink as well as readers to produce the separate types.
Now, the conversion to Flink is direct and doesn't go through Java date/time classes. That avoids all time zone calculations and should be quicker.
rdblue
commented
Mar 6, 2025
| LogicalTypeAnnotation annotation = primitive.getLogicalTypeAnnotation(); | ||
| if (annotation != null) { | ||
| Optional<ParquetValueWriter<?>> writer = | ||
| annotation.accept(new LogicalTypeWriterBuilder(fType, desc)); |
There was a problem hiding this comment.
Updated this to use the logical annotation visitor.
rdblue
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This updates Flink's Avro and Parquet readers to support new timestamp(9) and unknown types.
While enabling
DataTestcases, I found thatsupportsDefaultValueswas not enabled so default value tests were not running for Avro. After I enabled those tests, I also needed to update theRowDataassertions and also convert values to match Flink's object model in the readers by callingRowDataUtil.convertConstant.