Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[chery-pick](branch-2.1) Pick "[Fix](group commit) Fix group commit block queue mem estimate fault" #37379

Merged
merged 1 commit into from
Jul 7, 2024

Conversation

Yukang-Lian
Copy link
Collaborator

Pick [Fix](group commit) Fix group commit block queue mem estimate faule #35314

Proposed changes

Issue Number: close #xxx

Problem: When group commit=async_mode and NULL data is imported into a variant type column, it causes incorrect memory statistics for group commit backpressure, leading to a stuck issue. Cause: In group commit mode, blocks are first added to a queue in batches using add block, and then blocks are retrieved from the queue using get block. To track memory usage during backpressure, we add the block size to the memory statistics during add block and subtract the block size from the memory statistics during get block. However, for variant types, during the add block write to WAL, serialization occurs, which can merge types (e.g., merging int and bigint into bigint), thereby changing the block size. This results in a discrepancy between the block size during get block and add block, causing memory statistics to overflow.
Solution: Record the block size at the time of add block and use this recorded size during get block instead of the actual block size. This ensures consistency in the memory addition and subtraction.

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

Proposed changes

Issue Number: close #xxx

…pache#35314)

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

**Problem:** When `group commit=async_mode` and NULL data is imported
into a `variant` type column, it causes incorrect memory statistics for
group commit backpressure, leading to a stuck issue.
**Cause:** In group commit mode, blocks are first added to a queue in
batches using `add block`, and then blocks are retrieved from the queue
using `get block`. To track memory usage during backpressure, we add the
block size to the memory statistics during `add block` and subtract the
block size from the memory statistics during `get block`. However, for
`variant` types, during the `add block` write to WAL, serialization
occurs, which can merge types (e.g., merging `int` and `bigint` into
`bigint`), thereby changing the block size. This results in a
discrepancy between the block size during `get block` and `add block`,
causing memory statistics to overflow.
**Solution:** Record the block size at the time of `add block` and use
this recorded size during `get block` instead of the actual block size.
This ensures consistency in the memory addition and subtraction.

## Further comments

If this is a relatively large or complex change, kick off the discussion
at [[email protected]](mailto:[email protected]) by explaining why
you chose the solution you did and what alternatives you considered,
etc...
@Yukang-Lian
Copy link
Collaborator Author

run buildall

Copy link
Contributor

github-actions bot commented Jul 7, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.32% (9151/25194)
Line Coverage: 27.86% (74703/268115)
Region Coverage: 26.74% (38512/144033)
Branch Coverage: 23.45% (19521/83252)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c01253eb4b4d2ffd8b1987dda27c81fb7464124f_c01253eb4b4d2ffd8b1987dda27c81fb7464124f/report/index.html

@dataroaring dataroaring merged commit 7d423b3 into apache:branch-2.1 Jul 7, 2024
20 of 22 checks passed
@yiguolei yiguolei mentioned this pull request Jul 19, 2024
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants