[improve](cloud-mow) merge and remove old version of delete bitmap when cumulative compaction is done #40204
Conversation
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
hust-hhb
force-pushed
the
remove-delete-bitmap
branch
from
121f7b1 to
497f711
Compare
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
TPC-H: Total hot run time: 38188 ms |
TPC-DS: Total hot run time: 191474 ms |
ClickBench: Total hot run time: 31.94 s |
|
run buildall |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| return Status::OK(); | ||
| } | ||
|
|
||
| void CloudCumulativeCompaction::process_old_version_delete_bitmap() { |
There was a problem hiding this comment.
warning: function 'process_old_version_delete_bitmap' has cognitive complexity of 63 (threshold 50) [readability-function-cognitive-complexity]
void CloudCumulativeCompaction::process_old_version_delete_bitmap() {
^Additional context
be/src/cloud/cloud_cumulative_compaction.cpp:361: +1, including nesting penalty of 0, nesting level increased to 1
if (!pre_rowsets.empty()) {
^be/src/cloud/cloud_cumulative_compaction.cpp:387: +2, including nesting penalty of 1, nesting level increased to 2
do {
^be/src/cloud/cloud_cumulative_compaction.cpp:388: +3, including nesting penalty of 2, nesting level increased to 3
if (!new_delete_bitmap->empty()) {
^be/src/cloud/cloud_cumulative_compaction.cpp:392: +4, including nesting penalty of 3, nesting level increased to 4
DBUG_EXECUTE_IF("CloudCumulativeCompaction.modify_rowsets.get_mow_lock_failed", {
^be/src/util/debug_points.h:36: expanded from macro 'DBUG_EXECUTE_IF'
if (UNLIKELY(config::enable_debug_points)) { \
^be/src/cloud/cloud_cumulative_compaction.cpp:392: +5, including nesting penalty of 4, nesting level increased to 5
DBUG_EXECUTE_IF("CloudCumulativeCompaction.modify_rowsets.get_mow_lock_failed", {
^be/src/util/debug_points.h:38: expanded from macro 'DBUG_EXECUTE_IF'
if (dp) { \
^be/src/cloud/cloud_cumulative_compaction.cpp:398: +4, including nesting penalty of 3, nesting level increased to 4
if (get_st.ok()) {
^be/src/cloud/cloud_cumulative_compaction.cpp:402: +4, including nesting penalty of 3, nesting level increased to 4
if (!get_st.ok()) {
^be/src/cloud/cloud_cumulative_compaction.cpp:411: +4, including nesting penalty of 3, nesting level increased to 4
DBUG_EXECUTE_IF(
^be/src/util/debug_points.h:36: expanded from macro 'DBUG_EXECUTE_IF'
if (UNLIKELY(config::enable_debug_points)) { \
^be/src/cloud/cloud_cumulative_compaction.cpp:411: +5, including nesting penalty of 4, nesting level increased to 5
DBUG_EXECUTE_IF(
^be/src/util/debug_points.h:38: expanded from macro 'DBUG_EXECUTE_IF'
if (dp) { \
^be/src/cloud/cloud_cumulative_compaction.cpp:417: +4, including nesting penalty of 3, nesting level increased to 4
if (update_st.ok()) {
^be/src/cloud/cloud_cumulative_compaction.cpp:421: +4, including nesting penalty of 3, nesting level increased to 4
if (!update_st.ok()) {
^be/src/cloud/cloud_cumulative_compaction.cpp:428: +4, including nesting penalty of 3, nesting level increased to 4
DBUG_EXECUTE_IF("CloudCumulativeCompaction.modify_rowsets.remove_mow_lock_failed", {
^be/src/util/debug_points.h:36: expanded from macro 'DBUG_EXECUTE_IF'
if (UNLIKELY(config::enable_debug_points)) { \
^be/src/cloud/cloud_cumulative_compaction.cpp:428: +5, including nesting penalty of 4, nesting level increased to 5
DBUG_EXECUTE_IF("CloudCumulativeCompaction.modify_rowsets.remove_mow_lock_failed", {
^be/src/util/debug_points.h:38: expanded from macro 'DBUG_EXECUTE_IF'
if (dp) { \
^be/src/cloud/cloud_cumulative_compaction.cpp:434: +4, including nesting penalty of 3, nesting level increased to 4
if (remove_st.ok()) {
^be/src/cloud/cloud_cumulative_compaction.cpp:438: +4, including nesting penalty of 3, nesting level increased to 4
if (!remove_st.ok()) {
^be/src/cloud/cloud_cumulative_compaction.cpp:448: +2, including nesting penalty of 1, nesting level increased to 2
if (get_st.ok() && update_st.ok() && remove_st.ok()) {
^be/src/cloud/cloud_cumulative_compaction.cpp:448: +1
if (get_st.ok() && update_st.ok() && remove_st.ok()) {
^be/src/cloud/cloud_cumulative_compaction.cpp:453: +3, including nesting penalty of 2, nesting level increased to 3
for (auto it = new_delete_bitmap->delete_bitmap.begin();
^| return Status::OK(); | ||
| } | ||
|
|
||
| void CloudCumulativeCompaction::process_old_version_delete_bitmap() { |
There was a problem hiding this comment.
warning: function 'process_old_version_delete_bitmap' exceeds recommended size/complexity thresholds [readability-function-size]
void CloudCumulativeCompaction::process_old_version_delete_bitmap() {
^Additional context
be/src/cloud/cloud_cumulative_compaction.cpp:349: 110 lines including whitespace and comments (threshold 80)
void CloudCumulativeCompaction::process_old_version_delete_bitmap() {
^| } | ||
| } | ||
|
|
||
| void MetaServiceImpl::remove_delete_bitmap_update_lock( |
There was a problem hiding this comment.
warning: function 'remove_delete_bitmap_update_lock' exceeds recommended size/complexity thresholds [readability-function-size]
void MetaServiceImpl::remove_delete_bitmap_update_lock(
^Additional context
cloud/src/meta-service/meta_service.cpp:2138: 95 lines including whitespace and comments (threshold 80)
void MetaServiceImpl::remove_delete_bitmap_update_lock(
^
|
TeamCity be ut coverage result: |
|
run buildall |
| } | ||
| } | ||
|
|
||
| void MetaServiceImpl::remove_delete_bitmap_update_lock( |
There was a problem hiding this comment.
warning: function 'remove_delete_bitmap_update_lock' exceeds recommended size/complexity thresholds [readability-function-size]
void MetaServiceImpl::remove_delete_bitmap_update_lock(
^Additional context
cloud/src/meta-service/meta_service.cpp:2138: 101 lines including whitespace and comments (threshold 80)
void MetaServiceImpl::remove_delete_bitmap_update_lock(
^
|
TeamCity be ut coverage result: |
|
run buildall |
| return Status::OK(); | ||
| } | ||
|
|
||
| void CloudCumulativeCompaction::process_old_version_delete_bitmap() { |
There was a problem hiding this comment.
warning: function 'process_old_version_delete_bitmap' has cognitive complexity of 63 (threshold 50) [readability-function-cognitive-complexity]
void CloudCumulativeCompaction::process_old_version_delete_bitmap() {
^Additional context
be/src/cloud/cloud_cumulative_compaction.cpp:362: +1, including nesting penalty of 0, nesting level increased to 1
if (!pre_rowsets.empty()) {
^be/src/cloud/cloud_cumulative_compaction.cpp:388: +2, including nesting penalty of 1, nesting level increased to 2
do {
^be/src/cloud/cloud_cumulative_compaction.cpp:389: +3, including nesting penalty of 2, nesting level increased to 3
if (!new_delete_bitmap->empty()) {
^be/src/cloud/cloud_cumulative_compaction.cpp:393: +4, including nesting penalty of 3, nesting level increased to 4
DBUG_EXECUTE_IF("CloudCumulativeCompaction.modify_rowsets.get_mow_lock_failed", {
^be/src/util/debug_points.h:36: expanded from macro 'DBUG_EXECUTE_IF'
if (UNLIKELY(config::enable_debug_points)) { \
^be/src/cloud/cloud_cumulative_compaction.cpp:393: +5, including nesting penalty of 4, nesting level increased to 5
DBUG_EXECUTE_IF("CloudCumulativeCompaction.modify_rowsets.get_mow_lock_failed", {
^be/src/util/debug_points.h:38: expanded from macro 'DBUG_EXECUTE_IF'
if (dp) { \
^be/src/cloud/cloud_cumulative_compaction.cpp:399: +4, including nesting penalty of 3, nesting level increased to 4
if (get_st.ok()) {
^be/src/cloud/cloud_cumulative_compaction.cpp:403: +4, including nesting penalty of 3, nesting level increased to 4
if (!get_st.ok()) {
^be/src/cloud/cloud_cumulative_compaction.cpp:412: +4, including nesting penalty of 3, nesting level increased to 4
DBUG_EXECUTE_IF(
^be/src/util/debug_points.h:36: expanded from macro 'DBUG_EXECUTE_IF'
if (UNLIKELY(config::enable_debug_points)) { \
^be/src/cloud/cloud_cumulative_compaction.cpp:412: +5, including nesting penalty of 4, nesting level increased to 5
DBUG_EXECUTE_IF(
^be/src/util/debug_points.h:38: expanded from macro 'DBUG_EXECUTE_IF'
if (dp) { \
^be/src/cloud/cloud_cumulative_compaction.cpp:418: +4, including nesting penalty of 3, nesting level increased to 4
if (update_st.ok()) {
^be/src/cloud/cloud_cumulative_compaction.cpp:422: +4, including nesting penalty of 3, nesting level increased to 4
if (!update_st.ok()) {
^be/src/cloud/cloud_cumulative_compaction.cpp:429: +4, including nesting penalty of 3, nesting level increased to 4
DBUG_EXECUTE_IF("CloudCumulativeCompaction.modify_rowsets.remove_mow_lock_failed", {
^be/src/util/debug_points.h:36: expanded from macro 'DBUG_EXECUTE_IF'
if (UNLIKELY(config::enable_debug_points)) { \
^be/src/cloud/cloud_cumulative_compaction.cpp:429: +5, including nesting penalty of 4, nesting level increased to 5
DBUG_EXECUTE_IF("CloudCumulativeCompaction.modify_rowsets.remove_mow_lock_failed", {
^be/src/util/debug_points.h:38: expanded from macro 'DBUG_EXECUTE_IF'
if (dp) { \
^be/src/cloud/cloud_cumulative_compaction.cpp:435: +4, including nesting penalty of 3, nesting level increased to 4
if (remove_st.ok()) {
^be/src/cloud/cloud_cumulative_compaction.cpp:439: +4, including nesting penalty of 3, nesting level increased to 4
if (!remove_st.ok()) {
^be/src/cloud/cloud_cumulative_compaction.cpp:449: +2, including nesting penalty of 1, nesting level increased to 2
if (get_st.ok() && update_st.ok() && remove_st.ok()) {
^be/src/cloud/cloud_cumulative_compaction.cpp:449: +1
if (get_st.ok() && update_st.ok() && remove_st.ok()) {
^be/src/cloud/cloud_cumulative_compaction.cpp:454: +3, including nesting penalty of 2, nesting level increased to 3
for (auto it = new_delete_bitmap->delete_bitmap.begin();
^| return Status::OK(); | ||
| } | ||
|
|
||
| void CloudCumulativeCompaction::process_old_version_delete_bitmap() { |
There was a problem hiding this comment.
warning: function 'process_old_version_delete_bitmap' exceeds recommended size/complexity thresholds [readability-function-size]
void CloudCumulativeCompaction::process_old_version_delete_bitmap() {
^Additional context
be/src/cloud/cloud_cumulative_compaction.cpp:350: 110 lines including whitespace and comments (threshold 80)
void CloudCumulativeCompaction::process_old_version_delete_bitmap() {
^
|
TeamCity be ut coverage result: |
|
run buildall |
|
TeamCity be ut coverage result: |
fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgr.java
Outdated
Show resolved
Hide resolved
regression-test/suites/compaction/test_cu_compaction_remove_old_version_delete_bitmap.groovy
Show resolved
Hide resolved
|
TeamCity be ut coverage result: |
|
PR approved by at least one committer and no changes requested. |
github-actions
bot
added
the
approved
…en cumulative compaction is done (#40204) When the count of delete bitmap is big, it may lead to high cpu use rate, we can merge old version delete bitmap to reduce the count of delete bitmap when cumulative compaction is done to reduce the use rate of cpu. Here is an example: before cu compaciton ,rowset layout is "cumulative point": 10, "rowsets": [ "[0-1] 0 DATA NONOVERLAPPING 0200000000000000ffffffffffffffecffffffffffffffff 0", "[2-9] 1 DATA NONOVERLAPPING 0200000000002cc1a8447a4e0ba5850f773803ae2d534284 1.21 KB", "[10-10] 1 DATA NONOVERLAPPING 0200000000002d07a8447a4e0ba5850f773803ae2d534284 1.09 KB", "[11-11] 1 DATA NONOVERLAPPING 0200000000002d14a8447a4e0ba5850f773803ae2d534284 1.10 KB", "[12-12] 1 DATA NONOVERLAPPING 0200000000002d23a8447a4e0ba5850f773803ae2d534284 1.10 KB", "[13-13] 1 DATA NONOVERLAPPING 0200000000002d33a8447a4e0ba5850f773803ae2d534284 1.10 KB", "[14-14] 1 DATA NONOVERLAPPING 0200000000002d40a8447a4e0ba5850f773803ae2d534284 1.10 KB" ], base rowset is [2-9], now cu compaciton input rowset range is [10-14], after compaciton, rowset layout is "cumulative point": 10, "rowsets": [ "[0-1] 0 DATA NONOVERLAPPING 0200000000000000ffffffffffffffecffffffffffffffff 0", "[2-9] 1 DATA NONOVERLAPPING 0200000000002cc1a8447a4e0ba5850f773803ae2d534284 1.21 KB", "[10-14] 1 DATA NONOVERLAPPING 0200000000002d63a8447a4e0ba5850f773803ae2d534284 1.21 KB" ], 1. the delete bitmap of rowset [10-10] -> [11-11] -> [12-12] -> [13-13] -> [14-14] will be deleted when delete_expired_stale_rowsets 2. the delete bitmap of version 10~14, will do agg first, then update the agg result on verison 14 delete bitmap 3. if update sucess, add the delete bitmap of version 10~13 to the queue which name is to_remove_vec 4. when deleting delete bitmap on rowset [10-10] -> [11-11] -> [12-12] -> [13-13] -> [14-14], the delete bitmap which mention on step 3 will be deleted too.
…it when processing old version delete bitmap on cu compaction (#42379) pr #40204 support deleting old version delete bitmap when doing cu compaction, it will update delete bitmap with agg result, then deleting old version delete bitmap. Updating delete bitmap means overwriting existing keys, however delete bitmap may split into multiple kvs to store in fdb, so we should delete the existing keys and rewrite it to fdb instead of overwriting directly.
…on delete bitmap (#44300) ### What problem does this PR solve? Related PR: #40204 Problem Summary: pr #40204 support removing old delete bitmap, however it donesn't consider boundary which will lead to miss some delete bitmap should be removed on be , it only affect local delete bitmap, the delete bitmap store in fdb is right, and these missing delete bitmaps can only be deleted by next base compaciton before this pr.
…on delete bitmap (#44448) Problem Summary: pr #40204 support removing old delete bitmap, however it donesn't consider boundary which will lead to miss some delete bitmap should be removed on be , it only affect local delete bitmap, the delete bitmap store in fdb is right, and these missing delete bitmaps can only be deleted by next base compaciton before this pr. pick pr:#44300
1. check every delete bitmap in MS belongs to a visible rowset 2. check that #40204 works as expected
1. check every delete bitmap in MS belongs to a visible rowset 2. check that #40204 works as expected
…itmap when doing cu compaction (apache#42471) pr apache#40204 support deleting old version delete bitmap when doing cu compaction, it a new function and better to add a config to enable it or not.
When the count of delete bitmap is big, it may lead to high cpu use rate, we can merge old version delete bitmap to reduce the count of delete bitmap when cumulative compaction is done to reduce the use rate of cpu.
Here is an example:
before cu compaciton ,rowset layout is
base rowset is [2-9], now cu compaciton input rowset range is [10-14], after compaciton, rowset layout is