Fix SST files not being cleaned up in the locations folder

[danpi]

Descriptions of the changes in this PR:

Fix #4554

Motivation

Resolve issue with SST files remaining uncleared in the locations folder.

Changes

1. Reuse the cleanup logic of GarbageCollectorThread to trigger entryLocationCompact() when a major compaction occurs.
2. Add the entryLocationCompaction configuration option to control the above behavior, disabled by default.

[danpi]

I believe this issue is related to optimization #3653.

When using deleteRange to replace the one-by-one deletion method, only one action is written to RocksDB. If RocksDB is performing tiered compaction, this action is not guaranteed to be picked up, causing SST files to remain undeleted.

To resolve this issue:
Option 1: Revert to the previous logic of iteratively deleting each key.
Option 2: Use compactRange to forcefully clean up levels L0 to L6.
I chose to perform a compactRange synchronously when executing a major compaction.

[hangc0276]

I believe this issue is related to optimization #3653.

When using deleteRange to replace the one-by-one deletion method, only one action is written to RocksDB. If RocksDB is performing tiered compaction, this action is not guaranteed to be picked up, causing SST files to remain undeleted.

To resolve this issue: Option 1: Revert to the previous logic of iteratively deleting each key. Option 2: Use compactRange to forcefully clean up levels L0 to L6. I chose to perform a compactRange synchronously when executing a major compaction.

@danpi Great job! I think it should be a bug in RocksDB. We can use option 2. Please check the failed CI, thanks.

[danpi]

@hangc0276 Addressed the feedback above. PTAL, thanks.

[hangc0276]

Remove this one and use entryLocationCompactionInterval > 0 instead?

[hangc0276]

We'd better introduce a random factor for it. If we roll restart the BookKeeper cluster, and all the bookies start time are the same, which will make all the bookies triggered RocksDB compaction at the same time and impact the read latency.

[hangc0276]

The random factor can be any time in [0, entryLocationCompactionInterval]

[hangc0276]

@hangc0276 Addressed the feedback above. PTAL, thanks.

@danpi Thanks for your contribution. I left more comments; please take a look. Thanks.

[danpi]

@hangc0276 Addressed the feedback above. PTAL, thanks.

@danpi Thanks for your contribution. I left more comments; please take a look. Thanks.

@hangc0276 Add randomCompactionDelay to avoid all the bookies triggering compaction simultaneously. PTAL, thanks.