Use compaction dynamic config to enable compaction supervisors #17782
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
#16768 introduces the option to run compaction as a supervisor on Overlord instead of via coordinator duties.
It also packages several improvements over the traditional compaction duty, such as support for using
the MSQ engine to launch compaction tasks.
However, that patch relies on an Overlord runtime property to enable/disable this feature, which can be
cumbersome and error prone, given that both Coordinator and Overlord need to be aware of it.
It also leads to a poorer design since half of compaction-related features are controlled by the compaction
dynamic config whereas the other half is controlled by the runtime properties.
This patch reconciles the two by simply using compaction dynamic config for all purposes.
It has the added benefit of allowing a quick switch between Overlord supervisor and Coordinator duty, in
case something misbehaves or perhaps for testing purposes.
Changes
CompactionSupervisorConfig
useSupervisors
andengine
to cluster-level compaction dynamic configuseAutoScaleSlots
Release note
TODO
This PR has: