feat: support NodeCountScaler

[free6om]

fixed #6552
fixed #5795

Background

The requirement originated from here.
When the requirement was raised, the KB ClusterDefinition API had a WorkloadType field, where the underlying implementation relied on the native StatefulSet API of Kubernetes when the WorkloadType was Consensus, Replication, or Stateful. In this context, the requirement could be fulfilled by introducing a new WorkloadType similar to Daemon.
Starting from version 0.8, KB introduced a new API architecture where there is no longer a field like WorkloadType. In the final implementation, all components are supported through the InstanceSet, which is a common Workload API. The InstanceSet directly manages more fundamental Kubernetes objects such as Pods and PVCs.
In the new API architecture, adding support for the Daemon type of workload is no longer as straightforward as before. The specific design is as follows.

Design

Goals

1. KB supports the ability to be aware of the number of nodes, i.e., dynamically adjusting the Replicas value of certain ComponentSpecs under the Cluster.
2. KB supports enforcing a one-to-one relationship between Pods and Nodes through settings such as Pod anti-affinity, Node affinity, and taint tolerations.
3. KB supports elevating the priority of Pods by setting PriorityClassName, reducing the risk of Pods being evicted due to insufficient Node resources.

The above three points are the core features of DaemonSet.

API

For points 2 and 3 mentioned above, KB currently supports them, and they can be configured through relevant fields in APIs like Cluster.
For point 1, it is achieved by introducing an experimental API called NodeCountScalar.

apiVersion: experimental.kubeblocks.io/v1alpha1
kind: NodeCountScalar
metadata:
  name: my-cluster-scaler
spec:
  targetClusterName: "my-cluster"
  targetComponentNames: ["component-0"]
status:
  componentStatuses:
  - name: "component-0"
    currentReplicas: 3
    readyReplicas: 3
    availableReplicas: 3
    desiredReplicas: 3
  conditions:
  - type: ScaleReady
    status: True
    reason: ScaleReady
    message: "scale ready"
    lastTransitionTime: 2024-05-07 16:10:01
  lastScaleTime: 2024-05-07 16:05:15

Implementation

The NodeCountScalar Controller watches for Node and Cluster objects.
When a Create or Delete event occurs for a Node, it triggers a full-scale tuning of all NodeCountScalar objects.
When a Create or Update event occurs for a Cluster, it triggers the tuning of the corresponding NodeCountScalar object (if present).

During tuning, the NodeCountScalar Controller retrieves the current list of Nodes, where the length of the list becomes the DesiredReplicas.
Then, the Replicas value of the target ComponentSpec in the target Cluster object is updated to DesiredReplicas, achieving the goal of dynamically adjusting Replicas.
During the tuning process of the Cluster and its secondary resources, the Status field of the corresponding NodeCountScalar object is updated in real-time to observe the progress of dynamic adjustments.

Testing

1. Create the target Cluster object
   Create the Cluster using the following command. At this time, the Replicas value of the ComponentSpec named "mysql" is 0, and the Cluster is in the Stopped state.

kbcli cluster create mytest --cluster-definition=apecloud-mysql --set replicas=0

2. Create the NodeCountScalar object
   Run the following command to create the NodeAwareScaler:

kubectl create -f -<<EOF
apiVersion: experimental.kubeblocks.io/v1alpha1
kind: NodeCountScalar
metadata:
  name: mytest
spec:
  targetClusterName: mytest
  targetComponentNames: ["mysql"]
EOF

Expected results

1. The Replicas value in the ComponentSpec named "mysql" in the Cluster object should be changed to the same value as the number of Nodes.
2. After the Cluster tuning is complete (i.e., the Phase in the Status is Running), when you retrieve the NodeCountScalar object named "mytest" using kubectl, you should see the following output:

# kubectl get ncs mytest
# #output should be like this:
NAME     TARGET-CLUSTER-NAME   READY   REASON     MESSAGE            LAST-SCALE-TIME
mytest   mytest                True    Ready      scale ready        4m55s

[codecov]

Codecov Report

Attention: Patch coverage is 54.31034% with 106 lines in your changes are missing coverage. Please review.

Project coverage is 64.88%. Comparing base (a89c4b2) to head (07f16ba).
Report is 30 commits behind head on main.

Files	Patch %	Lines
controllers/experimental/tree_loader.go	42.85%	16 Missing and 8 partials ⚠️
...ntrollers/experimental/reconciler_update_status.go	70.66%	17 Missing and 5 partials ⚠️
...rollers/experimental/nodecountscaler_controller.go	0.00%	16 Missing ⚠️
controllers/experimental/cluster_handler.go	0.00%	14 Missing ⚠️
...rs/experimental/reconciler_scale_target_cluster.go	62.16%	9 Missing and 5 partials ⚠️
controllers/experimental/node_scaling_handler.go	0.00%	12 Missing ⚠️
pkg/controllerutil/util.go	75.00%	2 Missing and 1 partial ⚠️
pkg/controller/instanceset/instance_util.go	83.33%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7258      +/-   ##
==========================================
- Coverage   64.96%   64.88%   -0.09%     
==========================================
  Files         337      345       +8     
  Lines       41723    41936     +213     
==========================================
+ Hits        27107    27210     +103     
- Misses      12260    12345      +85     
- Partials     2356     2381      +25     

Flag	Coverage Δ
unittests	64.88% <54.31%> (-0.09%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[weicao]

If there are taints on some nodes, which cannot be tolerated by the Component, which situation will happen?

• Some Pods are in pending status.
• Some Nodes have multiple Pods deployed.

[weicao]

seems MergeList should be moved into a more suitable package.

[free6om]

/cherry-pick release-0.9

[github-actions]

🤖 says: cherry pick action finished successfully 🎉!
See: https://github.com/apecloud/kubeblocks/actions/runs/8984506513