From 1ab716e200dacc7781abf1e6a0282e5128125488 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Edgar=20Hern=C3=A1ndez?= <23639005+israel-hdez@users.noreply.github.com> Date: Fri, 7 Feb 2025 14:11:10 -0600 Subject: [PATCH] Better compatibility with in-place upgrades MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit For in-place upgrades the manifests of the new version are applied in a cluster where KServe may already be installed. The old version of the kserve-controller is terminated and the new version is deployed. The default strategy is Rolling. This has the consequence that during an in-place upgrade, there will be two different versions of kserve-controller running in parallel. Most importantly, the two instances will be running with different configs, which include references to different versions of kserve-agent, kserve-router, storage-initializer, etc. During an in-place upgrade, the new version of the controller will upgrade the deployed models to use the new versions of the images. However, since the old version of kserve-controller is running in parallel, this old version will do rollback. The two instances will conflict with each other, leading to Deployments/KSVCs being changed very quickly which will lead to the cluster spawning pods without control. Eventually, the cluster will terminate the old version of kserve-controller. However, because of the very, very quick updates to deployments the resources of the cluster can be exhausted leading to instability. By changing the `strategy` to `Recreate`, during an in-place upgrade the cluster will make sure that the old version of kserve-controller is fully terminated before starting the new version. This prevents cluster instability. Signed-off-by: Edgar Hernández <23639005+israel-hdez@users.noreply.github.com> --- config/manager/manager.yaml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/config/manager/manager.yaml b/config/manager/manager.yaml index c75dbddb54b..85d706242e8 100644 --- a/config/manager/manager.yaml +++ b/config/manager/manager.yaml @@ -12,6 +12,9 @@ spec: matchLabels: control-plane: kserve-controller-manager controller-tools.k8s.io: "1.0" + strategy: + type: Recreate + rollingUpdate: nil template: metadata: labels: