Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG]vscale ops of pg cluster is stuck in the running state, with INVALID restart Annotation #9012

Closed
tianyue86 opened this issue Mar 6, 2025 · 1 comment · Fixed by #9014
Assignees
Labels
kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Milestone

Comments

@tianyue86
Copy link

tianyue86 commented Mar 6, 2025

Describe the bug

Kubernetes: v1.31.1-aliyun.1
KubeBlocks: 1.0.0-beta.32
kbcli: 1.0.0-beta.15

To Reproduce
Steps to reproduce the behavior:

  1. Create pg cluster with the following yaml
apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: postgres-ocgaqt
  namespace: default
spec:
  clusterDef: postgresql
  topology: replication
  terminationPolicy: Delete
  componentSpecs:
    - name: postgresql
      serviceVersion: 15.7.0
      labels:
        apps.kubeblocks.postgres.patroni/scope: postgres-ocgaqt-postgresql
      replicas: 2
      disableExporter: true
      resources:
        limits:
          cpu: 100m
          memory: 0.5Gi
        requests:
          cpu: 100m
          memory: 0.5Gi
      volumeClaimTemplates:
        - name: data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
  1. vscale the cluster
kbcli cluster vscale postgres-ocgaqt --auto-approve --force=true                 --components postgresql                 --cpu 200m                 --memory 0.6Gi --namespace default
OpsRequest postgres-ocgaqt-verticalscaling-8wmzh created successfully, you can view the progress:
	kbcli cluster describe-ops postgres-ocgaqt-verticalscaling-8wmzh -n default
  1. The process is stuck at this ops
kbcli cluster list-ops postgres-ocgaqt --status all  --namespace default
NAME                                    NAMESPACE   TYPE              CLUSTER           COMPONENT    STATUS    PROGRESS   CREATED-TIME                 
postgres-ocgaqt-verticalscaling-8wmzh   default     VerticalScaling   postgres-ocgaqt   postgresql   Running   0/2        Mar 06,2025 16:04 UTC+0800

k describe opsrequest postgres-ocgaqt-verticalscaling-8wmzh
Name:         postgres-ocgaqt-verticalscaling-8wmzh
Namespace:    default
Labels:       app.kubernetes.io/instance=postgres-ocgaqt
              app.kubernetes.io/managed-by=kubeblocks
              operations.kubeblocks.io/ops-type=VerticalScaling
Annotations:  <none>
API Version:  operations.kubeblocks.io/v1alpha1
Kind:         OpsRequest
Metadata:
  Creation Timestamp:  2025-03-06T08:04:51Z
  Finalizers:
    opsrequest.kubeblocks.io/finalizer
  Generate Name:  postgres-ocgaqt-verticalscaling-
  Generation:     2
  Owner References:
    API Version:     apps.kubeblocks.io/v1
    Kind:            Cluster
    Name:            postgres-ocgaqt
    UID:             036c86b9-2d3c-4bbe-8f6c-978b384d563d
  Resource Version:  67651601
  UID:               1e4df749-de9f-400d-9271-46af8fa2d592
Spec:
  Cluster Name:                    postgres-ocgaqt
  Enqueue On Force:                false
  Force:                           true
  Pre Condition Deadline Seconds:  0
  Type:                            VerticalScaling
  Vertical Scaling:
    Component Name:  postgresql
    Limits:
      Cpu:     200m
      Memory:  644245094400m
    Requests:
      Cpu:     200m
      Memory:  644245094400m
Status:
  Cluster Generation:  3
  Components:
    Postgresql:
      Phase:  Updating
      Progress Details:
        Message:     Start to vertical scale: Pod/postgres-ocgaqt-postgresql-1 in Component: postgresql
        Object Key:  Pod/postgres-ocgaqt-postgresql-1
        Start Time:  2025-03-06T08:04:51Z
        Status:      Processing
        Object Key:  Pod/postgres-ocgaqt-postgresql-0
        Status:      Pending
  Conditions:
    Last Transition Time:  2025-03-06T08:04:51Z
    Message:               wait for the controller to process the OpsRequest: postgres-ocgaqt-verticalscaling-8wmzh in Cluster: postgres-ocgaqt
    Reason:                WaitForProgressing
    Status:                True
    Type:                  WaitForProgressing
    Last Transition Time:  2025-03-06T08:04:51Z
    Message:               OpsRequest: postgres-ocgaqt-verticalscaling-8wmzh is validated
    Reason:                ValidateOpsRequestPassed
    Status:                True
    Type:                  Validated
    Last Transition Time:  2025-03-06T08:04:51Z
    Message:               Start to vertical scale resources in Cluster: postgres-ocgaqt
    Reason:                VerticalScalingStarted
    Status:                True
    Type:                  VerticalScaling
  Last Configuration:
    Components:
      Postgresql:
        Limits:
          Cpu:     100m
          Memory:  512Mi
        Requests:
          Cpu:      100m
          Memory:   512Mi
  Phase:            Running
  Progress:         0/2
  Start Timestamp:  2025-03-06T08:04:51Z
Events:
  Type    Reason                    Age                From                    Message
  ----    ------                    ----               ----                    -------
  Normal  WaitForProgressing        17m (x2 over 17m)  ops-request-controller  wait for the controller to process the OpsRequest: postgres-ocgaqt-verticalscaling-8wmzh in Cluster: postgres-ocgaqt
  Normal  ValidateOpsRequestPassed  17m (x2 over 17m)  ops-request-controller  OpsRequest: postgres-ocgaqt-verticalscaling-8wmzh is validated
  Normal  VerticalScalingStarted    17m (x2 over 17m)  ops-request-controller  Start to vertical scale resources in Cluster: postgres-ocgaqt
  Normal  Processing                17m                ops-request-controller  Start to vertical scale: Pod/postgres-ocgaqt-postgresql-1 in Component: postgresql

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

@tianyue86 tianyue86 added the kind/bug Something isn't working label Mar 6, 2025
@tianyue86 tianyue86 added this to the Release 1.0.0 milestone Mar 6, 2025
@tianyue86 tianyue86 changed the title [BUG]vscale ops of pg is stuck in the running state [BUG]vscale ops of pg cluster is stuck in the running state Mar 6, 2025
@shanshanying
Copy link
Contributor

Image Image

@shanshanying shanshanying changed the title [BUG]vscale ops of pg cluster is stuck in the running state [BUG]vscale ops of pg cluster is stuck in the running state, with INVALID restart Annotation Mar 6, 2025
@shanshanying shanshanying added the severity/major Great chance user will encounter the same problem label Mar 6, 2025
@sophon-zt sophon-zt linked a pull request Mar 6, 2025 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants