Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG]Failed to scale-in of asmysql-orc cluster #9046

Open
tianyue86 opened this issue Mar 13, 2025 · 0 comments
Open

[BUG]Failed to scale-in of asmysql-orc cluster #9046

tianyue86 opened this issue Mar 13, 2025 · 0 comments
Assignees
Labels
kind/bug Something isn't working
Milestone

Comments

@tianyue86
Copy link

tianyue86 commented Mar 13, 2025

Describe the bug

Kubernetes: v1.31.1-aliyun.1
KubeBlocks: 1.0.0-beta.33
kbcli: 1.0.0-beta.15

To Reproduce
Steps to reproduce the behavior:

  1. Create asmysql-orc cluster with yaml below
apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: orcm-hcagop
  namespace: default
spec:
  clusterDef: orchestrator
  topology: raft
  terminationPolicy: WipeOut
  services:
    - name: orchestrator
      componentSelector: orchestrator
      spec:
        ports:
          - name: orc-http
            port: 80
  componentSpecs:
    - name: orchestrator
      serviceVersion: 3.2.6
      replicas: 3
      resources:
        requests:
          cpu: 500m
          memory: 0.5Gi
        limits:
          cpu: 500m
          memory: 0.5Gi
      volumeClaimTemplates:
        - name: data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
-------------------------------------
apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: asmysql-hcagop
  namespace: default
spec:
  clusterDef: mysql
  topology: orc
  terminationPolicy: WipeOut
  componentSpecs:
    - name: mysql
      serviceVersion: 8.0.30
      
      disableExporter: true
      replicas: 2
      resources:
        limits:
          cpu: 500m
          memory: 0.5Gi
        requests:
          cpu: 500m
          memory: 0.5Gi
      volumeClaimTemplates:
        - name: data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
      serviceRefs:
      - name: orchestrator
        namespace: default
        clusterServiceSelector:
          cluster: orcm-hcagop
          service:
            component: orchestrator
            service: orchestrator
            port: orc-http
          credential:
            component: orchestrator
            name: orchestrator

2.scale-out and scale-in

kbcli cluster scale-out asmysql-hcagop --auto-approve --force=true --components mysql --replicas 1  --namespace default
OpsRequest asmysql-hcagop-horizontalscaling-vw7pn created successfully, you can view the progress:
	kbcli cluster describe-ops asmysql-hcagop-horizontalscaling-vw7pn -n default

kbcli cluster list-instances asmysql-hcagop --namespace default        
NAME                     NAMESPACE   CLUSTER          COMPONENT   STATUS    ROLE        ACCESSMODE   AZ                 CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE     NODE                                   CREATED-TIME                 
asmysql-hcagop-mysql-0   default     asmysql-hcagop   mysql       Running   primary                  cn-zhangjiakou-c   500m / 500m          512Mi / 512Mi           data:20Gi   cn-zhangjiakou.10.0.0.128/10.0.0.128   Mar 13,2025 17:06 UTC+0800   
asmysql-hcagop-mysql-1   default     asmysql-hcagop   mysql       Running   secondary                cn-zhangjiakou-c   500m / 500m          512Mi / 512Mi           data:20Gi   cn-zhangjiakou.10.0.0.128/10.0.0.128   Mar 13,2025 17:06 UTC+0800   
asmysql-hcagop-mysql-2   default     asmysql-hcagop   mysql       Running   secondary                cn-zhangjiakou-c   500m / 500m          512Mi / 512Mi           data:20Gi   cn-zhangjiakou.10.0.0.128/10.0.0.128   Mar 13,2025 17:10 UTC+0800   

kbcli cluster scale-in asmysql-hcagop --auto-approve --force=true --components mysql --replicas 1  --namespace default 
OpsRequest asmysql-hcagop-horizontalscaling-7txhb created successfully, you can view the progress:
	kbcli cluster describe-ops asmysql-hcagop-horizontalscaling-7txhb -n default
  1. Check ops status ===> scale-in is always in running state
kbcli cluster list-ops asmysql-hcagop --status all  --namespace default
NAME                                     NAMESPACE   TYPE                CLUSTER          COMPONENT   STATUS    PROGRESS   CREATED-TIME                 
asmysql-hcagop-horizontalscaling-vw7pn   default     HorizontalScaling   asmysql-hcagop   mysql       Succeed   1/1        Mar 13,2025 17:10 UTC+0800   
asmysql-hcagop-horizontalscaling-7txhb   default     HorizontalScaling   asmysql-hcagop   mysql       Running   0/1        Mar 13,2025 17:13 UTC+0800
  1. See error
    2025-03-13T09:21:05.111Z INFO build error: requeue after: 1s as: [http error occurred when executing action memberLeave at pod asmysql-hcagop-mysql-2: unexpected http status code: 503 Service Unavailable] {"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"asmysql-hcagop-mysql","namespace":"default"}, "namespace": "default", "name": "asmysql-hcagop-mysql", "reconcileID": "0fe55c21-431a-4b75-bd2a-ca3cf0b0ab5b", "component": {"name":"asmysql-hcagop-mysql","namespace":"default"}}
    2025-03-13T09:21:05.118Z INFO reconcile object *v1.Component with action STATUS OK {"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"asmysql-hcagop-mysql","namespace":"default"}, "namespace": "default", "name": "asmysql-hcagop-mysql", "reconcileID": "0fe55c21-431a-4b75-bd2a-ca3cf0b0ab5b", "component": {"name":"asmysql-hcagop-mysql","namespace":"default"}}
    2025-03-13T09:21:05.450Z INFO expected cm object[default/sidecar-mysql8.0-reload-script-asmysql-vknysl] has been unchanged

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

@tianyue86 tianyue86 added the kind/bug Something isn't working label Mar 13, 2025
@tianyue86 tianyue86 added this to the Release 1.0.0 milestone Mar 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants