Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG]the last ordinal pod can not be online #9000

Open
lancelot1989 opened this issue Mar 4, 2025 · 0 comments
Open

[BUG]the last ordinal pod can not be online #9000

lancelot1989 opened this issue Mar 4, 2025 · 0 comments
Assignees
Labels
kind/bug Something isn't working vip

Comments

@lancelot1989
Copy link
Contributor

lancelot1989 commented Mar 4, 2025

Using yaml's below to create a cluster.

apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: kafka2-ext-zk-descriptor
  namespace: default
spec:
  terminationPolicy: Delete
  # Specifies the name of the ClusterDefinition to use when creating a Cluster.
  # Note: DO NOT UPDATE THIS FIELD
  # The value must be `kafaka` to create a Kafka Cluster
  clusterDef: kafka
  # Specifies the name of the ClusterTopology to be used when creating the
  # Cluster.
  # - combined: combined Kafka controller (KRaft) and broker in one Component
  # - combined_monitor: combined mode with monitor component
  # - separated: separated KRaft and Broker Components.
  # - separated_monitor: separated mode with monitor component
  # Valid options are: [combined,combined_monitor,separated,separated_monitor,kafka2-external-zk]
  topology: kafka2-external-zk
  services:
    - name: bootstrap
      serviceName: bootstrap
      componentSelector: kafka-broker
      spec:
        type: ClusterIP
        ports:
          - name: kafka-client
            targetPort: 9092
            port: 9092
  componentSpecs:
    - name: kafka-broker
      componentDef: kafka27-broker
      serviceRefs:
        - name: kafkaZookeeper
          namespace: default
          serviceDescriptor: kafka-cluster-zookeeper-service
      tls: false
      replicas: 3
      env:
        - name: KB_BROKER_DIRECT_POD_ACCESS
          value: 'true'
        - name: KAFKA_HEAP_OPTS
          value: "-Xmx500m -Xms500m"
        - name: KB_KAFKA_ZK_SUB_PATH
          value: "test"   # set sub path to use for sharing zk, default value is $(CLUSTER_NAME), set empty string for root
      resources:
        limits:
          cpu: '0.5'
          memory: 0.5Gi
        requests:
          cpu: '0.5'
          memory: 0.5Gi
      volumeClaimTemplates:
        - name: data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
        - name: metadata
          spec:
            storageClassName: null
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 5Gi
---
apiVersion: apps.kubeblocks.io/v1
kind: ServiceDescriptor
metadata:
  name: kafka-cluster-zookeeper-service
  namespace: default
spec:
  # Specifies the type or nature of the service.
  # Should represent a well-known application cluster type, such as {mysql, redis, zookeeper}.
  serviceKind: zookeeper
  # Represents the version of the service reference.
  serviceVersion: 3.8.5
  # Represents the endpoint of the service connection credential.
  endpoint:
    # your external zk endpoints here
    value: "zookeeper-cluster-zookeeper-0.zookeeper-cluster-zookeeper-headless.default:2181,zookeeper-cluster-zookeeper-1.zookeeper-cluster-zookeeper-headless.default:2181,zookeeper-cluster-zookeeper-2.zookeeper-cluster-zookeeper-headless.default:2181"
  # Represents the port of the service connection credential.
  port:
    value: "2181"

Using below yaml to do an OpsRequest to offline the last ordinal pod.

apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  generateName: offline-instance-
spec:
  clusterName: kafka2-ext-zk-descriptor
  horizontalScaling:
  - componentName: kafka-broker
    scaleIn:
      onlineInstancesToOffline:
      - kafka2-ext-zk-descriptor-kafka-broker-2
  type: HorizontalScaling

Using below yaml to do an OpsRequest to try to online the last ordinal pod.

apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  generateName: online-instance-
spec:
  clusterName: kafka2-ext-zk-descriptor
  horizontalScaling:
  - componentName: kafka-broker
    scaleOut:
      offlineInstancesToOnline:
      - kafka2-ext-zk-descriptor-kafka-broker-2
  type: HorizontalScaling

Then OpsRequest of online will stuck.

k get ops
NAME                           TYPE                CLUSTER                    STATUS    PROGRESS   AGE
offline-instance-6vkml         HorizontalScaling   kafka2-ext-zk-descriptor   Succeed   1/1        7m1s
online-instance-znk4k          HorizontalScaling   kafka2-ext-zk-descriptor   Running   0/1        6m10s

kafka2-ext-zk-descriptor-kafka-broker-2 can not be onlined.

k get pod
NAME                                      READY   STATUS    RESTARTS     AGE
kafka2-ext-zk-descriptor-kafka-broker-0   2/2     Running   0            8m43s
kafka2-ext-zk-descriptor-kafka-broker-1   2/2     Running   0            8m43s

Cluster's offlineInstances is cleaned but replicas does not increment.

k get cluster
NAME                       CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS     AGE
kafka2-ext-zk-descriptor   kafka                Delete               Running    10m
k get cluster kafka2-ext-zk-descriptor -oyaml|grep -C5 replicas
    - name: KB_KAFKA_ZK_SUB_PATH
      value: test
    name: kafka-broker
    replicas: 2
    resources:
      limits:
        cpu: 500m
        memory: 512Mi
      requests:

KB version: v1.0.0-beta.21
K8S version: v1.30.4

@lancelot1989 lancelot1989 added the kind/bug Something isn't working label Mar 4, 2025
@shanshanying shanshanying added this to the Release 1.0.0 milestone Mar 4, 2025
@shanshanying shanshanying removed this from the Release 1.0.0 milestone Mar 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Something isn't working vip
Projects
None yet
Development

No branches or pull requests

3 participants