Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]pulsar cluster's bookies-recovery-0 pod is always in Init status #9011

Open
tianyue86 opened this issue Mar 6, 2025 · 0 comments
Open

[BUG]pulsar cluster's bookies-recovery-0 pod is always in Init status #9011

tianyue86 opened this issue Mar 6, 2025 · 0 comments
Assignees
@shanshanying
Labels
kind/bug Something isn't working
Milestone
Release 1.0.0

Comments

@tianyue86
Copy link

Describe the bug

Kubernetes: v1.31.1-aliyun.1
KubeBlocks: 1.0.0-beta.32
kbcli: 1.0.0-beta.15

To Reproduce
Steps to reproduce the behavior:

  1. Create pulsar cluster with following yaml. ---> it works well in beta.30
apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: pulsar-zuqtwe
  namespace: default
  labels:
    app.kubernetes.io/instance: pulsar-zuqtwe
  annotations:
    resource.kubeblocks.io/ignore-constraint: "true"
    kubeblocks.io/enabled-pod-ordinal-svc: broker
spec:
  clusterDef: pulsar
  topology: pulsar-enhanced-cluster
  terminationPolicy: DoNotTerminate
  services:
    - name: broker-bootstrap
      serviceName: broker-bootstrap
      componentSelector: broker
      spec:
        type: ClusterIP
        ports:
          - name: pulsar
            port: 6650
            targetPort: 6650
          - name: http
            port: 80
            targetPort: 8080
          - name: kafka-client
            port: 9092
            targetPort: 9092
    - name: zookeeper
      serviceName: zookeeper
      componentSelector: zookeeper
      spec:
        type: ClusterIP
        ports:
          - name: client
            port: 2181
            targetPort: 2181
  componentSpecs:
    - name: proxy
      serviceVersion: 2.11.2
      replicas: 1
      resources:
        limits:
          cpu: 500m
          memory: 0.5Gi
        requests:
          cpu: 500m
          memory: 0.5Gi
    - name: bookies-recovery
      serviceVersion: 2.11.2
      replicas: 1
      resources:
        limits:
          cpu: 500m
          memory: 0.5Gi
        requests:
          cpu: 500m
          memory: 0.5Gi
    - name: broker
      serviceVersion: 2.11.2
      replicas: 3
      env:
        - name: KB_PULSAR_BROKER_NODEPORT
          value: "false"
      
      resources:
        limits:
          cpu: 500m
          memory: 0.5Gi
        requests:
          cpu: 500m
          memory: 0.5Gi
    - name: bookies
      serviceVersion: 2.11.2
      replicas: 3
      resources:
        limits:
          cpu: 500m
          memory: 0.5Gi
        requests:
          cpu: 500m
          memory: 0.5Gi
      volumeClaimTemplates:
        - name: ledgers
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
        - name: journal
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
    - name: zookeeper
      serviceVersion: 2.11.2
      replicas: 3
      resources:
        limits:
          cpu: 500m
          memory: 0.5Gi
        requests:
          cpu: 500m
          memory: 0.5Gi
      volumeClaimTemplates:
        - name: data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
  1. Check cluster status
k get cluster -A
NAMESPACE   NAME             CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS     AGE
default     pulsar-zuqtwe    pulsar               DoNotTerminate       Updating   87m

k get pod
NAME                                          READY   STATUS             RESTARTS        AGE
pulsar-zuqtwe-bookies-0                       2/2     Running            0               60m
pulsar-zuqtwe-bookies-1                       2/2     Running            0               83m
pulsar-zuqtwe-bookies-2                       2/2     Running            0               83m
pulsar-zuqtwe-bookies-recovery-0              0/1     Init:0/1           0               81m
pulsar-zuqtwe-broker-0                        2/2     Running            0               60m
pulsar-zuqtwe-broker-1                        2/2     Running            0               83m
pulsar-zuqtwe-broker-2                        2/2     Running            0               81m
pulsar-zuqtwe-proxy-0                         1/1     Running            0               60m
pulsar-zuqtwe-zookeeper-0                     1/1     Running            0               85m
pulsar-zuqtwe-zookeeper-1                     1/1     Running            0               60m
pulsar-zuqtwe-zookeeper-2                     1/1     Running            0               71m

k describe pod pulsar-zuqtwe-bookies-recovery-0
Name:             pulsar-zuqtwe-bookies-recovery-0
Namespace:        default
Priority:         0
Service Account:  kb-pulsar-bookies-recovery-2-1.0.0-alpha.0
Node:             cn-zhangjiakou.10.0.0.128/10.0.0.128
Start Time:       Thu, 06 Mar 2025 14:06:23 +0800
Labels:           app.kubernetes.io/instance=pulsar-zuqtwe
                  app.kubernetes.io/managed-by=kubeblocks
                  apps.kubeblocks.io/component-name=bookies-recovery
                  apps.kubeblocks.io/pod-name=pulsar-zuqtwe-bookies-recovery-0
                  controller-revision-hash=6b475dbd67
                  workloads.kubeblocks.io/instance=pulsar-zuqtwe-bookies-recovery
                  workloads.kubeblocks.io/managed-by=InstanceSet
Annotations:      k8s.aliyun.com/pod-ips: 10.0.0.193
Status:           Pending
IP:               10.0.0.193
IPs:
  IP:           10.0.0.193
Controlled By:  InstanceSet/pulsar-zuqtwe-bookies-recovery
Init Containers:
  check-bookies:
    Container ID:  containerd://57633df0c170c599b31c3f2737042a6fba2b85bc76224d4121e6b9593a3f6b62
    Image:         apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar:2.11.2
    Image ID:      apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar@sha256:2e53735894e7c6ed28a0efc27b5cb170ddb44bd1809b1167adcd7f1c7c556ab7
    Port:          <none>
    Host Port:     <none>
    Command:
      /kb-scripts/check-bookies.sh
    State:          Running
      Started:      Thu, 06 Mar 2025 14:06:24 +0800
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     0
      memory:  0
    Requests:
      cpu:     0
      memory:  0
    Environment Variables from:
      pulsar-zuqtwe-bookies-recovery-env  ConfigMap  Optional: false
    Environment:                          <none>
    Mounts:
      /kb-scripts from scripts (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vhskk (ro)
Containers:
  bookies-recovery:
    Container ID:  
    Image:         apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar:2.11.2
    Image ID:      
    Ports:         8000/TCP, 3181/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /kb-scripts/start-bookies-recovery.sh
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  512Mi
    Requests:
      cpu:     500m
      memory:  512Mi
    Environment Variables from:
      pulsar-zuqtwe-bookies-recovery-env  ConfigMap  Optional: false
    Environment:
      SERVICE_PORT:             8000
      httpServerEnabled:        true
      httpServerPort:           8000
      prometheusStatsHttpPort:  8000
      useHostNameAsBookieID:    true
    Mounts:
      /kb-scripts from scripts (rw)
      /opt/pulsar/conf from bkrecovery-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vhskk (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 False 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  bkrecovery-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      pulsar-zuqtwe-bookies-recovery-bkrecovery-config
    Optional:  false
  scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      pulsar-zuqtwe-bookies-recovery-pulsar-scripts
    Optional:  false
  kube-api-access-vhskk:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>
  1. check cmp
k get cmp
NAME                             DEFINITION                                SERVICE-VERSION   STATUS     AGE
pulsar-zuqtwe-bookies            pulsar-bookkeeper-2-1.0.0-alpha.0         2.11.2            Running    87m
pulsar-zuqtwe-bookies-recovery   pulsar-bookies-recovery-2-1.0.0-alpha.0   2.11.2            Creating   84m
pulsar-zuqtwe-broker             pulsar-broker-2-1.0.0-alpha.0             2.11.2            Running    87m
pulsar-zuqtwe-proxy              pulsar-proxy-2-1.0.0-alpha.0              2.11.2            Running    84m
pulsar-zuqtwe-zookeeper          pulsar-zookeeper-2-1.0.0-alpha.0          2.11.2            Running    88m

k describe cmp pulsar-zuqtwe-bookies-recovery
Status:
  Conditions:
    Last Transition Time:  2025-03-06T06:06:02Z
    Message:               The operator has started the provisioning of Cluster: pulsar-zuqtwe-bookies-recovery
    Observed Generation:   1
    Reason:                PreCheckSucceed
    Status:                True
    Type:                  ProvisioningStarted
    Last Transition Time:  2025-03-06T06:06:04Z
    Message:               the component phase is Creating
    Observed Generation:   1
    Reason:                Unavailable
    Status:                False
    Type:                  Available
  Observed Generation:     1
  Phase:                   Creating
Events:                    <none>
  1. Scroll down to '....'
  2. See errorExpected behavior
    A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.
@tianyue86 tianyue86 added the kind/bug Something isn't working label Mar 6, 2025
@tianyue86 tianyue86 added this to the Release 1.0.0 milestone Mar 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@shanshanying shanshanying

Labels
kind/bug Something isn't working
Projects
None yet
Milestone
Release 1.0.0
Development

No branches or pull requests
2 participants
@shanshanying @tianyue86