Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] delete restore failed cluster Deleting hang #8985

Open
JashBook opened this issue Feb 27, 2025 · 0 comments
Open

[BUG] delete restore failed cluster Deleting hang #8985

JashBook opened this issue Feb 27, 2025 · 0 comments
Assignees
Labels
kind/bug Something isn't working
Milestone

Comments

@JashBook
Copy link
Collaborator

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

  1. create cluster
kubectl apply -f -<<EOF
apiVersion: apps.kubeblocks.io/v1alpha1
kind: Cluster
metadata:
  name: mongo-bopyvl
  namespace: default
spec:
  terminationPolicy: WipeOut
  componentSpecs:
    - name: mongodb
      componentDef: mongodb
      serviceVersion: 7.0.12
      replicas: 3
      resources:
        requests:
          cpu: 100m
          memory: 0.5Gi
        limits:
          cpu: 100m
          memory: 0.5Gi
      updateStrategy: BestEffortParallel
      volumeClaimTemplates:
        - name: data
          spec:
            storageClassName:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
EOF
  1. backup
kbcli cluster backup mongo-bopyvl --method datafile

kubectl get backup backup-default-mongo-bopyvl-20250227170014
NAME                                         POLICY                               METHOD     REPO                    STATUS      TOTAL-SIZE   DURATION   CREATION-TIME          COMPLETION-TIME        EXPIRATION-TIME
backup-default-mongo-bopyvl-20250227170014   mongo-bopyvl-mongodb-backup-policy   datafile   backuprepo-kbcli-test   Completed   272315       13s        2025-02-27T09:00:15Z   2025-02-27T09:00:27Z   
  1. restore
kbcli cluster restore mongo-backup --backup backup-default-mongo-bopyvl-20250227170014

➜  ~ kubectl get cluster mongo-backup         
NAME           CLUSTER-DEFINITION   VERSION   TERMINATION-POLICY   STATUS     AGE
mongo-backup                                  WipeOut              Creating   96s

➜  ~ kubectl get ops
NAME           TYPE      CLUSTER        STATUS    PROGRESS   AGE
mongo-backup   Restore   mongo-backup   Running   -/-        100s

➜  ~ kbcli cluster list-instances mongo-backup
NAME                     NAMESPACE   CLUSTER        COMPONENT   STATUS    ROLE      ACCESSMODE   AZ              CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE     NODE                      CREATED-TIME                 
mongo-backup-mongodb-0   default     mongo-backup   mongodb     Running   primary   <none>       cn-shanghai-b   100m / 100m          512Mi / 512Mi           data:20Gi   172.16.0.44/172.16.0.44   Feb 27,2025 17:01 UTC+0800   
mongo-backup-mongodb-1   default     mongo-backup   mongodb     Running   <none>    <none>       cn-shanghai-b   100m / 100m          512Mi / 512Mi           data:20Gi   172.16.0.7/172.16.0.7     Feb 27,2025 17:01 UTC+0800   
mongo-backup-mongodb-2   default     mongo-backup   mongodb     Running   <none>    <none>       cn-shanghai-b   100m / 100m          512Mi / 512Mi           data:20Gi   172.16.0.37/172.16.0.37   Feb 27,2025 17:01 UTC+0800   
  1. delete restore cluster
kbcli cluster delete mongo-backup --auto-approve 
Cluster mongo-backup deleted
  1. See error
➜  ~ kubectl get ops
NAME           TYPE      CLUSTER        STATUS    PROGRESS   AGE
mongo-backup   Restore   mongo-backup   Running   -/-        2m26s
➜  ~ 
➜  ~ kubectl get cluster mongo-backup 
NAME           CLUSTER-DEFINITION   VERSION   TERMINATION-POLICY   STATUS     AGE
mongo-backup                                  WipeOut              Deleting   2m34s

logs kubeblocks

2025-02-27T09:03:27.074Z	ERROR	delete *v1alpha1.InstanceSet error: mongo-backup-mongodb	{"controller": "instanceset", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "InstanceSet", "InstanceSet": {"name":"mongo-backup-mongodb","namespace":"default"}, "namespace": "default", "name": "mongo-backup-mongodb", "reconcileID": "b02a1775-439f-4ba7-8326-3c090f0d7404", "InstanceSet": {"name":"mongo-backup-mongodb","namespace":"default"}, "error": "Operation cannot be fulfilled on instancesets.workloads.kubeblocks.io \"mongo-backup-mongodb\": StorageError: invalid object, Code: 4, Key: /ccs6u8ohu7elrn2npa5j0/workloads.kubeblocks.io/instancesets/default/mongo-backup-mongodb, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 26d53d4b-3e5e-4b35-b10f-f5fd6c27073b, UID in object meta: "}
2025-02-27T09:03:27.162Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"mongo-backup","namespace":"default"}, "namespace": "default", "name": "mongo-backup", "reconcileID": "3e27d61c-e9e7-48a8-9e6e-bb0890968b9d", "opsRequest": {"name":"mongo-backup","namespace":"default"}, "opsRequest": {"name":"mongo-backup","namespace":"default"}}
2025-02-27T09:03:27.173Z	INFO	deleting the sub-resource kinds: [OpsRequest]	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"mongo-backup","namespace":"default"}, "namespace": "default", "name": "mongo-backup", "reconcileID": "6363367c-124b-4609-9d95-5790b59fc67a", "cluster": {"name":"mongo-backup","namespace":"default"}}
2025-02-27T09:03:27.786Z	INFO	deleting the sub-resource kinds: [OpsRequest]	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"mongo-backup","namespace":"default"}, "namespace": "default", "name": "mongo-backup", "reconcileID": "b92a8537-0f3e-48ea-9210-388870697b67", "cluster": {"name":"mongo-backup","namespace":"default"}}
2025-02-27T09:03:28.823Z	INFO	deleting the sub-resource kinds: [OpsRequest]	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"mongo-backup","namespace":"default"}, "namespace": "default", "name": "mongo-backup", "reconcileID": "0428b3ac-a390-4b68-9c3f-496f6352a52c", "cluster": {"name":"mongo-backup","namespace":"default"}}
2025-02-27T09:03:29.859Z	INFO	deleting the sub-resource kinds: [OpsRequest]	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"mongo-backup","namespace":"default"}, "namespace": "default", "name": "mongo-backup", "reconcileID": "5c477060-d30c-44a3-b14a-04e87bb6612e", "cluster": {"name":"mongo-backup","namespace":"default"}}
2025-02-27T09:03:30.896Z	INFO	deleting the sub-resource kinds: [OpsRequest]	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"mongo-backup","namespace":"default"}, "namespace": "default", "name": "mongo-backup", "reconcileID": "f2fd73c6-d511-4de4-8866-a103e0dd9948", "cluster": {"name":"mongo-backup","namespace":"default"}}
2025-02-27T09:03:31.934Z	INFO	deleting the sub-resource kinds: [OpsRequest]	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"mongo-backup","namespace":"default"}, "namespace": "default", "name": "mongo-backup", "reconcileID": "64ba12b3-96b4-4b20-9d0c-c81c5293c39c", "cluster": {"name":"mongo-backup","namespace":"default"}}
2025-02-27T09:03:32.973Z	INFO	deleting the sub-resource kinds: [OpsRequest]	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"mongo-backup","namespace":"default"}, "namespace": "default", "name": "mongo-backup", "reconcileID": "be0f4a04-156e-4348-8a8a-d4de0fadb809", "cluster": {"name":"mongo-backup","namespace":"default"}}
2025-02-27T09:03:34.011Z	INFO	deleting the sub-resource kinds: [OpsRequest]	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"mongo-backup","namespace":"default"}, "namespace": "default", "name": "mongo-backup", "reconcileID": "3dcf546e-9c06-4ece-949f-faf6605f1496", "cluster": {"name":"mongo-backup","namespace":"default"}}
2025-02-27T09:03:35.049Z	INFO	deleting the sub-resource kinds: [OpsRequest]	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"mongo-backup","namespace":"default"}, "namespace": "default", "name": "mongo-backup", "reconcileID": "e1179c8b-4a8d-4e30-854e-8958f45e93d9", "cluster": {"name":"mongo-backup","namespace":"default"}}
2025-02-27T09:03:36.086Z	INFO	deleting the sub-resource kinds: [OpsRequest]	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"mongo-backup","namespace":"default"}, "namespace": "default", "name": "mongo-backup", "reconcileID": "d6ab5aa4-fbdf-450c-a98a-1e6ee4464574", "cluster": {"name":"mongo-backup","namespace":"default"}}
2025-02-27T09:03:37.123Z	INFO	deleting the sub-resource kinds: [OpsRequest]	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"mongo-backup","namespace":"default"}, "namespace": "default", "name": "mongo-backup", "reconcileID": "a4f7df16-6011-4bbd-8526-4e32d7393b38", "cluster": {"name":"mongo-backup","namespace":"default"}}
2025-02-27T09:03:38.161Z	INFO	deleting the sub-resource kinds: [OpsRequest]	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"mongo-backup","namespace":"default"}, "namespace": "default", "name": "mongo-backup", "reconcileID": "838cc473-7979-436c-9b76-15ea4a76ea62", "cluster": {"name":"mongo-backup","namespace":"default"}}
2025-02-27T09:03:39.199Z	INFO	deleting the sub-resource kinds: [OpsRequest]	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"mongo-backup","namespace":"default"}, "namespace": "default", "name": "mongo-backup", "reconcileID": "6cabfeb7-833d-483d-bdf4-fb8510d2e678", "cluster": {"name":"mongo-backup","namespace":"default"}}
2025-02-27T09:03:40.237Z	INFO	deleting the sub-resource kinds: [OpsRequest]	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"mongo-backup","namespace":"default"}, "namespace": "default", "name": "mongo-backup", "reconcileID": "f6287c15-4aa0-41fe-87b6-2e175cc09681", "cluster": {"name":"mongo-backup","namespace":"default"}}
2025-02-27T09:03:41.273Z	INFO	deleting the sub-resource kinds: [OpsRequest]	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"mongo-backup","namespace":"default"}, "namespace": "default", "name": "mongo-backup", "reconcileID": "f9e707d2-c4a0-4061-bd16-2b21646a7895", "cluster": {"name":"mongo-backup","namespace":"default"}}
2025-02-27T09:03:42.310Z	INFO	deleting the sub-resource kinds: [OpsRequest]	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"mongo-backup","namespace":"default"}, "namespace": "default", "name": "mongo-backup", "reconcileID": "b3833d15-306c-451b-8095-5fa9e937cde7", "cluster": {"name":"mongo-backup","namespace":"default"}}
2025-02-27T09:03:43.348Z	INFO	deleting the sub-resource kinds: [OpsRequest]	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"mongo-backup","namespace":"default"}, "namespace": "default", "name": "mongo-backup", "reconcileID": "bbf4b269-cfbd-452d-9d27-43740ffe84bb", "cluster": {"name":"mongo-backup","namespace":"default"}}
2025-02-27T09:03:44.387Z	INFO	deleting the sub-resource kinds: [OpsRequest]	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"mongo-backup","namespace":"default"}, "namespace": "default", "name": "mongo-backup", "reconcileID": "a25f4cca-06d3-44b9-9cc7-2b178dc0a1f7", "cluster": {"name":"mongo-backup","namespace":"default"}}
2025-02-27T09:03:45.425Z	INFO	deleting the sub-resource kinds: [OpsRequest]	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"mongo-backup","namespace":"default"}, "namespace": "default", "name": "mongo-backup", "reconcileID": "d575fb1b-a7cf-4ce9-b64e-302cab9763dc", "cluster": {"name":"mongo-backup","namespace":"default"}}
2025-02-27T09:03:46.464Z	INFO	deleting the sub-resource kinds: [OpsRequest]	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"mongo-backup","namespace":"default"}, "namespace": "default", "name": "mongo-backup", "reconcileID": "fe3df60a-c5b1-4cd1-bb20-64492c794daf", "cluster": {"name":"mongo-backup","namespace":"default"}}
2025-02-27T09:03:47.505Z	INFO	deleting the sub-resource kinds: [OpsRequest]	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"mongo-backup","namespace":"default"}, "namespace": "default", "name": "mongo-backup", "reconcileID": "637ed389-2613-4a9c-81cc-870c97083f66", "cluster": {"name":"mongo-backup","namespace":"default"}}
2025-02-27T09:03:48.544Z	INFO	deleting the sub-resource kinds: [OpsRequest]	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"mongo-backup","namespace":"default"}, "namespace": "default", "name": "mongo-backup", "reconcileID": "e3b59f20-a9ad-436f-b52b-7cbc84a0b319", "cluster": {"name":"mongo-backup","namespace":"default"}}
2025-02-27T09:03:49.581Z	INFO	deleting the sub-resource kinds: [OpsRequest]	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"mongo-backup","namespace":"default"}, "namespace": "default", "name": "mongo-backup", "reconcileID": "41f0fcca-078a-412e-a143-97ca47a4d4b7", "cluster": {"name":"mongo-backup","namespace":"default"}}
2025-02-27T09:03:50.619Z	INFO	deleting the sub-resource kinds: [OpsRequest]	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"mongo-backup","namespace":"default"}, "namespace": "default", "name": "mongo-backup", "reconcileID": "e54bfdc5-0355-49d2-82e9-0d93254d8720", "cluster": {"name":"mongo-backup","namespace":"default"}}
2025-02-27T09:03:51.654Z	INFO	deleting the sub-resource kinds: [OpsRequest]	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"mongo-backup","namespace":"default"}, "namespace": "default", "name": "mongo-backup", "reconcileID": "6af88196-b38e-40a5-9b4c-82bdac245281", "cluster": {"name":"mongo-backup","namespace":"default"}}
2025-02-27T09:03:52.692Z	INFO	deleting the sub-resource kinds: [OpsRequest]	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"mongo-backup","namespace":"default"}, "namespace": "default", "name": "mongo-backup", "reconcileID": "8a83e750-4970-4cd2-999a-52271c04afc5", "cluster": {"name":"mongo-backup","namespace":"default"}}
2025-02-27T09:03:53.729Z	INFO	deleting the sub-resource kinds: [OpsRequest]	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"mongo-backup","namespace":"default"}, "namespace": "default", "name": "mongo-backup", "reconcileID": "51a97fa9-b79c-4c83-b6a7-a22b87ce88d7", "cluster": {"name":"mongo-backup","namespace":"default"}}
2025-02-27T09:03:54.768Z	INFO	deleting the sub-resource kinds: [OpsRequest]	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"mongo-backup","namespace":"default"}, "namespace": "default", "name": "mongo-backup", "reconcileID": "92e38581-2db2-4731-8360-77941330a3ef", "cluster": {"name":"mongo-backup","namespace":"default"}}
2025-02-27T09:03:55.806Z	INFO	deleting the sub-resource kinds: [OpsRequest]	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"mongo-backup","namespace":"default"}, "namespace": "default", "name": "mongo-backup", "reconcileID": "7c1161fb-0036-4a91-901d-0a217697e691", "cluster": {"name":"mongo-backup","namespace":"default"}}
2025-02-27T09:03:56.845Z	INFO	deleting the sub-resource kinds: [OpsRequest]	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"mongo-backup","namespace":"default"}, "namespace": "default", "name": "mongo-backup", "reconcileID": "ce126d40-c545-4c7b-a5aa-f28a2c5a92be", "cluster": {"name":"mongo-backup","namespace":"default"}}

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

@JashBook JashBook added the kind/bug Something isn't working label Feb 27, 2025
@JashBook JashBook added this to the Release 0.9.4 milestone Feb 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants