Kubernetes – Kubelet Unable to attach or mount volumes – timed out waiting for the condition
Tags: kubernetes
Published: 2021-10-11 10:07:58

---
title: Kubernetes – Kubelet Unable to attach or mount volumes – timed out waiting for the condition
published: true
date: 2021-09-30 11:40:00 UTC
tags: Kubernetes
canonical_url: https://veducate.co.uk/kubelet-unable-attach-volumes/#utm_source=rss&utm_medium=rss&utm_campaign=kubelet-unable-attach-volumes
---

###### The Issue

When I updated my Kasten application in my Kubernetes cluster, I found that one of the pods was stuck in “init” status.

```
dean@dean [~] (⎈ |tkg-wld-01-admin@tkg-wld-01:default) # k get pods -n kasten-io -w
NAME READY STATUS RESTARTS AGE
aggregatedapis-svc-78564d4697-wl9wg 1/1 Running 0 3m9s
auth-svc-7977b9684b-zph27 1/1 Running 0 3m11s
catalog-svc-7ff7779b75-kmvsr 0/2 Init:0/2 0 2m43s
```

[![kubectl get pods - status init](https://i0.wp.com/veducate.co.uk/wp-content/uploads/2021/09/kubectl-get-pods-status-init.jpg?resize=602%2C104&ssl=1)](https://i0.wp.com/veducate.co.uk/wp-content/uploads/2021/09/kubectl-get-pods-status-init.jpg?ssl=1)

Running a describe on that pod pointed to the fact the volume could not be attached.

```
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m58s default-scheduler Successfully assigned kasten-io/catalog-svc-7ff7779b75-kmvsr to tkg-wld-01-md-0-54598b8d99-rpqjf
Warning FailedMount 55s kubelet Unable to attach or mount volumes: unmounted volumes=[catalog-persistent-storage], unattached volumes=[k10-k10-token-lbqpw catalog-persistent-storage]: timed out waiting for the condition
```

###### [![kubelet Unable to attach or mount volumes- unmounted volumes=[catalog-persistent-storage], unattached volumes=[k10-k10-token-lbqpw catalog-persistent-storage]- timed out waiting for the condition](https://i1.wp.com/veducate.co.uk/wp-content/uploads/2021/09/kubelet-Unable-to-attach-or-mount-volumes-unmounted-volumescatalog-persistent-storage-unattached-volumesk10-k10-token-lbqpw-catalog-persistent-storage-timed-out-waiting-for-the-condition.jpg?resize=604%2C43&ssl=1)](https://i1.wp.com/veducate.co.uk/wp-content/uploads/2021/09/kubelet-Unable-to-attach-or-mount-volumes-unmounted-volumescatalog-persistent-storage-unattached-volumesk10-k10-token-lbqpw-catalog-persistent-storage-timed-out-waiting-for-the-condition.jpg?ssl=1)

###### The Cause

Some where along the line I found some stale [volumeattachments](https://docs.openshift.com/container-platform/4.8/rest_api/storage_apis/volumeattachment-storage-k8s-io-v1.html)linked to Kubernetes node that no longer exist in my cluster. This looks to be causing some confusion in the cluster who should be attaching the volume

The image below shows:

- Find the Persistent Volume name linked to the associated claim for the failure in the pod events
- Map this to the available VolumeAttachments
- Reference VolumeAttachments for each node to available nodes in the cluster
  - I’ve highlighted the missing node in the red box

[![kubectl get pv - get volumeattachment - get nodes](https://i2.wp.com/veducate.co.uk/wp-content/uploads/2021/09/kubectl-get-pv-get-volumeattachment-get-nodes.jpg?resize=604%2C234&ssl=1)](https://i2.wp.com/veducate.co.uk/wp-content/uploads/2021/09/kubectl-get-pv-get-volumeattachment-get-nodes.jpg?ssl=1)

###### The Fix

The fix is to remove the stale VolumeAttachment.

```
kubectl delete volumeattachment [volumeattachment_name]
```

[![kubectl delete volumeattachment](https://i2.wp.com/veducate.co.uk/wp-content/uploads/2021/09/kubectl-delete-volumeattachment.jpg?resize=604%2C28&ssl=1)](https://i2.wp.com/veducate.co.uk/wp-content/uploads/2021/09/kubectl-delete-volumeattachment.jpg?ssl=1)

After this your pod should eventually pick up and retry, or you could remove the pod and let Kubernetes replace it for you (so long as it’s part of a deployment or other configuration managing your application).

Regards

[Follow @Saintdle](https://twitter.com/saintdle)<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

[Dean Lewis](https://uk.linkedin.com/in/saintdle?trk=profile-badge)

The post [Kubernetes – Kubelet Unable to attach or mount volumes – timed out waiting for the condition](https://veducate.co.uk/kubelet-unable-attach-volumes/) appeared first on [vEducate.co.uk](https://veducate.co.uk).