EKS cluster upgrade fail with Kubelet version of Fargate pods must be updated to match cluster version!!
Tags: eks, upgrade, fargate, kubernetes
Published: 2023-12-30 17:04:24


![EKS Cluster - Could not update cluster version](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0ofc2hy5tavjj6b4q91q.png)

You might encounter the following error message within the AWS EKS console when attempting to upgrade the Kubernetes version of your cluster.

> Kubelet version of Fargate pods must be updated to match cluster version 1.25 before updating cluster version; Please recycle all offending pod replicas.

For instance, if you are in the process of incrementally upgrading the Kubernetes version from 1.24 to 1.27.

The upgrade from version 1.24 to 1.25 will proceed without encountering errors. However, please note that it won’t automatically update the kubelet version of the Fargate nodes/Pods until a restart is performed.

![AWS EKS Console (Kubernetes version 1.25)](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/id6othycyqfnk2t2e35t.png)

But when you check Fargate nodes versions,
![Fargate Node details](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/088pwpeuqo6reht93kol.png)

Or when you check from terminal,

```
# To list all nodes
kubectl get nodes
```

![Fargate Nodes details — Terminal](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qwvilrvk4916clon2iij.png)

**Solutions..**

If the Fargate pods’ versions are not compatible with the upgraded EKS cluster, you might need to recreate the pods using a compatible version.

- You can do this by updating the deployment or stateful set YAML files with the correct version and applying the changes to the cluster. Kubernetes will automatically create new pods with the updated version.

- You can get all deployment, statefulset from all namespaces and restart it as shown below.

```
# To list all deployments
kubectl get deployment -A

# To list all deployments
kubectl get statefulset -A

# To restart deployments
kubectl rollout restart deployment coredns -n kube-system
kubectl rollout restart deployment metrics-server -n kube-system
kubectl rollout restart deployment aws-load-balancer-controller -n kube-system
kubectl rollout restart deployment ebs-csi-controller -n kube-system

# To restart statefulset
kubectl rollout restart statefulset adot-collector -n fargate-container-insights
```
Please note that this solution will work exclusively with Fargate nodes. If you are using other cluster modes, please refer to the links provided at the end of this article for alternative solutions.

---
Also don’t forget to update kubectl as well.

```
# To check kubectl version
kubectl version --short --client
```
Link to update kubectl:
[Installing or updating kubectl](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html?source=post_page-----7fb0403bf9b6--------------------------------)

For more details about Amazon EKS Kubernetes versions upgrade:
[Amazon EKS Kubernetes versions](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html?source=post_page-----7fb0403bf9b6--------------------------------)
[Updating an Amazon EKS cluster Kubernetes version](https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html?source=post_page-----7fb0403bf9b6--------------------------------)

---
I’ll always strive to provide simple explanations. If you found this article useful, follow me for more similar articles in the future. Your support keeps me motivated to create content like this. :handshake::blush: