How do you gracefully shut down Pods in Kubernetes?
Tags: kubernetes, devops
Published: 2023-02-20 13:16:07

When you type `kubectl delete pod`, the pod is deleted, and the endpoint controller removes its IP address and port (endpoint) from the Services and etcd.

You can observe this with `kubectl describe service`.

![Listing endpoints with kubectl describe](https://res.cloudinary.com/learnk8s/image/upload/v1676872994/threads/graceful-shutdown-2_l2gafi.png)

But that's not enough!

**Several components sync a local list of endpoints:**

- kube-proxy keeps a local list of endpoints to write iptables rules.
- CoreDNS uses the endpoint to reconfigure the DNS entries.

And the same is true for the Ingress controller, Istio, etc.

![Endpoints propagation in Kubernetes](https://res.cloudinary.com/learnk8s/image/upload/v1676872994/threads/graceful-shutdown-3_oxakmq.png)

All those components will (eventually) remove the previous endpoint so that no traffic can ever reach it again.

At the same time, the kubelet is also notified of the change and deletes the pod.

_What happens when the kubelet deletes the pod before the rest of the components?_

![Endpoints are not propagated and removed at the same time](https://res.cloudinary.com/learnk8s/image/upload/v1676872994/threads/graceful-shutdown-4_m1pjvc.png)

**Unfortunately, you will experience downtime** because components such as kube-proxy, CoreDNS, the ingress controller, etc., still use that IP address to route traffic.

_So what can you do?_

Wait!

![The kubelet will immediately delete the pod, even if the endpoint is not propagated](https://res.cloudinary.com/learnk8s/image/upload/v1676872994/threads/graceful-shutdown-5_dkdoaw.png)

**If you wait long enough before deleting the pod, the in-flight traffic can still resolve, and the new traffic can be assigned to other pods.**

_How are you supposed to wait?_

![The kubelet should wait for the endpoints to propagate before deleting the pod](https://res.cloudinary.com/learnk8s/image/upload/v1676872995/threads/graceful-shutdown-6_oezur9.png)

When the kubelet deletes a pod, it goes through the following steps:

- Triggers the `preStop` hook (if any).
- Sends the SIGTERM.
- Sends the SIGKILL signal (after 30 seconds).

![The kubelet deleting the pod goes through 3 steps: preStop hook, SIGTERM and SIGKILL](https://res.cloudinary.com/learnk8s/image/upload/v1676872995/threads/graceful-shutdown-7_ijesks.png)

**You can use the `preStop` hook to insert an artificial delay.**

![You can use a preStop hook to delay deleting a pod](https://res.cloudinary.com/learnk8s/image/upload/v1676872995/threads/graceful-shutdown-8_mocv2u.png)

**You can listen to the SIGTERM signal in your app and wait.**

Also, you can gracefully stop the process and exit when you are done waiting.

Kubernetes gives you 30s to do so (configurable).

![You can catch the SIGTERM signal in your app and wait](https://res.cloudinary.com/learnk8s/image/upload/v1676872995/threads/graceful-shutdown-9_ddmpkm.png)

_Should you wait 10 seconds, 20 or 30s?_

There's no single answer.

While propagating endpoints could only take a few seconds, Kubernetes doesn't guarantee any timing nor that all of the components will complete it at the same time.

![Endpoint propagation timeline in Kubernetes](https://res.cloudinary.com/learnk8s/image/upload/v1676872995/threads/graceful-shutdown-10_n6swjw.png)

If you want to explore more, here are a few links:

- https://learnk8s.io/graceful-shutdown
- https://freecontent.manning.com/handling-client-requests-properly-with-kubernetes/
- https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods
- https://medium.com/tailwinds-navigator/kubernetes-tip-how-to-gracefully-handle-pod-deletion-b28d23644ccc
- https://medium.com/flant-com/kubernetes-graceful-shutdown-nginx-php-fpm-d5ab266963c2
- https://www.openshift.com/blog/kubernetes-pods-life

And finally, if you've enjoyed this thread, you might also like:

- The Kubernetes workshops that we run at Learnk8s https://learnk8s.io/training
- This collection of past threads https://twitter.com/danielepolencic/status/1298543151901155330
- The Kubernetes newsletter I publish every week, "Learn Kubernetes weekly" https://learnk8s.io/learn-kubernetes-weekly