Zero Downtime Django (gunicorn) Deployments on GKE
Tags: django, gunicorn
Published: 2024-01-29 22:15:30

---
title: Zero Downtime Django (gunicorn) Deployments on GKE
published: true
date: 2024-01-06 05:39:46 UTC
tags: ["django", "gunicorn"]
canonical_url: https://developerzen.com/zero-downtime-django-gunicorn-deployments-on-gke/
---

We recently switched to [Twingate’s](https://www.twingate.com) GKE load balancer to use Google’s new [Container-native load balancer](https://cloud.google.com/kubernetes-engine/docs/concepts/container-native-load-balancing "Container-native load balancing"). The premise was good - LB talks directly to pods and saves an extra network hops, (with classic LB, traffic goes from LB to a GKE node which then, based on `iptables` configured by [kube-proxy](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/), get routed to the pod) and should perform better, support more features, and in general we’d rather be on google’s maintained side and not on legacy tech.

However, immediately after making the switch, we started noticing short bursts of 502 errors whenever we’d deploy a new release of our services to the cluster. We tracked it down to the following behavior described in the [Container-native load balancing through Ingress docs](https://cloud.google.com/kubernetes-engine/docs/how-to/container-native-load-balancing#traffic_does_not_reach_endpoints "Google Docs: Traffic does not reach endpoints"):

> 502 errors and rejected connections can also be caused by a container that doesn’t handle SIGTERM.  
> If a container doesn’t explicitly handle SIGTERM, it immediately terminates and stops handling requests. The load balancer continues to send incoming traffic to the terminated container, leading to errors.

## Why do we get 502s on pod restarts?  [#](#why-do-we-get-502s-on-pod-restarts "Link to section: Why do we get 502s on pod restarts?")

The legacy load balancer relied on Kubernetes’s [kube-proxy](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/) to do the routing.  
[kube-proxy](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/) configures the `iptables` on all the cluster’s node with rules on how to distribute traffic to nodes.  
When the load balancer receives a request, it sends it to a random node on the cluster which then routes it to the pod (which might be on a different node).  
[kube-proxy](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/) is aware of the different pod’s states and when a pod changed state to `Terminating` it immediately updates the routing information.

With Container-native load balancing, traffic is routed directly to pods.  
This eliminates the extra networking hop but at a cost that it is not aware of the pods state and relies on healthchecks to know when a pod is terminating.

We were getting these 502s bursts because once we deployed a new version, old pods were being terminated and when receiving SIGTERM they’d stop processing new requests. The load balancer, however, would still send them requests until healthcheck fails (it was set to 10s in our case) and it removes it from circulation.

To solve this we need to be able to gracefully terminate our pods - we need some sort of a toggle to tell the pod to start failing its healthcheck while it continues processing other requests regularly for enough time for the load balancer to stop sending traffic its way.

In order to address this issue, we must find a way to gracefully terminate our pods.  
This requires some kind of switch that instructs the pod to begin failing its health check, while simultaneously maintaining regular processing of other requests for enough time to allow the load balancer to mark the pod as done and stop sending traffic its way.

To understand how to do this, lets first take a step back and understand Kubernetes’s process for terminating pods…

### Whats the termination process for Kubernetes Pod

#### 1. Pod is set to “Terminating” state

The pod is then removed from endpoints list of all services and [kube-proxy](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/) updates routing rules on all nodes so that they shouldn’t receive traffic.

#### 2. _preStop_ Hook is called

The [preStop Hook](https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#hook-details) is a command executed on the containers in the pod.

#### 3. SIGTERM signal is sent to pod

Kubernetes sends a SIGTERM to the containers in the pod to let them know they need to shut down soon.

#### 4. Kubernetes waits for containers to gracefully terminate

Kubernetes wait for a specified time, called _termination grace period_ for containers to gracefully terminate. By default, this period is set to _30 seconds_ but it can be customized by setting `terminationGracePeriodSeconds` value as part of the pod spec:

```
apiVersion: v1
kind: Pod
metadata:
 name: example-pod
spec:
 terminationGracePeriodSeconds: 60
 containers:
 - name: app
 image: busybox

```

#### 5. SIGKILL signal is sent to pod and it’s removed

If containers are still running after the grace period, they are sent SIGKILL signal and are forcibly removed. Kubernetes then cleans up its objects store.

## Gracefully Terminating Django (gunicorn)  [#](#gracefully-terminating-django-gunicorn "Link to section: Gracefully Terminating Django (gunicorn)")

Gunicorn has its own definition for [graceful timeout](https://docs.gunicorn.org/en/stable/settings.html#graceful-timeout) - when it receives a SIGTERM it will give workers a grace period (_30s_ by default) to finish processing the _current request_ they’re processing and exit. In our case we need gunicorn to continue serving requests for some time before shutting the worker down:

1. When pod is terminating, toggle health check (we’re using `/health`) view
2. Wait for 25 seconds (We set the LB to healthcheck every 5s and consider a pod down after 2 consecutive failures so 25s should give it enough time to fail)
3. Send SIGTERM to gunicorn

The simplest way to signal Django to start failing the healthcheck is by using a file - `/tmp/shutdown` - if the file exists we should start failing the healthcheck.  
(We can’t use a variable and\or http call because gunicorn runs multiple workers and doing some multiprocess memory sharing magic is too complex)

So the detailed graceful shutdown process is as follows:

1. Kubernetes sets pod to “Terminating state”
2. Kubernetes calls preStop hook 2.1. Create a `/tmp/shutdown` file 2.2. Sleep for `25s` - enough time for load balancer to refresh
3. Kubernetes sends SIGTERM to container and gunicorn shuts down workers

Our `preStop` hook is pretty simple: (Note that our LB are configured to healthcheck every _5s_ and remove target if it fails twice so we need to sleep for _at least 10s_ to make sure pod is removed. These settings may differ on your system…)

```
lifecycle:
 preStop:
 exec:
 command:
 - sh
 - -c
 - echo "shutting down - $(date +%s)" >> /tmp/shutdown && sleep 25

```

Our Django healthcheck view:

```
SHUTDOWN_FILE = "/tmp/shutdown" # nosec

def is_shutting_down() -> bool:
 return os.path.exists(SHUTDOWN_FILE)

@internal_only_view
def health_check(_request):
 if is_shutting_down():
 return HttpResponse("Shutting Down...", status=503)

 ... Some extra healthcheck logic ...
 
 return HttpResponse("OK")

```

## References  [#](#references "Link to section: References")

- [External Application Load Balancer overview](https://cloud.google.com/load-balancing/docs/https) - Google classic load balancer vs. the new container-native ones
- [Kubernetes preStop Hook](https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#hook-details)
- [Container-native load balancing](https://cloud.google.com/kubernetes-engine/docs/concepts/container-native-load-balancing "Container-native load balancing")