Nginx Ingress Controller Load Balancing and Retry with proxy_next_upstream
Tags: devops, nginx, kubernetes, ingressnginx
Published: 2020-01-20 14:57:42

---
title: Nginx Ingress Controller Load Balancing and Retry with proxy_next_upstream
published: true
description: Nginx configuration that makes sense with proxy_next_upstream
tags: devops,nginx,kubernetes,ingress-nginx
---

By default, Nginx/Nginx Ingress provides [sensible default settings](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#configuration-options). In the real world, there are a few combinations of configurations that you must deliberately change according to different upstreams (your backend apps) behaviour.

## Log format

Structured logging is always better if JSONified. This is documented [here](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/log-format/).

```yaml
log-format-upstream: '{"ts": "$time_iso8601", "requestID": "$req_id", "upstreamAddr": "$upstream_addr", "proxyUpstreamName": "$proxy_upstream_name", "proxyAlternativeUpstreamName": "$proxy_alternative_upstream_name", "requestMethod": "$request_method", "requestUrl": "$host$uri?$args", "status": $status, "upstreamStatus": "$upstream_status", "requestSize": "$request_length", "responseSize": "$upstream_response_length", "userAgent": "$http_user_agent", "remoteIp": "$remote_addr", "referer": "$http_referer", "latency": "$upstream_response_time"}'
```

Unfortunately, error logs can not be customised in Nginx.


## The power of [proxy-next-upstream](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream)


In Kubernetes, upstreams are Pod IPs from each Kubernetes Services’ Endpoints in your Ingress configurations. They are watched and fetched periodically from the Kubernetes API. This way, a controller does not need to reload its `nginx.conf` every time an upstream Pod is restarted or deleted. See [Avoiding reloads](https://github.com/kubernetes/ingress-nginx/blob/master/docs/how-it-works.md#avoiding-reloads) for reference. This means that Nginx Ingress Controller does not depend on DNS.

### Example Scenario


Now, what if a request hits an Nginx Pod upstream that is being restarted or was deleted? Obviously, we expect Nginx to load balance properly to the next available Pod right? Yes, this is how it works.


- A Pod with IP `10.8.9.11` is deleted.
- Nginx tries to connect to this IP until the `proxy_connect_timeout`.
- Nginx will then look at the `proxy_next_upstream` settings and see if the error matches any of its value. If there is a match, the request is retried to the **next upstream** (other IPs from the Kubernetes Service endpoints).


**Retry configurations**


config | custom | default
--- | --- | ---
proxy-connect-timeout | 3 (seconds) | 5 (seconds)
proxy-next-upstream | error timeout http_502 http_503 http_504 | error timeout
proxy-next-upstream-tries | 3 | 3
proxy-next-upstream-timeout | 0 (no limit) | 0 (no limit)


**Gotchas**

If you are going to customise `proxy-next-upstream-timeout`, the value should be greater than the `proxy-connect-timeout`.


**Observation**

The first error logs.

```
2020/01/20 13:20:48 [error] 2224#2224: *133630 connect() failed (111: Connection refused) while connecting to upstream, client: 10.8.9.11 ...
2020/01/20 13:20:48 [error] 2224#2224: *133630 connect() failed (111: Connection refused) while connecting to upstream, client: 10.8.9.11 ...
2020/01/20 13:20:48 [error] 2224#2224: *133630 connect() failed (111: Connection refused) while connecting to upstream, client: 10.8.9.11 ...

... 3 seconds later ...

13:20:56 [error] 2190#2190: *133679 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.8.9.11...
```

And then finally, Nginx still managed to make the request successful.

You should see something in the logs like.

```json
{
  "upstreamAddr": "10.8.9.11:8080, 10.8.8.2:8080",
  "status": 200,
  "upstreamStatus": "504, 200",
  "latency": "3.001, 0.005"
}
```


Pod IP | status code | latency
--- | --- | ---
10.8.9.11 | 504 (failed) | 3.001
10.8.8.2 | 200 | 0.005


**Summary**


* The `$upstream_addr` displays the upstream Pod IPs used during the retry. This means that in a single request context, multiple Pod IPs can be used.
* `$upstream_status` shows that in the first upstream the response code is 504 which is included in our `proxy_next_upstream` settings as `http_504`.
* `$upstream_response_time` shows the response time for each upstream that was used. The first one is 3 seconds which is also our `proxy_connect_timeout` duration.


### Known Issues


Sometimes, identical IPs are used in the `$upstream_addr`. See https://github.com/kubernetes/ingress-nginx/issues/4944#issuecomment-575754005.