☸️ Kubernetes NGINX Ingress Controller: 10+ Complementary Configurations for Web Applications
Tags: nginx, kubernetes, devops, cloud
Published: 2023-10-16 07:55:12

---
title: '☸️ Kubernetes NGINX Ingress Controller: 10+ Complementary Configurations for Web Applications'
tags:
  - nginx
  - kubernetes
  - devops
  - cloud
license: public-domain
cover_image: 'https://raw.githubusercontent.com/bcouetil/articles/main/images/kubernetes/boat/nginx-1.jpg'
published: true
id: 1500484
date: '2023-10-16T07:55:12Z'
---

<!--
TODO:

Ajouter la taille des headers:

2024/01/19 14:52:52 [error] 5943#5943: *185755772 upstream sent too big header while reading response header from upstream, client: 152.228.196.162, server: app.staging.pleenk.com, request: "GET /payment?pw_beneficiary_0=0045adf7-73e7-4203-bdea-6a98a1b4f434|12&pw_callback=https://marketplace-app.staging.pleenk.com/callback&pw_description=&pw_funds_blocking_period=PT1M&pw_marketplace_fee=5.5&pw_metadata=&pw_privacy_scope=515238e9-8b6c-475e-8d0c-738874d32c8b&pw_transaction_ref=123&signature=MIGHAkEpf6dGjMb8AvOUrPjQzO3SQK7ru_AqVKBiIpSGlsU-eb5qmYjxLE4KLYBV_ktqXsjfOz-tLbT6LXMrRUS91v06HgJCAXyV-Uw14rjrNiVUnIEepwLI7RiuJRyBf5r2BOS3kV0IZgBlb9XDwPuf4_XHeXACetq7z0e8B58ErNzj4CsCCmwG HTTP/2.0", upstream: "http://10.2.11.51:3000/payment?pw_beneficiary_0=0045adf7-73e7-4203-bdea-6a98a1b4f434%7C12&pw_callback=https://marketplace-app.staging.pleenk.com/callback&pw_description=&pw_funds_blocking_period=PT1M&pw_marketplace_fee=5.5&pw_metadata=&pw_privacy_scope=515238e9-8b6c-475e-8d0c-738874d32c8b&pw_transaction_ref=123&signature=MIGHAkEpf6dGjMb8AvOUrPjQzO3SQK7ru_AqVKBiIpSGlsU-eb5qmYjxLE4KLYBV_ktqXsjfOz-tLbT6LXMrRUS91v06HgJCAXyV-Uw14rjrNiVUnIEepwLI7RiuJRyBf5r2BOS3kV0IZgBlb9XDwPuf4_XHeXACetq7z0e8B58ErNzj4CsCCmwG", host: "app.staging.pleenk.com"

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: webapp-direct
  annotations:
    nginx.ingress.kubernetes.io/whitelist-source-range: $DIRECT_IP_AND_CIDR_WHITELIST
    nginx.ingress.kubernetes.io/proxy-buffer-size: "16k"
spec:
  rules:
    - host: $WEBAPP_DIRECT_URL # for CDN & co which does not manage to connect to IP with virtual hosts


-->

<!--
🚀 Découvrez 12 configurations complémentaires pour l'Ingress Controller NGINX dans Kubernetes ☸️

Dans ce nouvel article, nous partageons des astuces pour optimiser le déploiement d'applications Web sur Kubernetes. L'Ingress Controller NGINX est un outil puissant, et ces configurations avancées vous aideront à tirer le meilleur parti de cette technologie.

Si vous connaissez d'autres usages essentiels, n'hésitez pas à les partager, l'article continue de vivre 🤓

Lien vers l'article: https://lnkd.in/eNJE7Kvq

#kubernetes #nginx #cloud @zenika
-->

<!-- TOC start (generated with https://github.com/derlin/bitdowntoc) -->

- [Initial thoughts](#initial-thoughts)
- [Helm chart Installation/update](#helm-chart-installationupdate)
- [1. Set NGINX as the default ingress controller](#1-set-nginx-as-the-default-ingress-controller)
- [2. Set SSL/TLS termination on AWS load balancer](#2-set-ssltls-termination-on-aws-load-balancer)
- [3. Use professional error pages](#3-use-professional-error-pages)
- [4. Redirect users HTTP calls to HTTPS port](#4-redirect-users-http-calls-to-https-port)
- [5. Rewrite internal redirects from HTTP to HTTPS](#5-rewrite-internal-redirects-from-http-to-https)
- [6. Set a valid default certificate when TLS is terminated on NGINX Ingress Controller](#6-set-a-valid-default-certificate-when-tls-is-terminated-on-nginx-ingress-controller)
- [7. Allow large file transfer](#7-allow-large-file-transfer)
- [8. Autoscale the ingress controller](#8-autoscale-the-ingress-controller)
- [9. Stick user session to the same targeted pod](#9-stick-user-session-to-the-same-targeted-pod)
- [10. Have access to real client IP in applications](#10-have-access-to-real-client-ip-in-applications)
- [11. Set maintenance mode](#11-set-maintenance-mode)
- [12. Disable NGINX access logs for a particular service](#12-disable-nginx-access-logs-for-a-particular-service)
- [13. Whitelist IP for a particular Ingress](#13-whitelist-ip-for-a-particular-ingress)
- [Wrapping up](#wrapping-up)
- [Further reading](#further-reading)

<!-- TOC end -->

# Initial thoughts

Kubernetes is an open-source container orchestration platform used to manage and automate the deployment and scaling of containerized applications. It has gained popularity in recent years due to its ability to provide a consistent experience across different cloud providers and on-premises environments.

The [NGINX ingress controller](https://docs.nginx.com/nginx-ingress-controller/intro/overview/) is a production‑grade ingress controller that runs NGINX Open Source in a Kubernetes environment. The daemon monitors Kubernetes ingress resources to discover requests for services that require ingress load balancing.

![NGINX architecture](https://raw.githubusercontent.com/bcouetil/articles/main/images/kubernetes/nginx-architecture.png)

In this article, we will dig into the versatility and simplicity of this ingress controller to implement different common use cases. You will find some other ones in different articles (such as [Kubernetes NGINX Ingress: 10 Useful Configuration Options](https://loft.sh/blog/kubernetes-nginx-ingress-10-useful-configuration-options/)) but none of them has both described and regrouped the ones below, yet these are widely used for web applications in production.

These apply to multiple Cloud providers, at least AWS, GCP and OVHCloud, except when a specific Cloud provider is mentioned.

These are also fully compatible with each other, except when architecture differ (for example, TLS termination on load balancer versus termination on NGINX pods).

_As future experiences demand, we'll augment its content with additional use cases, ensuring its relevance continues to flourish_

# Helm chart Installation/update

Everything in the YAML snippets below — except for ingress configuration — relates to configuring the [NGINX ingress controller](https://github.com/nginxinc/kubernetes-ingress). This includes customizing the [default configuration](https://github.com/kubernetes/ingress-nginx/blob/master/charts/ingress-nginx/values.yaml).

To begin, make sure your Helm distribution is aware of the chart using this command:

```shell
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx && helm repo update
```

After preparing or updating your custom `nginx.helm.values.yml` file, deploy or update the Helm deployment using this command:

```shell
helm -n system upgrade --install ngx ingress-nginx/ingress-nginx --version 4.3.0 --create-namespace -f nginx.helm.values.yml
```

Replace 4.3.0 with the latest version found on [ArtifactHUB](https://artifacthub.io/packages/helm/ingress-nginx/ingress-nginx), and proceed according to your upgrade strategy.

![(green ship wheel), helm, tropical sea, sunny day, (green)](https://raw.githubusercontent.com/bcouetil/articles/main/images/kubernetes/boat/nginx-2.jpg)

# 1. Set NGINX as the default ingress controller

By default, you have to specify the class in each of your ingress:

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example
spec:
  ingressClassName: nginx
```

But if you have a single ingress controller in your cluster, just configure it to be the default:

_nginx.helm.values.yml_

```yaml
controller:
  ingressClass:
    create: true # default
    setAsDefaultIngress: true
```

No more need for the `ingressClassName` field. Ever.

# 2. Set SSL/TLS termination on AWS load balancer

By default with Kubernetes incoming traffic, SSL/TLS termination has to be handled by target application, one by one. Another application, another TLS termination handling, with certificate handling.

A simple yet powerful way of abstracting TLS handling is to terminate on load balancer, and have HTTP inside the cluster by default.

![lb tls termination](https://raw.githubusercontent.com/bcouetil/articles/main/images/kubernetes/lb-ssl-termination.png)

As a pre-requisite, you have to [request a public ACM certificate in AWS](https://repost.aws/knowledge-center/terminate-https-traffic-eks-acm).

Once you have the certificate ARN, use it in below configuration under `service.beta.kubernetes.io/aws-load-balancer-ssl-cert` annotation:

_nginx.helm.values.yml_

```yaml
controller:
  service:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:eu-west-1:94xxxxxxx:certificate/2c0c2512-a829-4dd5-bc06-b3yyyyy
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https" # If you don't specify this annotation, controller creates TLS listener for all the service ports
      service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
      service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
      service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
      service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
```

# 3. Use professional error pages

By default, NGINX ingress controller gives you neutral yet boring error pages:

![NGINX 404 error page](https://res.cloudinary.com/practicaldev/image/fetch/s--Ub7zlLav--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://bartoszgordon.com/gatsby-404-nginx-production.png)

These ones can be replaced with nice polished and animated ones, such as this one from [tarampampam's repository](https://github.com/tarampampam/error-pages):

![NGINX 404 error page](https://camo.githubusercontent.com/8b88a4e05cbe8ba1329d15dc429fbe5b5814193b74d547e961d9690e6ecf75b3/68747470733a2f2f6873746f2e6f72672f776562742f6c662f6c6e2f78382f6c666c6e783866757934726f66786a7533347474736b696a6473752e676966)

It has some nice side features, like automatic light/dark modes, and routing details displayable fore debugging purpose.

Examples from multiple themes are showcased [here](https://tarampampam.github.io/error-pages/) for everyone to choose from.

Once you found your theme, configure your favorite ingress controller:

_nginx.helm.values.yml_

```yaml
controller:
  config:
    custom-http-errors: 404,408,500,501,502,503,504,505

# Prepackaged default error pages from https://github.com/tarampampam/error-pages/wiki/Kubernetes-&-ingress-nginx
# multiple themes here: https://tarampampam.github.io/error-pages/
defaultBackend:
  enabled: true
  image:
    repository: ghcr.io/tarampampam/error-pages
    tag: 2.21 # latest as of 01/04/2023 here: https://github.com/tarampampam/error-pages/pkgs/container/error-pages
  extraEnvs:
    - name: TEMPLATE_NAME
      value: lost-in-space # one of: app-down, cats, connection, ghost, hacker-terminal, l7-dark, l7-light, lost-in-space, matrix, noise, shuffle
    - name: SHOW_DETAILS # Optional: enables the output of additional information on error pages
      value: "false"
```

# 4. Redirect users HTTP calls to HTTPS port

Once you have all your web routes configured to handled SSL/TLS/HTTPS, HTTP routes have no reason to be, and is even dangerous to keep, security-wise.

Instead of disabling the port, which can be annoying to your users, you can automatically redirect HTTP to HTTPS with this configuration:

_nginx.helm.values.yml_

```yaml
controller:
  containerPort:
    http: 80
    https: 443
    tohttps: 2443 # from https://github.com/kubernetes/ingress-nginx/issues/8017

  service:
    enableHttp: true
    enableHttps: true
    targetPorts:
      http: tohttps # from https://github.com/kubernetes/ingress-nginx/issues/8017
      https: https

  # Will add custom configuration options to Nginx ConfigMap
  config:
    # from https://github.com/kubernetes/ingress-nginx/issues/8017
    http-snippet: |
      server{
        listen 2443;
        return 308 https://$host$request_uri;
      }
    use-forwarded-headers: "true" # from https://github.com/kubernetes/ingress-nginx/issues/1957
```

# 5. Rewrite internal redirects from HTTP to HTTPS

When you terminate your TLS on the load balancer or the ingress controller, application does not know of the TLS incoming calls: everything inside the cluster is HTTP. Hence, when an application needs to redirect you somewhere else inside the cluster to another path, it might try to redirect you on HTTP, same as it received.

For each ingress redirecting internally, apply this configuration:

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: auth-server
  annotations:
    nginx.ingress.kubernetes.io/proxy-redirect-from: http://
    nginx.ingress.kubernetes.io/proxy-redirect-to: https://
spec:
  # [...]
```

# 6. Set a valid default certificate when TLS is terminated on NGINX Ingress Controller

When you don't have the option to terminate TLS on load balancer, NGINX Ingress Controller can be used to do the TLS termination. It would be too long to detail here, if needed you can find litterature on internet, such as [kubernetes + ingress + cert-manager + letsencrypt = https](https://medium.com/andcloudio/kubernetes-ingress-cert-manager-letsencrypt-https-e475f5ee0228), or [Installing an NGINX Ingress controller with a Let's Encrypt certificate manager](https://cloud.yandex.com/en/docs/managed-kubernetes/tutorials/ingress-cert-manager), or else [How to Set Up an Nginx Ingress with Cert-Manager on DigitalOcean Kubernetes](https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nginx-ingress-with-cert-manager-on-digitalocean-kubernetes).

When this scenario is in place, each ingress route get it's own certificate, it can be the same certificate. It can also be the same secret, if the services are in the same namespace.

But the default NGINX certificate, for non-configured routes, will still be the NGINX auto-signed certificate.

To fix that, you can reuse a matching wildcard certificate that you already have somewhere in the cluster, generated using Cert-Manager. NGINX ingress controller can be configured to target it, even from another namespace:

_nginx.helm.values.yml_

```yaml
controller:
  extraArgs:
    default-ssl-certificate: "my-namespace/my-certificate"
```

# 7. Allow large file transfer

By default, the NGINX ingress controller allow a maximum of 1 Mb payload transfer.

For each ingress route where you need more, apply this configuration:

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 100m
[...]
```

![(green ship wheel), helm, tropical sea, sunny day, (green)](https://raw.githubusercontent.com/bcouetil/articles/main/images/kubernetes/boat/nginx-3.jpg)

# 8. Autoscale the ingress controller

Eventually, the traffic of your web application will grow, and the ingress controller initial configuration may become obsolete.

One way to do an easy autoscaling is using a _daemonset_, one _pod_ for each _node_:

_nginx.helm.values.yml_

```yaml
controller:
  kind: DaemonSet # Deployment or DaemonSet
```

Another way is autoscaling on NGINX CPU and memory:

_nginx.helm.values.yml_

```yaml
controller:
  autoscaling:
    enabled: true
    minReplicas: 1
    maxReplicas: 3
    targetCPUUtilizationPercentage: 200
    targetMemoryUtilizationPercentage: 200
```

If this is not sufficient, gather your incoming connections metrics and autoscale based on them. This needs complex operations, so we just forward you to the excellent article [Autoscaling Ingress controllers in Kubernetes](https://dev.to/danielepolencic/autoscaling-ingress-controllers-in-kubernetes-1kgn) by [Daniele Polencic](https://dev.to/danielepolencic)

# 9. Stick user session to the same targeted pod

Applications in Kubernetes cluster must be mostly stateless, by often there is still an ephemeral session depending on the pod the user is reaching. If the users ends up on another pod, the session can be disrupted. In this case we need a "sticky session".

The enabling of sticky sessions is on the ingress side:

_nginx.helm.values.yml_

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    # sticky session, from documentation: https://kubernetes.github.io/ingress-nginx/examples/affinity/cookie/
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/affinity-mode: "persistent" # change to "balanced" (default) to redistribute some sessions when scaling pods
    nginx.ingress.kubernetes.io/session-cookie-name: "name-distinguishing-services"
    nginx.ingress.kubernetes.io/session-cookie-max-age: "172800" # in seconds, equivalent to 48h
[...]
```

# 10. Have access to real client IP in applications

By default for managed load balancers, the client IP visible to your application is not the one from the real client.

You can have it defined in `X-Real-Ip` request header by setting this NGINX ingress controller configuration:

For AWS:

_nginx.helm.values.yml_

```yaml
controller:
  service:
    externalTrafficPolicy: "Local"
```

Or for OVHCloud, from [official documentation](https://help.ovhcloud.com/csm/en-public-cloud-kubernetes-getting-source-ip-behind-loadbalancer?id=kb_article_view&sysparm_article=KB0049765):

_nginx.helm.values.yml_

```yaml
controller:
  service:
    externalTrafficPolicy: "Local"
    annotations:
      service.beta.kubernetes.io/ovh-loadbalancer-proxy-protocol: "v2"
  config:
    use-proxy-protocol: "true"
    real-ip-header: "proxy_protocol"
    proxy-real-ip-cidr: "xx.yy.zz.aa/nn"
```

This will be effective on Helm `install` but not always on `upgrade`, depending on the status of your release ; sometimes you have to edit the NGINX LoadBalancer service to define the value in `spec.externalTrafficPolicy`, and then restart NGINX pods to use the config part (targeting the configmap).

More information in [Kubernetes Documentation](https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-typeloadbalancer).

# 11. Set maintenance mode

You may have already wondered how you could have your users know that you are currently deploying, to help them patiently wait for your website to be available again.

There are multiple lightweight ways to do that, and some of them involve NGINX ingress controller.

_DevOps Directive_ has made an awesome job in this field described the article [Kubernetes Maintenance Page](https://devopsdirective.com/posts/2022/10/kubernetes-maintenance-page/). The solution uses a dedicated deployment + a service without any custom Docker image, that you can target with any ingress during maintenance.

# 12. Disable NGINX access logs for a particular service

In cases where you're dealing with a massively used ingress that's drowning out your NGINX logs, there's a solution. This often crops up in development environments, especially when a high-frequency tool like an APM server comes into play. These tools trigger frequent calls, even during idle user moments.

To combat this, leverage the `nginx.ingress.kubernetes.io/enable-access-log` annotation:

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: apm-server
  labels:
    app: apm-server
  annotations:
    nginx.ingress.kubernetes.io/enable-access-log: "false"
spec:
  rules:
    - host: apm.my-app.com
```

# 13. Whitelist IP for a particular Ingress

To restrict access to a particular Ingress per source IP, you can set NGINX [whitelist-source-range](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#whitelist-source-range) annotation with some IPs and/or CIDR. For example:

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: webapp-direct
  annotations:
    nginx.ingress.kubernetes.io/whitelist-source-range: 10.0.0.0/24,172.10.0.1
spec:
  rules:
    - host: my.website.com
      http:
        paths:
          - path: "/"
            pathType: Prefix
            backend:
              service:
                name: webapp
                port:
                  number: 80
```

# Wrapping up

We have covered multiple NGINX ingress controller use-cases for web application, that we can use for a large variety of situations.

If you think one or two others are common and missing here, don't hesitate to comment in the section below 🤓

![(green ship wheel), helm, tropical sea, sunny day, (green)](https://raw.githubusercontent.com/bcouetil/articles/main/images/kubernetes/boat/nginx-4.jpg)

_Illustrations generated locally by Automatic1111 using Lyriel model_

# Further reading

{% embed https://dev.to/zenika/why-managed-kubernetes-is-a-viable-solution-for-even-modest-but-actively-developed-applications-357g %}

{% embed https://dev.to/zenika/kubernetes-from-your-docker-compose-file-to-a-cluster-with-kompose-1gn0 %}

{% embed https://dev.to/zenika/kubernetes-a-pragmatic-kubectl-aliases-collection-17oc %}

{% embed https://dev.to/zenika/web-application-on-kubernetes-a-tutorial-to-observability-with-the-elastic-stack-2p8a %}

{% embed https://dev.to/zenika/kubernetes-awesome-maintained-links-you-will-keep-using-next-year-48o8 %}

{% embed https://dev.to/zenika/managed-kubernetes-our-dev-is-on-aws-our-prod-is-on-ovh-3nbf %}

{% embed https://dev.to/zenika/how-to-deploy-a-secured-ovh-managed-kubernetes-cluster-using-terraform-in-2023-17o7 %}

{% embed https://dev.to/zenika/how-to-deploy-a-cost-efficient-awseks-kubernetes-cluster-using-terraform-in-2023-3903 %}

{% embed https://dev.to/zenika/eks-10-tips-to-reduce-the-bill-up-to-90-on-aws-managed-kubernetes-clusters-epe %}

_This article was enhanced with the assistance of an AI language model to ensure clarity and accuracy in the content, as English is not my native language._