How to setup nginx Ingress w/ automatically generated LetsEncrypt certificates on Kubernetes
Tags: kubernetes, docker, letsencrypt, devops
Published: 2019-12-07 12:38:41

---
title: How to setup nginx Ingress w/ automatically generated LetsEncrypt certificates on Kubernetes
published: true
tags: kubernetes,docker,letsencrypt,devops
cover_image: https://chrisme.fra1.digitaloceanspaces.com/ingress_letsencrypt_kubernetes.png
---

> Background: After first searching around there are plenty of tutorials to show how to do this, but none of them just worked w/o modifications, they were outdated or used helmet (which is not needed and I wanted to use as few tools as possible), so I had to look up multiple sources for getting it running. Basically this got a compilation of the articles found online that worked for me.

I'm omitting much of the background information or explanations here, as these can be found in the linked tutorials. Instead this shall serve as a brief set of instructions to get it working.

**As a bonus we'll also setup port forwarding for any arbitrary service (non http/https - like databases) to become accessible from the internet over the same IP/LoadBalancer as the websites.**

I'm using the Kubernetes offered by DigitalOcean, so if your new I'd be happy for using my link [here ](https://m.do.co/c/5bfd32255e83) to sign up. You can spin up a k8s cluster starting at 10$/month.

## Prerequisites

- Basic understanding of kubernetes objects / types
- A k8s cluster, with kubectl ready / setup
- Access to a DNS Provider to setup some DNS entries to point to your cluster

## Sources

Credits to the sources used used in this article (no particular order):

- https://kubernetes.github.io/ingress-nginx/deploy/
- https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nginx-ingress-with-cert-manager-on-digitalocean-kubernetes
- https://cert-manager.io/docs/installation/kubernetes/
- https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/

## Create some dummy 'echo' deployments

We want some dummy webservices to just respond to http requests. Create the following two files:

```yaml
# echo1.yaml
apiVersion: v1
kind: Service
metadata:
  name: echo1
spec:
  ports:
  - port: 80
    targetPort: 5678
  selector:
    app: echo1
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo1
spec:
  selector:
    matchLabels:
      app: echo1
  replicas: 2
  template:
    metadata:
      labels:
        app: echo1
    spec:
      containers:
      - name: echo1
        image: hashicorp/http-echo
        args:
        - "-text=echo1"
        ports:
        - containerPort: 5678
```

```yaml
# echo2.yaml
apiVersion: v1
kind: Service
metadata:
  name: echo2
spec:
  ports:
  - port: 80
    targetPort: 5678
  selector:
    app: echo2
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo2
spec:
  selector:
    matchLabels:
      app: echo2
  replicas: 1
  template:
    metadata:
      labels:
        app: echo2
    spec:
      containers:
      - name: echo2
        image: hashicorp/http-echo
        args:
        - "-text=echo2"
        ports:
        - containerPort: 5678
```

Apply both deployments:

```bash
$ kubectl apply -f echo1.yaml
$ kubectl apply -f echo2.yaml
```

This will create 2 deployments along with 2 services, listening on cluster internal port 80:

```bash
$ kubectl get service
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
echo1        ClusterIP   10.245.164.177   <none>        80/TCP    1h
echo2        ClusterIP   10.245.77.216    <none>        80/TCP    1h
```

## Setup Ingress nginx

Getting ingress-nginx up and running only requires two commands:

```bash
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml

namespace/ingress-nginx created
configmap/nginx-configuration created
configmap/tcp-services created
configmap/udp-services created
serviceaccount/nginx-ingress-serviceaccount created
clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole created
role.rbac.authorization.k8s.io/nginx-ingress-role created
rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding created
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding created
deployment.apps/nginx-ingress-controller created
```

```bash
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud-generic.yaml

service/ingress-nginx created
```

At this point nginx-ingress is setup and a Load Balancer (caution: cloud providers charge fees for every Load Balancer) will spin up. This can take some minutes to show up. You can check the status with the following command:


```bash
$ kubectl get -n ingress-nginx service
NAME            TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE
ingress-nginx   LoadBalancer   10.245.209.25   155.241.87.123   80:30493/TCP,443:30210/TCP   1h
```

UPDATE:

There's an open issue regarding network routing causing problems later when a pod tries to request certificates and runs into timeout trying to self check the requested domain.

Due to this we have to change the 'externalTrafficPolicy' of the just created 'ingress-nginx' Service. Create a file 'Service_ingress-nginx.yaml' w/ the following content:

```yaml
# Service_ingress-nginx.yaml
kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  # default for externalTrafficPolicy = 'Local', but due to an issue
  # (https://stackoverflow.com/questions/59286126/kubernetes-cluterissuer-challenge-timeouts,
  # https://github.com/danderson/metallb/issues/287)
  # it has to be 'Cluster' for now
  externalTrafficPolicy: Cluster
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
    - name: https
      port: 443
      protocol: TCP
      targetPort: https
```

When you diff / apply this file, the difference should be that the externalTrafficPolicy now is 'Cluster'. Disadvantage of this is, that you can loose track of the original IP requesting services, but for me this does not matter and resolves the certificate requesting issue.

```bash
kubectl diff -f Service_ingress-nginx.yaml # show differences
kubectl apply -f Service_ingress-nginx.yaml # activate changes
```

Note down your external IP as we'll use it as our one gate for services we want to make publicly accessible.

Now go to your DNS provider and create two entries (A records of any kind) to point to your external IP.

Create a file 'echo_ingress.yaml' and adjust it to match the DNS entries just created:

```yaml
# echo_ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: echo-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"    
spec:
  rules:
  - host: echo1.yourdomain.com
    http:
      paths:
      - backend:
          serviceName: echo1
          servicePort: 80
  - host: echo2.yourdomain.com
    http:
      paths:
      - backend:
          serviceName: echo2
          servicePort: 80
```

Create the ingress by applying it:

```bash
$ kubectl apply -f echo_ingress.yaml
```

The two sites should now be avaible via http. Go ahead and try visiting them in the browser.

## cert-manager

The next step is to install cert-manager, which will later use Issuers get our certificates.

```bash
$ kubectl create namespace cert-manager
$ kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.12.0/cert-manager.yaml
```

This installs cert-manager. You can check for running pods:


```bash
$ kubectl get pods --namespace cert-manager
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-5c47f46f57-n9bb6              1/1     Running   0          31s
cert-manager-cainjector-6659d6844d-6zjh5   1/1     Running   0          31s
cert-manager-webhook-547567b88f-9hj5f      1/1     Running   0          31s
```

To check if cert-manager runs correctly, we can now issue self signed certificates for testing. Generate a file named 'test-cert-manager.yaml':

```yaml
# test-cert-manager.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: cert-manager-test
---
apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
  name: test-selfsigned
  namespace: cert-manager-test
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: selfsigned-cert
  namespace: cert-manager-test
spec:
  commonName: example.com
  secretName: selfsigned-cert-tls
  issuerRef:
    name: test-selfsigned
```

Apply it and check the output:

```bash
$ kubectl apply -f test-cert-manager.yaml
$ kubectl describe certificate -n cert-manager-test
```

The events of the describe output should state something like 'Certificate issued successfully'.

If it's OK, delete the test resources:

```bash
$ kubectl delete -f test-cert-manager.yaml
```

With cert-manager up and running, we're missing one final piece; an Issuer (or in our case, we'll pick a ClusterIssuer so that we don't need to specify namespaces and it just works globaly) for generating valid certificates.


## ClusterIssuer's

We'll create two issuers. The first one (staging) is to test if everything works correctly. Otherwise, if something is wrong like a wrong DNS setting and we just use the production issuer, we could get temporarly get rejected from letsencrypt's servers.

Create the following two files and adjust your e-mail address:

```yaml
# staging_issuer.yaml
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    # The ACME server URL
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: me@example.com
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-staging
    # Enable the HTTP-01 challenge provider
    solvers:
    - http01:
        ingress:
          class:  nginx
```

```yaml
# prod_issuer.yaml
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: me@example.com
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-prod
    # Enable the HTTP-01 challenge provider
    solvers:
    - http01:
        ingress:
          class:  nginx
```

Create the issuers:

```bash
$ kubectl create -f staging_issuer.yaml
$ kubectl create -f prod_issuer.yaml
```

Extend the previously created 'echo_ingress.yaml' so that it looks like this:

```yaml
# echo_ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: echo-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"    
    cert-manager.io/cluster-issuer: "letsencrypt-staging"
spec:
  tls:
  - hosts:
    - echo1.yourdomain.com
    - echo2.yourdomain.com
    secretName: letsencrypt-staging
  rules:
  - host: echo1.yourdomain.com
    http:
      paths:
      - backend:
          serviceName: echo1
          servicePort: 80
  - host: echo2.yourdomain.com
    http:
      paths:
      - backend:
          serviceName: echo2
          servicePort: 80
```

Apply the adjustments:

```bash
$ kubectl apply -f echo_ingress.yaml
```

Now, in my case it took several minutes until the certificate got issued. Check the events section of the folling command:

```bash
$ kubectl describe certificate letsencrypt-staging
```

After some time it should state something like 'Certificate issued successfully'. Then you can reload the sites in your browser and check the certificates. They should now be issued by letsencrypt fake authorities.

If this succeeds, we can finally come to the last step of substituting the issuer with the production one. Adjust 'echo_ingress.yaml' one more time, switching to 'letsencrypt-prod':

```yaml
# echo_ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: echo-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"    
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  tls:
  - hosts:
    - echo1.yourdomain.com
    - echo2.yourdomain.com
    secretName: letsencrypt-prod
  rules:
  - host: echo1.yourdomain.com
    http:
      paths:
      - backend:
          serviceName: echo1
          servicePort: 80
  - host: echo2.yourdomain.com
    http:
      paths:
      - backend:
          serviceName: echo2
          servicePort: 80
```

Apply the adjustments again:

```bash
$ kubectl apply -f echo_ingress.yaml
```

Now, again after some possibly minutes, your site should be issued a valid certificate.

```bash
$ kubectl describe certificate letsencrypt-prod
```

**Congratulations!** We finally have our sites running with valid certificates. The nice thing at this point is, that it's scalable. From now on it's easy to add or remove sites.

## Troubleshooting certificate issueing

When at some point certificates are not issued it can help to know which components are involved until a cert is successfully issued. In my case for example challenges could not be fullfilled. Therefore just follow the guide on the official cert-manager site until the root is identified:

https://cert-manager.io/docs/faq/acme/

## Bonus - make other services avaiable outside of cluster

Say you have some service other then websites - like a database - running inside your cluster and want to make it accessible from outside.

One method would be to just attach a 'NodePort' service to it, but that would come with restrictions, like only ports beyond 30000 and if one of your nodes go down or get replaced, the ip to access the service would need to be adjusted.

Instead, we can just add arbitrary ports to our existing load-balanced nginx-ingress service (TCP or UDP). Therefore, only some small modifications must be made.

Here's an example of opening TCP port 9000 and forwarding it to some service running internal at port 27017.

Previously we just applied the resources from the URL 'https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml' and 'https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud-generic.yaml'.

They contain two resources that are active but need to be adjusted and re-applied. First, create a file 'ingress-nginx-tcp-configmap.yaml':

```yaml
# ingress-nginx-tcp-configmap.yaml
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
data:
  9000: "default/someservice:27017"
```

Second, create a file called 'ingress-nginx-service.yaml':

```yaml
# ingress-nginx-service.yaml
kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  externalTrafficPolicy: Local
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
    - name: https
      port: 443
      protocol: TCP
      targetPort: https
    - name: proxied-tcp-9000
      port: 9000
      targetPort: 9000
      protocol: TCP

```

Apply the adjustments.

```bash
$ kubectl apply -f ingress-nginx-tcp-configmap.yaml
$ kubectl apply -f ingress-nginx-service.yaml
```

Your service sould no be accessible from outside of the cluster with the same IP as the websites.