Install Certificate Manager Controller in Kubernetes
Tags: certificate, kubernetes, certmanager, ca
Published: 2021-05-02 19:37:09

<img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1e7czag0bi7ht3q4f2bc.png" />

<sup><b>Credits:</b> Logo by [cert-manager.io](https://cert-manager.io/)</sup>

Automate the process of issuing public key certificates from multiple sources, ensuring they are valid, up to date and renew before expiration.

- [K8s Controller](#k8s-controller)
   - [Prepare](#prepare)
   - [Install](#install)
   - [Uninstall](#uninstall)
- [Self Signed](#self-signed)
   - [Issuer](#self-signed-issuer)
   - [Certificate](#self-signed-certificate)
   - [Secrets](#cert-secrets)
   - [Trust](#cert-trust) 
- [Advanced](#advanced)
   - [Share Secrets between Namespaces](#namespace-secrets)
- [Summary](#summary)

<p>&nbsp;</p>

> **Note:** This post is a quick start guide for deploying and using `cert-manager` on a Kubernetes cluster.

<p>&nbsp;</p>

<h3 id="prerequisites">Prerequisites</h3>

- [Install Rancher K3s on Raspberry Pi Cluster](https://dev.to/zachinachshon/install-rancher-k3s-on-raspberry-pi-cluster-1i06)
- [Installing Helm, Kubernetes Package Manager](https://dev.to/zachinachshon/installing-helm-kubernetes-package-manager-3fi0)

<p>&nbsp;</p>

**Why do we need to worry about certificates?** When declaring a domain name i.e `my-website.domain.com` and addressing it from either internal network and/or public internet, the devices used to perform the call (web browsers, internal services, containers etc..) would require to check its validity. In order to do that, the domain name should have a certificate that is issued and trusted to operate securely.

**Why do we need a certificate manager?** Certificate validity has its expiration date, which means certificates have to get renewed. It might be a cumbersome task when there are many certificates to handle. This is the reason `cert-manager` exists, to help with issuing certificates from a variety of sources, such as [Let’s Encrypt](https://letsencrypt.org/), a simple signing key pair or self signed. It will ensure certificates are valid, up to date and attempt to renew certificates at a configured time before expiry.

> **Note:** The domain referenced in this post is `MY_DOMAIN`, please change accordingly. If you interested in a local-only work mode, you don't have to pay for a new domain, just decide on a name and use it. For example, if your desired domain is `homelab.com`, replace `MY_DOMAIN` with `homelab`.

<p>&nbsp;</p>

<h2 id="k8s-controller">K8s Controller</h2>

`cert-manager` is a Kubernetes controller that manage the certificate aspect of a cluster state. It looks after the state of certificates on a specific cluster and issue new ones or request to renew existing ones before expiration.

<h3 id="prepare">Prepare</h3>

> **Tip:** This preparation step is relevant only if you wish to deploy the `cert-manager` pod on a specific cluster node such as master for example, otherwise please continue to the next Install step.

We want Kubernetes to create the `cert-manager` pod on the master node. In order to do that, we'll have to label that node and use `nodeSelector` attribute when installing `cert-manager` Helm chart.

1. Get all nodes names and labels

    ```bash
    kubectl get nodes --show-labels
    ```
    <p>&nbsp;</p>

1. Label `kmaster` node with `node-type=master`

    ```bash
    kubectl label nodes kmaster node-type=master
    ```
    <p>&nbsp;</p>
   
    > **Note:** I'll refer to the master node named `kmaster` as specified in the [RPi cluster installation post](https://dev.to/zachinachshon/setting-up-a-raspberry-pi-cluster-1oa0), replace the name with the one you have assigned to the master node on your Kubernetes cluster.

1. Verify that label had been created successfully

    ```bash
    kubectl get nodes --show-labels | grep node-type
    ```
    <p>&nbsp;</p>

<h3 id="install">Install</h3>

1. Create a `cert-manager` namespace

    ```bash
    kubectl create namespace cert-manager
    ```
    <p>&nbsp;</p>

1. Disable resource validation on the `cert-manager` namespace

    ```bash
    kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true
    ```
    <p>&nbsp;</p>
	
    > **Note:** `cert-manager` deploys a webhook component to perform resource validations on `Issuer`, `ClusterIssuer` and `Certificate`. This webhook shouldn't run on the same namespace the `cert-manager` is deployed on.

1. Add the required Helm repository

    ```bash
    helm repo add jetstack https://charts.jetstack.io
    ```
    <p>&nbsp;</p>

1. Update your local Helm chart repository cache

    ```bash
    helm repo update
    ```
    <p>&nbsp;</p>

1. Search for latest `jetstack/cert-manager` official Helm chart version

    ```bash
    helm search repo cert-manager
   
    # NAME                      CHART VERSION	  APP VERSION
    # jetstack/cert-manager     v1.2.0       	  v1.2.0
    ```
    <p>&nbsp;</p>

1. Install the `cert-manager` Helm chart using the version from previous step

    Without node affinity:
   
    ```bash
    helm upgrade --install cert-manager \
        --namespace cert-manager \
        --version v1.2.0 \
        jetstack/cert-manager \
        --set installCRDs=true
    ```
    <p>&nbsp;</p>
   
    With node affinity (if you have followed the `Prepare` step):
   
    ```bash
    helm upgrade --install cert-manager \
        --namespace cert-manager \
        --version v1.2.0 \
        --set nodeSelector.node-type=master \
        --set webhook.nodeSelector.node-type=master \
        --set cainjector.nodeSelector.node-type=master \
        jetstack/cert-manager \
        --set installCRDs=true
    ```
    <p>&nbsp;</p>

    > **Note:** Make sure you are running on Kubernetes > `v1.19` to have the `installCRDs` flag available.

1. Verify installation

    ```bash
    # Make sure all cert-manager deployed pods are running
    kubectl get pods --namespace cert-manager
   
    # Make sure custom resources *.cert-manager.io were created successfully 
    kubectl get crd | grep cert-manager
    
    # Verify that ClusterIssuer is non-namespaced scoped ('false')
    # so it can be used to issue Certificates across all namespaces
    kubectl api-resources | grep clusterissuers
    ```
    <p>&nbsp;</p>

<h3 id="uninstall">Uninstall</h3>

1. Remove `cert-manager` from the cluster

    ```bash
    helm uninstall cert-manager --namespace cert-manager
    ```
    <p>&nbsp;</p>
   
1. Clear the namespace

    ```bash
    kubectl delete namespaces cert-manager
    ```
    <p>&nbsp;</p>

---

<h2 id="self-signed">Self Signed</h2>
<p>&nbsp;</p>

**What?** The self signed issuer does not represent a [certificate authority](https://en.wikipedia.org/wiki/Certificate_authority) as such, but instead denotes that certificates will be signed through “self signing” using a given private key. 

**Why?** This means that the provided private key of the resulting certificate will be used to sign its own certificate.

**When?** This Issuer type is useful for bootstrapping the CA certificate key pair for some Private Key Infrastructure (PKI), or for otherwise creating simple certificates. Clients consuming these certificates have no way to trust this certificate since there is no CA signer apart from itself, and as such, would be forced to trust the certificate as is.

**Who?** Clients consuming these certificates could be services deployed to our local Kubernetes cluster that are exposed from outside the cluster such as Kubernetes dashboard, Jenkins UI, Private Docker Registry UI etc..

<h3 id="self-signed-issuer">Issuer</h3>

1. Create a certificate `ClusterIssuer`
  
    ```yaml
    cat <<EOF | kubectl apply -f -
    apiVersion: cert-manager.io/v1alpha2
    kind: ClusterIssuer
    metadata:
      name: MY_DOMAIN-ca-issuer
    spec:
      selfSigned: {}
    EOF
    ```
    <p>&nbsp;</p>
   
    > **Important:** `cert-manager` issues certificates through an `Issuer`. The `Issuer` can issue certificates for the namespace it is created on, but a `ClusterIssuer` can create certificates for any namespace.

1. Wait for status to become `Ready`

    ```bash
    kubectl get clusterissuers MY_DOMAIN-ca-issuer -o wide
    ```
    <p>&nbsp;</p>

<h3 id="self-signed-certificate">Certificate</h3>

When creating a new certificate, make sure to create one on a named namespace. It'll get verified by `cert-manager` even-though it exists on a different namespace since we're referencing a `ClusterIssuer`.

> **Tip:** Replace `MY_NAMESPACE` with the namespace of your choise.

1. Create a **_namespaced_** `X.509` certificate (check [here](https://cert-manager.io/docs/concepts/certificate/) for official schema)

    ```yaml
    cat <<EOF | kubectl apply -f -
    apiVersion: cert-manager.io/v1alpha2
    kind: Certificate
    metadata:
      name: MY_DOMAIN-com-cert
      namespace: MY_NAMESPACE
    spec:
      secretName: MY_DOMAIN-com-cert-secret
      isCA: true
      commonName: '*.MY_DOMAIN.com'
      organization:
        - MY_DOMAIN
      dnsNames:
        - MY_DOMAIN.com
        - '*.MY_DOMAIN.com'
      keySize: 2048
      keyAlgorithm: rsa
      issuerRef:
        name: MY_DOMAIN-ca-issuer
        kind: ClusterIssuer
    EOF
    ```
    <p>&nbsp;</p>

1. Check certificate status is `Issued`

    ```bash
    kubectl describe certificate MY_DOMAIN-com-cert --namespace MY_NAMESPACE
    ```
    <p>&nbsp;</p>

1. Check that secret `MY_DOMAIN-com-cert-secret` was created successfully

    ```bash
    kubectl get secret --namespace MY_NAMESPACE
    kubectl get secret MY_DOMAIN-com-cert-secret -o yaml --namespace MY_NAMESPACE
    ```
    <p>&nbsp;</p>

1. **(Optional)**: When in need to delete a certificate

    ```bash
    # Delete certificate
    kubectl delete certificate MY_DOMAIN-com-cert --namespace MY_NAMESPACE
   
    # Delete the auto generated secret
    kubectl delete secret MY_DOMAIN-com-cert-secret --namespace MY_NAMESPACE
    ```
    <p>&nbsp;</p>

<h3 id="cert-secrets">Secrets</h3>

Follow these instructions to export the `cert-manager` generated certificate secrets as local files.

> **Note:** Sometimes there is a need to use the secret values from outside the Kubernetes cluster. Such example is setting the `cert_file` as a trusted certificate on a client machine (laptop / desktop), specially when using a self signed certificate.

1. Create a local destination folder

    ```bash
    mkdir -p $HOME/temp/MY_NAMESPACE/cert-secrets
    export MY_DOMAIN=<insert-domain-name-here>
    export MY_NAMESPACE=<insert-namespace-here>
    ```
    <p>&nbsp;</p>

1. Export the certificate secrets

    **cert_file** - client certificate path used for authentication

    ```bash
    kubectl get secret ${MY_DOMAIN}-com-cert-secret \
       --namespace ${MY_NAMESPACE} \
       -o jsonpath='{.data.tls\.crt}' | base64 -D \
       > $HOME/temp/${MY_NAMESPACE}/cert-secrets/cert_file.crt
    ```
    <p>&nbsp;</p>

    **key_file** - client key path used for authentication

    ```bash
    kubectl get secret ${MY_DOMAIN}-com-cert-secret \
       --namespace ${MY_NAMESPACE} \
       -o jsonpath='{.data.tls\.key}' | base64 -D \
       > $HOME/temp/${MY_NAMESPACE}/cert-secrets/key_file.key
    ```
    <p>&nbsp;</p>

    **ca_file** - CA certificate path used to verify the remote server cert file

    ```bash
    kubectl get secret ${MY_DOMAIN}-com-cert-secret \
       --namespace ${MY_NAMESPACE} \
       -o jsonpath='{.data.ca\.crt}' | base64 -D \
       > $HOME/temp/${MY_NAMESPACE}/cert-secrets/ca_file.crt
    ```
    <p>&nbsp;</p>

1. Check that x3 secrets exported successfully

    ```bash
    ls -lah $HOME/temp/${MY_NAMESPACE}/cert-secrets
    ```
    <p>&nbsp;</p>
   
1. Clear exported variables

    ```bash
    unset MY_DOMAIN MY_NAMESPACE
    ```
    <p>&nbsp;</p>

<h3 id="cert-trust">Trust</h3>

When addressing a Kubernetes ingress controller resource that had been signed with a self signed certificate secret, clients such as web-browsers would warn us of an invalid certificate authority or invalid certificate.

**ERR_CERT_AUTHORITY_INVALID**

<img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lh9ijea1nciv53xmns1q.png" alt="err-cert-authority-invalid" />

**ERR_CERT_INVALID**

<img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/b3ll5h42k5xx3nd6rklu.png" alt="err-cert-invalid" />
<p>&nbsp;</p>

Since we are the ones that created the certificate we would like to tell our clients (laptop/desktop) to trust it. This is how you trust a certificate on **macOS**, for other operating systems please refer to official documentation.

1. Open the certificate file in macOS `Keychain Access`

   - CLI: `open $HOME/temp/${MY_NAMESPACE}/cert-secrets/cert_file.crt`
   - Other: double click on the `cert_file.crt`

1. Double click on the `*.MY_DOMAIN.com` certificate and expand the `Trust` section

1. Within the `When using this certificate` select `Always Trust`

1. Open a web-browser and navigate to one of the domain names using the self signed certificate and verify that there are no errors

---

<h2 id="advanced">Advanced</h2>

This section contains non conventional tip & tricks, use it cautiously.

<h3 id="namespace-secrets">Share Secrets between Namespaces</h3>

**When?** Use when in need to copy a `cert-manager` generated certificate secret to a different namespace than the one it was created on.

**Why?** Useful when you need to reference the secret in some kind of ingress controller under its authentication spec which exists on a different namespace.

In order to solve this limitation we simple need to copy the secret to a different namespace as follows:

```bash
kubectl get secret MY_DOMAIN-com-cert-secret -n SOURCE_NAMESPACE -o yaml \
| sed s/"namespace: SOURCE_NAMESPACE"/"namespace: DESTINATION-NAMESPACE"/\
| kubectl apply -n DESTINATION-NAMESPACE -f -
```
<p>&nbsp;</p>

> **Important:** It is not advised to copy secrets manually since `cert-manager` would lose track of the secret as it was copied manually and when the time to renew is due, the secret would become invalid.

---

<h2 id="summary">Summary</h2>

This post was a sneak peak on how to create certificates and manage their state using `cert-manager` Kubernetes controller. 

**What now?** Now that you have a local self signed certificate, you can continue to create an ingress controller and expose internal web services within your home network such as Kubernetes dashboard, Jenkins UI, your web application UI with their own virtual hosted names i.e. `kubernetes.MY_DOMAIN.com`, `jenkins.MY_DOMAIN.com`,  `mydashboard.MY_DOMAIN.com` etc..

Please leave your comment, suggestion or any other input you think is relevant to this post in the discussion below.

---

Like this post?
You can find more by:

Checking out my blog: https://blog.zachinachshon.com
Following me on twitter: @zachinachshon

Thanks for reading! ❤️