K3s Update TLS SANs
Tags: k3s, kubernetes, homelab, tls
Published: 2024-03-09 21:02:44

## Introduction
When I was following the [official guide of adding an external cluster load balancer](https://docs.k3s.io/datastore/cluster-loadbalancer#setup-load-balancer) for K3s using HAProxy to have some HAs when connecting to the cluster. I ran into a problem: `kubectl` refuses to connect to the cluster due to some certificate issues:

>`couldn't get current server API group list: Get "https://balancer_ip:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate is valid for [list of control plan ips and cluster + Kube API server service ip], ::1, not load_balancer_ip`

This is due to my load balancer's IP not being in the cluster's API server certificate. During cluster initialization and when adding new control plane node to the cluster, K3s automatically update this certificate to include the new control plane node. However, this is not the case when adding an external cluster load balancer.

---
## Table of Contents
{%- # TOC start (generated with https://github.com/derlin/bitdowntoc) -%}

- [Solution 1: Reinitialize the Cluster With --tls-san](#solution-1-reinitialize-the-cluster-with-tlssan)
- [Solution 2: Editing K3s Configuration File](#solution-2-editing-k3s-configuration-file)
   * [Step 1: Edit the Config File](#step-1-edit-the-config-file)
   * [Step 2: Delete the Existing Kube API Server Cert](#step-2-delete-the-existing-kube-api-server-cert)
   * [Step 3: Restart K3s](#step-3-restart-k3s)
   * [Step 4: Inform the DynamicListener About the Change](#step-4-inform-the-dynamiclistener-about-the-change)
      + [Alternative Method](#alternative-method)
- [Worker / Control Plane Node Reconfiguration (Optional)](#worker-control-plane-node-reconfiguration-optional)

{%- # TOC end -%}

---
## Solution 1: Reinitialize the Cluster With --tls-san
According to the [K3s official documentation](https://docs.k3s.io/cli/server#listeners), you can pass the `--tls-san` flag when initializing the cluster:
```
sh k3s_install.sh --tls-san additional_ip_1 --tls-san additional_ip_1 --cluster-init
```

However, my cluster is already running and I don't want to reinstall it.

---
## Solution 2: Editing K3s Configuration File
The K3s configuration file is located under `/etc/rancher/k3s/config.yaml` on all control plane nodes. If the file doesn't exist, you have to create it manually.


### Step 1: Edit the Config File
In the config file, you can simply add the following entries:
```yaml
tls-san:
  - ip_1
  - hostname_1
  // additional entries
```
This can be done on any control plane node. 

### Step 2: Delete the Existing Kube API Server Cert
On any control plane node run
```bash
sudo kubectl -n kube-system delete secrets/k3s-serving
```
to delete the existing Kubernetes API server certificate.

This can be done anywhere as long as you can access the cluster's API server.

### Step 3: Restart K3s
Restart K3s on the same control plane node:
```bash
sudo systemctl restart k3s
```

### Step 4: Inform the DynamicListener About the Change
[DynamicListener](https://github.com/rancher/dynamiclistener) is a component of K3s that handles automatic updates/renewal of the API server certificate, including when new control plan nodes join the cluster. 

According to one of the K3s contributor, [Brad Davidson](https://github.com/brandond) in one of the [GitHub issue](https://github.com/k3s-io/k3s/issues/7312#issuecomment-1652536728):

> Dynamiclistener adds SANs for any hostname or IP address requested via a HTTP Host header or TLS SNI handshake. It is designed to allow you to add or remove servers without having to manually regenerate the certificate. The downside (as you noted) is that this allows for SAN stuffing.

The documentation about it is not very clear and the official README is still in progress. This is as far as I can go about it.

For each new hostname / IP you added to the config file, run the following command on the control plan node:
```bash
curl -k --resolve your_new_hostname_or_ip:6443:127.0.0.1  https://your_new_hostname_or_ip:6443/ping
```

This method is also mentioned in another GitHub issue comment [here](https://github.com/k3s-io/k3s/issues/3369#issuecomment-849005179).

To confirm that the API server certificate has been updated, run the following command:
```bash
kubectl get secret/k3s-serving -n kube-system -o yaml
```
This can be done anywhere as long as you have access to the cluster's API server.

In the `metadata.annotations` part of the output, you should see your newly added hostname/ip as one of the annotations: 
```
listener.cattle.io/cn-ur_hostnameor__ip: ur_hostname_or_ip
```

#### Alternative Method
I found one Medium post [here](https://taozhi.medium.com/k3s-apiserver-unable-to-connect-to-the-server-x509-certificate-is-valid-for-10-43-0-1-8ec1f8c2097f) that uses another method to update the DynamicListener.


After you delete the existing API server certificate on a control plane node using
```bash
sudo kubectl -n kube-system delete secrets/k3s-serving
```

You can move or delete a file called `dynamic-cert.json` located under `/var/lib/rancher/k3s/server/tls` on the same control plane node
```bash
sudo mv /var/lib/rancher/k3s/server/tls/dynamic-cert.json /tmp/dynamic-cert.json
```
**BEFORE** restarting K3s on the same control plane node.

I have tested both methods and they all worked.


## Worker / Control Plane Node Reconfiguration (Optional)
Now we have to update our worker / control plane nodes to use the new endpoint. 

The fastest way is to update the K3s systemd service file.

You will have to do this on each node that you want to reconfigure.

For control plane nodes the file is located at
```bash
/etc/systemd/system/k3s.service
```

For worker nodes the file is located at
```
/etc/systemd/system/k3s-agent.service
```

Update the value of the `--server` flag in the file to use the new hostname / IP of the balancer.


After making the changes, run `sudo systemctl daemon-reload` to reload the changes.

Finally run `sudo systemctl restart k3s` to restart the K3s daemon for the changes to take effect.