Cook up a k3s cluster on Raspberry pies with Ansible
Tags: ansible, k3s, kubernetes, rancher
Published: 2023-03-07 21:49:00

Cook up and scale a k3s Kubernetes cluster on Raspberry pies with a single command.

All the code is available at my Github repo [PiClusterChef](https://github.com/hatati/PiClusterChef)

## Prerequisites

- Install the Raspbian OS (64-bit) on your Raspberry Pies, connect them to your local network, give them a hostname, and enable their SSH servers. There is an excellent guide by w3Schools on how to do this [here](https://www.w3schools.com/nodejs/nodejs_raspberrypi.asp). (I prefer to use the official [Raspberry Pi Imager](https://www.raspberrypi.com/software/) to flash the SD Cards instead of Etcher, which is used by w3schools, but you can use either.)
- Install [Ansible](https://docs.ansible.com/ansible/latest/installation_guide/index.html) on your local machine. (I use the WSL 2 subsystem in Windows to run Ansible.)
- Install [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/).
- **Optional**: Install [Helm](https://helm.sh/docs/intro/install/).

## Ansible setup and components

To provision our cluster with ansible we need an *inventory* file, *playbook* file(s), and some initial SSH setup since ansible uses SSH to connect to nodes.

### Inventory file

In Ansible, the inventory file is a file that contains a list of target hosts or nodes that Ansible can manage. In this case, it will contain all our Raspberry Pis.

```yaml
workers:
  hosts:
    nerminworker1:
      ansible_host: 192.168.0.66
      ansible_user: nermin

masters:
  hosts:

# This hostgroup is designed to only contains the initial bootstrap master node
bootstrapMaster:
  hosts:
    nerminmaster:
      ansible_host: 192.168.0.67
      ansible_user: nermin

pies:
  children:
    bootstrapMaster:
    masters:
    workers:
  vars:
    k3s_version: v1.24.10+k3s1
```

We have four groups in our setup:

- `bootstrapMaster`: This is the first node that will get a k3s server and will create the token used by other nodes to register with the cluster.
- `masters`: These are the remaining k3s server nodes that will register themselves via the `bootstrapMaster`.
- `workers`: These are the k3s worker/agent nodes that will run workloads in the cluster. They also register - themselves via the `bootstrapMaster` and the generated k3s token.
- `pies`: This group contains all the masters and workers combined.

I have two Raspberry Pis in my setup, even though the picture shows three. The third Raspberry Pi has another purpose not related to the k3s cluster. Both nodes have the user `nermin`. `nermin` is used to execute commands on each of the hosts. A single variable `k3s_version` is used to determine which version of k3s we want to install.



### SSH connections

Add your public SSH key to each Raspberry Pi to avoid being prompted for passwords when creating sessions.

```bash
$ ssh-copy-id <user>@<host>
```

In this specific case:

```bash
$ ssh-copy-id nermin@nerminmaster
$ ssh-copy-id nermin@nerminworker1
```

You can also use the IP address as the host instead of the hostname.

> **Note**: Make sure you have generated an SSH key pair in your WSL2 system. You can check by running:
>
> ```bash
> $ ls ~/.ssh/id_*.pub
> ```
>
> If you don't see any files listed, you'll need to generate an SSH key pair. You can do this by running:
>
> ```bash
> $ ssh-keygen
>```
>
> Follow the prompts to generate a new key pair. Make sure to keep your private key secure and do not share it with others.

### Install Playbook

The *install-k3s-playbook.yaml* file contains plays for installing k3s masters and workers on the Raspberry Pis. This includes enabling memory cgroups as required by the [documentation](https://docs.k3s.io/advanced#raspberry-pi). The playbook also retrieves the kubeconfig file from the bootstrap node and places it in the current directory with the name k3sconfig.

```yaml
- name: Enable cgroups
  become: true
  hosts: pies
  tasks:
  - name: Ping hosts
    ansible.builtin.ping:
  - name: Check if cgroups are enabled
    command: cat /boot/cmdline.txt
    register: cmdlineContent
  - name: Enable cgroups
    command: sed -i -e 's/$/ cgroup_memory=1 cgroup_enable=memory/' /boot/cmdline.txt
    when: "'cgroup_memory=1 cgroup_enable=memory' not in cmdlineContent.stdout"
    notify: 
    - Restart pi
  handlers:
  - name: Restart pi
    ansible.builtin.reboot:

- name: Install k3s bootstrap server
  become: true
  hosts: bootstrapMaster
  tasks:
  - name: Ping host
    ansible.builtin.ping:
  - name: Install k3s bootstrap server
    shell: curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION={{ k3s_version }} K3S_NODE_NAME={{ inventory_hostname }} K3S_KUBECONFIG_MODE="644" sh -s - server --cluster-init
  - name: Extract K3S_TOKEN from server output
    command: cat /var/lib/rancher/k3s/server/node-token
    register: k3s_token
    failed_when: k3s_token is failed or k3s_token.stdout is undefined
  - name: Set K3S_TOKEN as a fact
    set_fact:
      k3s_token: "{{ k3s_token.stdout }}"

- name: Install k3s servers
  become: true
  hosts: masters
  tasks:
  - name: Ping hosts
    ansible.builtin.ping:
  - name: Install k3s servers
    shell: curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION={{ k3s_version }} K3S_URL=https://{{ hostvars['nerminmaster']['ansible_default_ipv4'].address }}:6443 K3S_TOKEN={{ hostvars['nerminmaster']['k3s_token'] }} K3S_NODE_NAME={{ inventory_hostname }} sh -s - server

- name: Install k3s workers
  become: true
  hosts: workers
  tasks:
  - name: Ping hosts
    ansible.builtin.ping:
  - name: Install k3s workers
    shell: curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION={{ k3s_version }} K3S_URL=https://{{ hostvars['nerminmaster']['ansible_default_ipv4'].address }}:6443 K3S_TOKEN={{ hostvars['nerminmaster']['k3s_token'] }} K3S_NODE_NAME={{ inventory_hostname }} sh -

- name: Fetch k3s kubeconfig
  become: true
  hosts: bootstrapMaster
  tasks:
  - name: Fetch kubeconfig
    fetch:
      src: /etc/rancher/k3s/k3s.yaml
      dest: k3sconfig
      flat: true
```

### Uninstall Playbook

The *uninstall-k3s-playbook.yaml* uninstalls the k3s services and scripts on each Raspberry Pi. This enables easy cleanup.

```yaml
- name: Uninstall k3s on workers
  become: true
  hosts: workers
  tasks:
  - name: Ping hosts
    ansible.builtin.ping:
  - name: Uninstall k3s agent
    command: /usr/local/bin/k3s-agent-uninstall.sh

- name: Uninstall k3s on servers
  become: true
  hosts: masters
  tasks:
  - name: Ping hosts
    ansible.builtin.ping:
  - name: Uninstall k3s server
    command: /usr/local/bin/k3s-uninstall.sh

- name: Uninstall k3s on bootstrap servers
  become: true
  hosts: bootstrapMaster
  tasks:
  - name: Ping hosts
    ansible.builtin.ping:
  - name: Uninstall k3s server
    command: /usr/local/bin/k3s-uninstall.sh
```

## Create the k3s cluster

Now for the fun part. Create the k3s cluster by executing the install playbook:

```bash
$ ansible-playbook -i inventory.yaml install-k3s-playbook.yaml
```

That's it! The output should look like this (my Raspberry Pis have cgroups enabled and are therefore skipping that task):

```bash
PLAY [Enable cgroups] ******************************************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************************************************************************************
ok: [nerminworker1]
ok: [nerminmaster]

TASK [Ping hosts] **********************************************************************************************************************************************************************************************************************************************************************
ok: [nerminmaster]
ok: [nerminworker1]

TASK [Check if cgroups are enabled] ****************************************************************************************************************************************************************************************************************************************************
changed: [nerminworker1]
changed: [nerminmaster]

TASK [Enable cgroups] ******************************************************************************************************************************************************************************************************************************************************************
skipping: [nerminmaster]
skipping: [nerminworker1]

PLAY [Install k3s bootstrap server] ****************************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************************************************************************************
ok: [nerminmaster]

TASK [Ping host] ***********************************************************************************************************************************************************************************************************************************************************************
ok: [nerminmaster]

TASK [Install k3s bootstrap server] ****************************************************************************************************************************************************************************************************************************************************
changed: [nerminmaster]

TASK [Extract K3S_TOKEN from server output] ********************************************************************************************************************************************************************************************************************************************
changed: [nerminmaster]

TASK [Set K3S_TOKEN as a fact] *********************************************************************************************************************************************************************************************************************************************************
ok: [nerminmaster]

PLAY [Install k3s servers] *************************************************************************************************************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [Install k3s workers] *************************************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************************************************************************************
ok: [nerminworker1]

TASK [Ping hosts] **********************************************************************************************************************************************************************************************************************************************************************
ok: [nerminworker1]

TASK [Install k3s workers] *************************************************************************************************************************************************************************************************************************************************************
changed: [nerminworker1]

PLAY [Fetch k3s kubeconfig] ************************************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************************************************************************************
ok: [nerminmaster]

TASK [Fetch kubeconfig] ****************************************************************************************************************************************************************************************************************************************************************
changed: [nerminmaster]

PLAY RECAP *****************************************************************************************************************************************************************************************************************************************************************************
nerminmaster               : ok=10   changed=4    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0
nerminworker1              : ok=6    changed=2    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0
```

The playbook downloaded the kubeconfig file from the bootstrap master to the current directory on the local machine with the name *k3sconfig*. The file looks like this:

```yaml
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ...
    server: https://127.0.0.1:6443
  name: default
contexts:
- context:
    cluster: default
    user: default
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
  user:
    client-certificate-data: ...
    client-key-data: ...
```

Replace the localhost IP in the `server` value with the IP or hostname of the bootstrap master node.

```yaml
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ...
    server: https://nerminmaster:6443  # <----- This value here.
  name: default
contexts:
- context:
    cluster: default
    user: default
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
  user:
    client-certificate-data: ...
    client-key-data: ...
```

Verify that the cluster is up and running:

```bash
$ kubectl get nodes --kubeconfig k3sconfig
NAME            STATUS   ROLES                       AGE     VERSION
nerminmaster    Ready    control-plane,etcd,master   8m42s   v1.24.10+k3s1
nerminworker1   Ready    <none>                      8m7s    v1.24.10+k3s1
```

Success.

## Optional: Deploy rancher server to k3s

It would be nice if we had a UI we could use to view and manage our cluster. Rancher server comes with this UI!

To deploy the Rancher server, execute the following:

```bash
## CERT MANAGER

# Add jetstack helm repo
helm repo add jetstack https://charts.jetstack.io
helm repo update

# Create cert-manager namespace
kubectl create namespace cert-manager --kubeconfig k3sconfig

# Install cert-manager
helm upgrade --install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --version v1.8.2 \
  --set installCRDs=true \
  --wait \
  --kubeconfig k3sconfig


## RANCHER SERVER

# Create cattle-system namespace
kubectl create ns cattle-system --kubeconfig k3sconfig

# Add rancher-latest helm repo
helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
helm repo update

# Install rancher
helm upgrade --install rancher rancher-latest/rancher \
  --version 2.7.0 \
  --namespace cattle-system \
  --set hostname=nermin.cluster \
  --set replicas=1 \
  --set rancherImageTag=v2.7.2-rc5-linux-arm64 \
  --wait \
  --kubeconfig k3sconfig
```

Create a record in your hosts file that points to a worker node's IP. On Windows, this file is located at `C:/Windows/System32/drivers/etc/hosts`. In the file, add the record:

```txt
192.168.0.66   nermin.cluster
```

Notice that this domain is the same as the `hostname` value in the Rancher chart.

Go to the domain [`nermin.cluster`](nermin.cluster). There is not a trusted certificate, so the browser will warn us against an untrusted site. Press *Advanced* and then *Continue to nermin.cluster (unsafe)*. Now, you will enter the login page.


![Rancher front page](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/520sf8vro5ejqnxwzfr1.png)

To acquire the bootstrap password, run:

```bash
$ kubectl get secret \
  --namespace cattle-system bootstrap-secret \
  -o go-template='{{.data.bootstrapPassword|base64decode}}{{"\n"}}' \
  --kubeconfig k3sconfig

qstl5v7n4j6tmmr89rnhhb6qcf69krfljnwxm96sds2xpgbm4zp4zf
```

Copy and paste the password into the input field, and you will be allowed to enter the Rancher server UI.


![Rancher workloads](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uinkz3m4es8pa6tgwron.png)

## Clean up

Run the *uninstall-k3s-playbook* to remove k3s from all the nodes:

```bash
$ ansible-playbook -i inventory.yaml uninstall-k3s-playbook.yaml
```

## Remarks

This setup allows easy scaling of the cluster up and down by only adding or removing master and worker nodes from the inventory file. However, there are some important points to be aware of:

- **Dynamic IPs**: The Raspberry Pis have dynamic IPs on the local network, meaning that their IPs can change when their current IP lease expires. This can disrupt the execution of the playbook. Consider assigning static IPs to the nodes.
- **Local domain**: Because the `nermin.cluster` domain has been defined in the hosts file, it can be resolved but only on the local machine. This is good for testing purposes, but if the cluster needs to be available to others on the same network, a real domain should be purchased and used.
- **Untrusted certificate**: The `nermin.cluster` domain is only resolvable on the local machine. No certificate has been signed by a trusted party for this domain, and therefore, it isn't trusted by the browser. If you plan on allowing access to other users, you should change the domain and acquire a certificate. The [cert-manager](https://cert-manager.io/docs/) chart can help you get free Let's Encrypt certificates.
- **Hardcoded values**: In order to acquire information from the bootstrap node like its IP address and the k3s token, we reference them by hardcoding the hostname in the hostvars dict `hostvars['nerminmaster']`. This means that a change to the hostname in the `bootstrapMaster` group will make the playbook fail. A change in any of the places requires a change in both the *inventory.yaml* and *install-k3s-playbook.yaml*.