Install RKE2 with Cilium and Metallb
Tags: rancher, metallb, cilium, kubernetes
Published: 2024-04-03 15:55:46


## Introduction
In today’s rapidly evolving technological landscape, container orchestration has become a critical component for managing complex applications. Kubernetes is an open-source platform that revolutionizes how we deploy, scale, and manage containerized workloads. Its advantages are manifold: it provides seamless scalability, high availability, and declarative configuration. With a thriving ecosystem and a robust community, Kubernetes empowers developers to focus on innovation rather than infrastructure intricacies.

But what about simplicity and security? That’s where Rancher RKE2 steps in. This lightweight Kubernetes distribution offers a streamlined experience without compromising on safety. With a single binary installation, RKE2 simplifies setup and maintenance. Its built-in high availability, enhanced security features, and operator-friendly design make it an attractive choice for modern deployments. Whether you’re a seasoned DevOps engineer or a curious developer, both Kubernetes and RKE2 provide the tools needed to thrive in this dynamic era of software development.
<br/>


## Prerequisites
First things first, we need to prepare our machines in order for rke2 cluster to work properly.
These include operating system, networking, and firewall that will be applied to all nodes.

Also, we will be deploying our cluster on 6 Linux servers with the following specifications:
* **Control-Plane-1** => IP: 192.168.100.11 , OS: Debian 12 , Hostname: kuber-master-1

* **Control-Plane-2** => IP: 192.168.100.12 , OS: Debian 12 , Hostname: kuber-master-2

* **Control-Plane-3** => IP: 192.168.100.13 , OS: Debian 12 , Hostname: kuber-master-3

* **Worker-1** => IP: 192.168.100.14 , OS: Debian 12 , Hostname: kuber-worker-1

* **Worker-2** => IP: 192.168.100.15 , OS: Debian 12 , Hostname: kuber-worker-2

* **Worker-3** => IP: 192.168.100.16 , OS: Debian 12 , Hostname: kuber-worker-3

* **L4 Load Balancer** => ip: 192.168.100.100
<br/>

**NOTE**: We've assumed that there is an external load balancer, which could be a cloud load balancer or an on-premise node. The load balancer is also accessible with the following domains:
* rancher.arman-projects.com
* kubernetes.arman-projects.com
* rke2.arman-projects.com
* k8s.arman-projects.com
<br/>

**NOTE**: In this article, we use the terms server node which refers to the control-plane node, and agent node which refers to the worker node. We use these terms interchangeably.
<br/>

### Operating System Pre-requisites

The very basic thing to do is to update the servers.
```bash
apt-get update && apt-get upgrade -y
```
next reboot the server.
```bash
reboot
```
<br/>

### Networking Pre-requisites

First of all, we must enable 2 kernel modules.
Run the following:
```bash
modprobe br_netfilter
modprobe overlay
cat <<EOF | tee /etc/modules-load.d/k8s.conf
br_netfilter
overlay
EOF
```

Now configure sysctl to prepare OS for Kubernetes and of course for security purposes.

```bash
cat <<EOF | tee /etc/sysctl.conf
net.ipv4.ip_forward=1
net.ipv4.conf.all.send_redirects=0
net.ipv4.conf.default.send_redirects=0
net.ipv4.conf.default.accept_source_route=0
net.ipv4.conf.all.accept_redirects=0
net.ipv4.conf.default.accept_redirects=0
net.ipv4.conf.all.log_martians=1
net.ipv4.conf.default.log_martians=1
net.ipv4.conf.all.rp_filter=1
net.ipv4.conf.default.rp_filter=1
net.ipv6.conf.all.accept_ra=0
net.ipv6.conf.default.accept_ra=0
net.ipv6.conf.all.accept_redirects=0
net.ipv6.conf.default.accept_redirects=0
kernel.keys.root_maxbytes=25000000
kernel.keys.root_maxkeys=1000000
kernel.panic=10
kernel.panic_on_oops=1
vm.overcommit_memory=1
vm.panic_on_oom=0
net.ipv4.ip_local_reserved_ports=30000-32767
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-arptables=1
net.bridge.bridge-nf-call-ip6tables=1
EOF
```
After sysctl is configured, 
For changes to take effect run sysctl command below:
```bash
sysctl --system
```
<br/>


### Firewall Pre-requisites
There are some ports used by rke2 and Kubernetes services that must be opened between nodes. The list of ports can be found at "https://docs.rke2.io/install/requirements".
For simplicity, we've allowed all traffic between cluster nodes and just allowed some ports access only through the Load balancer.
First, flush all nftables rules.
```bash
nft flush ruleset
```
Then add these contents to the nftables default config file.
```bash
#!/usr/sbin/nft -f

flush ruleset

table inet filter {
  chain input {
    type filter hook input priority filter; policy accept;
    tcp dport 22 accept;
    ct state established,related accept;
    iifname "lo" accept;
    ip protocol icmp accept;
    ip daddr 8.8.8.8 tcp dport 53 accept;
    ip daddr 8.8.8.8 udp dport 53 accept;
    ip daddr 8.8.4.4 tcp dport 53 accept;
    ip daddr 8.8.4.4 udp dport 53 accept;
    ip daddr 1.1.1.1 tcp dport 53 accept;
    ip daddr 1.1.1.1 udp dport 53 accept;
    ip saddr 192.168.100.11 accept;
    ip saddr 192.168.100.12 accept;
    ip saddr 192.168.100.13 accept;
    ip saddr 192.168.100.14 accept;
    ip saddr 192.168.100.15 accept;
    ip saddr 192.168.100.16 accept;
    ip saddr 192.168.100.100 tcp dport {9345,6443,443,80} accept;
    ip saddr 192.168.100.100 udp dport {9345,6443,443,80} accept;
    counter packets 0 bytes 0 drop;
  }

  chain forward {
    type filter hook forward priority filter; policy accept;
  }

  chain output {
    type filter hook output priority filter; policy accept;
  }
}
```
In the firewall config above, all traffic from cluster nodes is allowed to reach each other and just 4 ports are supposed to access the cluster through the Load balancer.
**NOTE**: The ssh port is 22 and it's benn opened from all sources.
Enable and restart the nftables service.
```bash
systemctl enable nftables
systemctl restart nftables
```
<br/>


## Install Rancher rke2

Till now, we've prepared our machines. It's time to deploy rke2.
The subsequent commands will configure and install Rancher rke2.
<br/>


## Control Plane 1
<br/>


### Set up Rancher configs

Create Rancher configuration directory.
```bash
mkdir -p /etc/rancher/rke2/
```
We need to change the default options and arguments of rke2. In order to do that, create a file named **config.yaml** and put the below lines in it.
```yaml
write-kubeconfig-mode: "0644"
advertise-address: 192.168.100.100
tls-san:
  - 192.168.100.100
  - rancher.arman-projects.com
  - kubernetes.arman-projects.com
  - rke2.arman-projects.com
  - k8s.arman-projects.com
cni: none
cluster-cidr: 10.100.0.0/16
service-cidr: 10.110.0.0/16
cluster-dns: 10.110.0.10
cluster-domain: arman-projects.com
etcd-arg: "--quota-backend-bytes 2048000000"
etcd-snapshot-schedule-cron: "0 3 * * *"
etcd-snapshot-retention: 10
disable:
  - rke2-ingress-nginx
disable-kube-proxy: true
kube-apiserver-arg:
  - '--default-not-ready-toleration-seconds=30'
  - '--default-unreachable-toleration-seconds=30'
kube-controller-manager-arg:
  - '--node-monitor-period=4s'
kubelet-arg:
  - '--node-status-update-frequency=4s'
  - '--max-pods=100'
egress-selector-mode: disabled
protect-kernel-defaults: true
```
Let's explain the above options:

* **write-kubeconfig-mode**: The permission of the generated kubeconfig file.
* **advertise-address**: Kubernetes API server address that all nodes must connect to.
* **tls-san**: Valid addresses for Kubernetes client certificate. kubectl trusts these addresses and any others are not trusted.
*  **cni**: The CNI plugin that must get installed. Here we put none which indicates no CNI should be installed.
*  **cluster-cidr**: The CIDR used in pods.
*  **service-cidr**: The CIDR used in services.
*  **cluster-dns**: Coredns service address.
*  **cluster-domain**: The Kubernetes cluster domain. The default value is cluster.local
*  **etcd-arg**: etcd database arguments. Here we've just increased the default etcd allowed memory usage.
*  **etcd-snapshot-schedule-cron**: Specifies when to perform an etcd snapshot.
*  **etcd-snapshot-retention**: Specifies how many snapshots will be kept.
*  **disable**: instructs rke2 to not deploy the specified add-ons. Here we've disabled the nginx ingress add-on.
*  **disable-kube-proxy**: Whether or not to use kube-proxy for pod networking. Here we have disabled the kube proxy so as to use Cilium instead.
*  **kube-apiserver-arg**: Specifies kube api server arguments. Here we've set default-not-ready-toleration-seconds and default-unreachable-toleration-seconds to 30 seconds. The default value is 300 seconds, so in order to reschedule pods faster and maintain service availability, the default values have been decreased.
*  **kube-controller-manager-arg**: Specifies the interval of kubelet health monitoring time.
*  **kubelet-arg**: Sets kubelet arguments. node-status-update-frequency specifies the time in which node status is updated and max-pods is the maximum number of pods allowed to run on that node.
*  **egress-selector-mode**: Disable rke2 egress mode. By default this value is set to agent and Rancher rke2 servers establish a tunnel to communicate with nodes. This behavior is due to prevent opening several connections over and over. In some cases, enabling this mode will cause some routing issues in your cluster, so it's been disabled in our scenario.
*  **protect-kernel-defaults**: compare kubelet default parameters and OS kernel. If they're different, the container is killed.
<br/>

###(Optional) Taint Control Plane nodes
It's preferable to taint the Control Plane nodes so the workload pods won't get scheduled on those. To do so, edit `/etc/rancher/rke2/config.yaml` and add the following line inside it.
```yaml
node-taint:
  - "CriticalAddonsOnly=true:NoExecute"
```
<br/>

###(Optional) Set up offline files
In order to speed up the bootstrapping process or to install Rancher in an air-gapped environment, we may want to download image files and put them in our nodes.
**NOTE**: Using offline files is useful but optional. In case you've got a poor internet connection or installing in an air-gapped environment, you should use the offline files.

Go to "https://github.com/rancher/rke2/releases" and select a release. After that, you should download ***rke2-images-core.linux-amd64.tar.gz***.
Create the Rancher data directory to put the compressed offline images.
```bash
mkdir -p /var/lib/rancher/rke2/agent/images
```
Put the compressed files in the Rancher data directory.
```bash
mv  rke2-images-core.linux-amd64.tar.gz /var/lib/rancher/rke2/agent/images/
```
<br/>

### Install rke2

Download rke2 installer script.
```bash
curl -sfL https://get.rke2.io > install_rke2.sh
```
Execute the binary with your desired arguments.
Here we've specified two arguments, the rke2 version and installation type:
```bash
chmod ug+x install_rke2.sh
INSTALL_RKE2_VERSION="v1.29.4+rke2r1" INSTALL_RKE2_TYPE="server" ./install_rke2.sh
```
**NOTE**: In this tutorial, we've installed rke2 version 1.29.4. You can change it in your environment.

Now you must have 2 new services available on your system, rke2-server and rke2-agent.

For server nodes, we just need rke2-server, so you should disable and mask the rke2-agent service:
```bash
systemctl disable rke2-agent && systemctl mask rke2-agent
```
It's time to initiate our first cluster node. Start rke2-server service. The rke2-server service will bootstrap your control-plane node.
```bash
systemctl enable --now rke2-server.service
```
**NOTE**: This will take some time depending on your network bandwidth.

If no error occurred, verify the rke2 status:
```bash
systemctl status rke2-server
```
In case a problem has shown up, check logs to troubleshoot it:
```bash
journalctl -u rke2-server -f
```

To check the pods in the cluster, you'll need to use kubectl.
The Rancher installation has downloaded the necessary binaries(e.g. kubectl, ctr, containerd,...) and put them in /var/lib/rancher/rke2/bin. Add the path to the default OS path.
```bash
PATH=$PATH:/var/lib/rancher/rke2/bin
```
Also, Rancher has generated **KUBECONFIG** which can be used by kubectl. By default, the KUBECONFIG has been set to use 127.0.0.1 as the API server which in our case should be set to an external address that is our load balancer(i.e. 192.168.100.100). Proceed with the subsequent steps:
```bash
mkdir ~/.kube
cp /etc/rancher/rke2/rke2.yaml ~/.kube/config
sed -i 's/127.0.0.1/192.168.100.100/g' ~/.kube/config
```
If you cat `~/.kube/config`, you'll have something like this:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/asgv4z41iammn30rwuq5.png)
You should now be able to use kubectl to check the status of your cluster.

**NOTE**: If you use kubectl, you'll see that nodes are in **NotReady** state and the reason for that, is due to the absence of a CNI plugin.
<br/>

### Install Cilium

We're ready to install Cilium as the CNI in our cluster. 

The first step is to install Cilium CLI. To do so run the following commands to download and install the CLI.
```bash
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
curl -L --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
```
**NOTE**: As of this day, our Cilium stable version is 1.15.4 and the version of Cilium CLI is 0.16.6.


Add the following config to a file named **cilium.yaml**:
```bash
cluster:
  name: cluster-1
  id: 10
prometheus:
  enabled: true
  serviceMonitor:
    enabled: false
dashboards:
  enabled: true
hubble:
  metrics:
    enabled:
    - dns:query;ignoreAAAA
    - drop
    - tcp
    - flow
    - icmp
    - http
    dashboards:
      enabled: true
  relay:
    enabled: true
    prometheus:
      enabled: true
  ui:
    enabled: true
    baseUrl: "/"
version: 1.15.4
operator:
  prometheus:
    enabled: true
  dashboards:
    enabled: true
```
**NOTE**: The Version of Cilium is 1.15.4. You can change it if there is a newer version, but be cautious to see the changelog of each version.

Apply the config:
```bash
cilium install -f cilium.yaml
```

Verify the status of Cilium:
```bash
cilium status
```
Now your node should be in **ready** state:
```
kubectl get nodes
```

By default, Cilium uses its own IP CIDR for pods and not the one configured during Cluster bootstrapping. To change this behavior, edit the Cilium config map:
```bash
kubectl -n kube-system edit cm cilium-config
```
There is a line like this:
```yaml
ipam: cluster
```
Change the value to "kubernetes":
```yaml
ipam: kubernetes
```
Save your changes and exit.

The already running pods are still using the default Cilium ipam. To apply yours, either restart the server or restart Cilium resources.
```bash
kubectl -n kube-system rollout restart deployment cilium-operator
kubectl -n kube-system rollout restart ds cilium
```
**NOTE**: check other pods too. If they're using the old IPs, restart them too.

Our first Control plane node is ready. 

Next, we'll proceed to join the other nodes to our initialized cluster.
<br/>


## Control Plane 2 & Control Plane 3

The procedure to join other server nodes is mostly the same as bootstrapping the first server node. The only difference is when we set which server node our new node should join to. So you should be able to follow the tutorial for all of the Server nodes 2, 3, etc.

Let's go through the steps
<br/>

### Set up Rancher configs

We need to change the default options and arguments of rke2.

Create the Rancher configuration directory and create and edit the Rancher config file:
```bash
mkdir -p /etc/rancher/rke2/
vim /etc/rancher/rke2/config.yaml
```
The configuration options are the same as the first server node. The only difference is the first 2 lines: **server** and **token**:

* **server**: This is a fixed registration address. It's a layer 4 load balancer and will distribute requests on our server nodes(the nodes that are running rke2-server service).
* **token**: The registration address. It's been generated by the first server node we bootstraped the cluster. The token value can be obtained from `/var/lib/rancher/rke2/server/node-token`.

Add these lines to the config.yaml:
```yaml
server: https://192.168.100.100:9345
token: XXXXXXXXXX
.
.
.
.
```
**NOTE**: We've omitted the next lines as they're the same as the first node.
<br/>

###(Optional) Set up offline files

Go to "https://github.com/rancher/rke2/releases" and download ***rke2-images-core.linux-amd64.tar.gz***.

Create the Rancher data directory to put the compressed offline images.
```bash
mkdir -p /var/lib/rancher/rke2/agent/images
mv  rke2-images-core.linux-amd64.tar.gz /var/lib/rancher/rke2/agent/images/
```
<br/>


### Install rke2

Download rke2 installer script. Make it executable and install the rke2 server:
```bash
curl -sfL https://get.rke2.io > install_rke2.sh
chmod ug+x install_rke2.sh
INSTALL_RKE2_VERSION="v1.29.4+rke2r1" INSTALL_RKE2_TYPE="server" ./install_rke2.sh
```
Disable and mask rke2-agent service:
```bash
systemctl disable rke2-agent && systemctl mask rke2-agent
```
Start rke2-server service:
```bash
systemctl enable --now rke2-server.service
```
If no error occurred, verify the Rancher status:
```bash
systemctl status rke2-server
```
In case a problem has arised, check logs to troubleshoot it:
```bash
journalctl -u rke2-server -f
```
<br/>


## Worker 1

It's time to join our first agent(worker) node to the cluster. All Kubernetes workloads are handled by our worker nodes.
<br/>

### Set up Rancher configs

Create the Rancher configuration directory.
```bash
mkdir -p /etc/rancher/rke2/
```
We need to change the default options and arguments of the rke2 agent. For that purpose, create the Rancher config file named **config.yaml** and put the below lines in it.
```yaml
server: https://192.168.100.100:9345
token: XXXXXXXXXX
node-name: kuber-worker-1
kubelet-arg:
  - '--node-status-update-frequency=4s'
  - '--max-pods=100'
```
Let's explain the above options:
* **server**: As we mentioned in previous sections, This is a fixed registration address. It's the layer 4 load balancer and will distribute requests on our server nodes(the nodes that are running the rke2-server service).
* **token**: The registration address. It's been generated by the first server node we bootstraped the cluster. The token value can be obtained from `/var/lib/rancher/rke2/server/node-token`.
* **node-name**: A unique name for this worker node. This name is used by Rancher to identify node and must be unique.
* **kubelet-arg**: Kubelet specific arguments. As you've seen, all server nodes have the same kubelet arguments as this agent node. The kubelet arguments are the same across the cluster. Here we've specified two arguments for kubelet, one for the frequency of status updates reported to the kube-api-server and the other one for the maximum allowed pods on a single node.

###(Optional) Set up offline files
Like the previous parts, we can provide offline files to install in an air-gapped environment or to make bootstrapping faster.

Go to "https://github.com/rancher/rke2/releases" and select a release. After that you should download ***rke2-images-core.linux-amd64.tar.gz***.
Create Rancher data directory to put the compressed offline images.
```bash
mkdir -p /var/lib/rancher/rke2/agent/images
```
Put the compressed files in the Rancher data directory.
```bash
mv  rke2-images-core.linux-amd64.tar.gz /var/lib/rancher/rke2/agent/images/
```
<br/>

### Install rke2

Download the rke2 installer script.
```bash
curl -sfL https://get.rke2.io > install_rke2.sh
```
Execute the binary with your arguments. Note that as we're installing on the agent node, you must specify **agent** mode in the arguments.
```bash
chmod ug+x install_rke2.sh
INSTALL_RKE2_VERSION="v1.29.4+rke2r1" INSTALL_RKE2_TYPE="agent" ./install_rke2.sh
```
Disable and mask the rke2-server service as there is no need for it:
```bash
systemctl disable rke2-server && systemctl mask rke2-server
```
Now we're about to join the first agent node to the cluster. Start rke2-agent service.
```bash
systemctl enable --now rke2-agent.service
```
**NOTE**: This will take some time depending on your network bandwidth.

If no error occurred, verify the Rancher status:
```bash
systemctl status rke2-agent
```
In case an error has shown, check logs to troubleshoot it:
```bash
journalctl -u rke2-agent -f
```
<br/>


## Worker 2 & Worker 3

The instructions on how to join the second and other worker nodes are exactly the same as the first one. The only difference is the node name.

In the Rancher configuration file located in `/etc/rancher/rke2/config.yaml`, the **node-name** option must be unique and different from the other nodes(e.g. kuber-worker-2, kuber-worker-3). For other parts, proceed with **Worker 1** guideline.


Congratulations! Now we've got a working Rancher Kubernetes cluster with 3 control plane and 3 worker nodes. 
<br/>


## Deploy MetalLB

The last part is about deploying Metallb as a load balancer and IP pool management service. If you've got a cloud load balancer, then skip the rest of the tutorial.
<br/>

### Install MetalLB

For this part, we'll be using Helm to deploy MetalLB. The first thing to do is to add the MetalLB helm repo:
```bash
helm repo add metallb https://metallb.github.io/metallb
```
Now Install MetalLB:
```bash
helm install --create-namespace --namespace metallb-system metallb metallb/metallb
```
Verify the pods are all in running state.
```bash
kubectl get pods -n metallb-system
```
<br/>

### Configure IP Address Pool

Let's assume we have a range IP starting from 192.168.100.50 and ending with 192.168.100.55. To create a pool for this range and have MetalLB assign IPs to services, create a yaml file(e.g. main-pool.yaml) and add the below content to it:
```yaml
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: main-pool
  namespace: metallb-system
spec:
  avoidBuggyIPs: true
  addresses:
  - 192.168.100.50-192.168.100.55
  serviceAllocation:
    serviceSelectors:
      - matchLabels:
         ip-pool: main-pool
```
**NOTE**: We've provided **serviceSelectors** section to only assign IPs to services that have the matching label.

Apply the pool:
```bash
kubectl apply -f main-pool.yaml
```
MetalLB will assign IP to any service of type LoadBalancer that has the label "main-pool".
The only remaining part is to advertise our IP pool in the entire cluster. Use **L2Advertisement** kind provided by MetalLB to do so.

Create a yaml file(e.g. l2advertisement.yaml) and add the following manifest in it:
```yaml
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: main-advertisement
  namespace: metallb-system
spec:
  ipAddressPools:
    - main-pool
```
**NOTE**: If there was more than one pool, we could've advertised them all by this manifest and passed the pool names to **ipAddressPools** section as a list.

Apply the manifest:
```bash
kubectl apply -f l2advertisement.yaml
```

Everything is ready for you to deploy your workload into this cluster.
Good luck:)
<br/>


## Summary
In this essay, we showed how to use Rancher rke2 to deploy a Kubernetes cluster with 6 Debian nodes with firewall enabled. We've also covered deploying Cilium as a CNI for our cluster and have it completely replace kube-proxy so as to increase speed and gain more observability via Cilium tools.
This article also showed how to deploy Metallb to manage IP pools and load balance traffic for those IP pools.
Throughout this guide, we assumed that we have an external load balancer that will distribute traffic to our workload and control plane nodes.
<br/>

For further information please visit rke2 and MetalLB official documents:
* https://docs.rke2.io/
* https://metallb.universe.tf/configuration/
* https://artifacthub.io/packages/helm/metallb/metallb
<br/>

You may also want to visit my other article about deploying Rancher rke2 with Cilium and Metallb if you're interested:
https://dev.to/arman-shafiei/deploy-nginx-load-balancer-for-rancher-lgk
<br/>


Thank you for reading this post and please leave a comment if you have any suggestions or figured out an issue with this article.