Setting Up Kubernetes Cluster with K3S
Tags: kubernetes, k3s, opensource
Published: 2023-04-18 11:51:55

Deploying a high availability Kubernetes cluster using k3s with 3 masters and ETCD as the storage, the backend is a reliable way to ensure that your applications run seamlessly, even when one or more nodes fail. In this article, we will guide you through the process of setting up such a cluster with detailed examples.

## Prerequisites

Before you start deploying a k3s cluster with high availability, you will need to prepare the following:
* Three or more servers with 3 running any Linux distribution (these servers will be used as Kubernetes nodes).
* A user account with sudo privileges on each server.
* A working network connection between the servers.
* A firewall is installed on each server, allowing traffic on ports 6443, 2379, 2380, and 8472.

## NEW CLUSTER

To run K3s in this mode, you must have an odd number of server nodes. We recommend starting with three nodes. The odd number of nodes is to prevent a split brain. A split-brain is a state of a server cluster where nodes diverge from each other and have conflicts when handling incoming I/O operations. The servers may record the same data inconsistently or compete for resources. In a situation where there is an odd number, there will always be a majority vote, which can automatically be elected as the source of truth for the conflict.
To get started, first launch a server node with the cluster-init flag to enable clustering and a token that will be used as a shared secret to join additional servers to the cluster. 

**NOTE** Replace the token with something secure and keep it safe from third-party access

```bash

curl -sfL https://get.k3s.io |K3S_TOKEN=123456789 sh -s - server --cluster-init


```

After launching the first server, join the second and third servers to the cluster using the shared secret with the server IP address of the first node (the one used in the initial step), in this scenario, we will assume that IP is 10.51.10.10 :

```bash

curl -sfL https://get.k3s.io | K3S_TOKEN=123456789 sh -s - server --server https://10.51.10.10:6443

```

Now you have a highly available control plane. Any successfully clustered servers can be used in the --server argument to join additional server and worker nodes. Joining additional worker nodes to the cluster follows the same procedure as a single server cluster.
There are a few config flags that must be the same in all server nodes (You don't have to worry about these if you use the above method as it automatically configures all these for you):
* Network-related flags: --cluster-dns, --cluster-domain, --cluster-cidr, --service-cidr
* Flags controlling the deployment of certain components: --disable-helm-controller, --disable-kube-proxy, --disable-network-policy and any component passed to --disable
* Feature-related flags: --secrets-encryption

## Cluster Access

The `kubeconfig` file is stored at `/etc/rancher/k3s/k3s.yaml` is used to configure access to the Kubernetes cluster. If you have installed upstream Kubernetes command line tools such as kubectl or helm you will need to configure them with the correct kubeconfig path. This can be done by either exporting the `KUBECONFIG` environment variable or invoking the --kubeconfig command line flag. Refer to the examples below for details.
The configuration file `/etc/rancher/k3s/k3s.yaml` can be found on any of the master servers in your cluster. Copy this file to your local machine and install `kubectl` on your local dev machine:
Check the official Kubernetes website for instructions on how to install kubectl on your operating system.
Leverage the `KUBECONFIG` environment variable:

```bash

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

kubectl get pods --all-namespaces

```

If you get a list of namespaces on your cluster, then you are good to go.
While this setup is sufficient to get you started, we also need to have a highly available storage service. K3S comes with hostPath storage, which is not ideal for HA workloads.

## Longhorn

[Longhorn](https://longhorn.io) delivers simplified, easy-to-deploy and upgrade, 100% open source, cloud-native persistent block storage without the cost overhead of open core or proprietary alternatives.
The Longhorn block storage can easily be added to your cluster with the following command

```bash

kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/longhorn.yaml

```

## Conclusion

You have now finally deployed an enterprise-grade Kubernetes cluster with k3s. You can now deploy some work on this cluster. Some components to take note of are for ingress, you already have [Traefik](https://traefik.io) installed, [longhorn](https://longhorn.io) will handle storage and [Containerd](https://containerd.io) as the container runtime engine.