How to create your own private cloud : part 1
Tags: ubuntu, raspberrypi, k3s, kubernetes
Published: 2021-08-03 01:15:39

Hello folks,

Have you ever wondered how do cloud providers build their cloud ? How do circle-ci, github actions, gitlab-ci do run your tests and builds your projects in their cloud ?

Don't wonder anymore, because in this series, i'll show to build your own mini-cloud at home.

**Disclaimer**:
- it will require some hardware investment,
- it will give you a glimpse of hardware architecture, but not full picture (we will not cover spine/leaf nodes architecture used in cloud datacenters)

## What hardware do you need ?
- At least two raspberry pi with 4GB each (official price [55$](https://www.raspberrypi.org/products/raspberry-pi-4-model-b/) each),
- One MicroSD card for each Raspberry pi (i recommend taking at least a 32GB card),
- [optional] A Raspberry pi POE+ hat for each Raspberry pi (official price [20$](https://www.raspberrypi.org/products/poe-hat/) each)
- [optional] A POE+ ethernet switch (or the official raspberry pi [power supply](https://www.raspberrypi.org/documentation/hardware/raspberrypi/power/README.md) if you can't afford a POE switch and want to use wifi for networking)
- [Optional] A cluster case for a clean setup.

I made one based on 4 Raspberry pi's with POE+ hat and a POE+ switch. Total Budget : 400$.  
If you go with only two RPI's over WIFI, it will cost you around 150$.  

## What it looks like?

### night mode

![night cluster](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/t9t9i3vor1cdqtzfxh84.jpg)

### day mode

![day cluster](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/v8t1u3spq96m2n0lx11r.jpg)

## Install Ubuntu server ARM64 on SD card

We could install the official Raspberry pi OS on each card, but unfortunately, raspberry only support 32 bit images officially (their 64 bit OS is still broken from time to time), and we want to use the full power of Raspberry pi 4 which is 64 bit. See this benchmark to compare for yourself the huge difference :
![ARM32 vs ARM64](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9yqaknmdagonr1022y0s.png)
So here i'll recommend using Ubuntu Server 20.04 LTS ARM64 build provided by [Raspberry pi Imager](https://www.raspberrypi.org/software/).

![imager](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x8fhjjwta4d95ktoju7o.png)

Select `Other general purpose OS`>`Ubuntu`> `Ubuntu server 20.04 LTS`
![Other](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kmjijzw4k7j5zm99mcnm.png)
![ubuntu](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5xjw6wfkni2yu7ibl3kr.png)

And click on the `Write` button.

## SD Card Modification

Before putting your SD card on your raspberry pi, you first need to make a few changes to allow you to use it in Headless mode (ie: without a screen and keyboard linked to it).

With your SD card still inserted, you should see two new disks :
- bootfs
- writable

### Enable SSH
Still at the root of `bootfs` disk, create an empty file named `ssh` :
```bash
touch ssh
```
This will enable `ssh` connections on your Raspberry pi.

### Enable container cpu and memory isolation

Open a Terminal and go to the `bootfs` filesystem of your SD Card, and open the `cmdline.txt` file.
You should see a single line, append this at the end of the line :
```txt
cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory
```
This will enable you to use kubernetes (k3s to be precise) on your cluster.

### **Optional** Enable POE+ hat driver

Open `userconfig.txt` and append these lines at the end of the file :
```txt
dtoverlay=rpi-poe
dtparam=poe_fan_temp0=45000
dtparam=poe_fan_temp1=55000
dtparam=poe_fan_temp2=65000
dtparam=poe_fan_temp3=75000
```
This will enable POE+ hat driver and fan configuration to cool down your Raspberry pi.

## Rename your Rasberrys
Now go to the **writable** disk and edit as an admin the `etc/hosts` and append at the end:
```txt
127.0.1.1 <your-new-name>
```
For the first SD card, i suggest naming it to recognize that's it will be the master node of your cluster. Mine is called **`k3s-main`**. For the other ones, you can choose a predictable name like **`k3s-worker-01`** ...

Also edit `etc/hostname` and change the current name (ubuntu) to the name you just choose.

## Insert your SD card and boot

Now you can insert your SD card in your Raspberry pi's. put power on and let them boot.

### Connect to each of them with ssh

Now you need to connect to each of your booted Raspberry pi's. If you have wifi and a dhcp server, you might be able to connect to them using their name with .local appended to it. For exemple :
```bash
> ssh ubuntu@k3s-main.local
```
When prompted for password, just use `ubuntu`. it will ask you to change the default password after a successfull login.
Once the password changed, it will disconnect you.

If it does not work, don't worry. First run `ifconfig` to get information about your network.
```bash
> ifconfig
etho0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet <YOUR.LOCAL.IP>  netmask 255.255.0.0  broadcast 172.19.255.255
        ether <MAC:ADDRESS>  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
```
This should give you your local IP. replace the last two digits by `00` and run nmap :
```bash
> nmap -sn <YOUR.LOCAL.IP.ZERO>/24
```
This should return your your IP, plus the one of your Raspberry pi's. you should now be able to login using ssh and the raspberry IP
```bash
> ssh ubuntu@<Rasbpberry.IP>
```
If you have already setup a ssh key, i recommand using it to automatically connect to your Raspberry pi's without password prompt. To do this :
```bash
> ssh-copy-id -i ~/.ssh/id_rsa.pub ubuntu@k3s-main.local
```
Enter password for the last time when prompted.

### Update your Raspberry pi's

On each Raspberry pi's within the SSH session, run :
```bash
> sudo apt update & sudo apt upgrade
```

## Install k3s

### Main node

On your main node (k3s-main for me), run the following command :
```bash
> curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644
```
Once the script finished execution, you should be able to see your node running :
```bash
> kubectl get nodes
NAME      STATUS    ROLES     AGE     VERSION
k3s-main   Ready    control-plane,master    1m      v1.21.0
```
### Save your cluster token

save your token to allow your workers to join the main node
```bash
> cat /var/lib/rancher/k3s/server/node-token
<YOUR-TOKEN>
```

### Join main node on workers

On each of your worker nodes (k3s-worker-01, k3s-worker-02, k3s-worker-03 for me) join the main node by running the following command :
```bash
> curl -sfL https://get.k3s.io | K3S_URL=https://k3s-main.local:6443 K3S_TOKEN=<YOUR-TOKEN> sh -
```

Once the scripts finished execution, you should be able to see your nodes running (execute on main node):
```bash
> kubectl get nodes
NAME      STATUS    ROLES     AGE     VERSION
k3s-main   Ready    control-plane,master    15m      v1.21.0
k3s-worker-01   Ready     <none>    3m      v1.21.0
k3s-worker-02   Ready     <none>    2m      v1.21.0
k3s-worker-03   Ready     <none>    1m      v1.21.0
```

## Conclusion

So we have installed a cluster on raspberry pi's. what now ?

Next time, we'll see how to install [rancher](https://rancher.com/docs/rancher/v2.5/en/installation/install-rancher-on-k8s/) on your cluster to monitor and administrate it with a beautifull UI.