K8s cluster with OCI free-tier and Raspberry Pi4 (part 4)
Tags: kubernetes, k8s, raspberrypi, oci
Published: 2023-02-15 19:31:21

This long read is a multiple part tutorial for building a Kubernetes cluster (using k3s) with 4 x OCI free-tier ARM instances and 4 x Raspberry Pi 4. Plus some applications needed for installation (Terraform and Ansible) and a lot of things installed on the cluster.
**Part 4 is adding applications to the cluster.
GitHub repository is [here](https://github.com/liviux/k8s-cluster-oci-rpi4)**
I'll just install the apps and access them. But you need to configure to use all of them.
##Lens
First thing installed is a beautiful dashboard - [Lens](https://k8slens.dev/). Install the desktop app on your PC, go to _File > Add Cluster_. You have to paste here all that you receive when running `kubectl config view --minify --raw` on server. Edit _127.0.0.1:6443_ from that result with your server IP, in my case _10.20.30.1:6443_.

##MetalLB
Metal LB will work as our load balancer, it will give an external IP to every service type _LoadBalancer_. Install it with `kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.7/config/manifests/metallb-native.yaml`. Create a file _config-metallb.yaml_ and write this in it:

```
---
apiVersion: metallb.io/v1beta1

kind: IPAddressPool
metadata:
  name: default
  namespace: metallb-system
spec:
  addresses:
  - 192.168.0.150 - 192.168.0.250

---
apiVersion: metallb.io/v1beta1

kind: L2Advertisement
metadata:
  name: default
  namespace: metallb-system
spec:
  ipAddressPools:
  - default
```
Edit IP range with your values from home network. Apply it with `kubectl apply -f config-metallb.yaml`. Now for every time in the rest of the guide you acces a service with node ip (10.20.30.1 -10.20.30.8:port) you could only use the external IP given by metallb.

##Helm & Arkade
On the server it seems git is not installed. So `sudo apt install` git first. Plus KUBECONFIG env variable wasn't configured until now. So add this line to _~/.bashrc_.
```
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
```
Helm is the package manager for Kubernetes. Installing helm is very easy, just run following commands on server:
```
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh
```
Arkade is an Open Source marketplace For Kubernetes. Installation is just `curl -sLS https://get.arkade.dev | sudo sh`.

##Bash completion

To run commands faster and easier you can use auto-completion for kubectl. Run `apt install bash-completion` and then `echo 'source <(kubectl completion bash)' >>~/.bashrc`. Now after every kubectl you can hit tab and it will autocomplete for you. Try with a running pod, just write `kubectl get pod` and hit TAB.

##Traefik dashboard
First write a yaml file _traefik-crd.yaml_ and apply it. It should have this content:

```
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    additionalArguments:
      - "--api"
      - "--api.dashboard=true"
      - "--api.insecure=true"
      - "--log.level=DEBUG"
    ports:
      traefik:
        expose: true
    providers:
      kubernetesCRD:
        allowCrossNamespace: true
```
This should be available at _http://10.20.30.4:9000/dashboard/#/_ in your browser (or any ip from those 8 that are nodes).

##Longhorn
Longhorn is a cloud native distributed block storage for Kubernetes. First create a new _install-longhorn.yml_ file for Ansible playbook. Paste in it :

```
---
- hosts: all
  tasks:
  - name: Install some package for Longhorns
    apt:
      name:
        - nfs-common
        - open-iscsi
        - util-linux
...
```
Run it with `ansible-playbook install-longhorn.yml -K -b` to install some extra components on nodes. Now run `ansible -a "lsblk -f" all` to find what are the names of the drives.
Move to server. Run `helm repo add longhorn https://charts.longhorn.io` then `helm repo update`
and then `helm install longhorn longhorn/longhorn --namespace longhorn-system --create-namespace`. It will last a while, ~7 minutes. Now create a _longhorn-service.yaml_ file and paste this:
```
apiVersion: v1
kind: Service
metadata:
  name: longhorn-ingress-lb
  namespace: longhorn-system
spec:
  selector:
    app: longhorn-ui
  type: LoadBalancer
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: http
```
Run it with `kubectl apply -f longhorn-service.yaml`. Now you can access the Longhorn Dashboard in your browser in _10.20.30.1:port_ (or any of your nodes IPs). The port you get it from running `bectl describe svc longhorn-ingress-lb -n longhorn-system | grep NodePort`
Now to make Longhorn default StorageClass. Run `kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'` and now you should have only one default in `kubectl get storageclass`.
##Portainer
This app is a container management platform. We'll be installed using again helm. Run `helm repo add portainer https://portainer.github.io/k8s/` and then `helm repo update`.
Now just run `helm install --create-namespace -n portainer portainer portainer/portainer`. You can acces portainer UI from 10.20.30.1:30777 (or any other IP from your Wireguard network 10.20.30.1 - 10.20.30.8 :30777). It uses 10 GB, check with `kubectl get pvc -n portainer` (you can check in your Longhorn Dashboard the new volume created). There you will create a user and password. This is what I have after clicking on _Get Started_:

![portainer](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/v4ggrycif2pqk6n4a1o5.png)

##ArgoCD
Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes.
Installation very easy with `kubectl create namespace argocd` then `kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml` and wait a few minutes, or check progress with `kubectl get all -n argocd`. Now to access the UI run `kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'`. Find your port with `kubectl describe service/argocd-server -n argocd | grep NodePort` and access the UI from 10.20.30.1:port (or another IP form your network). User is _admin _and password is stored in `kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo`.

##Prometheus & Grafana 
Prometheus is probably the best monitoring system out there and Grafana will be used as it's dashboard. Will be installed from the ArgoCD UI using a official helm chart _kube-prometheus-stack_ . Open _Applications_ and click _New App_. Edit:
- App Name : _kube-prometheus-stack_
- Project Name : _default_
- Sync Policy : _Automatic_
- check _Auto Create Namespace_ and _Server Side Apply_
- Repository URL : _https://prometheus-community.github.io/helm-charts_ (select _Helm_)
- Chart : _kube-prometheus-stack_
- Cluster URL : _https://kubernetes.default.svc_
- Namespace : _kube-prometheus-stack_
- alertmanager.service.type : _LoadBalancer_
- prometheus.service.type : _LoadBalancer_
- prometheusOperator.service.type : _LoadBalancer_
- grafana.ingress.enabled : _true_.

Now hit _Create_. We configured the services as _LoadBalancer_, because by default they are _ClusterIP_ and if you wanted to access them you have to do port-forwarding every time. You can acces the Prometheus UI from node:port (port you get it with `kubectl describe svc kube-prometheus-stack-prometheus -n kube-prometheus-stack | grep NodePort`). The same for Prometheus Alert Manager (get port with `kubectl describe svc kube-prometheus-stack-alertmanager -n kube-prometheus-stack | grep NodePort`). For Grafana Dashboard you need to go to any _nodeIP/kube-prometheus-stack-grafana:80_. User is admin and password get it with `kubectl get secret --namespace kube-prometheus-stack kube-prometheus-stack-grafana -o jsonpath='{.data.admin-password}' | base64 -d`.