Kubernetes on CRI-O (CentOS)
Tags: kubernetes, linux, docker, opensource
Published: 2020-12-14 17:45:00

## Docker is now deprecated in Kubernetes
**Yes, it is true.** 

Docker support in the kubelet is now deprecated and will be removed in a future release. The kubelet uses a module called "dockershim" which implements CRI support for Docker and it has seen maintenance issues in the Kubernetes community.

So, Now Kubernetes is encouraging to use different Container Runtime Interface instead of docker.
  
## Why CRI-O ?
CRI-O is an implementation of the Kubernetes CRI (Container Runtime Interface) to enable using OCI (Open Container Initiative) compatible runtimes.

It is a lightweight alternative to using Docker, Moby or rkt as the runtime for Kubernetes.

CRI-O is a CRI runtime mainly developed by Red Hat folks. In fact, this runtime is used in Red Hat OpenShift now. Yes, they do not depend on Docker anymore.

Interestingly, RHEL 7 does not officially support Docker either. Instead, they provide Podman, Buildah and CRI-O for container environment.

## Let's get started 

Here we are deploying a 3 node cluster:
- one master node
- two worker nodes

### Installing CRI-O (should be done on all nodes)

- Create a enviroment variable for OS.
```
$ export OS=CentOS_7
```
> Note: If you are using CentOS 8 then give variable name as CentOS_8

- Create another environment variable for VERSION of crio.
```
$ export VERSION=1.17
```
> Note: If you are using CentOS 8 then you can also install crio version 1.18

- Configure REPO for cri-o.
```
$ sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/devel:kubic:libcontainers:stable.repo
```
```
$ sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo
```
- Install cri-o. 
```
$ sudo yum install cri-o -y
```
- Start and enable the cri-o service.
```
$ sudo systemctl start cri-o 
```
- If any issue with starting cri-o, refer troubleshooting section !!
```
$ sudo systemctl enable cri-o
```
### Troubleshooting

#### Not able to start cri-o service:
```
$ sudo systemctl status cri-o

crio.service - Container Runtime Interface for OCI (CRI-O)
   Loaded: loaded (/usr/lib/systemd/system/crio.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2020-12-11 05:42:49 EST; 29s ago
     Docs: https://github.com/cri-o/cri-o
  Process: 1259 ExecStart=/usr/bin/crio $CRIO_CONFIG_OPTIONS $CRIO_RUNTIME_OPTIONS $CRIO_STORAGE_OPTIONS $CRIO_NETWORK_OPTIONS $CRIO_METRICS_OPTIONS (code=exited, status=1/FAILURE)
 Main PID: 1259 (code=exited, status=1/FAILURE)

Dec 11 05:42:49 ip-172-31-89-94.ec2.internal systemd[1]: Starting Container Runtime Interface for OCI (CRI-O)...
Dec 11 05:42:49 ip-172-31-89-94.ec2.internal crio[1259]: time="2020-12-11 05:42:49.409813214-05:00" level=fatal msg="Validating root config: failed to get store to set defaults: kernel does not support overlay fs: overlay: the backing xfs filesystem is formatted without d_type support, which leads to incorrect behavior. Reformat the filesystem with ftype=1 to enable d_type support. Running without d_type is not supported.: driver not supported"
Dec 11 05:42:49 ip-172-31-89-94.ec2.internal systemd[1]: crio.service: main process exited, code=exited, status=1/FAILURE
Dec 11 05:42:49 ip-172-31-89-94.ec2.internal systemd[1]: Failed to start Container Runtime Interface for OCI (CRI-O).
Dec 11 05:42:49 ip-172-31-89-94.ec2.internal systemd[1]: Unit crio.service entered failed state.
Dec 11 05:42:49 ip-172-31-89-94.ec2.internal systemd[1]: crio.service failed.
```
#### Root Cause:

The default container storage i.e. /var/lib/containers/storage is mounted with ftype=0, so d_type is disabled ( d_type enables mapping of filename to its inode ). For overlay, **ftype=1 is a requirement !!**
```
$ sudo xfs_info /var/lib/containers/storage | grep ftype

naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
```

#### Solution:

- Create a Logical Volume (LV).
```
$ sudo fdisk <disk_name>
```

```
$ sudo partprobe <disk_name>
```
```
$ sudo vgcreate vgname <partition>
```

```
$ sudo lvcreate -l 100%FREE -n lvname vgname
```

```
$ sudo lvs

LV     VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
root   centos -wi-ao----  <6.67g
swap   centos -wi-ao---- 820.00m
lvname vgname -wi-a-----  <5.00g
```
- Format LV with XFS filesystem 
```
$ sudo mkfs.xfs /dev/mapper/vgname-lvname

meta-data=/dev/mapper/vgname-lvname isize=512    agcount=4, agsize=327424 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=1309696, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
```
- Now ftype is set to 1.
- Mount LV on /var/lib/containers/storage permanently.
```
$ sudo cat /etc/fstab | grep /var/lib/containers/storage

/dev/mapper/vgname-lvname /var/lib/containers/storage xfs defaults 0 0
```

```
$ sudo mount -a
```

```
$ sudo df -hT | grep /var/lib/containers/storage

/dev/mapper/vgname-lvname xfs       5.0G   33M  5.0G   1% /var/lib/containers/storage
```
- Change runroot path and un-hash both runroot & root in /etc/crio/crio.conf.
```
$ sudo cat /etc/crio/crio.conf | grep /var/lib/containers/storage

root = "/var/lib/containers/storage"
runroot = "/var/lib/containers/storage"
```
- Then start and enable the cri-o service.
```
$ sudo systemctl start cri-o
```
```
$ sudo systemctl enable cri-o
```

### Install and configure prerequisites for Kubernetes: (should be done on all nodes)

- Load overlay module.
```
$ sudo modprobe overlay
```
- Load br_netfilter module.
```
$ sudo modprobe br_netfilter
```
This module is enabled so that iptables, ip6tables and arptables can filter bridged IPv4/IPv6/ARP packets and makes the firewall transparent.

- Set up required sysctl params, these persist across reboots.
```
$ cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
```
- Apply sysctl params without reboot.
```
$ sudo sysctl --system
```
- Disabling firewall. 
```
$ sudo systemctl stop firewalld && sudo systemctl disable firewalld
```
- Set SELinux in permissive mode (effectively disabling it).
```
$ sudo setenforce 0
```
```
$ sudo sed -i -e s/SELINUX=enforcing/SELINUX=permissive/g /etc/sysconfig/selinux
```
- Switching off swap.
```
$ sudo swapoff -a
```
- Also disable the swap in /etc/fstab (comment swap entry).
```
$ sudo cat /etc/fstab | grep swap

#/dev/mapper/centos-swap                                swap                    swap    defaults        0 0
```
- Change cgroup_manager of crio in /etc/crio/crio.conf.
```
$ sudo cat /etc/crio/crio.conf | grep cgroup_manager

cgroup_manager = "cgroupfs"
```
- Configure Kubernetes repo.
```
$ sudo cat /etc/yum.repos.d/kube.repo

[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
```

### Deploying Kubernetes Cluster
#### COMMANDS ON MASTER

- Install kubeadm package.
```
$ sudo yum install kubeadm -y
```
- Start and enable kubelet.
```
$ sudo systemctl start kubelet && sudo systemctl enable kubelet
```
- Now initialize the Kubernetes control-plane and master.
```
$ sudo kubeadm init
```
> Note: If giving error of iptables does not exist, just load br_netfilter module again and run the command.  

- Output 
```
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.46.157:6443 --token 7xlnp0.9uv4z0qr4wvzhtqn \
    --discovery-token-ca-cert-hash sha256:4a1a412d2e682556df0bf10dc380c744a98eb99e8c927fa58eb025d5ff7dc694
```
- To make kubectl work for your non-root user, run these commands, which are also part of the kubeadm init output:
```
$ mkdir -p $HOME/.kube
```
```
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
```
```
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
```
- These commands store your Kubernetes configuration file in home directory.
- Make a record of the kubeadm join command that kubeadm init outputs. You need this command to join nodes to your cluster.
- We need to install CNI plugin. So that pods can communicate with each other. 
- There are different CNI plugins for different purpose. Here we are using **Weave Net** CNI plugin.
```
$ kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
```   
Weave Net creates a virtual network that connects containers across multiple hosts and enables their automatic discovery.
- Once the CNI plugin has been installed, you can confirm that it is working by checking that the CoreDNS Pod is up and running.
```
$ kubectl get pods -n kube-system
```
- Our master is ready !!
```
$ kubectl get nodes

NAME     STATUS   ROLES                  AGE   VERSION
master   Ready    control-plane,master   14m   v1.20.0
```
- Now we need to add worker nodes to the cluster. 

#### COMMANDS ON WORKER NODES

- Install kubeadm package.
```
$ sudo yum install kubeadm -y
``` 
- Start and enable kubelet.
```
$ sudo systemctl start kubelet && sudo systemctl enable kubelet
```
- Run the kubeadm join command. (command in the output of kubeadm init)
```
$ sudo kubeadm join 192.168.46.157:6443 --token 7xlnp0.9uv4z0qr4wvzhtqn \
    --discovery-token-ca-cert-hash sha256:4a1a412d2e682556df0bf10dc380c744a98eb99e8c927fa58eb025d5ff7dc694
``` 
- Output
```
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
```
- Now our worker nodes have joined the cluster !!

#### COMMAND ON MASTER

- Check whether nodes are ready.
```
$ kubectl get nodes

NAME      STATUS   ROLES                  AGE     VERSION
master    Ready    control-plane,master   15m     v1.20.0
worker1   Ready    <none>                 4m44s   v1.20.0
worker2   Ready    <none>                 4m18s   v1.20.0
```


### HURRAH !! Kubernetes cluster is ready

**THANK YOU!!**