Kubernetes Installation Tutorial: Kubespray
Tags: kubernetes, kubespray, ansible
Published: 2022-12-15 07:36:08

The Kubespray distribution brings the power of Ansible for configuration, setup, and maintenance of a Kubernetes cluster. Starting from an inventory file, you define which nodes are part of the cluster and which role they should play. Then, additional configuration files fine-tune the settings of the various Kubernetes components. By applying the playbook - Ansible jargon for the installation/setup scripts that consume your configuration - the desired state is manifested on the target server. Using Kubespray means to manifest your cluster as true Infrastructure as code: All subsequent runs lead to the very same desired state.

This article is a hands-on tutorial to install a 3 node Kubernetes cluster with a single controller node. We start with the prerequisite setup of the nodes, then install the control plane, and add worker nodes. Finally, we will also see how to upgrade the cluster.

_The original article appeared at my blog [admantium.com](https://admantium.com/blog/kube12_k8s_with_kubespray_tutorial/)_.

## Prerequisites

To follow along, you need this:

- 3 server with a [compatible OS](https://kubespray.io/#/?id=supported-linux-distributions)
- Regarding hardware resources, controller nodes need at least 1.5 GB RAM, worker nodes 1GB of RAM
- SSH access to these servers

Just as before, I will do the installation on my preferred cloud environment at Hetzner. The hardware configuration is this:

- 1x CX11 node (1 Intel CPU, 2GB RAM, 40GB SSD)
- 2x CX21 nodes (2 Intel CPU, 4GB RAM, 40GB SSD)

## Configure Kubespray

### 1. Kubespray Controller Setup

The Kubespray controller is a machine on which you will install the Kubespray environment and from which you will have access to all the servers which are going to be a part of your cluster. Preferably, this controller is not also part of your cluster.

On this controller, following the [official documentation](https://github.com/kubernetes-sigs/kubespray/blob/controller/docs/ansible.md#installing-ansible), we will clone the Kubespray repo and install ansible first in a Python virtual environment - a separated workspace, similar to a container, that encapsulates all required software versions into one.

Run the following commands:

```bash
git clone --depth=1 https://github.com/kubernetes-sigs/kubespray.git

KUBESPRAYDIR=$(pwd)/kubespray
VENVDIR="$KUBESPRAYDIR/.venv"
ANSIBLE_VERSION=2.12

virtualenv  --python=$(which python3) $VENVDIR

source $VENVDIR/bin/activate
cd $KUBESPRAYDIR

pip install -U -r requirements-$ANSIBLE_VERSION.txt
```

You will see a long list of downloaded and installed packages:

```bash
Collecting ansible==5.7.1
  Downloading ansible-5.7.1.tar.gz (35.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 35.7/35.7 MB 8.6 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting ansible-core==2.12.5
  Downloading ansible-core-2.12.5.tar.gz (7.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.8/7.8 MB 9.6 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting cryptography==3.4.8
  Downloading cryptography-3.4.8-cp36-abi3-macosx_10_10_x86_64.whl (2.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 9.7 MB/s eta 0:00:00
...
Successfully built ansible ansible-core
Installing collected packages: ruamel.yaml, resolvelib, pbr, netaddr, jmespath, ruamel.yaml.clib, PyYAML, pyparsing, pycparser, MarkupSafe, packaging, jinja2, cffi, cryptography, ansible-core, ansible
```

When you see the success message, we can continue to define the inventory.

### 2. Creating the Inventory

The very first step when using Kubespray is to define the inventory - a core concept of Ansible, which is a list of servers, with hostnames or IP address, and the role they have on your setup.

First, copy the sample inventory definitions from the repo...

```bash
cp -rfp inventory/sample inventory/mycluster
```

... then bootstrap the inventory file details by running a command that lists all the IPs of your server (from controller to workers):

```python
declare -a IPS=(167.235.49.54 159.69.35.153 167.235.73.16)
CONFIG_FILE=inventory/mycluster/hosts.yml python3 contrib/inventory_builder/inventory.py ${IPS[@]}
```

This command will name the nodes linearly as `node1`, `node2`, and `node3`. For a one controller node setup, edit the created file `inventory/mycluster/hosts.yml` file, change the name of the first node to `controller`, rename other nodes, and only use `controller` in the kube_control_plane group.

My final inventory file than looked like this:

```yaml
all:
  hosts:
    controller:
      ansible_host: 167.235.49.54
      ip: 167.235.49.54
      access_ip: 167.235.49.54
    node1:
      ansible_host: 159.69.35.153
      ip: 159.69.35.153
      access_ip: 159.69.35.153
    node2:
      ansible_host: 167.235.73.16
      ip: 167.235.73.16
      access_ip: 167.235.73.16
  children:
    kube_control_plane:
      hosts:
        controller:
    kube_node:
      hosts:
        controller:
        node1:
        node2:
    etcd:
      hosts:
        controller:
        node1:
        node2:
    k8s_cluster:
      children:
        kube_control_plane:
        kube_node:
    calico_rr:
      hosts: {}
```

This basic inventory file has a lot more options: You can also use it to manage an etcd server, and much more. Check the [official documentation](https://kubespray.io/#/docs/getting-started?id=building-your-own-inventory) for more information.

### 3. Test Run

Lets run the installation script in check mode, which will only print the changes, not apply them directly. Use the `-u user` flag to define the name of the user that is used for the SSH connection, and then this user needs to become a root user.

```bash
ansible-playbook \
   -i inventory/mycluster/hosts.yml  \
   --private-key=~/.ssh/id_rsa_k8s_kubeadm.key  \
   --check  \
   -u devcon  \
   --become \
   cluster.yml
```

If you see any error here, there are most luckily related to running in check mode - ignore them for now. At the end of the command, you should see a summary of the tasks:

```bash
Wednesday 28 September 2022  20:45:20 +0200 (0:00:01.114)       0:00:53.828 ***
===============================================================================
kubernetes/preinstall : Create kubernetes directories ------------------------------------------- 5.06s
kubernetes/preinstall : Remove swapfile from /etc/fstab ----------------------------------------- 2.24s
bootstrap-os : Install dbus for the hostname module --------------------------------------------- 2.01s
Gather necessary facts (hardware) --------------------------------------------------------------- 1.90s
bootstrap-os : Assign inventory name to unconfigured hostnames (non-CoreOS, non-Flatcar, Suse and ClearLinux, non-Fedora) --- 1.88s
Gather necessary facts (network) ---------------------------------------------------------------- 1.82s
bootstrap-os : Gather host facts to get ansible_os_family --------------------------------------- 1.81s
Gather minimal facts ---------------------------------------------------------------------------- 1.42s
adduser : User | Create User -------------------------------------------------------------------- 1.33s
kubernetes/preinstall : Ensure ping package ----------------------------------------------------- 1.33s
bootstrap-os : Create remote_tmp for it is used by another module ------------------------------- 1.33s
bootstrap-os : Ensure bash_completion.d folder exists ------------------------------------------- 1.32s
kubernetes/preinstall : set is_fedora_coreos ---------------------------------------------------- 1.29s
kubernetes/preinstall : Create other directories ------------------------------------------------ 1.29s
kubernetes/preinstall : check if booted with ostree --------------------------------------------- 1.23s
adduser : User | Create User Group -------------------------------------------------------------- 1.23s
kubernetes/preinstall : check swap -------------------------------------------------------------- 1.22s
kubernetes/preinstall : Check if kubernetes kubeadm compat cert dir exists ---------------------- 1.18s
kubernetes/preinstall : get content of /etc/resolv.conf ----------------------------------------- 1.16s
kubernetes/preinstall : check status of /etc/resolv.conf ---------------------------------------- 1.16s
```

### 4. Optionally: Kubernetes Component Configuration

In my installation attempt, I did not change any configuration. Ansible playbooks can be configured through the configuration files `inventory/mycluster/group_vars/all/all.yml` and `inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml`.

## Cluster Installation and Troubleshooting

The cluster installation is started by executing the ansible playbook `cluster.yaml` and applying required command line flags. It looks like this:

```yaml
ansible-playbook \
   -i inventory/mycluster/hosts.yml  \
   --private-key=~/.ssh/id_rsa_k8s_kubeadm.key  \
   -u root \
   --become \
   cluster.yml
```

The installation was tricky - I needed several attempts to get it working. Basically, I proceeded by running the command with the `-vvv` flag, and then check the Ansible tasks and configuration.

The most persistent error was about the Calico CNI. The error message looks like this:

```bash
fatal: [controller]: FAILED! => {"changed": true, "cmd": "/usr/local/bin/calicoctl.sh replace -f-", "delta": "0:00:00.054945", "end": "2022-09-29 19:26:13.445232", "msg": "non-zero return code", "rc": 1, "start": "2022-09-29 19:26:13.390287", "stderr": "Hit error: error with field RouteReflectorClusterID = '1' (Reason: failed to validate Field: RouteReflectorClusterID because of Tag: ipv4 )", "stderr_lines": ["Hit error: error with field RouteReflectorClusterID = '1' (Reason: failed to validate Field: RouteReflectorClusterID because of Tag: ipv4 )"], "stdout": "Partial success: replaced the first 1 out of 1 'Node' resources:", "stdout_lines": ["Partial success: replaced the first 1 out of 1 'Node' resources:"]}
```

The CNI Calico could not be installed. After several attempts, I needed to define two variables in `kubespray/extra_playbooks/inventory/mycluster/group_vars/all/all.yml`:

```yaml
cluster_id: 1
route_reflector_cluster_id: 2
```

And, I also needed to change a task definition in `roles/network_plugin/calico/rr/tasks/update-node.yml` to use the `route_reflector_cluster_id`:

```yaml
  - name: Calico-rr | Set route reflector cluster ID
    set_fact:
      calico_rr_node_patched: >-
        {{ calico_rr_node.stdout | from_json | combine({ 'spec': { 'bgp':
        { 'routeReflectorClusterID': route_reflector_cluster_id }}}, recursive=True) }}
```

Now the installation was successful.

```bash
PLAY RECAP ***********************************************************************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
controller                 : ok=709  changed=33   unreachable=0    failed=0    skipped=1243 rescued=0    ignored=4
node2                      : ok=507  changed=6    unreachable=0    failed=0    skipped=770  rescued=0    ignored=2
node3                      : ok=507  changed=6    unreachable=0    failed=0    skipped=770  rescued=0    ignored=2
```

Let's see how to access the cluster.

## Access your Kubernetes Cluster (and Troubleshooting)

To work with your cluster, you need the `kubectl` binary, ideally with the very same version, and the `kubeconfig` file. A Kubespray installation puts these on the controller nodes. So, you can perform a SSH login to your cluster and work from there, or use your kubespray controller on which you manually install the `kubectl` binary.

Let’s see some information about the cluster and its nodes. To my surprise, several errors occurred: the worker nodes were not ready, the coredns deployment could not schedule a pod, and the calico pods were crashing.

```bash
$> kubectlubectl get nodes
NAME      STATUS     ROLES           AGE   VERSION
controller    Ready      control-plane   37h   v1.24.6
worker1   NotReady   <none>          37h   v1.24.6
worker2   NotReady   <none>          37h   v1.24.6

$> kubectlubectl get deploy -A
NAMESPACE     NAME             READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   coredns          1/2     2            1           22h
kube-system   dns-autoscaler   1/1     1            1           22h
```

Again, I needed some troubleshooting attempts, and for the same reasons, I will also document them.

### Error: Worker Nodes not ready

Looking at the Kubernetes log files, I could see this:

```bash
Oct 01 08:44:53 worker1 kubelet[1176]: I1001 08:44:53.877700    1176 reconciler.go:254] "operationExecutor.MountVolume started for volume \"etc-nginx\" (UniqueName: \"kubernetes.io/host-path/9e50e3cca2a34ce5d8f21c4508ba5796-etc-nginx\") pod \"nginx-proxy-node2\" (UID: \"9e50e3cca2a34ce5d8f21c4508ba5796\") " pod="kube-system/nginx-proxy-node2"
Oct 01 08:44:53 worker1 kubelet[1176]: I1001 08:44:53.877928    1176 operation_generator.go:703] "MountVolume.SetUp succeeded for volume \"etc-nginx\" (UniqueName: \"kubernetes.io/host-path/9e50e3cca2a34ce5d8f21c4508ba5796-etc-nginx\") pod \"nginx-proxy-node2\" (UID: \"9e50e3cca2a34ce5d8f21c4508ba5796\") " pod="kube-system/nginx-proxy-node2"
Oct 01 08:44:53 worker1 kubelet[1176]: E1001 08:44:53.965399    1176 kubelet.go:2424] "Error getting node" err="node \"node2\" not found"
```

I needed to remove and add the nodes again, following the [documentation](https://github.com/kubernetes-sigs/kubespray/blob/controller/docs/nodes.md):

- run the remove-node playbook, passing the node names
- on the controller node, run `kubectl delete node`
- confirm that `kubectl get nodes` only shows the controller node
- update `hosts.yaml` with the new and correct hostname
- run the `cluster.yml` playbook again

After this, the above error message vanished.

### Erro: CoreDNS Pod not scheduled

Let’s check why a coredns pod cannot be scheduled. Looking at the pods, the nominated node for coredns is `<none>` - thats strange.

```bash
NAME                              READY   STATUS                  RESTARTS       AGE   IP              NODE      NOMINATED NODE   READINESS GATES
calico-node-7g8dh                 0/1     Init:CrashLoopBackOff   10 (22s ago)   27m   159.69.35.153   worker1   <none>           <none>
calico-node-bcwhh                 0/1     Init:CrashLoopBackOff   14 (47s ago)   27m   167.235.73.16   worker2   <none>           <none>
calico-node-mprfp                 1/1     Running                 1 (120m ago)   40h   167.235.49.54   controller    <none>           <none>
coredns-74d6c5659f-c7ddd          0/1     Pending                 0              25h   <none>          <none>    <none>           <none>
coredns-74d6c5659f-wjl7q          1/1     Running                 1 (120m ago)   25h   10.233.97.132   controller    <none>           <none>
```

I scaled the deployment to 1 replica, then it worked.

### Error: Caclico Pods keep crashing

Finally, let’s check why the calico Pods keep crashing, peeking into the log files:

a. calico-node consists of three init containers and a main container

```bash
$> kubectl logs calico-node-7g8dh
Defaulted container "calico-node" out of: calico-node, upgrade-ipam (init), install-cni (init), flexvol-driver (init)
Error from server (BadRequest): container "calico-node" in pod "calico-node-7g8dh" is waiting to start: PodInitializing
```

b. check the main container - its waiting to be created

```bash
$> kubectl logs calico-node-7g8dh -c calico-node
Error from server (BadRequest): container "calico-node" in pod "calico-node-7g8dh" is waiting to start: PodInitializing
```

c. check the upgrade-ipam init container - it executes correctly

```bash
$> kubectl logs calico-node-7g8dh -c upgrade-ipam
2022-10-01 10:53:43.347 [INFO][1] ipam_plugin.go 75: migrating from host-local to calico-ipam...
2022-10-01 10:53:43.349 [INFO][1] migrate.go 67: checking host-local IPAM data dir dir existence...
2022-10-01 10:53:43.349 [INFO][1] migrate.go 69: host-local IPAM data dir dir not found; no migration necessary, successfully exiting...
2022-10-01 10:53:43.349 [INFO][1] ipam_plugin.go 105: migration from host-local to calico-ipam complete node="worker1"
```

d. check the install-cni init container - it complains about a configuration error

```bash
$> kubectl logs calico-node-7g8dh -c install-cni
time="2022-10-01T11:27:59Z" level=info msg="Running as a Kubernetes pod" source="install.go:140"
2022-10-01 11:27:59.951 [INFO][1] cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/bandwidth"
#...
2022-10-01 11:28:00.215 [INFO][1] cni-installer/<nil> <nil>: Wrote Calico CNI binaries to /host/opt/cni/bin

2022-10-01 11:28:00.298 [INFO][1] cni-installer/<nil> <nil>: CNI plugin version: v3.23.3

2022-10-01 11:28:00.299 [INFO][1] cni-installer/<nil> <nil>: /host/secondary-bin-dir is not writeable, skipping
W1001 11:28:00.299160       1 client_config.go:617] Neither --kubeconfig nor --controller was specified.  Using the inClusterConfig.  This might not work.
time="2022-10-01T11:28:00Z" level=info msg="Using CNI config template from CNI_NETWORK_CONFIG_FILE" source="install.go:335"
time="2022-10-01T11:28:00Z" level=fatal msg="open /host/etc/cni/net.d/calico.conflist.template: no such file or directory" source="install.go:339"
```

e. to be sure, check the last init container flexvol-driver: Its als in pending state

```bash
$> kubectl logs calico-node-7g8dh -c flexvol-driver
Error from server (BadRequest): container "flexvol-driver" in pod "calico-node-7g8dh" is waiting to start: PodInitializing
```

Ok, the init container of the calico pod does not start because of a misconfiguration. Let’s see how this pod is started.

```yaml
install-cni:
    Image:      quay.io/calico/cni:v3.23.3
    Port:       <none>
    Host Port:  <none>
    Command:
      /opt/cni/bin/install
    Environment Variables from:
      kubernetes-services-endpoint  ConfigMap  Optional: true
    Environment:
      CNI_CONF_NAME:            10-calico.conflist
      UPDATE_CNI_BINARIES:      true
      CNI_NETWORK_CONFIG_FILE:  /host/etc/cni/net.d/calico.conflist.template
      SLEEP:                    false
      KUBERNETES_NODE_NAME:      (v1:spec.nodeName)
    Mounts:
      /host/etc/cni/net.d from cni-net-dir (rw)
      /host/opt/cni/bin from cni-bin-dir (rw)
   flexvol-driver:
..
Volumes:
   cni-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/cni/net.d
    HostPathType:
   cni-bin-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/cni/bin
    HostPathType:
```

So, on the worker nodes, there is no file at `/etc/cni/net.d/calico.conflist.template` - but it’s on the controller. Let’s just copy it to the controller nodes, and then the Calico pods could be started.

```bash
calico-node-7g8dh                 1/1     Running   0              48m   159.69.35.153   worker1   <none>           <none>
calico-node-bcwhh                 1/1     Running   0              48m   167.235.73.16   worker2   <none>           <none>
calico-node-mprfp                 1/1     Running   1 (141m ago)   40h   167.235.49.54   controller    <none>           <none>
```

### The Cluster works

And with this, the cluster is finally ready.

```bash
NAME      STATUS   ROLES           AGE   VERSION
controller    Ready    control-plane   40h   v1.24.6
worker1   Ready    <none>          48m   v1.24.6
worker2   Ready    <none>          48m   v1.24.6
```

This was another extensive troubleshooting lesson, but I'm happy that it finally worked.

## Upgrade the Kubernetes Version

Upgrading the cluster can be done on two ways. The unsafe method basically installs new versions of the Kubernetes component binaries on the server and restarts them. This can disrupt your cluster when e.g. a misconfigured component does not start anymore. The other way is to upgrade nodes step-by-step, and to include a node drain operation to move workloads to other nodes. In both cases, a manual step that you need to do is to update deprecated API versions as before.

Let’s upgrade the nodes the safe way, and start with the controller node. We will pass the parameter `upgrade_node_post_upgrade_confirm` so that the node will be cordoned before the upgrade according to this [task definition](https://github.com/kubernetes-sigs/kubespray/blob/controller/roles/upgrade/post-upgrade/tasks/main.yml).

Run the following commands:

```bash
ansible-playbook \
   -i inventory/mycluster/hosts.yml \
   --private-key=~/.ssh/id_rsa_k8s_kubeadm.key \
   -u root  \
   --become \
   --force \
   -v  \
   -e kube_version=v1.25.2 \
  --extra-vars "upgrade_node_post_upgrade_confirm=true" \
  --limit "kube_control_plane" \
   upgrade-cluster.yml
```

However, as before, I encountered errors.

```bash
TASK [download : prep_kubeadm_images | Check kubeadm version matches kubernetes version] ************************************
fatal: [controller]: FAILED! => {"msg": "The conditional check 'not kubeadm_version == downloads.kubeadm.version' failed. The error was: error while evaluating conditional (not kubeadm_version == downloads.kubeadm.version):

> Check kubeadm version matches kubernetes version
```

For this error, I could not find a workaround, all attempts of re-cloning the repo, selectively disabling check conditions in the Ansible task definitions, checking passed vars failed. I got the feeling of debugging Ansible, not managing a Kubernetes cluster.

So, as a last resort, I tried the unsafe upgrade version with this command:

```bash
ansible-playbook \
   -i inventory/mycluster/hosts.yml \
   --private-key=~/.ssh/id_rsa_k8s_kubeadm.key \
   -u root  \
   --become \
   --force \
   -v  \
   -e kube_version=v1.25.2 \
   -e upgrade_cluster_setup=true \
  --limit "kube_control_plane" \
   cluster.yml
```

This also failed with this error:

```bash
TASK [container-engine/runc : download_file | Starting download of file] ****************************************************
fatal: [controller]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: {{ download_defaults | combine(downloads.runc) }}
```

I could find a related open [Github issue](https://github.com/kubernetes-sigs/kubespray/issues/9206). At this point, I stopped to try an upgrade, and concluded using Kubespray.

## Conclusion

The Kubespray distribution offers complete configuration of a Kubernetes cluster with the power of Ansible. Starting from the server, continuing with the Kubernetes components, and ending at their individual configuration: Everything can be controlled as versioned infrastructure as code. This article showed you how to use Kubespray to set up a one controller, two worker nodes cluster. You learned about the Ansible base installation, the necessary steps so define the inventory, and finally how to install the cluster. However, my experience with Kubespray was tainted by extensive troubleshooting: First, to get the installation complete, I needed to set specific options in the Ansible configuration. Second, in the cluster, the worker nodes were not ready. Drilling into the Kubernetes log files and pod logs, I could trace one error to a missing configuration file that was present on the controller, but not in the worker. And third, I could not perform an update of the Kubernetes version.