Remote development with multi-node MicroK8s cluster and Skaffold
Tags: kubernetes, devops, skaffold, microk8s
Published: 2020-04-15 06:13:40

As part of an effort to bring parity between environments, my team recently switched to using [Kubernetes] for development as well as production orchestration with the help of [Skaffold].

While this reduces maintenance overhead and is a great idea on the whole, it also means that the application started outgrowing my laptop rather quickly. Luckily, we have an enterprise virtual machine provisioning environment where I decided to offload my development environment.

Now, it would've been easier to set up a single-node cluster with a large enough VM, but how basic would that have been? So, I went with a multi-node cluster. If you'd like to do the same, read on!

# Prerequisites

- A set of Ubuntu virtual machines. I used four machines with 16 GB RAM and 4 CPU cores each.

- Pick one of these instances as the primary node and the rest will be considered secondary. Let's assume that the primary node has the IP address `10.0.0.1`.

- A shared disk between the nodes so persistent volumes can be shared across them. For the following, we'll assume there's a reasonably sized disk mounted at `/data` on all nodes.

## Ports to expose

The primary node should have the following ports exposed:

- `25000` (so secondary nodes can join the cluster)

- `32000` (so the container registry can be accessed by secondary nodes)

- `16443` (if you want to use `kubectl` from a remote machine)

- Others as needed

## Software to install

Follow standard steps to install the following services/applications:

- [Docker] &mdash; on all nodes

- [Skaffold] &mdash; only on primary node

My VMs started off with Ubuntu Server 16.04, which I then upgraded to 18.04. This is not necessary, but important to note in case it changes some of the steps below.

# Install MicroK8s

Let's start by [installing MicroK8s](https://microk8s.io/#get-started) on all nodes:

``` bash
sudo snap install microk8s --classic --channel=1.18/stable
# check status
sudo microk8s status --wait-ready
```

No need to enable any addons for now as we'll be doing this in later steps.

**Warning:** You might run into an issue with [NFS and snap not playing nicely](https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1662552). I was only able to run MicroK8s commands as root.

## kubectl

MicroK8s comes with its own namespaced `kubectl` that can be invoked with `microk8s kubectl`. If you're used to working directly with `kubectl`, this might start to get tedious. There are a few ways to get around this:

- Connect existing `kubectl` to your MicroK8s instance by running

    ``` bash
    sudo microk8s kubectl config view --raw > $HOME/.kube/config
    ```

    The same file can also be used to access the cluster from a remote machine as long as appropriate ports are accessible. See [documentation](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig) for more information. I should point out that this is the option that I have tested, but the next two might also work.

- Use a good old bash alias: `alias kubectl='microk8s kubectl'`

- Use a snap alias: `sudo snap alias microk8s.kubectl kubectl`

# Install and configure MicroK8s addons

First, enable some basic [MicroK8s addons] that we're going to need:

``` bash
sudo microk8s enable dns ingress storage
```

If you need [Helm] support, be sure to add `helm` to the list above.

## Configure storage location

By default, the storage addon persists all volumes in `/var/snap/microk8s/common/default-storage`. Since we're going to be sharing storage across various nodes, we need to update this to write to our `/data` mounted disk instead. You can do this by editing the `hostpath-provisioner` deployment:

``` bash
sudo microk8s kubectl get -o yaml -n kube-system deploy hostpath-provisioner | \
  sed 's~/var/snap/microk8s/common/default-storage~/data/snap/microk8s/common/default-storage~g' | \
  sudo microk8s kubectl apply -f -

# restart microk8s for good measure
sudo microk8s stop && sudo microk8s start
```

## Enable internal registry

When you have a multi-node cluster, the easiest way to share development images is to push them to a private registry. Skaffold knows how to push and pull from private registries, when needed, but we'll need to set one up. Luckily, MicroK8s makes it relatively easy:

``` bash
sudo microk8s enable registry
```

This will start a registry on port `32000` that can be accessed by other nodes in the cluster via `10.0.0.1:32000`.

### Working with an insecure registry

Without additional configuration, the registry started in the step above is insecure. If you're not comfortable with that, you could look into securing it. For the purposes of this tutorial, we will continue to use it as is, which still requires some &ndash; though less involved &ndash; changes.

First, we need to make sure that Docker won't have any trouble pushing to this registry just because it is insecure. This can be done by [editing the Docker daemon configuration](https://docs.docker.com/registry/insecure/) at `/etc/docker/daemon.json` to add the following lines:

``` json
{
  "insecure-registries" : ["10.0.0.1:32000"]
}
```

Keep in mind that this file might need to be created if it's not already present. Once this is taken care of, restart the daemon for changes to take effect:

``` bash
sudo systemctl restart docker
```

Second, MicroK8s needs to be persuaded not to complain when pulling from this insecure registry. For this, find the file `/var/snap/microk8s/current/args/containerd-template.toml` and under `[plugins] -> [plugins.cri.registry] -> [plugins.cri.registry.mirrors]`, add:

``` toml
        [plugins.cri.registry.mirrors."10.0.0.1:32000"]
          endpoint = ["http://10.0.0.1:32000"]
```

Restart MicroK8s:

``` bash
sudo microk8s stop && sudo microk8s start
```

This needs to be done on all the nodes. See [official instructions](https://microk8s.io/docs/registry-private) for more details.

# Form the cluster

Finally, we're ready to form a cluster. Run the following on the master node:

``` bash
microk8s add-node
```

You'll get back something similar to:

``` text
Join node with: microk8s join 10.0.0.1:25000/<some-token>
```

Copy the `microk8s join...` command and run it on one of the secondary nodes. A new token will need to be generated for each secondary node that you wish to add to the cluster.

Now you can run `kubectl get nodes` on the primary and see that all nodes have joined. That's it, you now have your own fully functioning Kubernetes cluster!

# Use Skaffold for building and deployment

This part of the tutorial assumes some knowledge of [Skaffold]. If you aren't familiar with it, it's a very useful tool and I'd highly recommend checking it out. In my case, we're using Skaffold to simplify building Docker images and deploying our [Helm] charts during development.

With the above setup in place, we need to make some minor adjustments to our `skaffold.yaml` files in order to make them work with the multi-node cluster and private registry. I will also assume that your code has already been cloned to a location on your primary node.

First, list your private registry under `build.insecureRegistries`:

``` yaml
build:
  insecureRegistries: ["10.0.0.1:32000"]
```

Second, prefix image names for all your artifacts under `build` with your registry address like so:

``` yaml
build:
  artifacts:
  - image: 10.0.0.1:32000/my-image
```

The same goes for where you're using the image. This might vary depending on your own setup. As an example, if you're using Helm, you might end up with something similar to:

``` yaml
deploy:
  helm:
    releases:
    - values:
        imageName: 10.0.0.1:32000/my-image
```

Now you can build and deploy with `skaffold dev` or `skaffold run` as usual. Development will never be the same again!

# Troubleshooting

Along the way, I encountered some issues that might be peculiar to my setup, but are worth mentioning.

## Getting `Forbidden: disallowed by cluster policy` error

I ran into [this issue](https://github.com/ubuntu/microk8s/issues/749) when trying to install the Elasticsearch chart. The workaround for this was to do the following on the primary node:

``` bash
echo "--allow-privileged" | sudo tee -a /var/snap/microk8s/current/args/kube-apiserver
# restart microk8s for changes to take effect
sudo microk8s stop && sudo microk8s start
```

## Unable to connect to the internet from pods

By default, the internal DNS server points to Google's DNS servers. If, for whatever reason, this doesn't work for you, you would need to [update the CoreDNS configuration](https://microk8s.io/docs/addon-dns) to allow your pods to access the internet.

``` bash
kubectl edit -n kube-system configmaps coredns
# edit the configmap by replacing the line that starts with "forward" under data.Corefile with "forward . /etc/resolv.conf"
# if this doesn't work, you might try manually replacing Google's 8.8.8.8 and 8.8.4.4 servers with your own DNS servers
# save and close, the configuration should get reloaded
```

# Next steps

Here are some improvements to this system you could explore that I have not covered here:

- If [VS Code] is your IDE of choice, [remote SSH development](https://code.visualstudio.com/docs/remote/ssh) makes interacting with your remote code a breeze!

- Configure Helm and kubectl clients on your local machines to connect to your remote cluster by exporting the kubeconfig file described [earlier](#kubectl). This would allow you to store your code and interact with the cluster without leaving the comforts of your local machine.

- You might even try offloading your builds to the remote Docker host instead of building locally

Hope you have fun with it!

[Kubernetes]: https://kubernetes.io
[Docker]: http://www.docker.com
[Helm]: http://helm.sh
[Skaffold]: http://skaffold.dev
[MicroK8s addons]: https://microk8s.io/docs/addons
[VS Code]: https://code.visualstudio.com