Hacking in kind (Kubernetes in Docker)
Tags: kubernetes, kind
Published: 2023-11-17 19:17:15

---
title: Hacking in kind (Kubernetes in Docker)
published: true
description: I figured out how to add nodes dynamically to a cluster running in kind
tags: kubernetes,kind
# cover_image: https://direct_url_to_image.jpg
# Use a ratio of 100:42 for best results.
# published_at: 2023-11-17 19:14 +0000
---

> Originally published [on my blog](https://sklar.rocks/kind-hacking/)

# How to dynamically add nodes to a kind cluster

[Kind](https://kind.sigs.k8s.io/) allows you to run a Kubernetes cluster inside Docker.  This is incredibly useful for developing Helm charts, Operators, or even just testing out different k8s features in a safe way.

I've recently been working on an operator (built using the [operator-sdk](https://github.com/operator-framework/operator-sdk)) that manages cluster node lifecycles.  Kind allows you to [spin up clusters with multiple nodes](https://kind.sigs.k8s.io/docs/user/configuration/#nodes), using a Docker container per-node and joining them using a common Docker network.  However, the `kind` executable does not allow you to modify an existing cluster by adding or removing a node.

I wanted to see if this was possible using a simple shell script, and it turns out that it's actually not too difficult!

## Creating the node

Using my favorite diff tool, [DiffMerge](https://sourcegear.com/diffmerge/), and `docker inspect` to compare an existing kind node's state to a new container's, I experimented with various `docker run` flags until I got something that's close enough to the kind node.

```bash
docker run \
--restart on-failure \
-v /lib/modules:/lib/modules:ro \
--privileged \
-h $NODE_NAME \
-d \
--network kind \
--network-alias $NODE_NAME \
--tmpfs /run \
--tmpfs /tmp \
--security-opt seccomp=unconfined \
--security-opt apparmor=unconfined \
--security-opt label=disable \
-v /var \
--name $NODE_NAME \
--label io.x-k8s.kind.cluster=kind \
--label io.x-k8s.kind.role=worker \
--env KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER \
kindest/node:v1.25.2@sha256:9be91e9e9cdf116809841fc77ebdb8845443c4c72fe5218f3ae9eb57fdb4bace
```

## Joining to the cluster

You can join new nodes to a k8s cluster by using the `kubeadm join` command.  In this case, we can use `docker exec` to execute this command on our node after its container has started up.

This command won't work out of the box because kind uses a `kubeadm.conf` that does not exist in the node docker image.  It is injected into the container by the kind executable.

Again, using my trusty DiffMerge tool, I compared two `/kind/kubeadm.conf` files in existing kind nodes and found very few differences.  This allowed me to just grab one from any worker node to use as a template.

```bash
docker exec --privileged kind-worker cat /kind/kubeadm.conf > $LOCAL_KUBEADM
```

From here, I needed to set the node's unique IP in its `kubeadm.conf`.  We can use `docker inspect` to grab any node IP address we need.  Since I'm working in bash, I just decided to use a simple sed replacement to replace the template node's IP address with my new node's IP in my local copy of `kubeadm.conf`.

```bash
TEMPLATE_IP=$(docker inspect kind-worker | jq -r '.[0].NetworkSettings.Networks.kind.IPAddress')
NODE_IP=$(docker inspect $NODE_NAME | jq -r '.[0].NetworkSettings.Networks.kind.IPAddress')

ESCAPED_TEMPLATE_IP=$(echo $TEMPLATE_IP | sed 's/\./\\./g' )
ESCAPED_NODE_IP=$(echo $NODE_IP | sed 's/\./\\./g')

sed -i.bkp "s/${ESCAPED_TEMPLATE_IP}/${ESCAPED_NODE_IP}/g" $LOCAL_KUBEADM
```

Now that our `kubeadm.conf` is prepared, we need to copy it to the new node:
```bash
docker exec --privileged -i $NODE_NAME cp /dev/stdin /kind/kubeadm.conf < $LOCAL_KUBEADM
```

Finally, we can join our node to the cluster:

```bash
docker exec --privileged $NODE_NAME kubeadm join --config /kind/kubeadm.conf --skip-phases=preflight --v=6
```

## Node Tags

Since you have complete control of the new node's `kubeadm.conf`, it is possible to configure many of its properties for further testing.  For example, to add additional labels to the new node, you can run something like this:

```bash
sed -i.bkp "s/node-labels: \"\"/node-labels: \"my-label-key=my-label-value\"/g" $LOCAL_KUBEADM
```

This will add the `my-label-key=my-label-value` label to the node once it joins the cluster.

## Future Work

Based on this script, I believe it's possible to add a `kind create node` subcommand to add a node to an existing cluster.  Stay tuned for that...