KWOK - Kubernetes without Kubelet
Tags: kubernetes, experimental
Published: 2023-04-10 12:13:08

#### Premise
Kubernetes is an open-source container orchestration platform that automates the deployment, scaling, and management of containerized applications. While kubelet is an essential component of Kubernetes, it is not necessary for running Kubernetes. Kubelet is responsible for managing the state of each node in the Kubernetes cluster and ensuring that containers are running on the node as expected. However, there are alternative ways to manage Kubernetes clusters without kubelet, such as using external node agents or third-party tools. These tools can perform similar functions as kubelet, including managing node resources and monitoring container health. Therefore, while kubelet is an integral part of Kubernetes, it is not a requirement for running a functional and efficient Kubernetes cluster.

KWOK, which stands for Kubernetes Without Kubelet, is a new experimental project that aims to manage Kubernetes clusters in a different way. Normally, Kubernetes uses a component called kubelet to manage the state of each node in the cluster and ensure that containers are running correctly. However, KWOK explores the possibility of replacing kubelet with other tools that can do similar jobs, to make Kubernetes clusters more lightweight and customizable.

Even though the KWOK project is still in early development and not ready for production use, it has shown promising results. By using KWOK, users can have more control over how they manage their Kubernetes clusters and tailor them to their specific needs. KWOK also allows for more compatibility with other containerization platforms and unique environments.

---
#### Component hierarchy
```
                 +-----------------------+
                 | Kubernetes API Server |
                 +-----------------------+
                            |
                            v
                  +----------------+
                  | Etcd           |
                  +----------------+
                            |
                            v
   +-------------------------------------+
   | Kubernetes Controller Manager       |
   +-------------------------------------+
                            |
                            v
   +-------------------------------------+
   | Kubernetes Scheduler                |
   +-------------------------------------+
                            |
                            v
   +----------------------------------------+
   | External Node Agent (replacing kubelet)|
   +----------------------------------------+
                            |
                            v
   +-------------------------------------+
   |    Container Runtime (containerd)   |
   +-------------------------------------+
   
```
[Architecture](https://kwok.sigs.k8s.io/docs/design/architecture/)

---
#### Need
Try KWOK if you want to (one or more of):
- simulate kubernetes resources (1000s of nodes, pods ...)
- create ultra-fast clusters for testing/simulation
- verify manifests across kube-api server
- play around with taints and tolerations without actual nodes
- just get a feel of kubernetes

#### Install - https://kwok.sigs.k8s.io/docs/user/install/
```sh
# Install the main cli
❯ brew install kwok
==> Downloading https://formulae.brew.sh/api/formula.jws.json
...
❯ kwok --version
kwok version v0.1.1

# Install the command to control cluster (m1 mac in my case)
❯ wget https://github.com/kubernetes-sigs/kwok/releases/download/v0.1.1/kwok-darwin-arm64
❯ kwokctl --version
kwokctl version v0.1.1
```

#### Cluster administration
*To create clusters and builds you will need to [install docker](https://docs.docker.com/install/#supported-platforms).*
```sh
# assumption: container images are already pulled on the host
❯ time kwokctl create cluster --name=c-one
...
Status: Downloaded newer image for registry.k8s.io/kube-scheduler:v1.26.0
registry.k8s.io/kube-scheduler:v1.26.0
Starting cluster                                                                                                                      cluster="c-one"
[+] Running 6/6
 ⠿ Network kwok-c-one                            Created                                                                                         0.0s
 ⠿ Container kwok-c-one-etcd                     Started                                                                                         0.5s
 ⠿ Container kwok-c-one-kube-apiserver           Started                                                                                         0.7s
 ⠿ Container kwok-c-one-kwok-controller          Started                                                                                         1.2s
 ⠿ Container kwok-c-one-kube-controller-manager  Started                                                                                         1.2s
 ⠿ Container kwok-c-one-kube-scheduler           Started                                                                                         1.0s
Cluster is created                                                                                                         cluster="c-one" elapsed=1s
You can now use your cluster with:

    kubectl config use-context kwok-c-one

Thanks for using kwok!
kwokctl create cluster --name=c-one  1.25s user 0.52s system 32% cpu 5.397 total

# images pulled
❯ docker image ls | grep registry.k8s
registry.k8s.io/kwok/kwok                                          v0.1.1    b4aa6a246bed   3 weeks ago     61.6MB
registry.k8s.io/kube-apiserver                                     v1.26.0   646e01ef5206   4 months ago    129MB
registry.k8s.io/kube-scheduler                                     v1.26.0   1c1f5ebdeb5a   4 months ago    54.8MB
registry.k8s.io/kube-controller-manager                            v1.26.0   8c4e5a3e4d95   4 months ago    120MB
registry.k8s.io/etcd                                               3.5.6-0   ef2458028240   4 months ago    181MB

# list clusters
❯ kwokctl get clusters
c-one

# get kubeconfig
❯ kwokctl get kubeconfig --name c-one > kubeconfig

# Delete cluster
❯ kwokctl delete cluster --name=c-one
Stopping cluster                                                                                                                      cluster="c-one"
[+] Running 6/6
 ⠿ Container kwok-c-one-kwok-controller          Removed                                                                                         0.2s
 ⠿ Container kwok-c-one-kube-controller-manager  Removed                                                                                         0.2s
 ⠿ Container kwok-c-one-kube-scheduler           Removed                                                                                         0.2s
 ⠿ Container kwok-c-one-kube-apiserver           Removed                                                                                         1.3s
 ⠿ Container kwok-c-one-etcd                     Removed                                                                                         0.2s
 ⠿ Network kwok-c-one                            Removed                                                                                         0.1s
Deleting cluster                                                                                                                      cluster="c-one"
Cluster deleted
```

#### Using the cluster
```sh
❯ kubectl config use-context kwok-c-one
Switched to context "kwok-c-one".

# create nodes
❯ cat /tmp/node.yaml
apiVersion: v1
kind: Node
metadata:
  annotations:
    kwok.x-k8s.io/node: fake
  name: node-02
spec:
  taints: []
❯ k apply -f /tmp/node.yaml
node/node-01 created
❯ k get nodes
NAME      STATUS   ROLES    AGE   VERSION
node-01   Ready    <none>   32s   fake

# run a pod
❯ k run nginx-pod --image=nginx
pod/nginx-pod created
❯ k get pods
NAME        READY   STATUS    RESTARTS   AGE
nginx-pod   1/1     Running   0          4s

# create a service
❯ k expose pod nginx-pod --port 8080 --target-port 80
service/nginx-pod exposed
❯ k get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
kubernetes   ClusterIP   10.0.0.1     <none>        443/TCP    40m
nginx-pod    ClusterIP   10.0.0.20    <none>        8080/TCP   5s

# access a service - is still TODO
❯ k port-forward svc/nginx-pod --address=0.0.0.0 8080:8080
Forwarding from 0.0.0.0:8080 -> 80
Handling connection for 8080
# from another terminal
❯ http localhost:8080
http: error: ConnectionError: ('Connection aborted.', BadStatusLine('TODO: PortForward("default/nginx-pod", \'P\')')) while doing a GET request to URL: http://localhost:8080/

❯ k delete pod nginx-pod
pod "nginx-pod" deleted
```

---
#### Conclusion
Very interesting concept, instantaneous turn-around + some added goodies like monitoring, audits and admission control. It is not yet fit for end-user use, [kind](https://dev.to/ashokan/development-cluster-with-kind-3429) is preferred for now

---
#### Further reading
- [Roadmap](https://github.com/orgs/kubernetes-sigs/projects/21/views/2)
- [Documentation](https://kwok.sigs.k8s.io/)