Writing a kubernetes controller in Go with kubebuilder
Tags: kubernetes, go, operators, controllers
Published: 2020-11-17 23:49:05

---
title: Writing a kubernetes controller in Go with kubebuilder
published: true
description: A simple kubernetes job controller written in Go using kubebuilder
tags: kubernetes,golang,operators,controllers
cover_image: https://dev-to-uploads.s3.amazonaws.com/i/01mnpzqu0t0nhu0n15gb.jpg
---

In this post we'll implement a simple kubernetes controller using the [kubebuilder](https://github.com/kubernetes-sigs/kubebuilder)

### Installing kubebuilder
The installation instructions can be found on kubebuilder [installation docs](https://book.kubebuilder.io/quick-start.html#installation)

### Install kustomize
Since kubebuilder internally relies on kustomize, hence we need to install that from the [instructions here](https://kubernetes-sigs.github.io/kustomize/installation/)

Now we're all set up to start building our controller

---

### Start the project
I have initialized an empty go project
```
go mod init cnat
```

Next we'll initialize the controller project
```
kubebuilder init --domain ishankhare.dev
```

Now we'll ask kubebuilder to setup all the necessary scaffolding for our project
```
kubebuilder create api --group cnat --version v1alpha1 --kind At
```

Kubebuilder will ask us whether to create `Resource` and `Controller` directories, to which we can answer `y` each, since we want these to be created

```
Create Resource [y/n]
y
Create Controller [y/n]
y
Writing scaffold for you to edit...
```


#### Test install scaffold
If we now run `make install`, kubebuilder should generate the base `CRDs` under `config/crd/bases` and a few other files for us.

Running `make run` should now allow us to launch the operator locally. This is really helpful for local testing while we will be implementing and debugging the business logic of our operator code. The live logs will give us insights about how our codebase changes reflect to what's happening with the operator.

#### Enable status subresource
In the file `api/v1alpha1/at_types.go` add the following comment above `At` struct.
```
// +kubebuilder:object:root=true
// +kubebuilder:subresource:status

// At is the Schema for the ats API
type At struct {
.
.
.
```

This will enable us to use privilege separation between `spec` and `status` fields.

#### Background
A little background on what we're going to build here. Our controller is basically a 'Cloud Native At' (cnat) for short – a cloud native version of the Unix's `at` command. We provide a `CRD` manifest containing 2 main things:

1. A command to execute
2. Time to execute the command

Our controller will basically wait and watch till the appropriate time, then would spawn a `Pod` and run the given command in that `Pod`. All this will it will also keep updating the `Status` of that `CRD` which we can view with `kubectl`

***

So let's begin

#### `CRD` outline

This section basically describes how we want our `CRD` to look like. Given the two points we listed above, we can basically represent them as:

```
apiVersion: cnat.ishankhare.dev/v1alpha1
kind: At
metadata:
  name: at-sample
spec:
  schedule: "2020-11-16T10:12:00Z"
  command: "echo hello world!"
```

This is enough information for our controller to decide 'when' and 'what' to execute.

#### Defining the `CRD`
Now that we know what we expect out of the `CRD` let's implement that. Goto the file `api/v1alpha1/at_types.go` and add the following fields to the code:
```
const (
	PhasePending = "PENDING"
	PhaseRunning = "RUNNING"
	PhaseDone    = "DONE"
)
```

Also we edit the existing `AtSpec` and `AtStatus` structs as follows:

```
type AtSpec struct {
  Schedule string `json:"schedule,omitempty"`
  Command  string `json:"command,omitempty"`
}

type AtStatus struct {
  Phase string `json:"phase,omitempty"`
}
```

We save this as we run
```
$ make manifests
go: creating new go.mod: module tmp
go: found sigs.k8s.io/controller-tools/cmd/controller-gen in sigs.k8s.io/controller-tools v0.2.5
/Users/ishankhare/godev/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases

$ make install    # this will install the generated CRD into k8s
```

We should now be able to see an exhaustive `CustomResourceDefinition` generated for us at `config/crd/bases/cnat.ishankhare.dev_ats.yaml`. For people new to `CRD`'s, they will basically help kubernetes identify what our `kind: At` means. Unlike builtin resource types like `Deployments`, `Pods` etc. we need to define custom types for kubernetes through a `CustomResourceDefinition`. Let's try to create an `At` resource:

```
cat >sample-at.yaml<<EOF
apiVersion: cnat.ishankhare.dev/v1alpha1
kind: At
metadata:
  name: at-sample
spec:
  schedule: "2020-11-16T10:12:00Z"
  command: "echo hello world!"
EOF
```

If we now do a `kubectl apply -f` to the previously shown CRD:

```
kubectl apply -f sample-at.yaml
kubectl get at
NAME        AGE
at-sample   124m
```

So now kubernetes recognizes our custom type, but it still does NOT know what to do with it. The logic part of the controller is still missing, but we have the skeleton ready to start writing our logic around it.

***

#### Implementing the Controller logic

Now we're coming to the fun part – implementing the logic for our operator. This will basically tell kubernetes what to do with our generated CRD. As few say its like "making kubernetes do tricks!".

Most of the scaffolding as been already generated for us by kubebuilder in `main.go` and `controllers/`. Inside the `controllers/at_controller.go` file we have the function `Reconcile`. Consider this is the reconcile loop's body.

A small refresher – a controller basically runs a periodic loop over the said objects and watches for any changes to those objects. When changes happen the loop body is triggered and the relevant logic can be executed. This function `Reconcile` is that logical piece, the entry-point for all our controller logic.

Let's start adding a basic structuring around it:

```
func (r *AtReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
    reqLogger := r.Log.WithValues("at", req.NamespacedName)
	reqLogger.Info("=== Reconciling At")

    instance := &cnatv1alpha1.At{}
	err := r.Get(context.TODO(), req.NamespacedName, instance)
	if err != nil {
		if errors.IsNotFound(err) {
			// object not found, could have been deleted after
			// reconcile request, hence don't requeue
			return ctrl.Result{}, nil
		}

		// error reading the object, requeue the request
		return ctrl.Result{}, err
	}

	// if no phase set, default to Pending
	if instance.Status.Phase == "" {
		instance.Status.Phase = cnatv1alpha1.PhasePending
	}

    // state transition PENDING -> RUNNING -> DONE


	return ctrl.Result{}, nil
}
```

Here we basically create a logger and an empty instance of the `At` object, which will allow use to query kubernetes for existing `At` objects and read/write to their status etc. The function is supposed to return the result of the reconciliation logic and an error. The `ctrl.Result` object can optionally tell the reconciler to either `Requeue` immediately (bool) or `RequeueAfter` with a `time.Duration` – again these are optional.

Now lets come to the important part, the state diagram. The image below explains what basically happens when we create a new `At`:

![State diagram](https://dev-to-uploads.s3.amazonaws.com/i/02dsdoh5wnysmwar83v7.png)

Now, lets start implementing this logic in the function:
```
...
// state transition PENDING -> RUNNING -> DONE
switch instance.Status.Phase {
	case cnatv1alpha1.PhasePending:
		reqLogger.Info("Phase: PENDING")

		diff, err := schedule.TimeUntilSchedule(instance.Spec.Schedule)
		if err != nil {
			reqLogger.Error(err, "Schedule parsing failure")

			return ctrl.Result{}, err
		}

		reqLogger.Info("Schedule parsing done", "Result", fmt.Sprintf("%v", diff))

		if diff > 0 {
			// not yet time to execute, wait until scheduled time
			return ctrl.Result{RequeueAfter: diff * time.Second}, nil
		}

		reqLogger.Info("It's time!", "Ready to execute", instance.Spec.Command)
		// change state
		instance.Status.Phase = cnatv1alpha1.PhaseRunning
```

The `schedule.TimeUntilSchedule` function is simple to implement like so:
```
package schedule

import "time"

func TimeUntilSchedule(schedule string) (time.Duration, error) {
	now := time.Now().UTC()
	layout := "2006-01-02T15:04:05Z"
	scheduledTime, err := time.Parse(layout, schedule)
	if err != nil {
		return time.Duration(0), err
	}

	return scheduledTime.Sub(now), nil
}
```

The next `case` in our `switch` body is for `RUNNING` phase. This is the most complicated one and I've added comments wherever I can to explain it better.
```
case cnatv1alpha1.PhaseRunning:
		reqLogger.Info("Phase: RUNNING")

		pod := spawn.NewPodForCR(instance)
		err := ctrl.SetControllerReference(instance, pod, r.Scheme)
		if err != nil {
			// requeue with error
			return ctrl.Result{}, err
		}

		query := &corev1.Pod{}
		// try to see if the pod already exists
		err = r.Get(context.TODO(), req.NamespacedName, query)
		if err != nil && errors.IsNotFound(err) {
			// does not exist, create a pod
			err = r.Create(context.TODO(), pod)
			if err != nil {
				return ctrl.Result{}, err
			}

			// Successfully created a Pod
			reqLogger.Info("Pod Created successfully", "name", pod.Name)
			return ctrl.Result{}, nil
		} else if err != nil {
			// requeue with err
			reqLogger.Error(err, "cannot create pod")
			return ctrl.Result{}, err
		} else if query.Status.Phase == corev1.PodFailed ||
			query.Status.Phase == corev1.PodSucceeded {
			// pod already finished or errored out`
			reqLogger.Info("Container terminated", "reason", query.Status.Reason,
				"message", query.Status.Message)
			instance.Status.Phase = cnatv1alpha1.PhaseDone
		} else {
			// don't requeue, it will happen automatically when
			// pod status changes
			return ctrl.Result{}, nil
		}
```
We set the `ctrl.SetControllerReference` that basically tells kubernetes runtime that the created `Pod` is "owned" by this `At` instance. This is later going to come handy for us when watching on resources created by our Controller.
We also use another external function here `spawn.NewPodForCR` which is basically responsible for spawning the new `Pod` for our `At` custom resource spec. This is implemented as follows:
```
package spawn

import (
	cnatv1alpha1 "cnat/api/v1alpha1"
	"strings"

	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

func NewPodForCR(cr *cnatv1alpha1.At) *corev1.Pod {
	labels := map[string]string{
		"app": cr.Name,
	}

	return &corev1.Pod{
		ObjectMeta: metav1.ObjectMeta{
			Name:      cr.Name,
			Namespace: cr.Namespace,
			Labels:    labels,
		},
		Spec: corev1.PodSpec{
			Containers: []corev1.Container{
				{
					Name:    "busybox",
					Image:   "busybox",
					Command: strings.Split(cr.Spec.Command, " "),
				},
			},
			RestartPolicy: corev1.RestartPolicyOnFailure,
		},
	}
}
```

The last bits of code deal with the `DONE` phase and the `default` case of `switch`. We also do the updation of our `status` subresource outside the switch as a common part of code:

```
case cnatv1alpha1.PhaseDone:
		reqLogger.Info("Phase: DONE")
		// reconcile without requeuing
		return ctrl.Result{}, nil
	default:
		reqLogger.Info("NOP")
		return ctrl.Result{}, nil
	}

	// update status
	err = r.Status().Update(context.TODO(), instance)
	if err != nil {
		return ctrl.Result{}, err
	}

	return ctrl.Result{}, nil
```

Lastly we utilize the `SetControllerReference` by adding the following to the `SetupWithManager` function in the same file:
```
err := ctrl.NewControllerManagedBy(mgr).
		For(&cnatv1alpha1.At{}).
		Owns(&corev1.Pod{}).
		Complete(r)

	if err != nil {
		return err
	}

	return nil
```

The part specifying `Owns(&corev1.Pod{})` is important and tells the controller manager that pods created by this controller also needs to be watched for changes.

If you want to have a look at the entire final code implementation, head over the this github repo – [ishankhare07/kubebuilder-controller](https://github.com/ishankhare07/kubebuilder-controller).
With this in place, we are now done with implementation of our controller, we can run and test it on a cluster in our current kube context using `make run`
```
go: creating new go.mod: module tmp
go: found sigs.k8s.io/controller-tools/cmd/controller-gen in sigs.k8s.io/controller-tools v0.2.5
/Users/ishankhare/godev/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
/Users/ishankhare/godev/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
go run ./main.go
2020-11-18T05:05:45.531+0530    INFO    controller-runtime.metrics      metrics server is starting to listen    {"addr": ":8080"}
2020-11-18T05:05:45.531+0530    INFO    setup   starting manager
2020-11-18T05:05:45.531+0530    INFO    controller-runtime.manager      starting metrics server {"path": "/metrics"}
2020-11-18T05:05:45.531+0530    INFO    controller-runtime.controller   Starting EventSource    {"controller": "at", "source": "kind source: /, Kind="}
2020-11-18T05:05:45.732+0530    INFO    controller-runtime.controller   Starting EventSource    {"controller": "at", "source": "kind source: /, Kind="}
2020-11-18T05:05:46.232+0530    INFO    controller-runtime.controller   Starting Controller     {"controller": "at"}
2020-11-18T05:05:46.232+0530    INFO    controller-runtime.controller   Starting workers        {"controller": "at", "worker count": 1}
2020-11-18T05:05:46.232+0530    INFO    controllers.At  === Reconciling At      {"at": "cnat/at-sample"}
2020-11-18T05:05:46.232+0530    INFO    controllers.At  Phase: PENDING  {"at": "cnat/at-sample"}
2020-11-18T05:05:46.232+0530    INFO    controllers.At  Schedule parsing done   {"at": "cnat/at-sample", "Result": "-14053h23m46.232658s"}
2020-11-18T05:05:46.232+0530    INFO    controllers.At  It's time!      {"at": "cnat/at-sample", "Ready to execute": "echo YAY!"}
2020-11-18T05:05:46.436+0530    DEBUG   controller-runtime.controller   Successfully Reconciled {"controller": "at", "request": "cnat/at-sample"}
2020-11-18T05:05:46.443+0530    INFO    controllers.At  === Reconciling At      {"at": "cnat/at-sample"}
2020-11-18T05:05:46.443+0530    INFO    controllers.At  Phase: RUNNING  {"at": "cnat/at-sample"}
2020-11-18T05:05:46.718+0530    INFO    controllers.At  Pod Created successfully        {"at": "cnat/at-sample", "name": "at-sample"}
2020-11-18T05:05:46.718+0530    DEBUG   controller-runtime.controller   Successfully Reconciled {"controller": "at", "request": "cnat/at-sample"}
2020-11-18T05:05:46.722+0530    INFO    controllers.At  === Reconciling At      {"at": "cnat/at-sample"}
2020-11-18T05:05:46.722+0530    INFO    controllers.At  Phase: RUNNING  {"at": "cnat/at-sample"}
2020-11-18T05:05:46.722+0530    DEBUG   controller-runtime.controller   Successfully Reconciled {"controller": "at", "request": "cnat/at-sample"}
2020-11-18T05:05:46.778+0530    INFO    controllers.At  === Reconciling At      {"at": "cnat/at-sample"}
2020-11-18T05:05:46.778+0530    INFO    controllers.At  Phase: RUNNING  {"at": "cnat/at-sample"}
2020-11-18T05:05:46.778+0530    DEBUG   controller-runtime.controller   Successfully Reconciled {"controller": "at", "request": "cnat/at-sample"}
2020-11-18T05:05:46.846+0530    INFO    controllers.At  === Reconciling At      {"at": "cnat/at-sample"}
2020-11-18T05:05:46.847+0530    INFO    controllers.At  Phase: RUNNING  {"at": "cnat/at-sample"}
2020-11-18T05:05:46.847+0530    DEBUG   controller-runtime.controller   Successfully Reconciled {"controller": "at", "request": "cnat/at-sample"}
2020-11-18T05:05:49.831+0530    INFO    controllers.At  === Reconciling At      {"at": "cnat/at-sample"}
2020-11-18T05:05:49.831+0530    INFO    controllers.At  Phase: RUNNING  {"at": "cnat/at-sample"}
2020-11-18T05:05:49.831+0530    INFO    controllers.At  Container terminated    {"at": "cnat/at-sample", "reason": "", "message": ""}
2020-11-18T05:05:50.054+0530    DEBUG   controller-runtime.controller   Successfully Reconciled {"controller": "at", "request": "cnat/at-sample"}
2020-11-18T05:05:50.056+0530    INFO    controllers.At  === Reconciling At      {"at": "cnat/at-sample"}
2020-11-18T05:05:50.056+0530    INFO    controllers.At  Phase: DONE     {"at": "cnat/at-sample"}
2020-11-18T05:05:50.056+0530    DEBUG   controller-runtime.controller   Successfully Reconciled {"controller": "at", "request": "cnat/at-sample"}
```

If you're interested to see the output command we can run:
```
$ kubectl logs -f at-sample
hello world!
```

This post was first published on my personal blog [ishankhare.dev](https://ishankhare.dev)