How to create a Kubernetes Operator ?
Tags: kubernetes, devops, sre, tutorial
Published: 2023-06-12 13:21:00

In the first part of this serie about the **operator pattern**, we saw what it is and in which cases it can be highly helpful, especially for automation.

So today we will see **how to create a Kubernetes Operator**!

---

## Tools
As every open-source solution, if you need to do something, a bunch of tools exists with their own specificities. If you want to see the list, go check the [**Kubernetes documentation**](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/#writing-operator).

In this serie, we will use [**Operator Framework**](https://operatorframework.io/) and [**KubeBuilder**](https://book.kubebuilder.io/).

### Operator Framework
A few words about Operator Framework, we will use the **Go SDK** but you need to know that you can also use it with **Ansible** and **Helm**. 

---

## Setup
### Homebrew
If you are using **Homebrew**, you can install the **Operator Framework SDK** with the following command :
```bash
brew install operator-sdk
```

### From Github Release
```bash
# Define informations about your platform
export ARCH=$(case $(uname -m) in x86_64) echo -n amd64 ;; aarch64) echo -n arm64 ;; *) echo -n $(uname -m) ;; esac)
export OS=$(uname | awk '{print tolower($0)}')

# Download the binary for your platform
export OPERATOR_SDK_DL_URL=https://github.com/operator-framework/operator-sdk/releases/download/v1.28.0
curl -LO ${OPERATOR_SDK_DL_URL}/operator-sdk_${OS}_${ARCH}

# Install the binary
chmod +x operator-sdk_${OS}_${ARCH} && sudo mv operator-sdk_${OS}_${ARCH} /usr/local/bin/operator-sdk
```

---

## Create our first operator
### Initialize the project
The first thing to do is to initialize the project with the following command
> operator-sdk init --domain [YOUR DOMAIN] --repo [YOUR CODE REPOSITORY]

*Example*
```bash
operator-sdk init --domain adaendra.org --repo github.com/adaendra/test-operator
```

It will generate a folder structure like this

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cegsf30k0iwmpg93u4ln.png)

You will be able to find some generic files, a lot of common files *(like the Makefile or Dockerfile)* and the begining of your Golang project with `main.go`.


> Note : By default your namespace is able to watch resources everywhere in the cluster.
>So if you want to limit its vision, you can update the definition of the manager to add the `Namespace`  option.
> `mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{Namespace: "dummy_ns"})`
> 
> *For more informations about the scope of an operator, please check the [SDK documentation](https://sdk.operatorframework.io/docs/building-operators/golang/operator-scope/)*


### Create an API, a Controller and a CRD
In a lot of cases when we use an operator we want to create a **Custom Resource Definition** which will be used as reference for our tasks.

In this tutorial we will create a custom resource **MyProxy** in the **gateway** group which, for each instance, will deploy a **Nginx deployment**.

#### Command to generate the code
```console
operator-sdk create api --group gateway --version v1alpha1 --kind MyProxy --resource --controller
```
Once executed, you can see two new folders : **api** and **controllers**.

#### API
In this folder, the only file that will interest us is  `myproxy_types.go`. It's in this file where we will define all the fields that we need in our **Spec**, but it's also here we will define the **Status** structure!

For our example, we will just define a `Name` field in **MyProxy Spec**.
```go
type MyProxySpec struct {  
   Name string `json:"name,omitempty"`  
}
```

> **Important !!** This file is used as base to build numerous yaml files for your operator. So, every modification in this file, execute both commands :
> `make manifests` & `make generate`

#### Controller
In this folder, you will find every controllers related to your **Custom Resources** like `myproxy_controller.go` that we generated earlier. This folder is the central place about operations that your operator can do.

In every controller file, you will find two methods that we must update : **Reconcile** and **SetupWithManager**.

##### SetupWithManager
```go
// SetupWithManager sets up the controller with the Manager.
func (r *MyProxyReconciler) SetupWithManager(mgr ctrl.Manager) error {  
   return ctrl.NewControllerManagedBy(mgr).  
      For(&gatewayv1alpha1.MyProxy{}).  
      Owns(&appsv1.Deployment{}).  
      Complete(r)  
}
```
In this example *(which is also our implementation)*, we can see :

`ctrl.NewControllerManagedBy(mgr)` which creates a new controller with basic options. *(It's in this method where you can personalize your controller options like the number of reconciliations max you want in parallel)*

`For(&gatewayv1alpha1.MyProxy{})` will declare that we **want the reconciliation to be triggered if a add/update/delete event happen** on a specific kind of resource . *(Here MyProxy)* You can use it for **each kind of resource you want to watch.** *(Useful if you want to expose dynamically all the deployments through a Nginx for example)*

`Owns(&appsv1.Deployment{})`  is quite similar as `For`, so it will declare that we **want the reconciliation to be triggered if a add/update/delete event happen**. But it will also add a filter, because the **reconciliation will only be triggered if the operator own the resource with the event**. *(So if you update another deployment, nothing will happend in your operator)*

##### Reconcile
This method is  the heart of your operator, and is the one which will be executed **every time a reconciliation will be triggered**.

But before dive into the function, there is an important thing to see before. Above the method, you can see some comments starting with `//+kubebuilder`.  **This comments defines the rights for your operator!**

So it's really important to define them correctly! In our case, we need to add some rights to our operator to be able to read/create and update **Deployments**.

Each comment is defined as follow : 
> // +kubebuiler:rbac:groups=[group of the resource],resources=[resources name],verbs=[verbs]

The **group of the resource** must be only one value, but for **resources name** and **verbs**, you can define multiple values at once, joining all the values with `;`.

```go
// +kubebuilder:rbac:groups=gateway.adaendra.org,resources=myproxies,verbs=get;list;watch;create;update;patch;delete  
// +kubebuilder:rbac:groups=gateway.adaendra.org,resources=myproxies/status,verbs=get;update;patch  
// +kubebuilder:rbac:groups=gateway.adaendra.org,resources=myproxies/finalizers,verbs=update  
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete  
```

Now we can dive into the function code. As said earlier, this **function will be called each time a reconciliation is triggered**. As a result, we must be careful about what we are doing here!

For example, if we want to create resources, we must be sure that they are not already existing on the cluster! And if it already exists, we must check it and do some updates if required!

###### 1. Retrieve our custom resource
So the first step is to try to retrieve an instance of our custom resource *(here an instance of MyProxy)*. We need it to get its spec, and being able to update its status.

```go
    // Retrieve the resource
    myProxy := &gatewayv1alpha1.MyProxy{}
    err := r.Get(ctx, req.NamespacedName, myProxy)

    if err != nil {
	    // If we have an error and this error said "not found", we ignore the error
        if errors.IsNotFound(err) {
            log.Info("Resource not found. Error ignored as the resource must have been deleted.")
            return ctrl.Result{}, nil
        }

		// If it's another error, we return it
        log.Error(err, "Error while retrieving MyProxy instance")
        return ctrl.Result{}, err
    }
```

###### 2. Retrieve resources managed by the operator
Now that we have our "parent" resource, we are go retrieve our "child" resources. In our case it's a deployment, its name is defined with the  field `Name` from MyProxy and must be in the namespace  `test_ns`.

```go
found := &appsv1.Deployment{}  
err = r.Get(ctx, types.NamespacedName{Name: myProxy.Spec.Name, Namespace: "test_ns"}, found)
```

###### 3. Check if  the resource exists
The following step consists to  check what we get at the previous step. If the variable `err` is  a not found error, we  know that the resource doesn't exist, so we can create it! If it contains another error, we return it.

Our implementation will look like this

```go
if err != nil && errors.IsNotFound(err) {  
   // Define a new deployment  
   dep := r.deploymentForExample(myProxy)  
   log.Info("Creating a new Deployment", "Deployment.Namespace", dep.Namespace, "Deployment.Name", dep.Name)  
   err = r.Create(ctx, dep)  
   if err != nil {  
      log.Error(err, "Failed to create new Deployment", "Deployment.Namespace", dep.Namespace, "Deployment.Name", dep.Name)  
      return ctrl.Result{}, err  
   }  
   // Deployment created successfully - return and requeue  
   return ctrl.Result{Requeue: true}, nil  
} else if err != nil {  
   log.Error(err, "Failed to get Deployment")  
   return ctrl.Result{}, err  
}
```

*Here is a really simple example of `deploymentForExample`*
```go
func (r *MyProxyReconciler) deploymentForExample(myproxy *gatewayv1alpha1.MyProxy) *appsv1.Deployment {  
   dep := &appsv1.Deployment{}  
  
   dep.Namespace = "test_ns"  
   dep.Name = myproxy.Spec.Name  
  
   var replicas int32 = 2  
  
   labels := map[string]string{  
      "test_label": myproxy.Spec.Name,  
   }  
  
   dep.Spec = appsv1.DeploymentSpec{  
      Replicas: &replicas,  
      Template: corev1.PodTemplateSpec{  
         Spec: corev1.PodSpec{  
            Containers: []corev1.Container{  
               {  
                  Name:  "nginx",  
                  Image: "nginx",  
               },  
            },  
         },  
      },  
   }  
   dep.Labels = labels  
   dep.Spec.Template.Labels = labels  
  
   return dep  
}
```


###### 4. Update the resource
If we don't get an error while trying to retrieve the resource, it means that we were able to correctly get a resource. So we can check it's parameters and update it if some values has been changed.

```go
var size int32 = 2  
if *found.Spec.Replicas != size {  
   found.Spec.Replicas = &size  
   err = r.Update(ctx, found)  
   if err != nil {  
      log.Error(err, "Failed to update Deployment", "Deployment.Namespace", found.Namespace, "Deployment.Name", found.Name)  
      return ctrl.Result{}, err  
   }  
   // Spec updated - return and requeue  
   return ctrl.Result{Requeue: true}, nil  
}
```

In our  example, we will check that the number of pods is still equal to `2`. If it's not the case, we will try to update the resource, and manage the error if we get one.

##### Update generated files
Once we finished to update our controller, it's really important to execute the following command:
```bash
make manifests
```

We saw earlier that we can find some comments that defines RBAC rights for our controller. So we need to execute this command to *(at least)* **generate RBAC definitions files**.

### Operator build
Now that our operator is ready to be used, we can build it before deploy it.

#### Before the build
By default, the built image will be named `controller:latest` and can be push to `example.com/tmpoperator`. As you can imagine, it can generate some issues.

So, if you want to update these informations, you must : 
- update variables `IMG` and `IMAGE_TAG_BASE` in the **Makefile**
- update the image name in **config/manager/manager.yaml**

#### Build
To execute the build, use this command
```bash
make docker-build
```

And this one if you want to push the image to a remote docker registry
```bash
make docker-push
```

#### Deployment
To deploy your operator, you must execute 2 commands : 

**to deploy all your Custom Resource Definitions on your cluster**
```bash
make install
```

**to deploy your operator**
```bash
make deploy
```


### Test
When everything above is done, you can try to deploy an instance of `MyProxy` and you should see a nginx deployment appear!

*Example of a MyProxy instance definition*
```yaml
apiVersion: gateway.example.com/v1alpha1  
kind: MyProxy  
metadata:  
  labels:  
    app.kubernetes.io/name: myproxy  
    app.kubernetes.io/instance: myproxy-sample  
    app.kubernetes.io/part-of: tmpoperator  
    app.kubernetes.io/managed-by: kustomize  
    app.kubernetes.io/created-by: tmpoperator  
  name: myproxy-sample  
spec:  
  name: toto
```

---

This part was quite long, but it was necessary to show you how to create an operator and see what we can do with.

In the next part of this serie, we will see advanced configurations and features to help your operator to be more efficient!

I hope it will help you and if you have any questions *(there are not dumb questions)* or some points are not clear for you, don't hesitate to add your question in the comments or to contact me directly on  [LinkedIn](https://www.linkedin.com/in/maxime-guilbert-14b6a5128/).

---

<center><h3>You want to support me?</h3></center>
<a href="https://www.buymeacoffee.com/mxguilbert7" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/default-orange.png" alt="Buy Me A Coffee" height="41" width="174"></a>