How to Build A Kubernetes Operator
Tags: kubernetes, docker, devops
Published: 2020-06-19 18:45:34

This is the second part of our series focusing on Kubernetes Operators, and it shows how you can **build a Kubernetes Operator based on the Bitnami Apache Helm chart**. Note that you can refer to the steps in this tutorial to build an operator for your own applications.

#Prerequisites#

1.We assume you followed the **first part of the series**. Thus, you should have a **Kubernetes cluster**  (v1.7 or newer) with a control plane and two workers running on your computer. Also, the **Operator Lifecycle Manager** should be installed on your system.  You can enter the following command to verify that everything is set up:

```
kubectl get all --namespace olm
```

```
NAME                                    READY   STATUS    RESTARTS   AGE
pod/catalog-operator-64b6b59c4f-brck9   1/1     Running   0          3m28s
pod/olm-operator-844fb69f58-fn57f       1/1     Running   0          3m28s
pod/operatorhubio-catalog-5s8k2         1/1     Running   0          3m4s
pod/packageserver-65df5d5cc9-nz26h      1/1     Running   0          3m2s
pod/packageserver-65df5d5cc9-x7hwc      1/1     Running   0          3m2s

NAME                                       TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE
service/operatorhubio-catalog              ClusterIP   10.103.1.120   <none>        50051/TCP   3m3s
service/v1-packages-operators-coreos-com   ClusterIP   10.99.75.171   <none>        443/TCP     3m3s

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/catalog-operator   1/1     1            1           3m28s
deployment.apps/olm-operator       1/1     1            1           3m28s
deployment.apps/packageserver      2/2     2            2           3m2s

NAME                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/catalog-operator-64b6b59c4f   1         1         1       3m28s
replicaset.apps/olm-operator-844fb69f58       1         1         1       3m28s
replicaset.apps/packageserver-65df5d5cc9      2         2         2       3m2s
```

2.The **Operator SDK** is installed on your machine. For details about installing the Operator SDK, refer to the [Install the Operator SDK CLI page](https://github.com/operator-framework/operator-sdk/blob/master/doc/user/install-operator-sdk.md).
3.**Helm CLI** is installed on your computer. To install Helm CLI, follow the instructions from the [Installing Helm](https://helm.sh/docs/intro/install/) page.
4.**Docker**. For details about installing Docker, refer to the [Install Docker](https://docs.docker.com/v17.12/install///) page.
5.You need a **[quay.io](https://quay.io/)** account. 

#Set Up The Apache Operator#

In this section, we'll walk you through the process of **setting up the Apache Operator using Bitnami's Helm chart**.

1.Add the Bitnami Helm repository to your Helm client by running the following command:

```
helm repo add bitnami https://charts.bitnami.com
```

```
"bitnami" has been added to your repositories
```

2.Create a Helm-based Operator by running the `operator-sdk new` command, and passing it the following arguments:
* The name of your operator (`apache-operator`).
* The `--api-version` flag with the Kubernetes apiVersion. The format is `$GROUP_NAME/$VERSION`. In this tutorial, we'll be using `appfleet.com/v1alpha1`.
* The `--kind flag` with the name of the Kubernetes CRD (`Apache`)
* The `--type` flag with the type of operator. We'll be using `helm`. Other valid types are `Go` and `Ansible`
* The `helm-chart` flag with the name of the Helm chart (`bitnami/apache`)

```
operator-sdk new apache-operator --api-version=appfleet.com/v1alpha1 --kind=Apache --type=helm --helm-chart=bitnami/apache
```

```
INFO[0000] Creating new Helm operator 'apache-operator'.
INFO[0001] Created helm-charts/apache
INFO[0001] Generating RBAC rules
WARN[0001] The RBAC rules generated in deploy/role.yaml are based on the chart's default manifest. Some rules may be missing for resources that are only enabled with custom values, and some existing rules may be overly broad. Double check the rules generated in deploy/role.yaml to ensure they meet the operator's permission requirements.
INFO[0001] Created build/Dockerfile
INFO[0001] Created watches.yaml
INFO[0001] Created deploy/service_account.yaml
INFO[0001] Created deploy/role.yaml
INFO[0001] Created deploy/role_binding.yaml
INFO[0001] Created deploy/operator.yaml
INFO[0001] Created deploy/crds/appfleet.com_v1alpha1_apache_cr.yaml
INFO[0001] Generated CustomResourceDefinition manifests.
INFO[0001] Project creation complete.
```

This command creates the following directory structure:

```
tree apache-operator -L 2
```

```
apache-operator
├── build
│   └── Dockerfile
├── deploy
│   ├── crds
│   ├── operator.yaml
│   ├── role.yaml
│   ├── role_binding.yaml
│   └── service_account.yaml
├── helm-charts
│   └── apache
└── watches.yaml

```

Things to note:

* Kubernetes compares the actual state of the cluster with the desired state. Then, it takes action to match these states.  The Operators extend this pattern by watching a specific custom resource type and taking actions to match the spec in that resource. In this example, the Operator watches the Apache resource as defined in the **watches.yaml** file:

```
cat apache-operator/watches.yaml
```

```
---
- version: v1alpha1
  group: appfleet.com
  kind: Apache
  chart: helm-charts/apache
```

The `Dockerfile` uses the `quay.io/operator-framework/helm-operator:v0.15.1` image as the base, and then it copies `watches.yaml` file and the Helm charts:

```
cat apache-operator/build/Dockerfile
```

```
FROM quay.io/operator-framework/helm-operator:v0.15.1

COPY watches.yaml ${HOME}/watches.yaml
COPY helm-charts/ ${HOME}/helm-charts/
```

3.Now you can build the Apache Operator by moving into the `apache-operator` directory, and then entering the following `operator-sdk build` command: 

```
operator-sdk build apache-operator:v0.1
```

```
INFO[0000] Building OCI image apache-operator:v0.1
Sending build context to Docker daemon  64.51kB
Step 1/3 : FROM quay.io/operator-framework/helm-operator:v0.15.1
 ---> 450a3ca2d02d
Step 2/3 : COPY watches.yaml ${HOME}/watches.yaml
 ---> Using cache
 ---> db5c285c02fb
Step 3/3 : COPY helm-charts/ ${HOME}/helm-charts/
 ---> 50255ede17de
Successfully built 50255ede17de
Successfully tagged apache-operator:v0.1
INFO[0003] Operator build complete.
```

4.Verify that the Docker image was created with:

```
docker images | grep apache
```

```
apache-operator                            v0.1                50255ede17de        38 seconds ago      174MB
```

#Set Up quay.io with Docker#

You can think of Quay as something similar to GitHub, but for Docker images. It's a registry where you can host images and share them. There are a couple of ways to set up `quay.io` with Docker. For the sake of simplicity, you'll use the `docker login` command. It's important to note that the `docker login` command stores the password you enter as **plain-text**. Thus, you should first generate an encrypted password. 

1.Point your browser to https://quay.io/, and then navigate to *Account Settings*:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/6o6yrd4kbvvrstm74uqc.png)

2.From the *Account Settings* page select *Generate Encrypted Password*:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/0f5l5v6h5zgm8zsamw9o.png)

3.You will be prompted to enter your `quay.io` password:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/j0qdwhuyd4s7rd4h1gen.png)

4.Select *Docker Login* and then copy the command containing your Docker encrypted password:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/vqmycuy01x8cp63nsp0a.png)

5.In a terminal window, log in to `quay.io` by entering the following command:

```
docker login -u="<YOUR_USERNAME>" -p="<YOUR_ENCRYPTED_PASSWORD>" quay.io
```

```
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Login Succeeded
```

#Push the Apache Operator Image to quay.io#

To do this, you first need to properly tag the image with the hostname of the registry, and the name of your repository.  Then, you can push the image.

1.Enter the following command to tag the local image named  `apache-operator` into the `quay.io/andreipope` repository: 

```
docker tag apache-operator:v0.1 quay.io/andreipope/apache-operator:v0.1
```

Note that the image name is comprised by a slash-separated list of the:
* Registry hostname (`quay.io`)
* Repository (`andreipope`) 
* Operator name (`apache-version`). 

☞ The name of our repository is `andreipope`, but yours will be different.

2.To push the image that we created in the previous section, run the following command `docker push` command:

```
docker push quay.io/andreipope/apache-operator:v0.1
```

```
b8325e5fabd7: Pushed
2f4354fc6a73: Pushed
b496b494f6f9: Pushed
9fd48ecc1227: Pushed
0141daa77f22: Pushed
27cd2023d60a: Pushed
4b52dfd1f9d9: Pushed
v0.1: digest: sha256:7230926984fc3d688e02764748441a74907eeb772e3b51eb06b1bac225ba9f98 size: 1778
```

3.Point your browser to https://quay.io/, navigate to the apache-operator repository, and make repository public:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/eo1jgkzhhd64kgexkwga.png)

#Deploy the Apache Operator#

You are now ready to deploy the Apache Operator. Before that, you must customize the specs.

1.Open the `deploy/operator.yaml` file in a plain-text editor and update the placeholder `image: REPLACE_IMAGE`  with the location of your image (`quay.io/andreipope/apache-operator:v0.1`)

Your `deploy.operator.yaml` file should look similar to the following:

```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: apache-operator
spec:
  replicas: 1
  selector:
    matchLabels:
      name: apache-operator
  template:
    metadata:
      labels:
        name: apache-operator
    spec:
      serviceAccountName: apache-operator
      containers:
        - name: apache-operator
          # Replace this with the built image name
          image: quay.io/andreipope/apache-operator:v0.1
          imagePullPolicy: Always
          env:
            - name: WATCH_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: OPERATOR_NAME
              value: "apache-operator"

```

2.Enter these `kubectl create` commands to deploy the Apache Operator:

```
kubectl create -f deploy/service_account.yaml
kubectl create -f deploy/role.yaml
kubectl create -f deploy/role_binding.yaml
kubectl create -f deploy/operator.yaml
```

```
serviceaccount/expressjs-operator created
role.rbac.authorization.k8s.io/expressjs-operator created
rolebinding.rbac.authorization.k8s.io/expressjs-operator created
deployment.apps/expressjs-operator created
```

3.Check the status of the deployment:

```
kubectl get deployment
NAME              READY   UP-TO-DATE   AVAILABLE   AGE
apache-operator   1/1     1            1           24s
```

☞ Note that the deployment doesn't define the spec for the Apache cluster. You’ll describe the Apache cluster in the next section, once the Operator is running.

4.The Operator is a pod running in this deployment. To see it, type the following command:

```
kubectl get pods
```

```
kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE
apache-operator-6d5795f879-np6pr   1/1     Running   1          38s
```

#Deploy the Apache Cluster#

In the *Set Up the Apache Operator section*, you created a CRD  defining a new kind of resource, an Apache cluster. Now, you will apply that spec so that the Operator starts watching the Apache resources. Then, you will deploy the Apache cluster itself. 

1.First, let's deploy the CRD that defines the resources the Operator will monitor:

```
kubectl apply -f deploy/crds/appfleet.com_apaches_crd.yaml
customresourcedefinition.apiextensions.k8s.io/apaches.appfleet.com created
```

```
customresourcedefinition.apiextensions.k8s.io/apaches.appfleet.com created
```

2.At this point, you are ready to deploy the Apache cluster:

```
kubectl apply -f deploy/crds/appfleet.com_v1alpha1_apache_cr.yaml
```

```
apache.appfleet.com/example-apache created
```

3.The deployment takes a bit of time to complete. Once everything is ready, you should see a new pod running Apache:

```
kubectl get pods
```

```
NAME                               READY   STATUS    RESTARTS   AGE
apache-operator-6d5795f879-np6pr   1/1     Running   4          2m53s
example-apache-7cf789fc98-462dr    1/1     Running   0          39s
```

4.You can also check that the deployment was created by entering the following command:

```
kubectl get deployment
```

```
NAME              READY   UP-TO-DATE   AVAILABLE   AGE
apache-operator   1/1     1            1           5m43s
example-apache    2/2     2            2           3m29s
```

#Scaling Up#

At this point, you have a running Apache cluster. To add another instance, you must modify the `replicaCount` field  in the `deploy/crds/appfleet.com_v1alpha1_apache_cr.yamlapache.appfleet.com/example-apache`  file. Then, you need to apply the new spec.

1.Open the `deploy/crds/appfleet.com_v1alpha1_apache_cr.yamlapache.appfleet.com/example-apache` file in a plain-text editor, and specify `spec.replicaCount: 2`.
The updated file should look similar to the following:

```
apiVersion: appfleet.com/v1alpha1
kind: Apache
metadata:
  name: example-apache
spec:
  # Default values copied from <project_dir>/helm-charts/apache/values.yaml

  affinity: {}
  cloneHtdocsFromGit:
    enabled: false
    interval: 60
  git:
    pullPolicy: IfNotPresent
    registry: docker.io
    repository: bitnami/git
    tag: 2.25.0-debian-10-r0
  image:
    debug: false
    pullPolicy: IfNotPresent
    registry: docker.io
    repository: bitnami/apache
    tag: 2.4.41-debian-10-r0
  ingress:
    annotations: {}
    certManager: false
    enabled: false
    hostname: example.local
    secrets: null
    tls:
    - hosts:
      - example.local
      secretName: example.local-tls
  livenessProbe:
    enabled: true
    failureThreshold: 6
    initialDelaySeconds: 180
    path: /
    periodSeconds: 20
    port: http
    successThreshold: 1
    timeoutSeconds: 5
  metrics:
    enabled: false
    image:
      pullPolicy: IfNotPresent
      registry: docker.io
      repository: bitnami/apache-exporter
      tag: 0.7.0-debian-10-r0
    podAnnotations:
      prometheus.io/port: "9117"
      prometheus.io/scrape: "true"
    resources:
      limits: {}
      requests: {}
  nodeSelector: {}
  podAnnotations: {}
  readinessProbe:
    enabled: true
    failureThreshold: 6
    initialDelaySeconds: 30
    path: /
    periodSeconds: 10
    port: http
    successThreshold: 1
    timeoutSeconds: 5
  replicaCount: 2
  resources:
    limits: {}
    requests: {}
  service:
    annotations: {}
    externalTrafficPolicy: Cluster
    httpsPort: 443
    nodePorts:
      http: ""
      https: ""
    port: 80
    type: LoadBalancer
  tolerations: {}


```

2.You can apply the updated spec with:

```
kubectl apply -f deploy/crds/appfleet.com_v1alpha1_apache_cr.yaml
```

```
apache.appfleet.com/example-apache configured
```

3.After applying the updated spec, the state of the cluster differs from the desired state. The Operator starts a new instance of the Apache webserver to reconcile the two, scaling up the cluster:

```
kubectl get pods
```

```
NAME                               READY   STATUS              RESTARTS   AGE
apache-operator-6d5795f879-np6pr   1/1     Running             4          4m10s
example-apache-7cf789fc98-462dr    1/1     Running             0          116s
example-apache-7cf789fc98-fvttm    0/1     ContainerCreating   0          1s
```

Wait a bit until the second container is created:

```
kubectl get pods
```

```
NAME                               READY   STATUS    RESTARTS   AGE
apache-operator-6d5795f879-np6pr   1/1     Running   0          5m2s
example-apache-7cf789fc98-462dr    1/1     Running   0          2m48s
example-apache-7cf789fc98-fvttm    1/1     Running   0          53s
```

In the above output, note that two `example-apache` pods are running.

#Verify Your Installation#

The Apache Operator creates a Kubernetes service which is basically an endpoint where clients can obtain access to the group of pods running Apache. You will use this service to access your cluster using a web browser.

1.Make sure that the service is up with:

```
kubectl get service
```

```
NAME                      TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
apache-operator-metrics   ClusterIP      10.104.41.83   <none>        8686/TCP,8383/TCP            5m46s
example-apache            LoadBalancer   10.99.54.115   <pending>     80:31058/TCP,443:30362/TCP   3m51s
kubernetes                ClusterIP      10.96.0.1      <none>        443/TCP       
```

2.Forward the connections made to http://localhost:80 to the pod running the `service/example-apache` service:

```
kubectl port-forward service/example-apache 80:80
```

```
Forwarding from 127.0.0.1:80 -> 8080
Forwarding from [::1]:80 -> 8080
```

3.Point your browser to http://localhost:80. If everything worked well, you should see something like the following:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/awd2jzgoq1npborxjk6m.png)

Now you should have a good understanding of how Kubernetes Operators work. Furthermore, by completing this tutorial you learned how create an operator for your application using a Helm chart.