The developer's guide to Kubernetes Operators
Tags: kubernetes, go
Published: 2024-02-08 22:54:04

<p><a href="https://developers.redhat.com/topics/kubernetes/">Kubernetes</a> Operators are constructed from different parts and components. This cross-referenced guide will list components you need to know to get started developing operators using the <a href="https://operatorframework.io/">Operator Framework</a>. You'll find a handy list of the links used at the end.</p>

> Excerpts in this guide are in Go. Tools used are part of the Operator Framework.

#### TOC
- [What is an Operator?](#what-is-an-operator)
- [What is a Manager?](#what-is-a-manager)
- [What is an API?](#what-is-an-api)
- [What are Subresources?](#what-are-subresources)
- [What is a Controller?](#what-is-a-controller)
- [What are Finalizers?](#what-are-finalizers)
- [What is a Scheme?](#what-is-a-scheme)
- [What are Metrics?](#what-are-metrics)
- [What is a Metrics Server?](#what-is-a-metrics-server)
- [What is a Health Probe?](#what-is-a-health-probe)
- [What is a Webhook Server?](#what-is-a-webhook-server)
- [What about Versioning?](#what-about-versioning)
- [What is Helm?](#what-is-helm)
- [What is OLM?](#what-is-olm)
- [What is Kustomize?](#what-is-kustomize)
- [Quick Links](#quick-links)

## What is an Operator?

<p>An <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/operator/">Operator</a>—aka a Kubernetes-native application—is software running and configured in a Kubernetes-based cluster, adhering to the <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/operator/">Operator Pattern</a>. We can write operators in any language Kubernetes has a <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/operator/#writing-operator">client</a> for. The <a href="https://operatorframework.io/">Operator Framework</a> offers two SDKs: <a href="https://github.com/operator-framework/operator-sdk">Operator SDK</a> for <a href="https://developers.redhat.com/topics/go">Go</a> and <a href="https://github.com/operator-framework/java-operator-sdk">Java Operator SDK</a> for <a href="https://developers.redhat.com/java">Java</a>.</p>

<p>An operator typically describes an <a href="#what-is-an-api">API</a> and configures a <a href="#what-is-a-manager">manager</a> to run <a href="#what-is-a-controller">controllers</a>. Operators are deployed like any other application, using <a href="https://kubernetes.io/docs/concepts/cluster-administration/manage-deployment/">resources</a>. As such, operators can be deployed manually or using <a href="#what-is-helm">Helm</a>. With the Operator Framework, they are often installed using the framework's Operator Lifecycle Manager, <a href="#what-is-olm">OLM</a>.</p>

## What is a manager?

<p>A <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/manager#Manager">manager</a> is used for creating and running our operator. We create a manager using the <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime#NewManager">NewManager</a> utility function. We use the manager's <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/manager#Options">Options</a> for configuring the various aspects of our manager instance, i.e., <a href="#what-is-a-scheme">Scheme</a>, Cache, <a href="#what-are-metrics">Metrics</a>, LeaderElection, <a href="#what-is-a-health-probe">HealthProbe</a>, <a href="#what-is-a-webhook-server">WebhookServer</a>, etc.</p>

```go
kubeConfig := config.GetConfigOrDie()
mgr, err := ctrl.NewManager(kubeConfig, ctrl.Options{
	// manager options go here
})
```

<p>Once created, we use our manager instance to create one or more <a href="#what-is-a-controller">controllers</a>, a <a href="#what-is-a-health-probe">health probe</a>, and a <a href="#what-is-a-webhook-server">webhook server</a> before using our manager's <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/manager#RunnableFunc.Start">Start</a> receiver function to run everything, which marks the start of our operator run.</p>

```go
// we need to include the controllers and all configurations before we start
err := mgr.Start(ctx)
```

## What is an API?

<p>We extend <a href="https://kubernetes.io/docs/concepts/overview/kubernetes-api/">Kubernetes API</a> using <a href="https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/">CustomResourceDefinition</a> resources (CRD). An instance of a CRD type is called a <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/">CustomResource</a> (CR). Resources, in general, are identified using a <a href="https://book.kubebuilder.io/cronjob-tutorial/gvks">GroupVersionKind</a> (GVK). The <a href="#what-about-versioning">version</a> part is reflected in our <a href="https://sdk.operatorframework.io/docs/overview/project-layout/">code layout</a> as part of our API package, and we'll typically have a subpackage-per-version. From a code perspective, a CRD must implement the <a href="https://pkg.go.dev/k8s.io/apimachinery/pkg/runtime#Object">Object</a> interface and have a <code>metadata</code> field. We use <a href="https://pkg.go.dev/k8s.io/apimachinery/pkg/apis/meta/v1#ObjectMeta">ObjectMeta</a> as the <code>metadata</code> field, and <a href="https://pkg.go.dev/k8s.io/apimachinery/pkg/apis/meta/v1#TypeMeta">TypeMeta</a> for providing various functions related to GVK and marshaling, including the required <code>GetObjectKind</code> function (more on the Object interface next).</p>

> Note the marshaling markers used for the serialization of our types.

```go
type OurCustomAPI struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`
}
```

<p>We also need an API List type. Used for encapsulating a list of our type instances when fetching lists. The List type implements the <a href="https://pkg.go.dev/k8s.io/apimachinery/pkg/runtime#Object">Object</a> interface as well. We again use <a href="https://pkg.go.dev/k8s.io/apimachinery/pkg/apis/meta/v1#TypeMeta">TypeMeta</a> for the implementation of <code>GetObjectKind</code>. The <code>metadata</code> field is a <a href="https://pkg.go.dev/k8s.io/apimachinery/pkg/apis/meta/v1#ListMeta">ListMeta</a>. A List type also requires an <code>items</code> field for encapsulating an array of instances.</p>

```go
type OurCustomAPIList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata,omitempty"`
	Items           []OurCustomAPI `json:"items"`
}
```

<p>API types will typically have a <code>spec</code> field for declaratively describing requirements and a <code>status</code> field for <a href="#what-is-a-controller">controllers</a> to report back their operation status.</p>

```go
type OurCustomAPI struct {
	// meta objects removed for brevity
	Spec   OurCustomAPISpec   `json:"spec,omitempty"`
	Status OurCustomAPIStatus `json:"status,omitempty"`
}

type OurCustomAPISpec struct {
	// spec fields go here
}

type OurCustomAPIStatus struct {
	// status fields go here
}
```

<p>Next, we must implement the <code>DeepCopyObject</code> receiver function to implement the <a href="https://pkg.go.dev/k8s.io/apimachinery/pkg/runtime#Object">Object</a> interface fully. This is considered boilerplate and can be generated alongside other useful copy-related functions by including <a href="https://book.kubebuilder.io/reference/markers/object">object generation markers</a> in our code and running the <a href="https://book.kubebuilder.io/reference/controller-gen">controller-gen tool</a> to generate the boilerplate <em>Go</em> code in a file named <code>zz_generated.deepcopy.go</code> right next to our types code.</p>

```go
// +kubebuilder:object:root=true
type OurCustomAPI struct {
	// fields removed for brevity
}

// +kubebuilder:object:root=true
type OurCustomAPIList struct {
	// fields removed for brevity
}
```

<p>We also use <a href="https://book.kubebuilder.io/reference/markers/crd">CRD generation markers</a> and <a href="https://book.kubebuilder.io/reference/markers/crd-validation">CRD validation markers</a> in our code, running the <a href="https://book.kubebuilder.io/reference/controller-gen">controller-gen tool</a> to generate our CRD manifests, which will be later used for deploying our operator in the project's <code>config/crd</code> folder.</p>

```go
// +kubebuilder:object:root=true
// +kubebuilder:resource:scope=Namespaced,shortName=ocapi
type OurCustomAPI struct {
	// fields removed for brevity
	Spec OurCustomAPISpec `json:"spec,omitempty"`
}

type OurCustomAPISpec struct {
	// +kubebuilder:default=1
	// +kubebuilder:validation:Minimum=1
	// +kubebuilder:validation:Maximum=3
	Replicas *int32 `json:"replicas,omitempty"`
}
```

<p><strong>We must regenerate every time our types get any sort of modification.</strong></p>

<p>As described, a CR holds the requirements in its <code>spec</code> and triggers one or more <a href="#what-is-a-controller">controllers</a> that may reflect their operation status asynchronously in its <code>status</code> field. Further endpoints, called <a href="#what-are-subresources">subresources</a>, can be coupled with our custom resource.</p>

## What are subresources?

<p><a href="https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#subresources">Subresources</a> are specific simple resources that can be coupled with other resources. They are used as endpoints on top of enabled resources for performing particular operations without modifying the parent resource. In the context of CRDs, we need to be concerned with two, <code>/scale</code> and <code>/status</code><em>. </em></p>

<p>The <code>/scale</code><strong> </strong>endpoint is used for reading and writing scaling-related data. We must add the following directive to our CRD for configuring data paths to enable scaling. <strong>Note that scaling requires us to implement a <a href="#what-is-a-health-probe">health probe</a> to determine the health status of our operator.</strong></p>

> We should enable the scale subresource if our operator requires scaling.

```yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  ...
spec:
  ...
  versions:
  - name: v1alpha1
    ...
    subresources:
      scale:
        labelSelectorPath: .status.selector
        specReplicasPath: .spec.replicas
        statusReplicasPath: .status.replicas
```

<p>The corresponding code will be:</p>

```go
type OurCustomAPISpec struct {
	Replicas *int32 `json:"replicas,omitempty"`
}

type OurCustomAPIStatus struct {
	Replicas int32    `json:"replicas,omitempty"`
	Selector string `json:"selector,omitempty"`
}
```

<p>Next, the <code>/status</code> subresource is used for decoupling <code>status</code> reported by <a href="#what-is-a-controller">controllers</a> from <code>spec</code> declarations by consumers. We add an empty object to enable it. This will effectively make our <code>OurCustomAPIStatus</code> a subresource.</p>

> We should enable the <code>status</code> subresource if we need to decouple our status from our spec. This is also considered best practice.

```yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  ...
spec:
  ...
  versions:
  - name: v1alpha1
    ...
    subresources:
      status: {}
```

<p>Once we enable the <code>status</code> subresource, <strong>all API requests </strong>for creating/modifying/patching instances of the related CRD <strong>will ignore the </strong><code>status</code><strong> of any object pushed through</strong>. This means that our <code>status</code> field <strong>must be</strong> optional. We need to include the <code>omitempty</code> marshaling property for it. To update the status, we use the <code>/status</code> endpoint, which will update <strong>only</strong> the <code>status</code> part. Programmatically, we can get a <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/client#Client">Client</a> from our <a href="#what-is-a-manager">manager</a>. Clients provide a <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/client#StatusClient.Status">Status</a> API for creating/modifying/patching the <code>status</code> when it's set to be a subresource.</p>

```go
// when the status subresource is enabled,
// this will update only the spec and ignore the status of instanceofOurCustomAPI.
// when the status subresource is disabled,
// this will update both the spec and the status.
mgr.GetClient().Update(ctx, instanceofOurCustomAPI)

// when the status subresource is enabled,
// this will update only the status instanceofOurCustomAPI.
mgr.GetClient().Status().Update(ctx, instanceofOurCustomAPI)
```

<p>Both subresources can be automatically enabled in our generated CRD manifests using the <a href="https://book.kubebuilder.io/reference/generating-crd#scale">scale</a> and <a href="https://book.kubebuilder.io/reference/generating-crd#status">status</a> markers.</p>

```go
// +kubebuilder:subresource:status
// +kubebuilder:subresource:scale:specpath=.spec.replicas,statuspath=.status.replicas,selectorpath=.status.selector
type OurCustomAPI struct {
	// fields removed for brevity
}
```

## What is a controller?

<p>A <a href="https://kubernetes.io/docs/concepts/architecture/controller/">controller</a>, the center of the entire operator, watches the cluster's desired state for changes and attempts to reconcile the current state accordingly. We use the <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime#NewControllerManagedBy">NewControllerManagedBy</a> utility function to create a <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/builder#Builder">Builder</a> using our <a href="#what-is-a-manager">manager</a> instance, configure the <a href="#what-is-an-api">API</a> types triggering our controller, configure the <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler">Reconciler</a> implementation invoked for every triggering event, and build our controller. We can use <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/predicate#Predicate">Predicates</a> to fine-grain the events.</p>

```go
err := ctrl.NewControllerManagedBy(mgr).For(&amp;OurCustomApi{}).Complete(&amp;OurCustomReconciler{})
```

<p>A <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler">Reconciler</a> implementation exposes a <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler">Reconcile</a> function, also known as a reconciliation loop. This function will be invoked for events based on the controller's configuration. The invocation will include the name and namespace of the triggering resource in a <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Request">Request</a> object. For every invocation, we typically fetch the latest version of the resource in question, analyze the declarative <code>Spec</code>, reconcile the underlying application, and update the resource's <code>Status</code>. The reconcile function is expected to return a <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Result">Result</a> indicating whether a requeue of the request is required and its scheduling.</p>

> When turning off requeuing, our reconciler will be invoked again only for the next triggering event.

```go
type OurCustomReconciler struct {}

func (r *OurCustomReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	// reconciliation logic goes here
	return ctrl.Result{Requeue: true, RequeueAfter: time.Minute * 5}, nil
}
```

<p>When running inside a cluster, our operator will need permissions to various resources, such as our custom ones. This is achieved using <a href="https://kubernetes.io/docs/reference/access-authn-authz/rbac/">RBAC</a>. We include <a href="https://book.kubebuilder.io/reference/markers/rbac">RBAC generation markers</a> in our code and run the <a href="https://book.kubebuilder.io/reference/controller-gen">controller-gen tool</a> to generate our <code>Role</code>/<code>ClusterRole</code> manifests, which will be later used for deploying our operator, in the project's <code>config/rbac</code> folder. <strong>We must regenerate as our code evolves and require RBAC modifications</strong>.</p>

> _Bindings_ and _ServiceAccounts_ are not generated.

```go
// +kubebuilder:rbac:groups=group.example.com,resources=ourcustomapis,verbs=get;list;watch;create;update;patch
func (r *OurCustomReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	// removed for brevity
}
```

<p>When reconciling for every event, we also need to consider Delete events. Our operator might require cleanup. For cleanup logic, we use <a href="#what-are-finalizers">finalizers</a>.</p>

## What are finalizers?

<p><a href="https://kubernetes.io/docs/concepts/overview/working-with-objects/finalizers/">Finalizers</a> are a list of keys used as part of a mechanism for cleanup logic. When an object with a finalizer set to it is being deleted, <em>Kubernetes</em> will mark it for deletion by setting its <a href="https://pkg.go.dev/k8s.io/apimachinery/pkg/apis/meta/v1#ObjectMeta.DeletionTimestamp">DeletionTimestamp</a> and block it from deletion until all finalizers are removed. We, the operator's developers, must <strong>add our finalizer </strong>if we need to implement any cleanup logic. We use the <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/controller/controllerutil#AddFinalizer">AddFinalizer</a> utility function:</p>

```go
controllerutil.AddFinalizer(instanceofOurCustomAPI,ourOwnFinalizerString)
err := mgr.GetClient().Update(ctx, instanceofOurCustomAPI)
```

<p>It's also our responsibility to <strong>remove our finalizer</strong> once the cleanup is done so the object can be deleted. We use the <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/controller/controllerutil#RemoveFinalizer">RemoveFinalizer</a> utility function:</p>

```go
controllerutil.RemoveFinalizer(instanceofOurCustomAPI, ourOwnFinalizerString)
err := mgr.GetClient().Update(ctx, instanceofOurCustomAPI)
```

<p>We use the <a href="https://pkg.go.dev/k8s.io/apimachinery/pkg/apis/meta/v1#ObjectMeta.DeletionTimestamp">DeletionTimestamp</a> value and the <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/controller/controllerutil#ContainsFinalizer">ContainsFinalizer</a> utility function to decide if a cleanup is in order.</p>

```go
if !instanceofOurCustomAPI.DeletionTimestamp.IsZero() {
	if controllerutil.ContainsFinalizer(instanceofOurCustomAPI, ourOwnFinalizerString) {
		// cleanup code goes here
    }
}
```

## What is a scheme?

<p>We use a <a href="https://pkg.go.dev/k8s.io/apimachinery/pkg/runtime#Scheme">Scheme</a> reference when creating our <a href="#what-is-a-manager">manager</a> to register the various types we own/work with. A Scheme acts like an object mapper, introducing types to Kubernetes. Preparing a scheme is a two-step process. First, if our code introduces new types, we must create a <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/scheme#Builder">Builder</a> for our GroupVersion and set our types to <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/scheme#Builder.Register">Register</a> with it at runtime. A standard convention is encapsulating the Builder alongside the <a href="#what-is-an-api">API</a>, exposing its <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/scheme#Builder.AddToScheme">AddToScheme</a> receiver function. This decouples the types from whomever requires them.</p>

```go
var (
	groupVersion = schema.GroupVersion{Group: "group.example.com", Version: "v1"}
	schemeBuilder = &amp;scheme.Builder{GroupVersion: groupVersion}
	AddToScheme = schemeBuilder.AddToScheme
)

type OurCustomAPI struct {
	// fields removed for brevity
}

type OurCustomAPIList struct {
	// fields removed for brevity
}

func init() {
	schemeBuilder.Register(&amp;OurCustomAPI{}, &amp;OurCustomAPIList{})
}
```

<p>For the second step, in our operator code, just before we create our <a href="#what-is-a-manager">manager</a>, we need to initiate our scheme and load types onto it using the <code>AddToScheme</code> function mentioned above. We create our initial scheme with the <a href="https://pkg.go.dev/k8s.io/apimachinery/pkg/runtime#NewScheme">NewScheme</a> utility function.</p>

> We use our scheme with the [Scheme](https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/manager#Options.Scheme) option when we configure our [manager](#what_is_a_manager_).

```go
scheme := runtime.NewScheme()
err := ourapipkg.AddToScheme(scheme)
```

## What are metrics?

<p>Kubernetes metrics are <a href="https://prometheus.io/docs/concepts/metric_types/">Prometheus Metrics</a> used for analyzing various aspects of running applications. Every <a href="#what-is-a-manager">manager</a> exposes a <a href="#what-is-a-metrics-server">metrics server</a> by default. This can disabled or fine-grained. We use a preconfigured <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/metrics#Registry">Prometheus Registry</a> for registering metrics created using the <a href="https://pkg.go.dev/github.com/prometheus/client_golang/prometheus">prometheus package</a>. We then use these metrics to report our data that, in turn, will be reflected in the <a href="#what-is-a-metrics-server">metrics server</a>.</p>

```go
var OurCustomCounterMetric = *prometheus.NewCounterVec(prometheus.CounterOpts{
	Name: "our_custom_counter",
	Help: "Count the thingy",
}, []string{"thingy_added_label"})

func init() {
	metrics.Registry.MustRegister(OurCustomCounterMetric)
}
```

<p>Next, we can increment the metric from our code:</p>

```go
ourmetricspkg.OurCustomCounterMetric.WithLabelValues("labelvalue").Inc()
```

<p>It's worth noting that the different components of Kubernetes report <a href="https://kubernetes.io/docs/reference/instrumentation/metrics/">metrics</a> that can help analyze our application.</p>

## What is a metrics server?

<p>A metrics server is an HTTP server exposing textual data for scraping by <a href="https://prometheus.io/">Prometheus</a>. We can fine-grain our server configuration when configuring our <a href="#what-is-a-manager">manager</a> using the metric server's <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/metrics/server#Options">Options</a>.</p>

> We use these options with the [Metrics](https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/manager#Options.Metrics) field when we configure our [manager](#what_is_a_manager_).

```go
options := server.Options{BindAddress: "127.0.0.1:8080"}
```

<p>By default, this server has no authentication layer. This can be mitigated with the commonly used <a href="https://github.com/brancz/kube-rbac-proxy">kube-rbac-proxy</a> as a proxy layer enforcing authentication using TLS certificates or Tokens.</p>

## What is a health probe?

<p>Every <a href="#what-is-a-manager">manager</a> can enable a HealthProbe, which means serving endpoints for checking the health of the underlying application, our <a href="#what-is-an-operator">operator</a>. HealthProbe exposes two endpoints: <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/manager#Options.ReadinessEndpointName">readiness</a> and <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/manager#Options.LivenessEndpointName.">liveliness</a>. These are configured using our manager's receiver functions, <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/manager#Manager.AddHealthzCheck">AddHealthzCheck</a> and <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/manager#Manager.AddReadyzCheck">AddReadyzCheck</a>. Both take a <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/healthz#Checker">Checker</a>, a function that takes an HTTP <a href="https://pkg.go.dev/net/http#Request">Request</a> and returns an Error. We can use the pre-built Ping checker if a custom health check is not required. Failure tolerations can be fine-grained per endpoint when designing the <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/">deployment</a>.</p>

> Note that HealthProbe is mandatory for scaling.

```go
err := mgr.AddHealthzCheck("healthz", healthz.Ping)
err = mgr.AddReadyzCheck("readyz", healthz.Ping)
```

## What is a webhook server?

<p>Our <a href="#what-is-a-manager">manager</a> can expose a Webhook Server. An HTTP server serving endpoints for <a href="https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/">Dynamic Admission Controllers</a>, AKA Admission Webhooks. These controllers are used for verifying and mutating resources before applying them to the system. There are two types of Admission Webhooks: <a href="https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#validatingadmissionwebhook">Validating Admission Webhook</a> and <a href="https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#mutatingadmissionwebhook">Mutating Admission Webhook</a>. Kubernetes will invoke these webhooks for any admission of a CR. First, all the Validating Webhooks configured for the CRD will be invoked. They are expected to <strong>verify the validity</strong> of the admitted CR. Next, all the Mutating Webhooks configured for the CRD will be invoked. They are expected to <strong>mutate</strong> the CR, i.e., add labels, annotations, etc. Only after successfully invoking all webhooks, the CR will be applied to the system.</p>

<p>There are two approaches to implementing Admission Webhooks. The first one is by adding receiver functions on top of our <a href="#what-is-an-api">API</a> type to make it implement either or both the <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/webhook/admission#Validator">Validator</a> and the <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/webhook/admission#Defaulter">Defaulter</a> interfaces, essentially making our API type handle the logic for validation and mutation, respectively.</p>

```go
type OurCustomAPI struct {
	// fields removed for brevity
}

func (a *OurCustomAPI) Default() {
	// mutating logic goes here
}

func (a *OurCustomAPI) ValidateCreate() (warnings Warnings, err error) {
	// create validation logic goes here
	return nil, nil
}

func (a *OurCustomAPI) ValidateUpdate(old runtime.Object) (warnings Warnings, err error) {
	// update validation logic goes here
	return nil, nil
}

func (a *OurCustomAPI) ValidateDelete() (warnings Warnings, err error) {
	// delete validation logic goes here
	return nil, nil
}
```

<p>We then use the <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime#NewWebhookManagedBy">NewWebhookManagedBy</a> utility function to create a <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/builder#WebhookBuilder">WebhookBuilder</a> with our configured <a href="#what-is-a-manager">manager</a>, introduce our <a href="#what-is-an-api">API</a>, and build our webhook.</p>

```go
err := ctrl.NewWebhookManagedBy(mgr).For(&amp;OurCustomApi{}).Complete()
```

<p>As for the second implementation approach, If we prefer decoupling our webhooks from our types, we can use the <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/webhook/admission#CustomValidator">CustomValidator</a> and <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/webhook/admission#CustomDefaulter">CustomDefaulter</a> interfaces, supported by the <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/builder#WebhookBuilder.WithValidator">WithValidator</a> and <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/builder#WebhookBuilder.WithDefaulter">WithDefaulter</a> builder steps, respectively.</p>

```go
type OurCustomWebhook struct {}

func (w *OurCustomWebhook) Default(ctx context.Context, obj runtime.Object) error {
	// mutating logic goes here
	return nil
}

func (w *OurCustomWebhook) ValidateCreate(ctx context.Context, obj runtime.Object) (warnings Warnings, err error) {
	// create validation logic goes here
	return nil, nil
}

func (w *OurCustomWebhook) ValidateUpdate(ctx context.Context, oldObj, newObj runtime.Object) (warnings Warnings, err error) {
	// update validation logic goes here
	return nil, nil
}

func (w *OurCustomWebhook) ValidateDelete(ctx context.Context, obj runtime.Object) (warnings Warnings, err error) {
	// delete validation logic goes here
	return nil, nil
}
```

<p>And create our Webhook:</p>

```go
ourWebhook := &amp;OurCustomWebhook{}
err := ctrl.NewWebhookManagedBy(mgr).For(&amp;OurCustomApi{}).WithValidator(ourWebhook).WithDefaulter(ourWebhook).Complete()
```

<p>Next, we need to tell Kubernetes about our webhooks. This is achieved using the <a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#validatingwebhookconfiguration-v1-admissionregistration-k8s-io">ValidatingWebhookConfiguration</a> and <a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#mutatingwebhookconfiguration-v1-admissionregistration-k8s-io">MutatingWebhookConfiguration</a> APIs. We can include <a href="https://book.kubebuilder.io/reference/markers/webhook">webhook generation markers</a> in our code and run the <a href="https://book.kubebuilder.io/reference/controller-gen">controller-gen tool</a> in the project's folder to generate these manifests, which will be later used for deploying our operator, in the project's <code>config/webhook</code> folder. <strong>We must regenerate as our code evolves and API versions get bumped or modified.</strong></p>

```go
// +kubebuilder:webhook:verbs=create;update;delete,path=/validate-group-example-com-v1beta1-ourcustomapi,mutating=false,failurePolicy=fail,groups=group.example.com,resources=ourcustomapis,versions=v1beta1,name=ourcustomapi.group.example.com,sideEffects=None,admissionReviewVersions=v1
// +kubebuilder:webhook:verbs=create;update;delete,path=/mutate-group-example-com-v1beta1-ourcustomapi,mutating=true,failurePolicy=fail,groups=group.example.com,resources=ourcustomapis,versions=v1beta1,name=ourcustomapi.group.example.com,sideEffects=None,admissionReviewVersions=v1
type OurCustomWebhook struct {}
```

## What about versioning?

<p>We differentiate our operator versioning from <a href="https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/">CRD versioning</a>. For instance, our operator version 1.2.3 can support API <code>v1beta1</code> and <code>V1</code> while deprecating <code>v1alpha2</code> and not supporting <code>v1alpha1</code>. Each CRD version has three properties relating to versioning. These can be fine-grained in our generated CRD manifests using <a href="https://book.kubebuilder.io/reference/markers/crd">CRD generation markers</a>.</p>

<ul>
	<li><strong>served</strong> indicates whether this version is enabled.</li>
	<li><strong>storage</strong> indicates the version to be used for persisting data with etcd. A mandatory one and only one version can be marked for storage.</li>
	<li><strong>deprecated</strong> indicates whether this version is being deprecated.</li>
</ul>

```yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  ...
spec:
  ...
  versions:
  - name: v1alpha1
    ...
    served: false
    storage: false
  - name: v1alpha2
    ...
    served: true
    storage: false
    deprecated: true
  - name: v1beta1
    ...
    served: true
    storage: true
  - name: v1
    schema:
      ...
    served: true
    storage: false
```

## What is Helm?

<p><a href="https://helm.sh/">Helm</a> is used for packaging and managing Kubernetes applications. The package format is called <a href="https://helm.sh/docs/topics/charts/">charts</a>. The most notable files in a chart are the <a href="https://helm.sh/docs/topics/charts/#the-chartyaml-file">Chart.yaml file</a> containing metadata about the chart itself, the <code>values.yaml</code> containing default values for our application, and the <code>templates</code> folder containing template YAML files and helpers for constructing our application manifests files used for deploying.</p>

<p>Another part of the Helm project is the <a href="https://helm.sh/docs/intro/install/">Helm</a> CLI (command-line interface), which is used for installing, upgrading, uninstalling, and managing charts on clusters. A chart installed on a cluster is called a <a href="https://helm.sh/docs/glossary/#release">Release</a>. We use <a href="https://helm.sh/docs/intro/using_helm/">repositories</a> for sharing charts. We can install a chart from a repository using the Helm CLI. We can also override the various values used for configuring the underlying application from the command line.</p>

<p>Helm doesn't require any operator or application installed on the target cluster. The connection between the release's components is annotation and discovery-based.</p>

## What is OLM?

<p>The <a href="https://olm.operatorframework.io/">Operator Lifecycle Manager</a> is used for managing our operator's lifecycle. OLM encapsulates operators as Managed Services, providing over-the-air updates using catalogs, a dependency model, discoverability, scalability, stability, and much more.</p>

<p>We first need to discuss the <a href="https://olm.operatorframework.io/docs/concepts/crds/clusterserviceversion/">ClusterServiceVersion</a> (CSV) CRs to understand OLM. The OLM operator handles these. We use a CSV to describe our operator's deployments, requirements, metadata, supported install strategies, etc. We can use the <a href="https://sdk.operatorframework.io/docs/cli/operator-sdk/">operator-sdk CLI tool</a> and <a href="https://sdk.operatorframework.io/docs/building-operators/golang/references/markers/">API markers</a> to generate our CSV as a <a href="#what-is-kustomize">Kustomize</a> base. By convention, we generate it in the project's <code>config/manifests</code> folder.<strong> We must regenerate the CSV as our code evolves and requires such modifications</strong>.</p>

```go
//+operator-sdk:csv:customresourcedefinitions:displayName="Our Operator API"
type OurCustomAPI struct {
	// fields removed for brevity
	Spec OurCustomAPISpec `json:"spec,omitempty"`
}

type OurCustomAPISpec struct {
	//+operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Number of replicas",xDescriptors={"urn:alm:descriptor:com.tectonic.ui:podCount"}
    Replicas int `json:"replicas"`
}
```

<p>The next OLM component we'll cover is the <a href="https://olm.operatorframework.io/docs/tasks/creating-operator-bundle/">Bundle</a>. After generating our CSV, we'll include <a href="https://olm.operatorframework.io/docs/tasks/creating-operator-manifests/#packaging-additional-objects-alongside-an-operator">additional supported resources</a> in the <code>config</code> folder based on the <a href="https://sdk.operatorframework.io/docs/overview/project-layout/#common-base">common structure</a>, use <a href="#what-is-kustomize">kustomize</a>, and pipe the result into the <a href="https://sdk.operatorframework.io/docs/cli/operator-sdk/">operator-sdk cli tool</a> to generate our Bundle manifests. We use a bundle per operator <a href="#what-about-versioning">version</a>. Bundles are container images built from scratch, adhering to a specific folder layout:</p>

<ul>
	<li>The<code>/manifests</code> folder contains our operator's CSV mentioned above and other <a href="https://olm.operatorframework.io/docs/tasks/creating-operator-manifests/#packaging-additional-objects-alongside-an-operator">supported resources</a> required for our operator to run, i.e., CRDs, RBAC, etc.</li>
	<li>The <code>/metadata</code> folder contains a file called <code>annotations.yaml</code> with all the annotations used for configuring our Bundle, i.e. package and channel data. We can optionally include dependencies.yaml file to leverage OLM's <a href="https://olm.operatorframework.io/docs/concepts/olm-architecture/dependency-resolution/">dependency resolution</a> mechanism.</li>
	<li>The (optional) <code>/tests/scorecard</code> folder contains a <code>config.yaml</code> file, used for <a href="https://sdk.operatorframework.io/docs/testing-operators/scorecard/">statically validating and testing</a> our operator.</li>
</ul>

<p>After building and pushing our Bundle image to an accessible registry, we can potentially install or upgrade the Bundle directly from the image using the <a href="https://sdk.operatorframework.io/docs/cli/operator-sdk/">operator-sdk cli tool</a>. But the next logical step would be to include our Bundle in a <a href="https://olm.operatorframework.io/docs/tasks/creating-a-catalog/">Catalog</a>. The Catalog is a container image built from <a href="https://quay.io/repository/operator-framework/opm">OPM's Image</a> serving the Catalogs configured in its <code>/configs</code> folder. The configs are YAML files describing multiple schemas. The <code>olm.package</code> schema describes a package. The <code>olm.bundle</code> schema describes bundles for a package. Each Bundle represents a specific <a href="#what-abut-versioning">version</a> of our operator and targets a specific Bundle image. Next, the <code>olm.channel</code> schema is used for associating package bundles to a channel, i.e., selecting bundles for the stable and alpha channels. <a href="https://olm.operatorframework.io/docs/reference/catalog-templates/#examp">See example</a>. We use the <a href="https://github.com/operator-framework/operator-registry">opm cli tool</a> to generate the various components.</p>

> One of the more known open source catalogs is [OperatorHub.io](https://operatorhub.io/)

<p>After building and pushing our Catalog image to an accessible registry, consumers can create a <a href="https://olm.operatorframework.io/docs/concepts/crds/catalogsource/">CatalogSource</a> targeting our image and install packages from our Catalog. Consumers create <a href="https://olm.operatorframework.io/docs/concepts/crds/operatorgroup/">OperatorGroups</a>, setting the allowed permission for installed operators. And <a href="https://olm.operatorframework.io/docs/concepts/crds/subscription/">Subscriptions</a> for requesting a package to be installed by OLM. This will create an <a href="https://olm.operatorframework.io/docs/concepts/crds/installplan/">InstallPlan</a> describing the installation of the package. Once approved, manually or automatically, the underlying operator's Bundle selected for the target Channel of the desired Package will be installed based on its CSV.</p>

## What is Kustomize?

<p><a href="https://kustomize.io/">Kustomize</a> helps create and manage resource files template-free by merging overlays on top of bases and introducing various built-in and customizable <a href="https://kubectl.docs.kubernetes.io/references/kustomize/builtins/">transformers and generators</a>. We use a <a href="https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/">kustomize.yaml</a> file to declaratively orchestrate our manifests construction and the <a href="https://kubectl.docs.kubernetes.io/installation/kustomize/">kustomize cli tool</a> to build our manifests. Throughout this guide, we noticed that the generated file target is usually <code>config/X</code> in our project's root, we typically design our kustomization on top of the <code>config</code> folder.</p>

## Quick links

<ul>
	<li><a href="https://sdk.operatorframework.io/docs/overview/project-layout/">Project Common Layout</a></li>
	<li><a href="https://kubernetes.io/docs/concepts/extend-kubernetes/operator/">Kubernetes Operator Pattern</a></li>
	<li><a href="https://operatorframework.io/">Operator Framework</a> / <a href="https://sdk.operatorframework.io/">Operator SDK (Go)</a> / <a href="https://javaoperatorsdk.io/">Java Operator SDK</a></li>
	<li><a href="https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions">API Conventions</a></li>
	<li><a href="https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/">Admissions Controllers</a></li>
	<li><a href="https://prometheus.io/docs/concepts/metric_types/">Prometheus Metric Types</a></li>
	<li><a href="https://kubernetes.io/docs/reference/instrumentation/metrics/">Kubernetes Components Metrics List</a></li>
	<li><a href="https://olm.operatorframework.io/">Operator Lifecycle Manager (OLM)</a></li>
	<li><a href="https://olm.operatorframework.io/docs/tasks/creating-operator-manifests/#packaging-additional-objects-alongside-an-operator">Additional Resources in Bundle Manifests</a></li>
	<li><a href="https://olm.operatorframework.io/docs/concepts/olm-architecture/dependency-resolution/">OLM Dependency Resolution</a></li>
	<li><a href="https://operatorhub.io/">Operator Hub</a></li>
</ul>

### Development tools

<ul>
	<li><a href="https://book.kubebuilder.io/reference/controller-gen">controller-gen</a> generates CRD, RBAC, and Webhook manifests, as well as DeepCopy boilerplate functions.</li>
	<li><a href="https://sdk.operatorframework.io/docs/cli/operator-sdk/">operator-sdk</a> generates CSV and Bundles for OLM.</li>
	<li><a href="https://github.com/operator-framework/operator-registry">opm</a> composes Catalogs for OLM.</li>
	<li><a href="https://kubectl.docs.kubernetes.io/installation/kustomize/">kustomize</a> constructs&nbsp;Kubernetes manifests using overlays, transformers, and generators.</li>
	<li><a href="https://helm.sh/docs/intro/install/">helm</a>&nbsp;deployes and manages&nbsp;Helm Chart Releases.</li>
</ul>

### Development toolbox

<ul>
	<li><a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/client/config#GetConfigOrDie">GetConfigOrDie</a> is used for fetching&nbsp;the configuration for accessing the running server.</li>
	<li><a href="NewScheme">NewScheme</a> is used for creating&nbsp;a new scheme for introducing types to Kubernetes.</li>
	<li><a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime#NewManager">NewManager</a> is used for creating&nbsp;a new manager for creating controllers and running the operator.</li>
	<li><a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime#NewControllerManagedBy">NewControllerManagedBy</a> is used for configuring a new controller with a manager.</li>
	<li><a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime#NewWebhookManagedBy">NewWebhookManagedBy</a> is used for configuring a new admission webhook with a manager.</li>
	<li><a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/healthz#Checker">Ping Health Checker</a> is a useful health probe checker.</li>
	<li><a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/metrics#Registry">Prometheus Registry</a> Kubernetes' pre-configured Prometheus registry.</li>
	<li><a href="https://pkg.go.dev/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured#Unstructured">Unstructured Type </a>provides&nbsp;the Unstructured type.</li>
</ul>

### Useful packages

<ul>
	<li><a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/controller/controllerutil">Controller Utils Package</a>&nbsp;provides various utils for working with controllers.</li>
	<li><a href="https://pkg.go.dev/k8s.io/apimachinery/pkg/api/meta">Meta Package</a> contains valuable functions for working with objects meta.</li>
	<li><a href="https://pkg.go.dev/github.com/prometheus/client_golang/prometheus">Prometheus Package</a> for creating metrics.</li>
	<li><a href="https://pkg.go.dev/k8s.io/apimachinery/pkg/api/errors">Errors Package</a> provides useful utility functions for working with errors.</li>
	<li><a href="https://pkg.go.dev/k8s.io/component-base/cli">CLI Package</a> provides utility functions for creating <a href="https://github.com/spf13/cobra">Cobra</a> commands (common approach for creating operators).</li>
	<li><a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/log/zap">Zap Package</a> provides utility functions for createing loggers.</li>
	<li><a href="https://pkg.go.dev/k8s.io/apimachinery/pkg/util/wait">Wait Package</a> provides tools for listening for Conditions.</li>
	<li><a href="https://pkg.go.dev/k8s.io/apimachinery/pkg/util/json">JSON Package</a> provides utility function for marsharling and unmarshaling JSON.</li>
	<li><a href="https://pkg.go.dev/k8s.io/apimachinery/pkg/util/rand">Rand Package</a> provides tools related to randomization.</li>
	<li><a href="https://pkg.go.dev/k8s.io/apimachinery/pkg/util/version">Version Package</a> provides utility function for inspecting versions.</li>
	<li><a href="https://pkg.go.dev/k8s.io/apimachinery/pkg/util/yaml">Yaml Package</a>&nbsp;provides utility function for marsharling and unmarshaling Yaml.</li>
</ul>

### Common interfaces

<ul>
	<li><a href="https://pkg.go.dev/k8s.io/apimachinery/pkg/runtime#Object">Runtime Object</a></li>
	<li><a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler">Controller Reconciler</a></li>
	<li><a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/webhook/admission#Defaulter">Admission Defaulter</a> / <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/webhook/admission#CustomDefaulter">Custom Defaulter</a></li>
	<li><a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/webhook/admission#Validator">Admission Validator</a> / <a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/webhook/admission#CustomValidator">Custom Validator</a></li>
	<li><a href="https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/predicate#Predicate">Predicates</a></li>
</ul>

### Code markers

<ul>
	<li><a href="https://book.kubebuilder.io/reference/markers/object">Object Generation Markers</a></li>
	<li><a href="https://book.kubebuilder.io/reference/markers/crd">CRD Generation Markers</a></li>
	<li><a href="https://book.kubebuilder.io/reference/markers/crd-validation">CRD Validation Markers</a></li>
	<li><a href="https://book.kubebuilder.io/reference/generating-crd#subresources">CRD Subresources Generation Markers</a></li>
	<li><a href="https://book.kubebuilder.io/reference/markers/rbac">RBAC Generation Markers</a></li>
	<li><a href="https://book.kubebuilder.io/reference/markers/webhook">Webhook Generation Markers</a></li>
	<li><a href="https://sdk.operatorframework.io/docs/building-operators/golang/references/markers/">CSV API Markers</a></li>
</ul>