Writing a very basic kubernetes mutating admission webhook
Tags: programming, go, kubernetes, tutorial
Published: 2019-09-19 06:41:26

---
title: Writing a very basic kubernetes mutating admission webhook
published: true
tags: programming,go,kubernetes,tutorial
canonical_url: 
---

My findings when attempting to write a (very) simple kubernetes mutating admission webhook.

![](https://cdn-images-1.medium.com/max/1024/1*BkllrXSkhpMTulJIk1b0aw.jpeg)

#### Okay.. so, webhooks…?

Admission webhooks help you do some really cool stuff, there are two kinds of webhooks, _validating_ and _mutating_. We will concentrate on the mutating admission webhook in this post.

Mutating admission webhooks allow you to “_modify”_ a (e.g.) Pod (or any kubernetes resource) request. E.g. you can modify a Pod to use a particular scheduler, add / inject sidecar containers (think LinkerD sidecar), or even reject it if it doesn’t meet some security requirements, etc. etc. — all **without** having to write a full fledged “micro” service to do this. The webhook can live _anywhere,_ in practice, k8s just needs to know where _anywhere_ is.

### Setup

The setup is easy, but important, all you really need to make sure is that the MutatingAdminssionController is enabled in the k8s api-server. To check if your k8s cluster has this enabled, you can use

```
kubectl api-versions | grep admissionregistration
```

For development, I can recommend using Kubernetes-In-Docker (KinD), all you need is Docker and [KinD](https://kind.sigs.k8s.io/docs/user/quick-start/). KinD doesn’t auto-enable these, you can use this KinD configuration (kind.yaml)

```
---
kind: Cluster
apiVersion: kind.sigs.k8s.io/v1alpha3
kubeadmConfigPatches:
- |
 apiVersion: kubeadm.k8s.io/v1beta2
 kind: ClusterConfiguration
 metadata:
 name: config
 apiServer:
 extraArgs:
 "enable-admission-plugins": "NamespaceLifecycle,LimitRanger,ServiceAccount,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,PersistentVolumeClaimResize,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota"
nodes:
- role: control-plane
```

Spin up the KinD cluster with

```
kind create cluster --conifg kind.yaml
```

I trust you’ll be able to configure kubectl etc. to now use this cluster going forward.

This is it, the setup is done. Well done! :)

### Deployment

I’d like to start here, as this is the easiest part of this _“tutorial”_ (if you will). Deploying an admission webhook is in practice the same as deploying any other service onto a k8s cluster.

All we need is

- a Service
- a Deployment
- a MutatingWebhookConfiguration

The first two are simple, so I won’t go into those deeper, you can see what I’ve used on [Github](https://github.com/alex-leonhardt/k8s-mutate-webhook/blob/master/deploy/webhook.yaml).

Before you _just go ahead and deploy this to prod_ ... 

Please do think about a couple _what-if_s before using webhooks in a production environment:

- what if the request to your webhook fails (conn drop, conn reset, no dns, etc.)
- what if the deployment failed and the pod is stuck in a crash-loop
- what if the webhook has become mission critical and **_must be functional_** 100% of the time
- what if you create a circular dependency (my favourites!)
- what if you have a 3214 node cluster with many 10s of thousand resources & requests all depending on this webhook, which is scaled to 1 pod ;)

> Note: Webhooks are **only** called over SSL/TLS so your webhook **must** have a valid signed certificate (we do this further down the line)

The **MutatingWebhookConfiguration** is where we tell k8s which resource requests should be sent to our webhook. The configuration consists of the following properties:

- `apiVersion (at the time it is: admissionregistration.k8s.io/v1beta1)`
- `kind       (must be: MutatingWebhookConfiguration)`
- `metadata   (the usual: name, annotations, labels)`
- `webhooks   (a list of type webhook)`

The webhook (_type_) consists of these properties:

- `name`
- `clientConfig`
- `    - caBundle (we will get this from the k8s cluster itself)`
- `    - service to send the AdmissionReview requests to`
- `rules ( a list of rules that define which resource operations should be matched, these rules make sure that k8s resource requests are sent to your webhook )`
- `namespaceSelector (the usual: matchLabels: {“label_name”: “label_value”}`
- `…`

there are many more that could be used, for a simple non-production webhook to play with, the above will suffice.

The full list of properties can be seen [here](https://godoc.org/k8s.io/api/admissionregistration/v1beta1#MutatingWebhook).

A rule consists of the following:

- `operations  (a list of [operations](https://godoc.org/k8s.io/api/admissionregistration/v1beta1#OperationType) to match, in our case ["CREATE"])`
- `apiGroups   (in our case, empty [""])`
- `apiVersions (in our case, this is ["v1"])`
- `resources   (in our case, this is ["pods"])`

`apiGroups`, `apiVersions` and `resources` are all (kind of) dependent on each other, in this example it’s quite easy as `Pod` is part of the core api group so it doesn’t need specifying, the empty `[""]` is _matching_ the core api group.

Here is an example:

```
---
apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
 name: mutateme
 labels:
 app: mutateme
webhooks:
 - name: mutateme.default.svc.cluster.local
 clientConfig:
 caBundle: ${CA_BUNDLE}
 service:
 name: mutateme
 namespace: default
 path: "/mutate"
 rules:
 - operations: ["CREATE"]
 apiGroups: [""]
 apiVersions: ["v1"]
 resources: ["pods"]
 namespaceSelector:
 matchLabels:
 mutateme: enabled
```

the `${CA_BUNDLE}` above refers to the actual CA bundle retrieved from the k8s API, replace it with your own; you can get your cluster’s CA bundle with

```
kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}'
```

As webhooks can only be called over HTTPS (SSL/TLS) you will need to generate a new ssl key & certificate for it. Doing this is somewhat involved, so probably best to take a look [here](https://github.com/alex-leonhardt/k8s-mutate-webhook/blob/master/ssl/ssl.sh):

{% gist https://gist.github.com/alex-leonhardt/65c95394c0c1faab97c3ec60dd0b004f %}

and/or

- [https://github.com/morvencao/kube-mutating-webhook-tutorial/blob/master/deployment/webhook-create-signed-cert.sh](https://github.com/morvencao/kube-mutating-webhook-tutorial/blob/master/deployment/webhook-create-signed-cert.sh)

### The webhook

Pasting the entire code here is probably not very useful, so I’ll concentrate on the essentials that helped me understand a few things.

Conceptually, what a webhook has to do is relatively easy, it receives a `AdmissionReview`, and responds with an `AdmissionReview` :) — most blog posts I’ve found concentrate on the Request and Response objects only, which can be a bit confusing.

In essence, this is what needs to happen ..

- the K8S api server will send a AdmissionReview _“request”_ and expects a AdmissionReview _“response”_ ; this can get confusing, but i.e. it is something like this

```
{
 "kind": "AdmissionReview",
 "apiVersion": "admission.k8s.io/v1beta1",
 "request": {...}
}
```

- the AdmissionReview consists of AdmissionRequest and AdmissionResponse objects
- the webhook needs to _“unmarshal”_ the AdmissionReview from JSON format into some kind of object so it can read the AdmissionRequest and modify the AdmissionResponse object within it
- the webhook i.e. creates its own AdmissionResponse object, copies the UID from the AdmissionRequest object and replaces the AdmissionResponse object within the AdmissionReview with its own (overwrites it)
- responds with a AdmissionReview object in JSON

```
{
 "kind": "AdmissionReview",
 "apiVersion": "admission.k8s.io/v1beta1",
 "request":  { ... ORIGINAL REQUEST ... },
 "response": { ... OUR RESPONSE ... }
}
```

#### Where to start?

We will start with a basic web server, that supports SSL/TLS, and can read and respond in JSON format.

In practice, you can use whatever programming language you’d like for this, I have used Go, but this can easily be done in Python or any other compiled or interpreted language. Ideally though, use a language that already has K8S libraries so you don’t have to create our own objects/types; Go (naturally) has these, but there are also at least the Python libraries you could use.

[Here](https://github.com/alex-leonhardt/k8s-mutate-webhook/blob/master/cmd/main.go)’s what I did to accept requests on port 8443 with SSL/TLS set up to use a key & cert.

{% gist https://gist.github.com/alex-leonhardt/edd6fd402a81f9fee5944b8452b8f4d3 %}

The above example creates a server s with 2 handlers, one for / and one for /mutate which is the endpoint that will be called by k8s (which is what we specified in the `MutatingWebhookConfiguration`); it listens on port `:8443` and we use the ListenAndServeTLS method to serve requests over SSL/TLS.

I split up the logic of http request/response from the actual processing of the AdmissionReview request as I find it’ll be easier to test the function/s independently; so the `/mutate` handler really only does

- take the JSON http request and read in the BODY
- send the BODY to the Mutate function in the mutate package (m) — the unmarshalling from JSON into the appropriate object structure, modification and marshalling back to JSON is done here
- respond with a JSON message, either one that describes an Error or a AdmissionReview including our AdmissionResponse

So far so good, the potentially more challenging part is next.

#### pkg/mutate/mutate.go

The mutate package does the actual processing of AdmissionReview requests by …

- unmarshalling the received JSON payload into a AdmissionReview object
- using the AdmissionRequest object to decide what we should do
- creating the JSONPatch
- creating a new AdmissionResponse
- updating the AdmissionReview with our new AdmissionResponse
- marshal the final AdmissionReview back into JSON and return it

**API Resources, Types, etc.**

Something that I’m struggling the most with is finding the correct repository, version and even file/s that I need to do things with k8s or its supported resources.

Here’s what I’ve found out so far ..

- `Pod` and other core resource types (which are used to unmarshal JSON into) can be found in [kubernetes/api/blob/master/core/v1/types.go](https://github.com/kubernetes/api/blob/master/core/v1/types.go)
- `Metadata` objects (which usually are part of main object like Pod), seem to be kept in the `apimachinery` repo, it doesn’t have a description so 🤷🏻‍♂️[apimachinery/blob/master/pkg/apis/meta/v1/types.go](https://github.com/kubernetes/apimachinery/blob/master/pkg/apis/meta/v1/types.go)
- Other resources and their definitions seem to reside in kubernetes/api/{extensions,node,autoscaling,..}; the admission types are located here: [kubernetes/api/blob/master/admission/v1beta1/types.go](https://github.com/kubernetes/api/blob/master/admission/v1beta1/types.go) (for now)

However, that’s not how they’re imported (at least not in Go); for our example, we need these imports:

```
import (
 v1beta1 "k8s.io/api/admission/v1beta1"
 corev1  "k8s.io/api/core/v1"
 metav1  "k8s.io/apimachinery/pkg/apis/meta/v1"
)
```

IIRC they must be `k8s.io/` imports and **not** `github.com/` imports, even though the repos are hosted on Github.

**Marshal & Unmarshal**

These terms always get me confused, but I’ll try to explain

- marshal: when you convert a object into a JSON (or other, e.g. protobuf) equivalent representation
- unmarshal: when you convert JSON (or other) into your own object’s representation

For the webhook we need to

- unmarshal the JSON AdmissionReview into a Go AdmissionReview object, I called it `ar`
- check if there’s a _embedded raw object_ and if so, again, unmarshal it into the object type that we expect, in our case a `Pod`, which I called simply pod; if that fails, we should not try to continue and return an error
- marshal the `JSONPatch` map into a valid JSONPatch, which, needs to be valid JSON
- marshal the **final** (Go) AdmissionReview object back into a JSON AdmissionReview

this is why we needed to find those imports earlier, so we can create objects of the correct types and unmarshal JSON into them or marshal them into JSON.

The types we use/need are

- `AdmissionReview (v1beta1.AdmissionReview)`
- `AdmissionRequest (v1beta1.AdmissionRequest)`
- `Pod (corev1.Pod)`
- `PatchTypeJSONPatch (v1beta1.PatchTypeJSONPatch)`
- `AdmissionResponse (v1beta1.AdmissionResponse)`
- `Status (metav1.Status)` to set the `AdmissionResponse.Result`

**Why do we create a JSONPatch?**

This is because the mutating webhook **does not modify the k8s resource**, but responds with a _**patch**_, telling k8s **_how_** to modify the object for us. I found this rather counter intuitive, since we’re creating a _"Mutating Admission Webhook"_ but don’t actually _mutate_ anything.

More about JSONPatch expressions and how they work can be found [here](http://jsonpatch.com/), but essentially they consist of an operation (`op`), a `path` and a `value`, e.g.:

```
{
 "op": "replace",
 "path": "/spec/containers/0/image",
 "value": "debian"
}
```

In our case, it instructs how and what operations it should apply to the resource that was requested. You can have more than one operation/instruction, e.g. a list of operations `[{"op": ..},{"op": ..},{"op": ..}]` and I believe they’ll be executed in the same order. Something for you to try and find out ;).

I used the following to create a JSONPatch list for the webhook, it replaces any container image to use the Debian docker image instead of the originally requested .. _FWIW - it's a list as a Pod may have more than 1 container_.

```
// resp is the AdmissionReview.Response 
p := []map[string]string{}

for i := range pod.Spec.Containers {
  patch := map[string]string{
    "op": "replace",
    "path": fmt.Sprintf("/spec/containers/%d/image", i), 
    "value": "debian",
  }
  p = append(p, patch)
}

// parse the []map into JSON
resp.Patch, err = json.Marshal(p)
```

This is it!

You can find the all the code on Github, I’d encourage you to try it without looking first :) … except for the details on the SSL ca, key, cert signing. I’ve also added links to resource that I believe will be helpful to understand what’s going on.

Kudos to the IBM-Cloud blog on Medium (link below) that I used as inspiration for this post and also as part reference when I got stuck. It’s more complete but also more complicated (at least at the time it seemed like it); if you're intending to create a proper version of this, you should check it out.

[alex-leonhardt/k8s-mutate-webhook](https://github.com/alex-leonhardt/k8s-mutate-webhook)

> Tip: Use a IDE / Editor that can help with code completion, dependencies, etc. — I used VSCode, but there are also good Vim plugins.


### Resources

- [https://godoc.org/k8s.io/api/admissionregistration/v1beta1#MutatingWebhook](https://godoc.org/k8s.io/api/admissionregistration/v1beta1#MutatingWebhook)
- [https://godoc.org/k8s.io/api/admission/v1beta1](https://godoc.org/k8s.io/api/admission/v1beta1)
- [https://godoc.org/k8s.io/kubernetes/pkg/apis/admission](https://godoc.org/k8s.io/kubernetes/pkg/apis/admission)
- [https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.15/#mutatingwebhookconfiguration-v1beta1-admissionregistration-k8s-io](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.15/#mutatingwebhookconfiguration-v1beta1-admissionregistration-k8s-io)
- [https://github.com/morvencao/kube-mutating-webhook-tutorial](https://github.com/morvencao/kube-mutating-webhook-tutorial)

#### Kudos

[https://medium.com/ibm-cloud/diving-into-kubernetes-mutatingadmissionwebhook-6ef3c5695f74](https://medium.com/ibm-cloud/diving-into-kubernetes-mutatingadmissionwebhook-6ef3c5695f74)

* * *