Build your Service Mesh: Admission Controller
Tags: kubernetes, go, diy, tutorial
Published: 2024-06-22 20:26:49

## Creating the Admission Controller

Before kube-apiserver persists the object and later be scheduled to a node, 
the Admission Controller can validate and mutate the object.

The Mutation Admission Controller is going to mutate the pods that have the the
annotation `diy-service-mesh: true`.

Let's dig into how the Admission Controller works.


### Admission Controller Flow

![Admission Controller Flow](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0v10pr8ms1sm69n56win.png)

1. kube-controller or kubectl sends a request to the kube-apiserver to create a pod.
1. The kube-apiserver sends the request to the Admission Controller. In this case the proxy-injector.
1. The proxy-injector returns the mutated patch to the kube-apiserver.
1. Kube-apiserver persists the object in the etcd if the object is valid.
1. Kube-scheduler will schedule the pod to a node.
1. Kube-scheduler returns an available node to the kube-apiserver or an error if the pod can't be scheduled.
1. The kube-apiserver will store the object in the etcd with the selected node.
1. The kubelet in the selected node will create the pod in the container runtime.


### Admission Controller Code

Full code of the Admission Controller: [injector](https://github.com/ramonberrutti/diy-service-mesh/blob/main/cmd/injector/main.go)

The mutate function processes the AdmissionReview object and returns the AdmissionResponse object with the mutated patch.

```go
func mutate(ar *admissionv1.AdmissionReview) *admissionv1.AdmissionResponse {
	req := ar.Request
    // Ignore all requests other than pod creation.
	if req.Operation != admissionv1.Create || req.Kind.Kind != "Pod" {
		return &admissionv1.AdmissionResponse{
			UID:     req.UID,
			Allowed: true,
		}
	}

	var pod v1.Pod
    // Unmarshal the raw object to the pod.
	if err := json.Unmarshal(req.Object.Raw, &pod); err != nil {
		return &admissionv1.AdmissionResponse{
			UID: req.UID,
			Result: &metav1.Status{
				Message: err.Error(),
			},
		}
	}

	// Check if the pod contains the inject annotation.
	if v, ok := pod.Annotations["diy-service-mesh/inject"]; !ok || strings.ToLower(v) != "true" {
		return &admissionv1.AdmissionResponse{
			UID:     req.UID,
			Allowed: true,
		}
	}

	// Add the initContainer to the pod.
	pod.Spec.InitContainers = append(pod.Spec.InitContainers, v1.Container{
		Name:            "proxy-init",
		Image:           os.Getenv("IMAGE_TO_DEPLOY_PROXY_INIT"),
		ImagePullPolicy: v1.PullAlways,
		SecurityContext: &v1.SecurityContext{
			Capabilities: &v1.Capabilities{
				Add:  []v1.Capability{"NET_ADMIN", "NET_RAW"},
				Drop: []v1.Capability{"ALL"},
			},
		},
	})

	// Add the sidecar container to the pod.
	pod.Spec.Containers = append(pod.Spec.Containers, v1.Container{
		Name:            "proxy",
		Image:           os.Getenv("IMAGE_TO_DEPLOY_PROXY"),
		ImagePullPolicy: v1.PullAlways,
		SecurityContext: &v1.SecurityContext{
			RunAsUser:    func(i int64) *int64 { return &i }(1337),
			RunAsNonRoot: func(b bool) *bool { return &b }(true),
		},
	})

	patch := []map[string]any{
		{
			"op":    "replace",
			"path":  "/spec/initContainers",
			"value": pod.Spec.InitContainers,
		},
		{
			"op":    "replace",
			"path":  "/spec/containers",
			"value": pod.Spec.Containers,
		},
	}

	podBytes, err := json.Marshal(patch)
	if err != nil {
		return &admissionv1.AdmissionResponse{
			UID: req.UID,
			Result: &metav1.Status{
				Message: err.Error(),
			},
		}
	}

	patchType := admissionv1.PatchTypeJSONPatch
	return &admissionv1.AdmissionResponse{
		UID:     req.UID,
		Allowed: true,
		AuditAnnotations: map[string]string{
			"proxy-injected": "true",
		},
		Patch:     podBytes,
		PatchType: &patchType,
	}
}
```

`IMAGE_TO_DEPLOY_PROXY_INIT` and `IMAGE_TO_DEPLOY_PROXY` are environment variables that `tilt` will update with the last `proxy-init` and `proxy` image respectively.

For complex patch use thi library: https://github.com/evanphx/json-patch 

## Deploying the Admission Controller

`MutatingWebhookConfiguration` tells the kube-apiserver to send the pod creation requests to the injector.

```yaml
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: service-mesh-injector-webhook
webhooks:
- name: service-mesh-injector.service-mesh.svc
  clientConfig:
    service:
      name: service-mesh-injector
      namespace: service-mesh
      path: "/inject"
  rules:
  - operations: ["CREATE"]
    apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["pods"]
  admissionReviewVersions: ["v1"]
  sideEffects: None
  timeoutSeconds: 5
```

Some important points:

- `caBundle` in the `clientConfig` is missing. This is a necessary field because the kube-apiserver only calls the webhook if the certificate is valid.
- Two jobs, `injector-admission-create` `injector-admission-patch` are going generate the certificates and patch the `MutatingWebhookConfiguration` with the `caBundle`.
- The `rules` options allow to filter the objects that are going to be sent to the injector.

The file [injector.yaml](https://github.com/ramonberrutti/diy-service-mesh/blob/main/k8s/injector.yaml) contains the nesesary resources
including the Service Account, Role, RoleBinding, ClusterRole, ClusterRoleBinding, 
Service, Deployment and the Job to generate the certificates.

## Testing the Admission Controller

Let's modify the `http-client` and `http-server` deployments to add the annotation `diy-service-mesh/inject: "true"`.

```yaml
spec:
  replicas: 1
  selector:
    matchLabels:
      app: http-client
  template:
    metadata:
      labels:
        app: http-client
      annotations:
        diy-service-mesh/inject: "true"
    spec:
```

*Important:* the annotation needs to be added to the pod template and not to the deployment.