How to make exclusive locks in Kubernetes
Tags: kubernetes
Published: 2022-10-04 19:08:05

---
title: How to make exclusive locks in Kubernetes
published: true
date: 2022-10-01 16:58:02 UTC
tags: kubernetes
series: Klock
canonical_url: https://rnemet.dev/posts/projects/klock_exclusive_lock/
---

Klock: Exclusive Lock Primer

There is an application running in a Kubernetes cluster. Goal is to protect this application from any modifications,
except if those modifications are coming from predefined actor.

## Setup Requirements

To start let's set up the Kind cluster and [Klock](https://github.com/robert-nemet/klock). I assume you already have installed [Kind](https://kind.sigs.k8s.io/).
Now let's create a cluster and install all requirements:

1. Create cluster: `kind create cluster`
2. Install cert-manager: `kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.2/cert-manager.yaml`
3. Install Klock :

```shell
helm repo add rnemet https://rnemet.dev/helm-charts
helm repo update
helm install klock rnemet/klock
```

## Setup Scenario

My application has one Pod and one ConfigMap. By default the [Klock](https://github.com/robert-nemet/klock) supports locking Deployments, Pods, Secrets, and ConfigMaps. So, I'm covered there.

Let's deploy our application, one Pod and one ConfigMap:

```shell
kubectl run my-pod --image nginx --labels aura=red
kubectl create configmap my-cm --from-literal nikola=tesla
kubectl label cm my-cm aura=red
```

Add one more Pod:

```shell
kubectl run other-pod --image nginx
```

Check what we have in _default_ namespace:

```shell
❯ kubectl get pods,cm
NAME            READY   STATUS    RESTARTS   AGE
pod/my-pod      1/1     Running   0          5m52s
pod/other-pod   1/1     Running   0          112s

NAME                         DATA   AGE
configmap/kube-root-ca.crt   1      18m
configmap/my-cm              1      4m20s
```

While our application is:

```shell
❯ kubectl get pod,cm -A --selector aura=red
NAMESPACE   NAME         READY   STATUS    RESTARTS   AGE
default     pod/my-pod   1/1     Running   0          7m2s

NAMESPACE   NAME              DATA   AGE
default     configmap/my-cm   1      5m30s
```

Let's add one service account(SA) to be used for our so-called _operator_ to manage our application:

```shell
kubectl create serviceaccount jonny-op
```

If you try to list pods with __jonny-op__:

```shell
❯ kubectl run operator  --image roffe/kubectl --overrides='{"apiVersion":"v1","spec":{"serviceAccount":"jonny-op"}}' -it --restart Never -- /bin/sh -c "kubectl -n default get pods"
No resources found.
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:default:jonny-op" cannot list resource "pods" in API group "" in the namespace "default"
pod default/operator terminated (Error)
```

Here I'm trying to list all pods in default namespace with our fake operator. As operator run with SA __jonny-op__ it can not list pods. What about our application:

```shell
kubectl delete pod operator
❯ kubectl run operator  --image roffe/kubectl --overrides='{"apiVersion":"v1","spec":{"serviceAccount":"jonny-op"}}' -it --restart Never -- /bin/sh -c "kubectl get pod,cm  --selector aura=red"
No resources found.
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:default:jonny-op" cannot list resource "pods" in API group "" in the namespace "default"
Error from server (Forbidden): configmaps is forbidden: User "system:serviceaccount:default:jonny-op" cannot list resource "configmaps" in API group "" in the namespace "default"
pod default/operator terminated (Error)
```

Doh... Let's fix this by creating Role and RoleBinding:

```shell
kubectl create role app-op --verb list --verb get --verb create --verb update --verb delete --verb patch --resource pods --resource cm
kubectl create rolebinding jonny-app --role app-op --serviceaccount default:jonny-op
```

And check:

```shell
❯ kubectl delete pod operator
pod "operator" deleted
❯ kubectl run operator  --image roffe/kubectl --overrides='{"apiVersion":"v1","spec":{"serviceAccount":"jonny-op"}}' -it --restart Never -- /bin/sh -c "kubectl get pod,cm  --selector aura=red"
If you don't see a command prompt, try pressing enter.
NAME         READY     STATUS    RESTARTS   AGE
pod/my-pod   1/1       Running   0          26m

NAME              DATA      AGE
configmap/my-cm   1         25m
```

__Notice__: you need to delete _operator_ pod each time because it can not be updated. That I simulate operator with a Pod.

## Run Scenario

Me, as external actor can modify both Pod __my-pod__ and ConfigMap __my-cm__. Well, I created the cluster and I'm an admin in this case so that is natural.
As shown, we can do the same in the _default_ namespace running a workload with SA __jonny-op__, or operator. But my goal is just and only __jonny_op__
can do it. And only for my application.

All resources in my application are grouped with the label `aura=red`. So, let's create a lock:

```shell
kubectl apply -f - <<EOF
apiVersion: klock.rnemet.dev/v1
kind: Lock
metadata:
  name: lockred
spec:
  operations:
    - UPDATE
    - DELETE
  matcher:
    aura: red
  exclusive:
    name: jonny-op
EOF
```

Now let's try to update Pod __my-pod__:

```shell
❯ kubectl label pod my-pod aura=blue --overwrite
Error from server (denied, there is a lock: map[aura:red]): admission webhook "klocks.rnemet.dev" denied the request: denied, there is a lock: map[aura:red]
```

Ok... What about Pod __other-pod__:

```shell
❯ kubectl label pod other-pod aura=blue --overwrite
pod/other-pod labeled
```

So, I as admin can label(UPDATE) Pod that is not labeled with `aura:red`. What about SA __jonny-op__:

```shell
❯ kubectl run operator  --image roffe/kubectl --overrides='{"apiVersion":"v1","spec":{"serviceAccount":"jonny-op"}}' -it --restart Never -- /bin/sh -c "kubectl label pod my-pod here=was-jonny"
pod/my-pod labeled
❯ k get pods --show-labels
NAME        READY   STATUS      RESTARTS   AGE   LABELS
my-pod      1/1     Running     0          91m   aura=red,here=was-jonny
operator    0/1     Completed   0          25s   run=operator
other-pod   1/1     Running     0          87m   aura=blue,run=other-pod
```

What about ConfigMap __my-cm__?

```shell
❯ kubectl label cm my-cm admin=was-here
Error from server (denied, there is a lock: map[aura:red]): admission webhook "klocks.rnemet.dev" denied the request: denied, there is a lock: map[aura:red]
❯ k delete pods operator
pod "operator" deleted
❯ kubectl run operator  --image roffe/kubectl --overrides='{"apiVersion":"v1","spec":{"serviceAccount":"jonny-op"}}' -it --restart Never -- /bin/sh -c "kubectl label cm my-cm here=was-jonny"
configmap/my-cm labeled
❯ kubectl get cm --show-labels
NAME               DATA   AGE    LABELS
kube-root-ca.crt   1      108m   <none>
my-cm              1      94m    aura=red,here=was-jonny
```

Same. So, I managed to protect my application from other actors in the cluster, while SA __jonny-op__ can operate with it.

## Conclusion

Even if this is more manual example, it shows that is possible to make exclusive lock in Kubernetes with Klock.