Stop Messing with Kubernetes Finalizers
Tags: kubernetes, devops, python, programming
Published: 2022-06-01 20:25:50

We've all been there - it's frustrating seeing deletion of Kubernetes resource getting stuck, hang or take a very long time. You might have _"solved"_ this using the terrible advice of removing finalizers or running `kubectl delete ... --force --grace-period=0` to force immediate deletion. 99% of the time this is a horrible idea and in this article I will show you why.

## Finalizers

Before we get into why force-deletion is a bad idea, we first need to talk about _finalizers_.

Finalizers are values in resource metadata that signal required pre-delete operations - they tell resource controller what operations need to be performed before object is deleted.

The most common one would be:

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
  finalizers:
  - kubernetes.io/pvc-protection
...
```

Their purpose is to stop a resource from being deleted, while controller or Kubernetes Operator cleanly and gracefully cleans-up any dependant objects such as underlying storage devices.

When you delete an object which has a finalizer, `deletionTimestamp` is added to resource metadata making the object read-only. Only exception to the read-only rule, is that finalizers can be removed. Once all finalizers are gone, the object is queued to be deleted.

It's important to understand that finalizers are just items/keys in resource metadata. Finalizers don't specify the code to execute. They have to be added/removed by the resource controller.

Also, don't confuse finalizers with _Owner References_. `.metadata.OwnerReferences` field specify parent/child relations between objects such as _Deployment_ -> _ReplicaSet_ -> _Pod_. When you delete an object such as Deployment a whole tree of child objects can be deleted. This process (deletion) is automatic, unlike with finalizers, where controller needs to take some action and remove the finalizer field.

## What Could Go Wrong?

As mentioned earlier, the most common finalizer you might encounter is the one attached to _Persistent Volume (PV)_ or _Persistent Volume Claim (PVC)_. This finalizer protects the storage from being deleted while it's in use by a Pod. Therefore, if the PV or PVC doesn't want to delete, it most likely means that it's still mounted by a Pod. If you decide force-delete PV, be aware that backing storage in Cloud or any other infrastructure might not get deleted, therefore you might leave a dangling resource, which still costs you money.
  
Another example is a _Namespace_ which can get stuck in `Terminating` state because resources still exist in the namespace that the namespace controller is unable to remove. Forcing deletion of namespace can leave dangling resources in your cluster which include for example [Cloud provider's load balancer](https://github.com/kubernetes/kubernetes/issues/60807#issuecomment-658875036) which might be very hard to track down later.

While not necessarily related to finalizers, it's good to mention that resources can get stuck for many other reasons other than waiting for finalizers:

The simplest example would be Pod being stuck in `Terminating` state, which usually signals issue with Node on which the Pod runs. _"Solving"_ this with `kubectl delete pod --grace-period=0 --force ...` will remove the Pod from API server (`etcd`), but it might still be running on the Node, which is definitely not desirable.

Another example would be a _StatefulSet_, where Pod force-deletion can create problems because Pods have fixed identities (`pod-0`,`pod-1`). A distributed system might depend on these names/identities - if the Pod is force-deleted, but still runs on the node, you can end-up with 2 pods with same identity when StatefulSet controller replaces the original _"deleted"_ Pod. These 2 Pods might then attempt to access same storage, which can lead to corrupted data. More on this in [docs](https://kubernetes.io/docs/tasks/run-application/force-delete-stateful-set-pod).

## Finalizers in The Wild

We now know that we shouldn't mess with resources that have finalizers attacked to them, but which resources are these?

The 3 most common ones you will encounter in _"vanilla"_ Kubernetes are `kubernetes.io/pv-protection` and `kubernetes.io/pvc-protection` related to _Persistent Volumes_ and _Persistent Volume Claims_ respectively (plus [couple more](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolume-deletion-protection-finalizer) introduced in v1.23) as well as `kubernetes` finalizer present on _Namespaces_. The last one however isn't in `.metadata.finalizers` field but rather in `.spec.finalizers` - this special case is described in [architecture document](https://github.com/kubernetes/design-proposals-archive/blob/main/architecture/namespaces.md#finalizers).

Besides these _"vanilla"_ finalizers, you might encounter many more if you install Kubernetes Operators which often perform pre-deletion logic on their custom resources. A quick search through code of some popular projects turn up the following:

- [Istio](https://github.com/istio/istio/blob/master/operator/pkg/controller/istiocontrolplane/istiocontrolplane_controller.go#L65) - `istio-finalizer.install.istio.io`
- [Cert Manager](https://github.com/cert-manager/cert-manager/blob/ee8ec69fadff165afa96c2dd22264c16fdb7d065/internal/apis/acme/v1beta1/const.go#L20) - `finalizer.acme.cert-manager.io`
- [Strimzi (Kafka)](https://github.com/strimzi/strimzi-kafka-operator/commit/69e77ce8d5918c25048a253f91f4bca8e89028d9#diff-0f711d9ed233c37fbe749fd6c4aadce73849f48de3c414d86d9af89d51ea5ef7R317) - `service.kubernetes.io/load-balancer-cleanup`
- [Quay](https://github.com/quay/quay-operator/pull/405/files#diff-db06dd075ea792819f15dcbfb9c2376eea2e17832c2bd64ae6b381d3c947b57eR56) - `quay-operator/finalizer`
- [Ceph/Rook](https://github.com/rook/rook/blob/master/Documentation/ceph-teardown.md#removing-the-cluster-crd-finalizer) - `ceph.rook.io/disaster-protection`
- [ArgoCD](https://github.com/argoproj-labs/argocd-operator/pull/247/files#diff-1078fd6d90631dae21aebe2e5cb7b8f2e559f568d61b8277117dd19344462d47R188) - `argoproj.io/finalizer`
- [Litmus Chaos](https://github.com/litmuschaos/chaos-operator/blob/master/pkg/controller/chaosengine/chaosengine_controller.go#L57) - `chaosengine.litmuschaos.io/finalizer`

If you want to find all the finalizers that are present in your cluster, then you will have to run the following command against each resource type:

```bash
kubectl get some-resource -o custom-columns=Kind:.kind,Name:.metadata.name,Finalizers:.metadata.finalizers
```

You can use `kubectl api-resources` to get a list resources types available in your cluster. 

Regardless of which finalizer is stopping deletion of your resources, the negative effects of force-deleting those resources will be generally the same, which is something being left behind, be it storage, load balancer, or a simple pod.

Also, the proper solution will be generally the same, which is finding the finalizer that's stopping the deletion, figuring out what is its purpose - possibly by looking at the source code of the controller/operator - and resolving whatever is blocking the controller from removing the finalizer.

If you decide to force-delete the problematic resources anyway, then the solution would be:

```bash
kubectl patch some-resource/some-name \
    --type json \
    --patch='[ { "op": "remove", "path": "/metadata/finalizers" } ]'
```

One exception would be _Namespace_, which has `finalize` API method which is usually called when all resources in said Namespace are cleaned-up. If the Namespace refuses to delete even when there are no resources left to delete, then you can call the method yourself:

```bash
cat <<EOF | curl -X PUT \
  localhost:12345/api/v1/namespaces/my-namespace/finalize \
  -H "Content-Type: application/json" \
  --data-binary @-
{
  "kind": "Namespace",
  "apiVersion": "v1",
  "metadata": {
    "name": "my-namespace"
  },
  "spec": {
    "finalizers": null,
  }
}
EOF
```

## Building Your Own

Now that we know what they are and how they work, it should be clear that they're quite useful, so let's see how we can apply them to our own resources and workloads.

Kubernetes ecosystem is based around Go, but for simplicity's sake I will use Python here. If you're not familiar with Python Kubernetes client library, consider reading my previous article first - _[Automate All the Boring Kubernetes Operations with Python](https://martinheinz.dev/blog/73)_.

Before we start using finalizers, we first need to create some resource in a cluster - in this case a Deployment:

```python
# initialize the client library...

deployment_name = "my-deploy"
ns = "default"

v1 = client.AppsV1Api(api_client)

deployment_manifest = client.V1Deployment(
    api_version="apps/v1",
    kind="Deployment",
    metadata=client.V1ObjectMeta(name=deployment_name),
    spec=client.V1DeploymentSpec(
        replicas=3,
        selector=client.V1LabelSelector(match_labels={
            "app": "nginx"
        }),
        template=client.V1PodTemplateSpec(
            metadata=client.V1ObjectMeta(labels={"app": "nginx"}),
            spec=client.V1PodSpec(
                containers=[client.V1Container(name="nginx",
                                               image="nginx:1.21.6",
                                               ports=[client.V1ContainerPort(container_port=80)]
                                               )]))))

response = v1.create_namespaced_deployment(body=deployment_manifest, namespace=ns)
```

The above code creates a sample Deployment called `my-deploy`, at this time without any finalizer. To add a couple finalizers we will use following patch:

```python
finalizers = ["test/finalizer1", "test/finalizer2"]

v1.patch_namespaced_deployment(deployment_name, ns, {"metadata": {"finalizers": finalizers}})

while True:
    try:
        response = v1.read_namespaced_deployment_status(name=deployment_name, namespace=ns)
        if response.status.available_replicas != 3:
            print("Waiting for Deployment to become ready...")
            time.sleep(5)
        else:
            break
    except ApiException as e:
        print(f"Exception when calling AppsV1Api -> read_namespaced_deployment_status: {e}\n")
```

The important part here is a call to `patch_namespaced_deployment` which sets `.metadata.finalizers` to a list of finalizers we defined. Each of these must be _fully qualified_, meaning they must contain `/` as they need to adhere to `DNS-1123` specification. Ideally, to make them more understandable you should use format like `kubernetes.io/pvc-protection`, where you prefix it with hostname of your service which is related to the controller responsible for the finalizer.

Rest of the code in the above snippet simply makes sure that the replicas of the Deployment are available after which we can proceed with managing the finalizers:

```python
from kubernetes import client, watch

def finalize(deployment, namespace, finalizer):
    print(f"Do some pre-deletion task related to the {finalizer} present in {namespace}/{deployment}")
    ...

v1 = client.AppsV1Api(api_client)
w = watch.Watch()
for deploy in w.stream(partial(v1.list_namespaced_deployment, namespace=ns)):
    print(f"Deploy - Message: Event type: {deploy['type']}, Deployment {deploy['object']['metadata']['name']} was changed.")
    if deploy['type'] == "MODIFIED" and "deletionTimestamp" in deploy['object']['metadata']:

        fins = deploy['object']['metadata']['finalizers']
        f = fins[0]
        finalize(deploy['object']['metadata']['name'], ns, f)
        new_fins = list(set(fins) - {f})
        body = [{
            "kind": "Deployment",
            "apiVersion": "apps/v1",
            "metadata": {
                "name": deploy['object']['metadata']['name'],
            },
            "op": "replace",
            "path": f"/metadata/finalizers",
            "value": new_fins
        }]
        resp = v1.patch_namespaced_deployment(name=deploy['object']['metadata']['name'],
                                              namespace=ns,
                                              body=body,
                                              field_manager="json")
    elif deploy['type'] == "DELETED":
        print(f"{deploy['object']['metadata']['name']} successfully deleted.")
print("Finished namespace stream.")
```

The general sequence here is as follows:

We start by watching the desired resource - in this case a Deployment - for any changes/events. We then look for the events that relate to modifications to the resource and we specifically check whether `deletionTimestamp` is present. If it's there, we grab list of finalizers from resource's metadata and start processing first of them. We first perform all necessary pre-deletion tasks with `finalize` function, after which we apply patch to the resource with original list of finalizers minus the one we processed.

If the patch in Python looks complicated to you, then just know that it's an equivalent to the following `kubectl` command:

```bash
kubectl patch deployment/my-deploy \
  --type json \
  --patch='[ { "op": "replace", "path": "/metadata/finalizers", "value": [test/finalizer1] } ]'
```

If the patch is accepted, we will receive another modification event at which point we will process another finalizer. We repeat that until all finalizers are gone. At that point resource gets automatically deleted.

Be aware that you might receive the events more than once, therefore it's important to make the pre-deletion logic idempotent.

If you run the above code snippets and then execute `kubectl delete deployment my-deploy`, then you should see logs like:

```bash
# Finalizers added to Deployment
Deploy - Message: Event type: ADDED, Deployment my-deploy was changed.
# "kubectl delete" gets executed, "deletionTimestamp" is added
Deploy - Message: Event type: MODIFIED, Deployment my-deploy was changed.
# First finalizer is removed...
Do some pre-deletion task related to the test/finalizer1 present in default/my-deploy
# Another "MODIFIED" event comes in, Second finalizer is removed...
Deploy - Message: Event type: MODIFIED, Deployment my-deploy was changed.
Do some pre-deletion task related to the test/finalizer2 present in default/my-deploy
# Finalizers are gone "DELETED" event comes - Deployment is gone.
Deploy - Message: Event type: DELETED, Deployment my-deploy was changed.
my-deploy successfully deleted.
```

The above demonstration using Python works, but isn't exactly robust. In real-world scenario you'd most likely want to use Operator framework either through [`kopf`](https://kopf.readthedocs.io/en/stable/) in case of Python, or more usually with _Kubebuilder_ for Go. Kubebuilder docs also includes [whole page](https://book.kubebuilder.io/reference/using-finalizers.html) on how to use finalizers, including sample code.

If you don't want to implement whole Kubernetes Operator, you can also choose to build _Mutating Webhook_ which is described in [Dynamic Admission Control](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/) docs. The process there would be the same - receive the event, process your business logic and remove the finalizer.

## Conclusion

One think you should take away from this article is that you might want to think twice before using `--force --grace-period=0` or removing finalizers from resources. There might be situations when it's OK to ignore finalizer, but for your own sake, investigate before using the nuclear solution and be aware of possible consequences as doing so might hide a systemic problem in your cluster.