Detecting Kubernetes API Deprecations with pluto
Tags: kubernetes, api
Published: 2022-07-27 13:22:24

The Kubernetes API is changing all the time. With these changes come deprecations and eventual removals of parts of the API. To be able to keep an up-to-date Kubernetes cluster version, we have to identify deprecated APIs and update them. This may become tedious in larger clusters with hundreds of resources but tools like pluto can help.

## What does API deprecation mean in Kubernetes?

The operations governing the lifecycle of all Kubernetes resources are provided via RESTful API endpoints by the [Kubernetes API server](https://kubernetes.io/docs/reference/glossary/?all=true#term-kube-apiserver). In other words, the Kubernetes API is the frontend of the [Kubernetes control plane](https://kubernetes.io/docs/reference/glossary/?all=true#term-control-plane). 

![Kubernetes API components](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2uxxonyleah514n9934q.png)

Resource APIs are associated with URIs like

* `/apis/GROUP/VERSION/*` for **cluster-scoped** resources, and 
* `/apis/GROUP/VERSION/namespaces/NAMESPACE/*` for **namespace-scoped** resources.

Some resources are also grouped into the **core group** (or, **legacy group**). Those are available via the special API endpoint `/api/{version}` (see [*API groups*](https://kubernetes.io/docs/reference/using-api/#api-groups)). Pods, for example, are part of the core group. A request to list all pods in a given namespace `my-namespace` is mapped to the following HTTP GET request.

> GET /api/v1/namespaces/my-namespace/pods

(See the [API documentation for the Pod resource](https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/) for reference.)

## The problem with Kubernetes API deprecations

Kubernetes specifies a [deprecation policy](https://kubernetes.io/docs/reference/using-api/deprecation-policy/) that defines what it means if parts of an API become **deprecated**. Essentially, deprecation means that the associated endpoints of the Kubernetes API server are flagged for removal and subsequently deleted. Since the API server governs the resource lifecycle, using a resource with a removed API version, will prevent the deployment of that resource. Consequently, if we fail to update our resource API versions, we will either be stuck with an outdated Kubernetes version; or, updating to the new Kubernetes version will prevent deployments of certain resources. Both are undesirable states since we will either

1. continue using an unstable Kubernetes version, or
2. our Kubernetes deployment will be incomplete.

### Deploying resources with removed API versions

To get a clearer picture, let's have a look at the second problem and see how Kubernetes responds if we try to deploy a resource using a removed API version. To do this, we spin up a local Kubernetes cluster with [`k3d`](https://k3d.io/)

```shell
$ k3d cluster create
INFO[0000] Prep: Network
INFO[0004] Created network 'k3d-k3s-default'
INFO[0004] Created image volume k3d-k3s-default-images
INFO[0004] Starting new tools node...
INFO[0005] Pulling image 'ghcr.io/k3d-io/k3d-tools:5.4.3'
INFO[0007] Starting Node 'k3d-k3s-default-tools'
INFO[0007] Creating node 'k3d-k3s-default-server-0'
INFO[0009] Pulling image 'docker.io/rancher/k3s:v1.23.6-k3s1'
INFO[0027] Creating LoadBalancer 'k3d-k3s-default-serverlb'
INFO[0028] Pulling image 'ghcr.io/k3d-io/k3d-proxy:5.4.3'
INFO[0030] Using the k3d-tools node to gather environment information
INFO[0030] Starting new tools node...
INFO[0030] Starting Node 'k3d-k3s-default-tools'
INFO[0033] Starting cluster 'k3s-default'
INFO[0033] Starting servers...
INFO[0033] Starting Node 'k3d-k3s-default-server-0'
INFO[0040] All agents already running.
INFO[0040] Starting helpers...
INFO[0040] Starting Node 'k3d-k3s-default-serverlb'
INFO[0049] Injecting records for hostAliases (incl. host.k3d.internal) and for 3 network members into CoreDNS configmap...
INFO[0051] Cluster 'k3s-default' created successfully!
INFO[0051] You can now use it like this:
kubectl cluster-info
```

Then switch Kubernetes context e.g. using [`kubectx`](https://github.com/ahmetb/kubectx)

```shell
$ kubectx k3d-k3s-default
Switched to context "k3d-k3s-default".
```

Now, we take a look at which version the Kubernetes API server is running by executing `kubectl version`

```shell
$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.2", GitCommit:"f66044f4361b9f1f96f0053dd46cb7dce5e990a8", GitTreeState:"clean", BuildDate:"2022-06-15T14:22:29Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"darwin/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.6+k3s1", GitCommit:"418c3fa858b69b12b9cefbcff0526f666a6236b9", GitTreeState:"clean", BuildDate:"2022-04-28T22:16:18Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
```

As we can see from the _Server Version:_ output, our k3d cluster is running Kubernetes v1.23. 



By taking a look at the [API deprecation guide](https://kubernetes.io/docs/reference/using-api/deprecation-guide/#v1-22) we can see that two versions of the `Ingress` API, **extensions/v1beta1** and **networking.k8s.io/v1beta1**, were removed in v1.22 of Kubernetes. So let's try to deploy an `Ingress` resource with that API version and see what happens. Below we have an example manifest that I shamelessly stole from the [_The Ingress Resource_](https://kubernetes.io/docs/concepts/services-networking/ingress/#the-ingress-resource) section of the official Kubernetes documentation and put it in a file `ingress-pre-1.22.yaml`.

```yaml
# ingress-pre-1.22.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: minimal-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx-example
  rules:
  - http:
      paths:
      - path: /testpath
        pathType: Prefix
        backend:
          service:
            name: test
            port:
              number: 80
```

And then let's try to deploy it to our v1.23 Kubernetes cluster.

```shell
$ kubectl apply -f ingress-pre-1.22.yaml
error: resource mapping not found for name: "minimal-ingress" namespace: "" from "ingress-pre-1.22.yaml": no matches for kind "Ingress" in version "networking.k8s.io/v1beta1"
ensure CRDs are installed first
```

As we can see, the API returns an error indicating that `networking.k8s.io/v1beta1` does not include an `Ingress` type anymore. If we change the API version to _networking.k8s.io/v1_, however,

```
# ingress-1.22.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: minimal-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx-example
  rules:
  - http:
      paths:
      - path: /testpath
        pathType: Prefix
        backend:
          service:
            name: test
            port:
              number: 80
```
 our `Ingress` will be created without a hitch. 
```shell
$ kubectl apply -f ingress-1.22.yaml
ingress.networking.k8s.io/minimal-ingress created
```

## Detecting API deprecations with pluto

In a more realistic scenario, we already have resources deployed to our cluster and want to keep their API versions up to date so that we can update our cluster version safely. 



The question therefore is, how do we spot resources with deprecated and soon-to-be-removed API versions? One answer to that question is to check the deprecation guide mentioned earlier and check which resource API versions will be removed in the upcoming Kubernetes update. It is important to note, however, that if we skip several versions, we will have to repeat this check for all versions in between our current and target Kubernetes versions as well.



In large clusters with dozens of resource types and versions, this can become tedious and error-prone. Luckily, there are tools like [pluto](https://github.com/FairwindsOps/pluto) by *FairwindOps* which assist us in spotting deprecated and soon-to-be-removed resource API versions.



Let's for example deploy our `Ingress` resource to a Kubernetes cluster with an API server version earlier than v1.22 (e.g. v1.19). Creating a k3d cluster with a non-current Kubernetes API version is possible by passing the `--image` option to k3d specifying a k3s image for our desired Kubernetes version (e.g. [v1.19.16-k3s1](https://hub.docker.com/layers/k3s/rancher/k3s/v1.19.16-k3s1/images/sha256-14a70ad7fe4402c6639a679522417f3477f6b968f56a71bcec6a07510c0c02a3?context=explore), full list of images is available on [Dockerhub](https://hub.docker.com/r/rancher/k3s/tags)):

```shell
$ k3d cluster delete
INFO[0000] Deleting cluster 'k3s-default'
INFO[0002] Deleting cluster network 'k3d-k3s-default'
INFO[0005] Deleting 2 attached volumes...
WARN[0005] Failed to delete volume 'k3d-k3s-default-images' of cluster 'k3s-default': failed to find volume 'k3d-k3s-default-images': Error: No such volume: k3d-k3s-default-images -> Try to delete it manually
INFO[0005] Removing cluster details from default kubeconfig...
INFO[0005] Removing standalone kubeconfig file (if there is one)...
INFO[0005] Successfully deleted cluster k3s-default!
$ k3d cluster create --image rancher/k3s:v1.19.16-k3s1
INFO[0000] Prep: Network
INFO[0003] Created network 'k3d-k3s-default'
INFO[0003] Created image volume k3d-k3s-default-images
INFO[0003] Starting new tools node...
INFO[0003] Starting Node 'k3d-k3s-default-tools'
INFO[0006] Creating node 'k3d-k3s-default-server-0'
INFO[0009] Pulling image 'rancher/k3s:v1.19.16-k3s1'
INFO[0018] Creating LoadBalancer 'k3d-k3s-default-serverlb'
INFO[0018] Using the k3d-tools node to gather environment information
INFO[0018] Starting new tools node...
INFO[0018] Starting Node 'k3d-k3s-default-tools'
INFO[0021] Starting cluster 'k3s-default'
INFO[0021] Starting servers...
INFO[0021] Starting Node 'k3d-k3s-default-server-0'
INFO[0028] All agents already running.
INFO[0028] Starting helpers...
INFO[0028] Starting Node 'k3d-k3s-default-serverlb'
INFO[0037] Injecting records for hostAliases (incl. host.k3d.internal) and for 3 network members into CoreDNS configmap...
INFO[0039] Cluster 'k3s-default' created successfully!
INFO[0039] You can now use it like this:
kubectl cluster-info
```
We again confirm that the API server is running Kubernetes v1.19 with `kubectl version`

```shell
$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.2", GitCommit:"f66044f4361b9f1f96f0053dd46cb7dce5e990a8", GitTreeState:"clean", BuildDate:"2022-06-15T14:22:29Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"darwin/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.16+k3s1", GitCommit:"da16869555775cf17d4d97ffaf8a13b70bc738c2", GitTreeState:"clean", BuildDate:"2021-11-04T00:55:24Z", GoVersion:"go1.15.14", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.24) and server (1.19) exceeds the supported minor version skew of +/-1
```

Now, let's apply our `Ingress` with the manifest specifying the deprecated API version again.

```shell
$ kubectl apply -f ingress-pre-1.22.yaml
Warning: networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
ingress.networking.k8s.io/minimal-ingress created
```

As we can see, we already get a helpful deprecation warning that recommends using `networking.k8s.io/v1` instead of `networking.k8s.io/v1beta1`. For now, we assume, however, that our resources already have been deployed to a running cluster and we want to detect those resources (without applying them again).



This is where [tools like `pluto`](https://pluto.docs.fairwinds.com/installation/) become handy to detect the use of deprecated resource API version.

Pluto is available on multiple platforms and via a variety of package managers. We'll use [binenv](https://github.com/devops-works/binenv) here.

```shell
$ binenv install pluto
2022-07-21T19:47:18+02:00 WRN version for "pluto" not specified; using "5.8.0"
fetching pluto version 5.8.0 100% |█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| (11/11 MB, 9.578 MB/s)
2022-07-21T19:47:22+02:00 INF "pluto" (5.8.0) installed
```

There is a couple of ways we can hand over resource manifests to pluto in order to detect deprecated API versions: for example via directory scan or by direct input. Additionally, there is a convenient integration with Helm that checks our releases for deprecations. 


### Directory scan

Using a **directory scan** requires that we know where to find the Kubernetes manifests for our cluster. In our case, this is pretty simple since we only have two simple manifests and they are both in the current directory. The following command makes pluto run a directory scan and detect our deprecated api version. 

```shell
$ pluto detect-files --directory . --target-versions k8s=v1.22.0
NAME              KIND      VERSION                     REPLACEMENT            REMOVED   DEPRECATED
minimal-ingress   Ingress   networking.k8s.io/v1beta1   networking.k8s.io/v1   true      true
```

Note that we can specify our target version(s) using the `--target-versions` option. If we were to target Kubernetes v1.15.0 instead, pluto would return the empty list, because our CRD API version was deprecated only later on in v1.16.0.

```shell
$ pluto detect-files --directory . --target-versions k8s=v1.15.0
No output to display
```

### Direct input

Using **direct input**, we can just pipe a resource manifest directly into pluto. This is particularly useful if we want to scan the resources already deployed to the cluster (and going over the manifests is too complicated). Since our CRD is already deployed we can obtain the resource manifest with `kubectl get` and hand it to pluto.

```shell
$ kubectl get ingress minimal-ingress -o yaml | pluto detect -
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
NAME              KIND      VERSION              REPLACEMENT            REMOVED   DEPRECATED
minimal-ingress   Ingress   extensions/v1beta1   networking.k8s.io/v1   true      true
```

> :information_source: You may have noticed that pluto reports the deprecated API version as `extensions/v1beta1` instead of `networking.k8s.io/v1beta1`. This is the case because the API server in v1.19. apparently normalises `networking.k8s.io/v1beta1` to `extensions/v1beta1`. The manifest of the deployed `Ingress` that is returned by `kubectl get ingress minimal-ingress -o yaml` therefore has API version `extensions/v1beta1`.

### Helm releases

If we deploy our resources with [Helm](https://helm.sh/), Pluto also provides a `detect-helm` subcommand that checks our releases for deprecated API versions. 

## Summary

The Kubernetes API is constantly evolving. To keep our cluster up to date, we constantly have to watch for deprecated and soon-to-be-removed resource API versions. Manual checks are possible thanks to the Kubernetes deprecation guide, but they can become very tedious and error-prone. Tools like pluto allow us to automate API deprecation checks and simplify the effort involved in maintaining up-to-date resource API versions.