Learning by auditing Kubernetes manifests
Tags: devops, kubernetes, goodpractices, security
Published: 2022-07-04 17:34:36

Last year, I spoke at the [National DevOps Conference](https://www.devopsonline.co.uk/national-devops-conference/) that took place at the British Museum. I had already visited the museum before, but speaking there was a fantastic experience. Besides, we had the museum all for ourselves for a couple of hours. If you've ever visited the place, you know what I mean.

Anyway, I also attended a talk about [Checkov](https://www.checkov.io/):

>Checkov scans cloud infrastructure configurations to find misconfigurations before they're deployed.
>
>Checkov uses a common command line interface to manage and analyze infrastructure as code (IaC) scan results across platforms such as Terraform, CloudFormation, Kubernetes, Helm, ARM Templates and Serverless framework.

After the talk, I wanted to give it a try. Though I'm not a huge Cloud user, I'm using Kubernetes in a couple of my demos. I installed the CLI and launched it on this [folder](https://github.com/hazelcast-demos/zerodowntime/tree/master/infrastructure/kube), just for fun. Interestingly, I learned _a lot_.

Let's start with a quick summary:

```bash
checkov -d infrastructure/kube/ --quiet --compact | grep CKV | sort --unique
```

The command outputs all the rules that I broke.

I knew about some of them, even though I don't intend to fix them in my context. For example, requests and limits make sense in production, but I don't care about them for my demo.

Others are entirely new to me. I'm happy to share here my <abbr title="Things I Learned">TIL</abbr> (or not).

## Prefer digests to tags

* _Rule_: [CKV_K8S_43 - Ensure images are selected using a digest](https://docs.bridgecrew.io/docs/bc_k8s_39)
* _Severity_: Medium

I've been bitten by this one before. Even though I'll keep it as it is, it deserves an explanation. And the explanation is that **image tags are not immutable**.

Tags are just pointers to specific images. Publishers can point an existing tag to another image. In this case, builds are not idempotent, as the base image may change from one build to another.

For the demo, I'm building images on my local machine so that keeping the tag makes sense. I can update the dependencies' version, rebuild, and get the latest built image.

## Explicitly disallow privilege escalation

* _Rule_: [CKV_K8S_20 - Ensure containers do not run with AllowPrivilegeEscalation](https://docs.bridgecrew.io/docs/bc_k8s_19)
* _Severity_: Medium

This one came up as a bit of a surprise. To understand the issue, we need to read the relevant documentation:

>**AllowPrivilegeEscalation** - Gates whether or not a user is allowed to set the security context of a container to `allowPrivilegeEscalation=true`. This defaults to allowed so as to not break setuid binaries. Setting it to `false` ensures that no child process of a container can gain more privileges than its parent.
>
> -- [Pod Security Policies - Privilege Escalation](https://kubernetes.io/docs/concepts/policy/pod-security-policy/#privilege-escalation)

To summarize: **by default, a child process of a container can gain more privileges than its parent**.

It's a pretty serious issue. Note that the Pod Security Policy itself has been deprecated in v1.21 and will be removed in v1.25. Until that version, one should always explicitly set the attribute to `false`.

## Set the security context

* _Rule_: [CKV_K8S_30 - Ensure securityContext is applied to pods and containers](https://docs.bridgecrew.io/docs/bc_k8s_28)
* _Severity_: Low

Though the severity is marked as low, I believe this rule is essential. Each pod or container can have a `securityContext` section in the manifest. The section has many different fields:

* `runAsNonRoot`, `runAsUser` and `runAsGroup`: see _Don't run as root_ below
* `seccompProfile`: see _Set the seccomp profile_ below
* `fsGroup` and `fsGroupChangePolicy`
* `seLinuxOptions`
* `supplementalGroups`
* `sysctls`
* `windowsOptions`

Some of them are described in more detail in the sections below, but a complete description goes beyond an introductory blog post. If you want to dive deeper, please check the [relevant documentation](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.19/#securitycontext-v1-core).

Alternatively, Snykt wrote a [great explanatory article](https://snyk.io/blog/10-kubernetes-security-context-settings-you-should-understand/) on the most important ones.

## Don't run as root

* _Rule_: [CKV_K8S_23 - Minimize admission of root containers](https://docs.bridgecrew.io/docs/bc_k8s_22)
* _Severity_: Medium

I think this one is pretty well-known, but it's still worth repeating. A container is a Linux process that leverages kernel features:

* _Control groups_ limit, account for, and isolate resource usage, _e.g._ CPU, memory, disk I/O, network, etc.
* _Namespaces_ partition kernel resources such that one set of processes sees one set of resources while another set of processes sees a different set of resources

While both provide process isolation, it's not foolproof. If a container running as root is compromised, an attacker could use the extra permissions to attack further. To fix the issue, use the following snippet:

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: <name>
spec:
    securityContext:
    runAsNonRoot: true
    runAsUser: <user>
```

## Set a high UID user

* _Rule_: [CKV_K8S_40 - Ensure containers run with a high UID to avoid host conflict](https://docs.bridgecrew.io/docs/bc_k8s_37)
* _Severity_: Low

In the previous rule check, we didn't run as `root`, which is UID 1. However, even with other UIDs, we risk impersonating another user on the host system if the container is compromised. To drastically reduce the probability, configured users should only start from UID 10,000.

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: <name>
spec:
    securityContext:
    runAsUser: <+10,000>
```

## Set the seccomp profile

* _Rule_: [CKV_K8S_31 - Ensure seccomp profile is set to Docker/Default or Runtime/Default](https://docs.bridgecrew.io/docs/bc_k8s_30)
* _Severity_: Low

>Secure computing mode (`seccomp`) is a Linux kernel feature. You can use it to restrict the actions available within the container. The `seccomp()` system call operates on the seccomp state of the calling process. You can use this feature to restrict your application’s access.
>
>The default `seccomp` profile provides a sane default for running containers with seccomp and disables around 44 system calls out of 300+. It is moderately protective while providing wide application compatibility.
>
> -- [Seccomp security profiles for Docker](https://docs.docker.com/engine/security/seccomp/)

Addressing the issue depends on the version of your Kubernetes cluster:

* For Kubernetes up to 1.18, annotations to the rescue:

    ```yaml
    apiVersion: v1
    kind: Pod
    metadata:
      name: <name>
      annotations:
        seccomp.security.alpha.kubernetes.io/pod: "runtime/default"
    ```

* For Kubernetes 1.19+, the `securityContext` attribute features a `secompProfile`:

    ```yaml
    apiVersion: v1
    kind: Pod
    metadata:
      name: <name>
    spec:
      securityContext:
        seccompProfile:
          type: RuntimeDefault
      containers:
        ...
    ```

## Conclusion

If you want to enforce policies on `seccomp`, please check the documentation relative to [Pod Security Admission](https://kubernetes.io/docs/concepts/security/pod-security-admission/), or [Pod Security Policies](https://kubernetes.io/docs/concepts/policy/pod-security-policy/), depending on your Kubernetes version.

On a more general note, running an analysis/audit tool is a great way to get insight into a subject. I've done it on Kubernetes with Checkov, but the number of such tools is large enough to get an instant knowledge boost if you feel like it. Of course, it won't make you an instant subject matter expert, but it's a good step on the road.

**To go further:**

* [Checkov: Policy-as-code for everyone](https://www.checkov.io/)
* [Checkov: Kubernetes Policy Index](https://docs.bridgecrew.io/docs/kubernetes-policy-index)
* [Hardening Docker and Kubernetes with seccomp](https://martinheinz.dev/blog/41)
* [10 Kubernetes Security Context settings you should understand](https://snyk.io/blog/10-kubernetes-security-context-settings-you-should-understand/)
* [Pod Security Admission](https://kubernetes.io/docs/concepts/security/pod-security-admission/)

_Initially published at [A Java Geek](https://blog.frankel.ch/learning-auditing-kubernetes-manifests/) on July 4<sup>th</sup>, 2022_