Multi-tenancy in Kubernetes
Tags: kubernetes, devops
Published: 2023-04-10 12:28:23

_Should you have more than one team using the same Kubernetes cluster?_

_Can you run untrusted workloads safely from untrusted users?_

_Does Kubernetes do multi-tenancy?_

This article will explore the challenges of running a cluster with multiple tenants.

Multi-tenancy can be divided into:

1. **Soft multi-tenancy** for when you trust your tenants — like when you share a cluster with teams from the same company.
2. **Hard multi-tenancy** for when you don't trust tenants.

You can also have a mix!

![](https://res.cloudinary.com/learnk8s/image/upload/v1681109800/threads/multi-tenancy-2_ay5im8.png)

**The basic building block to share a cluster between tenants is the namespace.**

Namespaces group resources logically — they don't offer any security mechanisms nor guarantee that all resources are deployed in the same node.

![Namespaces are used to logically group resources](https://res.cloudinary.com/learnk8s/image/upload/v1681128917/threads/multi-tenancy-3_arwbxp.png)

**Pods in a namespace can still talk to all other pods in the cluster, make requests to the API, and use as many resources as they want.**

Out of the box, any user can access any namespace.

_How should you stop that?_

![Namespaces don't offer any mechanism for isolation](https://res.cloudinary.com/learnk8s/image/upload/v1681109800/threads/multi-tenancy-4_uwpv1x.png)

[With RBAC, you can limit what users and apps can do with and within a namespace.](https://learnk8s.io/rbac-kubernetes)

A common operation is to grant permissions to limited users.

![Limiting what users can do in a namespace](https://res.cloudinary.com/learnk8s/image/upload/v1681109800/threads/multi-tenancy-5_ojf1jt.png)

[With Quotas](https://kubernetes.io/docs/concepts/policy/resource-quotas/) and [LimitRanges](https://kubernetes.io/docs/concepts/policy/limit-range/), you can limit the resources deployed in the namespace and the memory, CPU, etc., that can be utilized.

**This is an excellent idea if you want to limit what a tenant can do with their namespace.**

![LimitRange and ResourceQuota](https://res.cloudinary.com/learnk8s/image/upload/v1681109800/threads/multi-tenancy-6_ibzbiv.png)

**By default, all pods can talk to any pod in Kubernetes.**

This is not great for multi-tenancy, but you can correct this with [NetworkPolicies.](https://github.com/ahmetb/kubernetes-network-policy-recipes)

Network policies are similar to firewall rules that let you segregate outbound and inbound traffic.

![Network policies in Kubernetes](https://res.cloudinary.com/learnk8s/image/upload/v1681109800/threads/multi-tenancy-7_tphsen.png)

_Great, is the namespace secure now?_

Not so fast.

While RBAC, NetworkPolicies, Quotas, etc., give you the basic building blocks for multi-tenancy is not enough.

**Kubernetes has several shared components.**

A good example is the Ingress controller, which is usually deployed once per cluster.

**If you submit an Ingress manifest with the same path, the last overwrites the definition and only one works.**

![A single Ingress controller shared across namespaces](https://res.cloudinary.com/learnk8s/image/upload/v1681109800/threads/multi-tenancy-8_qqxdmh.png)

It's a better idea to deploy a controller per namespace.

Another interesting challenge is CoreDNS.

_What if one of the tenants abuses the DNS?_

The rest of the cluster will suffer too.

You could limit requests with an extra plugin <https://github.com/coredns/policy>.

![Abusing of CoreDNS](https://res.cloudinary.com/learnk8s/image/upload/v1681109800/threads/multi-tenancy-9_zh3fv2.png)

The same challenge applies to the Kubernetes API server.

**Kubernetes isn't aware of the tenant, and if the API receives too many requests, it will throttle them for everyone.**

I don't know if there's a workaround for this!

![Abusing of the Kubernetes API](https://res.cloudinary.com/learnk8s/image/upload/v1681109801/threads/multi-tenancy-10_prbtfn.png)

Assuming you manage to sort out shared resources, there's also the challenge with the kubelet and workloads.

[As Philippe Bogaerts explains in this article,](https://xxradar.medium.com/exploiting-applications-using-livenessprobes-in-kubernetes-cdff6329d320) a tenant could take over nodes in the cluster just (ab)using liveness probes.

_The fix is not trivial._

![Exploiting Kubernetes with liveness probes](https://res.cloudinary.com/learnk8s/image/upload/v1681109801/threads/multi-tenancy-11_fneior.png)

You could have a linter as part of your CI/CD process or use admission controllers to verify that resources submitted to the cluster are safe.

[Here is a library or rules for the Open Policy Agent.](https://github.com/open-policy-agent/gatekeeper-library)

![Using admission controllers to lint and validate resources](https://res.cloudinary.com/learnk8s/image/upload/v1681109801/threads/multi-tenancy-12_jamopm.png)

**You also have containers that offer a weaker isolation mechanism than virtual machines.**

[Lewis Denham-Parry shows how to escape from a container in this video.](https://www.youtube.com/watch?v=JaMJJTb_bEE)

_How can you fix this?_

You could use a container sandbox like [gVisor](https://gvisor.dev/), light virtual machines as containers ([Kata containers](https://katacontainers.io/), [firecracker + containerd](https://github.com/firecracker-microvm/firecracker-containerd)) or full virtual machines ([virtlet as a CRI](https://github.com/Mirantis/virtlet)).

![gVisor, Kata containers, Firecracker + containerd, virtlet](https://res.cloudinary.com/learnk8s/image/upload/v1681109801/threads/multi-tenancy-13_xzqsue.png)

Hopefully, you've realized the complexity of the subject and how it's hard to provide rigid boundaries to separate networks, workloads, and controllers in Kubernetes.

[That's why providing hard multi-tenancy in Kubernetes is not recommended.](https://blog.jessfraz.com/post/hard-multi-tenancy-in-kubernetes/)

If you need hard multi-tenancy, the advice is to use multiple clusters or a Cluster-as-a-Service tool instead.

- [Cluster API](https://github.com/kubernetes-sigs/cluster-api)
- [HyperShift](https://github.com/openshift/hypershift)
- [Kamaji](https://github.com/clastix/kamaji)
- [Gardener](https://gardener.cloud/)

If you can tolerate the weaker multi-tenancy model in exchange for simplicity and convenience, you can roll out your RBAC, Quotas, etc. rules.

But there are a few tools that abstract those problems from you:

- [Virtual Cluster (wg-multitenancy)](https://github.com/kubernetes-sigs/cluster-api-provider-nested/tree/main/virtualcluster)
- [Vcluster](https://www.vcluster.com/)
- [Capsule](https://github.com/clastix/capsule)
- [Kiosk](https://github.com/loft-sh/kiosk)

And finally, if you've enjoyed this thread, you might also like:

- The Kubernetes workshops that we run at Learnk8s https://learnk8s.io/training
- This collection of past threads https://twitter.com/danielepolencic/status/1298543151901155330
- The Kubernetes newsletter I publish every week https://learnk8s.io/learn-kubernetes-weekly