Resource quotas to manage namespaces better
Tags: kubernetes, quota, policy, namespace
Published: 2022-03-14 12:57:19

Resource quotas prevent resource contention issues by rationing available resources (cpu, memory, gpu ..). Without these controls one rogue application can potentially make the cluster crawl.

Resource quotas provide constraints that limit resource consumption. 
They can be applied only at the namespace level, which means they can be applied to computing resources and limit the number of objects inside the namespace.

In Kubernetes quotas are implemented with a `ResourceQuota` object. It can be used to limit/control the following objects:
- ConfigMaps
- Persistent Volume Claims (PVCs)
- Pods
- Secrets
- Services

---

##### Create quotas
- Check: 
  - https://kubernetes.io/docs/concepts/policy/resource-quotas/#viewing-and-setting-quotas
  - https://raw.githubusercontent.com/kubernetes/website/main/content/en/examples/admin/resource/quota-objects.yaml

##### Apply quota to a namespace
```
❯ kubectl create -f compute-resources.yaml -n test-ns
resourcequota/compute-resources created
❯ kubectl create -f object-counts.yaml -n test-ns
resourcequota/object-counts created
```

##### Get quota details and status
```
❯ kubectl get resourcequotas -n test-ns
NAME                AGE   REQUEST                                                                                                                                                                         LIMIT
compute-resources   22s   requests.cpu: 0/10, requests.memory: 0/40Gi, requests.nvidia.com/gpu: 0/2                                                                                                       limits.cpu: 0/15, limits.memory: 0/60Gi
object-counts       11s   configmaps: 1/20, persistentvolumeclaims: 0/20, pods: 0/50, replicationcontrollers: 0/20, secrets: 1/20, services: 0/20, services.loadbalancers: 0/4, services.nodeports: 0/4
```

##### Dele quotas
```
❯ kubectl delete resourcequota/compute-resources resourcequota/object-counts
resourcequota "compute-resources" deleted
resourcequota "object-counts" deleted
```
##### Tests
```
❯ for i in $(seq 1 5); do
kubectl create deployment test-dep-${i} --image=alpine --replicas=1 -n test-ns
kubectl expose deployment test-dep-${i} --type=NodePort --port=80 -n test-ns
done
deployment.apps/test-dep-1 created
service/test-dep-1 exposed
deployment.apps/test-dep-2 created
service/test-dep-2 exposed
deployment.apps/test-dep-3 created
service/test-dep-3 exposed
deployment.apps/test-dep-4 created
service/test-dep-4 exposed
deployment.apps/test-dep-5 created
Error from server (Forbidden): services "test-dep-5" is forbidden: exceeded quota: object-counts, requested: services.nodeports=1, used: services.nodeports=4, limited: services.nodeports=4
```

Side effect:
  Even the pods are also not created as there are no memory, cpu requirements given
```
❯ kubectl get deployments.apps test-dep-1 -n test-ns
NAME         READY   UP-TO-DATE   AVAILABLE   AGE
test-dep-1   0/1     0            0           5m17s
❯ kubectl get rs test-dep-1-85d7f5dcdb -n test-ns
NAME                    DESIRED   CURRENT   READY   AGE
test-dep-1-85d7f5dcdb   1         0         0       5m44s
❯ kubectl describe rs test-dep-1-85d7f5dcdb -n test-ns | tail -2
  Warning  FailedCreate  4m40s                 replicaset-controller  Error creating: pods "test-dep-1-85d7f5dcdb-jh62r" is forbidden: failed quota: compute-resources: must specify limits.cpu,limits.memory,requests.cpu,requests.memory
  Warning  FailedCreate  118s (x7 over 4m39s)  replicaset-controller  (combined from similar events): Error creating: pods "test-dep-1-85d7f5dcdb-6679m" is forbidden: failed quota: compute-resources: must specify limits.cpu,limits.memory,requests.cpu,requests.memory
```

We add resource limits:
```
...
    spec:
      containers:
        - name: nginx
          image: nginx:1.14.2
          ports:
            - containerPort: 80
          resources:
            limits:
              memory: '1Gi'
              cpu: '800m'
            requests:
              memory: '256Mi'
              cpu: '100m'
```

Rerun the deployments (`replicaCount=3`)
```
❯ for i in $(seq 1 5); do
sed -e "s/nginx-dep/nginx-dep-${i}/" /tmp/dep.yaml | kubectl apply -f -
done
deployment.apps/nginx-dep-1 created
deployment.apps/nginx-dep-2 created
deployment.apps/nginx-dep-3 created
deployment.apps/nginx-dep-4 created
deployment.apps/nginx-dep-5 created
❯ k get deployments.apps -n test-ns
NAME          READY   UP-TO-DATE   AVAILABLE   AGE
nginx-dep-1   3/3     3            3           102s
nginx-dep-2   3/3     3            3           102s
nginx-dep-3   3/3     3            3           102s
nginx-dep-4   3/3     3            3           101s
nginx-dep-5   3/3     3            3           101s
❯ k get resourcequotas -n test-ns
NAME                AGE   REQUEST                                                                                                                                                                          LIMIT
compute-resources   22m   requests.cpu: 1500m/10, requests.memory: 3840Mi/40Gi, requests.nvidia.com/gpu: 0/2                                                                                               limits.cpu: 12/15, limits.memory: 15Gi/60Gi
object-counts       22m   configmaps: 1/20, persistentvolumeclaims: 0/20, pods: 15/50, replicationcontrollers: 0/20, secrets: 1/20, services: 0/20, services.loadbalancers: 0/4, services.nodeports: 0/4
```

Let us push beyond cpu limits
```
❯ for i in $(seq 6 10); do
sed -e "s/nginx-dep/nginx-dep-${i}/" /tmp/dep.yaml | kubectl apply -f -
deployment.apps/nginx-dep-6 created
deployment.apps/nginx-dep-7 created
deployment.apps/nginx-dep-8 created
deployment.apps/nginx-dep-9 created
deployment.apps/nginx-dep-10 created
done
❯ k get rs -n test-ns
NAME                      DESIRED   CURRENT   READY   AGE
...
nginx-dep-6-7fbd56f7d8    3         3         3       7m35s
nginx-dep-7-7fbd56f7d8    3         0         0       3m51s
nginx-dep-8-7fbd56f7d8    3         0         0       3m39s
nginx-dep-9-7fbd56f7d8    3         0         0       3m39s
```

It creates 6 deployments and stalls for deployment-7 .. 10
```
❯ kubectl describe rs nginx-dep-7-7fbd56f7d8 -n test-ns | tail -2
  Warning  FailedCreate  4m14s                replicaset-controller  Error creating: pods "nginx-dep-7-7fbd56f7d8-bv7hb" is forbidden: exceeded quota: compute-resources, requested: limits.cpu=800m, used: limits.cpu=14400m, limited: limits.cpu=15
  Warning  FailedCreate  92s (x7 over 4m13s)  replicaset-controller  (combined from similar events): Error creating: pods "nginx-dep-7-7fbd56f7d8-54shl" is forbidden: exceeded quota: compute-resources, requested: limits.cpu=800m, used: limits.cpu=14400m, limited: limits.cpu=15
```
---

##### Setting boundary values for cpu and memory
Use the `LimitRange` resource
- https://kubernetes.io/docs/concepts/policy/limit-range/

##### References
- https://kubernetes.io/docs/concepts/policy/resource-quotas/
- https://kubernetes.io/docs/tasks/administer-cluster/quota-api-object/