Repository: kubernetes
Issue: CPU Throttling on Linux kernel 5.4.0-1029-aws
Labels: kind/bug,priority/backlog,sig/node,lifecycle/frozen,triage/accepted

**What happened**:
After this thread: https://github.com/kubernetes/kubernetes/issues/67577 we upgraded our instance images to use Ubuntu 20.04 which has kernel `5.4.0-1029-aws`, later than 4.18 that introduced the fix on the CFS and 4.19 that fixed the regression that resulted.
https://engineering.indeedblog.com/blog/2019/12/cpu-throttling-regression-fix/
https://github.com/aws/containers-roadmap/issues/175#issuecomment-498774249

We have set CPU requests and limits for all of our pods on the cluster BUT the control-plane pods and node daemonsets such as network (calico) and kube-proxy, coredns and such, and we see high throttles for our nodejs applications.
```
 === ip-10-28-177-37.ec2.internal | production ===
  Namespace                   Name                                          CPU Requests  CPU Limits   Memory Requests  Memory Limits  AGE
  ---------                   ----                                          ------------  ----------   ---------------  -------------  ---
  kube-system                 calico-node-zbbwm                             100m (1%)     0 (0%)       0 (0%)           0 (0%)         45h
  kube-system                 datadog-sq584                                 0 (0%)        0 (0%)       0 (0%)           0 (0%)         44h
  kube-system                 ext-ingress-nginx-ingress-controller-9v6bg    0 (0%)        0 (0%)       0 (0%)           0 (0%)         45h
  kube-system                 kube-proxy-ip-10-28-177-37.ec2.internal       100m (1%)     0 (0%)       0 (0%)           0 (0%)         45h
  kube-system                 node-local-dns-chmhd                          25m (0%)      0 (0%)       5Mi (0%)         0 (0%)         45h
  production-rts              article-rts-66479cc6f7-hbgp4                  100m (1%)     1800m (22%)  51Mi (0%)        512Mi (1%)     18h
  production-rts              nodejs-app-54674bf85b-mtfmk                       2 (25%)       5500m (68%)  3Gi (9%)         4Gi (12%)      18h
```
We can see throttling is reduced when increasing the limits, and this does not make any sense since nodejs is single-threaded and we're giving more than 1 core limit for those containers.
I peeked a little bit on the nodes running on my cluster and saw that `/sys/fs/cgroup/cpu,cpuacct/kubepods/burstable/cpu.shares` is set to 1561 on a 5 core machine:

```bash
$ cd /sys/fs/cgroup/cpu,cpuacct/kubepods/burstable/
$ for i in cpu.{cfs_*,shares};do echo "$i: $(cat $i)";done
cpu.cfs_period_us: 100000
cpu.cfs_quota_us: -1
cpu.shares: 1561
```
Does that makes sense? I know cgroups are inherited by their parents and with 1561 shares on the `kubepods/burstable`, each pod without quota (cpu limits) will get 1561 shares?

I have a pod running [nodejs application without clustering enabled](https://amagiacademy.com/blog/posts/2020-02-26/node-container-cpu) is defined like this (truncated irrelevant data from the manifest):
```yaml
apiVersion: v1
kind: Pod
metadata:
  name: nodejs-app-54674bf85b-mtfmk
  namespace: production
spec:
  containers:
  - image: myimage:mytag
    name: nodejs-app
    resources:
      limits:
        cpu: 5500m
        memory: 4Gi
      requests:
        cpu: "2"
        memory: 3Gi
```
Based on that, the `cpu,cpuacct` is:
```bash
$ ps -ef | grep node
nobody   3831432 3831414  8 09:01 ?        00:06:21 node --max-old-space-size=8192 --max-http-header-size=16384 app.js
$ cat /proc/3831432/cgroup | grep 'cpu,cpuacct'
8:cpu,cpuacct:/kubepods/burstable/pod6b6b5d77-bb16-4b83-9f60-846167610391/813bc24643f7eef708c85ae5640c0aaf6360b0bf66da1c17c2b04ed6ea205b2f
$ cd /sys/fs/cgroup/cpu,cpuacct/kubepods/burstable/pod6b6b5d77-bb16-4b83-9f60-846167610391/813bc24643f7eef708c85ae5640c0aaf6360b0bf66da1c17c2b04ed6ea205b2f/
$ for i in cpu.{cfs_*,shares};do echo "$i: $(cat $i)";done
cpu.cfs_period_us: 100000
cpu.cfs_quota_us: 550000
cpu.shares: 2048
```
The application gets throttled:
https://pasteboard.co/JG5rUH7.png

**What you expected to happen**:
- Application won't get throttled when set a CPU limit that it doesn't reach.
- Set no more than 1 cpu limit for nodejs application and not get throttled.

**Environment**:
- Kubernetes version (use `kubectl version`):
```
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.0", GitCommit:"af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38", GitTreeState:"clean", BuildDate:"2020-12-13T18:42:23Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.12", GitCommit:"e2a822d9f3c2fdb5c9bfbe64313cf9f657f0a725", GitTreeState:"clean", BuildDate:"2020-05-06T05:09:48Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}
```
- Cloud provider or hardware configuration:
AWS bootstrapped with kOps
- OS (e.g: `cat /etc/os-release`):
```
NAME="Ubuntu"
VERSION="20.04.1 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.1 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
```
- Kernel (e.g. `uname -a`):
```
Linux ip-10-28-165-188 5.4.0-1029-aws #30-Ubuntu SMP Tue Oct 20 10:06:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
```
- Install tools:
- Network plugin and version (if this is a network-related bug): calico