Repository: kubernetes
Issue: “--system-reserved” kubelet option cannot work as expected
Labels: kind/bug,area/kubelet,sig/node,priority/important-longterm,triage/accepted

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!-->

**What happened**:

start kubelet with below config:
```
[root@app ~]# cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf 
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=8686"
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"
Environment="KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki"
ExecStart=/usr/bin/kubelet 
$KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS 
--docker-root=/SkyDiscovery/docker --enforce-node-allocatable=pods,kube-reserved,system-reserved 
--kube-reserved-cgroup=/kube.slice --kubelet-cgroups=/kube.slice 
--runtime-cgroups=/kube.slice --system-reserved-cgroup=/system.slice 
--kube-reserved=cpu=2,memory=1Gi --system-reserved=cpu=2,memory=1Gi 
--eviction-hard=memory.available<5%
[root@app ~]# 
```
check the related cgroup file, get the following output:
```
[root@app ~]# cat /sys/fs/cgroup/cpu/system.slice/cpu.shares 
2048
[root@app ~]# cat /sys/fs/cgroup/cpu/system.slice/cpu.cfs_period_us 
100000
[root@app ~]# cat /sys/fs/cgroup/cpu/system.slice/cpu.cfs_quota_us 
-1
[root@app ~]# cat /sys/fs/cgroup/memory/system.slice/memory.limit_in_bytes
1073741824
[root@app ~]# 
```

**What you expected to happen**:

`/sys/fs/cgroup/memory/system.slice/memory.limit_in_bytes` is not limited.

due to `memory.limit_in_bytes` is limited, may cause system services are killed by OOM, expect this config only affect the available memory resource for node(check with `kubectl describe node $node_name`)

`cpu.cfs_quota_us=-1` means not set limit for CPU usage,  but `memory.limit_in_bytes=1073741824`  means set limit for memory usage, why the function of reserved cpu and reserved memory is differnent?  

**How to reproduce it (as minimally and precisely as possible)**:

start kubelet service with config:
```
[root@app ~]# systemctl cat kubelet 
# /etc/systemd/system/kubelet.service
[Unit]
Description=kubelet: The Kubernetes Node Agent
Documentation=http://kubernetes.io/docs/
After=cephfs-mount.service
Requires=cephfs-mount.service

[Service]
ExecStartPre=/usr/bin/mkdir -p /sys/fs/cgroup/hugetlb/system.slice
ExecStartPre=/usr/bin/mkdir -p /sys/fs/cgroup/cpuset/system.slice
ExecStart=/usr/bin/kubelet
Restart=always
StartLimitInterval=0
RestartSec=10

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=8686"
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"
Environment="KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS --docker-root=/SkyDiscovery/docker --enforce-node-allocatable=pods,kube-reserved,system-reserved --kube-reserved-cgroup=/kube.slice --kubelet-cgroups=/kube.slice --runtime-cgroups=/kube.slice --system-reserved-cgroup=/system.slice --kube-reserved=cpu=2,memory=1Gi --system-reserved=cpu=2,memory=1Gi --eviction-hard=memory.available<5%
```

**Anything else we need to know?**:

i have read this doc: https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/
and get these info:
```
Be extra careful while enforcing system-reserved reservation since it can lead to critical system
services being CPU starved or OOM killed on the node.The recommendation is to enforce 
system-reserved only if a user has profiled their nodes exhaustively to come up with precise
estimates and is confident in their ability to recover if any process in that group is oom_killed.
```
so it is reasonable to limit the file  `/sys/fs/cgroup/memory/system.slice/memory.limit_in_bytes` ? 

if not use `--system-reserved`, the ` /sys/fs/cgroup/cpu/system.slice/cpu.shares ` will be `1024` by default, which will result in system services being CPU starved


**Environment**:
- Kubernetes version (use `kubectl version`):   v1.10.12
- Cloud provider or hardware configuration:  dell server PowerEdge R740
- OS (e.g. from /etc/os-release):  CentOS7.4
- Kernel (e.g. `uname -a`):  3.10.0-693.el7.centos.x86.64
- Install tools: 
- Others: