Repository: kubernetes
Issue: Kubelet doesn't support dynamic CPU offlining/onlining
Labels: sig/node,kind/feature,needs-triage

<!-- This form is for bug reports and feature requests ONLY!

If you're looking for help check [Stack Overflow](https://stackoverflow.com/questions/tagged/kubernetes) and the [troubleshooting guide](https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/).

If the matter is security related, please disclose it privately via https://kubernetes.io/security/.
-->

**Is this a BUG REPORT or FEATURE REQUEST?**:

/kind bug

**What happened**:

Kubelet doesn't follow CPU hotplug events. For example, if I offline a CPU, guaranteed pods might still be assigned there even though there is no real capacity available. Also, the CPU manager static policy might try to assign containers to non-existent CPUs. This would lead to containers not starting, because the CRI cpuset assignment would fail.

**How to reproduce it (as minimally and precisely as possible)**:

On the master node check CPU capacity:

    $ kubectl get node worker-node -o json | jq '.status | .capacity | .cpu'
    "4"
    
On the worker node offline a CPU:

    # echo 0 > /sys/devices/system/cpu/cpu2/online
    
On the master node check CPU capacity again:

    $ kubectl get node worker-node -o json | jq '.status | .capacity | .cpu'
    "4"

**What you expected to happen**:

The cpu capacity should have gone to 3.

**Anything else we need to know?**:

The correct way to fix this would be to listen to udev hotplug events from a netlink socket. When an event telling that a CPU has been added or removed is received, kubelet should do a few things:
1. Inform API server about the new capacity in NodeStatus message.
2. Inform CPU manager that a new topology must be loaded. The containers must be reassigned to CPUs, because either the default pool grew or some container lost a CPU on which it was assigned to.

CPU hotplug functionality is needed for some pretty special use cases. One of them is the possibility to (dynamically) disable SMT support by offlining a sibling core. I know that this bug is not something that a regular user will meet on daily basis, but since kubernetes manages CPU usage and CPU allocations, this is something that it should take into account.

I would be happy to take a look at fixing this bug if there is some sort of consensus that such a patch might be accepted in kubernetes.

**Environment**:
- Kubernetes version (use `kubectl version`):

        Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", 
        GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:53:20Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
        Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", 
        GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:43:26Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

- Cloud provider or hardware configuration: Local VMs
- OS (e.g. from /etc/os-release):

        NAME=Fedora
        VERSION="28 (Workstation Edition)"
- Kernel (e.g. `uname -a`):

        4.17.12-200.fc28.x86_64