Repository: kubernetes
Issue: scheduler being topology-unaware can cause runaway pod creation
Labels: kind/bug,priority/backlog,sig/scheduling,sig/node,triage/accepted

**What happened**:
With "--topology-manager-policy=single-numa-node" enabled on kubelet, creating a ReplicaSet (or other entity which automatically creates pods) resulted in hundreds of pods with a status of "Topology Affinity Error".

controller-0:~$ kubectl get pod
NAME                         READY   STATUS                    RESTARTS   AGE
exclusive-replicaset-27dzq   0/1     Topology Affinity Error   0          19s
exclusive-replicaset-2f4mz   0/1     Topology Affinity Error   0          11s
exclusive-replicaset-57sk2   0/1     Topology Affinity Error   0          7s
exclusive-replicaset-5qvnt   0/1     Topology Affinity Error   0          6s
exclusive-replicaset-67z22   0/1     Topology Affinity Error   0          19s
exclusive-replicaset-6zlcj   0/1     Topology Affinity Error   0          19s
exclusive-replicaset-7nn5k   0/1     Topology Affinity Error   0          17s
exclusive-replicaset-7q7w2   0/1     Topology Affinity Error   0          11s
exclusive-replicaset-8npgb   0/1     Topology Affinity Error   0          17s
exclusive-replicaset-8ts7f   0/1     Topology Affinity Error   0          1s
exclusive-replicaset-8zcng   0/1     Topology Affinity Error   0          8s
exclusive-replicaset-959m7   0/1     Topology Affinity Error   0          5s
exclusive-replicaset-9q4rn   0/1     Topology Affinity Error   0          6s
exclusive-replicaset-bnzxl   0/1     Topology Affinity Error   0          1s
exclusive-replicaset-bvrvb   0/1     Topology Affinity Error   0          17s
exclusive-replicaset-dcbjg   0/1     Topology Affinity Error   0          2s
exclusive-replicaset-dq84c   0/1     Topology Affinity Error   0          7s


**What you expected to happen**:
Ideally the scheduler shouldn't try to place a pod onto a node where it cannot possibly run due to topology policy.  This requires making the schedule topology-aware.  Until then, rather than going to the error state perhaps the pod should be sent back to the scheduler for re-scheduling to another node (possibly with a delay to ensure we don't waste CPU time constantly rescheduling a pod which can't fit anywhere in the whole cluster).

**How to reproduce it (as minimally and precisely as possible)**:
On a multi-numa-node host, run kubelet with "--cpu-manager-policy=static --topology-manager-policy=single-numa-node".  Attempt to create a ReplicaSet with a "Guaranteed" QoS pod having a container that requests more CPUs than are available on a single NUMA node, but less than are available on the whole host.

The scheduler will send the pod to the node (because it's not topology-aware), but kubelet will determine it cannot fit the pod on a single NUMA node and put the pod into status "Topology Affinity Error".  The ReplicaSet sees that the pod failed, creates another pod, and the same thing happens again...and again...and again.

**Anything else we need to know?**:

**Environment**:
- Kubernetes version (use `kubectl version`):  v1.16.2
- Cloud provider or hardware configuration:  multi-numa-node
- OS (e.g: `cat /etc/os-release`):  centos 7
- Kernel (e.g. `uname -a`): Linux controller-0 3.10.0-957.21.3.rt56.935.el7.tis.2.x86_64 #1 SMP PREEMPT RT Thu Oct 24 16:34:45 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux