Repository: kubernetes
Issue: race condition detected during the scheduling with preemption 
Labels: kind/bug,sig/scheduling,lifecycle/frozen

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!

If the matter is security related, please disclose it privately via https://kubernetes.io/security/
-->


**What happened**:
race condition detected during the scheduling with preemption 

**What you expected to happen**:
the preemptor with high priority should be scheduled successfully.

**How to reproduce it (as minimally and precisely as possible)**:
In the case of preemption,  with the below assumption, 

- node1 has the replication controller with couple of pods running, and consider to be the candidate node for the preemption, 

  ![image](https://user-images.githubusercontent.com/6007417/88760460-a6671480-d19f-11ea-9cec-38aa9e849448.png)




- node2 also has some pods running on it but will not be considered to be the candidate node due to either priority or topology constraints, 
![image](https://user-images.githubusercontent.com/6007417/88651160-f2607d80-d0fb-11ea-8c4f-a009e6fd88ad.png)


the scheduling sequence very likely will hit the race condition like below, 

step1: preemptor pod (pod6) is coming and found no fit node to schedule
step2: error is recorded and the pod6 is added into the unschedulableQ
step3: preemption is happen on node1 and pod1 is deleted
step4: pod1 is added to activeQ
step5: field of `NominatedNodeName` is patched to pod6
step6: next scheduling cycle is started and the pod1 is only pod in the `activeQ`
step7: going to schedule pod in the `activeQ` (pod1) and the candidate node is node1
step8: event of `OnUpdate` of the preemptor pod (pod6) is captured and pod6 is moved from `unschedulableQ` to `activeQ`
step9: pod1 is scheduled on the node1
step10: next scheduling cycle is started and the pod6 is the only pod in the `activeQ`
step11: preemption is happened again and pod2 is removed this time.
step12: error is recorded again and the pod is moved to `podBackoffQ`, and the `nominatedNode` is still node1
step13: pod `condition` is not changed, `nominatedNode` is not changed, so the pod is not updated, pod6 is still in the `podBackoffQ`.
step14:  pod6 needs to wait for backoff time expired before getting moved from  `podBackoffQ` to `activeQ`
step15:  pod2 is scheduled on node1.

looping forever from here !!!  along with pod deletion and preemption on node1.
...
...

The root cause is step8 `OnUpdate`  event is not guaranteed to be finished before step7, they are handled asynchronously.

PR: https://github.com/kubernetes/kubernetes/pull/93179 is happened to partially addressed this issue by updating the `nominatedNode` to `nil` in step13,  the pod is moved from `podBackoffQ` to `activeQ` by the change, this makes it possible to break the infinite loop since high priority preemptor got a chance to be added into the `activeQ` before the next scheduling cycle is started.

The final solution per my understanding is we need to add a check here, maybe hold for an while util we found the preemptor is added into `activeQ`

https://github.com/kubernetes/kubernetes/blob/e79e352d36258abc5e5659289ec0fb13634bcbe7/pkg/scheduler/scheduler.go#L336-L343





**Anything else we need to know?**:

**Environment**:
- Kubernetes version (use `kubectl version`):
- Cloud provider or hardware configuration:
- OS (e.g: `cat /etc/os-release`):
- Kernel (e.g. `uname -a`):
- Install tools:
- Network plugin and version (if this is a network-related bug):
- Others: