Repository: kubernetes
Issue: sts pod pending when we delete this pod when scheduling
Labels: kind/bug,sig/scheduling,lifecycle/stale,triage/needs-information,needs-triage

### What happened?

1 when we have 3 sts , but when one node is down ,and then add a new node

2 we delete the sts pod every 90s when it is pending,  and occasionally the sts pod is still  pending when the pvc is bounded, and even though we delete the sts pod, it can not revover，we have to restart the kube-scheduler, and then the sts pod can be scheduled 

### What did you expect to happen?

The sts pod can be sechduled normally, even when we delete the sts pod

### How can we reproduce it (as minimally and precisely as possible)?

add a new node for sts , and delete the sts pod every 90s

### Anything else we need to know?

1. I found the sts pod has already boudded to master0 node as the kube-scheduler logs show :
```
2023-11-09T18:40:21.332222821+08:00 stderr F I1109 10:40:21.332166       1 binder.go:671] "All PVCs for pod are bound" pod="ccos-mysql/ccos-mysql-2-0"
2023-11-09T18:40:21.336490774+08:00 stderr F I1109 10:40:21.336471       1 schedule_one.go:266] "Successfully bound pod to node" pod="ccos-mysql/ccos-mysql-2-0" node="503-g09-12u-master0" evaluatedNodes=22 feasibleNodes=1
2023-11-09T18:40:46.324358822+08:00 stderr F I1109 10:40:46.324303       1 schedule_one.go:826] "Unable to schedule pod; no fit; waiting" pod="ccos-mysql/ccos-mysql-2-0" err="0/22 nodes are available: 1 node(s) didn't match pod anti-affinity rules, 19 node(s) didn't match Pod's node affinity/selector, 2 node(s) had volume node affinity conflict. preemption: 0/22 nodes are available: 1 No preemption victims found for incoming pod, 21 Preemption is not helpful for scheduling."

```
2. and the kube-scheduler cache also indicates the "ccos-mysql-2-0" pod has already bounded to this node:
```
        Node name: 503-g09-12u-master0
        Deleted: false
        Requested Resources: &{MilliCPU:39984 Memory:97883189376 EphemeralStorage:0 AllowedPodNumber:0 ScalarResources:map[]}
        Allocatable Resources:&{MilliCPU:79500 Memory:402498994176 EphemeralStorage:430489415787 AllowedPodNumber:500 ScalarResources:map[hugepages-1Gi:0 hugepages-2Mi:0]}
        Scheduled Pods(number: 319):
        name: ccos-mysql-2-0, namespace: ccos-mysql, uid: 45d96618-9739-49aa-a8d5-08d170fb91c8, phase: Pending, nominated node: 

```
3. but the scheduling queue show there is another ccos-mysql-2-0 pod is also scheduling, but uid is different
```
I1113 07:13:01.730818       1 dumper.go:62] "Dump of scheduling queue" pods=<
        name: ccos-mysql-2-0, namespace: ccos-mysql, uid: 10989100-926b-41b0-9652-8bed846efc10, phase: Pending, nominated node: 
```

4. we found the pvc is already bounded 
```
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: ccos.io/local
    volume.kubernetes.io/selected-node: 503-g09-12u-master0
    volume.kubernetes.io/storage-provisioner: ccos.io/local
    volumequota.ccos.io/ignore-on-nonstandard-node: "true"
  creationTimestamp: "2023-11-09T10:37:42Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    member: ccos-mysql-2
  name: rds-ccos-mysql-2-0
  namespace: ccos-mysql
  resourceVersion: "235564503"
  uid: dd88f185-171b-4b10-adde-83ad425391b2
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    limits:
      storage: 50Gi
    requests:
      storage: 50Gi
  storageClassName: ccos-hostpath-data-meta
  volumeMode: Filesystem
  volumeName: pvc-dd88f185-171b-4b10-adde-83ad425391b2
```

I believe this has something to do with  the the deletion of  mysql sts pods when it is still pending

### Kubernetes version

<details>

```console
$ kubectl version
# paste output here
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.8-1+087d0f619-dirty", GitCommit:"087d0f619", GitTreeState:"dirty", BuildDate:"2023-09-27T07:43:23Z", GoVersion:"go1.19.3", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25+", GitVersion:"v1.25.8-28+7a988ab22c9159-dirty", GitCommit:"7a988ab22c91599b564e1faee58b4245722db11a", GitTreeState:"dirty", BuildDate:"2023-09-25T13:17:17Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
```

</details>


### Cloud provider

<details>
no
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here
6.15.67-6.cl9.x86_64
# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>