Repository: kubernetes
Issue: Sts pod pending in scheduler cache
Labels: kind/bug,sig/scheduling,lifecycle/stale,needs-triage

### What happened?

1 . we found the mysql pod is pending , but its pvc is already bounded, the pods show as follows
```
[root@sphere-node-1 ~]# kubectl get po -n mysql
NAME                                   READY   STATUS    RESTARTS   AGE
mysql-0-0                         0/1     Pending   0          4h4m
mysql-1-0                         1/1     Running   0          15d
mysql-2-0                         1/1     Running   0          5h7m
mysql-exporter-7c45cbc7b7-9dfh4   1/1     Running   0          15d
[root@sphere-node-1 ~]# kubectl get pvc -n mysql
NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                        AGE
mysql-0-0   Bound    pvc-114695a3-885e-4a87-9e7d-75d76cf059e5   50Gi       RWO            hostpath-data-meta             15d
mysql-1-0   Bound    pvc-0f186afd-bb87-4b3a-97a7-5aae6f071c3f   50Gi       RWO            hostpath-data-meta             15d
mysql-2-0   Bound    pvc-cd92c657-3c88-46e8-8352-22d5ecb46fe9   50Gi       RWO            hostpath-data-meta             5h7m



I1016 06:29:42.137250       1 schedule_one.go:826] "Unable to schedule pod; no fit; waiting" pod="mysql/mysql-0-0" err="0/5 nre available: 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match Pod's node affinity/selector, 2 node(s) had volumeaffinity conflict. preemption: 0/5 nodes are available: 1 node(s) didn't match pod anti-affinity rules, 4 Preemption is not helpful foruling."

```

2. But the sheduler cache info shows the  one mysql-2-0  is Running in node-6, and one mysql-2-0  is pending in node-3
```
Node name:  node-6
name: ccos-mysql-2-0, namespace: ccos-mysql, uid: cd835665-9889-4e04-8c75-05b7ae984dc9, phase: Running, nominated node:

Node name:  node-3
name: ccos-mysql-2-0, namespace: ccos-mysql, uid: 2a3ab7ba-a9c1-404d-b4e4-e7dcfd91578c, phase: Pending, nominated node:

 1 dumper.go:62] "Dump of scheduling queue" pods=<
        name: ccos-mysql-0-0, namespace: ccos-mysql, uid: f10be0d0-c41d-4629-a8ce-4002ecf8b4c3, phase: Pending, nominated node:

```

### What did you expect to happen?

The sts pod should not exists in Running and Pending state at the same time

### How can we reproduce it (as minimally and precisely as possible)?

remove one sts pod and try to create one new

related to https://github.com/kubernetes/kubernetes/issues/121866

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
# paste output here
```
1.25.8

</details>


### Cloud provider

<details>

</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```
5.15.131-9
</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>