Repository: kubernetes
Issue: A completed status pod which request 1 nvidia.com/gpu resource updated to UnexpectedAdmissionError when restart kubelet service
Labels: kind/bug,sig/node,triage/accepted

### What happened?

I have a k8s cluster (version v1.23.6) with a GPU node which has 4  GPUs.

Firstly, I submitted a job which request 1 nvidia.com/gpu, and after a while the job successfully finished with a  Completed pod status.

Secondly, I submitted a deployment which request 4 nvidia.com/gpu, it keeps ruuning on the GPU node.

Thirdly, I restart the kubelet service on the GPU node, then the job pod updated pod status from Completed to UnexpectedAddmissionError.

```
kubectl get pods
NAME              READY   STATUS                                  RESTARTS   AGE
job-9s7jw         0/1        UnexpectedAdmissionError   0          5m1s
deploy-6d485b4676-fbv69   1/1        Running                                 0          3m14s
deploy-6d485b4676-9gfzg   1/1        Running                                 0          3m14s
deploy-6d485b4676-q7lr9   1/1        Running                                 0          3m14s
deploy-6d485b4676-w5qmx1/1        Running                                 0          3m14s
```

Describing the job pod:
```
...
Status:              Failed
Reason:            UnexpectedAdmissionError
Message:         Pod Allocate failed due to requested number of devices unavailable for nvidia.com/gpu. Requested:1, Available: 0, which is unexpected
```

### What did you expect to happen?

The job  has completed successfully and release the gpu resource, kubelet should not try to reallocate a GPU resource to a Completed status pod when the kubelet service is restarted.

I expect that the job pod should keep Completed status when kubelet service is restarted.

### How can we reproduce it (as minimally and precisely as possible)?

(1) submitted a job which request 1 nvidia.com/gpu resource, waiting for the job completed,pod status is Completed.
(2) submtted a deployment which request all nvidia.com/gpu resource.
(3) restart kubelet service on the gpu node, and you can see that the job pod updated pod status to UnexpectedAdmissionError.

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.6", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2022-04-14T14:58:59Z", GoVersion:"go1.17.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.23.6", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2022-04-14T14:50:46Z", GoVersion:"go1.17.9", Compiler:"gc", Platform:"linux/amd64"}
```

</details>


### Cloud provider

<details>
On prem environment
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here
```

</details>


### Install tools

<details>
kubeadm
</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>