Repository: kubernetes
Issue: kubelet: Device plugin fails to connect to new instance of plugin if previous one is terminated
Labels: kind/bug,sig/node,triage/accepted

### What happened?

The kubelet fails to connect to the device plugin if a previously running device plugin was running on the node with the same resource name.

Kubelets can get into a situation of running two pods advertising the same device plugin for a brief short period of time. For example, consider the following situation of a vendor hypothetically "upgrading" the device plugin:

1. kubelet runs`first-device-plugin` advertising resource `example.com/resource`
2. device vendor upgrades device-plugin to `second-plugin` and user deploys it
3. kubelet switches to use `second-device-plugin` advertising resource `example.com/resource`
4. user deletes `first-device-plugin`

Prior to v1.25.0, kubelet would handle this situation, and on step 4, disconnect of `first-device-plugin` would ensure that `first-device-plugin` would be disconnected, and maintain connection to `second-device-plugin` (since it was most recently connected).

This regressed as part of https://github.com/kubernetes/kubernetes/pull/109016 where as part of the deletion of the `first-device-plugin` (step 4), kubelet would disconnect from "the most recently" connected plugin, namely `second-device-plugin`, and thus would get into a situation where neither the first or second device plugin would be connected. This would result in kubelet failing to adverting resources for the device. This is because of a bug where kubelet only stores the state of the most recently connected plugin and on disconnect, disconnects the most recently connected plugin.

### What did you expect to happen?

I expected kubelet to either maintain the same behavior as pre 1.25: when deleting a device plugin pod, to disconnect from the device plugin that was deleted rather then the most recently connected plugin.

I expect that when a new device plugin connects and the device manager registers it, if an existing device plugin is already running kubelet should handle this situation by either rejecting the registration request or disconnecting from the existing device plugin and connecting to the new one. 





### How can we reproduce it (as minimally and precisely as possible)?

Reproduced on master and v1.27.3 in kind:

Create kind cluster (using single worker node v1.27.3):

```
kind_config="$(cat << EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  ipFamily: ipv4
nodes:
# the control plane node
- role: control-plane
- role: worker
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        v: "6"
        read-only-port: "10255"
EOF
)"

$ kind create cluster --config <(printf '%s\n' "${kind_config}") --image kindest/node:v1.27.3
```

(one kubelet, kind-worker for repro)
```
$ kubectl get nodes
NAME                 STATUS   ROLES           AGE   VERSION
kind-control-plane   Ready    control-plane   27s   v1.27.3
kind-worker          Ready    <none>          8s    v1.27.3
```

Verify initial node capacity and node allocatable (device is not advertised as expected, since device plugin isn't installed):

```
$ kubectl get node kind-worker -o json | jq '.status| {allocatable: .allocatable, capacity: .capacity}'
{
  "allocatable": {
    "cpu": "12",
    "ephemeral-storage": "421481144Ki",
    "hugepages-1Gi": "0",
    "hugepages-2Mi": "0",
    "memory": "65425280Ki",
    "pods": "110"
  },
  "capacity": {
    "cpu": "12",
    "ephemeral-storage": "421481144Ki",
    "hugepages-1Gi": "0",
    "hugepages-2Mi": "0",
    "memory": "65425280Ki",
    "pods": "110"
  }
}
```

Apply the first device plugin pod.

```
https://gist.github.com/bobbypage/897296ad4726c0a7b09ea1a6bba47062
$ kubectl apply -f first-device-plugin.yaml
pod/first-device-plugin created

```

Verify it went to running.

```
$ kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=kind-worker
NAMESPACE     NAME                  READY   STATUS    RESTARTS   AGE   IP            NODE          NOMINATED NODE   READINESS GATES
default       first-device-plugin   1/1     Running   0          8s    10.244.1.2    kind-worker   <none>           <none>
kube-system   kindnet-42ww9         1/1     Running   0          39s   192.168.8.3   kind-worker   <none>           <none>
kube-system   kube-proxy-k7pgh      1/1     Running   0          39s   192.168.8.3   kind-worker   <none>           <none>
```

The new resource `example.com/resource`, is advertised 

```

$ kubectl get node kind-worker -o json | jq '.status| {allocatable: .allocatable, capacity: .capacity}'
{
  "allocatable": {
    "cpu": "12",
    "ephemeral-storage": "421481144Ki",
    "example.com/resource": "2",
    "hugepages-1Gi": "0",
    "hugepages-2Mi": "0",
    "memory": "65425280Ki",
    "pods": "110"
  },
  "capacity": {
    "cpu": "12",
    "ephemeral-storage": "421481144Ki",
    "example.com/resource": "2",
    "hugepages-1Gi": "0",
    "hugepages-2Mi": "0",
    "memory": "65425280Ki",
    "pods": "110"
  }
}
```

Create the second device plugin pod. Consider a scenario where this is a "new version" of the device plugin.

```
# https://gist.github.com/bobbypage/18d526daf4c1b72bdf659e65ed3890d2
$ kubectl apply -f second-device-plugin.yaml
pod/second-device-plugin created
```

Verify the pod went to running.

```
$ kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=kind-worker
NAMESPACE     NAME                   READY   STATUS    RESTARTS   AGE   IP            NODE          NOMINATED NODE   READINESS GATES
default       first-device-plugin    1/1     Running   0          42s   10.244.1.2    kind-worker   <none>           <none>
default       second-device-plugin   1/1     Running   0          11s   10.244.1.3    kind-worker   <none>           <none>
kube-system   kindnet-42ww9          1/1     Running   0          73s   192.168.8.3   kind-worker   <none>           <none>
kube-system   kube-proxy-k7pgh       1/1     Running   0          73s   192.168.8.3   kind-worker   <none>           <none>
```

`example.com/resource` is still advertised as 2. 

```
$ kubectl get node kind-worker -o json | jq '.status| {allocatable: .allocatable, capacity: .capacity}'
{
  "allocatable": {
    "cpu": "12",
    "ephemeral-storage": "421481144Ki",
    "example.com/resource": "2",
    "hugepages-1Gi": "0",
    "hugepages-2Mi": "0",
    "memory": "65425280Ki",
    "pods": "110"
  },
  "capacity": {
    "cpu": "12",
    "ephemeral-storage": "421481144Ki",
    "example.com/resource": "2",
    "hugepages-1Gi": "0",
    "hugepages-2Mi": "0",
    "memory": "65425280Ki",
    "pods": "110"
  }
}
```

Now, delete the first device plugin

```
$ kubectl delete pod first-device-plugin
pod "first-device-plugin" deleted
```

`example.com/resource` allocable dropped to zero and will not recover!

```
$ kubectl get node kind-worker -o json | jq '.status| {allocatable: .allocatable, capacity: .capacity}'
{
  "allocatable": {
    "cpu": "12",
    "ephemeral-storage": "421481144Ki",
    "example.com/resource": "0",
    "hugepages-1Gi": "0",
    "hugepages-2Mi": "0",
    "memory": "65425280Ki",
    "pods": "110"
  },
  "capacity": {
    "cpu": "12",
    "ephemeral-storage": "421481144Ki",
    "example.com/resource": "2",
    "hugepages-1Gi": "0",
    "hugepages-2Mi": "0",
    "memory": "65425280Ki",
    "pods": "110"
  }
}
```

The second device plugin pod is still running, but the resource is no longer advertised. 

```
$ kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=kind-worker
NAMESPACE     NAME                   READY   STATUS    RESTARTS   AGE    IP            NODE          NOMINATED NODE   READINESS GATES
default       second-device-plugin   1/1     Running   0          63s    10.244.1.3    kind-worker   <none>           <none>
kube-system   kindnet-42ww9          1/1     Running   0          2m5s   192.168.8.3   kind-worker   <none>           <none>
kube-system   kube-proxy-k7pgh       1/1     Running   0          2m5s   192.168.8.3   kind-worker   <none>           <none>
```

Kubelet logs - https://gist.github.com/bobbypage/04424608bf89bb1a7043cf50dadf28e5
Containerd logs - https://gist.github.com/bobbypage/351c60f690687067a631a6fcb7ba0d77



### Anything else we need to know?

I verified that this regressed as part of v1.25.0, in commit 05e3919b4536560b38dae9ae54f47fa40840df2b this repro'd, but in previous commit, 2845122e14e4fa1291c3f6362c96236914ba33fb, it did not repro. 

### Kubernetes version

<details>

```console
$ kubectl version
# paste output here
```

</details>


### Cloud provider

<details>

</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>