Repository: kubernetes
Issue: Pods that consume "devices" via Device Plugin always fail when Node reboots even if it implements `plugins_registry` interface
Labels: kind/bug,sig/node,triage/accepted,wg/device-management

### What happened?

When node reboots, Pods that consume **devices** via Device Plugin always fail.

Even if I implement device plugin with ability to use `/var/lib/kubelet/plugins_registry` directory, it fails too.

Below is my flow of inspection to understand current implementation of `kubelet` related to the device plugin interface.

***

I tested below cases:
- Only `kubelet` restarts
- Only `containerd` restarts
- Both `kubelet` and `containerd` restart

All of 3 cases do not make any error. It only happens when node reboots.

***

Device Health status is provided using `healthDevices`.
- [When `kubelet` starts, it restores from checkpoint, but both `healthDevices` and `unhealthyDevices` set to empty set.](https://github.com/kubernetes/kubernetes/blob/release-1.31/pkg/kubelet/cm/devicemanager/manager.go#L506-L532)
- [`healthyDevices` will be supplied through this code block.](https://github.com/kubernetes/kubernetes/blob/release-1.31/pkg/kubelet/cm/devicemanager/manager.go#L259-L318)
    - Right now, there is no other code block that supplies `healthyDevices`.

About Device Plugin Registration
- [Device Plugin can be registered to `kubelet` through this code block.](https://github.com/kubernetes/kubernetes/blob/release-1.31/pkg/kubelet/cm/devicemanager/plugin/v1beta1/handler.go#L41-L44)
    - Plugin sends request to `kubelet`.
- After registration, dedicated client will be created and [run at another goroutine](https://github.com/kubernetes/kubernetes/blob/release-1.31/pkg/kubelet/cm/devicemanager/plugin/v1beta1/handler.go#L78-L80).
- [As I mentioned above, `healthyDevices` can only be injected through this code block](https://github.com/kubernetes/kubernetes/blob/release-1.31/pkg/kubelet/cm/devicemanager/manager.go#L271-L318).
    - However, initialization always happens at the beginning of this code block.
- To summarize it, **there is no way to supply health devices before device plugin registration**.

***

About `PluginManager` and `plugins_registry`
> `pluginmanager` runs a set of asynchronous loops that figure out which plugins need to be registered/unregistered based on this node and makes it so.
- [`plugins_registry` is used at `PluginManager`. The first usage can be found here](https://github.com/kubernetes/kubernetes/blob/release-1.31/pkg/kubelet/kubelet.go#L848-L851).
- [In `PluginManager`, registration of Device Plugin watcher (NOT the DEVICE PLUGIN) happens.](https://github.com/kubernetes/kubernetes/blob/release-1.31/pkg/kubelet/kubelet.go#L1581-L1582)
    - [For device plugin, this code block implements Device Plugin watcher interface.](https://github.com/kubernetes/kubernetes/blob/release-1.31/pkg/kubelet/cm/devicemanager/plugin/v1beta1/server.go#L40-L45)
- [After registration of watcher, `PluginManager` runs.](https://github.com/kubernetes/kubernetes/blob/release-1.31/pkg/kubelet/pluginmanager/plugin_manager.go#L108-L124)
    - Inside that code block, we can find `go pm.reconciler.Run(stopCh)`. We can jump to [this section](https://github.com/kubernetes/kubernetes/blob/release-1.31/pkg/kubelet/pluginmanager/reconciler/reconciler.go#L85-L91).
- [In the `reconcile()`](https://github.com/kubernetes/kubernetes/blob/release-1.31/pkg/kubelet/pluginmanager/reconciler/reconciler.go#L111-L165), it syncs `desiredStateOfWorld` and `actualStateOfWorld`.
- [In this code block, all unix sockets exist in the `/var/lib/kubelet/plugins_registry` are managed through `AddOrUpdatePlugin()` method.](https://github.com/kubernetes/kubernetes/blob/release-1.31/pkg/kubelet/pluginmanager/pluginwatcher/plugin_watcher.go#L187-L200)

All of above steps are executed BEFORE entering `syncLoop()`.

***

So my thought was, If I implement device plugin to deal with `plugins_registry`, it can be registered to `kubelet` before other pod run.
- [`GetInfo(...)`](https://github.com/kubernetes/kubernetes/blob/release-1.31/pkg/kubelet/pluginmanager/operationexecutor/operation_generator.go#L91) and [`NotifyRegistrationStatus(...)`](https://github.com/kubernetes/kubernetes/blob/release-1.31/pkg/kubelet/pluginmanager/operationexecutor/operation_generator.go#L166) GRPC interface have to be implemented.

However, even if I implemented above things, same problem happens.
- When node reboots, all containers are in stopped state.
- `kubelet` is in charge of re-run stopped containers in that node.
- Before entering `syncLoop()`, 2 GRPC interface will be called.
    - However, because device plugin pod is in stopped state, above GRPC call fails.
    - Due to failure of above GRPC call, device plugin registration before entering `syncLoop()` also fails too.

To solve this problem, I have to run the device plugin using other method (e.g. systemd daemon), that can't be managed by kubelet.

### What did you expect to happen?

To be honest, I know the current problem but don't know the nice solution.

### How can we reproduce it (as minimally and precisely as possible)?

Prepare device plugin (e.g. [`NVIDIA/k8s-device-plugin`](https://github.com/NVIDIA/k8s-device-plugin)) and run any Pod consumes that device (e.g. NVIDIA GPU).

After Pod successfully created and run without any problem, DO node reboot (e.g. just type `sudo reboot`).

After node reboots, you can see error message like below.
```
Allocate failed due to no healthy devices present; cannot allocate unhealthy devices nvidia.com/gpu, which is unexpected
```

### Anything else we need to know?

This issue has been discussed in [this Slack thread](https://kubernetes.slack.com/archives/C0BP8PW9G/p1709610711920639).


### Kubernetes version

<details>

```console
$ kubectl version
Client Version: v1.30.4
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.5
```

</details>


### Cloud provider

<details>
Bare Metal
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

$ uname -a
Linux MS03-CEO-008 6.8.0-45-generic #45~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Sep 11 15:25:05 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
```

</details>


### Install tools

<details>
kubeadm version: &version.Info{Major:"1", Minor:"30", GitVersion:"v1.30.5", GitCommit:"74e84a90c725047b1328ff3d589fedb1cb7a120e", GitTreeState:"clean", BuildDate:"2024-09-12T00:17:07Z", GoVersion:"go1.22.6", Compiler:"gc", Platform:"linux/amd64"}
</details>


### Container runtime (CRI) and version (if applicable)

<details>
containerd containerd.io 1.7.22 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>