Repository: kubernetes
Issue: Deletion of csi-node-plugin Pod causes driver entry to be removed from CSINode object; kube-scheduler schedules more than driver's allocatable
Labels: kind/bug,sig/storage,triage/accepted

### What happened?

With @plkokanov we hit in our environments the following issue multiple times:
```
% k -n shoot--foo--bar describe po etcd-main-0

Events:
  Type     Reason              Age                 From                     Message
  ----     ------              ----                ----                     -------
  Warning  FailedAttachVolume  35s (x47 over 80m)  attachdetach-controller  AttachVolume.Attach failed for volume "pv-shoot--shik--shak-1234" : rpc error: code = Internal desc = Attach volume /subscriptions/<omitted>/resourceGroups/shoot--shik--shak/providers/Microsoft.Compute/disks/pv-shoot--shik--shak-1234 to instance shoot--shik--shak-cpu-worker-etcd-z1-6c956-mhcw5 failed with Retriable: false, RetryAfter: 0s, HTTPStatusCode: 409, RawError: {\r
  "error": {\r
    "code": "OperationNotAllowed",\r
    "message": "The maximum number of data disks allowed to be attached to a VM of this size is 16.",\r
    "target": "dataDisks"\r
  }\r
}
```

When we checked, the CSINode object for the corresponding Node was reporting the correct volume attachment limit:
```
% k get csinode shoot--shik--shak-cpu-worker-etcd-z1-6c956-mhcw5 -o yaml

spec:
  drivers:
  # ...
  - allocatable:
      count: 16
    name: disk.csi.azure.com
    nodeID: shoot--shik--shak-cpu-worker-etcd-z1-6c956-mhcw5
    topologyKeys:
    - topology.disk.csi.azure.com/zone
    - topology.kubernetes.io/zone
```

According to the Node status and according to the VM state in Azure, it had already 16 disks attached.
```
% k get no shoot--shik--shak-cpu-worker-etcd-z1-6c956-mhcw5 -o json | jq '.status.volumesAttached | length'
16

% az vm show -g shoot--shik--shak --name shoot--shik--shak-cpu-worker-etcd-z1-6c956-mhcw5 | jq '.storageProfile.dataDisks | length'
16
```

kube-scheduler wrongly sheduled a Pod with a volume to the `shoot--shik--shak-cpu-worker-etcd-z1-6c956-mhcw5` Node as the Node already had its volume attachments limit reached.

We found that this happens due to csi-node-plugin Pod deletion. On deletion of this Pod, we see that the driver section from the CSINode object is removed and it is added only when the new csi-node-plugin Pod starts.

The corresponding handling in kube-scheduler's NodeVolumeLimits plugin is: https://github.com/kubernetes/kubernetes/blob/a7242fcff768658019f878cb691583dcbcfefb2d/pkg/scheduler/framework/plugins/nodevolumelimits/csi.go#L197-L201

If we fail to fetch the limit from the CSINode object, kube-scheduler let's the Pod to be scheduled.

### What did you expect to happen?

kube-scheduler to do not schedule Pods with volumes to Nodes that already have their volume attachment limit reached.

### How can we reproduce it (as minimally and precisely as possible)?

To reproduce the removal of the driver section after csi-node-plugin Pod deletion:

1. Start watching the CSINode object:
```
 k get csinode "$NODENAME" -w -o yaml > "~/csinode-$NODENAME.yaml"
```

2. In a new terminal window delete the csi-node-plugin Pod

3. Make sure that the CSINode object watch is as follows:
```yaml
apiVersion: storage.k8s.io/v1
kind: CSINode
metadata:
  name: node-1
spec:
  drivers:
  - allocatable:
      count: 8
    name: disk.csi.azure.com
    nodeID: node-1
    topologyKeys:
    - topology.disk.csi.azure.com/zone
    - topology.kubernetes.io/zone
---
apiVersion: storage.k8s.io/v1
kind: CSINode
metadata:
  name: node-1
spec:
  drivers: null
---
apiVersion: storage.k8s.io/v1
kind: CSINode
metadata:
  name: node-1
spec:
  drivers:
  - allocatable:
      count: 8
    name: disk.csi.azure.com
    nodeID: node-1
    topologyKeys:
    - topology.disk.csi.azure.com/zone
    - topology.kubernetes.io/zone
```

Note: The non-relevant fields are removed from the example output above.

You can see that until the new csi-node-plugin is scheduled and started, the CSINode object does not contain any information about the driver. We see that during that time kube-scheduler schedules new Pods with volumes to that Node.
The kube-scheduler plugin for respecting the Node volume limit is: https://github.com/kubernetes/kubernetes/blob/a7242fcff768658019f878cb691583dcbcfefb2d/pkg/scheduler/framework/plugins/nodevolumelimits/csi.go


### Anything else we need to know?

We use VPA to scale the csi-node-plugin Pod. That's why it can be evicted and restarted.

### Kubernetes version

<details>

```console
$ kubectl version

Server Version: v1.29.4
```

</details>


### Cloud provider

<details>
Azure and AWS
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>
Gardener
</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>
[azuredisk-csi-driver](https://github.com/kubernetes-sigs/azuredisk-csi-driver): v1.30.0
</details>