Repository: kubernetes
Issue: failed to assign quota after kubelet was restarted
Labels: kind/bug,sig/storage,needs-triage

### What happened?

After kubelet was restarted, kubelet starts to output quota related errors and the total count of entries in `/etc/projects` and `/etc/projid` decreased.

```console
I0416 17:38:13.086019  157568 empty_dir.go:306] Set quota on /var/lib/kubelet/pods/e5149231-3a81-41e0-87be-579846f6caea/volumes/kubernetes.io~configmap/config failed assign quota FAILED exit status 1
I0416 17:38:13.086051  157568 operation_generator.go:838] "MountVolume.markVolumeErrorState leaving volume uncertain" volumeName=kubernetes.io/configmap/e5149231-3a81-41e0-87be-579846f6caea-config
E0416 17:38:13.086130  157568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e5149231-3a81-41e0-87be-579846f6caea-config podName:e5149231-3a81-41e0-87be-579846f6caea nodeName:}" failed. No retries permitted until 2024-04-16 17:40:15.086106798 +0800 CST m=+376.261198810 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "config" (UniqueName: "kubernetes.io/configmap/e5149231-3a81-41e0-87be-579846f6caea-config") pod "xxx-yyy" (UID: "e5149231-3a81-41e0-87be-579846f6caea") : assign quota FAILED exit status 1
I0416 17:38:13.086156  157568 event.go:294] "Event occurred" object="xxx/yyy" fieldPath="" kind="Pod" apiVersion="v1" type="Warning" reason="FailedMount" message="MountVolume.SetUp failed for volume \"config\" : assign quota FAILED exit status 1"
```

Similar issues: #115309

### What did you expect to happen?

-

### How can we reproduce it (as minimally and precisely as possible)?

1. ext4 fs with quota enabled and enforced `tune2fs -O project -Q prjquota /dev/xxx; mount -o prjquota ...`
2. enable featuregate `LocalStorageCapacityIsolationFSQuotaMonitoring` in kubelet config
3. register this node to k8s (start kubelet): works well
4. restart kubelet: reproduced

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.7", GitCommit:"84e1fc493a47446df2e155e70fca768d2653a398", GitTreeState:"clean", BuildDate:"2023-07-19T12:23:27Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.7", GitCommit:"84e1fc493a47446df2e155e70fca768d2653a398", GitTreeState:"clean", BuildDate:"2023-07-19T12:16:45Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}
```

</details>


### Cloud provider

<details>
IDC
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
NAME="AlmaLinux"
VERSION="9.3 (Shamrock Pampas Cat)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.3"
PLATFORM_ID="platform:el9"
PRETTY_NAME="AlmaLinux 9.3 (Shamrock Pampas Cat)"
ANSI_COLOR="0;34"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:almalinux:almalinux:9::baseos"
HOME_URL="https://almalinux.org/"
DOCUMENTATION_URL="https://wiki.almalinux.org/"
BUG_REPORT_URL="https://bugs.almalinux.org/"

ALMALINUX_MANTISBT_PROJECT="AlmaLinux-9"
ALMALINUX_MANTISBT_PROJECT_VERSION="9.3"
REDHAT_SUPPORT_PRODUCT="AlmaLinux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.3"

$ uname -a
Linux xxx 5.14.0-362.24.2.el9_3.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Mar 30 14:11:54 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>
containerd 1.6.31
</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>