Repository: kubernetes
Issue: static pod stuck in "Waiting for volumes to unmount for pod" for a longtime on single node by chance
Labels: kind/bug,sig/storage,sig/node,triage/accepted

### What happened?

1. when update static pod（kube-apiserver）yaml, the static pod maybe stuck for 20 minutes to 2 hours, some logs show as follow
```
Aug 15 20:18:20 node1 hyperkube[1974327]: I0815 20:18:20.604518 1974327 actual_state_of_world.go:973] "Pod mounted volumes" uniquePodName=9d45620a-ae62-4ee3-bdb7-139998904a99 mountedVolume=[{MountedVolume:{PodName:9d45620a-ae62-4ee3-bdb7-139998904a99 VolumeName:kubernetes.io/host-path/9d45620a-ae62-4ee3-bdb7-139998904a99-positions InnerVolumeSpecName:positions OuterVolumeSpecName:positions PluginName:kubernetes.io/host-path PodUID:9d45620a-ae62-4ee3-bdb7-139998904a99 Mounter:0xc006092940 BlockVolumeMapper:<nil> VolumeGidValue: VolumeSpec:0xc0032f7a70 DeviceMountPath: SELinuxMountContext:}} {MountedVolume:{PodName:9d45620a-ae62-4ee3-bdb7-139998904a99 VolumeName:kubernetes.io/projected/9d45620a-ae62-4ee3-bdb7-139998904a99-kube-api-access-wd5vj InnerVolumeSpecName:kube-api-access-wd5vj OuterVolumeSpecName:kube-api-access-wd5vj PluginName:kubernetes.io/projected PodUID:9d45620a-ae62-4ee3-bdb7-139998904a99 Mounter:0xc005345100 BlockVolumeMapper:<nil> VolumeGidValue: VolumeSpec:0xc0032f7b48 DeviceMountPath: SELinuxMountContext:}} {MountedVolume:{PodName:9d45620a-ae62-4ee3-bdb7-139998904a99 VolumeName:kubernetes.io/host-path/9d45620a-ae62-4ee3-bdb7-139998904a99-var-log-test-apiserver InnerVolumeSpecName:var-log-test-apiserver OuterVolumeSpecName:var-log-test-apiserver PluginName:kubernetes.io/host-path PodUID:9d45620a-ae62-4ee3-bdb7-139998904a99 Mounter:0xc003817600 BlockVolumeMapper:<nil> VolumeGidValue: VolumeSpec:0xc0032f7ab8 DeviceMountPath: SELinuxMountContext:}} {MountedVolume:{PodName:9d45620a-ae62-4ee3-bdb7-139998904a99 VolumeName:kubernetes.io/host-path/9d45620a-ae62-4ee3-bdb7-139998904a99-var-log-kube-apiserver InnerVolumeSpecName:var-log-kube-apiserver OuterVolumeSpecName:var-log-kube-apiserver PluginName:kubernetes.io/host-path PodUID:9d45620a-ae62-4ee3-bdb7-139998904a99 Mounter:0xc005f320c0 BlockVolumeMapper:<nil> VolumeGidValue: VolumeSpec:0xc0032f7ae8 DeviceMountPath: SELinuxMountContext:}} {MountedVolume:{PodName:9d45620a-ae62-4ee3-bdb7-139998904a99 VolumeName:kubernetes.io/host-path/9d45620a-ae62-4ee3-bdb7-139998904a99-var-log-oauth-apiserver InnerVolumeSpecName:var-log-oauth-apiserver OuterVolumeSpecName:var-log-oauth-apiserver PluginName:kubernetes.io/host-path PodUID:9d45620a-ae62-4ee3-bdb7-139998904a99 Mounter:0xc00ae8edc0 BlockVolumeMapper:<nil> VolumeGidValue: VolumeSpec:0xc0032f7b18 DeviceMountPath: SELinuxMountContext:}}]
Aug 15 20:18:29 node1 hyperkube[1974327]: I0815 20:18:29.606947 1974327 file.go:202] "Reading config file" path="/etc/kubernetes/manifests/kube-apiserver-pod.yaml"
Aug 15 20:18:29 node1 hyperkube[1974327]: I0815 20:18:29.607559 1974327 common.go:69] "Generated UID" pod="test-kube-apiserver/kube-apiserver" podUID=d6c287e864be7616ad8256abb9e925b1 source="/etc/kubernetes/manifests/kube-apiserver-pod.yaml"
Aug 15 20:18:29 node1 hyperkube[1974327]: I0815 20:18:29.607570 1974327 common.go:73] "Generated pod name" pod="test-kube-apiserver/kube-apiserver-node1" podUID=d6c287e864be7616ad8256abb9e925b1 source="/etc/kubernetes/manifests/kube-apiserver-pod.yaml"
Aug 15 20:18:29 node1 hyperkube[1974327]: I0815 20:18:29.607581 1974327 common.go:78] "Set namespace for pod" pod="test-kube-apiserver/kube-apiserver-node1" source="/etc/kubernetes/manifests/kube-apiserver-pod.yaml"
Aug 15 20:18:31 node1 hyperkube[1974327]: I0815 20:18:31.381574 1974327 volume_manager.go:469] "Some volumes still mounted for pod" pod="test-kube-apiserver/kube-apiserver-node1" mountedVolumes=[audit-dir cert-dir resource-dir]
Aug 15 20:18:31 node1 hyperkube[1974327]: I0815 20:18:31.381589 1974327 kubelet.go:1976] "SyncTerminatedPod exit" pod="test-kube-apiserver/kube-apiserver-node1" podUID=bb8823c6e574ba7c1215633f4bf0f7d7
Aug 15 20:18:31 node1 hyperkube[1974327]: E0815 20:18:31.381600 1974327 pod_workers.go:1256] "Error syncing pod, skipping" err="mounted volumes=[audit-dir cert-dir resource-dir]: timed out waiting for the condition" pod="test-kube-apiserver/kube-apiserver-node1" podUID=bb8823c6e574ba7c1215633f4bf0f7d7
Aug 15 20:18:31 node1 hyperkube[1974327]: I0815 20:18:31.381628 1974327 pod_workers.go:1293] "Processing pod event done" pod="test-kube-apiserver/kube-apiserver-node1" podUID=bb8823c6e574ba7c1215633f4bf0f7d7 updateType="terminated"
Aug 15 20:18:31 node1 hyperkube[1974327]: I0815 20:18:31.381641 1974327 pod_workers.go:1188] "Processing pod event" pod="test-kube-apiserver/kube-apiserver-node1" podUID=bb8823c6e574ba7c1215633f4bf0f7d7 updateType="terminated"
Aug 15 20:18:31 node1 hyperkube[1974327]: I0815 20:18:31.508902 1974327 kubelet.go:2261] "SyncLoop (SYNC) pods" total=5 pods=[test-kube-apiserver/kube-apiserver-node1 test-doko/doko-ingress-proxy-tp5d5 test-cnv/hci-compute-fileserver-5874bc85df-gpph2 test-vnet-operator/vnet-operator-controller-manager-6cdcbdb49-bcrkq test-cluster-node-tuning-operator/cluster-node-tuning-operator-5bbb5fd999-r9nkg]
Aug 15 20:18:31 node1 hyperkube[1974327]: I0815 20:18:31.508930 1974327 pod_workers.go:931] "Notifying pod of pending update" pod="test-kube-apiserver/kube-apiserver-node1" podUID=d6c287e864be7616ad8256abb9e925b1 workType="sync"
Aug 15 20:18:31 node1 hyperkube[1974327]: I0815 20:18:31.509044 1974327 pod_workers.go:1138] "Pod cannot start yet" pod="test-kube-apiserver/kube-apiserver-node1" podUID=d6c287e864be7616ad8256abb9e925b1
Aug 15 20:18:31 node1 hyperkube[1974327]: I0815 20:18:31.517993 1974327 kubelet.go:1965] "SyncTerminatedPod enter" pod="test-kube-apiserver/kube-apiserver-node1" podUID=bb8823c6e574ba7c1215633f4bf0f7d7
Aug 15 20:18:31 node1 hyperkube[1974327]: I0815 20:18:31.518002 1974327 kubelet_pods.go:1605] "Generating pod status" pod="test-kube-apiserver/kube-apiserver-node1"
Aug 15 20:18:31 node1 hyperkube[1974327]: I0815 20:18:31.518048 1974327 kubelet_pods.go:1615] "Got phase for pod" pod="test-kube-apiserver/kube-apiserver-node1" oldPhase=Pending phase=Pending
Aug 15 20:18:31 node1 hyperkube[1974327]: I0815 20:18:31.518100 1974327 status_manager.go:532] "updateStatusInternal" version=1 pod="test-kube-apiserver/kube-apiserver-node1" podUID=bb8823c6e574ba7c1215633f4bf0f7d7 containers="(kube-apiserver state=waiting previous=<none>) (kube-apiserver-cert-regeneration-controller state=waiting previous=<none>) (kube-apiserver-cert-syncer state=waiting previous=<none>) (kube-apiserver-check-endpoints state=waiting previous=<none>) (kube-apiserver-insecure-readyz state=waiting previous=<none>) (setup state=waiting previous=<none>)"
Aug 15 20:18:31 node1 hyperkube[1974327]: I0815 20:18:31.518170 1974327 status_manager.go:552] "Status Manager: adding pod with new status to podStatusChannel" pod="test-kube-apiserver/kube-apiserver-node1" podUID=bb8823c6e574ba7c1215633f4bf0f7d7 statusVersion=1 status={Phase:Pending Conditions:[{Type:Initialized Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2024-08-15 20:18:31 +0800 CST Reason:ContainersNotInitialized Message:containers with incomplete status: [setup]} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2024-08-15 20:18:31 +0800 CST Reason:ContainersNotReady Message:containers with unready status: [kube-apiserver kube-apiserver-cert-syncer kube-apiserver-cert-regeneration-controller kube-apiserver-insecure-readyz kube-apiserver-check-endpoints]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2024-08-15 20:18:31 +0800 CST Reason:ContainersNotReady Message:containers with unready status: [kube-apiserver kube-apiserver-cert-syncer kube-apiserver-cert-regeneration-controller kube-apiserver-insecure-readyz kube-apiserver-check-endpoints]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2024-08-15 19:57:00 +0800 CST Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.253.166.72 PodIP:10.253.166.72 PodIPs:[{IP:10.253.166.72}] StartTime:2024-08-15 19:57:00 +0800 CST InitContainerStatuses:[{Name:setup State:{Waiting:&ContainerStateWaiting{Reason:PodInitializing,Message:,} Running:nil Terminated:nil} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:image.test.io/test-ceaedge/ceaedge@sha256:b161fe4e21adfa95e7620c778536dd656a9437482d89fa44941f05c0e101fe28 ImageID: ContainerID: Started:<nil>}] ContainerStatuses:[{Name:kube-apiserver State:{Waiting:&ContainerStateWaiting{Reason:PodInitializing,Message:,} Running:nil Terminated:nil} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:image.test.io/test-ceaedge/ceaedge@sha256:b161fe4e21adfa95e7620c778536dd656a9437482d89fa44941f05c0e101fe28 ImageID: ContainerID: Started:0xc011eba16d} {Name:kube-apiserver-cert-regeneration-controller State:{Waiting:&ContainerStateWaiting{Reason:PodInitializing,Message:,} Running:nil Terminated:nil} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:image.test.io/test-ceaedge/ceaedge@sha256:382da2faa65c152f4d930f42f6d729fe219d337b243584f8ae13788829730024 ImageID: ContainerID: Started:0xc011eba16e} {Name:kube-apiserver-cert-syncer State:{Waiting:&ContainerStateWaiting{Reason:PodInitializing,Message:,} Running:nil Terminated:nil} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:image.test.io/test-ceaedge/ceaedge@sha256:382da2faa65c152f4d930f42f6d729fe219d337b243584f8ae13788829730024 ImageID: ContainerID: Started:0xc011eba16f} {Name:kube-apiserver-check-endpoints State:{Waiting:&ContainerStateWaiting{Reason:PodInitializing,Message:,} Running:nil Terminated:nil} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:image.test.io/test-ceaedge/ceaedge@sha256:382da2faa65c152f4d930f42f6d729fe219d337b243584f8ae13788829730024 ImageID: ContainerID: Started:0xc011eba1a0} {Name:kube-apiserver-insecure-readyz State:{Waiting:&ContainerStateWaiting{Reason:PodInitializing,Message:,} Running:nil Terminated:nil} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:image.test.io/test-ceaedge/ceaedge@sha256:382da2faa65c152f4d930f42f6d729fe219d337b243584f8ae13788829730024 ImageID: ContainerID: Started:0xc011eba1a1}] QOSClass:Burstable EphemeralContainerStatuses:[]}
Aug 15 20:18:31 node1 hyperkube[1974327]: I0815 20:18:31.518183 1974327 volume_manager.go:448] "Waiting for volumes to unmount for pod" pod="test-kube-apiserver/kube-apiserver-node1"
Aug 15 20:18:35 node1 hyperkube[1974327]: I0815 20:18:35.509137 1974327 actual_state_of_world.go:973] "Pod mounted volumes" uniquePodName=9d45620a-ae62-4ee3-bdb7-139998904a99 mountedVolume=[{MountedVolume:{PodName:9d45620a-ae62-4ee3-bdb7-139998904a99 VolumeName:kubernetes.io/host-path/9d45620a-ae62-4ee3-bdb7-139998904a99-positions InnerVolumeSpecName:positions OuterVolumeSpecName:positions PluginName:kubernetes.io/host-path PodUID:9d45620a-ae62-4ee3-bdb7-139998904a99 Mounter:0xc006092940 BlockVolumeMapper:<nil> VolumeGidValue: VolumeSpec:0xc0032f7a70 DeviceMountPath: SELinuxMountContext:}} {MountedVolume:{PodName:9d45620a-ae62-4ee3-bdb7-139998904a99 VolumeName:kubernetes.io/projected/9d45620a-ae62-4ee3-bdb7-139998904a99-kube-api-access-wd5vj InnerVolumeSpecName:kube-api-access-wd5vj OuterVolumeSpecName:kube-api-access-wd5vj PluginName:kubernetes.io/projected PodUID:9d45620a-ae62-4ee3-bdb7-139998904a99 Mounter:0xc005345100 BlockVolumeMapper:<nil> VolumeGidValue: VolumeSpec:0xc0032f7b48 DeviceMountPath: SELinuxMountContext:}} {MountedVolume:{PodName:9d45620a-ae62-4ee3-bdb7-139998904a99 VolumeName:kubernetes.io/host-path/9d45620a-ae62-4ee3-bdb7-139998904a99-var-log-test-apiserver InnerVolumeSpecName:var-log-test-apiserver OuterVolumeSpecName:var-log-test-apiserver PluginName:kubernetes.io/host-path PodUID:9d45620a-ae62-4ee3-bdb7-139998904a99 Mounter:0xc003817600 BlockVolumeMapper:<nil> VolumeGidValue: VolumeSpec:0xc0032f7ab8 DeviceMountPath: SELinuxMountContext:}} {MountedVolume:{PodName:9d45620a-ae62-4ee3-bdb7-139998904a99 VolumeName:kubernetes.io/host-path/9d45620a-ae62-4ee3-bdb7-139998904a99-var-log-kube-apiserver InnerVolumeSpecName:var-log-kube-apiserver OuterVolumeSpecName:var-log-kube-apiserver PluginName:kubernetes.io/host-path PodUID:9d45620a-ae62-4ee3-bdb7-139998904a99 Mounter:0xc005f320c0 BlockVolumeMapper:<nil> VolumeGidValue: VolumeSpec:0xc0032f7ae8 DeviceMountPath: SELinuxMountContext:}} {MountedVolume:{PodName:9d45620a-ae62-4ee3-bdb7-139998904a99 VolumeName:kubernetes.io/host-path/9d45620a-ae62-4ee3-bdb7-139998904a99-var-log-oauth-apiserver InnerVolumeSpecName:var-log-oauth-apiserver OuterVolumeSpecName:var-log-oauth-apiserver PluginName:kubernetes.io/host-path PodUID:9d45620a-ae62-4ee3-bdb7-139998904a99 Mounter:0xc00ae8edc0 BlockVolumeMapper:<nil> VolumeGidValue: VolumeSpec:0xc0032f7b18 DeviceMountPath: SELinuxMountContext:}}]

Aug 15 20:18:35 node1 hyperkube[1974327]: I0815 20:18:35.509810 1974327 actual_state_of_world.go:973] "Pod mounted volumes" uniquePodName=9d45620a-ae62-4ee3-bdb7-139998904a99 mountedVolume=[{MountedVolume:{PodName:9d45620a-ae62-4ee3-bdb7-139998904a99 VolumeName:kubernetes.io/host-path/9d45620a-ae62-4ee3-bdb7-139998904a99-var-log-test-apiserver InnerVolumeSpecName:var-log-test-apiserver OuterVolumeSpecName:var-log-test-apiserver PluginName:kubernetes.io/host-path PodUID:9d45620a-ae62-4ee3-bdb7-139998904a99 Mounter:0xc003817600 BlockVolumeMapper:<nil> VolumeGidValue: VolumeSpec:0xc0032f7ab8 DeviceMountPath: SELinuxMountContext:}} {MountedVolume:{PodName:9d45620a-ae62-4ee3-bdb7-139998904a99 VolumeName:kubernetes.io/host-path/9d45620a-ae62-4ee3-bdb7-139998904a99-var-log-kube-apiserver InnerVolumeSpecName:var-log-kube-apiserver OuterVolumeSpecName:var-log-kube-apiserver PluginName:kubernetes.io/host-path PodUID:9d45620a-ae62-4ee3-bdb7-139998904a99 Mounter:0xc005f320c0 BlockVolumeMapper:<nil> VolumeGidValue: VolumeSpec:0xc0032f7ae8 DeviceMountPath: SELinuxMountContext:}} {MountedVolume:{PodName:9d45620a-ae62-4ee3-bdb7-139998904a99 VolumeName:kubernetes.io/host-path/9d45620a-ae62-4ee3-bdb7-139998904a99-var-log-oauth-apiserver InnerVolumeSpecName:var-log-oauth-apiserver OuterVolumeSpecName:var-log-oauth-apiserver PluginName:kubernetes.io/host-path PodUID:9d45620a-ae62-4ee3-bdb7-139998904a99 Mounter:0xc00ae8edc0 BlockVolumeMapper:<nil> VolumeGidValue: VolumeSpec:0xc0032f7b18 DeviceMountPath: SELinuxMountContext:}} {MountedVolume:{PodName:9d45620a-ae62-4ee3-bdb7-139998904a99 VolumeName:kubernetes.io/host-path/9d45620a-ae62-4ee3-bdb7-139998904a99-positions InnerVolumeSpecName:positions OuterVolumeSpecName:positions PluginName:kubernetes.io/host-path PodUID:9d45620a-ae62-4ee3-bdb7-139998904a99 Mounter:0xc006092940 BlockVolumeMapper:<nil> VolumeGidValue: VolumeSpec:0xc0032f7a70 DeviceMountPath: SELinuxMountContext:}} {MountedVolume:{PodName:9d45620a-ae62-4ee3-bdb7-139998904a99 VolumeName:kubernetes.io/projected/9d45620a-ae62-4ee3-bdb7-139998904a99-kube-api-access-wd5vj InnerVolumeSpecName:kube-api-access-wd5vj OuterVolumeSpecName:kube-api-access-wd5vj PluginName:kubernetes.io/projected PodUID:9d45620a-ae62-4ee3-bdb7-139998904a99 Mounter:0xc005345100 BlockVolumeMapper:<nil> VolumeGidValue: VolumeSpec:0xc0032f7b48 DeviceMountPath: SELinuxMountContext:}}]
Aug 15 20:18:46 node1 hyperkube[1974327]: I0815 20:18:46.508306 1974327 kubelet.go:2261] "SyncLoop (SYNC) pods" total=5 pods=[test-cnv/vic-image-registry-5c64975fdd-jww6l test-cnv/yum-repo-87d95477-4jfn7 test-kube-apiserver/kube-apiserver-node1 default/grafana-6598f98dd-hkzmf test-cluster-node-tuning-operator/cluster-node-tuning-operator-5bbb5fd999-r9nkg]
Aug 15 20:18:46 node1 hyperkube[1974327]: I0815 20:18:46.508354 1974327 pod_workers.go:931] "Notifying pod of pending update" pod="test-kube-apiserver/kube-apiserver-node1" podUID=d6c287e864be7616ad8256abb9e925b1 workType="sync"
Aug 15 20:18:46 node1 hyperkube[1974327]: I0815 20:18:46.508440 1974327 pod_workers.go:1138] "Pod cannot start yet" pod="test-kube-apiserver/kube-apiserver-node1" podUID=d6c287e864be7616ad8256abb9e925b1
Aug 15 20:18:49 node1 hyperkube[1974327]: I0815 20:18:49.607098 1974327 file.go:202] "Reading config file" path="/etc/kubernetes/manifests/kube-apiserver-pod.yaml"
Aug 15 20:18:49 node1 hyperkube[1974327]: I0815 20:18:49.607697 1974327 common.go:69] "Generated UID" pod="test-kube-apiserver/kube-apiserver" podUID=d6c287e864be7616ad8256abb9e925b1 source="/etc/kubernetes/manifests/kube-apiserver-pod.yaml"
Aug 15 20:18:49 node1 hyperkube[1974327]: I0815 20:18:49.607708 1974327 common.go:73] "Generated pod name" pod="test-kube-apiserver/kube-apiserver-node1" podUID=d6c287e864be7616ad8256abb9e925b1 source="/etc/kubernetes/manifests/kube-apiserver-pod.yaml"

Aug 15 20:18:49 node1 hyperkube[1974327]: I0815 20:18:49.607717 1974327 common.go:78] "Set namespace for pod" pod="test-kube-apiserver/kube-apiserver-node1" source="/etc/kubernetes/manifests/kube-apiserver-pod.yaml"
Aug 15 20:19:01 node1 hyperkube[1974327]: I0815 20:19:01.508509 1974327 kubelet.go:2261] "SyncLoop (SYNC) pods" total=7 pods=[test-controller-manager/vm-scheduler-848d87568f-xtdw7 test-kube-controller-manager/kube-controller-manager-node1 test-doko/doko-agent-6hkd8 default/hp-volume-kpbnp test-kube-apiserver/kube-apiserver-node1 test-logging/logging-operator-548564d9d9-rcncd test-monitoring/alertmanager-main-0]
Aug 15 20:19:01 node1 hyperkube[1974327]: I0815 20:19:01.508578 1974327 pod_workers.go:931] "Notifying pod of pending update" pod="test-kube-apiserver/kube-apiserver-node1" podUID=d6c287e864be7616ad8256abb9e925b1 workType="sync"
Aug 15 20:19:01 node1 hyperkube[1974327]: I0815 20:19:01.508643 1974327 pod_workers.go:1138] "Pod cannot start yet" pod="test-kube-apiserver/kube-apiserver-node1" podUID=d6c287e864be7616ad8256abb9e925b1
Aug 15 20:19:09 node1 hyperkube[1974327]: I0815 20:19:09.607309 1974327 file.go:202] "Reading config file" path="/etc/kubernetes/manifests/kube-apiserver-pod.yaml"
Aug 15 20:19:09 node1 hyperkube[1974327]: I0815 20:19:09.607920 1974327 common.go:69] "Generated UID" pod="test-kube-apiserver/kube-apiserver" podUID=d6c287e864be7616ad8256abb9e925b1 source="/etc/kubernetes/manifests/kube-apiserver-pod.yaml"
Aug 15 20:19:09 node1 hyperkube[1974327]: I0815 20:19:09.607932 1974327 common.go:73] "Generated pod name" pod="test-kube-apiserver/kube-apiserver-node1" podUID=d6c287e864be7616ad8256abb9e925b1 source="/etc/kubernetes/manifests/kube-apiserver-pod.yaml"
Aug 15 20:19:09 node1 hyperkube[1974327]: I0815 20:19:09.607944 1974327 common.go:78] "Set namespace for pod" pod="test-kube-apiserver/kube-apiserver-node1" source="/etc/kubernetes/manifests/kube-apiserver-pod.yaml"
Aug 15 20:19:12 node1 hyperkube[1974327]: I0815 20:19:12.508790 1974327 kubelet.go:2261] "SyncLoop (SYNC) pods" total=2 pods=[test-kube-apiserver/kube-apiserver-node1 test-vnet-operator/vnet-operator-controller-manager-6cdcbdb49-bcrkq]
Aug 15 20:19:12 node1 hyperkube[1974327]: I0815 20:19:12.508808 1974327 pod_workers.go:931] "Notifying pod of pending update" pod="test-kube-apiserver/kube-apiserver-node1" podUID=d6c287e864be7616ad8256abb9e925b1 workType="sync"
Aug 15 20:19:12 node1 hyperkube[1974327]: I0815 20:19:12.508896 1974327 pod_workers.go:1138] "Pod cannot start yet" pod="test-kube-apiserver/kube-apiserver-node1" podUID=d6c287e864be7616ad8256abb9e925b1


Aug 15 20:19:27 node1 hyperkube[1974327]: I0815 20:19:27.508926 1974327 kubelet.go:2261] "SyncLoop (SYNC) pods" total=6 pods=[test-cnv/hci-compute-fileserver-5874bc85df-gpph2 test-vnet-operator/vnet-operator-controller-manager-6cdcbdb49-bcrkq test-base-image-registry-operator/base-image-registry-69ff9bd484-64527 test-node-label-operator/node-label-operator-6cd767b464-q4dz6 test-kube-controller-manager/kube-controller-manager-node1 test-kube-apiserver/kube-apiserver-node1]
Aug 15 20:19:27 node1 hyperkube[1974327]: I0815 20:19:27.509009 1974327 pod_workers.go:931] "Notifying pod of pending update" pod="test-kube-apiserver/kube-apiserver-node1" podUID=d6c287e864be7616ad8256abb9e925b1 workType="sync"
Aug 15 20:19:27 node1 hyperkube[1974327]: I0815 20:19:27.509099 1974327 pod_workers.go:1138] "Pod cannot start yet" pod="test-kube-apiserver/kube-apiserver-node1" podUID=d6c287e864be7616ad8256abb9e925b1
Aug 15 20:19:29 node1 hyperkube[1974327]: I0815 20:19:29.607409 1974327 file.go:202] "Reading config file" path="/etc/kubernetes/manifests/kube-apiserver-pod.yaml"
Aug 15 20:19:29 node1 hyperkube[1974327]: I0815 20:19:29.607998 1974327 common.go:69] "Generated UID" pod="test-kube-apiserver/kube-apiserver" podUID=d6c287e864be7616ad8256abb9e925b1 source="/etc/kubernetes/manifests/kube-apiserver-pod.yaml"
Aug 15 20:19:29 node1 hyperkube[1974327]: I0815 20:19:29.608009 1974327 common.go:73] "Generated pod name" pod="test-kube-apiserver/kube-apiserver-node1" podUID=d6c287e864be7616ad8256abb9e925b1 source="/etc/kubernetes/manifests/kube-apiserver-pod.yaml"
Aug 15 20:19:29 node1 hyperkube[1974327]: I0815 20:19:29.608017 1974327 common.go:78] "Set namespace for pod" pod="test-kube-apiserver/kube-apiserver-node1" source="/etc/kubernetes/manifests/kube-apiserver-pod.yaml"
Aug 15 20:19:40 node1 hyperkube[1974327]: I0815 20:19:40.511401 1974327 actual_state_of_world.go:973] "Pod mounted volumes" uniquePodName=9d45620a-ae62-4ee3-bdb7-139998904a99 mountedVolume=[{MountedVolume:{PodName:9d45620a-ae62-4ee3-bdb7-139998904a99 VolumeName:kubernetes.io/host-path/9d45620a-ae62-4ee3-bdb7-139998904a99-var-log-test-apiserver InnerVolumeSpecName:var-log-test-apiserver OuterVolumeSpecName:var-log-test-apiserver PluginName:kubernetes.io/host-path PodUID:9d45620a-ae62-4ee3-bdb7-139998904a99 Mounter:0xc003817600 BlockVolumeMapper:<nil> VolumeGidValue: VolumeSpec:0xc0032f7ab8 DeviceMountPath: SELinuxMountContext:}} {MountedVolume:{PodName:9d45620a-ae62-4ee3-bdb7-139998904a99 VolumeName:kubernetes.io/host-path/9d45620a-ae62-4ee3-bdb7-139998904a99-var-log-kube-apiserver InnerVolumeSpecName:var-log-kube-apiserver OuterVolumeSpecName:var-log-kube-apiserver PluginName:kubernetes.io/host-path PodUID:9d45620a-ae62-4ee3-bdb7-139998904a99 Mounter:0xc005f320c0 BlockVolumeMapper:<nil> VolumeGidValue: VolumeSpec:0xc0032f7ae8 DeviceMountPath: SELinuxMountContext:}} {MountedVolume:{PodName:9d45620a-ae62-4ee3-bdb7-139998904a99 VolumeName:kubernetes.io/host-path/9d45620a-ae62-4ee3-bdb7-139998904a99-var-log-oauth-apiserver InnerVolumeSpecName:var-log-oauth-apiserver OuterVolumeSpecName:var-log-oauth-apiserver PluginName:kubernetes.io/host-path PodUID:9d45620a-ae62-4ee3-bdb7-139998904a99 Mounter:0xc00ae8edc0 BlockVolumeMapper:<nil> VolumeGidValue: VolumeSpec:0xc0032f7b18 DeviceMountPath: SELinuxMountContext:}} {MountedVolume:{PodName:9d45620a-ae62-4ee3-bdb7-139998904a99 VolumeName:kubernetes.io/host-path/9d45620a-ae62-4ee3-bdb7-139998904a99-positions InnerVolumeSpecName:positions OuterVolumeSpecName:positions PluginName:kubernetes.io/host-path PodUID:9d45620a-ae62-4ee3-bdb7-139998904a99 Mounter:0xc006092940 BlockVolumeMapper:<nil> VolumeGidValue: VolumeSpec:0xc0032f7a70 DeviceMountPath: SELinuxMountContext:}} {MountedVolume:{PodName:9d45620a-ae62-4ee3-bdb7-139998904a99 VolumeName:kubernetes.io/projected/9d45620a-ae62-4ee3-bdb7-139998904a99-kube-api-access-wd5vj InnerVolumeSpecName:kube-api-access-wd5vj OuterVolumeSpecName:kube-api-access-wd5vj PluginName:kubernetes.io/projected PodUID:9d45620a-ae62-4ee3-bdb7-139998904a99 Mounter:0xc005345100 BlockVolumeMapper:<nil> VolumeGidValue: VolumeSpec:0xc0032f7b48 DeviceMountPath: SELinuxMountContext:}}]
Aug 15 20:19:40 node1 hyperkube[1974327]: I0815 20:19:40.512098 1974327 actual_state_of_world.go:973] "Pod mounted volumes" uniquePodName=9d45620a-ae62-4ee3-bdb7-139998904a99 mountedVolume=[{MountedVolume:{PodName:9d45620a-ae62-4ee3-bdb7-139998904a99 VolumeName:kubernetes.io/host-path/9d45620a-ae62-4ee3-bdb7-139998904a99-var-log-test-apiserver InnerVolumeSpecName:var-log-test-apiserver OuterVolumeSpecName:var-log-test-apiserver PluginName:kubernetes.io/host-path PodUID:9d45620a-ae62-4ee3-bdb7-139998904a99 Mounter:0xc003817600 BlockVolumeMapper:<nil> VolumeGidValue: VolumeSpec:0xc0032f7ab8 DeviceMountPath: SELinuxMountContext:}} {MountedVolume:{PodName:9d45620a-ae62-4ee3-bdb7-139998904a99 VolumeName:kubernetes.io/host-path/9d45620a-ae62-4ee3-bdb7-139998904a99-var-log-kube-apiserver InnerVolumeSpecName:var-log-kube-apiserver OuterVolumeSpecName:var-log-kube-apiserver PluginName:kubernetes.io/host-path PodUID:9d45620a-ae62-4ee3-bdb7-139998904a99 Mounter:0xc005f320c0 BlockVolumeMapper:<nil> VolumeGidValue: VolumeSpec:0xc0032f7ae8 DeviceMountPath: SELinuxMountContext:}} {MountedVolume:{PodName:9d45620a-ae62-4ee3-bdb7-139998904a99 VolumeName:kubernetes.io/host-path/9d45620a-ae62-4ee3-bdb7-139998904a99-var-log-oauth-apiserver InnerVolumeSpecName:var-log-oauth-apiserver OuterVolumeSpecName:var-log-oauth-apiserver PluginName:kubernetes.io/host-path PodUID:9d45620a-ae62-4ee3-bdb7-139998904a99 Mounter:0xc00ae8edc0 BlockVolumeMapper:<nil> VolumeGidValue: VolumeSpec:0xc0032f7b18 DeviceMountPath: SELinuxMountContext:}} {MountedVolume:{PodName:9d45620a-ae62-4ee3-bdb7-139998904a99 VolumeName:kubernetes.io/host-path/9d45620a-ae62-4ee3-bdb7-139998904a99-positions InnerVolumeSpecName:positions OuterVolumeSpecName:positions PluginName:kubernetes.io/host-path PodUID:9d45620a-ae62-4ee3-bdb7-139998904a99 Mounter:0xc006092940 BlockVolumeMapper:<nil> VolumeGidValue: VolumeSpec:0xc0032f7a70 DeviceMountPath: SELinuxMountContext:}} {MountedVolume:{PodName:9d45620a-ae62-4ee3-bdb7-139998904a99 VolumeName:kubernetes.io/projected/9d45620a-ae62-4ee3-bdb7-139998904a99-kube-api-access-wd5vj InnerVolumeSpecName:kube-api-access-wd5vj OuterVolumeSpecName:kube-api-access-wd5vj PluginName:kubernetes.io/projected PodUID:9d45620a-ae62-4ee3-bdb7-139998904a99 Mounter:0xc005345100 BlockVolumeMapper:<nil> VolumeGidValue: VolumeSpec:0xc0032f7b48 DeviceMountPath: SELinuxMountContext:}}]
```
2. restart kubelet will start the static pod quickly

3. related PR:
- https://github.com/kubernetes/kubernetes/pull/113145
- https://github.com/kubernetes/kubernetes/issues/117745
- https://github.com/kubernetes/kubernetes/pull/117751
- https://github.com/kubernetes/kubernetes/pull/116995

### What did you expect to happen?

The static pods will start soon when edit the static pod yaml

### How can we reproduce it (as minimally and precisely as possible)?

this happend by chance , now can be reproduced in K8s  1.25 and 1.29  on single node

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version

# paste output here
```
k8s 1.25
</details>


### Cloud provider

<details>

</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>