Repository: kubernetes
Issue: kubelet crash loop panic with SIGSEGV
Labels: kind/bug,priority/important-soon,sig/node,lifecycle/rotten,needs-triage

### What happened?

I was running some pods on my node as usual.
At some point (23:41:35.127989 in the log), the kubelet crashed and then went into a crash loop.

Log showing the first crash and several thereafter: [kubelet-crash.txt](https://github.com/kubernetes/kubernetes/files/15312195/kubelet-crash.txt)


The errors I see shortly before the crash are:
```
projected.go:292] Couldn't get configMap default/kube-root-ca.crt: object "default"/"kube-root-ca.crt" not registered
projected.go:198] Error preparing data for projected volume kube-api-access-lwp5v for pod default/run-fluid-wtw5n-99kwf: object "default"/"kube-root-ca.crt" not registered
nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6cfc866-a5b2-4e16-9b83-884de8552d45-kube-api-access-lwp5v podName:e6cfc866-a5b2-4e16-9b83-884de8552d45 nodeName:}" failed. No retries permitted until 2024-05-13 23:41:38.8756138 +0000 UTC m=+9176.959811707 (durationBeforeRet
ry 2m2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lwp5v" (UniqueName: "kubernetes.io/projected/e6cfc866-a5b2-4e16-9b83-884de8552d45-kube-api-access-lwp5v") pod "run-fluid-wtw5n-99kwf" (UID: "e6cfc866-a5b2-4e16-9b83-884de8552d45") : object "default"/"kube-root-ca.crt" not registered
```

Stack trace from the first crash (later stack traces are a bit different):
```
goroutine 401 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x3d005c0?, 0x6d89410})
        vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x99
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x480e338?})
        vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x75
panic({0x3d005c0, 0x6d89410})
        /usr/local/go/src/runtime/panic.go:884 +0x213
errors.As({0x12, 0x0}, {0x3eac4c0?, 0xc0014e6590?})
        /usr/local/go/src/errors/wrap.go:109 +0x215
k8s.io/apimachinery/pkg/util/net.IsConnectionReset(...)
        vendor/k8s.io/apimachinery/pkg/util/net/util.go:45
k8s.io/client-go/rest.(*Request).request.func2(0x6d2e8a0?, {0x12, 0x0})
        vendor/k8s.io/client-go/rest/request.go:1007 +0x79
k8s.io/client-go/rest.IsRetryableErrorFunc.IsErrorRetryable(...)
        vendor/k8s.io/client-go/rest/with_retry.go:43
k8s.io/client-go/rest.(*withRetry).IsNextRetry(0xc001587f40, {0x0?, 0x0?}, 0x0?, 0xc001ef5f00, 0xc001bb1560, {0x12, 0x0}, 0x480e328)
        vendor/k8s.io/client-go/rest/with_retry.go:169 +0x170
k8s.io/client-go/rest.(*Request).request.func3(0xc001bb1560, 0xc002913af8, {0x4c44840?, 0xc001587f40?}, 0x0?, 0x0?, 0x39efc40?, {0x12?, 0x0?}, 0x480e328)
        vendor/k8s.io/client-go/rest/request.go:1042 +0xba
k8s.io/client-go/rest.(*Request).request(0xc00199f200, {0x4c42e00, 0xc00227b0e0}, 0x2?)
        vendor/k8s.io/client-go/rest/request.go:1048 +0x4e5
k8s.io/client-go/rest.(*Request).Do(0xc00199f200, {0x4c42dc8, 0xc000196010})
        vendor/k8s.io/client-go/rest/request.go:1063 +0xc9
k8s.io/client-go/kubernetes/typed/core/v1.(*nodes).Get(0xc001db6740, {0x4c42dc8, 0xc000196010}, {0x7ffe6a355c42, 0x1d}, {{{0x0, 0x0}, {0x0, 0x0}}, {0x4c03920, ...}})
        vendor/k8s.io/client-go/kubernetes/typed/core/v1/node.go:77 +0x145
k8s.io/kubernetes/pkg/kubelet.(*Kubelet).tryUpdateNodeStatus(0xc000289400, {0x4c42dc8, 0xc000196010}, 0x44ead4?)
        pkg/kubelet/kubelet_node_status.go:561 +0xf6
k8s.io/kubernetes/pkg/kubelet.(*Kubelet).updateNodeStatus(0xc000289400, {0x4c42dc8, 0xc000196010})
        pkg/kubelet/kubelet_node_status.go:536 +0xfc
k8s.io/kubernetes/pkg/kubelet.(*Kubelet).syncNodeStatus(0xc000289400)
        pkg/kubelet/kubelet_node_status.go:526 +0x105
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc002913f28?)
        vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x4c19ae0, 0xc0008f2000}, 0x1, 0xc000180360)
        vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x2540be400, 0x3fa47ae147ae147b, 0x0?, 0x0?)
        vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x89
created by k8s.io/kubernetes/pkg/kubelet.(*Kubelet).Run
        pkg/kubelet/kubelet.go:1606 +0x58a
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x1a pc=0x47a935]
```

### What did you expect to happen?

Ideally no crash, but at least a useful error message.

### How can we reproduce it (as minimally and precisely as possible)?

Really not sure.

### Anything else we need to know?

_No response_

### Kubernetes version

Client Version: v1.28.6
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.6

### Cloud provider

Bare metal.

### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

kubespray

### Container runtime (CRI) and version (if applicable)

containerd

containerd github.com/containerd/containerd v1.7.13 7c3aca7a610df76212171d200ca3811ff6096eb8

### Related plugins (CNI, CSI, ...) and versions (if applicable)

CNI = calico v3.26.4