Repository: kubernetes
Issue: Kubelet crashed due to thread exhaustion
Labels: kind/bug,priority/backlog,sig/node,triage/accepted

### What happened?

Below was stack trace of crash:
```
kubelet[2767716]: runtime: program exceeds 10000-thread limit
kubelet[2767716]: fatal error: thread exhaustion
 kubelet[2767716]: runtime stack:
 kubelet[2767716]: runtime.throw({0x468f19a?, 0x46cba0?})
kubelet[2767716]:         /usr/lib/go-1.18/src/runtime/panic.go:992 +0x71
 kubelet[2767716]: runtime.checkmcount()
 kubelet[2767716]:         /usr/lib/go-1.18/src/runtime/proc.go:763 +0x8c
 kubelet[2767716]: runtime.mReserveID()
 kubelet[2767716]:         /usr/lib/go-1.18/src/runtime/proc.go:779 +0x36
 kubelet[2767716]: runtime.startm(0xc0001a8a00, 0x0)
```
From kubelet logs, Validated around 10000 threads, majority are with this stack trace
```
 kubelet[2767716]: goroutine 63494 [select, 39 minutes]:
 kubelet[2767716]: k8s.io/kubernetes/vendor/github.com/godbus/dbus/v5.(*signalChannelData).deferredDeliver(0xc00201c1e0, 0xc0008fa778?)
 kubelet[2767716]:         vendor/github.com/godbus/dbus/v5/default_handler.go:332 +0x69
 Aug 04 18:36:09  kubelet[2767716]: created by k8s.io/kubernetes/vendor/github.com/godbus/dbus/v5.(*signalChannelData).deliver
 kubelet[2767716]:         vendor/github.com/godbus/dbus/v5/default_handler.go:327 +0xea
```
```
Top shows kubelet using ~80GB of memory (as well as enormous CPU utilization)

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
2849677 root      20   0  264.2g  78.5g  64520 S  5983   3.9  10338:12 kubelet
```
### What did you expect to happen?

Kubelet not to crash and keep memory and thread footprint low


### How can we reproduce it (as minimally and precisely as possible)?

By injecting error in loginD or dbus logic, this should be reproducible.


### Anything else we need to know?

Traced down the kubelet where [these threads ](https://github.com/kubernetes/kubernetes/blob/master/vendor/github.com/godbus/dbus/v5/default_handler.go#L331)are getting created in Loop:
Reference to kubelet code: https://github.com/kubernetes/kubernetes/blob/release-1.24/pkg/kubelet/nodeshutdown/nodeshutdown_manager_linux.go#L181
 ```
// Start starts the node shutdown manager and will start watching the node for shutdown events.
func (m *managerImpl) Start() error {
	stop, err := m.start()
	if err != nil {
		return err
	}
	go func() {
		for { >>> NEVER ENDING FOR LOOP, Till m.start send stop()
			if stop != nil {
				<-stop
			}

			time.Sleep(dbusReconnectPeriod)
			klog.V(1).InfoS("Restarting watch for node shutdown events")
			stop, err = m.start()
			if err != nil {
				klog.ErrorS(err, "Unable to watch the node for shutdown events")
			}
		}
	}()

func (m *managerImpl) start() (chan struct{}, error) {
	systemBus, err := systemDbus() >>>> Created Dbus thread to deliver signal which we see in logs
	if err != nil {
		return nil, err
	}
	m.dbusCon = systemBus

	currentInhibitDelay, err := m.dbusCon.CurrentInhibitDelay()
	if err != nil {
		return nil, err        >>> If Error happens here or anywhere below, Dbus connections are not closed, thereby threads of dbus just keeps hanging
	}

      .....
       .....
	........
}
```
systemDbus() --> systemd.NewDBusCon() --> [dbus.SystemBus](https://github.com/kubernetes/kubernetes/blob/release-1.24/pkg/kubelet/nodeshutdown/systemd/inhibit_linux.go#L52C15-L52C31)()  --> [ConnectSystemBus()](https://github.com/kubernetes/kubernetes/blob/master/vendor/github.com/godbus/dbus/v5/conn.go#L132) --> [conn.Auth()](https://github.com/kubernetes/kubernetes/blob/master/vendor/github.com/godbus/dbus/v5/conn.go#L159) --> [go.conn.inworker()](https://github.com/kubernetes/kubernetes/blob/master/vendor/github.com/godbus/dbus/v5/auth.go#L118) --> [conn.HandleSign()](https://github.com/kubernetes/kubernetes/blob/master/vendor/github.com/godbus/dbus/v5/conn.go#L432) --> [conn.SignalHandler.DeliverSignal()](https://github.com/kubernetes/kubernetes/blob/master/vendor/github.com/godbus/dbus/v5/conn.go#L472) --> [signalChannelData.deliverSignal](https://github.com/kubernetes/kubernetes/blob/master/vendor/github.com/godbus/dbus/v5/default_handler.go#L320) (this is what we see in stack trace)

### Kubernetes version

1.24
<details>

```console
$ kubectl version
# paste output here
```

</details>


### Cloud provider

<details>

</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>