Repository: kubernetes
Issue: kubelet starts workloads on a node that is currently in process of being shut down after it is restarted
Labels: kind/bug,sig/node,priority/important-longterm,triage/accepted

### What happened?

Currently, when the [Graceful Node Shutdown](https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/2000-graceful-node-shutdown/README.md) is enabled, kubelet will then connect to Dbus and subscribe to the `org.freedesktop.systemd1.Manager` interface (using object path `/org/freedesktop/login1`) to listen for the `PrepareForShutdown` event.

Additionally, aside from listening for an event, kubelet will also place a so-called "inhibitor lock" (see [Inhibitor Locks](https://www.freedesktop.org/wiki/Software/systemd/inhibit) for more information) that allows for it to stagger a potential system shutdown or restart up to some configured amount of time, and as such, allowing for workloads to complete their work and terminate gracefully.

Thus, should there be a shutdown or a restart of the underlying node initiated using compliant mechanisms (such as more modern tools that correctly handle the interaction between systemd, systemd-inhibit and Dbus), then kubelet would receive the `PrepareForShutdown` with its boolean attribute set to `true`—this would be a clear indication that the node is due to interrupt its operation, either a shutdown or a restart, and kubelet should remove ability for the node to receive new workloads (scheduling on it will be disabled) and then start gracefully terminating current workloads.

Then, once the last workload is terminated, kubelet will release the "inhibitor lock", allowing for the shutdown or restart to progress uninterrupted. This mechanism works well, given no interruptions to Dbus or kubelet operations.

However, if kubelet had been restarted, let's say as part of an ongoing upgrade or crash, then it would start to schedule workloads again on a currently shutting down node. This might not be desirable and problematic, as some nodes can have a substantial number of workloads (in terms of volume of Pods and containers running), or the workloads can be sparse but very resource intensive, especially since the node is about to be shut down, restarted or even completely retired.

This is because kubelet does not detect the current status (in terms of a potentially pending shutdown or restart) of the underlying node and only listens for an incoming event, which might have already been sent.

That said, depending on how the node will be shut down or terminated, there is a potential saving grace since if the shutdown or restart is a scheduled event, then for the remainder of the time before the main event, the `PrepareForShutdown` event might be broadcasted again (but might also not). But there is a delay between each new event broadcast, during which kubelet could promptly schedule workloads back on the node before it notices the current node status again.

Thus, this might still not be enough to ensure that workloads won't be scheduled again on such a node.

### What did you expect to happen?

kubelet should avoid starting workloads, both static and otherwise, again on a node that is currently being shut down or waiting to be restarted. The underlying node should also not be marked as schedulable again.

### How can we reproduce it (as minimally and precisely as possible)?

Only a few steps are needed to reproduce this behaviour:

- Configure kubelet on a single worker node to enable Graceful Node Shutdown

This can be done using the following kubelet configuration properties:


```yaml
shutdownGracePeriod: 3600s
shutdownGracePeriodCriticalPods: 1800s
```

A very simple kubelet configuration file as an example:

<details>

```yaml
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 0s
    cacheUnauthorizedTTL: 0s
cgroupDriver: systemd
clusterDNS:
- 172.17.0.10
clusterDomain: cluster.local
containerRuntimeEndpoint: ""
cpuManagerReconcilePeriod: 0s
evictionPressureTransitionPeriod: 0s
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageMaximumGCAge: 0s
imageMinimumGCAge: 0s
kind: KubeletConfiguration
logging:
  flushFrequency: 0
  options:
    json:
      infoBufferSize: "0"
  verbosity: 0
memorySwap: {}
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
resolvConf: /run/systemd/resolve/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 0s
shutdownGracePeriod: 3600s
shutdownGracePeriodCriticalPods: 1800s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s
```

</details>

- Prepare a simple test workload both as a static and normal Pod

An example `test.yaml`:

<details>

```yaml
---
apiVersion: v1
kind: Pod
metadata:
  name: test
  namespace: default
spec:
  containers:
    - name: test
      image: docker.io/library/ubuntu:22.04
      command:
        - /bin/sh
        - -c
        - |
          ( while true ; do date ; sleep 1 ; done ) &
          pid=$!
          echo "Starting!"
          _term () {
              kill $pid
              echo "Caught SIGTERM!"
          }
          trap _term TERM
          wait $pid
          trap - TERM
          wait $pid
          exit 0
  restartPolicy: Always
```

</details>

Usually, static Pods manifests are stored within the `/etc/kubernetes/manifests` directory.

- Simulate the node shutdown

Emit the `org.freedesktop.login1.Manager.PrepareForShutdown` signal to simulate the node being shut down:

An example using the `gdbus` utility:

```console
# gdbus emit --system --object-path /org/freedesktop/login1 --signal 'org.freedesktop.login1.Manager.PrepareForShutdown' true
```
You should see the following entries in the kubelet log (provided a sufficient log level is set):

<details>

```console
Jan 10 07:28:44 worker-node01 kubelet[6188]: I0110 07:28:44.397678    6188 nodeshutdown_manager_linux.go:265] "Shutdown manager detected new shutdown event, isNodeShuttingDownNow" event=true
Jan 10 07:28:44 worker-node01 kubelet[6188]: I0110 07:28:44.397692    6188 nodeshutdown_manager_linux.go:273] "Shutdown manager detected new shutdown event" event="shutdown"
Jan 10 07:28:44 worker-node01 kubelet[6188]: I0110 07:28:44.397703    6188 nodeshutdown_manager_linux.go:322] "Shutdown manager processing shutdown event"
Jan 10 07:28:44 worker-node01 kubelet[6188]: I0110 07:28:44.398563    6188 event.go:376] "Event occurred" object="worker-node01" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="Shutdown" message="Shutdown manager detected shutdown event"
````

</details>

Plus, kubelet should already be working towards terminating workloads on the given node.

Alternatively, you can schedule an actual restart, albeit delayed. This can be done using the following command:

```console
shutdown -r +10
```

The above would defer node restart to **10 minutes** in the future from the time when the command was issued.

- Restart the kubelet process

Provided that the static Pod manifest is in place, you can restart kubelet using the `systemctl restart kubelet` command.

This should restart the current kubelet process—it will start a new one, a fresh one. Shortly after the restart, kubelet should bring the static Pod up alongside other workloads that should already start being scheduled again on the given node.

Why static Pod? Sometimes, as I have noticed (and which might be a potential issue), certain workloads are often immediately scheduled on the node that is supposed to be shutting down, and as such, on a smaller cluster, this can result in scheduling failures (and if the CNI-related workloads are affected, then nothing would work as the overlay network will be most likely down). A static Pod does not have this problem, of course.

### Anything else we need to know?

There do not seem to be a lot of reliable mechanisms presented to kubelet from Dbus or systemd that would allow it to ascertain the current status of the node. The API offered by Dbus is somewhat lacking in that regard.

kubelet could pick the current status again, provided that the `PrepareForShutdown` event will be broadcast again—which is the case for scheduled shutdowns. However, this is not very reliable and might still result in kubelet starting workloads again.

An idea would be for kubelet to check either the `PreparingForShutdown` property of the `org.freedesktop.login1.Manager` interface, which will be eventually set to `true` to indicate the current status or the `ScheduledShutdown` property (same interface) that when non-empty (a reason and future time has been set) could be used as an indicator that the node stats is due to change.

That said, the `PreparingForShutdown` does not seem to be reliably handled—for example, would cancelling a pending node restart clear this property (set to back to `false`)? Should kubelet clear this property, if at all possible, should the shutdown be cancelled? The `ScheduledShutdown` property seems a little more reliable as it will be set to an empty state should the scheduled shutdown be cancelled.

A different approach would be to persist some property via the API Server as part of the Node object or use a "tag file" somewhere safe on the file system and check for its presence on startup. Both methods would require some way to reconcile the state to avoid stale information and such.

The `org.freedesktop.login1.Manager` interface for reference:

<details>

```
interface org.freedesktop.login1.Manager {
  methods:
    GetSession(in  s session_id,
               out o object_path);
    GetSessionByPID(in  u pid,
                    out o object_path);
    GetUser(in  u uid,
            out o object_path);
    GetUserByPID(in  u pid,
                 out o object_path);
    GetSeat(in  s seat_id,
            out o object_path);
    ListSessions(out a(susso) sessions);
    ListUsers(out a(uso) users);
    ListSeats(out a(so) seats);
    ListInhibitors(out a(ssssuu) inhibitors);
    @org.freedesktop.systemd1.Privileged("true")
    CreateSession(in  u uid,
                  in  u pid,
                  in  s service,
                  in  s type,
                  in  s class,
                  in  s desktop,
                  in  s seat_id,
                  in  u vtnr,
                  in  s tty,
                  in  s display,
                  in  b remote,
                  in  s remote_user,
                  in  s remote_host,
                  in  a(sv) properties,
                  out s session_id,
                  out o object_path,
                  out s runtime_path,
                  out h fifo_fd,
                  out u uid,
                  out s seat_id,
                  out u vtnr,
                  out b existing);
    @org.freedesktop.systemd1.Privileged("true")
    ReleaseSession(in  s session_id);
    ActivateSession(in  s session_id);
    ActivateSessionOnSeat(in  s session_id,
                          in  s seat_id);
    LockSession(in  s session_id);
    UnlockSession(in  s session_id);
    LockSessions();
    UnlockSessions();
    KillSession(in  s session_id,
                in  s who,
                in  i signal_number);
    KillUser(in  u uid,
             in  i signal_number);
    TerminateSession(in  s session_id);
    TerminateUser(in  u uid);
    TerminateSeat(in  s seat_id);
    SetUserLinger(in  u uid,
                  in  b enable,
                  in  b interactive);
    AttachDevice(in  s seat_id,
                 in  s sysfs_path,
                 in  b interactive);
    FlushDevices(in  b interactive);
    PowerOff(in  b interactive);
    PowerOffWithFlags(in  t flags);
    Reboot(in  b interactive);
    RebootWithFlags(in  t flags);
    Halt(in  b interactive);
    HaltWithFlags(in  t flags);
    Suspend(in  b interactive);
    SuspendWithFlags(in  t flags);
    Hibernate(in  b interactive);
    HibernateWithFlags(in  t flags);
    HybridSleep(in  b interactive);
    HybridSleepWithFlags(in  t flags);
    SuspendThenHibernate(in  b interactive);
    SuspendThenHibernateWithFlags(in  t flags);
    CanPowerOff(out s result);
    CanReboot(out s result);
    CanHalt(out s result);
    CanSuspend(out s result);
    CanHibernate(out s result);
    CanHybridSleep(out s result);
    CanSuspendThenHibernate(out s result);
    ScheduleShutdown(in  s type,
                     in  t usec);
    CancelScheduledShutdown(out b cancelled);
    Inhibit(in  s what,
            in  s who,
            in  s why,
            in  s mode,
            out h pipe_fd);
    CanRebootParameter(out s result);
    SetRebootParameter(in  s parameter);
    CanRebootToFirmwareSetup(out s result);
    SetRebootToFirmwareSetup(in  b enable);
    CanRebootToBootLoaderMenu(out s result);
    SetRebootToBootLoaderMenu(in  t timeout);
    CanRebootToBootLoaderEntry(out s result);
    SetRebootToBootLoaderEntry(in  s boot_loader_entry);
    SetWallMessage(in  s wall_message,
                   in  b enable);
  signals:
    SessionNew(s session_id,
               o object_path);
    SessionRemoved(s session_id,
                   o object_path);
    UserNew(u uid,
            o object_path);
    UserRemoved(u uid,
                o object_path);
    SeatNew(s seat_id,
            o object_path);
    SeatRemoved(s seat_id,
                o object_path);
    PrepareForShutdown(b start);
    PrepareForSleep(b start);
  properties:
    @org.freedesktop.DBus.Property.EmitsChangedSignal("false")
    @org.freedesktop.systemd1.Privileged("true")
    readwrite b EnableWallMessages = true;
    @org.freedesktop.DBus.Property.EmitsChangedSignal("false")
    @org.freedesktop.systemd1.Privileged("true")
    readwrite s WallMessage = '';
    @org.freedesktop.DBus.Property.EmitsChangedSignal("const")
    readonly u NAutoVTs = 6;
    @org.freedesktop.DBus.Property.EmitsChangedSignal("const")
    readonly as KillOnlyUsers = [];
    @org.freedesktop.DBus.Property.EmitsChangedSignal("const")
    readonly as KillExcludeUsers = [];
    @org.freedesktop.DBus.Property.EmitsChangedSignal("const")
    readonly b KillUserProcesses = false;
    @org.freedesktop.DBus.Property.EmitsChangedSignal("false")
    readonly s RebootParameter = '';
    @org.freedesktop.DBus.Property.EmitsChangedSignal("false")
    readonly b RebootToFirmwareSetup = false;
    @org.freedesktop.DBus.Property.EmitsChangedSignal("false")
    readonly t RebootToBootLoaderMenu = 18446744073709551615;
    @org.freedesktop.DBus.Property.EmitsChangedSignal("false")
    readonly s RebootToBootLoaderEntry = '';
    @org.freedesktop.DBus.Property.EmitsChangedSignal("const")
    readonly as BootLoaderEntries = [];
    readonly b IdleHint = false;
    readonly t IdleSinceHint = 0;
    readonly t IdleSinceHintMonotonic = 0;
    readonly s BlockInhibited = '';
    readonly s DelayInhibited = 'shutdown';
    @org.freedesktop.DBus.Property.EmitsChangedSignal("const")
    readonly t InhibitDelayMaxUSec = 300000000;
    @org.freedesktop.DBus.Property.EmitsChangedSignal("const")
    readonly t UserStopDelayUSec = 10000000;
    @org.freedesktop.DBus.Property.EmitsChangedSignal("const")
    readonly s HandlePowerKey = 'poweroff';
    @org.freedesktop.DBus.Property.EmitsChangedSignal("const")
    readonly s HandleSuspendKey = 'suspend';
    @org.freedesktop.DBus.Property.EmitsChangedSignal("const")
    readonly s HandleHibernateKey = 'hibernate';
    @org.freedesktop.DBus.Property.EmitsChangedSignal("const")
    readonly s HandleLidSwitch = 'suspend';
    @org.freedesktop.DBus.Property.EmitsChangedSignal("const")
    readonly s HandleLidSwitchExternalPower = '';
    @org.freedesktop.DBus.Property.EmitsChangedSignal("const")
    readonly s HandleLidSwitchDocked = 'ignore';
    @org.freedesktop.DBus.Property.EmitsChangedSignal("const")
    readonly t HoldoffTimeoutUSec = 30000000;
    @org.freedesktop.DBus.Property.EmitsChangedSignal("const")
    readonly s IdleAction = 'ignore';
    @org.freedesktop.DBus.Property.EmitsChangedSignal("const")
    readonly t IdleActionUSec = 1800000000;
    @org.freedesktop.DBus.Property.EmitsChangedSignal("false")
    readonly b PreparingForShutdown = true;
    @org.freedesktop.DBus.Property.EmitsChangedSignal("false")
    readonly b PreparingForSleep = false;
    @org.freedesktop.DBus.Property.EmitsChangedSignal("false")
    readonly (st) ScheduledShutdown = ('', 0);
    @org.freedesktop.DBus.Property.EmitsChangedSignal("false")
    readonly b Docked = false;
    @org.freedesktop.DBus.Property.EmitsChangedSignal("false")
    readonly b LidClosed = false;
    @org.freedesktop.DBus.Property.EmitsChangedSignal("false")
    readonly b OnExternalPower = true;
    @org.freedesktop.DBus.Property.EmitsChangedSignal("const")
    readonly b RemoveIPC = true;
    @org.freedesktop.DBus.Property.EmitsChangedSignal("const")
    readonly t RuntimeDirectorySize = 410025984;
    @org.freedesktop.DBus.Property.EmitsChangedSignal("const")
    readonly t RuntimeDirectoryInodesMax = 100104;
    @org.freedesktop.DBus.Property.EmitsChangedSignal("const")
    readonly t InhibitorsMax = 8192;
    @org.freedesktop.DBus.Property.EmitsChangedSignal("false")
    readonly t NCurrentInhibitors = 1;
    @org.freedesktop.DBus.Property.EmitsChangedSignal("const")
    readonly t SessionsMax = 8192;
    @org.freedesktop.DBus.Property.EmitsChangedSignal("false")
    readonly t NCurrentSessions = 1;
};
```

</details>

### Kubernetes version

<details>

```console
$ kubectl version
Client Version: v1.29.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.0
```

</details>


### Cloud provider

<details>

```
N/A
```

</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
$ uname -a
Linux worker-node01 5.15.0-83-generic #92-Ubuntu SMP Mon Aug 14 09:30:42 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
```

</details>


### Install tools

<details>

```console
$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"29", GitVersion:"v1.29.0", GitCommit:"3f7a50f38688eb332e2a1b013678c6435d539ae6", GitTreeState:"clean", BuildDate:"2023-12-13T08:50:10Z", GoVersion:"go1.21.5", Compiler:"gc", Platform:"linux/amd64"}
```

</details>


### Container runtime (CRI) and version (if applicable)

<details>

```console
$ crio --version
crio version 1.29.0
Version:        1.29.0
GitCommit:      d59bbdc252837107c9f5d235b8fb2650ff2b9d93
GitCommitDate:  2023-12-21T16:23:14Z
GitTreeState:   clean
BuildDate:      1970-01-01T00:00:00Z
GoVersion:      go1.21.1
Compiler:       gc
Platform:       linux/amd64
Linkmode:       static
BuildTags:      
  static
  netgo
  osusergo
  exclude_graphdriver_btrfs
  exclude_graphdriver_devicemapper
  seccomp
  apparmor
  selinux
LDFlags:          unknown
SeccompEnabled:   true
AppArmorEnabled:  false
```

```console
$ conmon --version
conmon version 2.1.10
commit: 09bcded8e9c49cf1ff1fda403feac5a08f22535f-dirty
````

```console
$ crun --version
crun version 1.12
commit: ce429cb2e277d001c2179df1ac66a470f00802ae
rundir: /run/user/1000/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
````

```console
$ runc --version
runc version 1.1.11
commit: v1.1.11-0-g4bccb38c
spec: 1.0.2-dev
go: go1.20.12
libseccomp: 2.5.4
````



</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

```console
$ crictl exec -it ee8826f196d0a calicoctl version
Client Version:    v3.25.0
Git commit:        3f7fe4d29
Cluster Version:   v3.25.0
Cluster Type:      k8s,kdd,bgp,kubeadm
```

</details>