Repository: kubernetes
Issue: DaemonSet controller and Graceful Node Shutdown manager disagree when making workloads placement decision
Labels: sig/node,kind/feature,sig/apps,priority/important-longterm,triage/accepted

### What happened?

Currently, when the [Graceful Node Shutdown](https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/2000-graceful-node-shutdown/README.md) is enabled, as the node is shutting down, workloads currently running will be terminated in order of the workload type and priority, while taking into the account the termination grace period that a given workload can have set.

However, even though the **DaemonSet** type workloads were also successfully vacated from a node that is currently shutting down, there are some other lingering issues with this specific workload type.

When a node shutdown is detected, the Shutdown Manager will then proceed to update the node status to "NotReady" (this does not set the "Unschedulable" flag similarly to what cordoning a node would), adds necessary taints, and then proceeds to terminate currently runnings workloads taking into the account priorities for different type of workloads and also ensuring that crucial system workloads (often run as DaemonSets) are to be terminated the last—attempting to ensure that node will not lose its network access, the metrics can still be sent, and monitoring won't be interrupted to the very end.

Part of the process of preparing the underlying node for shutdown is to add an admission callback that has a singular purpose—reject everything and anything attempting to start on this specific node. The code is very simple and makes no attempts to handle different types of workloads; everything in unison will be rejected during the time when the node is shutting down.

_(an example taken from [kubernetes/kubernetes/pkg/kubelet/nodeshutdown/nodeshutdown_manager_linux.go#L146-L157](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/nodeshutdown/nodeshutdown_manager_linux.go#L146-L157))_

<details>

https://github.com/kubernetes/kubernetes/blob/9c1c603fbef4abb1bfd257a02a32d7d16832a8a4/pkg/kubelet/nodeshutdown/nodeshutdown_manager_linux.go#L146-L157

</details>

This is where the DaemonSet controller and Graceful Node Shutdown manager disagree—the former sees that its previously managed workloads were removed, assumes that this is an invalid state and attempts to reconcile it starting missing Pods again, whereas the latter sees a new Pod being scheduled on the given node that is currently shutting down, and rejects it (like any other workload of any type) from being run. This cycle will repeat either until the workloads restart policy and errors budget tolerance will allow or indefinitely (or for as long as the node is still pending its shutdown or restart).

_(an example taken from [kubernetes/kubernetes/pkg/controller/daemon/daemon_controller.go#L1278-L1301](https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/daemon/daemon_controller.go#L1278-L1301))_

<details>

https://github.com/kubernetes/kubernetes/blob/9c1c603fbef4abb1bfd257a02a32d7d16832a8a4/pkg/controller/daemon/daemon_controller.go#L1278-L1301

</details>

_(an example taken from [kubernetes/kubernetes/pkg/controller/daemon/daemon_controller.go#L1304-L1313](https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/daemon/daemon_controller.go#L1304-L1313))_

<details>

https://github.com/kubernetes/kubernetes/blob/9c1c603fbef4abb1bfd257a02a32d7d16832a8a4/pkg/controller/daemon/daemon_controller.go#L1304-L1313

</details>

As the worker node can run workloads that take a long time to stop and terminate, the number of workloads with scheduling failure can keep increasing over time, leaving these failed attempts to have to be cleaned manually, such as the "NodeShutdown" (do these type of statuses need manual clean up too?), or even being permanently stuck in the "Pending" or "Terminating" states. Workloads terminated with the "Completed" should, provided that everything works as intended, be cleaned up and rescheduled automatically, especially once the node is back up and running again.

Some other workloads might report scheduling failures, leaving workloads in an "Error" state that would also require manual clean-up.

An example of this taking place during a node shutdown:

[Peek 2024-01-17 17-21.webm](https://github.com/kubernetes/kubernetes/assets/250723/26362a85-b46c-4824-8e0e-a57cd00d9ae5)

### What did you expect to happen?

Most of the workload types (Pods, Deployments, ReplicaSets, DaemonSets, etc.) are to be correctly removed from the underlying node, and then the remaining Pods terminated.

### How can we reproduce it (as minimally and precisely as possible)?

Only a few steps are needed to reproduce this behaviour:

- Configure kubelet on a single worker node to enable Graceful Node Shutdown

This can be done using the following kubelet configuration properties:

```yaml
shutdownGracePeriod: 3600s
shutdownGracePeriodCriticalPods: 1800s
```

A very simple kubelet configuration file as an example:

<details>

```yaml
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 0s
    cacheUnauthorizedTTL: 0s
cgroupDriver: systemd
clusterDNS:
- 172.17.0.10
clusterDomain: cluster.local
containerRuntimeEndpoint: ""
cpuManagerReconcilePeriod: 0s
evictionPressureTransitionPeriod: 0s
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageMaximumGCAge: 0s
imageMinimumGCAge: 0s
kind: KubeletConfiguration
logging:
  flushFrequency: 0
  options:
    json:
      infoBufferSize: "0"
  verbosity: 0
memorySwap: {}
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
resolvConf: /run/systemd/resolve/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 0s
shutdownGracePeriod: 3600s
shutdownGracePeriodCriticalPods: 1800s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s
```

</details>

- Prepare a simple test workload (a simple DaemonSet)

An example `test.yaml`:

<details>

```yaml
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: test
  namespace: default
  labels:
    app.kubernetes.io/name: test
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: test
  template:
    metadata:
      labels:
        app.kubernetes.io/name: test
    spec:
      nodeSelector:
        kubernetes.io/os: linux
      tolerations:
        - operator: Exists
      containers:
        - name: test
          image: docker.io/library/bash:latest
          command:
            - /bin/sh
            - -c
            - |
              ( while true ; do date ; sleep 1 ; done ) &
              pid=$!
              echo "Starting!"
              _term () {
                  kill $pid
                  echo "Caught SIGTERM!"
              }
              trap _term TERM
              wait $pid
              trap - TERM
              wait $pid
              exit 0
          lifecycle:
            preStop:
              exec:
                command:
                  - /bin/sh
                  - -c
                  - |
                    echo BEGIN preStop > /proc/1/fd/1
                    sleep 10
                    echo END preStop > /proc/1/fd/1
      terminationGracePeriodSeconds: 30
      restartPolicy: Always
```

</details>

- Simulate the node shutdown

Emit the `org.freedesktop.login1.Manager.PrepareForShutdown` signal to simulate the node being shut down:

An example using the `gdbus` utility:

```console
# gdbus emit --system --object-path /org/freedesktop/login1 --signal 'org.freedesktop.login1.Manager.PrepareForShutdown' true
```
You should see the following entries in the kubelet log (provided a sufficient log level is set):

<details>

```console
Jan 10 07:28:44 worker-node01 kubelet[6188]: I0110 07:28:44.397678    6188 nodeshutdown_manager_linux.go:265] "Shutdown manager detected new shutdown event, isNodeShuttingDownNow" event=true
Jan 10 07:28:44 worker-node01 kubelet[6188]: I0110 07:28:44.397692    6188 nodeshutdown_manager_linux.go:273] "Shutdown manager detected new shutdown event" event="shutdown"
Jan 10 07:28:44 worker-node01 kubelet[6188]: I0110 07:28:44.397703    6188 nodeshutdown_manager_linux.go:322] "Shutdown manager processing shutdown event"
Jan 10 07:28:44 worker-node01 kubelet[6188]: I0110 07:28:44.398563    6188 event.go:376] "Event occurred" object="worker-node01" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="Shutdown" message="Shutdown manager detected shutdown event"
````

</details>

Plus, kubelet should already be working towards terminating workloads on the given node.

After a few moments, the DaemonSet controller would attempt to schedule workloads, including the test one, back on the worker node that is currently shutting down.

An excerpt from logs captured when this takes place per:

_(the affected Pod name is **test-cj4n2**)_

<details>

[kwilczynski/a194305c0b2324aa1d254cccc40da5b3](https://gist.github.com/kwilczynski/a194305c0b2324aa1d254cccc40da5b3)

</details>


_(the affected Pod name is **node-exporter-9cvc2**)_


<details>

[kwilczynski/b2baa49657501c2ab8ad98e09d853ac2](https://gist.github.com/kwilczynski/b2baa49657501c2ab8ad98e09d853ac2)

</details>


### Anything else we need to know?

An important characteristic of a DaemonSet is the ability to ignore most default worker node statuses (such as "NotReady" and "SchedulingDisabled"), ignore the "Unschedulable" flag set on a given node, and also ignore most of the default node taints. The basic idea is to allow workloads run as DaemonSets to run everywhere, including nodes that should not potentially schedule any other types of workloads.

This design decision, which is also the key feature of a DaemonSet, is something that can interfere with the Graceful Node Shutdown feature.

How to fix this problem? Some viable fixes would be:

- Do not remove DaemonSets from the node that is due to be shut down (not too dissimilar to what cordoning and draining a node would do)
- Teach the DaemonSet controller about "NodeShutdown" status for a node so that it avoids attempting to schedule Pods there
- Add a new type of taint and teach the DaemonSet controller to include it as part of its scheduling decision making
- Teach the Graceful Node Shutdown manager to ignore DaemonSets until the very last moment before these types of workloads are due to be evicted

Some of the above could be controlled via a new setting for kubelet, like the behaviour that controls whether to evict DaemonSets and when.

Additionally, the "Unschedulable" could be set to "true" for a good measure, to ensure that absolutely no workloads will attempt to schedule themselves. Or to make it harder to do so.

---

Recently, there have been issues with status updates and synchronisation of different types of workloads when a node was shutting down. This has since appears to have been fixed via the following Pull Requests:

- https://github.com/kubernetes/kubernetes/pull/98005
- https://github.com/kubernetes/kubernetes/pull/117073
- https://github.com/kubernetes/kubernetes/issues/109450

However, some newer reports claim the contrary per:

- https://github.com/kubernetes/kubernetes/issues/98004
- https://github.com/kubernetes/kubernetes/issues/100184
- https://github.com/kubernetes/kubernetes/issues/122122



### Kubernetes version

<details>

- Kubernetes

```console
$ kubectl version
Client Version: v1.29.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.0
```

- OpenShift

```console
$ oc version 
Client Version: 4.14.4
Kustomize Version: v5.0.1
Server Version: 4.14.4
Kubernetes Version: v1.27.6+d548052
```

</details>


### Cloud provider

<details>

```
N/A
```

</details>


### OS version

<details>

- Kubernetes

```console
# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
$ uname -a
Linux worker-node01 5.15.0-83-generic #92-Ubuntu SMP Mon Aug 14 09:30:42 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
```

- OpenShift

```console
# On Linux:
$ cat /etc/os-release 
NAME="Red Hat Enterprise Linux CoreOS"
ID="rhcos"
ID_LIKE="rhel fedora"
VERSION="414.92.202311222314-0"
VERSION_ID="4.14"
VARIANT="CoreOS"
VARIANT_ID=coreos
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux CoreOS 414.92.202311222314-0 (Plow)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::coreos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://docs.openshift.com/container-platform/4.14/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="OpenShift Container Platform"
REDHAT_BUGZILLA_PRODUCT_VERSION="4.14"
REDHAT_SUPPORT_PRODUCT="OpenShift Container Platform"
REDHAT_SUPPORT_PRODUCT_VERSION="4.14"
OPENSHIFT_VERSION="4.14"
RHEL_VERSION="9.2"
OSTREE_VERSION="414.92.202311222314-0"
```

</details>

### Install tools

<details>

- Kubernetes

```console
$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"29", GitVersion:"v1.29.0", GitCommit:"3f7a50f38688eb332e2a1b013678c6435d539ae6", GitTreeState:"clean", BuildDate:"2023-12-13T08:50:10Z", GoVersion:"go1.21.5", Compiler:"gc", Platform:"linux/amd64"}
```

- OpenShift

```console
$ openshift-install version
openshift-install 4.14.4
built from commit 82051a1c3d42e394be353bbe2eacf348ccb75970
release image registry.ci.openshift.org/ocp/release@sha256:e6e1d90b492d50438034e6edb46bdafa6c86ae3b80ef3328685912d89681fdee
release architecture amd64
```

</details>


### Container runtime (CRI) and version (if applicable)

<details>

- Kubernetes

```console
$ crio --version
crio version 1.29.0
Version:        1.29.0
GitCommit:      d59bbdc252837107c9f5d235b8fb2650ff2b9d93
GitCommitDate:  2023-12-21T16:23:14Z
GitTreeState:   clean
BuildDate:      1970-01-01T00:00:00Z
GoVersion:      go1.21.1
Compiler:       gc
Platform:       linux/amd64
Linkmode:       static
BuildTags:      
  static
  netgo
  osusergo
  exclude_graphdriver_btrfs
  exclude_graphdriver_devicemapper
  seccomp
  apparmor
  selinux
LDFlags:          unknown
SeccompEnabled:   true
AppArmorEnabled:  false
```

```console
$ conmon --version
conmon version 2.1.10
commit: 09bcded8e9c49cf1ff1fda403feac5a08f22535f-dirty
````

```console
$ crun --version
crun version 1.12
commit: ce429cb2e277d001c2179df1ac66a470f00802ae
rundir: /run/user/1000/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
````

```console
$ runc --version
runc version 1.1.11
commit: v1.1.11-0-g4bccb38c
spec: 1.0.2-dev
go: go1.20.12
libseccomp: 2.5.4
````

- OpenShift

```console
$ crio --version
crio version 1.27.2-2.rhaos4.14.git9d684e2.el9
Version:        1.27.2-2.rhaos4.14.git9d684e2.el9
GitCommit:      unknown
GitCommitDate:  unknown
GitTreeState:   clean
GoVersion:      go1.20.10
Compiler:       gc
Platform:       linux/amd64
Linkmode:       dynamic
BuildTags:      
  rpm_crashtraceback
  libtrust_openssl
  selinux
  seccomp
  exclude_graphdriver_devicemapper
  exclude_graphdriver_btrfs
  containers_image_ostree_stub
LDFlags:           -compressdwarf=false -B 0x8c7c7209f406ead112dcaacb10b18ba6826efc5e -extldflags '-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 '
SeccompEnabled:   true
AppArmorEnabled:  false
```

```console
$ conmon --version
conmon version 2.1.7
commit: a3013c55c8bfc2d0597ffe8beedadba8aab5bbf6
```

```console
$ runc --version
runc version 1.1.10
spec: 1.0.2-dev
go: go1.20.10
libseccomp: 2.5.2
```

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

- Kubernetes

```console
$ crictl exec -it ee8826f196d0a calicoctl version
Client Version:    v3.25.0
Git commit:        3f7fe4d29
Cluster Version:   v3.25.0
Cluster Type:      k8s,kdd,bgp,kubeadm
```

- OpenShift

A lot of things. Probably too many to list. See the following for details:

- [OpenShift Container Platform 4.14 release notes](https://docs.openshift.com/container-platform/4.14/release_notes/ocp-4-14-release-notes.html)

</details>