Repository: kubernetes/autoscaler

Issue #6452: cluster-autoscaler still thinks node group has taints (after untaint) and refuses to scale back up from zero count
Created: 2024-01-17T07:22:02Z
Which component are you using?: cluster-autoscaler
What version of the component are you using?: cluster-autoscaler
Component version: helm chart 9.26.0 cluster-autoscaler 1.28.2
What k8s version are you using (kubectl version)?: 1.28
kubectl version OutputClient Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.5-eks-5e0fdde

What environment is this in?: AWS EKS; managed node groups
What did you expect to happen?: Untainted node group be able launch nodes and schedule pods again.
What happened instead?: autoscaler still thinks node group has the long-gone node with taint and thus won't launch another node instance, even though the taint has been removed from node group.
│ Events:                                                                                                              │
│   Type     Reason             Age                     From                Message                                    │
│   ----     ------             ----                    ----                -------                                    │
│   Normal   NotTriggerScaleUp  22m (x385 over 19h)     cluster-autoscaler  pod didn't trigger scale-up: 3 max node gr │
│ oup size reached, 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) had untolerated taint {cost: true}  │
│   Normal   NotTriggerScaleUp  12m (x179 over 19h)     cluster-autoscaler  pod didn't trigger scale-up: 3 max node gr │
│ oup size reached, 1 node(s) had untolerated taint {cost: true}, 1 node(s) didn't match Pod's node affinity/selector  │
│   Normal   NotTriggerScaleUp  7m11s (x1152 over 19h)  cluster-autoscaler  pod didn't trigger scale-up: 1 node(s) had │
│  untolerated taint {cost: true}, 1 node(s) didn't match Pod's node affinity/selector, 3 max node group size reached  │
│   Warning  FailedScheduling   5m (x161 over 19h)      default-scheduler   0/8 nodes are available: 8 node(s) didn't  │
│ match Pod's node affinity/selector. preemption: 0/8 nodes are available: 8 Preemption is not helpful for scheduling. │
│ .                                                                                                                    │
│   Normal   NotTriggerScaleUp  2m10s (x2508 over 19h)  cluster-autoscaler  pod didn't trigger scale-up: 1 node(s) did │
│ n't match Pod's node affinity/selector, 1 node(s) had untolerated taint {cost: true}, 3 max node group size reached  │
│   Normal   TriggeredScaleUp   50s                     cluster-autoscaler  pod triggered scale-up: [{eks-ZeroNodes-ce │
│ c688d4-02dd-c7f5-6bd1-be1a14735f61 0->1 (max: 1)}]                                                                   │
│   Normal   Scheduled          3s                      default-scheduler   Successfully assigned default/nginx-test t │
│ o ip-10-0-76-17.ap-southeast-1.compute.internal                                                                      │
│   Normal   Pulling            2s                      kubelet             Pulling image "nginx:latest"               │


How to reproduce it (as minimally and precisely as possible):
Taint node group to evict pods
aws eks update-nodegroup-config --cluster-name eks-cluster --nodegroup-name ZeroNodes --taints addOrUpdateTaints={key=cost,value=true,effect=NO_EXECUTE}
Pods get evicted and nodes eventually terminated to zero count.
Untaint node group to re-host pods.
aws eks update-nodegroup-config --cluster-name eks-cluster --nodegroup-name ZeroNodes --taints removeTaints={key=cost,value=true,effect=NO_EXECUTE}
Pod remains in perpetual pending state, as per events above, with autoscaler thinking the the old node with taint still around and refuses to launch another node in its place (without the taint).
Anything else we need to know?:
Workaround to deliberately kill autoscaler pod, so replacement autoscaler pod with no memory of the past can correctly auto-discover the node group and launch node to host the pod as per events above.
It seems like autoscaler wants to hang on to outdated historical data of terminated nodes in node group "1 node(s) had untolerated taint". It should be describing the node groups afresh to determine where to launch a new node.

Duplicate issue #7991: Cluster Autoscaler is not scheduling nodes; they only get scheduled after restarting autoscaler pod
Created: 2025-03-28T11:34:51Z
Which component are you using?:

cluster-autoscaler using image registry.k8s.io/autoscaling/cluster-autoscaler:v1.26.2
cluster-autoscaler using image registry.k8s.io/autoscaling/cluster-autoscaler:v1.29.3

What version of the component are you using?:
Server Version: version.Info{Major:"1", Minor:"29+",

Component version:
What k8s version are you using (kubectl version)?:
registry.k8s.io/autoscaling/cluster-autoscaler:v1.29.3
kubectl version Output$ kubectl version
Server Version: version.Info{Major:"1", Minor:"29+",

What environment is this in?:
AWS EKS
What did you expect to happen?:
The pods should be scheduled. But no new nodes are being created in the particular node pool.
What happened instead?:
pods of scheduled jobs are in a pending state since nodes are not available. Once we restart the cluster autoscaler nodes are being created
How to reproduce it (as minimally and precisely as possible):
I'm not sure how to reproduce this because this problem is only happening in two clusters. We have one more cluster, and it's working fine.
Anything else we need to know?:
Upgrade the cluster autoscaler from 1.26.2 to 1.29.3 in one of the clusters but still have the same issue. This is primarily seen in different node pools, and it's intermittent. We are shutting down clusters during off hours but not shutting down kube-system. This was working fine till last week.

Closing comment (2025-04-01T06:46:21Z): Looks like it's the same as #6452