### Human: Can you tell about Nutanix Kubernetes Engine - Kubernetes cluster is unhealthy after reducing the number of worker nodes to fewer than three. ### Assistant: In Kubernetes clusters deployed with NKE 2.7, reducing the number of worker nodes in a Production cluster to fewer than three causes the cluster to be reported as unhealthy due to an insufficient number of calico-typha replicas.. Nutanix Kubernetes Engine (NKE) is formerly known as Karbon or Karbon Platform Services.

Calico-typha is a component of the Calico Container Network Interface (CNI) used by NKE to increase scale by decreasing the impact Kubernetes nodes have on the datastore used by Calico. Starting with NKE 2.7, the number of replicas of calico-typha was increased from one to three in Kubernetes Production clusters. As a result of this change, decreasing the number of worker nodes in an NKE Production Kubernetes cluster from three or more nodes to fewer than three nodes will result in the Kubernetes cluster being reported as unhealthy. This is due to the cluster expecting to have three replicas of calico-typha instead of the one replica present after reducing the number of worker nodes to one or two.

When the number of nodes is reduced to fewer than three, the node removal task will hang for several minutes at the "Verifying health after removal of worker node" step until the task eventually fails and reports the following error in the NKE UI:

Remove worker failed to delete worker node: 1 nodes failed to be removed out of 1: failed to get the cluster health: Operation timed out: Calico: Calico unhealthy Max retries done: Failed to verify calico: Operation timed out: expecting 3 available replicas of calico-typha deployment in kube-system namespace. Currently running: 2

In the NKE UI, the status of the cluster is reported as Critical.

The Alerts page in the NKE UI will report the following two types of alerts:

Remove worker failed to delete worker node: 1 nodes failed to be removed out of 1: failed to get the cluster health: Operation timed out: Calico: Calico unhealthy Max retries done: Failed to verify calico: Operation timed out: expecting 3 available replicas of calico-typha deployment in kube-system namespace. Currently running: 2

In the Prism Central UI, the "Remove worker <worker node name> from node pool <node pool name>" task will show as Failed and report the following error:

Remove worker failed to delete worker node: 1 nodes failed to be removed out of 1: failed to get the cluster health: Operation timed out: Calico: Calico unhealthy Max retries done: Failed to verify calico: Operation timed out: expecting 3 available replicas of calico-typha deployment in kube-system namespace. Currently running: 2

The /home/nutanix/data/logs/karbon_core.out log on the Prism Central VM(s) will contain the following type of errors:

Remove worker failed to delete worker node: 1 nodes failed to be removed out of 1: failed to get the cluster health: Operation timed out: Calico: Calico unhealthy Max retries done: Failed to verify calico: Operation timed out: expecting 3 available replicas of calico-typha deployment in kube-system namespace. Currently running: 2

On the Prism Element cluster, the VM removal task for the worker node is successful, and the worker VM is deleted.

Note that this issue does not impact Kubernetes clusters deployed with NKE versions prior to 2.7, as those clusters would have been deployed with a single replica of calico-typha. This issue also does not impact Development clusters deployed with NKE 2.7, as Development clusters are deployed with a single calico-typha replica.. To resolve this issue, increase the number of worker nodes to three or more. For more information on adding worker nodes, see Creating a Node Pool in the Nutanix Kubernetes Engine 2.7 Guide.

Decreasing the number of worker nodes in an NKE Production Kubernetes cluster to fewer than three is not recommended and should be avoided. The NKE UI does not prevent deploying a Production cluster with one or two worker nodes, but deploying a new Production cluster with fewer than three nodes is also not recommended and should be avoided.

If the number of worker nodes cannot be increased to three or more, contact Nutanix Support (login required) for assistance.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.