vania-janet/multiturn-rag-retrieval-data huggingface dataset:
If they are in a Normal state, add back any deleted components and re-create any reverted changes, one by one, until you can determine which configuration or component caused the worker node disruption.
7. If the issue is still not resolved, follow the steps to [gather your worker node data](https://cloud.ibm.com/docs/openshift?topic=openshift-ts-critical-notreadyts-critical-notready-gather) and open a support ticket.







 If worker nodes switch between normal and critical states 

If your worker nodes switch between a Normal and Critical or NotReady state, check the following components for any issues or recent changes that may disrupt your worker nodes.



1. For classic clusters, check your firewalls or gateways. If there is a bandwidth limit or any type of malfunction, resolve the issue. Then, check your worker nodes again.
2. Check if the applications, security, or monitoring components in your cluster are overloading the cluster apiserver with requests, which may cause disruptions for your worker nodes.
3. If you recently added any components to your cluster, remove them. If you made changes to any existing components in your cluster, revert the changes. Then, check the status of your worker nodes to see whether the new components or changes were causing the issue.
4. Check for changes on any cluster webhooks, which can disrupt apiserver requests or block a worker node's ability to connect with the apiserver. [Remove all webhooks](https://cloud.ibm.com/docs/containers?topic=containers-ts-delete-webhooks) that were added to the cluster after it was created.
5. If the issue is still not resolved, follow the steps to [gather your worker node data](https://cloud.ibm.com/docs/openshift?topic=openshift-ts-critical-notreadyts-critical-notready-gather) and open a support ticket.









 Gathering data for a support case 

If you are unable to resolve the issue with the troubleshooting steps, gather information about your worker nodes.
The worker node hosts do not have infrastructure listed in the Cluster column of the output, but instead have the name of the cluster.
3. Update your worker nodes individually by running the ibmcloud ks worker update command.

ibmcloud ks worker update -c CLUSTER_NAME_OR_ID --worker WORKER_ID
4. Confirm that the update is complete by reviewing the Kubernetes version of your worker nodes.

kubectl get nodes

If the update failed, you must [apply version updates by replacing hosts](https://cloud.ibm.com/docs/satellite?topic=satellite-host-update-workershost-update-workers-minor).







 Apply version updates to your worker node hosts with a ConfigMap 

You can roll out updates to all your worker node hosts with a ConfigMap. Specify which nodes to update by using labels. You can also specify



1. Optional: [Attach](https://cloud.ibm.com/docs/satellite?topic=satellite-attach-hosts) and [assign](https://cloud.ibm.com/docs/satellite?topic=satellite-assigning-hostshost-assign-manual) extra hosts to the service cluster to handle the compute capacity while your existing hosts are updating.
2. [Identify your worker node hosts](https://cloud.ibm.com/docs/satellite?topic=satellite-host-update-workershost-identify). Your worker node hosts are not listed as Infrastructure.
3. View the labels of a worker node. You can find the worker node labels in the Labels section of your CLI output. Every label consists of a NodeSelectorKey and a NodeSelectorValue. You can use the labels to specify which worker nodes to update.

kubectl get nodes -o yaml

Example output

labels:
arch: amd64
beta.kubernetes.io/arch: amd64
beta.kubernetes.io/instance-type: upi
beta.kubernetes.io/os: linux
Check to see if the worker node is removed.

ibmcloud oc worker get --cluster <cluster_name_or_ID> --worker <worker_node_ID>

Example output shows if the worker node removal is successful. The worker get operation fails because the worker node is deleted. The infrastructure permissions are correctly set up.

FAILED
The specified worker node could not be found. (E0011)
3. If the worker node is not removed, review the [State and Status fields](https://cloud.ibm.com/docs/openshift?topic=openshift-debug_worker_nodes) and the [common issues with worker nodes](https://cloud.ibm.com/docs/openshift?topic=openshift-common_worker_nodes_issues) to continue debugging.
4. If you manually set credentials and still can't see the cluster's worker nodes in your infrastructure account, you might check whether the [cluster is orphaned](https://cloud.ibm.com/docs/openshift?topic=openshift-cluster_infra_errorsorphaned).









 Unable to create or delete worker nodes due to incorrect account error 

Classic infrastructure

 What’s happening 

You can't manage worker nodes for your cluster or view the cluster worker nodes in your classic IBM Cloud infrastructure account. However, you can update and manage other clusters in the account.

Further, you verified that you have the [proper infrastructure credentials](https://cloud.ibm.com/docs/openshift?topic=openshift-cluster_infra_errorscs_credentials).

You might receive an error message in your worker node status similar to the following example.

incorrect account for worker - The 'classic' infrastructure user credentials changed and no longer match the worker node instance infrastructure account.

 Why it’s happening 

The cluster might be provisioned in a classic IBM Cloud infrastructure account that is no longer linked to your Red Hat OpenShift on IBM Cloud account. The cluster is orphaned. Because the resources are in a different account, you don't have the infrastructure credentials to modify the resources.
For more information, see [worker node resource reserves](https://cloud.ibm.com/docs/openshift?topic=openshift-planning_worker_nodesresource_limit_node).

Want to be sure that you always have enough worker nodes to cover your workload? Try out [the cluster autoscaler](https://cloud.ibm.com/docs/openshift?topic=openshift-cluster-scaling-classic-vpc).





 Why do my worker nodes have the master role? 

When you run oc get nodes or oc describe node <worker_node>, you might see that the worker nodes have master,worker roles. In OpenShift Container Platform clusters, operators use the master role as a nodeSelector so that OCP can deploy default components that are controlled by operators, such as the internal registry, in your cluster. No master node processes, such as the API server or Kubernetes scheduler, run on your worker nodes. For more information about master and worker node components, see [Red Hat OpenShift architecture](https://cloud.ibm.com/docs/openshift?topic=openshift-service-architectureservice-architecture-4).





 How can I check the operating system that my worker nodes run? 

When you create a worker pool, you choose the flavor, which describes the operating system along with the compute resources of the worker nodes. Supported operating systems are RHEL 7.

You can also log in to your cluster to check the operating system of the worker nodes.



1. [Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.](https://cloud.ibm.com/docs/containers?topic=containers-access_cluster)
2. List your worker nodes.

oc get nodes
3. Describe your worker node and check for the operating system label that IBM applies, or the OS Image and Operating System fields in the System Info section.

oc describe node <node>

Example output

NAME: 10.xxx.xx.xxx
Roles: <none>
Worker node states 

You can view the current worker node state by running the ibmcloud ks worker ls --cluster <cluster_name_or_ID> command and locating the State and Status fields.



 Critical state 

You can view the current worker node state by running the ibmcloud ks worker ls --cluster <cluster_name_or_ID> command and locating the State and Status fields.

A worker node can go into a Critical state for many reasons. See [Troubleshooting worker nodes in Critical or NotReady state](https://cloud.ibm.com/docs/containers?topic=containers-ts-critical-notready) for more information and troubleshooting steps.





 Deleting state 

You can view the current worker node state by running the ibmcloud ks worker ls --cluster <cluster_name_or_ID> command and locating the State and Status fields.

A Deleting state means that you requested to delete the worker node, possibly as part of resizing a worker pool or autoscaling the cluster. Other operations can't be issued against the worker node while the worker node deletes. You can't reverse the deletion process. When the deletion process completes, you are no longer billed for the worker nodes.





 Deleted state 

You can view the current worker node state by running the ibmcloud ks worker ls --cluster <cluster_name_or_ID> command and locating the State and Status fields.

A Deleted state means that your worker node is deleted, and no longer is listed in the cluster or billed. This state can't be undone. Any data that was stored only on the worker node, such as container images, are also deleted.





 Deployed state 

You can view the current worker node state by running the ibmcloud ks worker ls --cluster <cluster_name_or_ID> command and locating the State and Status fields.

Updates are successfully deployed to your worker node. After updates are deployed, IBM Cloud Kubernetes Service starts a health check on the worker node. After the health check is successful, the worker node goes into a Normal state.
You can use these example checks to define rules for worker nodes in a specific zone (zonecheck.json), region (regioncheck.json), or for all worker nodes that don't match any of the checks that you defined in the config map (defaultcheck.json). Use the check template to create your own check. For every check, to identify a worker node, you must choose one of the worker node labels that you retrieved in the previous step.

For every check, you can set only one value for NodeSelectorKey and NodeSelectorValue. If you want to set rules for more than one region, zone, or other worker node labels, create a new check. Define up to 15 checks in a config map. If you add more checks, only 1 worker node will be reloaded at a time until all workers requested are updated.

Example

apiVersion: v1
kind: ConfigMap
metadata:
name: ibm-cluster-update-configuration
namespace: kube-system
data:
drain_timeout_seconds: "120"
zonecheck.json: 
{
"MaxUnavailablePercentage": 30,
"NodeSelectorKey": "failure-domain.beta.kubernetes.io/zone",
"NodeSelectorValue": "dal13"
}
regioncheck.json: 
{
"MaxUnavailablePercentage": 20,
"NodeSelectorKey": "failure-domain.beta.kubernetes.io/region",
"NodeSelectorValue": "us-south"
}
defaultcheck.json: 
{
"MaxUnavailablePercentage": 20
}
<check_name>: 
{
"MaxUnavailablePercentage": <value_in_percentage>,
"NodeSelectorKey": "<node_selector_key>",
"NodeSelectorValue": "<node_selector_value>"
}
Show more
oc get nodes NODE-NAME -o yaml | grep zone
3. Label your Azure worker nodes with the zone value that you retrieved earlier. Replace <node_name> and <zone> with the node name and zone of your worker nodes. For example, if you have a worker node in zone: eastus-1, use the following commands to add eastus-1 as a label to the worker node in the eastus-1 zone.

oc label node <node_name> topology.kubernetes.io/zone-
oc label node <node_name> topology.kubernetes.io/zone=<zone> --overwrite
4. Repeat the previous steps for each worker node.
5. [Sign in to your Azure account](https://azure.microsoft.com/en-us/get-started/) and retrieve the required parameters. For more information about the parameters, see the [parameter reference](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-azurefile-csi-driverazurefile-csi-driver-parameter-reference).



Before you begin, review the [parameter reference](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-azurefile-csi-driverazurefile-csi-driver-parameter-reference) for the template version that you want to use.







 Creating and assigning a configuration in the console 



1. [From the Locations console](https://cloud.ibm.com/satellite/locations), select the location where you want to create a storage configuration.
2. Select Storage > Create storage configuration
3. Enter a name for your configuration.
4. Select the Storage type.
5. Select the Version and click Next
6. If the Storage type that you selected accepts custom parameters, enter them on the Parameters tab.
7. If the Storage type that you selected requires secrets, enter them on the Secrets tab.
8. On the Storage classes tab, review the storage classes that are deployed by the configuration or create a custom storage class.
9.
* [VLAN connections for worker nodes](https://cloud.ibm.com/docs/containers?topic=containers-plan_basicsworker-worker-nodes)
* [Worker node communication across subnets and VLANs](https://cloud.ibm.com/docs/containers?topic=containers-plan_basicsworker-worker-subnets)



* [Worker-to-master and user-to-master communication: Service endpoints](https://cloud.ibm.com/docs/containers?topic=containers-plan_basicsworkeruser-master)



* [Public service endpoint only](https://cloud.ibm.com/docs/containers?topic=containers-plan_basicsworkeruser-master-endpoint)
* [Public and private cloud service endpoints](https://cloud.ibm.com/docs/containers?topic=containers-plan_basicsworkeruser-master-pub-priv)
* [Private service endpoint only](https://cloud.ibm.com/docs/containers?topic=containers-plan_basicsworkeruser-master-private)



* [Worker communication to other IBM Cloud services or on-premises networks](https://cloud.ibm.com/docs/containers?topic=containers-plan_basicsworker-services-onprem)



* [Communication with other IBM Cloud services over the private or public network](https://cloud.ibm.com/docs/containers?topic=containers-plan_basicsworker-services-onprem-pub-priv)
* [IBM Cloud® Direct Link for communication over the private network with resources in on-premises data centers](https://cloud.ibm.com/docs/containers?topic=containers-plan_basicsworker-services-onprem-direct-link)
* [strongSwan IPSec VPN connection for communication over the public network with resources in on-premises data centers](https://cloud.ibm.com/docs/containers?topic=containers-plan_basicsworker-services-onprem-strongswan)
oc label nodes <worker-IP> <worker-IP> <worker-IP> "storage=local-block"

Example output

node/<worker-IP> labeled
node/<worker-IP> labeled
node/<worker-IP> labeled
3. Verify that the label is added to the worker nodes that you want to use. Run the following command to display the labels on your worker nodes and highlight the label that you added previous step.

oc get nodes --show-labels | grep --color=always storage=local-block



Before you begin, review the [parameter reference](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-local-volume-blocklocal-volume-block-parameter-reference) for the template version that you want to use.







 Creating and assigning a configuration in the console 



1. [From the Locations console](https://cloud.ibm.com/satellite/locations), select the location where you want to create a storage configuration.
2. Select Storage > Create storage configuration
3. Enter a name for your configuration.
4. Select the Storage type.
5. Select the Version and click Next
6. If the Storage type that you selected accepts custom parameters, enter them on the Parameters tab.
7. If the Storage type that you selected requires secrets, enter them on the Secrets tab.
8. On the Storage classes tab, review the storage classes that are deployed by the configuration or create a custom storage class.
9. On the Assign to service tab, select the service that you want to assign your configuration to.
10. Click Complete to assign your storage configuration.







 Creating a configuration in the CLI 



1. Log in to the IBM Cloud CLI.

ibmcloud login
2. List your Satellite locations and note the Managed from column.

ibmcloud sat location ls
3. Target the Managed from region of your Satellite location. For example, for wdc target us-east.
Debugging worker nodes with Kubernetes API 

If you have access to the cluster, you can debug the worker nodes by using the Kubernetes API on the Node resource.

Before you begin, make sure that you have the Manager service access role in all namespaces for the cluster, which corresponds to the cluster-admin RBAC role.



1. [Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.](https://cloud.ibm.com/docs/containers?topic=containers-access_cluster)
2. List the worker nodes in your cluster and note the NAME of the worker nodes that are not in a ReadySTATUS. Note that the NAME is the private IP address of the worker node.

kubectl get nodes
3. Describe the each worker node, and review the Conditions section in the output.



* Type: The type of condition that might affect the worker node, such as memory or disk pressure.
* LastTransitionTime: The most recent time that the status was updated. Use this time to identify when the issue with your worker node began, which can help you further troubleshoot the issue.



kubectl describe node <name>
4. Check the usage of the worker nodes.



1. In the Allocated resources output of the previous command, review the workloads that use the worker node's CPU and memory resources. You might notice that some pods don't set resource limits, and are consuming more resources than you expected. If so, adjust the resource usage of the pods.
2. Review the percentage of usage of CPU and memory across the worker nodes in your cluster. If the usage is consistently over 80%, [add more worker nodes](https://cloud.ibm.com/docs/containers?topic=containers-add_workers) to the cluster to support the workloads.



5. Check for custom admission controllers that are installed in your cluster. Admission controllers often block required pods from running, which might make your worker nodes enter a critical state.
Your Flow Logs for VPC gathers information from the VPC, VPC subnet, or VPC load balancer level. However, you can use the flow logs to gather information that is specific to your worker nodes. Separate flow log files are created for ingress and egress traffic.



1. In the CLI, find the ibm-cloud.kubernetes.io/instance-id label value for the worker node.

kubectl describe node <worker_node_ip> | grep instance-id

Example output.

ibm-cloud.kubernetes.io/instance-id=1010_a1aa1010-a1a0-1010-a1aa-aa1a1-a1-aa1
2. In the IBM Cloud UI, click your IBM Cloud Object Storage instance in the Resource list.
3. Click the bucket where your flow logs are collected.
4. Download and decompress the flow log object.
5. Open the file and navigate through the file directory until you reach directories that begin with instance-id=.
6. Find the file directory that contains the instance ID found in the first step. The ID is included at the end of the file directory name. Example.

instance-id=crn%3AV1%...%3Ainstance%3A1010_a1aa1010-a1a0-1010-a1aa-aa1a1-a1-aa1
7. In the instance=id= directory, locate the record-type=ingress and record-type=egress files. Your ingress and egress traffic logs are located here.
Any data that was stored only on the worker node, such as container images, are also deleted.





 Deployed state 

You can view the current worker node state by running the ibmcloud ks worker ls --cluster <cluster_name_or_ID> command and locating the State and Status fields.

Updates are successfully deployed to your worker node. After updates are deployed, IBM Cloud Kubernetes Service starts a health check on the worker node. After the health check is successful, the worker node goes into a Normal state. Worker nodes in a Deployed state usually are ready to receive workloads, which you can check by running kubectl get nodes and confirming that the state shows Normal.





 Deploying state 

You can view the current worker node state by running the ibmcloud ks worker ls --cluster <cluster_name_or_ID> command and locating the State and Status fields.

A Deploying state means that when you update the Kubernetes version of your worker node, your worker node is redeployed to install the updates. If you reload or reboot your worker node, the worker node is redeployed to automatically install the latest patch version. If your worker node is stuck in this state for a long time, check whether a problem occurred during the deployment.





 Deploy_failed state 

You can view the current worker node state by running the ibmcloud ks worker ls --cluster <cluster_name_or_ID> command and locating the State and Status fields.

A Deploy_failed state means that your worker node could not be deployed. List the details for the worker node to find the details for the failure by running ibmcloud ks worker get --cluster <cluster_name_or_id> --worker <worker_node_id>.





 Normal state 

You can view the current worker node state by running the ibmcloud ks worker ls --cluster <cluster_name_or_ID> command and locating the State and Status fields.

A Normal state means that your worker node is fully provisioned and ready to be used in the cluster. This state is considered healthy and does not require an action from the user.
1. List your Azure worker nodes and make a note of the name of each node.

oc get nodes
2. Get the details of each node and make a note of the zone that the node is in. For example: eastus-1.

oc get nodes <node-name> -o yaml | grep zone
3. Label your worker nodes with the zone value that you retrieved earlier. Replace <node-name> and <zone> with the node name and zone of your worker node. For example, if you have a worker node in zone: eastus-1, use the following commands to add eastus-1 as a label to the worker node in the eastus-1 zone.

oc label node <node-name> topology.kubernetes.io/zone-
oc label node <node-name> topology.kubernetes.io/zone=<zone> --overwrite
4. Repeat the previous steps for each worker node
5. [Sign in to your Azure account](https://azure.microsoft.com/en-us/get-started/) and retrieve the required parameters. For more information about the parameters, see [Cluster config](https://cloud-provider-azure.sigs.k8s.io/install/configs/cluster-config).



Before you begin, review the [parameter reference](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-azuredisk-csi-driver&interface=uiazuredisk-csi-driver-parameter-reference) for the template version that you want to use.







 Creating and assigning a configuration in the console 



1. [From the Locations console](https://cloud.ibm.com/satellite/locations), select the location where you want to create a storage configuration.
2. Select Storage > Create storage configuration
3. Enter a name for your configuration.
4. Select the Storage type.
5. Select the Version and click Next
6. If the Storage type that you selected accepts custom parameters, enter them on the Parameters tab.