vania-janet/multiturn-rag-retrieval-data huggingface dataset:
[Access your Red Hat OpenShift cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-access_cluster).



1. Log in to each worker node in your cluster by using the oc debug command and complete the following steps.



* Log in to the worker node. Replace <worker_node_IP> with the name of your worker node. To get the names of your worker nodes, run the oc get nodes command.

oc debug node/<node name> -- chroot /host rm -rvf /var/lib/rook /mnt/local-storage
* For each disk partition, clear the xfs file system on the worker node. If you don't clear the file system, the OSD is not created.

file -sL /dev/<partition>
wipefs -a /dev/<partition>
* Edit the /etc/kubernetes/kubelet.conf file and change the value of the EnableControllerAttachDetach parameter to true.

nano /etc/kubernetes/kubelet.conf
* Save and exit by using ctrl + X.
* Restart the kubelet.

systemctl restart kubelet
* Log out of the worker node.

exit



2. Repeat the previous steps to wipe the file system for each worker node that you want to use in your ODF deployment.







 Getting your device details 

Optional for add-on version 4.8 and later. You can use automatic disk discovery to find available devices for ODF. However, if you want to manually specify storage devices for ODF, complete the following steps to retrieve your storage device details.

Before you install ODF, get the details of the local disks on your worker nodes.



1. Log in to your cluster and get a list of available worker nodes. Make a note of the worker nodes that you want to use in your OCS deployment.

oc get nodes
2. Log in to each worker node that you want to use for your ODF.

oc debug node/<node-name>
If your worker nodes switch between a Normal and Critical or NotReady state, check the following components for any issues or recent changes that may disrupt your worker nodes.



1. For classic clusters, check your firewalls or gateways. If there is a bandwidth limit or any type of malfunction, resolve the issue. Then, check your worker nodes again.
2. Check if the applications, security, or monitoring components in your cluster are overloading the cluster apiserver with requests, which may cause disruptions for your worker nodes.
3. If you recently added any components to your cluster, remove them. If you made changes to any existing components in your cluster, revert the changes. Then, check the status of your worker nodes to see whether the new components or changes were causing the issue.
4. Check for changes on any cluster webhooks, which can disrupt apiserver requests or block a worker node's ability to connect with the apiserver. [Remove all webhooks](https://cloud.ibm.com/docs/containers?topic=containers-ts-delete-webhooks) that were added to the cluster after it was created.
5. If the issue is still not resolved, follow the steps to [gather your worker node data](https://cloud.ibm.com/docs/containers?topic=containers-ts-critical-notreadyts-critical-notready-gather) and open a support ticket.









 Gathering data for a support case 

If you are unable to resolve the issue with the troubleshooting steps, gather information about your worker nodes. Then, [open a support ticket](https://cloud.ibm.com/unifiedsupport/cases/form) and include the worker node information you gathered.
Note that for version 4.8 clusters and later, you can automatically find the available devices on your worker nodes by setting the auto-discover-devices=true parameter. However, if you have a 4.7 cluster, you must complete the following steps to retrieve the device paths for the disks on your worker nodes.

When you create your ODF configuration, you must specify the device paths of the disks that you want to use in your storage cluster. The storage cluster is comprised of the object storage daemon (OSD) pods and the monitoring (MON) pods. The devices that you specify as OSD devices are your storage devices where your app data is stored and the devices that you specify as MON devices are managed by the MON pod and used to store and maintain the storage cluster mapping and monitor storage events. For more information about the OSD and MON, see the [Ceph documentation](https://docs.ceph.com/en/latest/start/intro/).



1. Log in to your cluster and get a list of available worker nodes. Make a note of the worker nodes that you want to use in your ODF configuration.

oc get nodes
2. Log in to each worker node that you want to use for your ODF configuration.

oc debug node/<node-name>
3. When the debug pod is deployed on the worker node, run the following commands to list the available disks on the worker node.



1. Allow host binaries.

chroot /host
2. List your devices.

lsblk



4. Review the command output for available disks. Disks that can be used for your ODF configuration must be unmounted. In the following example output from the lsblk command, the sdc disk has two available, unformatted partitions that you can use for the OSD and MON device paths for this worker node. If your worker node has raw disks without partitions, you need one disk for the OSD and one disk for the MON. As a best practice, and to maximize storage capacity on this disk, you can specify the smaller partition or disk for the MON, and the larger partition or disk for the OSD.
How many worker nodes do I need to handle my workload? 

Now that you have a good idea of what your workload looks like, let's map the estimated usage onto your available cluster configurations.



1. Estimate the max worker node capacity, which depends on what type of cluster you have. You don't want to max out worker node capacity in case a surge or other temporary event happens.



* Single zone clusters: Plan to have at least three worker nodes in your cluster. Further, you want one extra node's worth of CPU and memory capacity available within the cluster.
* Multizone clusters: Plan to have at least two worker nodes per zone, so six nodes across three zones in total. Additionally, plan for the total capacity of your cluster to be at least 150% of your total workload's required capacity, so that if one zone goes down, you have resources available to maintain the workload.



2. Align the app size and worker node capacity with one of the [available worker node flavors](https://cloud.ibm.com/docs/openshift?topic=openshift-planning_worker_nodesplanning_worker_nodes). To see available flavors in a zone, run ibmcloud oc flavors --zone <zone>.



* Don't overload worker nodes: To avoid your pods competing for CPU or running inefficiently, you must know what resources your apps require so that you can plan the number of worker nodes that you need. For example, if your apps require less resources than the resources that are available on the worker node, you can limit the number of pods that you deploy to one worker node. Keep your worker node at around 75% capacity to leave space for other pods that might need to be scheduled. If your apps require more resources than you have available on your worker node, use a different worker node flavor that can fulfill these requirements. You know that your worker nodes are overloaded when they frequently report back a status of NotReady or evict pods due to the lack of memory or other resources.
* Larger versus smaller worker node flavors: Larger nodes can be more cost efficient than smaller nodes, particularly for workloads that are designed to gain efficiency when they process on a high-performance machine.
* [Worker node](https://cloud.ibm.com/docs/openshift?topic=openshift-securityworkernodes)



* [Who owns the worker node and am I responsible to secure it?](https://cloud.ibm.com/docs/openshift?topic=openshift-securityworker-node-owner)
* [How does my worker node setup look?](https://cloud.ibm.com/docs/openshift?topic=openshift-securityworker-node-setup)



* [Network](https://cloud.ibm.com/docs/openshift?topic=openshift-securitynetwork)



* [Network segmentation and privacy for classic clusters](https://cloud.ibm.com/docs/openshift?topic=openshift-securitynetwork_segmentation)
* [What network traffic is allowed for my Classic cluster by default?](https://cloud.ibm.com/docs/openshift?topic=openshift-securitydefault-network-traffic-allowed)
* [What is network segmentation and how can I set it up for a Classic cluster?](https://cloud.ibm.com/docs/openshift?topic=openshift-securitynetwork-segmentation-setup)
* [What else can I do to reduce the surface for external attacks for Classic clusters?](https://cloud.ibm.com/docs/openshift?topic=openshift-securityexternal-what-else)
* [What if I want to connect my cluster to an on-prem data center?](https://cloud.ibm.com/docs/openshift?topic=openshift-securityonprem-network-setup)
* [Network segmentation and privacy for VPC clusters](https://cloud.ibm.com/docs/openshift?topic=openshift-securitynetwork_segmentation_vpc)
* [What network traffic is allowed for my VPC cluster by default?]
* VPC clusters: Worker nodes are provisioned in to an IBM Cloud account that is owned by IBM to enable monitoring of malicious activities and apply security updates. You can't access your worker nodes by using the VPC dashboard. However, you can manage your worker nodes by using the IBM Cloud Kubernetes Service console, CLI, or API. The virtual machines that make up your worker nodes are dedicated to you and you are responsible to request timely updates so that your worker node OS and IBM Cloud Kubernetes Service components apply the latest security updates and patches.



For more information, see [Your responsibilities by using IBM Cloud Kubernetes Service](https://cloud.ibm.com/docs/containers?topic=containers-responsibilities_iks).

Use the ibmcloud ks worker update[command](https://cloud.ibm.com/docs/containers?topic=containers-kubernetes-service-clics_worker_update) regularly (such as monthly) to deploy updates and security patches to the operating system and to update the Kubernetes version that your worker nodes run. When updates are available, you are notified when you view information about the master and worker nodes in the IBM Cloud console or CLI, such as with the ibmcloud ks clusters ls or ibmcloud ks workers ls --cluster <cluster_name> commands. Worker node updates are provided by IBM as a full worker node image that includes the latest security patches. To apply the updates, the worker node must be reimaged and reloaded with the new image. Keys for the root user are automatically rotated when the worker node is reloaded.





 How does my worker node setup look? 

The following image shows the components that are set up for every worker node to protect your worker node from malicious attacks.

The image does not include components that ensure secure end-to-end communication to and from the worker node. For more information, see [network security](https://cloud.ibm.com/docs/containers?topic=containers-securitynetwork).

Zoom

![Worker node setup in IBM Cloud Kubernetes Service excluding network security.]
* [Updating classic worker nodes](https://cloud.ibm.com/docs/containers?topic=containers-updateworker_node).
* [Updating VPC worker nodes](https://cloud.ibm.com/docs/containers?topic=containers-updatevpc_worker_node).









 Updating classic worker nodes 

You notice that an update is available for your worker nodes in a [classic infrastructure](https://cloud.ibm.com/docs/containers?topic=containers-infrastructure_providers) cluster. What does that mean? As security updates and patches are put in place for the API server and other master components, you must be sure that the worker nodes remain in sync. You can make two types of updates: updating only the patch version, or updating the major.minor version with the patch version.

Applies to only classic clusters. Have a VPC cluster? See [Updating VPC worker nodes](https://cloud.ibm.com/docs/containers?topic=containers-updatevpc_worker_node) instead.

For the latest security patches and fixes, make sure to update your worker nodes to the latest patch as soon as possible after it is available. For more information about the latest updates, review the [change log](https://cloud.ibm.com/docs/containers?topic=containers-changelog).

To update Satellite worker nodes, see [Updating hosts that are assigned as worker nodes to Satellite-enabled services](https://cloud.ibm.com/docs/satellite?topic=satellite-host-update-workers).



* Patch: A worker node patch update includes security fixes. You can update the classic worker node to the latest patch by using the ibmcloud ks worker reload or update commands. Keep in mind that the update command also updates the worker node to the same major.minor version as the master and latest patch version, if a major.minor version update is also available.
* Major.minor: A major.minor update moves up the Kubernetes version of the worker node to the same version as the master.
ibmcloud ks worker ls --cluster CLUSTER
3. View the labels of a worker node. You can find the worker node labels in the Labels section of your CLI output. Every label consists of a NodeSelectorKey and a NodeSelectorValue.

kubectl describe node PRIVATE-WORKER-IP

Example output

NAME: 10.184.58.3
Roles: <none>
Labels: arch=amd64
beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
failure-domain.beta.kubernetes.io/region=us-south
failure-domain.beta.kubernetes.io/zone=dal12
ibm-cloud.kubernetes.io/encrypted-docker-data=true
ibm-cloud.kubernetes.io/iaas-provider=softlayer
ibm-cloud.kubernetes.io/machine-type=u3c.2x4.encrypted
kubernetes.io/hostname=10.123.45.3
privateVLAN=2299001
publicVLAN=2299012
Annotations: node.alpha.kubernetes.io/ttl=0
volumes.kubernetes.io/controller-managed-attach-detach=true
CreationTimestamp: Tue, 03 Apr 2022 15:26:17 -0400
Taints: <none>
Unschedulable: false
4. Create a config map and define the unavailability rules for your worker nodes. The following example shows four checks, the zonecheck.json, regioncheck.json, defaultcheck.json, and a check template. You can use these example checks to define rules for worker nodes in a specific zone (zonecheck.json), region (regioncheck.json), or for all worker nodes that don't match any of the checks that you defined in the config map (defaultcheck.json). Use the check template to create your own check.
Troubleshooting worker nodes in Critical or NotReady state 

Cluster worker nodes go into a Critical or NotReady state when they stop communicating with the cluster master. When this occurs, your worker nodes are marked as Critical in the IBM Cloud UI or when you run ibmcloud oc worker commands, and as NotReady in the Red Hat OpenShift dashboards and when you run oc get nodes. There are several reasons why communication stops between worker nodes and the cluster master. Follow these steps to troubleshoot worker nodes in these states.

Check the IBM Cloud health and status dashboard for any notifications or maintenance updates that might be relevant to your worker nodes. These notifications or updates might help determine the cause of the worker node failures.



 Check the common causes of worker node failures 

There are several reasons why communication stops between worker nodes and the cluster master. Check whether the following common issues are causing the disruption.

The worker was deleted, reloaded, updated, replaced, or rebooted
: Worker nodes might temporarily show a Critical or NotReady state when they are deleted, reloaded, updated, or replaced. If any of these actions have been initiated on your worker node, whether manually or as part of an automation setup such as cluster autoscaler, wait until the actions are complete. Then, check the status of your worker nodes again. If any workers remain in the Critical or NotReady state, [reload](https://cloud.ibm.com/docs/containers?topic=containers-kubernetes-service-cli&interface=uics_worker_reload) or [replace](https://cloud.ibm.com/docs/containers?topic=containers-kubernetes-service-cli&interface=uicli_worker_replace) the affected workers.
: If a worker node was reloaded or replaced and initially works correctly, but then after some time goes back into a Critical or NotReady state, then it is likely that some workload or component on the worker is causing the issue. See [Debugging worker nodes](https://cloud.ibm.com/docs/containers?topic=containers-debug-kube-nodes) to isolate the problem workload.
false 
 workerNodes Optional: Enter the names of the worker nodes that you want to use for your ODF deployment. Don't specify this parameter if you want to use all the worker nodes in your cluster. To retrieve your worker node name, run the oc get nodes command. N/A 
 clusterEncryption Available for add-on version 4.7.0 and later. Enter true or false to enable encryption. 
 autoDiscoverDevices Optional: Automatically discover the available disks on your worker nodes. Enter true or false. 









 Limitations 

You can't use both the ibmcloud-block-storage-plugin and the ODF add-on at the same time. To install ODF, you must first edit the /etc/kubernetes/kubelet.conf file and change the value of the EnableControllerAttachDetach parameter to true which changes the default volume attachment behavior for the cluster. This means you can't dynamically provision volumes by using the ibmc-block-* storage classes. Instead, you must create volumes by using the [ODF storage classes](https://cloud.ibm.com/docs/openshift?topic=openshift-ocs-sc-ref).
From the IBM Cloud console [resource list](https://cloud.ibm.com/resources), click on your cluster.
2. Click Worker nodes.
3. Note each Public VLAN used by the worker nodes in your cluster. Multiple worker nodes may use the same public VLAN.
4. From the [IBM Cloud console](https://cloud.ibm.com/) menu ![Menu icon](https://cloud.ibm.com/docs-content/v1/content/4d497aff9e27b92fbbc2ee4e343a9ec4549cb1d7/icons/icon_hamburger.svg), click Classic Infrastructure > IP Management > VLANs.
5. Click on each public VLAN to check if it is used by the worker nodes in your cluster.
6. For each public VLAN that is used by the worker nodes in your cluster, find the Subnets section and note each IP address included in the table. These are the IP addresses that you must include in your allowlist.







 Getting your subnet IP addresses in the CLI 



1. List the public VLANs your worker nodes use. The output is formatted as publicVLAN=<vlan_id>.

kubectl describe nodes | grep publicVLAN | sort | uniq
2. For each public VLAN, find the associated public subnets. In the output, note the subnet IP addresses in the identifier column.

ibmcloud sl subnet list | grep <vlan-id>

Example output of subnets associated with public VLAN with ID 2761690.

ID identifier type network_space datacenter vlan_id IPs hardware virtual_servers
1962263 169.62.46.56 SECONDARY_ON_VLAN PUBLIC wdc07 2761690 8 0 0
2008207 169.62.39.248 SECONDARY_ON_VLAN PUBLIC wdc07 2761690 8 0 0
2342562 169.62.2.128 ADDITIONAL_PRIMARY PUBLIC wdc07 2761690 16 0 5
3.
If the details include an error message, review the list of [common error messages for worker nodes](https://cloud.ibm.com/docs/openshift?topic=openshift-common_worker_nodes_issues) to learn how to resolve the problem.

ibmcloud oc worker get --cluster <cluster_name_or_id> --worker <worker_node_id>





 Step 4: Review the infrastructure provider for the worker node 

Review the infrastructure environment to check for other reasons that might cause the worker node issues.



1. Check with your networking team to make sure that no recent maintenance, such as firewall or subnet updates, might impact the worker node connections.
2. Review [IBM Cloud](https://cloud.ibm.com/status/) for Red Hat OpenShift on IBM Cloud and the underlying infrastructure provider, such as Virtual Servers for classic, VPC related components, or Satellite.
3. If you have access to the underlying infrastructure, such as classic Virtual Servers, review the details of the corresponding machines for the worker nodes.
1 root root 10 Feb 9 04:15 scsi-3600605b00d87b43027b3bbb603150cc6-part2 -> ../../sda2
lrwxrwxrwx. 1 root root 10 Feb 9 04:15 scsi-3600605b00d87b43027b3bbb603150cc6-part3 -> ../../sda3
lrwxrwxrwx. 1 root root 9 Feb 9 04:15 scsi-3600605b00d87b43027b3bbf306bc28a7 -> ../../sdb
lrwxrwxrwx. 1 root root 10 Feb 9 04:15 scsi-3600605b00d87b43027b3bbf306bc28a7-part1 -> ../../sdb1
lrwxrwxrwx. 1 root root 9 Feb 9 04:17 scsi-3600605b00d87b43027b3bc310a64c6c9 -> ../../sdc
lrwxrwxrwx. 1 root root 10 Feb 11 03:14 scsi-3600605b00d87b43027b3bc310a64c6c9-part1 -> ../../sdc1
lrwxrwxrwx. 1 root root 10 Feb 11 03:15 scsi-3600605b00d87b43027b3bc310a64c6c9-part2 -> ../../sdc2
7. Repeat the previous steps for each worker node that you want to use for your OpenShift Data Foundation deployment.
8.
Planning your worker node setup 

Red Hat® OpenShift® on IBM Cloud® provides different worker node flavors and isolation levels so that you can choose the flavor and isolation that best meet the requirements of the workloads that you want to run in the cloud.

A worker node flavor describes the compute resources, such as CPU, memory, and disk capacity that you get when you provision your worker node. Worker nodes of the same flavor are grouped in worker node pools. The total number of worker nodes in a cluster determine the compute capacity that is available to your apps in the cluster.

Want to save on your classic worker node costs? [Create a reservation](https://cloud.ibm.com/docs/containers?topic=containers-reservations) to lock in a discount over 1 or 3 year terms! Then, create your worker pool by using the reserved instances. Note that autoscaling can't be enable on worker pools that use reservations.

Trying to plan how many worker nodes your need in your cluster? Check out [Sizing your Kubernetes cluster to support your workload](https://cloud.ibm.com/docs/openshift?topic=openshift-strategysizing) to find information about the default worker node setup and how you can determine the resource requirements of your workloads.



 Available hardware for worker nodes 

The worker node flavors and isolation levels that are available to you depend on your container platform, cluster type, the infrastructure provider that you want to use, and the Red Hat OpenShift on IBM Cloud location where you want to create your cluster.

Zoom

![Hardware options for worker nodes in a standard cluster](https://cloud.ibm.com/docs-content/v1/content/bce0f0917a9eea684d1b4b704ac6343a1f65f446/openshift/images/cs_clusters_hardware.png)

Figure 1. Hardware options for worker nodes in a standard cluster



 What flavors are available to me?
Get the following logs to look for issues on the worker node.

/var/log/syslog
/var/log/containerd.log
/var/log/kubelet.log
/var/log/kern.log
5. Run debug commands to help you gather information and troubleshoot issues. Commands that you might use to debug, such as ip, ifconfig, nc, ping, and ps, are already available in the shell. You can also install other tools, such as dig, tcpdump, mtr, and curl, by running apk add <tool>. For example, to add dig, run apk add bind-tools.

Before you can use the tcpdump command, you must first move the binary to a new location that does not conflict with the install path for the binary on the host. You can use the following command to relocate the binary: mv /usr/sbin/tcpdump /usr/local/bin/.
6. Delete the host access pod that you created for debugging.

kubectl delete pod debug-${NODE}







 Debugging by creating a pod with root SSH access 

If you are unable to use the kubectl debug node or kubectl exec commands, such as if the VPN connection between the cluster master and worker nodes is down, you can create a pod that enables root SSH access and copies a public SSH key to the worker node for SSH access.

Allowing root SSH access is a security risk. Only allow SSH access when it is required and no other option is available to troubleshoot worker node issues. When you finish troubleshooting, be sure to follow the steps in the [Cleaning up after debugging](https://cloud.ibm.com/docs/containers?topic=containers-cs_ssh_workerssh-debug-cleanup) section to disable SSH access.



1. Choose an existing or create a new public SSH key.

ssh-keygen -t rsa -b 4096
ls /.ssh

id_rsa id_rsa.pub