vania-janet/multiturn-rag-retrieval-data huggingface dataset:
If the previous steps do not solve the issue, [reload](https://cloud.ibm.com/docs/containers?topic=containers-kubernetes-service-cli&interface=uics_worker_reload) or [replace](https://cloud.ibm.com/docs/containers?topic=containers-kubernetes-service-cli&interface=uicli_worker_replace) the affected workers one at a time.



If some, but not all, of your worker nodes frequently enter a Critical or NotReady state, consider enabling [worker autorecovery](https://cloud.ibm.com/docs/containers?topic=containers-health-monitorautorecovery) to automate these recovery steps.





 If all worker nodes in a single zone, subnet, or VLAN are affected 

If all worker nodes in a single zone, subnet, or VLAN are in a Critical or NotReady state, but all other worker nodes in the cluster are functioning normally, there might be an issue with a networking component. Follow the steps in [If all worker nodes in a cluster are affected](https://cloud.ibm.com/docs/containers?topic=containers-ts-critical-notreadyts-critical-notready-steps-all), especially to the steps regarding any networking components that may affect the zone, subnet or VLAN, such as firewall or gateway rules, ACLs or custom routes, or Calico and Kubernetes network policies.

If you checked your networking components and still cannot resolve the issue, [gather your worker node data](https://cloud.ibm.com/docs/containers?topic=containers-ts-critical-notreadyts-critical-notready-gather) and open a support ticket.





 If all worker nodes in a cluster are affected 

If all the worker nodes in your cluster show Criticalor NotReady at the same time, there might be a problem with either the cluster apiserver or the networking path between the workers and the apiserver. Follow these troubleshooting steps to determine the cause and resolve the issue.

Some steps are specific to a specialized area, such as networking or automation. Consult with the relevant administrator or team in your organization before completing these steps.
Optional: Remove the worker nodes with the previous network metadata from the worker pool.



1. In the output of the previous step, note the ID of the worker nodes that you want to remove from the worker pool.
2. Remove the worker node.

ibmcloud ks worker rm --cluster <cluster_name_or_ID> --worker <worker_name_or_ID>
3. Verify that the worker node is removed.

ibmcloud ks worker ls --cluster <cluster_name_or_ID> --worker-pool <pool_name>
4. Rebalance the worker pool.

ibmcloud ks worker-pool rebalance --cluster <cluster_name_or_ID> --worker-pool <pool_name>



8. Optional: You can repeat steps 2 - 7 for each worker pool in your cluster. After you complete these steps, all worker nodes in your cluster are set up with the new VLANs.
9. The default ALBs in your cluster are still bound to the old VLAN because their IP addresses are from a subnet on that VLAN. Because ALBs can't be moved across VLANs, you can instead [create ALBs on the new VLANs and disable ALBs on the old VLANs](https://cloud.ibm.com/docs/containers?topic=containers-ingress-alb-managemigrate-alb-vlan).
10. Optional: If you no longer need the subnets on the old VLANs, you can [remove them](https://cloud.ibm.com/docs/containers?topic=containers-subnetsremove-subnets).
* [About updating the master](https://cloud.ibm.com/docs/containers?topic=containers-updatemaster-about)
* [Steps to update the cluster master](https://cloud.ibm.com/docs/containers?topic=containers-updatemaster-steps)



* [Updating classic worker nodes](https://cloud.ibm.com/docs/containers?topic=containers-updateworker_node)



* [Prerequisites](https://cloud.ibm.com/docs/containers?topic=containers-updateworker-up-prereqs)
* [Updating classic worker nodes in the CLI with a configmap](https://cloud.ibm.com/docs/containers?topic=containers-updateworker-up-configmap)
* [Updating classic worker nodes in the console](https://cloud.ibm.com/docs/containers?topic=containers-updateworker_up_console)



* [Updating VPC worker nodes](https://cloud.ibm.com/docs/containers?topic=containers-updatevpc_worker_node)



* [Prerequisites](https://cloud.ibm.com/docs/containers?topic=containers-updatevpc_worker_prereqs)
* [Updating VPC worker nodes in the CLI](https://cloud.ibm.com/docs/containers?topic=containers-updatevpc_worker_cli)
* [Updating VPC worker nodes in the console](https://cloud.ibm.com/docs/containers?topic=containers-updatevpc_worker_ui)



* [Updating flavors (machine types)](https://cloud.ibm.com/docs/containers?topic=containers-updatemachine_type)
* [Updating cluster components](https://cloud.ibm.com/docs/containers?topic=containers-updatecomponents)
Before you begin, [Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.](https://cloud.ibm.com/docs/containers?topic=containers-access_cluster)



1. Retrieve the following details of your VPC instance. These parameters are used to apply labels to your worker nodes.



* <instanceID> - The VPC instance ID. To retrieve this value, run ibmcloud is ins.
* <node-name> - The names of the worker nodes. To retrieve this value, run kubectl get nodes.
* <region-of-instanceID> and <zone-of-instanceID> - The region and zone that your VPC instance is in. To retrieve these values, run ibmcloud is in <instanceID>. Example region value: eu-de. Example zone value: eu-de-1.



2. Copy the following shell script and save it to a file on your local machine called setup.sh

!/bin/bash

function help()
{
echo "Run the script in the following format..."
echo "./setup.sh <node-name> <instanceID> <region-of-instanceID> <zone-of-instanceID>"
exit 1
}

function apply_labels()
{
kubectl label nodes $1 "ibm-cloud.kubernetes.io/worker-id"=$2
kubectl label nodes $1 "failure-domain.beta.kubernetes.io/region"=$3
kubectl label nodes $1 "failure-domain.beta.kubernetes.io/zone"=$4
kubectl label nodes $1 "topology.kubernetes.io/region"=$3
kubectl label nodes $1 "topology.kubernetes.io/zone"=$4
}

function verify_node()
{
kubectl get nodes | grep $1
if (( $? == 0 ))
then
If the worker node doesn't have public network access, you can maintain a copy of the image for debugging in your own ICR repository or build a customized image with other tools to fit your needs.

oc debug node/<NODE_NAME>
3. Run debug commands to help you gather information and troubleshoot issues. Commands that you might use to debug, such as tcpdump, curl, ip, ifconfig, nc, ping, and ps, are already available in the shell. You can also install other tools, such as mtr, by running yum install <tool>.







 Debugging by using kubectl exec 

If you are unable to use the oc debug node command, you can create an Alpine pod with a privileged securityContext and use the kubectl exec command to run debug commands from the pod's interactive shell.



1. Get the name of the worker node that you want to access. The worker node name is its private IP address.

oc get nodes -o wide
2. Export the name in an environment variable.

export NODE=<NODE_NAME>
3. Create a debug pod on the worker node. The Docker alpine image here is used as an example. If the worker node doesn't have public network access, you can maintain a copy of the image for debugging in your own ICR repository or build a customized image with other tools to fit your needs.

kubectl apply -f - << EOF
apiVersion: v1
kind: Pod
metadata:
name: debug-${NODE}
namespace: default
spec:
containers:
- args: ["-c", "apk add tcpdump mtr curl; sleep 1d"]
command: ["/bin/sh"]
image: icr.io/armada-master/alpine:latest
imagePullPolicy: IfNotPresent
name: debug
resources: {}
securityContext:
privileged: true
runAsUser: 0
volumeMounts:
Adding worker nodes and zones to clusters 

To increase the availability of your apps, you can add worker nodes to an existing zone or multiple existing zones in your cluster. To help protect your apps from zone failures, you can add zones to your cluster.

When you create a cluster, the worker nodes are provisioned in a worker pool. After cluster creation, you can add more worker nodes to a pool by resizing it or by adding more worker pools. By default, the worker pool exists in one zone. Clusters that have a worker pool in only one zone are called single zone clusters. When you add more zones to the cluster, the worker pool exists across the zones. Clusters that have a worker pool that is spread across more than one zone are called multizone clusters.

If you have a multizone cluster, keep its worker node resources balanced. Make sure that all the worker pools are spread across the same zones, and add or remove workers by resizing the pools instead of adding individual nodes. After you set up your worker pool, you can [set up the cluster autoscaler](https://cloud.ibm.com/docs/openshift?topic=openshift-cluster-scaling-classic-vpc) to automatically add or remove worker nodes from your worker pools based on your workload resource requests.



 Adding worker nodes by resizing an existing worker pool 

You can add or reduce the number of worker nodes in your cluster by resizing an existing worker pool, regardless of whether the worker pool is in one zone or spread across multiple zones.

For example, consider a cluster with one worker pool that has three worker nodes per zone.



* If the cluster is single zone and exists in dal10, then the worker pool has three worker nodes in dal10. The cluster has a total of three worker nodes.
* If the cluster is multizone and exists in dal10 and dal12, then the worker pool has three worker nodes in dal10 and three worker nodes in dal12. The cluster has a total of six worker nodes.



For bare metal worker pools, keep in mind that billing is monthly. If you resize up or down, it impacts your costs for the month.
[Planning your worker node setup](https://cloud.ibm.com/docs/openshift?topic=openshift-planning_worker_nodesplanning_worker_nodes)



* [Available hardware for worker nodes](https://cloud.ibm.com/docs/openshift?topic=openshift-planning_worker_nodesshared_dedicated_node)



* [What flavors are available to me?](https://cloud.ibm.com/docs/openshift?topic=openshift-planning_worker_nodesavailable-flavors)
* [Can I combine different flavors in a cluster?](https://cloud.ibm.com/docs/openshift?topic=openshift-planning_worker_nodescombine-flavors)
* [How can I change worker node flavors?](https://cloud.ibm.com/docs/openshift?topic=openshift-planning_worker_nodeschange-flavors)
* [Are the worker nodes encrypted?](https://cloud.ibm.com/docs/openshift?topic=openshift-planning_worker_nodesencrypted-flavors)
* [How do I manage my worker nodes?](https://cloud.ibm.com/docs/openshift?topic=openshift-planning_worker_nodesflavor-manage)
* [What limitations do I need to be aware of?](https://cloud.ibm.com/docs/openshift?topic=openshift-planning_worker_nodesflavor-limitations)
* [Why do my worker nodes have the master role?](https://cloud.ibm.com/docs/openshift?topic=openshift-planning_worker_nodesflavor-master-role)
* [How can I check the operating system that my worker nodes run?](https://cloud.ibm.com/docs/openshift?topic=openshift-planning_worker_nodesflavor-os-check)
* If the cluster is single zone and exists in dal10, then the worker pool has three worker nodes in dal10. The cluster has a total of three worker nodes.
* If the cluster is multizone and exists in dal10 and dal12, then the worker pool has three worker nodes in dal10 and three worker nodes in dal12. The cluster has a total of six worker nodes.



For bare metal worker pools, keep in mind that billing is monthly. If you resize up or down, it impacts your costs for the month. When you add worker nodes by resizing a worker pool, the new worker nodes run the same major.minor version as the cluster master, but the latest worker node patch of that major.minor version.

Before you begin, make sure that you have the [Operator or Administrator IBM Cloud IAM platform access role](https://cloud.ibm.com/docs/openshift?topic=openshift-users).

To resize the worker pool, change the number of worker nodes that the worker pool deploys in each zone:



1. Get the name of the worker pool that you want to resize.

ibmcloud ks worker-pool ls --cluster <cluster_name_or_ID>
2. Resize the worker pool by designating the number of worker nodes that you want to deploy in each zone. The minimum value is 1. For more information, see [What is the smallest size cluster that I can make?](https://cloud.ibm.com/docs/containers?topic=containers-faqssmallest_cluster).

ibmcloud ks worker-pool resize --cluster <cluster_name_or_ID> --worker-pool <pool_name> --size-per-zone <number_of_workers_per_zone>
3. Verify that the worker pool is resized.

ibmcloud ks worker ls --cluster <cluster_name_or_ID> --worker-pool <pool_name>

Example output for a worker pool that is in two zones, dal10 and dal12, and is resized to two worker nodes per zone:
Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 1.17.14_1545

 Component Previous Current Description 

 Containerd v1.3.4 1.3.9 See the [containerd release notes](https://github.com/containerd/containerd/releases/tag/v1.3.9). The update resolves CVE-2020–15257 (see the [IBM security bulletin](https://www.ibm.com/support/pages/node/6387878)). 
 HAProxy 1.8.26-384f42 db4e6d Added provenance labels for source tracking. 
 Ubuntu 18.04 packages 4.15.0-123-generic 4.15.0-126-generic Updated worker node image with kernel and package updates for [CVE-2020-14351](https://nvd.nist.gov/vuln/detail/CVE-2020-14351) and [CVE-2020-4788](https://nvd.nist.gov/vuln/detail/CVE-2020-4788). 
 Ubuntu 16.04 packages 4.4.0-194-generic 4.4.0-197-generic Updated worker node image with kernel and package updates for [CVE-2020-0427](https://nvd.nist.gov/vuln/detail/CVE-2020-0427), [CVE-2020-12352](https://nvd.nist.gov/vuln/detail/CVE-2020-12352), [CVE-2020-14351](https://nvd.nist.gov/vuln/detail/CVE-2020-14351), [CVE-2020-25645](https://nvd.nist.gov/vuln/detail/CVE-2020-25645), and [CVE-2020-4788](https://nvd.nist.gov/vuln/detail/CVE-2020-4788).
[Manually add](https://cloud.ibm.com/docs/containers?topic=containers-utilitiesmanual_block) a 3 GB block storage device to at least 3 worker nodes in your cluster and find the device path. To find the device path after attaching the disk, log in to your worker node and run lsblk to list the devices on that node.
* Select one of the block storage devices that you added earlier and find the device path. To find the device path after attaching the disk, log in to your worker node and run lsblk to list the devices on that node.



5. [Attach the block storage](https://cloud.ibm.com/docs/containers?topic=containers-utilitiesattach_block) to your worker nodes.
6. Continue with your Portworx setup by [Setting up a key-value store for Portworx metadata](https://cloud.ibm.com/docs/containers?topic=containers-storage_portworx_kv_store).







 VPC clusters 

Virtual Private Cloud



1. Follow the [steps](https://cloud.ibm.com/docs/containers?topic=containers-utilitiesvpc_cli_attach) to create the Block Storage for VPC instances and attach these to each worker node that you want to add to the Portworx storage layer. For highly available data storage, Portworx requires at least 3 worker nodes with raw and unformatted block storage.
2. If you want to use [journal devices](https://docs.portworx.com/install-with-other/operate-and-maintain/performance-and-tuning/tuning/), choose from the following options. Journal devices allow Portworx to write logs directly to a local disk on your worker node.



* [Attach](https://cloud.ibm.com/docs/containers?topic=containers-utilitiesvpc_cli_attach) an additional 3 GB disk to at least 3 worker nodes in your cluster and find the device path.
Change log for worker node fix pack 1.18.6_1523, released 3 August 2020 

The following table shows the changes that are in the worker node fix pack 1.18.6_1523. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 1.18.6_1520

 Component Previous Current Description 

 Ubuntu 18.04 packages 4.15.0-111-generic 4.15.0-112-generic Updated worker node images with kernel and package updates for [CVE-2019-17514](https://nvd.nist.gov/vuln/detail/CVE-2019-17514), [CVE-2019-20907](https://nvd.nist.gov/vuln/detail/CVE-2019-20907), [CVE-2019-9674](https://nvd.nist.gov/vuln/detail/CVE-2019-9674), [CVE-2020-10713](https://nvd.nist.gov/vuln/detail/CVE-2020-10713), [CVE-2020-10757](https://nvd.nist.gov/vuln/detail/CVE-2020-10757), [CVE-2020-11935
* [Update the flavor](https://cloud.ibm.com/docs/containers?topic=containers-updatemachine_type) in your cluster by creating a worker pool and removing the previous worker pool.









 Modifying worker node settings to optimize performance 



 Modifying worker node settings by using the Node Tuning Operator 

You can use the node tuning operator to tune worker node performance by creating custom profiles. For more information, see the Red Hat [Node Tuning Operator](https://docs.openshift.com/container-platform/4.7/scalability_and_performance/using-node-tuning-operator.html) docs.

Red Hat SRE is not responsible for the tuning and performance management of nodes that are customized with the Node Tuning Operator (NTO). If changes made by the NTO cause the worker nodes to be unavailable for scheduling or unresponsive, then revert the changes made by using the NTO.



1. Save the following example Tuned resource to a file called tuned-node.yaml and include the specifications you want to use for your worker nodes. Note that you can use the recommend method to apply the settings to your worker nodes by using node labels. In this example, the profile is called tuned-node and applies to worker nodes with the label label: node-role.kubernetes.io/master label.

If you have custom labels on your worker nodes, or if you want to tune only certain workers in your cluster, you can [label your worker nodes](https://cloud.ibm.com/docs/openshift?topic=openshift-add_workersworker_pool_labels), then use those labels in your tuning configuration.

Example custom Tuned profile. For more information about customizing your Tuned profile, see [Node Tuning Operator](https://docs.openshift.com/container-platform/4.7/scalability_and_performance/using-node-tuning-operator.html) docs.

apiVersion: tuned.openshift.io/v1
kind: Tuned
metadata:
name: tuned-node
namespace: openshift-cluster-node-tuning-operator
spec:
profile:
By default, the cluster deploys Linux® worker nodes. If the variable is set to true, the values of both worker_node_min_count and worker_node_max_count should be equal because the current implementation doesn't support dynamic creation of worker nodes through Host Factory. No false 
 worker_node_instance_type Specify the virtual server instance or bare metal server profile type name to be used to create the worker nodes for the Spectrum Symphony cluster based on worker_node_type. The worker nodes are the ones where the workload execution takes place, and the choice should be made according to the characteristic of the workloads. For more information, see [virtual server instance profiles](https://cloud.ibm.com/docs/vpc?topic=vpc-profiles) and [bare metal server profiles](https://cloud.ibm.com/docs/vpc?topic=vpc-bare-metal-servers-profile&interface=ui). Note: If dedicated_host_enabled == true, available instance prefix (for example, bx2 and cx2) can be limited depending on your target region. You can verify with the following IBM Cloud CLI commands: ibmcloud target -r {region_name} and ibmcloud is dedicated-host-profiles. No bx2-4x16 
 worker_node_max_count The maximum number of virtual server instance or bare metal server worker nodes that can be provisioned in the cluster. To take advantage of the auto-scale feature from [Host Factory](https://www.ibm.com/docs/en/spectrum-symphony/7.3.1?topic=factory-overview), the value needs to be greater than the worker_node_min_count. Enter a value in the range 1 - 500. if you are using bare metal servers, the value needs to match the worker_node_min_count, and the permitted value is in the range 1 - 16. Note: If you plan to use Spectrum Scale storage, the value for this parameter should be equal to worker_node_min_count. No 10
For a list of available worker node flavors, run ibmcloud oc flavors --zone <zone> --provider classic, or see [Classic flavors](https://cloud.ibm.com/docs/containers?topic=containers-classic-flavors&interface=ui).

hardware
: The level of hardware isolation for your worker nodes. Use dedicated to have available physical resources dedicated to you only, or shared to allow physical resources to be shared with other IBM customers. This option is available for virtual machine worker nodes flavors only.

public_vlan_id and private_vlan_id
: Optional. The ID of the public or private VLAN that you want to use for your worker nodes. To find available VLANs and subnets, run ibmcloud oc vlans --zone <zone>.

subnet_id
: Optional. The ID of an existing subnet that you want to use for your worker nodes. To find existing subnets, run ibmcloud oc subnets --provider classic --zone <zone>.

default_pool_size
: The number of worker nodes that you want to add to the default worker pool.
3. In the CLI, navigate to your Terraform directory.

cd <terraform_directory>
4. Run the commands to initialize and plan your Terraform actions. Review the plan output to make sure the correct actions are performed.

terraform init

terraform plan
5. Apply the Terraform files to create the cluster. Then, navigate to the IBM Cloud console to check that the cluster is provisioning.

terraform apply
If you are updating your worker nodes to a new major version, such as from 4.11 to 4.12, update the cluster master first.

ibmcloud oc cluster master update --cluster CLUSTER [--version MAJOR.MINOR.PATCH] [--force-update] [-f] [-q]

Example command:

ibmcloud oc cluster master update --cluster mycluster --version 4.11.42 --force-update
2. Wait until the master update finishes.







 Step 2: Determine which worker nodes you want to update 

Major

Minor



1. List your worker nodes by using the oc get nodes command and determining which worker nodes you want to update.

oc get nodes

Example output

NAME STATUS ROLES AGE VERSION
10.241.0.4 Ready master,worker 106s v1.21.6+4b61f94
10.241.128.4 Ready master,worker 22d v1.21.6+bb8d50a
10.241.64.4 Ready master,worker 22d v1.21.6+bb8d50a







 Step 3: Scale down OpenShift Data Foundation 

Major

Minor



1. For each worker node that you found in the previous step, find the rook-ceph-mon and rook-ceph-osd deployments.

oc get pods -n openshift-storage -o wide | grep -i <node_name>
2. Scale down the deployments that you found in the previous step.

oc scale deployment rook-ceph-mon-c --replicas=0 -n openshift-storage
oc scale deployment rook-ceph-osd-2 --replicas=0 -n openshift-storage
oc scale deployment --selector=app=rook-ceph-crashcollector,node_name=NODE-NAME --replicas=0 -n openshift-storage







 Step 4: Cordon and drain the worker node 

Major

Minor



1. Cordon the node. Cordoning the node prevents any pods from being scheduled on this node.
Change log for worker node fix pack 1.25.9_1544, released 9 May 2023 

The following table shows the changes that are in the worker node fix pack 1.25.9_1544. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 1.25.9_1543

 Component Previous Current Description 

 Ubuntu 18.04 packages 4.15.0-209-generic 4.15.0-210-generic Worker node kernel & package updates for [CVE-2023-1786](https://nvd.nist.gov/vuln/detail/CVE-2023-1786), [CVE-2023-0464](https://nvd.nist.gov/vuln/detail/CVE-2023-0464), [CVE-2023-0466](https://nvd.nist.gov/vuln/detail/CVE-2023-0466), [CVE-2023-1829](https://nvd.nist.gov/vuln/detail/CVE-2023-1829), [CVE-2023-25652](https://nvd.nist.gov/vuln/detail/CVE-2023-25652), [CVE-2023-25815](https://nvd.nist.gov/vuln/detail/CVE-2023-25815), [CVE-2023-29007](https://nvd.nist.gov/vuln/detail/CVE-2023-29007).