vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Step 12: Replace each worker node to pick up configuration changes 



1. Get a list of nodes in your cluster.

ibmcloud oc worker ls -c CLUSTER
2. Run the ibmcloud oc worker replace to replace each worker node in your cluster.

ibmcloud oc worker replace -c CLUSTER --worker WORKER-NODE







 Step 13: Update the registries.conf file on each noderegistries.conf file on each node 

After replacing each worker node, start a debug pod on each node and update the registries.conf file.



1. List your worker nodes with oc get nodes.

oc get nodes
2. Start a debug pod on one of the nodes.

oc debug node/NODE-NAME
3. Allow host binaries.

chroot /host
4. Open the registries.conf file.

vi etc/containers/registries.conf
5. Append the following image mappings to the registries.conf file.

[registry]]
location = "registry.redhat.io/odf4"
insecure = false
blocked = false
mirror-by-digest-only = false
prefix = ""

[registry.mirror]]
location = "us.icr.io/NAMESPACE/odf4"
insecure = false

[registry]]
location = "registry.redhat.io/openshift4"
insecure = false
blocked = false
mirror-by-digest-only = false
prefix = ""

[registry.mirror]]
location = "us.icr.io/NAMESPACE/openshift4"
insecure = false

[registry]]
location = "registry.redhat.io/ocs4"
insecure = false
blocked = false
mirror-by-digest-only = false
prefix = ""

[registry.mirror]]
location = "us.icr.io/NAMESPACE/ocs4"
insecure = false
Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 1.17.11_1538

 Component Previous Current Description 

 HAProxy 1.8.25-384f42 1.8.26-561f1a See the [HAProxy change log](https://www.haproxy.org/download/1.8/src/CHANGELOG). 
 Ubuntu 18.04 packages 4.15.0-112-generic 4.15.0-117-generic Updated worker node image with kernel and package updates for [CVE-2020-14344](https://nvd.nist.gov/vuln/detail/CVE-2020-14344), [CVE-2020-14363](https://nvd.nist.gov/vuln/detail/CVE-2020-14363), and [CVE-2020-14386](https://nvd.nist.gov/vuln/detail/CVE-2020-14386). 
 Ubuntu 16.04 packages 4.4.0-187-generic 4.4.0-189-generic Updated worker node image with kernel and package updates for [CVE-2020-14344](https://nvd.nist.gov/vuln/detail/CVE-2020-14344) and [CVE-2020-14363](https://nvd.nist.gov/vuln/detail/CVE-2020-14363). 







 Change log for worker node fix pack 1.17.11_1538, released 31 August 2020 

The following table shows the changes that are in the worker node fix pack 1.17.11_1538. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 1.17.11_1537

 Component Previous Current Description
If you plan to deploy only static worker nodes in the LSF cluster, for example, when you use Spectrum Scale storage, the value for this parameter should be equal to worker_node_min_count. Enter a value in the range 1 - 500. No 10 
 worker_node_min_count The minimum number of worker nodes. This is the number of static worker nodes that will be provisioned at the time the cluster is created. If using NFS storage, enter a value in the range 0 - 500. If using Spectrum Scale storage, enter a value in the range 1 - 64. Note: Spectrum Scale requires a minimum of 3 compute nodes (combination of primary, secondary, management, and worker nodes) to establish a [quorum](https://www.ibm.com/docs/en/spectrum-scale/5.1.2?topic=failure-quorumnodequo) and maintain data consistency in the event of a node failure. Therefore, the minimum value of 1 might need to be larger if the value specified for management_node_count is less than 2. No 0 
 zone IBM Cloud zone name within the selected region where the Spectrum LSF cluster should be deployed. To get a full list of zones within a region, see [Get zones by using the CLI](https://cloud.ibm.com/docs/vpc?topic=vpc-creating-a-vpc-in-a-different-regionget-zones-using-the-cli). Yes None
Change log for worker node fix pack 1.17.17_1565, released 7 June 2021 

The following table shows the changes that are in the worker node fix pack 1.17.17_1565. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 1.17.17_1564

 Component Previous Current Description 

 HA proxy 26c5cc 700dc6 Updated the image for [CVE-2021-27219](https://nvd.nist.gov/vuln/detail/CVE-2021-27219). 
 TCP keepalive optimization for VPC N/A N/A Set the net.ipv4.tcp_keepalive_time setting to 180 seconds for compatibility with VPC gateways.
If the issue persists, see [Debugging your worker node](https://cloud.ibm.com/docs/containers?topic=containers-debug_worker_nodesworker-debug-steps) for additional steps.



1. [Update your cluster and worker nodes to the latest version](https://cloud.ibm.com/docs/containers?topic=containers-updateupdate).
2. [Update your command line tools](https://cloud.ibm.com/docs/containers?topic=containers-cli-update).
3. [Run tests in the Diagnostics and Debug Tool add-on](https://cloud.ibm.com/docs/containers?topic=containers-debug-tool).







 Debugging your worker node 



 Step 1: Get the worker node state 

If your cluster is in a Critical, Delete failed, or Warning state, or is stuck in the Pending state for a long time, review the state of your worker nodes.

ibmcloud ks worker ls --cluster <cluster_name_or_id>





 Step 2: Review the worker node state 

Review the State and Status field for every worker node in your CLI output.

For more information, see [Worker node states](https://cloud.ibm.com/docs/containers?topic=containers-worker-node-state-reference).





 Step 3: Get the details for each worker node 

Get the details for the worker node. If the details include an error message, review the list of [common error messages for worker nodes](https://cloud.ibm.com/docs/containers?topic=containers-common_worker_nodes_issues) to learn how to resolve the problem.

ibmcloud ks worker get --cluster <cluster_name_or_id> --worker <worker_node_id>





 Step 4: Review the infrastructure provider for the worker node 

Review the infrastructure environment to check for other reasons that might cause the worker node issues.



1. Check with your networking team to make sure that no recent maintenance, such as firewall or subnet updates, might impact the worker node connections.
List your workers using the ibmcloud ks worker ls[command](https://cloud.ibm.com/docs/containers?topic=containers-kubernetes-service-clics_workers).
3. Classic clusters: Ensure you have at least two worker nodes in the VLANs where your ALBs are deployed. See [Adding worker nodes and zones to clusters](https://cloud.ibm.com/docs/containers?topic=containers-add_workers).
4. VPC clusters: Ensure you have at least two worker nodes in the zones where your ALBs are deployed. See [Adding worker nodes and zones to clusters](https://cloud.ibm.com/docs/containers?topic=containers-add_workers).
5. Ensure that your workers are healthy. For more information, see [Worker node states](https://cloud.ibm.com/docs/containers?topic=containers-worker-node-state-reference).
6. Ensure your nodes are not tainted or cordoned. For more information, see [Taints and Tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) and [Safely Drain a Node](https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/).



4. If you notice pod restarts, follow the steps.



1. Get the logs for the failing pod.

kubectl logs --previous -n kube-system <POD NAME>
2. Review the logs and adjust the Ingress resource configurations or the Ingress ConfigMap in the kube-system namespace.
* Attach an additional disk to at least 3 worker nodes use for the KVDB, add a 3 GB block storage device to at least 3 worker nodes in your cluster and find the device path. To find the device path after attaching the disk, log in to your worker node with oc debug <node> and run lsblk to list the devices on that node.
* Select one of the block storage devices that you added earlier to use for the KVDB and find the device path. To find the device path after attaching the disk, log in to your worker node with oc debug <node> and run lsblk to list the devices on that node.



5. Continue with your Portworx setup by [Setting up a key-value store for Portworx metadata](https://cloud.ibm.com/docs/containers?topic=containers-storage_portworx_kv_store).
(https://cloud.ibm.com/docs/containers?topic=containers-access_cluster)
2. List your worker nodes.

oc get nodes
3. Describe your worker node and check for the operating system label that IBM applies, or the OS Image and Operating System fields in the System Info section.

oc describe node <node>

Example output

NAME: 10.xxx.xx.xxx
Roles: <none>
Labels: arch=amd64
...
ibm-cloud.kubernetes.io/os=UBUNTU_18_64
...
kubernetes.io/arch=amd64
kubernetes.io/hostname=10.189.33.198
kubernetes.io/os=linux
...
System Info:
OS Image: Ubuntu 18.04.5 LTS
Operating System: linux
Architecture: amd64
...









 Virtual machines 

With VMs, you get greater flexibility, quicker provisioning times, and more automatic scalability features than bare metal, at a more cost-effective price. You can use VMs for most general-purpose use cases such as testing and development environments, staging, and prod environments, microservices, and business apps. However, there is a tradeoff in performance. If you need high-performance computing for data- or RAM-intensive workloads, consider creating classic clusters with [bare metal](https://cloud.ibm.com/docs/openshift?topic=openshift-planning_worker_nodesbm) worker nodes.



 Planning considerations for VMs 



 Do I want to use shared or dedicated hardware? 

When you create a standard classic cluster, you must choose whether you want the underlying hardware to be shared by multiple IBM customers (multi tenancy) or to be dedicated to you only (single tenancy). VPC standard clusters can be provisioned on shared infrastructure (multi tenancy) only.
Change log for worker node fix pack 1.16.15_1552, released 23 November 2020 

The following table shows the changes that are in the worker node fix pack 1.16.15_1552. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 1.16.15_1551

 Component Previous Current Description 

 Ubuntu 18.04 packages 4.15.0-122-generic 4.15.0-123-generic Updated worker node image with kernel and package updates for [CVE-2020-25692](https://nvd.nist.gov/vuln/detail/CVE-2020-25692), [CVE-2020-25709](https://nvd.nist.gov/vuln/detail/CVE-2020-25709), [CVE-2020-25710](https://nvd.nist.gov/vuln/detail/CVE-2020-25710), [CVE-2020-28196](https://nvd.nist.gov/vuln/detail/CVE-2020-28196), and [CVE-2020-8694](https://nvd.nist.gov/vuln/detail/CVE-2020-8694).
: Optional: Specify the ID of the VPC in which to create the worker pool's worker nodes. The value must match the VPC ID that the cluster is in. To list the cluster's VPC ID, run ibmcloud ks cluster get -c <cluster_name_or_ID>. If this option is not provided, then the worker pool defaults to the VPC ID of existing worker pools in the cluster.

-l, --label KEY1=VALUE1
: Optional: Apply key-value labels to each worker node in the worker pool. To specify multiple labels, use multiple options, such as -l key1=value1 -l key2=value2.

--kms-account-id ID
: Optional: The ID of the account that contains the KMS instance you want to use for local disk or secret encryption.

--kms-instance KMS_INSTANCE_ID
: Optional: Include the ID of a key management service (KMS) instance to use to encrypt the local disk on the worker nodes in the default worker pool. To list available KMS instances, run ibmcloud ks kms instance ls. If you include this option, you must also include the --crk option. For more information including steps to create, see [Managing encryption for the worker nodes in your cluster](https://cloud.ibm.com/docs/containers?topic=containers-encryptionworker-encryption).

--crk ROOT_KEY
: Optional: Include the ID of the root key in the KMS instance to use to encrypt the local disk on the worker nodes in this worker pool. To list available root keys, run ibmcloud ks kms crk ls --instance-id. If you include this option, you must also include the --kms-instance option. Before you can use KMS encryption, you must create a KMS instance and set up the required service authorization in IAM. See [Managing encryption for the worker nodes in your cluster](https://cloud.ibm.com/docs/containers?topic=containers-encryptionworker-encryption).

-q
After you create a worker pool, you might notice that the worker node flavor has .encrypted in the name, such as b3c.4x16.encrypted.





 How do I manage my worker nodes? 

Worker nodes in classic clusters are provisioned into your IBM Cloud account. You can manage your worker nodes by using Red Hat OpenShift on IBM Cloud, but you can also use the [classic infrastructure dashboard](https://cloud.ibm.com/classic/) in the IBM Cloud console to work with your worker node directly.

Unlike classic clusters, the worker nodes of your VPC cluster are not listed in the [VPC infrastructure dashboard](https://cloud.ibm.com/vpc/overview). Instead, you manage your worker nodes with Red Hat OpenShift on IBM Cloud only. However, your worker nodes might be connected to other VPC infrastructure resources, such as VPC subnets or VPC Block Storage. These resources are in the VPC infrastructure dashboard and can be managed separately from there.





 What limitations do I need to be aware of? 

Kubernetes limits the maximum number of worker nodes that you can have in a cluster. Review [worker node and pod quotas](https://kubernetes.io/docs/setup/best-practices/cluster-large/) for more information.

[Reserved capacity and reserved instances](https://cloud.ibm.com/docs/virtual-servers?topic=virtual-servers-provisioning-reserved-capacity-and-instances) are not supported.

Red Hat OpenShift on IBM Cloud also sets compute resource reserves that limit available compute resources on each worker node. For more information, see [worker node resource reserves](https://cloud.ibm.com/docs/openshift?topic=openshift-planning_worker_nodesresource_limit_node).

Want to be sure that you always have enough worker nodes to cover your workload? Try out [the cluster autoscaler](https://cloud.ibm.com/docs/openshift?topic=openshift-cluster-scaling-classic-vpc).





 Why do my worker nodes have the master role?
chmod 400 /.ssh/id_rsa_worker_private
4. Use the private key to SSH into the worker node that you found in step 2.

ssh -i /.ssh/id_rsa_worker_private root@<WORKER_PRIVATE_IP>







 Public network classic clusters that are connected to a public VLAN only 

Debug classic clusters that are connected to a public VLAN by logging in to your worker nodes.



1. [Install and configure the Calico CLI, and set the context for your cluster to run Calico commands](https://cloud.ibm.com/docs/openshift?topic=openshift-network_policiescli_install).
2. Create a Calico global network policy that is named ssh-open to allow inbound SSH traffic on port 22.

calicoctl apply [-c <path_to_calicoctl_cfg>/calicoctl.cfg] -f - <<EOF
apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
name: ssh-open
spec:
selector: ibm.role == 'worker_public'
ingress:
- action: Allow
protocol: TCP
destination:
ports:
- 22
- action: Allow
protocol: UDP
destination:
ports:
- 22
order: 1500
EOF
Show more
3. Get the public IP address of your worker node.

oc get nodes -o wide
4. SSH into the worker node via its public IP address.

ssh root@<WORKER_PUBLIC_IP>
5. Run debug commands to help you gather information and troubleshoot issues, such as ip, ifconfig, nc, tcpdump, ping, ps, and curl. You can also install other tools, such as mtr, by running yum install <tool>.







 Cleaning up after debugging 

After you finish debugging, clean up resources to disable SSH access.
* [Apply version updates to worker node hosts one at a time](https://cloud.ibm.com/docs/satellite?topic=satellite-host-update-workershost-update-workers-individually)
* [Apply version updates to your worker node hosts with a ConfigMap](https://cloud.ibm.com/docs/satellite?topic=satellite-host-update-workershost-update-workers-rolling)



* [Applying version updates to worker nodes by replacing hosts](https://cloud.ibm.com/docs/satellite?topic=satellite-host-update-workershost-update-workers-minor)
* [Updating worker node hosts in the Red Hat OpenShift on IBM Cloud console](https://cloud.ibm.com/docs/satellite?topic=satellite-host-update-workershost-update-roks-console)
* [Determining if the worker node version update is a major, minor, or patch update](https://cloud.ibm.com/docs/satellite?topic=satellite-host-update-workershost-update-workers-type)



[Copying a location](https://cloud.ibm.com/docs/satellite?topic=satellite-location-copylocation-copy)

[Configuring an HTTP proxy for your Satellite hosts](https://cloud.ibm.com/docs/satellite?topic=satellite-config-http-proxyconfig-http-proxy)



* [What type of location do I need to use HTTP proxy?](https://cloud.ibm.com/docs/satellite?topic=satellite-config-http-proxyconsider-http-proxy)
* [What type of hosts can I use?](https://cloud.ibm.com/docs/satellite?topic=satellite-config-http-proxyconsider-http-proxy-host)
* [What else do I need to know about HTTP proxy?](https://cloud.ibm.com/docs/satellite?topic=satellite-config-http-proxyadditional-http-proxy)
1 root root 10 Feb 9 04:15 scsi-3600605b00d87b43027b3bbb603150cc6-part2 -> ../../sda2
lrwxrwxrwx. 1 root root 10 Feb 9 04:15 scsi-3600605b00d87b43027b3bbb603150cc6-part3 -> ../../sda3
lrwxrwxrwx. 1 root root 9 Feb 9 04:15 scsi-3600605b00d87b43027b3bbf306bc28a7 -> ../../sdb
lrwxrwxrwx. 1 root root 10 Feb 9 04:15 scsi-3600605b00d87b43027b3bbf306bc28a7-part1 -> ../../sdb1
lrwxrwxrwx. 1 root root 9 Feb 9 04:17 scsi-3600605b00d87b43027b3bc310a64c6c9 -> ../../sdc
lrwxrwxrwx. 1 root root 10 Feb 11 03:14 scsi-3600605b00d87b43027b3bc310a64c6c9-part1 -> ../../sdc1
lrwxrwxrwx. 1 root root 10 Feb 11 03:15 scsi-3600605b00d87b43027b3bc310a64c6c9-part2 -> ../../sdc2
7. Repeat the previous steps for each worker node that you want to use for your ODF configuration.
Change log for worker node fix pack 1.14.10_1554, released 11 May 2020 

The following table shows the changes that are in the worker node fix pack 1.14.10_1554. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 1.14.10_1553

 Component Previous Current Description 

 Ubuntu 18.04 packages 4.15.0-96-generic 4.15.0-99-generic Updated worker node images with kernel and package updates for [CVE-2019-19768](https://nvd.nist.gov/vuln/detail/CVE-2019-19768), [CVE-2020-11884](https://nvd.nist.gov/vuln/detail/CVE-2020-11884), and [CVE-2020-12243](https://nvd.nist.gov/vuln/detail/CVE-2020-12243). 
 Ubuntu 16.04 packages 4.4.0-177-generic 4.4.0-178-generic Updated worker node images with package and kernel updates for [CVE-2019-19768](https://nvd.nist.gov/vuln/detail/CVE-2019-19768) and [CVE-2020-12243](https://nvd.nist.gov/vuln/detail/CVE-2020-12243). 







 Change log for worker node fix pack 1.14.10_1553, released 27 April 2020 

The following table shows the changes that are in the worker node fix pack 1.14.10_1553. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 1.14.10_1551

 Component Previous Current Description
* If the cluster is single zone and exists in dal10, then the worker pool has three worker nodes in dal10. The cluster has a total of three worker nodes.
* If the cluster is multizone and exists in dal10 and dal12, then the worker pool has three worker nodes in dal10 and three worker nodes in dal12. The cluster has a total of six worker nodes.



For bare metal worker pools, keep in mind that billing is monthly. If you resize up or down, it impacts your costs for the month. When you add worker nodes by resizing a worker pool, the new worker nodes run the same major.minor version as the cluster master, but the latest worker node patch of that major.minor version.

Before you begin, make sure that you have the [Operator or Administrator IBM Cloud IAM platform access role](https://cloud.ibm.com/docs/openshift?topic=openshift-users).

To resize the worker pool, change the number of worker nodes that the worker pool deploys in each zone:



1. Get the name of the worker pool that you want to resize.

ibmcloud oc worker-pool ls --cluster <cluster_name_or_ID>
2. Resize the worker pool by designating the number of worker nodes that you want to deploy in each zone. The minimum value is 2. For more information, see [What is the smallest size cluster that I can make?](https://cloud.ibm.com/docs/openshift?topic=openshift-faqssmallest_cluster).

ibmcloud oc worker-pool resize --cluster <cluster_name_or_ID> --worker-pool <pool_name> --size-per-zone <number_of_workers_per_zone>
3. Verify that the worker pool is resized.

ibmcloud oc worker ls --cluster <cluster_name_or_ID> --worker-pool <pool_name>

Example output for a worker pool that is in two zones, dal10 and dal12, and is resized to two worker nodes per zone:
Change log for worker node fix pack 1.15.12_1549, released 3 August 2020 

The following table shows the changes that are in the worker node fix pack 1.15.12_1549. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 1.15.12_1546

 Component Previous Current Description 

 Ubuntu 18.04 packages 4.15.0-111-generic 4.15.0-112-generic Updated worker node images with kernel and package updates for [CVE-2019-17514](https://nvd.nist.gov/vuln/detail/CVE-2019-17514), [CVE-2019-20907](https://nvd.nist.gov/vuln/detail/CVE-2019-20907), [CVE-2019-9674](https://nvd.nist.gov/vuln/detail/CVE-2019-9674), [CVE-2020-10713](https://nvd.nist.gov/vuln/detail/CVE-2020-10713), [CVE-2020-10757](https://nvd.nist.gov/vuln/detail/CVE-2020-10757), [CVE-2020-11935
If you want to attach raw, unformatted block storage to a classic worker node, you must install the [IBM Cloud Block Storage attacher plug-in](https://cloud.ibm.com/docs/containers?topic=containers-utilitiesblock_storage_attacher).

Before you begin:

[Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.](https://cloud.ibm.com/docs/containers?topic=containers-access_cluster)



1. Check which region and zone your VPC worker node is in.

ibmcloud ks worker ls -c <cluster_name>
2. Decide on the [Block Storage for Classic profile](https://cloud.ibm.com/docs/vpc?topic=vpc-block-storage-profiles) that best meets the capacity and performance requirements that you have.
3. [Provision a Block Storage for Classic volume](https://cloud.ibm.com/docs/vpc?topic=vpc-creating-block-storage). The volume that you provision must be in the same resource group, region, and zone as the worker node.
4. Retrieve your IAM token.

ibmcloud iam oauth-tokens
5. Retrieve the ID of the worker node that you want to attach to the Block Storage for Classic instance. Make sure to select a worker node that is located in the same zone as your Block Storage for Classic volume.

ibmcloud ks worker ls --cluster <cluster_name_or_ID>
6. Use a POST request to attach your Block Storage for Classic volume to the worker node.

Example request