vania-janet/multiturn-rag-retrieval-data huggingface dataset:
evicting pod "router-default-6fc798b98b-9m6kh"
evicting pod "prometheus-adapter-5b77ffdd5f-hzqrp"
evicting pod "alertmanager-main-1"
evicting pod "prometheus-k8s-0"
evicting pod "network-check-source-66c7fbb86-2r78z"
3. Wait until draining finishes, then complete the following steps to replace the worker node.







 Step 5: Update the worker node 

Major

Minor



1. List your worker nodes by using ibmcloud oc worker ls and find the worker node that you cordoned and drained in the previous step.

ibmcloud oc worker ls -c CLUSTER

Example output

ID Primary IP Flavor State Status Zone Version
kube-c85ra07w091uv4nid9ug-vpcoc-default-000001c1 10.241.128.4 bx2.4x16 normal Ready us-east-3 4.8.29_1544_openshift
kube-c85ra07w091uv4nid9ug-vpcoc-default-00000288 10.241.0.4 bx2.4x16 normal Ready us-east-1 4.8.29_1544_openshift
kube-c85ra07w091uv4nid9ug-vpcoc-default-00000352 10.241.64.4 bx2.4x16 normal Ready us-east-2 4.8.29_1544_openshift
2. Update the worker node.

ibmcloud oc worker update -c CLUSTER --worker kube-

Example output

The replacement worker node is created in the same zone with the same flavor, but gets new public or private IP addresses. During the replacement, all pods might be rescheduled onto other worker nodes and data is deleted if not stored outside the pod.
ibmcloud ks cluster config --cluster <cluster_name_or_ID> --admin --network
2. For the calico-node pods that are stuck in the CrashLoopBackOff status, note the NODE IP addresses.

kubectl -n kube-system get pods -o wide

In this example output, the calico-node pod can't start on worker node 10.176.48.106.

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
...
calico-kube-controllers-656c5785dd-kc9x2 1/1 Running 0 25h 10.176.48.107 10.176.48.107 <none> <none>
calico-node-mkqbx 0/1 CrashLoopBackOff 1851 25h 10.176.48.106 10.176.48.106 <none> <none>
coredns-7b56dd58f7-7gtzr 0/1 ContainerCreating 0 25h 172.30.99.82 10.176.48.106 <none> <none>
3. Get the IDs of the calico-node worker node entries. Copy the IDs for only the worker node IP addresses that you retrieved in the previous step.

calicoctl get nodes -o wide
4. Use the IDs to delete the worker node entries. After you delete the worker node entries, the Calico controller reschedules the calico-node pods on the new worker nodes.

calicoctl delete node <node_ID>
5. Verify that the Kubernetes component pods, including the calico-node pods, are now running. It might take a few minutes for the calico-node pods to be scheduled and for new component pods to be created.

kubectl -n kube-system get pods



To prevent this error in the future, never delete all worker nodes in your cluster. Always run at least one worker node in your cluster, and if you use Ingress to expose apps, run at least two worker nodes per zone.
ibmcloud oc cluster config --cluster <cluster_name_or_ID> --admin --network
2. For the calico-node pods that are stuck in the CrashLoopBackOff status, note the NODE IP addresses.

oc -n kube-system get pods -o wide

In this example output, the calico-node pod can't start on worker node 10.176.48.106.

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
...
calico-kube-controllers-656c5785dd-kc9x2 1/1 Running 0 25h 10.176.48.107 10.176.48.107 <none> <none>
calico-node-mkqbx 0/1 CrashLoopBackOff 1851 25h 10.176.48.106 10.176.48.106 <none> <none>
coredns-7b56dd58f7-7gtzr 0/1 ContainerCreating 0 25h 172.30.99.82 10.176.48.106 <none> <none>
3. Get the IDs of the calico-node worker node entries. Copy the IDs for only the worker node IP addresses that you retrieved in the previous step.

calicoctl get nodes -o wide
4. Use the IDs to delete the worker node entries. After you delete the worker node entries, the Calico controller reschedules the calico-node pods on the new worker nodes.

calicoctl delete node <node_ID>
5. Verify that the Kubernetes component pods, including the calico-node pods, are now running. It might take a few minutes for the calico-node pods to be scheduled and for new component pods to be created.

oc -n kube-system get pods



To prevent this error in the future, never delete all worker nodes in your cluster. Always run at least one worker node in your cluster, and if you use Ingress or routes to expose apps, run at least two worker nodes per zone.
If your worker node comes with compute resources in multiple tiers, a percentage of your PIDs, CPU and memory resources is reserved for each tier.

Red Hat OpenShift version 4 : Clusters also have process ID (PID) reservations and limits, to prevent a pod from using too many PIDs or ensure that enough PIDs exist for the kubelet and other Red Hat OpenShift on IBM Cloud system components. If the PID reservations or limits are reached, Kubernetes does not create or assign new PIDs until enough processes are removed to free up existing PIDs. The total amount of PIDs on a worker node approximately corresponds to 8,000 PIDs per GB of memory on the worker node. For example, a worker node with 16 GB of memory has approximately 128,000 PIDs (16 × 8,000 = 128,000).

To review how much compute resources are currently used on your worker node, run [oc top node](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commandstop).

Worker node memory reserves by tier

Worker node CPU reserves by tier

Worker node PID reserves by tier



Worker node memory reserves by tier

 Memory tier % or amount reserved b3c.4x16 worker node (16 GB) example mg1c.28x256 worker node (256 GB) example 

 First 4 GB (0 - 4 GB) 25% of memory 1 GB 1 GB 
 Next 4 GB (5 - 8 GB) 20% of memory 0.8 GB 0.8 GB 
 Next 8 GB (9 - 16 GB) 10% of memory 0.8 GB 0.8 GB 
 Next 112 GB (17 - 128 GB) 6% of memory N/A 6.72 GB 
 Remaining GB (129 GB+) 2% of memory N/A 2.54 GB 
 Additional reserve for [kubelet eviction](https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/) 100 MB 100 MB (flat amount) 100 MB (flat amount) 
 Total reserved (varies) 2.7 GB of 16 GB total 11.96 GB of 256 GB total
Red Hat OpenShift 4.5.19 4.5.21 See the [Red Hat OpenShift release notes](https://docs.openshift.com/container-platform/4.4/release_notes/ocp-4-4-release-notes.htmlocp-4-5-21). 
 RHEL 7 Packages N/A N/A Updated worker node image with package updates. 







 Change log for worker node fix pack 4.5.19_1521_openshift, released 23 November 2020 

The following table shows the changes that are in the worker node fix pack update 4.5.19_1521_openshift. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 4.5.17_1519_openshift

 Component Previous Current Description 

 Ephemeral storage reservations N/A N/A Local ephemeral storage is reserved on the Kubernetes data disk for system components. For more information, see [Worker node resource reserves](https://cloud.ibm.com/docs/openshift?topic=openshift-planning_worker_nodesresource_limit_node). 
 Red Hat OpenShift node 4.5.17 4.5.19 See the [Red Hat OpenShift release notes](https://docs.openshift.com/container-platform/4.5/release_notes/ocp-4-5-release-notes.htmlocp-4-5-19).
containers-kubernetes.cluster.update containers-kubernetes.containers-kubernetes.storage.attachment.delete 







 Worker nodes and worker pools 

Review the following worker node and worker pool API methods, their corresponding actions in IBM Cloud IAM, and the events that are sent to IBM Cloud Activity Tracker for Red Hat OpenShift on IBM Cloud.



Worker node and worker pool API methods, IAM actions, and Activity Tracker events.

 API Method Description IAM action for the API Activity Tracker event 

 DELETE/v1/clusters/{idOrName}/workerpools/{poolidOrName} Remove a worker pool from a cluster. containers-kubernetes.cluster.operate containers-kubernetes.workerpool.delete 
 DELETE/v1/clusters/{idOrName}/workerpools/{poolidOrName}/zones/{zoneid} Remove a zone from a worker pool. containers-kubernetes.cluster.operate containers-kubernetes.zone.delete 
 DELETE/v1/clusters/{idOrName}/workers/{workerId} Delete a worker node from a cluster. containers-kubernetes.cluster.operate containers-kubernetes.worker.delete 
 GET/v1/clusters/{idOrName}/workerpools/{poolidOrName} View details for a worker pool. containers-kubernetes.cluster.read N/A 
 GET/v1/clusters/{idOrName}/workers List all worker nodes in a cluster. containers-kubernetes.cluster.read N/A 
 GET/v1/clusters/{idOrName}/workers/{workerId} View details of a worker node. containers-kubernetes.cluster.read N/A 
 GET/v2/classic/getWorker View details of a worker node for classic cluster. containers-kubernetes.cluster.read N/A 
 GET/v2/classic/getWorkerPool View details of a worker pool for a classic cluster. containers-kubernetes.cluster.read N/A
Red Hat OpenShift on IBM Cloud. 4.11.20 4.11.22 For more information, see the [change log](https://docs.openshift.com/container-platform/4.11/release_notes/ocp-4-11-release-notes.htmlocp-4-11-22). 







 Change log for worker node fix pack 4.11.20_1537_openshift, released 02 January 2023 

The following table shows the changes that are in the worker node fix pack 4.11.20_1537_openshift. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 4.11.18_1536_openshift

 Component Previous Current Description 

 RHEL 8 Packages N/A N/A N/A 
 Red Hat OpenShift on IBM Cloud. 4.11.18 4.11.20 For more information, see the [change log](https://docs.openshift.com/container-platform/4.11/release_notes/ocp-4-11-release-notes.htmlocp-4-11-20). 







 Change log for worker node fix pack 4.11.18_1536_openshift, released 19 December 2022 

The following table shows the changes that are in the worker node fix pack 4.11.18_1536_openshift. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 4.11.17_1534_openshift

 Component Previous Current Description 

 RHEL 8 Packages N/A N/A N/A 
 Red Hat OpenShift on IBM Cloud. 4.11.17 4.11.18 For more information, see the [change log](https://docs.openshift.com/container-platform/4.11/release_notes/ocp-4-11-release-notes.htmlocp-4-11-18).
Component Previous Current Description 

 VPC Gen 2 RHEL 8 worker image disk size N/A N/A Fixed regression on previous version where root filesystem partition was 25 GB instead of 100GB for VPC Gen 2 







 Change log for worker node fix pack 4.9.50_1559_openshift, released 25 October 2022 

The following table shows the changes that are in the worker node fix pack 4.9.50_1559_openshift. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 4.9.49_1558_openshift

 Component Previous Current Description 

 RHEL 7 Packages N/A N/A N/A 
 Red Hat OpenShift on IBM Cloud 4.9.49 4.9.50 See the [change logs](https://docs.openshift.com/container-platform/4.9/release_notes/ocp-4-9-release-notes.htmlocp-4-9-50). 







 Change log for worker node fix pack 4.9.49_1558_openshift, released 10 October 2022 

The following table shows the changes that are in the worker node fix pack 4.9.49_1558_openshift. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 4.9.48_1557_openshift

 Component Previous Current Description 

 RHEL 7 Packages N/A N/A N/A 
 RHEL 8 Packages N/A N/A N/A 
 Red Hat OpenShift on IBM Cloud. 4.9.48 4.9.49 For more information, see the [change log](https://docs.openshift.com/container-platform/4.9/release_notes/ocp-4-9-release-notes.htmlocp-4-9-49). 







 Change log for master fix pack 4.9.48_1556_openshift, released 26 September 2022
Automatic storage volume discovery auto-discover-devices Config Set to true if you want to automatically discover and use the storage volumes on your worker nodes. true false 
 Node Label Key label-key Config The key of the worker node key=value label. true N/A 
 Node Label Key Value label-value Config The value of the worker node key=value label. true N/A 
 Device Path devicepath Config The local storage device path. Example: /dev/sdc. Required when auto-discover-devices is set to false. false N/A 







 4.11 parameter reference 



Table 4. 4.11 parameter reference

 Display name CLI option Type Description Required? Default value 

 Automatic storage volume discovery auto-discover-devices Config Set to true if you want to automatically discover and use the storage volumes on your worker nodes. true false 
 Node Label Key label-key Config The key of the worker node key=value label. true N/A 
 Node Label Key Value label-value Config The value of the worker node key=value label. true N/A 
 Device Path devicepath Config The local storage device path. Example: /dev/sdc. Required when auto-discover-devices is set to false. false N/A 







 4.12 parameter reference 



Table 5. 4.12 parameter reference

 Display name CLI option Type Description Required? Default value 

 Automatic storage volume discovery auto-discover-devices Config Set to true if you want to automatically discover and use the storage volumes on your worker nodes. true false 
 Node Label Key label-key Config The key of the worker node key=value label. true N/A 
 Node Label Key Value label-value Config The value of the worker node key=value label. true N/A 
 Device Path devicepath Config The local storage device path. Example: /dev/sdc. Required when auto-discover-devices is set to false. false N/A 









 Storage class reference for local block storage 

Review the Satellite storage classes for local block storage.
Change log for worker node fix pack 4.9.42_1547_openshift, released 18 July 2022 

The following table shows the changes that are in the worker node fix pack 4.9.42_1547_openshift. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 4.9.40_1546_openshift

 Component Previous Current Description 

 RHEL 7 Packages 3.10.0-1160.66.1 3.10.0-1160.71.1 Worker node package updates for [CVE-2020-26116](https://nvd.nist.gov/vuln/detail/CVE-2020-26116), [CVE-2020-26137](https://nvd.nist.gov/vuln/detail/CVE-2020-26137), [CVE-2021-3177](https://nvd.nist.gov/vuln/detail/CVE-2021-3177), [CVE-2022-0391](https://nvd.nist.gov/vuln/detail/CVE-2022-0391), [CVE-2022-1729](https://nvd.nist.gov/vuln/detail/CVE-2022-1729), [CVE-2022-1966](https://nvd.nist.gov/vuln/detail/CVE-2022-1966). 
 Red Hat OpenShift on IBM Cloud. 4.9.40 4.9.42 For more information, see the [change log](https://docs.openshift.com/container-platform/4.9/release_notes/ocp-4-9-release-notes.htmlocp-4-9-42). 







 Change log for worker node fix pack 4.9.40_1546_openshift, released 11 July 2022 

The following table shows the changes that are in the worker node fix pack 4.9.40_1546_openshift.
Change log for worker node fix pack 4.10.21_1527_openshift, released 18 July 2022 

The following table shows the changes that are in the worker node fix pack 4.10.21_1527_openshift. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 4.10.20_1526_openshift

 Component Previous Current Description 

 RHEL 7 Packages 3.10.0-1160.66.1 3.10.0-1160.71.1 Worker node package updates for [CVE-2020-26116](https://nvd.nist.gov/vuln/detail/CVE-2020-26116), [CVE-2020-26137](https://nvd.nist.gov/vuln/detail/CVE-2020-26137), [CVE-2021-3177](https://nvd.nist.gov/vuln/detail/CVE-2021-3177), [CVE-2022-0391](https://nvd.nist.gov/vuln/detail/CVE-2022-0391), [CVE-2022-1729](https://nvd.nist.gov/vuln/detail/CVE-2022-1729), [CVE-2022-1966](https://nvd.nist.gov/vuln/detail/CVE-2022-1966). 
 Red Hat OpenShift on IBM Cloud. 4.10.20 4.10.21 For more information, see the [change log](https://docs.openshift.com/container-platform/4.10/release_notes/ocp-4-10-release-notes.htmlocp-4-10-21). 







 Change log for worker node fix pack 4.10.20_1526_openshift, released 11 July 2022 

The following table shows the changes that are in the worker node fix pack 4.10.20_1526_openshift.
Change log for worker node fix pack 4.7.54_1572_openshift, released 18 July 2022 

The following table shows the changes that are in the worker node fix pack 4.7.54_1572_openshift. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 4.7.53_1571_openshift

 Component Previous Current Description 

 RHEL 7 Packages 3.10.0-1160.66.1 3.10.0-1160.71.1 Worker node package updates for [CVE-2020-26116](https://nvd.nist.gov/vuln/detail/CVE-2020-26116), [CVE-2020-26137](https://nvd.nist.gov/vuln/detail/CVE-2020-26137), [CVE-2021-3177](https://nvd.nist.gov/vuln/detail/CVE-2021-3177), [CVE-2022-0391](https://nvd.nist.gov/vuln/detail/CVE-2022-0391), [CVE-2022-1729](https://nvd.nist.gov/vuln/detail/CVE-2022-1729), [CVE-2022-1966](https://nvd.nist.gov/vuln/detail/CVE-2022-1966). 
 Red Hat OpenShift on IBM Cloud. 4.7.53 4.7.54 For more information, see the [change log](https://docs.openshift.com/container-platform/4.7/release_notes/ocp-4-7-release-notes.htmlocp-4-7-54). 







 Change log for worker node fix pack 4.7.53_1571_openshift, released 11 July 2022 

The following table shows the changes that are in the worker node fix pack 4.7.53_1571_openshift.
Change log for worker node fix pack 4.8.46_1565_openshift, released 18 July 2022 

The following table shows the changes that are in the worker node fix pack 4.8.46_1565_openshift. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 4.8.44_1564_openshift

 Component Previous Current Description 

 RHEL 7 Packages 3.10.0-1160.66.1 3.10.0-1160.71.1 Worker node package updates for [CVE-2020-26116](https://nvd.nist.gov/vuln/detail/CVE-2020-26116), [CVE-2020-26137](https://nvd.nist.gov/vuln/detail/CVE-2020-26137), [CVE-2021-3177](https://nvd.nist.gov/vuln/detail/CVE-2021-3177), [CVE-2022-0391](https://nvd.nist.gov/vuln/detail/CVE-2022-0391), [CVE-2022-1729](https://nvd.nist.gov/vuln/detail/CVE-2022-1729), [CVE-2022-1966](https://nvd.nist.gov/vuln/detail/CVE-2022-1966). 
 Red Hat OpenShift on IBM Cloud. 4.8.44 4.8.46 For more information, see the [change log](https://docs.openshift.com/container-platform/4.8/release_notes/ocp-4-8-release-notes.htmlocp-4-8-46). 







 Change log for worker node fix pack 4.8.44_1564_openshift, released 11 July 2022 

The following table shows the changes that are in the worker node fix pack 4.8.44_1564_openshift.
Red Hat OpenShift on IBM Cloud sets compute resource reserves that limit available compute resources on each worker node. Reserved memory, CPU resources, and process IDs (PIDs) can't be used by pods on the worker node, and reduces the allocatable resources on each worker node. When you initially deploy pods, if the worker node does not have enough allocatable resources, the deployment fails. Further, if pods exceed the worker node resource limit for memory and CPU, the pods are evicted. In Kubernetes, this limit is called a [hard eviction threshold](https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/hard-eviction-thresholds). For pods that exceed the PID limit, the pods receive as many PIDs as allocatable, but are not evicted based on PIDs.

If less PIDs, CPU or memory is available than the worker node reserves, Kubernetes starts to evict pods to restore sufficient compute resources and PIDs. The pods reschedule onto another worker node if a worker node is available. If your pods are evicted frequently, add more worker nodes to your cluster or set [resource limits](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/resource-requests-and-limits-of-pod-and-container) on your pods.

The resources that are reserved on your worker node depend on the amount of PIDs, CPU and memory that your worker node comes with. Red Hat OpenShift on IBM Cloud defines PIDs, CPU and memory tiers as shown in the following tables. If your worker node comes with compute resources in multiple tiers, a percentage of your PIDs, CPU and memory resources is reserved for each tier.

Red Hat OpenShift version 4 : Clusters also have process ID (PID) reservations and limits, to prevent a pod from using too many PIDs or ensure that enough PIDs exist for the kubelet and other Red Hat OpenShift on IBM Cloud system components. If the PID reservations or limits are reached, Kubernetes does not create or assign new PIDs until enough processes are removed to free up existing PIDs.
An NLB service makes your app available over both the NLB service IP address and the worker nodes' node ports.



1. Get the node port that the NLB assigned to the worker nodes. The node port is in the 30000 - 32767 range.

kubectl get svc -o wide

In the following example output, the node port is 31024:

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
webserver-lb 172.21.xxx.xxx 169.xx.xxx.xxx 80:31024/TCP 2m run=webserver
2. For classic clusters, get the Public IP address of a worker node. For VPC clusters, get the Private IP address instead.

ibmcloud oc worker ls --cluster <cluster_name>

Example output

ID Public IP Private IP Machine Type State Status Zone Version
kube-dal10-cr18e61e63c6e94b658596ca93d087eed9-w1 169.xx.xxx.xxx 10.176.48.67 u3c.2x4.encrypted normal Ready dal10 1.26_1513
kube-dal10-cr18e61e63c6e94b658596ca93d087eed9-w2 169.xx.xxx.xxx 10.176.48.79 u3c.2x4.encrypted normal Ready dal10 1.26_1513
kube-dal10-cr18e61e63c6e94b658596ca93d087eed9-w3 169.xx.xxx.xxx 10.176.48.78 u3c.2x4.encrypted normal Ready dal10 1.26_1513
3. Copy the public IP of the worker node and the node port into your text cheat sheet to use in later lessons.
4. Verify that you can access the public IP address the worker node through the node port.
Change log for worker node fix pack 4.7.59_1583_openshift, released 26 September 2022 

The following table shows the changes that are in the worker node fix pack 4.7.59_1583_openshift. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 4.7.56_1581_openshift

 Component Previous Current Description 

 RHEL 7 Packages N/A N/A N/A 
 RHEL 8 Packages 4.18.0-372.19.1 4.18.0-372.26.1 N/A 
 Red Hat OpenShift on IBM Cloud. 4.7.56 4.7.59 For more information, see the [change log](https://docs.openshift.com/container-platform/4.7/release_notes/ocp-4-7-release-notes.htmlocp-4-7-59). 







 Change log for worker node fix pack 4.7.56_1581_openshift, released 12 September 2022 

The following table shows the changes that are in the worker node fix pack 4.7.56_1581_openshift. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 4.7.56_1578_openshift

 Component Previous Current Description 

 RHEL 7 Packages N/A N/A N/A 
 Red Hat OpenShift on IBM Cloud N/A N/A N/A 







 Change log for master fix pack 4.7.56_1580_openshift, released 1 September 2022 

The following table shows the changes that are in the master fix pack 4.7.56_1580_openshift. Master patch updates are applied automatically.



Changes since version 4.7.53_1574_openshift

 Component Previous Current Description