vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Unable to create or delete worker nodes due to endpoints error 

 What’s happening 

You can't manage worker nodes for your cluster, and you receive an error message similar to one of the following.

Worker deploy failed due to network communications failing to master or registry endpoints. Please verify your network setup is allowing traffic from this subnet then attempt a worker replace on this worker

Pending endpoint gateway creation

 Why it’s happening 

Worker nodes can communicate with the Kubernetes master through the cluster's virtual private endpoint (VPE).

One VPE gateway resource is created per cluster in your VPC. If the VPE gateway for your cluster is not correctly created in your VPC, the VPE gateway is deleted from your VPC, or the IP address that is reserved for the VPE is deleted from your VPC subnet, worker nodes lose connectivity with the Kubernetes master.

 How to fix it 

Re-establish the VPE connection between your worker nodes and Kubernetes master.



1. To check the VPE gateway for your cluster in the VPC infrastructure console, open the [Virtual private endpoint gateways for VPC dashboard](https://cloud.ibm.com/vpc-ext/network/endpointGateways) and look for the VPE gateway in the format iks-<cluster_ID>.



* If the gateway for your cluster is not listed, continue to the next step.
* If the gateway for your cluster is listed but its status is not Stable, [open a support case](https://cloud.ibm.com/docs/containers?topic=containers-get-helphelp-support). In the case details, include the cluster ID.
* If the gateway for your cluster is listed and its status is Stable, you might have firewall or security group rules that are blocking worker node communication to the cluster master. [Configure your security group rules to allow outgoing traffic to the appropriate ports and IP addresses](https://cloud.ibm.com/docs/containers?topic=containers-vpc-security-group).



2. Refresh the cluster master.
: Optional: The public VLAN that was specified when the cluster was created. If you want your worker nodes to exist on a private VLAN only, don't provide a public VLAN ID. Private VLAN routers always begin with bcr (back-end router) and public VLAN routers always begin with fcr (front-end router). When you create a cluster and specify the public and private VLANs, the number and letter combination after those prefixes must match. If worker nodes are set up with a private VLAN only, you must allow worker nodes and the cluster master to communicate by [enabling the private cloud service endpoint](https://cloud.ibm.com/docs/containers?topic=containers-cs_network_clusterset-up-private-se) or [configuring a gateway appliance](https://cloud.ibm.com/docs/containers?topic=containers-plan_basicsworkeruser-master).

--disable-disk-encrypt
: Worker nodes feature AES 256-bit disk encryption by default; [learn more](https://cloud.ibm.com/docs/openshift?topic=openshift-securityencrypted_disk). To disable encryption, include this option.

--operating-system SYSTEM
: Optional. The operating system of the worker nodes you want to provision in your cluster. For a list of available operating systems by cluster version, see [Red Hat OpenShift on IBM Cloud version information](https://cloud.ibm.com/docs/openshift?topic=openshift-openshift_versions).

-q
: Optional: Do not show the message of the day or update reminders.



 Example worker add command 

ibmcloud oc worker add --cluster my_cluster --workers 3 --public-vlan my_public_VLAN_ID --private-vlan my_private_VLAN_ID --flavor b3c.4x16 --hardware shared







 ibmcloud oc worker get 

Virtual Private Cloud

Classic infrastructure

View the details of a worker node.
Ingress Subdomain: mycluster-<hash>-0000.us-south.containers.appdomain.cloud
Ingress Secret: mycluster-<hash>-0000
Workers: 6
Worker Zones: dal10, dal12
Version: 1.26_1524
Owner: owner@email.com
Resource Group ID: a8a12accd63b437bbd6d58fb6a462ca7
Resource Group Name: Default









 Deprecated: Adding stand-alone worker nodes 

If you have a cluster that was created before worker pools were introduced, you can use the deprecated commands to add stand-alone worker nodes.

If you have a cluster that was created after worker pools were introduced, you can't add stand-alone worker nodes. Instead, you can [create a worker pool](https://cloud.ibm.com/docs/containers?topic=containers-add_workersadd_pool), [resize an existing worker pool](https://cloud.ibm.com/docs/containers?topic=containers-add_workersresize_pool), or [add a zone to a worker pool](https://cloud.ibm.com/docs/containers?topic=containers-add_workersadd_zone) to add worker nodes to your cluster.



1. List available zones and pick the zone where you want to add worker nodes.

ibmcloud ks zone ls
2. List available VLANs in that zone and note their ID.

ibmcloud ks vlan ls --zone <zone>
3. List available flavors in that zone.

ibmcloud ks flavors --zone <zone>
4. Add stand-alone worker nodes to the cluster. For bare metal flavors, specify dedicated.

ibmcloud ks worker add --cluster <cluster_name_or_ID> --workers <number_of_worker_nodes> --public-vlan <public_VLAN_ID> --private-vlan <private_VLAN_ID> --flavor <flavor> --hardware <shared_or_dedicated>
5.
The worker node IP address remains the same after the reboot operation.

Rebooting a worker node can cause data corruption on the worker node. Use this command with caution and when you know that a reboot can help recover your worker node. In all other cases, [reload your worker node](https://cloud.ibm.com/docs/containers?topic=containers-kubernetes-service-clics_worker_reload) instead.

Before you reboot your worker node, make sure that you have enough capacity in other worker nodes to reschedule the pods on the worker node. Rescheduling pods helps to avoid downtime for your app or data corruption on your worker node.



1. List all worker nodes in your cluster and note the name of the worker node that you want to remove.

kubectl get nodes

The name that is returned in this command is the private IP address that is assigned to your worker node. You can find more information about your worker node when you run the ibmcloud ks worker ls --cluster <cluster_name_or_ID> command and look for the worker node with the same Private IP address.
2. Force pods to be removed from your worker node and rescheduled onto remaining worker nodes in the cluster. The worker node is also cordoned, or marked as unavailable for future pod scheduling. Replace <worker_name> with the private IP address of the worker node that you previously retrieved.

kubectl drain <worker_name>

This process can take a few minutes.
3. Reboot the worker node. Use the worker ID that is returned from the ibmcloud ks worker ls --cluster <cluster_name_or_ID> command.

ibmcloud ks worker reboot --cluster <cluster_name_or_ID> --worker <worker_name_or_ID>
4. Wait about 5 minutes before you make your worker node available for pod scheduling to ensure that the reboot is finished. During the reboot, the state of your worker node does not change. The reboot of a worker node is usually completed in a few seconds.
5. Make your worker node available for pod scheduling.
The worker node IP address remains the same after the reboot operation.

Rebooting a worker node can cause data corruption on the worker node. Use this command with caution and when you know that a reboot can help recover your worker node. In all other cases, [reload your worker node](https://cloud.ibm.com/docs/cli?topic=cli-kubernetes-service-clics_worker_reload) instead.

Before you reboot your worker node, make sure that you have enough capacity in other worker nodes to reschedule the pods on the worker node. Rescheduling pods helps to avoid downtime for your app or data corruption on your worker node.



1. List all worker nodes in your cluster and note the name of the worker node that you want to remove.

kubectl get nodes

The name that is returned in this command is the private IP address that is assigned to your worker node. You can find more information about your worker node when you run the ibmcloud ks worker ls --cluster <cluster_name_or_ID> command and look for the worker node with the same Private IP address.
2. Force pods to be removed from your worker node and rescheduled onto remaining worker nodes in the cluster. The worker node is also cordoned, or marked as unavailable for future pod scheduling. Replace <worker_name> with the private IP address of the worker node that you previously retrieved.

kubectl drain <worker_name>

This process can take a few minutes.
3. Reboot the worker node. Use the worker ID that is returned from the ibmcloud ks worker ls --cluster <cluster_name_or_ID> command.

ibmcloud ks worker reboot --cluster <cluster_name_or_ID> --worker <worker_name_or_ID>
4. Wait about 5 minutes before you make your worker node available for pod scheduling to ensure that the reboot is finished. During the reboot, the state of your worker node does not change. The reboot of a worker node is usually completed in a few seconds.
5.
HAPROXY c619f4 508bf6 [CVE-2016-3709](https://nvd.nist.gov/vuln/detail/CVE-2016-3709), [CVE-2022-42898](https://nvd.nist.gov/vuln/detail/CVE-2022-42898), [CVE-2022-1304](https://nvd.nist.gov/vuln/detail/CVE-2022-1304). 







 Change log for worker node fix pack 4.7.60_1590_openshift, released 21 November 2022 

The following table shows the changes that are in the worker node fix pack 4.7.60_1590_openshift. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 4.7.59_1588_openshift

 Component Previous Current Description
When you have more than 14 worker nodes, another subnet is ordered, so the subnets that you need to allow can change. To reduce the frequency of change, create worker pools with worker node flavors of higher CPU and memory resources so that you don't need to add worker nodes as often.



1. List the worker nodes in your cluster.

ibmcloud oc worker ls --cluster <cluster_name_or_ID>
2. From the output of the previous step, note all the unique network IDs (first three octets) of the Public IP for the worker nodes in your cluster. In the following output, the unique network IDs are 169.xx.178 and 169.xx.210.

ID Public IP Private IP Machine Type State Status Zone Version
kube-dal10-crb2f60e9735254ac8b20b9c1e38b649a5-w31 169.xx.178.101 10.xxx.xx.xxx b3c.4x16.encrypted normal Ready dal10 1.26
kube-dal10-crb2f60e9735254ac8b20b9c1e38b649a5-w34 169.xx.178.102 10.xxx.xx.xxx b3c.4x16.encrypted normal Ready dal10 1.26
kube-dal12-crb2f60e9735254ac8b20b9c1e38b649a5-w32 169.xx.210.101 10.xxx.xx.xxx b3c.4x16.encrypted normal Ready dal12 1.26
kube-dal12-crb2f60e9735254ac8b20b9c1e38b649a5-w33 169.xx.210.102 10.xxx.xx.xxx b3c.4x16.encrypted normal Ready dal12 1.26
3. List the VLAN subnets for each unique network ID.

ibmcloud sl subnet list | grep -e <networkID1> -e <networkID2>
Change log for worker node fix pack 1.13.12_1549, released 3 February 2020 

The following table shows the changes that are in the worker node fix pack 1.13.12_1549.



Changes since version 1.13.12_1548

 Component Previous Current Description 

 Ubuntu 18.04 packages 4.15.0-74-generic 4.15.0-76-generic Updated worker node images with kernel and package updates for [CVE-2019-19906](https://nvd.nist.gov/vuln/detail/CVE-2019-19906), [CVE-2019-5188](https://nvd.nist.gov/vuln/detail/CVE-2019-5188), [CVE-2019-20367](https://nvd.nist.gov/vuln/detail/CVE-2019-20367), [CVE-2020-7053](https://nvd.nist.gov/vuln/detail/CVE-2020-7053), [CVE-2019-14615](https://nvd.nist.gov/vuln/detail/CVE-2019-14615), [CVE-2019-15796](https://nvd.nist.gov/vuln/detail/CVE-2019-15796), [CVE-2019-15795
* [Normal state](https://cloud.ibm.com/docs/openshift?topic=openshift-worker-node-state-referenceworker-node-normal)
* [Provisioned state](https://cloud.ibm.com/docs/openshift?topic=openshift-worker-node-state-referenceworker-node-provisioned)
* [Provisioning state](https://cloud.ibm.com/docs/openshift?topic=openshift-worker-node-state-referenceworker-node-provisioning)
* [Provision pending state](https://cloud.ibm.com/docs/openshift?topic=openshift-worker-node-state-referenceworker-node-provision-pending)
* [Provision_failed state](https://cloud.ibm.com/docs/openshift?topic=openshift-worker-node-state-referenceworker-node-provision-failed)
* [Reloading state](https://cloud.ibm.com/docs/openshift?topic=openshift-worker-node-state-referenceworker-node-reloading)
* [Reloading_failed state](https://cloud.ibm.com/docs/openshift?topic=openshift-worker-node-state-referenceworker-node-reloading-failed)
* [Reload_pending state](https://cloud.ibm.com/docs/openshift?topic=openshift-worker-node-state-referenceworker-node-reload-pending)
* [Unknown state](https://cloud.ibm.com/docs/openshift?topic=openshift-worker-node-state-referenceworker-node-unknown)
* [Warning state](https://cloud.ibm.com/docs/openshift?topic=openshift-worker-node-state-referenceworker-node-warning)
If any workers remain in the Critical or NotReady state, [reload](https://cloud.ibm.com/docs/containers?topic=containers-kubernetes-service-cli&interface=uics_worker_reload) or [replace](https://cloud.ibm.com/docs/containers?topic=containers-kubernetes-service-cli&interface=uicli_worker_replace) the affected workers.
: If a worker node was reloaded or replaced and initially works correctly, but then after some time goes back into a Critical or NotReady state, then it is likely that some workload or component on the worker is causing the issue. See [Debugging worker nodes](https://cloud.ibm.com/docs/containers?topic=containers-debug-kube-nodes) to isolate the problem workload.

A worker node might end up in a Critical or NotReady state if it was rebooted without first being cordoned and drained. If this is the case, waiting for the reboot to complete does not resolve the issue. [Reload](https://cloud.ibm.com/docs/containers?topic=containers-kubernetes-service-cli&interface=uics_worker_reload) or [replace](https://cloud.ibm.com/docs/containers?topic=containers-kubernetes-service-cli&interface=uicli_worker_replace) the affected worker. If the issue persists, continue with the troubleshooting steps.

The worker node was unintentionally powered down
:
Classic clusters

In the [IBM Cloud console resource list](https://cloud.ibm.com/resources), worker nodes on classic infrastructure are classified as compute resources, or virtual machines. Sometimes a user might not realize that these resources function as cluster worker nodes, and might unintentionally power the worker nodes down. Worker nodes that are powered down might show up in the Critical or NotReady state. Ensure that the affected worker nodes are not powered down.





 Troubleshooting steps
For more information, see [Updating the master](https://cloud.ibm.com/docs/containers?topic=containers-updatemaster).

Worker nodes in standard clusters are provisioned in to your IBM Cloud infrastructure account. The worker nodes are dedicated to your account and you are responsible to request timely updates to the worker nodes to ensure that the worker node OS and Red Hat OpenShift on IBM Cloud components apply the latest security updates and patches. Security updates and patches are made available by IBM Site Reliability Engineers (SREs) who continuously monitor the Linux image that is installed on your worker nodes to detect vulnerabilities and security compliance issues. For more information, see [Updating worker nodes](https://cloud.ibm.com/docs/containers?topic=containers-updateworker_node).
* Are the master and worker nodes highly available?

The Red Hat OpenShift on IBM Cloud architecture and infrastructure is designed to ensure reliability, low processing latency, and a maximum uptime of the service. By default, every cluster in Red Hat OpenShift on IBM Cloud is set up with multiple Red Hat OpenShift master instances to ensure availability and accessibility of your cluster resources, even if one or more instances of your Red Hat OpenShift master are unavailable.

You can make your cluster even more highly available and protect your app from a downtime by spreading your workloads across multiple worker nodes in multiple zones of a region. This setup is called a [multizone cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-ha_clustersmz-clusters) and ensures that your app is accessible, even if a worker node or an entire zone is not available.

To protect against an entire region failure, create [multiple clusters and spread them across IBM Cloud regions](https://cloud.ibm.com/docs/openshift?topic=openshift-ha_clustersmultiple-clusters-glb). By setting up a network load balancer (NLB) for your clusters, you can achieve cross-region load balancing and cross-region networking for your clusters.
To reload multiple worker nodes, use multiple options, such as -w worker1_id -w worker2_id.

--skip-master-healthcheck
: Skip a health check of your master before reloading or rebooting your worker nodes.

-f
: Optional: Force the command to run with no user prompts.

-q
: Optional: Do not show the message of the day or update reminders.



 Example worker reboot command 

ibmcloud oc worker reboot --cluster my_cluster -w kube-dal10-cr18a61a63a6a94b658596aa93d087aaa9-w1 -w kube-dal10-cr18a61a63a6a94b658596aa93d087aaa9-w2







 ibmcloud oc worker reload 

Classic infrastructure

Reload the configurations for a Classic worker node. To reload a worker node in a VPC cluster, use the [ibmcloud oc worker replace command](https://cloud.ibm.com/docs/openshift?topic=openshift-kubernetes-service-clicli_worker_replace) instead.

A reload can be useful if your worker node experiences problems, such as slow performance or if your worker node is stuck in an unhealthy state. During the reload, your worker node machine is updated with the latest image and data is deleted if not [stored outside the worker node](https://cloud.ibm.com/docs/openshift?topic=openshift-storage-plan). The worker node public and private IP address remain the same after the reload operation.

Reloading a worker node applies patch version updates to your worker node, but not major or minor updates. To see the changes from one patch version to the next, review the [Version change log](https://cloud.ibm.com/docs/openshift?topic=openshift-openshift_versions) documentation.
Change log for worker node fix pack 1.14.10_1546, released 3 February 2020 

The following table shows the changes that are in the worker node fix pack 1.14.10_1546.



Changes since version 1.14.10_1545

 Component Previous Current Description 

 Ubuntu 18.04 packages 4.15.0-74-generic 4.15.0-76-generic Updated worker node images with kernel and package updates for [CVE-2019-19906](https://nvd.nist.gov/vuln/detail/CVE-2019-19906), [CVE-2019-5188](https://nvd.nist.gov/vuln/detail/CVE-2019-5188), [CVE-2019-20367](https://nvd.nist.gov/vuln/detail/CVE-2019-20367), [CVE-2020-7053](https://nvd.nist.gov/vuln/detail/CVE-2020-7053), [CVE-2019-14615](https://nvd.nist.gov/vuln/detail/CVE-2019-14615), [CVE-2019-15796](https://nvd.nist.gov/vuln/detail/CVE-2019-15796), [CVE-2019-15795
Change log for worker node fix pack 1.22.3_1530, released 22 November 2021 

The following table shows the changes that are in the worker node fix pack patch update 1.22.3_1530. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 1.22.2_1528

 Component Previous Current Description 

 Kubernetes 1.22.2 1.22.3 For more information, see the [change log](https://github.com/kubernetes/kubernetes/releases/tag/v1.22.3). 
 HA proxy 07f1e9e 3b8663 Contains fixes for [CVE-2021-20231](https://nvd.nist.gov/vuln/detail/CVE-2021-20231), [CVE-2021-20232](https://nvd.nist.gov/vuln/detail/CVE-2021-20232), [CVE-2021-3580](https://nvd.nist.gov/vuln/detail/CVE-2021-3580), [CVE-2021-22946](https://nvd.nist.gov/vuln/detail/CVE-2021-22946), [CVE-2021-22947](https://nvd.nist.gov/vuln/detail/CVE-2021-22947), [CVE-2021-22876](https://nvd.nist.gov/vuln/detail/CVE-2021-22876), [CVE-2021-22898](https://nvd.nist.gov/vuln/detail/CVE-2021-22898), [CVE-2021-22925](https://nvd.nist.gov/vuln/detail/CVE-2021-22925), [CVE-2019-20838](https://nvd.nist.gov/vuln/detail/CVE-2019-20838), [CVE-2020
To check if a bare metal worker node has a TPM chip, review the Trustable field after running the ibmcloud ks flavors --zone[command](https://cloud.ibm.com/docs/containers?topic=containers-kubernetes-service-clics_machine_types). 







 Change log for worker node fix pack 1.10.8_1528, released 26 October 2018 

The following table shows the changes that are in the worker node fix pack 1.10.8_1528.



Table 1. Changes since version 1.10.8_1527

 Component Previous Current Description 

 OS interrupt handling N/A N/A Replaced the interrupt request (IRQ) system daemon with a more performant interrupt handler. 







 Change log for master fix pack 1.10.8_1527, released 15 October 2018 

The following table shows the changes that are in the master fix pack 1.10.8_1527.



Table 1. Changes since version 1.10.8_1524

 Component Previous Current Description 

 Calico configuration N/A N/A Fixed calico-node container readiness probe to better handle node failures. 
 Cluster update N/A N/A Fixed problem with updating cluster add-ons when the master is updated from an unsupported version. 







 Change log for worker node fix pack 1.10.8_1525, released 10 October 2018 

The following table shows the changes that are in the worker node fix pack 1.10.8_1525.



Table 1. Changes since version 1.10.8_1524

 Component Previous Current Description 

 Kernel 4.4.0-133 4.4.0-137 Updated worker node images with kernel update for [CVE-2018-14633, CVE-2018-17182](https://changelogs.ubuntu.com/changelogs/pool/main/l/linux/linux_4.4.0-137.163/changelog). 
 Inactive session timeout N/A N/A Set the inactive session timeout to 5 minutes for compliance reasons. 







 Change log for 1.10.8_1524, released 2 October 2018
Containerd N/A N/A N/A 
 Haproxy af5031 8398d1 [CVE-2023-23916](https://nvd.nist.gov/vuln/detail/CVE-2023-23916). 







 Change log for worker node fix pack 1.23.16_1568, released 13 March 2023 

The following table shows the changes that are in the worker node fix pack 1.23.16_1568. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 1.23.16_1563

 Component Previous Current Description 

 Ubuntu 20.04 packages 5.4.0-139 5.4.0-144 Worker node kernel & package updates for [CVE-2022-29154](https://nvd.nist.gov/vuln/detail/CVE-2022-29154), [CVE-2022-3545](https://nvd.nist.gov/vuln/detail/CVE-2022-3545), [CVE-2022-3821](https://nvd.nist.gov/vuln/detail/CVE-2022-3821), [CVE-2022-40898](https://nvd.nist.gov/vuln/detail/CVE-2022-40898), [CVE-2022-41218](https://nvd.nist.gov/vuln/detail/CVE-2022-41218), [CVE-2022-4139](https://nvd.nist.gov/vuln/detail/CVE-2022-4139),