vania-janet/multiturn-rag-retrieval-data huggingface dataset:
[Why doesn't replacing a worker node create a worker node?](https://cloud.ibm.com/docs/openshift?topic=openshift-auto-rebalance-offauto-rebalance-off)

[Classic: Why is the bare metal instance ID inconsistent with worker records?](https://cloud.ibm.com/docs/openshift?topic=openshift-bm_machine_idbm_machine_id)

[After deleting all worker nodes, why don't my pods start on new worker nodes?](https://cloud.ibm.com/docs/openshift?topic=openshift-zero_nodes_calico_failurezero_nodes_calico_failure)

[After a worker node updates or reloads, why do duplicate nodes and pods appear?](https://cloud.ibm.com/docs/openshift?topic=openshift-cs_duplicate_nodescs_duplicate_nodes)

[Classic: Why can't I add worker nodes with an invalid VLAN ID?](https://cloud.ibm.com/docs/openshift?topic=openshift-suspendedsuspended)

[Classic: Why do I see a timeout error when I try to log in to a pod on a new worker node?](https://cloud.ibm.com/docs/openshift?topic=openshift-cs_nodes_duplicate_ipcs_nodes_duplicate_ip)

[Why can't I SSH into my worker node?](https://cloud.ibm.com/docs/openshift?topic=openshift-cs_ssh_workercs_ssh_worker)



* [Debugging by using oc debug](https://cloud.ibm.com/docs/openshift?topic=openshift-cs_ssh_workeroc-debug)
Why can't I log in to my worker nodes or debug them with oc debug command? 

  What’s happening 

When you log in to the terminal of worker node or run the oc debug node command from the CLI, you receive the following error.

oc debug node/10.200.0.00
Starting pod/10200000-debug ...

To use host binaries, run chroot /host
Pod IP: 10.200.0.10
If you don't see a command prompt, try pressing enter.

Removing debug pod ...
Error from server: error dialing backend: dial tcp 10.200.0.00:12345: connect: connection timed out

  Why it’s happening 

There is a faulty node that is causing DNS issues to the cluster.

  How to fix it 

To resolve this issue, determine which node is causing the error. Then, stop that node and try running oc debug node. See [Debugging clusters](https://cloud.ibm.com/docs/satellite?topic=satellite-ts-clusters-debug).

If you cannot determine which node is causing the issue, [reboot your Satellite location control plane hosts](https://cloud.ibm.com/docs/satellite?topic=satellite-host-update-location) one at a time and try running oc debug node. If you are still receiving the error, then [reboot one worker node at a time](https://cloud.ibm.com/docs/satellite?topic=satellite-host-update-workers).

If you can determine that a particular worker node is causing the problem and the oc debug node command is successful on other worker nodes, then replace the host where the oc debug node command fails.
Change log for worker node fix pack 1.23.10_1545, released 12 September 2022 

The following table shows the changes that are in the worker node fix pack 1.23.10_1545. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 1.23.10_1543

 Component Previous Current Description 

 Ubuntu 18.04 packages 4.15.0-191 4.15.0-192 Worker node kernel & package updates for [CVE-2021-33656](https://nvd.nist.gov/vuln/detail/CVE-2021-33656), [CVE-2022-35252](https://nvd.nist.gov/vuln/detail/CVE-2022-35252). 
 Kubernetes N/A N/A N/A 







 Change log for master fix pack 1.23.10_1544, released 1 September 2022 

The following table shows the changes that are in the master fix pack 1.23.10_1544. Master patch updates are applied automatically.



Changes since version 1.23.9_1539

 Component Previous Current Description 

 Cluster health image v1.3.9 v1.3.10 Updated Go dependencies and to Go version 1.18.5. 
 CoreDNS 1.8.7 1.8.6 See the [CoreDNS release notes](https://coredns.io/2021/10/07/coredns-1.8.6-release/). 
 Gateway-enabled cluster controller 1792 1823 Updated to Go version 1.17.13. 
 GPU device plug-in and installer d8f1be0 c58c299 Enable DRM module and update Go to version 1.18.5. 
 IBM Calico extension 997 1006 Updated to Go version 1.17.13. 
 IBM Cloud Controller Manager v1.23.8-7 v1.23.9-3 Updated vpcctl binary to version 3367. Updated to support the Kubernetes 1.23.10 release.
: Optional: The public VLAN that was specified when the cluster was created. If you want your worker nodes to exist on a private VLAN only, don't provide a public VLAN ID. Private VLAN routers always begin with bcr (back-end router) and public VLAN routers always begin with fcr (front-end router). When you create a cluster and specify the public and private VLANs, the number and letter combination after those prefixes must match. If worker nodes are set up with a private VLAN only, you must allow worker nodes and the cluster master to communicate by [enabling the private cloud service endpoint](https://cloud.ibm.com/docs/containers?topic=containers-cs_network_clusterset-up-private-se) or [configuring a gateway appliance](https://cloud.ibm.com/docs/containers?topic=containers-plan_basicsworkeruser-master).

--disable-disk-encrypt
: Worker nodes feature AES 256-bit disk encryption by default; [learn more](https://cloud.ibm.com/docs/containers?topic=containers-securityencrypted_disk). To disable encryption, include this option.

--operating-system SYSTEM
: Optional. The operating system of the worker nodes you want to provision in your cluster. For a list of available operating systems by cluster version, see [IBM Cloud Kubernetes Service version information](https://cloud.ibm.com/docs/containers?topic=containers-cs_versions).

-q
: Optional: Do not show the message of the day or update reminders.



 Example worker add command 

ibmcloud ks worker add --cluster my_cluster --workers 3 --public-vlan my_public_VLAN_ID --private-vlan my_private_VLAN_ID --flavor b3c.4x16 --hardware shared







 ibmcloud ks worker get 

Virtual Private Cloud

Classic infrastructure

View the details of a worker node.
The worker_node_min_count variable specifies the number of worker nodes that are provisioned at the time that the cluster is created, which will exist throughout the life of the cluster. The delta between those two variables specifies the maximum number of worker nodes that can either be created or destroyed by the Symphony Host Factory auto-scaling feature.

The Spectrum Symphony offering supports both bare metal worker nodes and Spectrum Scale storage nodes. The following combinations of values are supported:



* If worker_node_type is set as baremetal, a maximum of 16 bare metal nodes are supported.
* If spectrum_scale_enabled is set to true and storage_type is set as persistent, a maximum of 10 bare metal nodes are supported.



For more information, see [Deployment values](https://cloud.ibm.com/docs/hpc-spectrum-symphony?topic=hpc-spectrum-symphony-deployment-values).

When creating or deleting a cluster with many worker nodes, you might encounter VPC resource provisioning or deletion failures. In those cases, running the Schematics apply or destroy operation again might result in the remaining resources being successfully provisioned or deleted. If you continue to see errors, see [Getting help and support](https://cloud.ibm.com/docs/hpc-spectrum-symphony?topic=hpc-spectrum-symphony-getting-help-and-support).
* Why are there two different resource group parameters that can be specified in the IBM Cloud catalog tile?

The first resource group parameter entry in the Configure your workspace section in the IBM Cloud catalog applies to the resource group where the Schematics workspace is provisioned on your IBM Cloud account. The value for this parameter can be different than the one used for the second entry in the Parameters with default values section in the catalog. The second entry applies to the resource group where VPC resources are provisioned. As specified in the description for this second resource_group parameter, note that only the default resource group is supported for use of the Symphony Host Factory auto-scaling feature.
* Can I use the Spectrum Symphony Host Factory feature for auto scaling on any cluster deployed with this offering?
The following image shows the components that are set up for every worker node to protect your worker node from malicious attacks.

The image does not include components that ensure secure end-to-end communication to and from the worker node. For more information, see [network security](https://cloud.ibm.com/docs/containers?topic=containers-securitynetwork).

Zoom

![Worker node setup in IBM Cloud Kubernetes Service excluding network security.](https://cloud.ibm.com/docs-content/v1/content/4d497aff9e27b92fbbc2ee4e343a9ec4549cb1d7/containers/images/cs_worker_setup.png)

Figure 2. Worker node setup in IBM Cloud Kubernetes Service excluding network security

CIS-compliant Linux image
: Every worker node is set up with an Ubuntu operating system that implements the benchmarks that are published by the Center of Internet Security (CIS). The user or the owner of the machine can't change this operating system to another operating system. To review the current Ubuntu version, run kubectl get nodes -o wide. IBM works with internal and external security advisory teams to address potential security compliance vulnerabilities. Security updates and patches for the operating system are made available through IBM Cloud Kubernetes Service and must be installed by the user to keep the worker node secure. IBM Cloud Kubernetes Service uses an Ubuntu Linux kernel for worker nodes. You can run containers based on any Linux distribution in IBM Cloud Kubernetes Service. Check with your container image vendor to verify that your container images can run on an Ubuntu kernel.

Continuous monitoring by Site Reliability Engineers (SREs)
: The image that is installed on your worker nodes is continuously monitored by IBM Site Reliability Engineers (SREs) to detect vulnerabilities and security compliance issues. To address vulnerabilities, SREs create security patches and fix packs for your worker nodes. Make sure to apply these patches when they are available to ensure a secure environment for your worker nodes and the apps that you run on them.
Kubernetes NodeLocal DNS cache configuration N/A N/A Increased the pod termination grace period. 







 Change log for worker node fix pack 1.18.8_1525, released 17 August 2020 

The following table shows the changes that are in the worker node fix pack 1.18.8_1525. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 1.18.6_1523

 Component Previous Current Description 

 Kubernetes v1.18.6 v1.18.8 See the [Kubernetes change logs](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.mdv1188).
Change log for worker node fix pack 1.11.3_1531, released 26 October 2018 

The following table shows the changes that are in the worker node fix pack 1.11.3_1531.



Changes since version 1.11.3_1525

 Component Previous Current Description 

 OS interrupt handling N/A N/A Replaced the interrupt request (IRQ) system daemon with a more performant interrupt handler. 







 Change log for master fix pack 1.11.3_1527, released 15 October 2018 

The following table shows the changes that are in the master fix pack 1.11.3_1527.



Changes since version 1.11.3_1524

 Component Previous Current Description 

 Calico configuration N/A N/A Fixed calico-node container readiness probe to better handle node failures. 
 Cluster update N/A N/A Fixed problem with updating cluster add-ons when the master is updated from an unsupported version. 







 Change log for worker node fix pack 1.11.3_1525, released 10 October 2018 

The following table shows the changes that are in the worker node fix pack 1.11.3_1525.



Changes since version 1.11.3_1524

 Component Previous Current Description 

 Kernel 4.4.0-133 4.4.0-137 Updated worker node images with kernel update for [CVE-2018-14633, CVE-2018-17182](https://changelogs.ubuntu.com/changelogs/pool/main/l/linux/linux_4.4.0-137.163/changelog). 
 Inactive session timeout N/A N/A Set the inactive session timeout to 5 minutes for compliance reasons. 







 Change log for 1.11.3_1524, released 2 October 2018 

The following table shows the changes that are in patch 1.11.3_1524.



Changes since version 1.11.3_1521

 Component Previous Current Description
The worker node instance can't be found. Review '<provider>' infrastructure user permissions.

The worker node instance can't be identified. Review '<provider>' infrastructure user permissions.

The IAM token exchange request failed with the message: <message>
IAM token exchange request failed: <message>

The cluster could not be configured with the registry. Make sure that you have the Administrator role for Container Registry.

 Why it’s happening 

The infrastructure credentials that are set for the region and resource group are missing the appropriate [infrastructure permissions](https://cloud.ibm.com/docs/containers?topic=containers-classic-roles).

The user's infrastructure permissions are most commonly stored as an [API key](https://cloud.ibm.com/docs/containers?topic=containers-access-creds) for the region and resource group. More rarely, if you use a [different IBM Cloud account type](https://cloud.ibm.com/docs/containers?topic=containers-access-credsunderstand_infra), you might have [set infrastructure credentials manually](https://cloud.ibm.com/docs/containers?topic=containers-access-credscredentials).

 How to fix it 

The account owner must set up the infrastructure account credentials properly. The credentials depend on what type of infrastructure account you are using.

Before you begin, [Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.](https://cloud.ibm.com/docs/containers?topic=containers-access_cluster).



1. Identify what user credentials are used for the region and resource group's infrastructure permissions.



1. Check the API key for a region and resource group of the cluster.

ibmcloud ks api-key info --cluster <cluster_name_or_ID>

Example output

Getting information about the API key owner for cluster <cluster_name>...
OK
Name Email
<user_name> <name@email.com>
2.
Ubuntu 16.04 kernel and packages 4.4.0-161-generic 4.4.0-164-generic Updated worker node images with kernel and package updates for [CVE-2019-14835](https://nvd.nist.gov/vuln/detail/CVE-2019-14835). 







 Change log for worker node fix pack 1.13.10_1536, released 16 September 2019 

The following table shows the changes that are in the worker node fix pack 1.13.10_1536.



Table 1. Changes since version 1.13.10_1535

 Component Previous Current Description 

 containerd v1.2.8 v1.2.9 See the [containerd release notes](https://github.com/containerd/containerd/releases/tag/v1.2.9). Update resolves [CVE-2019-9515](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-9515).
The worker node IP address remains the same after the reboot operation.

Rebooting a worker node can cause data corruption on the worker node. Use this command with caution and when you know that a reboot can help recover your worker node. In all other cases, [reload your worker node](https://cloud.ibm.com/docs/openshift?topic=openshift-kubernetes-service-clics_worker_reload) instead.

Before you reboot your worker node, make sure that you have enough capacity in other worker nodes to reschedule the pods on the worker node. Rescheduling pods helps to avoid downtime for your app or data corruption on your worker node.



1. List all worker nodes in your cluster and note the name of the worker node that you want to remove.

oc get nodes

The name that is returned in this command is the private IP address that is assigned to your worker node. You can find more information about your worker node when you run the ibmcloud oc worker ls --cluster <cluster_name_or_ID> command and look for the worker node with the same Private IP address.
2. Force pods to be removed from your worker node and rescheduled onto remaining worker nodes in the cluster. The worker node is also cordoned, or marked as unavailable for future pod scheduling. Replace <worker_name> with the private IP address of the worker node that you previously retrieved.

oc adm drain <worker_name>

This process can take a few minutes.
3. Reboot the worker node. Use the worker ID that is returned from the ibmcloud oc worker ls --cluster <cluster_name_or_ID> command.

ibmcloud oc worker reboot --cluster <cluster_name_or_ID> --worker <worker_name_or_ID>
4. Wait about 5 minutes before you make your worker node available for pod scheduling to ensure that the reboot is finished. During the reboot, the state of your worker node does not change. The reboot of a worker node is usually completed in a few seconds.
5.
To reload multiple worker nodes, use multiple options, such as -w worker1_id -w worker2_id.

--skip-master-healthcheck
: Skip a health check of your master before reloading or rebooting your worker nodes.

-f
: Optional: Force the command to run with no user prompts.

-q
: Optional: Do not show the message of the day or update reminders.



 Example worker reboot command 

ibmcloud ks worker reboot --cluster my_cluster -w kube-dal10-cr18a61a63a6a94b658596aa93d087aaa9-w1 -w kube-dal10-cr18a61a63a6a94b658596aa93d087aaa9-w2







 ibmcloud ks worker reload 

Classic infrastructure

Reload the configurations for a Classic worker node. To reload a worker node in a VPC cluster, use the [ibmcloud ks worker replace command](https://cloud.ibm.com/docs/cli?topic=cli-kubernetes-service-clicli_worker_replace) instead.

A reload can be useful if your worker node experiences problems, such as slow performance or if your worker node is stuck in an unhealthy state. During the reload, your worker node machine is updated with the latest image and data is deleted if not [stored outside the worker node](https://cloud.ibm.com/docs/containers?topic=containers-storage-plan). The worker node public and private IP address remain the same after the reload operation.

Reloading a worker node applies patch version updates to your worker node, but not major or minor updates. To see the changes from one patch version to the next, review the [Version change log](https://cloud.ibm.com/docs/containers?topic=containers-changelog) documentation.

Before you reload your worker node, make sure that you have enough capacity in other worker nodes to reschedule the pods on the worker node.
Ubuntu 16.04 kernel and packages 4.4.0-161-generic 4.4.0-164-generic Updated worker node images with kernel and package updates for [CVE-2019-14835](https://nvd.nist.gov/vuln/detail/CVE-2019-14835). 







 Change log for worker node fix pack 1.14.6_1533, released 16 September 2019 

The following table shows the changes that are in the worker node fix pack 1.14.6_1533.



Table 1. Changes since version 1.14.6_1532

 Component Previous Current Description 

 containerd v1.2.8 v1.2.9 See the [containerd release notes](https://github.com/containerd/containerd/releases/tag/v1.2.9). Update resolves [CVE-2019-9515](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-9515).
To reload multiple worker nodes, use multiple options, such as -w worker1_id -w worker2_id.

--skip-master-healthcheck
: Skip a health check of your master before reloading or rebooting your worker nodes.

-f
: Optional: Force the command to run with no user prompts.

-q
: Optional: Do not show the message of the day or update reminders.



 Example worker reboot command 

ibmcloud ks worker reboot --cluster my_cluster -w kube-dal10-cr18a61a63a6a94b658596aa93d087aaa9-w1 -w kube-dal10-cr18a61a63a6a94b658596aa93d087aaa9-w2







 ibmcloud ks worker reload 

Classic infrastructure

Reload the configurations for a Classic worker node. To reload a worker node in a VPC cluster, use the [ibmcloud ks worker replace command](https://cloud.ibm.com/docs/containers?topic=containers-kubernetes-service-clicli_worker_replace) instead.

A reload can be useful if your worker node experiences problems, such as slow performance or if your worker node is stuck in an unhealthy state. During the reload, your worker node machine is updated with the latest image and data is deleted if not [stored outside the worker node](https://cloud.ibm.com/docs/containers?topic=containers-storage-plan). The worker node public and private IP address remain the same after the reload operation.

Reloading a worker node applies patch version updates to your worker node, but not major or minor updates. To see the changes from one patch version to the next, review the [Version change log](https://cloud.ibm.com/docs/containers?topic=containers-changelog) documentation.

Before you reload your worker node, make sure that you have enough capacity in other worker nodes to reschedule the pods on the worker node.
(https://cloud.ibm.com/docs/containers?topic=containers-bm_machine_idbm_machine_id)

[After deleting all worker nodes, why don't my pods start on new worker nodes?](https://cloud.ibm.com/docs/containers?topic=containers-zero_nodes_calico_failurezero_nodes_calico_failure)

[After a worker node updates or reloads, why do duplicate nodes and pods appear?](https://cloud.ibm.com/docs/containers?topic=containers-cs_duplicate_nodescs_duplicate_nodes)

[Classic: Why can't I add worker nodes with an invalid VLAN ID?](https://cloud.ibm.com/docs/containers?topic=containers-suspendedsuspended)

[Classic: Why do I see a timeout error when I try to log in to a pod on a new worker node?](https://cloud.ibm.com/docs/containers?topic=containers-cs_nodes_duplicate_ipcs_nodes_duplicate_ip)

[Why can't I SSH into my worker node?](https://cloud.ibm.com/docs/containers?topic=containers-cs_ssh_workercs_ssh_worker)



* [Debugging by using kubectl debug](https://cloud.ibm.com/docs/containers?topic=containers-cs_ssh_workeroc-debug)
* [Debugging by using kubectl exec](https://cloud.ibm.com/docs/containers?topic=containers-cs_ssh_workerkubectl-exec)
* [Debugging by creating a pod with root SSH access](https://cloud.ibm.com/docs/containers?topic=containers-cs_ssh_workerpod-ssh)



* [Private network](https://cloud.ibm.com/docs/containers?topic=containers-cs_ssh_workerssh-private-network)