vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Debugging worker nodes with Kubernetes API 

If you have access to the cluster, you can debug the worker nodes by using the Kubernetes API on the Node resource.

Before you begin, make sure that you have the Manager service access role in all namespaces for the cluster, which corresponds to the cluster-admin RBAC role.



1. [Access your Red Hat OpenShift cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-access_cluster).
2. List the worker nodes in your cluster and note the NAME of the worker nodes that are not in a ReadySTATUS. Note that the NAME is the private IP address of the worker node.

oc get nodes
3. Describe the each worker node, and review the Conditions section in the output.



* Type: The type of condition that might affect the worker node, such as memory or disk pressure.
* LastTransitionTime: The most recent time that the status was updated. Use this time to identify when the issue with your worker node began, which can help you further troubleshoot the issue.



oc describe node <name>
4. Check the usage of the worker nodes.



1. In the Allocated resources output of the previous command, review the workloads that use the worker node's CPU and memory resources. You might notice that some pods don't set resource limits, and are consuming more resources than you expected. If so, adjust the resource usage of the pods.
2. Review the percentage of usage of CPU and memory across the worker nodes in your cluster. If the usage is consistently over 80%, [add more worker nodes](https://cloud.ibm.com/docs/openshift?topic=openshift-add_workers) to the cluster to support the workloads.



5. Check for custom admission controllers that are installed in your cluster. Admission controllers often block required pods from running, which might make your worker nodes enter a critical state. If you have custom admission controllers, try removing them with oc delete. Then, check if the worker node issue resolves.
The Docker alpine image here is used as an example. If the worker node doesn't have public network access, you can maintain a copy of the image for debugging in your own ICR repository or build a customized image with other tools to fit your needs.

kubectl debug node/<NODE_NAME> --image=docker.io/library/alpine:latest -it
3. Run debug commands to help you gather information and troubleshoot issues. Commands that you might use to debug, such as ip, ifconfig, nc, ping, and ps, are already available in the shell. You can also install other tools, such as mtr, tcpdump, and curl, by running apk add <tool>.







 Debugging by using kubectl exec 

If you are unable to use the kubectl debug node command, you can create an Alpine pod with a privileged securityContext and use the kubectl exec command to run debug commands from the pod's interactive shell.



1. Get the name of the worker node that you want to access. The worker node name is its private IP address.

kubectl get nodes -o wide
2. Export the name in an environment variable.

export NODE=<NODE_NAME>
3. Create a debug pod on the worker node. The Docker alpine image here is used as an example. If the worker node doesn't have public network access, you can maintain a copy of the image for debugging in your own ICR repository or build a customized image with other tools to fit your needs.

kubectl apply -f - << EOF
apiVersion: v1
kind: Pod
metadata:
name: debug-${NODE}
namespace: default
spec:
containers:
- args: ["-c", "apk add tcpdump mtr curl; sleep 1d"]
command: ["/bin/sh"]
image: icr.io/armada-master/alpine:latest
imagePullPolicy: IfNotPresent
For a list of available worker node flavors, run ibmcloud ks flavors --zone <zone> --provider classic, or see [Classic flavors](https://cloud.ibm.com/docs/containers?topic=containers-classic-flavors&interface=ui).

hardware
: The level of hardware isolation for your worker nodes. Use dedicated to have available physical resources dedicated to you only, or shared to allow physical resources to be shared with other IBM customers. This option is available for virtual machine worker nodes flavors only.

public_vlan_id and private_vlan_id
: Optional. The ID of the public or private VLAN that you want to use for your worker nodes. To find available VLANs and subnets, run ibmcloud ks vlans --zone <zone>.

subnet_id
: Optional. The ID of an existing subnet that you want to use for your worker nodes. To find existing subnets, run ibmcloud ks subnets --provider classic --zone <zone>.

default_pool_size
: The number of worker nodes that you want to add to the default worker pool.
3. In the CLI, navigate to your Terraform directory.

cd <terraform_directory>
4. Run the commands to initialize and plan your Terraform actions. Review the plan output to make sure the correct actions are performed.

terraform init

terraform plan
5. Apply the Terraform files to create the cluster. Then, navigate to the IBM Cloud console to check that the cluster is provisioning.

terraform apply
After a worker node updates or reloads, why do duplicate nodes and pods appear? 

Virtual Private Cloud

Classic infrastructure

  What’s happening 

When you run oc get nodes, you see duplicate worker nodes with the status NotReady. The worker nodes with NotReady have public IP addresses, while the worker nodes with Ready have private IP addresses.

  Why it’s happening 

Older clusters listed worker nodes by the cluster's public IP address. Now, worker nodes are listed by the cluster's private IP address. When you reload or update a node, the IP address is changed, but the reference to the public IP address remains.

  How to fix it 

Service is not disrupted due to these duplicates, but you can remove the old worker node references from the API server.

oc delete node <node_name1> <node_name2>
* See [Getting help](https://cloud.ibm.com/docs/get-support?topic=get-support-using-avatar) for more details about using the forums.









 Contacting support 

Before you open a support case, gather relevant information about your cluster environment.



1. Get your cluster details.

ibmcloud ks cluster get -c <cluster_name_or_ID>
2. If your issue involves worker nodes, get the worker node details.



1. List all worker nodes in the cluster, and note the ID of any worker nodes with an unhealthy State or Status.

ibmcloud ks worker ls -c <cluster_name_or_ID>
2. Get the details of the unhealthy worker node.

ibmcloud ks worker get -w <worker_ID> -c <cluster_name_or_ID>



3. For issues with resources within your cluster such as pods or services, log in to the cluster and use the Kubernetes API to get more information about them.

You can also use the [IBM Cloud Kubernetes Service Diagnostics and Debug Tool](https://cloud.ibm.com/docs/containers?topic=containers-debug-tool) to gather and export pertinent information to share with IBM Support.
4. Contact IBM Support by [opening a case](https://cloud.ibm.com/unifiedsupport/cases/form). To learn about opening an IBM support case, or about support levels and case severities, see [Contacting support](https://cloud.ibm.com/docs/get-support?topic=get-support-using-avatar).
5. For the Problem type, search for or select Kubernetes Service.
6. For the Case details, provide a descriptive title and include the details that you previously gathered. From the Resources, you can also select the cluster that the issue is related to.







 Requesting access to allowlisted features 



1. [Create a ticket](https://cloud.ibm.com/unifiedsupport/cases/form).
If you are updating your worker nodes to a new major version, such as from 4.11 to 4.12, update the cluster master first.

ibmcloud oc cluster master update --cluster CLUSTER [--version MAJOR.MINOR.PATCH] [--force-update] [-f] [-q]

Example command:

ibmcloud oc cluster master update --cluster mycluster --version 4.11.42 --force-update
2. Wait until the master update finishes.







 Step 2: Determine which storage nodes you want to update 

Major

Minor



1. List your worker nodes by using oc get nodes and determine which storage nodes you want to update.

oc get nodes

Example output

NAME STATUS ROLES AGE VERSION
10.241.0.4 Ready master,worker 106s v1.21.6+4b61f94
10.241.128.4 Ready master,worker 22d v1.21.6+bb8d50a
10.241.64.4 Ready master,worker 22d v1.21.6+bb8d50a







 Step 3: Scale down OpenShift Data Foundation 

Major

Minor



1. For each worker node that you found in the previous step, find the rook-ceph-mon and rook-ceph-osd deployments.

oc get pods -n openshift-storage -o wide | grep -i <node_name>
2. Scale down the deployments that you found in the previous step.

oc scale deployment rook-ceph-mon-c --replicas=0 -n openshift-storage
deployment.apps/rook-ceph-mon-c scaled
oc scale deployment rook-ceph-osd-2 --replicas=0 -n openshift-storage
deployment.apps/rook-ceph-osd-2 scaled
oc scale deployment --selector=app=rook-ceph-crashcollector,node_name=NODE-NAME --replicas=0 -n openshift-storage







 Step 4: Cordon and drain the worker node 

Major

Minor
Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 1.16.14_1545

 Component Previous Current Description 

 HAProxy 1.8.25-384f42 1.8.26-561f1a See the [HAProxy change log](https://www.haproxy.org/download/1.8/src/CHANGELOG). 
 Ubuntu 18.04 packages 4.15.0-112-generic 4.15.0-117-generic Updated worker node image with kernel and package updates for [CVE-2020-14344](https://nvd.nist.gov/vuln/detail/CVE-2020-14344), [CVE-2020-14363](https://nvd.nist.gov/vuln/detail/CVE-2020-14363), and [CVE-2020-14386](https://nvd.nist.gov/vuln/detail/CVE-2020-14386). 
 Ubuntu 16.04 packages 4.4.0-187-generic 4.4.0-189-generic Updated worker node image with kernel and package updates for [CVE-2020-14344](https://nvd.nist.gov/vuln/detail/CVE-2020-14344) and [CVE-2020-14363](https://nvd.nist.gov/vuln/detail/CVE-2020-14363). 







 Change log for worker node fix pack 1.16.14_1545, released 31 August 2020 

The following table shows the changes that are in the worker node fix pack 1.16.14_1545. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 1.16.14_1544

 Component Previous Current Description
You can monitor the reloading process in the Status column of your CLI output. If your worker node is stuck in this state for a long time, check whether a problem occurred during the reloading.





 Reloading_failed state 

You can view the current worker node state by running the ibmcloud ks worker ls --cluster <cluster_name_or_ID> command and locating the State and Status fields.

A Reloading_failed state means that your worker node could not be reloaded. List the details for the worker node to find the details for the failure by running ibmcloud ks worker get --cluster <cluster_name_or_id> --worker <worker_node_id>.





 Reload_pending state 

You can view the current worker node state by running the ibmcloud ks worker ls --cluster <cluster_name_or_ID> command and locating the State and Status fields.

You requested to reload or to update the Kubernetes version of your worker node. When reloading begins, the state changes to Reloading.





 Unknown state 

You can view the current worker node state by running the ibmcloud ks worker ls --cluster <cluster_name_or_ID> command and locating the State and Status fields.

An Unknown state means that the Kubernetes master is not reachable for one of the following reasons:



* You requested an update of your Kubernetes master. The state of the worker node can't be retrieved during the update. If the worker node remains in this state for an extended period of time even after the Kubernetes master is successfully updated, try to [reload the worker node](https://cloud.ibm.com/docs/containers?topic=containers-kubernetes-service-clics_worker_reload).
* You might have another firewall that is protecting your worker nodes, or changed firewall settings recently. IBM Cloud Kubernetes Service requires certain IP addresses and ports to be opened to allow communication from the worker node to the Kubernetes master and vice versa.
: Required: The name of the worker node pool that you want to view the details of. To list available worker pools, run ibmcloud ks worker-pool ls --cluster <cluster_name_or_ID>.

-f
: Optional: Force the command to run with no user prompts.

-q
: Optional: Do not show the message of the day or update reminders.



 Example worker-pool label rm command 

ibmcloud ks worker-pool label rm --worker-pool pool1 --cluster my_cluster







 ibmcloud ks worker-pool label set 

Virtual Private Cloud

Classic infrastructure

Set custom Kubernetes labels in the format key=value for all the worker nodes in the worker pool. To keep any existing custom labels on the worker pool, you must include those labels in this command.

ibmcloud ks worker-pool label set --cluster CLUSTER --label LABEL [--label LABEL ...] --worker-pool POOL [-f] [-q]

Minimum required permissions
: Operator platform access role for the cluster in IBM Cloud Kubernetes Service

Command options:

-c, --cluster CLUSTER
: Required: The name or ID of the cluster where the worker pool is located.

--label LABEL
: Required: A custom label in the format key=value to set for all the worker nodes in the worker pool. For multiple labels, repeat this option. To keep any existing custom labels on the worker pool, include those labels with this option. You can list the existing custom labels on worker nodes in the worker pool by running ibmcloud ks worker-pool get -c <cluster_name_or_ID> --worker-pool <pool>.

-p, --worker-pool WORKER_POOL
: Required: The name of the worker node pool that you want to view the details of. To list available worker pools, run ibmcloud ks worker-pool ls --cluster <cluster_name_or_ID>.

-f
: Optional: Force the command to run with no user prompts.

-q
: Optional: Do not show the message of the day or update reminders.
Change log for worker node fix pack 1.18.19_1553, released 24 May 2021 

The following table shows the changes that are in the worker node fix pack 1.18.19_1553. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 1.18.18_1551

 Component Previous Current Description 

 HA proxy e0fa2f 26c5cc Updated image with fixes for [CVE-2020-26116](https://nvd.nist.gov/vuln/detail/CVE-2020-26116), [CVE-2020-27619](https://nvd.nist.gov/vuln/detail/CVE-2020-27619), [CVE-2021-23336](https://nvd.nist.gov/vuln/detail/CVE-2021-23336), [CVE-2021-3177](https://nvd.nist.gov/vuln/detail/CVE-2021-3177), [CVE-2019-3842](https://nvd.nist.gov/vuln/detail/CVE-2019-3842), [CVE-2020-13776](https://nvd.nist.gov/vuln/detail/CVE-2020-13776), [CVE-2019-18276](https://nvd.nist.gov/vuln/detail/CVE-2019-18276), [CVE-
For highly available data storage, Portworx requires at least 3 worker nodes with raw and unformatted block storage.
3. If you want to use [journal devices](https://docs.portworx.com/install-with-other/operate-and-maintain/performance-and-tuning/tuning/), choose from the following options. Journal devices allow Portworx to write logs directly to a local disk on your worker node.



* Attach an additional disk to at least 3 worker nodes to use for the journal. [Manually add](https://cloud.ibm.com/docs/containers?topic=containers-utilitiesmanual_block) a 3 GB block storage device to a worker node in your cluster and find the device path. To find the device path after attaching the disk, log in to your worker node and run lsblk to list the devices on that node.
* Select one of the block storage devices that you added earlier to use for the journal and find the device path. To find the device path after attaching the disk, log in to your worker node and run lsblk to list the devices on that node.
* Let Portworx automatically create a journal partition during installation.



4. If you want to use a specific device for the internal Portworx key value database (KVDB), choose from the following options.



* Attach an additional disk to at least 3 worker nodes to use for key value database (KVDB). [Manually add](https://cloud.ibm.com/docs/containers?topic=containers-utilitiesmanual_block) a 3 GB block storage device to at least 3 worker nodes in your cluster and find the device path. To find the device path after attaching the disk, log in to your worker node and run lsblk to list the devices on that node.
* Select one of the block storage devices that you added earlier and find the device path. To find the device path after attaching the disk, log in to your worker node and run lsblk to list the devices on that node.



5.
* [Firewall error](https://cloud.ibm.com/docs/containers?topic=containers-common_worker_nodes_issuesfirewall-error)
* [Hard reboot](https://cloud.ibm.com/docs/containers?topic=containers-common_worker_nodes_issueshard-reboot)
* [Instance can't be found](https://cloud.ibm.com/docs/containers?topic=containers-common_worker_nodes_issuesinstance-not-found)



[Troubleshooting worker nodes in Critical or NotReady state](https://cloud.ibm.com/docs/containers?topic=containers-ts-critical-notreadyts-critical-notready)



* [Check the common causes of worker node failures](https://cloud.ibm.com/docs/containers?topic=containers-ts-critical-notreadyts-critical-notready-common)
* [Troubleshooting steps](https://cloud.ibm.com/docs/containers?topic=containers-ts-critical-notreadyts-critical-notready-steps)



* [If one or some worker nodes are affected](https://cloud.ibm.com/docs/containers?topic=containers-ts-critical-notreadyts-critical-notready-steps-some)
* [If all worker nodes in a single zone, subnet, or VLAN are affected](https://cloud.ibm.com/docs/containers?topic=containers-ts-critical-notreadyts-critical-notready-steps-zone)
* [If all worker nodes in a cluster are affected](https://cloud.ibm.com/docs/containers?topic=containers-ts-critical-notreadyts-critical-notready-steps-all)
* [If worker nodes switch between normal and critical states](https://cloud.ibm.com/docs/containers?topic=containers-ts-critical-notreadyts-critical-notready-steps-switch)
Change log for worker node fix pack 1.16.9_1531, released 11 May 2020 

The following table shows the changes that are in the worker node fix pack 1.16.9_1531. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 1.16.9_1530

 Component Previous Current Description 

 Ubuntu 18.04 packages 4.15.0-96-generic 4.15.0-99-generic Updated worker node images with kernel and package updates for [CVE-2019-19768](https://nvd.nist.gov/vuln/detail/CVE-2019-19768), [CVE-2020-11884](https://nvd.nist.gov/vuln/detail/CVE-2020-11884), and [CVE-2020-12243](https://nvd.nist.gov/vuln/detail/CVE-2020-12243). 
 Ubuntu 16.04 packages 4.4.0-177-generic 4.4.0-178-generic Updated worker node images with package and kernel updates for [CVE-2019-19768](https://nvd.nist.gov/vuln/detail/CVE-2019-19768) and [CVE-2020-12243](https://nvd.nist.gov/vuln/detail/CVE-2020-12243). 







 Change log for worker node fix pack 1.16.9_1530, released 27 April 2020 

The following table shows the changes that are in the worker node fix pack 1.16.9_1530. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 1.16.8_1528

 Component Previous Current Description 

 containerd v1.3.3 v1.3.4 See the [containerd release notes](https://github.com/containerd/containerd/releases/tag/v1.3.4).
Complete the [prerequisite steps](https://cloud.ibm.com/docs/containers?topic=containers-updateworker-up-prereqs) and [set up a config map](https://cloud.ibm.com/docs/containers?topic=containers-updateworker_node) to control how your worker nodes are updated.
2. From the [IBM Cloud console](https://cloud.ibm.com/) menu ![Menu icon](https://cloud.ibm.com/docs-content/v1/content/4d497aff9e27b92fbbc2ee4e343a9ec4549cb1d7/icons/icon_hamburger.svg), click Kubernetes.
3. From the Clusters page, click your cluster.
4. From the Worker Nodes tab, select the checkbox for each worker node that you want to update. An action bar is displayed over the table header row.
5. From the action bar, click Update.



If you have Portworx installed in your cluster, you must restart the Portworx pods on updated worker nodes. For more information, see [Portworx limitations](https://cloud.ibm.com/docs/containers?topic=containers-storage_portworx_planportworx_limitations).







 Updating VPC worker nodes 

You notice that an update is available for your worker nodes in a [VPC infrastructure cluster](https://cloud.ibm.com/docs/containers?topic=containers-infrastructure_providers). What does that mean? As security updates and patches are put in place for the API server and other master components, you must be sure that the worker nodes remain in sync. You can make two types of updates: updating only the patch version, or updating the major.minor version with the patch version.

Applies to only VPC clusters. Have a classic cluster? See [Updating classic worker nodes](https://cloud.ibm.com/docs/containers?topic=containers-updateworker_node) instead.
Debugging worker nodes 

Virtual Private Cloud

Classic infrastructure

Review the options to debug your worker nodes and find the root causes for failures.



 Check worker node notifications and maintenance updates 

Check the IBM Cloud health and status dashboard for any notifications or maintenance updates that might be relevant to your worker nodes. These notifications or updates might help determine the cause of the worker node failures.



1.
Classic clusters

Check the [health dashboard](https://cloud.ibm.com/gen1/infrastructure/health-dashboard) for any IBM Cloud emergency maintenance notifications that might affect classic worker nodes in your account. Depending on the nature of the maintenance notification, you might need to reboot or reload your worker nodes.
2. Check the IBM Cloud [status dashboard](https://cloud.ibm.com/status) for any known problems that might affect your worker nodes or cluster. If any of the following components show an error status, that component might be the cause of your worker node disruptions.



* For all clusters, check the Kubernetes Service and Container Registry components.
* For VPC clusters, check the Virtual Private Cloud, Virtual Private Endpoint and Virtual Server for VPC components.
* For Classic clusters, check the Classic Infrastructure Provisioning and Virtual Servers components.









 Quick steps to resolve worker node issues 

If your worker node is not functioning as expected, you can follow these steps to update your cluster and command line tools or run diagnostic tests. If the issue persists, see [Debugging your worker node](https://cloud.ibm.com/docs/containers?topic=containers-debug_worker_nodesworker-debug-steps) for additional steps.



1. [Update your cluster and worker nodes to the latest version](https://cloud.ibm.com/docs/containers?topic=containers-updateupdate).
2. [Update your command line tools](https://cloud.ibm.com/docs/containers?topic=containers-cli-update).
3.