vania-janet/multiturn-rag-retrieval-data huggingface dataset:
If all worker nodes in a cluster are affected 

If all the worker nodes in your cluster show Criticalor NotReady at the same time, there might be a problem with either the cluster apiserver or the networking path between the workers and the apiserver. Follow these troubleshooting steps to determine the cause and resolve the issue.

Some steps are specific to a specialized area, such as networking or automation. Consult with the relevant administrator or team in your organization before completing these steps.



1. Check if there were any recent changes to your cluster, environment, or account that might impact your worker nodes. If so, revert the changes and then check the worker node status to determine if the changes caused the issue.



* For classic clusters, check any firewall or gateway, such as Virtual Router Appliance, Vyatta, or Juniper that manages traffic for cluster workers. Look for changes or issues that might drop or redirect traffic from cluster workers.
* For VPC clusters, check if any changes were made to the default security group and ACLs on the VPC or the worker nodes. If any modifications were made, ensure that you are allowing all necessary traffic from the cluster worker nodes to the cluster master, container registry, and other critical services. For more information, see [Controlling Traffic with VPC Security Groups](https://cloud.ibm.com/docs/containers?topic=containers-vpc-security-group&interface=ui) and [Controlling traffic with ACLs](https://cloud.ibm.com/docs/containers?topic=containers-vpc-acls&interface=ui).
* For VPC clusters, check any custom routing rules for changes that might be blocking traffic from the cluster apiserver.
* Check any Calico or Kubernetes network policies that are applied to the cluster and make sure that they do not block traffic from the worker node to the cluster apiservice, container registry, or other critical services.



2. Check if the applications, security, or monitoring components in your cluster are overloading the cluster apiserver with requests, which may cause disruptions for your worker nodes.
3. If you recently added any components to your cluster, remove them.
Although the worker nodes might be normal, other infrastructure resources, such as [networking](https://cloud.ibm.com/docs/containers?topic=containers-coredns_lameduck) and [storage](https://cloud.ibm.com/docs/containers?topic=containers-debug_storage_file), might still need attention. If you just created the cluster, some parts of the cluster that are used by other services such as Ingress secrets or registry image pull secrets, might still be in process.





 Pending 

Review the following description of the Pending cluster state. To view the state of your cluster, run ibmcloud ks cluster get --cluster <cluster_name_or_ID>.

The Kubernetes master is deployed. The worker nodes are provisioning and aren't available in the cluster yet. You can access the cluster, but you can't deploy apps to the cluster.





 Requested 

Review the following description of the Requested cluster state. To view the state of your cluster, run ibmcloud ks cluster get --cluster <cluster_name_or_ID>.

A request to create the cluster and order the infrastructure for the Kubernetes master and worker nodes is sent. When the deployment of the cluster starts, the cluster state changes to Deploying. If your cluster is stuck in the Requested state for a long time, open an [IBM Cloud support case](https://cloud.ibm.com/docs/containers?topic=containers-get-help).





 Updating 

Review the following description of the Updating cluster state. To view the state of your cluster, run ibmcloud ks cluster get --cluster <cluster_name_or_ID>.

The Kubernetes API server that runs in your Kubernetes master is being updated to a new Kubernetes API version. During the update, you can still access and change the cluster. However, you cannot initiate concurrent master operations, such as enabling API server auditing. Worker nodes, apps, and resources that the user deployed aren't modified and continue to run. Wait for the update to complete to review the health of your cluster.





 Unsupported
To retrieve these values, run oc get nodes -o jsonpath="{.items[].metadata.labels.ibm-cloud.kubernetes.io/os}{'n'}". Store the operating system of the worker nodes in an environment variable. For Red Hat OpenShift on IBM Cloud clusters, run SET WORKER_OS="redhat" and SET PLATFORM="openshift".

REGION
: The region of the worker nodes. To retrieve this value, run oc get nodes -o yaml | grep 'ibm-cloud.kubernetes.io/region'. Store the region of the worker nodes in an environment variable by running SET REGION="< region>".



 Updating the IBM Cloud Object Storage plug-in 

You can upgrade the existing IBM Cloud Object Storage plug-in to the most recent version.



1. Get the name of your IBM Cloud Object Storage plug-in Helm release and the version of the plug-in in your cluster.

helm ls -A | grep object

Example output

NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
<release_name> <namespace> 1 2020-02-13 16:05:58.599679 -0500 EST deployed ibm-object-storage-plugin-1.1.2 1.1.2
2. Update the IBM Cloud Helm repo to retrieve the most recent version of all Helm charts in this repo.

helm repo update
3. Update the IBM Cloud Object Storage ibmc Helm plug-in to the most recent version.

helm ibmc --update
4. Install the most recent version of the ibm-object-storage-plugin for your operating system.

Example helm ibmc upgrade command for OS X and Linux.

helm ibmc upgrade ibm-object-storage-plugin ibm-helm/ibm-object-storage-plugin --set license=true [--set bucketAccessPolicy=false]

Example helm upgrade command for Windows.
{
"workspace_id": "us-south.workspace.basic-cos-storage.535f45d6",
"workspace_name": "basic-cos-storage",
"job_id": "a9c02050ed5b8d06e524757016beebae",
"status_code": "job_finished",
"updated_at": "2022-11-23T14:37:27.596469934Z"
}
],
"updated_at": "2022-11-23T14:39:03.495851031Z"
}
},
"log_summary": {
"log_start_at": "0001-01-01T00:00:00Z",
"log_analyzed_till": "0001-01-01T00:00:00Z",
"repo_download_job": {},
"workspace_job": {},
"flow_job": {
"workitems_completed": 2,
"workitems": [
{
"workspace_id": "us-south.workspace.basic-resource-group.483b1ea2",
"job_id": "a1dc8ad75a0e779796c0d2a97e12e9b1",
"log_url": "https://schematics.cloud.ibm.com/v1/workspaces/us-south.workspace.basic-resource-group.483b1ea2/runtime_data/IBM-DefaultResourceGroup-f3ffa820-efbb-4f/log_store/actions/a1dc8ad75a0e779796c0d2a97e12e9b1"
},
{
"workspace_id": "us-south.workspace.basic-cos-storage.535f45d6",
Change log for worker node fix pack 1.27.2_1528, released 19 June 2023 

The following table shows the changes that are in the worker node fix pack 1.27.2_1528. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 1.27.2_1527

 Component Previous Current Description 

 Ubuntu 18.04 packages N/A N/A Worker node package updates for [CVE-2020-11080](https://nvd.nist.gov/vuln/detail/CVE-2020-11080),[CVE-2023-24329](https://nvd.nist.gov/vuln/detail/CVE-2023-24329),[CVE-2023-2603](https://nvd.nist.gov/vuln/detail/CVE-2023-2603),[CVE-2023-2609](https://nvd.nist.gov/vuln/detail/CVE-2023-2609),[CVE-2023-2610](https://nvd.nist.gov/vuln/detail/CVE-2023-2610),[CVE-2023-32681](https://nvd.nist.gov/vuln/detail/CVE-2023-32681)
You can monitor the reloading process in the Status column of your CLI output. If your worker node is stuck in this state for a long time, check whether a problem occurred during the reloading.





 Reloading_failed state 

You can view the current worker node state by running the ibmcloud oc worker ls --cluster <cluster_name_or_ID> command and locating the State and Status fields.

A Reloading_failed state means that your worker node could not be reloaded. List the details for the worker node to find the details for the failure by running ibmcloud oc worker get --cluster <cluster_name_or_id> --worker <worker_node_id>.





 Reload_pending state 

You can view the current worker node state by running the ibmcloud oc worker ls --cluster <cluster_name_or_ID> command and locating the State and Status fields.

You requested to reload or to update the Kubernetes version of your worker node. When reloading begins, the state changes to Reloading.





 Unknown state 

You can view the current worker node state by running the ibmcloud oc worker ls --cluster <cluster_name_or_ID> command and locating the State and Status fields.

An Unknown state means that the Kubernetes master is not reachable for one of the following reasons:



* You requested an update of your Kubernetes master. The state of the worker node can't be retrieved during the update. If the worker node remains in this state for an extended period of time even after the Kubernetes master is successfully updated, try to [reload the worker node](https://cloud.ibm.com/docs/openshift?topic=openshift-kubernetes-service-clics_worker_reload).
* You might have another firewall that is protecting your worker nodes, or changed firewall settings recently. Red Hat OpenShift on IBM Cloud requires certain IP addresses and ports to be opened to allow communication from the worker node to the Kubernetes master and vice versa.
Change log for worker node fix pack 1.18.19_1554, released 7 June 2021 

The following table shows the changes that are in the worker node fix pack 1.18.19_1554. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 1.18.19_1553

 Component Previous Current Description 

 HA proxy 26c5cc 700dc6 Updated the image for [CVE-2021-27219](https://nvd.nist.gov/vuln/detail/CVE-2021-27219). 
 TCP keepalive optimization for VPC N/A N/A Set the net.ipv4.tcp_keepalive_time setting to 180 seconds for compatibility with VPC gateways.
Change log for worker node fix pack 1.25.10_1548, released 19 June 2023 

The following table shows the changes that are in the worker node fix pack 1.25.10_1548. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 1.25.10_1547

 Component Previous Current Description 

 Ubuntu 18.04 packages N/A N/A Worker node package updates for [CVE-2020-11080](https://nvd.nist.gov/vuln/detail/CVE-2020-11080),[CVE-2023-24329](https://nvd.nist.gov/vuln/detail/CVE-2023-24329),[CVE-2023-2603](https://nvd.nist.gov/vuln/detail/CVE-2023-2603),[CVE-2023-2609](https://nvd.nist.gov/vuln/detail/CVE-2023-2609),[CVE-2023-2610](https://nvd.nist.gov/vuln/detail/CVE-2023-2610),[CVE-2023-32681](https://nvd.nist.gov/vuln/detail/CVE-2023-32681)
After you create the cluster, you can get the endpoint by running ibmcloud ks cluster get --cluster <cluster_name_or_ID>.

--workers WORKER
: Optional: Specify the number of worker nodes to include in the cluster. If you don't specify this option, a cluster with the minimum value of 1 is created. For more information, see [What is the smallest size cluster that I can make?](https://cloud.ibm.com/docs/containers?topic=containers-faqssmallest_cluster). If you create a cluster with only one worker node per zone, you might experience issues with Ingress. For high availability, create a cluster with at least two workers per zone. Every worker node is assigned a unique worker node ID and domain name that must not be manually changed after the cluster is created. Changing the ID or domain name prevents the Kubernetes master from managing your cluster.

--disable-disk-encrypt
: Worker nodes feature AES 256-bit disk encryption by default; [learn more](https://cloud.ibm.com/docs/containers?topic=containers-securityencrypted_disk). To disable encryption, include this option.

--pod-subnet SUBNET
: All pods that are deployed to a worker node are assigned a private IP address in the 172.30.0.0/16 range by default. If you plan to connect your cluster to on-premises networks through IBM Cloud® Direct Link or a VPN service, you can avoid subnet conflicts by specifying a custom subnet CIDR that provides the private IP addresses for your pods.
: When you choose a subnet size, consider the size of the cluster that you plan to create and the number of worker nodes that you might add in the future. The subnet must have a CIDR of at least /23, which provides enough pod IPs for a maximum of four worker nodes in a cluster. For larger clusters, use /22 to have enough pod IP addresses for eight worker nodes, /21 to have enough pod IP addresses for 16 worker nodes, and so on.
: The subnet that you choose must be within one of the following ranges:
For example, to verify that the app pod deployed to a worker node on a specific VLAN, view the VLAN that the worker node is on by running ibmcloud oc worker get --cluster <cluster_name_or_ID> --worker <worker_ID>.
4. In the output, verify that the worker node with the private IP address that you identified in the previous step is deployed in this worker pool.









 Deploying an app on a GPU machine 

If you have a [GPU machine type](https://cloud.ibm.com/docs/openshift?topic=openshift-planning_worker_nodesplanning_worker_nodes), you can accelerate the processing time required for compute intensive workloads such as AI, machine learning, inferencing and more.

IBM Cloud VPC worker nodes with GPUs are available for allowlisted accounts only. To request that your account be allowlisted, see [Requesting access to allowlisted features](https://cloud.ibm.com/docs/openshift?topic=openshift-get-help). Be sure to include the data centers, the VPC infrastructure profile, and the number of workers that you want use. For example 12 worker nodes in us-east-1 of VPC profile gx2-16x128xv100.

In the following steps, you learn how to deploy workloads that require the GPU. You can also deploy apps that don't need to process their workloads across both the GPU and CPU.

In the following steps, you learn how to deploy workloads that require the GPU. You can also deploy apps that don't need to process their workloads across both the GPU and CPU. After, you might find it useful to play around with mathematically intensive workloads such as the [TensorFlow](https://www.tensorflow.org/) machine learning framework with [this Kubernetes demo](https://github.com/pachyderm/pachyderm/tree/master/examples/ml/tensorflow).

Before you begin
Review the requirements to [install Portworx](https://docs.portworx.com/install-portworx/prerequisites/).

For production environments, choose one of the [SDS worker node flavors](https://cloud.ibm.com/docs/containers?topic=containers-planning_worker_nodessds-table) for best performance.





 How can I make sure that my data is stored highly available? 

You need at least three worker nodes in your Portworx cluster so that Portworx can replicate your data across nodes. By replicating your data across worker nodes, Portworx can ensure that your stateful app can be rescheduled to a different worker node in case of a failure without losing data. For even higher availability, use a [multizone cluster](https://cloud.ibm.com/docs/containers?topic=containers-ha_clustersmz-clusters) and replicate your volumes across worker nodes in 3 or more zones.





 What volume topology offers the best performance for my pods? 

One of the biggest challenges when you run stateful apps in a cluster is to make sure that your container can be rescheduled onto another host if the container or the entire host fails. In Docker, when a container must be rescheduled onto a different host, the volume does not move to the new host. Portworx can be configured to run hyper-converged to ensure that your compute resources and the storage are always placed onto the same worker node. When your app must be rescheduled, Portworx moves your app to a worker node where one of your volume replicas resides to ensure local-disk access speed and best performance for your stateful app. Running hyper-converged offers the best performance for your pods, but requires storage to be available on all worker nodes in your cluster.

You can also choose to use only a subset of worker nodes for your Portworx storage layer. For example, you might have a worker pool with SDS worker nodes that come with local raw block storage, and another worker pool with virtual worker nodes that don't come with local storage.
Run the following command to list all of the agents that are registered on the cluster:

kubectl get workeragents

The following information is listed for each agent:



* Name: The name that is registered for this worker.
* ServiceID: The serviceId of this worker.
* Agent: Indicates whether the worker agent is operating correctly on the cluster by specifying OK, or displaying an error message.
* Registered: The registration state of the worker, such as Succeeded or Unregistered.
* Version: The version of this worker for the current version that is used by the framework, such as OK, Outdated, or Unsupported.
* Auth: The validity of the API key that is used by the worker, such as OK or Error.
* Constrained: Indicates whether any cluster issues exist that affect the worker by specifying OK, or displaying an error message.
* Paused: Indicates whether the agent is actively searching for new tasks by specifying true or false.





To verify the overall status of a registered private worker pool, check the Overview page of the Delivery Pipeline private worker from the toolchain where it is used.



1. From the IBM Cloud console, click the menu icon ![hamburger icon](https://cloud.ibm.com/docs-content/v1/content/562806c0b18119f4d21cb66f5ae6d051634b4a16/ContinuousDelivery/images/icon_hamburger.svg) and select DevOps. On the Toolchains page, click a toolchain to open its Overview page. Alternatively, on your app's Overview page, on the Continuous delivery card, click View toolchain.
2. On the Delivery pipelines card, click the private workers tool integration for which you want to view a list of all of the registered workers. This worker pool contains one or more private workers that are installed on different clusters, but share a ServiceId. When work is assigned to a private worker pool, any of the private workers that are listed in the pool are eligible to run the workload.

The following information is provided for each worker that is registered in this pool:



* Name: The name of the registered worker.
Note that to attach block storage to your cluster, you must have a worker node available in the same zone as the volume that you want to attach. To find the zones of your worker nodes, run ibmcloud ks worker ls -c <cluster>. For example, wdc04.

storage
: Enter the storage size of the existing block storage volume that you want to attach to your cluster. The storage size must be written in gigabytes, for example, 20Gi (20 GB) or 1000Gi (1 TB). To retrieve your volume details, run ibmcloud sl block volume-list to get the volume ID, and then run ibmcloud sl block volume-detail <volume_ID> to get the details of your volume.

fsType
: Enter the file system type that is configured for your existing block storage. Choose between ext4 or xfs. If you don't specify this option, the PV defaults to ext4. When the wrong fsType is defined, then the PV creation succeeds, but the mounting of the PV to a pod fails. To retrieve your volume details, run ibmcloud sl block volume-list to get the volume ID, then run ibmcloud sl block volume-detail <volume_ID> to get the details of your volume.

Lun
: Enter the LUN ID of your block storage volume. To retrieve your volume details, run ibmcloud sl block volume-list to get the volume ID, then run ibmcloud sl block volume-detail <volume_ID> to get the details of your volume.

TargetPortal
: Enter the IP address of your block storage. To retrieve the TargetPortal parameter, run ibmcloud sl block volume-list to get the volume ID, then run ibmcloud sl block volume-detail <volume_ID> and make a note of the Target IP in the output.

VolumeId
: Enter the ID of your block storage. To retrieve your volume details, run ibmcloud sl block volume-list.

volumeName
: Enter the same value as your PV name. For example, block-storage-pv.
5. Create the PV in your cluster.
Nodes: 2 node(s) with storage (2 online), 1 node(s) without storage (1 online)
IP ID StorageNode Used Capacity Status StorageStatus Version Kernel OS
10.184.58.11 10.184.58.11 Yes 3.0 GiB 20 GiB Online Up 1.5.0.0-bc1c580 4.4.0-133-generic Ubuntu 18.04.5 LTS
10.176.48.67 10.176.48.67 Yes 3.0 GiB 20 GiB Online Up (This node) 1.5.0.0-bc1c580 4.4.0-133-generic Ubuntu 18.04.5 LTS
10.176.48.83 10.176.48.83 No 0 B 0 B Online No Storage 1.5.0.0-bc1c580 4.4.0-133-generic Ubuntu 18.04.5 LTS
Global Storage Pool
Total Used : 6.0 GiB
Total Capacity : 40 GiB
3. Verify that all worker nodes that you wanted to include in your Portworx storage layer are included by reviewing the StorageNode column in the Cluster Summary section of your CLI output. Worker nodes that are in the storage layer are displayed with Yes in the StorageNode column.

Because Portworx runs as a DaemonSet in your cluster, existing worker nodes are automatically inspected for raw block storage and added to the Portworx data layer when you deploy Portworx. If you add worker nodes to your cluster and add raw block storage to those workers, restart the Portworx pods on the new worker nodes so that your storage volumes are detected by the DaemonSet.
4. Verify that each storage node is listed with the correct amount of raw block storage by reviewing the Capacity column in the Cluster Summary section of your CLI output.
5. Review the Portworx I/O classification that was assigned to the disks that are part of the Portworx cluster. During the setup of your Portworx cluster, every disk is inspected to determine the performance profile of the device.
For example, check the size requirements for [Istio](https://cloud.ibm.com/docs/containers?topic=containers-istioistio_install).<br> * Worker nodes in your cluster are not in a Normal state. [Review the worker nodes' state and status](https://cloud.ibm.com/docs/containers?topic=containers-debug_worker_nodes).<br><br><br> 
 <br><br> * <br><br><br> Enabling The add-on is currently deploying to the cluster. Note that the add-on might take up to 15 minutes to install. 
 H1512 Addon daemonset may not be available on all Ready nodes. For the static route add-on: The static route operator DaemonSet is not available on any worker nodes, which prevents you from applying static route resources. Your worker nodes can't run the static route operator DaemonSet for the following reasons:<br><br><br><br> * One or more worker nodes reached their [resource limits](https://cloud.ibm.com/docs/containers?topic=containers-debug_worker_nodes).<br> * One or more worker nodes are running the [maximum number of pods per worker node](https://cloud.ibm.com/docs/containers?topic=containers-limitationsclassic_limits).<br><br><br> 



Still having trouble with the Istio add-on? Check out these [troubleshooting topics](https://cloud.ibm.com/docs/containers?topic=containers-istioistio-ts) to debug your Istio deployment.