vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Debugging clusters 

Virtual Private Cloud

Classic infrastructure

Review the options to debug your clusters and find the root causes for failures.



1.  List your cluster and find the State of the cluster.

ibmcloud ks cluster ls
2.  Review the State of your cluster. If your cluster is in a Critical, Delete failed, or Warning state, or is stuck in the Pending state for a long time. For more information, see [cluster states](https://cloud.ibm.com/docs/containers?topic=containers-cluster-states-reference).
3.  Review the state of each worker node. For more information, see [worker node states](https://cloud.ibm.com/docs/containers?topic=containers-worker-node-state-reference).

ibmcloud ks worker ls -c CLUSTER
4.  Review the [common worker node issues](https://cloud.ibm.com/docs/containers?topic=containers-common_worker_nodes_issues).
5.  Start [debugging the worker nodes](https://cloud.ibm.com/docs/containers?topic=containers-debug_worker_nodes).
6.  Review the site map for additional [troubleshooting information](https://cloud.ibm.com/docs/containers?topic=containers-cs_sitemapsitemap_troubleshooting).
To protect against this failure, you must [set up a multizone cluster to spread worker nodes across three zones, or create multiple single zone clusters](https://cloud.ibm.com/docs/containers?topic=containers-ha_clustersha_clusters) in different zones.

Potential failure point 3: Cluster availability
: The Kubernetes master is the main component that keeps your cluster up and running. The master stores cluster resources and their configurations in the etcd database that serves as the single point of truth for your cluster. The Kubernetes API server is the main entry point for all cluster management requests from the worker nodes to the master, or when you want to interact with your cluster resources. To protect your cluster master from a zone failure, review the following docs. Create a cluster in a multizone location, which spreads the master across zones. Or, consider setting up a second cluster in another zone. For more information, see [Setting up highly available clusters](https://cloud.ibm.com/docs/containers?topic=containers-ha_clustersha_clusters).

Potential failure point 4: Zone availability
: A zone failure affects all physical compute hosts and NFS storage. Failures include power, cooling, networking, or storage outages, and natural disasters, like flooding, earthquakes, and hurricanes. To protect against a zone failure, you must have clusters in two different zones that are load balanced by an external load balancer. For more information, see [Setting up highly available clusters](https://cloud.ibm.com/docs/containers?topic=containers-ha_clustersha_clusters).

Potential failure point 5: Region availability
: Every region is set up with a highly available load balancer that is accessible from the region-specific API endpoint. The load balancer routes incoming and outgoing requests to clusters in the regional zones. The likelihood of a full regional failure is low. However, to account for this failure, you can set up multiple clusters in different regions and connect them by using an external load balancer. If an entire region fails, the cluster in the other region can take over the workload.
Planning your cluster for high availability 

Design your standard cluster for maximum availability and capacity for your app with IBM Cloud® Kubernetes Service.

Your users are less likely to experience downtime when you distribute your apps across multiple worker nodes, zones, and clusters. Built-in capabilities, like load balancing and isolation, increase resiliency against potential failures with hosts, networks, or apps. Review these potential cluster setups that are ordered with increasing degrees of availability.

Zoom

![High availability for clusters](https://cloud.ibm.com/docs-content/v1/content/4d497aff9e27b92fbbc2ee4e343a9ec4549cb1d7/containers/images/cs_cluster_ha_roadmap_multizone_public.png)

Figure 1. High availability for clusters



1. A [single zone cluster](https://cloud.ibm.com/docs/containers?topic=containers-ha_clusterssingle_zone) with multiple worker nodes in a worker pool.
2. A [multizone cluster](https://cloud.ibm.com/docs/containers?topic=containers-ha_clustersmz-clusters) that spreads worker nodes across zones within one region.
3. Clusters with public network connectivity: [Multiple clusters](https://cloud.ibm.com/docs/containers?topic=containers-ha_clustersmultiple-clusters-glb) that are set up across zones or regions and that are connected via a global load balancer.





 Single zone clusters 

Single zone clusters can be created in one of the supported [single zone locations](https://cloud.ibm.com/docs/containers?topic=containers-regions-and-zoneszones-sz). To improve availability for your app and to allow failover for the case that one worker node is not available in your cluster, add additional worker nodes to your single zone cluster.

VPC clusters are supported only in [multizone metro locations](https://cloud.ibm.com/docs/containers?topic=containers-regions-and-zoneszones-vpc).
<Timestamp>2012-09-27 12:46:08</Timestamp>
</Header>
<DataDictionary numberOfFields="5">
<DataField name="class" optype="categorical" dataType="string">
<Value value="Iris-setosa"/>
<Value value="Iris-versicolor"/>
<Value value="Iris-virginica"/>
</DataField>
<DataField name="sepal_length" optype="continuous" dataType="double">
<Interval closure="closedClosed" leftMargin="4.3" rightMargin="7.9"/>
</DataField>
<DataField name="sepal_width" optype="continuous" dataType="double">
<Interval closure="closedClosed" leftMargin="2" rightMargin="4.4"/>
</DataField>
<DataField name="petal_length" optype="continuous" dataType="double">
<Interval closure="closedClosed" leftMargin="1" rightMargin="6.91"/>
</DataField>
<DataField name="petal_width" optype="continuous" dataType="double">
<Interval closure="closedClosed" leftMargin="0.1" rightMargin="2.5"/>
</DataField>
</DataDictionary>
<TreeModel modelName="RPart_Model" functionName="classification" algorithmName="rpart" splitCharacteristic="binarySplit" missingValueStrategy="defaultChild">
<MiningSchema>
<MiningField name="class" usageType="predicted"/>
<MiningField name="sepal_length" usageType="supplementary"/>
<MiningField name="sepal_width" usageType="supplementary"/>
* [Registering a Secrets Manager instance when creating a cluster](https://cloud.ibm.com/docs/containers?topic=containers-secrets-mgrsecrets-mgr_cluster_create)









 Setting up storage 

[Planning for storage](https://cloud.ibm.com/docs/containers?topic=containers-storage-planstorage-plan)



* [Non-persistent storage](https://cloud.ibm.com/docs/containers?topic=containers-storage-planstorage-plan-non-persistent)
* [Persistent storage for single zone clusters](https://cloud.ibm.com/docs/containers?topic=containers-storage-planstorage-plan-single-zone)



* [Classic File Storage for Classic](https://cloud.ibm.com/docs/containers?topic=containers-storage-planstorage-plan-classic-file)
* [Classic Block Storage for Classic](https://cloud.ibm.com/docs/containers?topic=containers-storage-planstorage-plan-classic-block)
* [File Storage for VPC](https://cloud.ibm.com/docs/containers?topic=containers-storage-planstorage-plan-vpc-file)
* [Block Storage for VPC](https://cloud.ibm.com/docs/containers?topic=containers-storage-planstorage-plan-vpc-block)



* [Persistent storage options for multizone clusters](https://cloud.ibm.com/docs/containers?topic=containers-storage-planstorage-plan-multizone)



* [IBM Cloud Object Storage](https://cloud.ibm.com/docs/containers?topic=containers-storage-planstorage-plan-cos)
* [Portworx](https://cloud.ibm.com/docs/containers?topic=containers-storage-planstorage-plan-portworx)
* [Databases](https://cloud.ibm.com/docs/containers?topic=containers-storage-planstorage-plan-databases)
curl "https://$ACCOUNT.cloudant.com/_api/v2/monitoring/rate/status_code?cluster=myclustername&format=json"

See example results (abbreviated) from a rate/status_code monitoring request:

{
"start": 1436194902,
"end": 1436195202,
"target_responses": [
{
"target": "myclustername 2xx",
"datapoints":

null,
1436194910
],

36.0,
1436194920
],
...

0.0,
1436195200
]
]
},
{
"target": "myclustername 3xx",
"datapoints":

null,
1436194910
],

0.0,
1436194920
],
...

null,
1436195200
]
]
},
{
"target": "myclustername 4xx",
"datapoints":

null,
1436194910
],

0.0,
1436194920
],
...

0.0,
1436195200
]
]
},
{
"target": "myclustername 5xx",
"datapoints":

null,
1436194910
],

0.0,
1436194920
],
...

null,
1436195200
]
]
}
]
}
Show more





 rate/verb 

See an example of a rate/verb monitoring request:

curl "https://$ACCOUNT.cloudant.com/_api/v2/monitoring/rate/verb?cluster=myclustername&format=json"
Adding clusters to Cyber Recovery instances 

You can add clusters to Cyber Recovery instances to expand the compute and storage capacity. Within a cluster, you can manage VMware ESXi™ servers for better resource allocation and high availability.



 Before you add clusters to Cyber Recovery instances 



* Whenever possible, add clusters by using the IBM Cloud® for VMware Solutions console and not the VMware vSphere® Web Client. Changes that you make on the vSphere Web Client are not synchronized with the VMware Solutions console. If you want to add clusters to Cyber Recovery instances by using the vSphere Web Client, do so only for on-premises clusters or clusters that you don't manage in the VMware Solutions console.
* The number of clusters, hosts, and virtual machines (VMs) determines the maximum number of clusters that you can add. You must remain within the VMware® sizing guidelines and limits for your deployment. For more information, see [VMware Configuration Maximums](https://configmax.vmware.com/home).
* You can add a cluster while another cluster is being created or deleted.







 Cluster type 

Select the cluster type, either Workload cluster or Gateway cluster.





 System settings for workload clusters 

When you add a workload cluster to a Cyber Recovery instance, you must specify the following settings.



 Cluster name 

The cluster name is set to vcs-lj-workload_xx_ by default, where xx represents two randomly generated alphabetic characters.

You can also specify a new cluster name that meets the following requirements:



* Only lowercase alphabetic, numeric, and dash (-) characters are allowed.
* The cluster name must start with a lowercase alphabetic character.
* The cluster name must end with a lowercase alphabetic or numeric character.
* The maximum length of the cluster name is 30 characters.
* The cluster name must be unique within the vCenter Server instance.







 Licensing settings 

Review the following information and specify the licensing setting for the Cyber Recovery component in the cluster.



* For Business Partner users, the vSphere license (Enterprise Plus edition) is included and purchased on your behalf.
* [Overview of cluster encryption](https://cloud.ibm.com/docs/openshift?topic=openshift-encryptionencrypt_ov)
* [Understanding Key Management Service (KMS) providers](https://cloud.ibm.com/docs/openshift?topic=openshift-encryptionkms)



* [Available KMS providers](https://cloud.ibm.com/docs/openshift?topic=openshift-encryptionkms-providers)
* [Controlling encryption](https://cloud.ibm.com/docs/openshift?topic=openshift-encryptionkms-encrypt-control)
* [Features and limitations of KMS providers](https://cloud.ibm.com/docs/openshift?topic=openshift-encryptionkms-keyprotect-features)



* [Encrypting the Kubernetes secrets by using a KMS provider](https://cloud.ibm.com/docs/openshift?topic=openshift-encryptionkeyprotect)



* [Prerequisites](https://cloud.ibm.com/docs/openshift?topic=openshift-encryptionkms_prereqs)
* [Enabling KMS encryption for the cluster through the CLI](https://cloud.ibm.com/docs/openshift?topic=openshift-encryptionkms_cli)
* [Enabling KMS encryption for the cluster through the console](https://cloud.ibm.com/docs/openshift?topic=openshift-encryptionkms_ui)
* [Rotating the root key for your cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-encryptionkms_rotate)



* [Verifying secret encryption](https://cloud.ibm.com/docs/openshift?topic=openshift-encryptionverify_kms)
* [Context](https://cloud.ibm.com/docs/containers?topic=containers-cs_uc_retailuc_data-share_context)
* [Solution](https://cloud.ibm.com/docs/containers?topic=containers-cs_uc_retailuc_data-share_solution)
* [Results](https://cloud.ibm.com/docs/containers?topic=containers-cs_uc_retailuc_data-share_results)



* [Traditional grocer increases customer traffic and sales with digital insights](https://cloud.ibm.com/docs/containers?topic=containers-cs_uc_retailuc_grocer)



* [Context](https://cloud.ibm.com/docs/containers?topic=containers-cs_uc_retailuc_grocer_context)
* [Results](https://cloud.ibm.com/docs/containers?topic=containers-cs_uc_retailuc_grocer_results)





[Transportation use cases for IBM Cloud](https://cloud.ibm.com/docs/containers?topic=containers-cs_uc_transportcs_uc_transport)



* [Shipping company increases availability of worldwide systems for business partner ecosystem](https://cloud.ibm.com/docs/containers?topic=containers-cs_uc_transportuc_shipping)



* [Context](https://cloud.ibm.com/docs/containers?topic=containers-cs_uc_transportuc_shipping_context)
* [Solution](https://cloud.ibm.com/docs/containers?topic=containers-cs_uc_transportuc_shipping_solution)
* [Results](https://cloud.ibm.com/docs/containers?topic=containers-cs_uc_transportuc_shipping_results)



* [Airline delivers innovative Human Resources (HR) benefits site in under 3 weeks](https://cloud.ibm.com/docs/containers?topic=containers-cs_uc_transportuc_airline)
* [Configuration, subscription, cluster, cluster group, and resource](https://cloud.ibm.com/docs/satellite?topic=satellite-iamiam-resource-config)
* [Link](https://cloud.ibm.com/docs/satellite?topic=satellite-iamiam-resource-link)
* [Other services](https://cloud.ibm.com/docs/satellite?topic=satellite-iamiam-resource-services)



* [Platform and service roles for Red Hat OpenShift clusters](https://cloud.ibm.com/docs/satellite?topic=satellite-iamiam-roles-clusters)



* [Common use cases and roles in IBM Cloud](https://cloud.ibm.com/docs/satellite?topic=satellite-iamiam-roles-usecases)



[Assigning access with IBM Cloud IAM](https://cloud.ibm.com/docs/satellite?topic=satellite-iam-assign-accessiam-assign-access)



* [Access policies](https://cloud.ibm.com/docs/satellite?topic=satellite-iam-assign-accessiam-roles-policies)
* [Overview of the process to set up access to IBM Cloud Satellite in IBM Cloud IAM](https://cloud.ibm.com/docs/satellite?topic=satellite-iam-assign-accessiam-assign-overview)
* [Assigning access policy to access group by using the console](https://cloud.ibm.com/docs/satellite?topic=satellite-iam-assign-accessiam-assign-ui)
* [Assigning access policy to access group with the CLI](https://cloud.ibm.com/docs/satellite?topic=satellite-iam-assign-accessiam-assign-cli)
* [Checking user permissions](https://cloud.ibm.com/docs/satellite?topic=satellite-iam-assign-accesschecking-perms)
Additionally, the type of cluster and the number of worker pools that you have can impact the size of your cluster.



* Single zone clusters: [Create a cluster](https://cloud.ibm.com/docs/containers?topic=containers-clusters) with 1 worker node in the default worker pool.
* Multizone clusters: You must [create a cluster](https://cloud.ibm.com/docs/containers?topic=containers-clusters) with 1 worker node per zone in the worker pool. Later, you can [remove zones](https://cloud.ibm.com/docs/containers?topic=containers-kubernetes-service-clics_zone_rm) from the worker pool or [remove individual worker nodes](https://cloud.ibm.com/docs/containers?topic=containers-kubernetes-service-clics_worker_rm) so that your cluster size reduces to the minimum size of 1.
* Worker pools: For any type of cluster, each worker pool must always have at least 1 worker node. For the smallest size cluster possible, you can have only 1 worker pool.



Keep in mind that some services such as Ingress might require multiple worker nodes for high availability, and you might not be able to run these services or your apps in the smallest size cluster possible.





 Which Kubernetes versions does the service support? 

IBM Cloud Kubernetes Service concurrently supports multiple versions of Kubernetes. When a new version (n) is released, versions up to 2 behind (n-2) are supported. Versions more than 2 behind the latest (n-3) are first deprecated and then unsupported.

For more information about supported versions and update actions that you must take to move from one version to another, see [Kubernetes version information](https://cloud.ibm.com/docs/containers?topic=containers-cs_versions)





 Which worker node operating systems does the service support? 

For a list of supported worker node operated systems by cluster version, see [Kubernetes version information](https://cloud.ibm.com/docs/containers?topic=containers-cs_versions).





 Where is the service available?
For example, my-static-pv.
labels:
failure-domain.beta.kubernetes.io/region: "<region>" Example us-east.
failure-domain.beta.kubernetes.io/zone: "<zone>" Example: wdc04. See /docs/containers?topic=containers-regions-and-zoneszones-sz
spec:
capacity:
storage: "<storage>"
accessModes:
- ReadWriteOnce
flexVolume:
driver: "ibm/ibmc-block"
fsType: "<fs_type>" Enter ext or xfs
options:
"Lun": "<Lun_ID>"
"TargetPortal": "<TargetPortal>"
"VolumeID": "<VolumeID>"
"volumeName": "block-storage-pv" Enter the same value as your PV name from metadata.name
Show more

name
: Give your PV a name. For example, block-storage-pv. Note that you must also enter this value in spec.FlexVolume.options as the volumeName.

labels
: Enter the region and the zone that you retrieved earlier. You must have at least one worker node in the same region and zone as your persistent storage to mount the storage in your cluster. To retrieve your volume details, run ibmcloud sl block volume-list to get the volume ID, then run ibmcloud sl block volume-detail <volume_ID> to get the details of your volume.

region
: Enter the region where your block storage is located. Note that your cluster and block storage must be in the same region. To find your cluster location, run ibmcloud oc cluster ls. For more information about the available regions and zones, see [regions and zones](https://cloud.ibm.com/docs/openshift?topic=openshift-regions-and-zoneszones-sz). For example, us-east.

zone
: Enter the zone where your storage volume is located.
containers-kubernetes.cluster-worker-pool.get The details of a worker pool in the cluster are returned. 
 containers-kubernetes.cluster-worker-pool.list The worker pools for a cluster are listed. 
 containers-kubernetes.cluster-worker-pool.rebalance A worker pool is rebalanced. 
 containers-kubernetes.cluster-worker-pool.resize A worker pool is resized, to add or decrease the number of worker nodes in the pool. 
 containers-kubernetes.cluster-worker-pool-autoscale.disable Autoscaling the worker pool is disabled. 
 containers-kubernetes.cluster-worker-pool-autoscale.enable Autoscaling the worker pool is enabled. 
 containers-kubernetes.cluster-worker-pool-label.set Kubernetes labels for a worker pool are set. Existing and future worker nodes in the worker pool inherit the label. 
 containers-kubernetes.cluster-worker-pool-taint.set Kubernetes taints for a worker pool are set. Existing and future worker nodes in the worker pool inherit the taint. 
 containers-kubernetes.cluster-worker-pool-zone.create A zone is added to a worker pool. 
 containers-kubernetes.cluster-worker-pool-zone.delete A zone is deleted from a worker pool. 
 containers-kubernetes.cluster-worker-pool-zone.get The details of a zone that a worker pool spans in the cluster are returned as part of cluster autoscaler operations. 
 containers-kubernetes.cluster-worker-pool-zone.list The worker pools for a cluster in a particular zone are listed as part of cluster autoscaler operations. 
 containers-kubernetes.cluster-worker-pool-zone.resize A worker node is added to or removed from a zone that the worker pool spans. 
 containers-kubernetes.cluster-worker-pool-zone-network.add The networking data, such as public and private VLAN data, is added for a zone that the worker pool spans.
You can't create a cluster with a tag in the CLI.
* Add tags to an existing cluster in the console or CLI.





 Adding tags to clusters with the console 



1. Log in to the [IBM Cloud clusters console](https://cloud.ibm.com/kubernetes/clusters).
2. Select a cluster with existing tags.
3. Next to the cluster name and status, click the Edit tags pencil icon. If your cluster does not have any existing tags, you don't have an Edit tags pencil icon. Instead, use the [resource list](https://cloud.ibm.com/docs/account?topic=account-tag) or CLI.
4. Enter the tag that you want to add to your cluster. To assign a key-value pair, use a colon such as costctr:1234.







 Adding tags to clusters with the CLI 



1. Log in to the [IBM Cloud CLI](https://cloud.ibm.com/docs/cli/reference/ibmcloud?topic=cli-ibmcloud_cliibmcloud_login).

ibmcloud login [--sso]
2. [Tag your cluster](https://cloud.ibm.com/docs/cli/reference/ibmcloud?topic=cli-ibmcloud_commands_resourceibmcloud_resource_tag_attach). Replace the --resource-name with the name of your cluster. To list available clusters, run ibmcloud oc cluster ls. If you want to check your existing tags so as not to duplicate any, run ibmcloud resource tags.

ibmcloud resource tag-attach --resource-name <cluster_name> --tag-names <tag1,tag2>

If you have more than one resource of the same name in your IBM Cloud account, the error message lists the resource CRNs and details, and instructs you to try again with the --resource-idoption.









 Adding labels to existing worker pools
* If you are running Kubernetes 1.20 or earlier , don't use the following reserved ranges: 172.16.0.0/16, 172.18.0.0/16, 172.19.0.0/16, and 172.20.0.0/16.



3. Check whether you have a public gateway in the zones where you want to create a cluster. Within one VPC, you can create only one public gateway per zone, but that public gateway can be attached to multiple subnets within the zone.

ibmcloud is public-gateways

Example output

ID Name VPC Zone Floating IP Created Status Resource group
26426426-6065-4716-a90b-ac7ed7917c63 test-pgw testvpc(36c8f522-.) us-south-1 169.xx.xxx.xxx(26466378-.) 2019-09-20T16:27:32-05:00 available -
2ba2ba2b-fffa-4b0c-bdca-7970f09f9b8a pgw-73b62bc0-b53a-11e9-9838-f3f4efa02374 team3(ff537d43-.) us-south-2 169.xx.xxx.xxx(2ba9a280-.) 2019-08-02T10:30:29-05:00 available -



* If you already have a public gateway in each zone, note the IDs of the public gateways.
* If you don't have a public gateway in each zone, create a public gateway. Consider naming the public gateway in the format <cluster>-<zone>-gateway. In the output, note the public gateway's ID.



ibmcloud is public-gateway-create <gateway_name> <VPC_ID> <zone>

Example output

ID 26466378-6065-4716-a90b-ac7ed7917c63
Name mycluster-us-south-1-gateway