vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Adding static routes to worker nodes 

Create static routes on your worker nodes by enabling the static routes add-on for IBM Cloud® Kubernetes Service clusters.



 About static routes 

When you configure your cluster network to communicate with other networks over the private network, you might need to add custom static routes on your worker nodes. Static routes allow worker nodes to successfully re-route response packets from your cluster to an IP address in on-premises data center through a VPN or a gateway appliance.

For example, you might use a VPN to connect your cluster to an on-premises data center over the private network. Additionally, your private VPN connection preserves the source IP address in requests between the cluster and the data center.

When an on-premises service sends a request to an app in your cluster, the worker node that your app pod is on drops the response due to reverse path filtering (RPF). RPF is a Linux kernel feature that drops any responses to IP addresses that are not listed in the worker node's routing tables. The response is dropped because the on-premises IP address that was preserved in the request is not routable by the worker node.

In this case, you can create a static route so that when a source IP address from a request is from one of your on-premises subnets, the response is sent to the VPN gateway IP address instead. By adding this static route to the worker nodes' routing tables, you ensure the following:



* Response packets are not dropped due to RPF, because a routing rule that points to the on-premises IP address exists.
* Response packets are successfully routed first through the VPN gateway IP address, and then re-routed to your on-premises IP address.



The static route cluster add-on can be used to apply and manage static routes only. You are responsible for configuring and managing your own VPN, gateway appliance, or IBM Cloud® Direct Link connection.





 Enabling the static route add-on 

To get started with static routes in IBM Cloud Kubernetes Service, enable the static route add-on.

Before you begin
If you delete the worker nodes or add worker nodes to the cluster, you must update your allowlist accordingly.

ibmcloud oc worker ls --cluster <cluster_name_or_ID>



2. Add the subnet CIDRs or individual worker node IP addresses to your service's allowlist or your on-premises allowlist for outbound traffic.
3. Repeat these steps for each cluster that you want to allow traffic to or from.









 Updating IAM allowlists for Kubernetes Service IP addresses 

By default, all IP addresses can be used to log in to the IBM Cloud console and perform actions to manage your cluster, such as creating, updating, deleting or viewing credentials. In the IBM Cloud Identity and Access Management (IAM) console, you can [create an allowlist by specifying which IP addresses have access](https://cloud.ibm.com/docs/account?topic=account-ips), and all other IP addresses are restricted.

In your allowlist, you must also add the CIDRs of the Red Hat OpenShift on IBM Cloud control plane for the zones in the region where your cluster is located so that Red Hat OpenShift on IBM Cloud can create or access the following components:



* Ingress ALBs
* VPC LoadBalancers
* VPC security groups
* VPE gateways
* Red Hat OpenShift web console (requires all control plane IP addresses)



Before you begin, the following steps require you to change the IAM allowlist for the user whose credentials are used for the cluster's region and resource group infrastructure permissions. If you are the credentials owner, you can change your own IAM allowlist settings. If you are not the credentials owner, but you are assigned the Editor or Administrator IBM Cloud IAM platform access role for the [User Management service](https://cloud.ibm.com/docs/account?topic=account-account-services), you can update the restricted IP addresses for the credentials owner.



1. Identify what user credentials are used for the cluster's region and resource group infrastructure permissions.



1. Check the API key for a region and resource group of the cluster.
Classic clusters: Why does source IP preservation fail when using tainted nodes? 

Classic infrastructure

 What’s happening 

In a classic cluster, you enabled source IP preservation for a [version 1.0 load balancer](https://cloud.ibm.com/docs/openshift?topic=openshift-loadbalancerlb_source_ip) service by changing externalTrafficPolicy to Local in the service's configuration file.

However, no traffic reaches the back-end service for your app.

 Why it’s happening 

When you enable source IP preservation for load balancer services, the source IP address of the client request is preserved.

The service forwards traffic to app pods on the same worker node only to ensure that the request packet's IP address isn't changed. Typically, load balancer service pods are deployed to the same worker nodes that the app pods are deployed to. However, some situations exist where the service pods and app pods might not be scheduled onto the same worker node. If you use [Kubernetes taints](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) on worker nodes, any pods that don't have a taint toleration are prevented from running on the tainted worker nodes. Source IP preservation might not be working based on the type of taint you used:



* Edge node taints: You [added the dedicated=edge label](https://cloud.ibm.com/docs/openshift?topic=openshift-edgeedge_nodes) to two or more worker nodes on each public VLAN in your cluster to ensure that load balancer pods deploy to those worker nodes only. Then, you also [tainted those edge nodes](https://cloud.ibm.com/docs/openshift?topic=openshift-edgeedge_workloads) to prevent any other workloads from running on edge nodes. However, you didn't add an edge node affinity rule and toleration to your app deployment. Your app pods can't be scheduled on the same tainted nodes as the service pods, and no traffic reaches the back-end service for your app.
Classic clusters: Why does source IP preservation fail when using tainted nodes? 

Classic infrastructure

 What’s happening 

In a classic cluster, you enabled source IP preservation for a [version 1.0 load balancer](https://cloud.ibm.com/docs/containers?topic=containers-loadbalancerlb_source_ip) service by changing externalTrafficPolicy to Local in the service's configuration file.

However, no traffic reaches the back-end service for your app.

 Why it’s happening 

When you enable source IP preservation for load balancer services, the source IP address of the client request is preserved.

The service forwards traffic to app pods on the same worker node only to ensure that the request packet's IP address isn't changed. Typically, load balancer service pods are deployed to the same worker nodes that the app pods are deployed to. However, some situations exist where the service pods and app pods might not be scheduled onto the same worker node. If you use [Kubernetes taints](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) on worker nodes, any pods that don't have a taint toleration are prevented from running on the tainted worker nodes. Source IP preservation might not be working based on the type of taint you used:



* Edge node taints: You [added the dedicated=edge label](https://cloud.ibm.com/docs/containers?topic=containers-edgeedge_nodes) to two or more worker nodes on each public VLAN in your cluster to ensure that load balancer pods deploy to those worker nodes only. Then, you also [tainted those edge nodes](https://cloud.ibm.com/docs/containers?topic=containers-edgeedge_workloads) to prevent any other workloads from running on edge nodes. However, you didn't add an edge node affinity rule and toleration to your app deployment. Your app pods can't be scheduled on the same tainted nodes as the service pods, and no traffic reaches the back-end service for your app.
Because Portworx runs as a DaemonSet in your cluster, existing worker nodes are automatically inspected for raw block storage and added to the Portworx data layer when you deploy Portworx. If you add or update worker nodes to your cluster and add raw block storage to those workers, restart the Portworx pods on the new or updated worker nodes so that your storage volumes are detected by the DaemonSet. 
 VPC clusters Storage volume reattachment required when updating worker nodes. When you update a worker node in a VPC cluster, the worker node is removed from your cluster and replaced with a new worker node. If Portworx volumes are attached to the worker node that is replaced, you must attach the volumes to the new worker node. You can attach storage volumes with the [API](https://cloud.ibm.com/docs/containers?topic=containers-utilitiesvpc_api_attach) or the [CLI](https://cloud.ibm.com/docs/containers?topic=containers-kubernetes-service-clics_storage_att_cr). Note this limitation does not apply to Portworx deployments that are using cloud drives. 
 The Portworx experimental InitializerConfiguration feature is not supported. IBM Cloud Kubernetes Service does not support the [Portworx experimental InitializerConfiguration admission controller](https://docs.portworx.com/portworx-install-with-kubernetes/storage-operations/hyperconvergence/initializer-experimental-feature-in-stork-v1-1). 
 Private clusters To install Portworx in a cluster that doesn't have VRF or access to private cloud service endpoints (CSEs), you must create a rule in the default security group to allow inbound and outbound traffic for the following IP addresses: 166.9.24.81, 166.9.22.100, and 166.9.20.178.
Reviewing issues and status 



1. To see whether IBM Cloud is available, [check the IBM Cloud status page](https://cloud.ibm.com/status?selected=status).
2. Filter for the Kubernetes Service component.
3. Review the [limitations and known issues documentation](https://cloud.ibm.com/docs/containers?topic=containers-limitations).
4. For issues in open source projects that are used by IBM Cloud, see the [IBM Open Source and Third Party policy](https://www.ibm.com/support/pages/node/737271). For example, you might check the Red Hat OpenShift [Bugzilla](https://bugzilla.redhat.com/).







 Feedback and questions 



1. Post in the Kubernetes Service Slack.



* If you are an external user, post in the

[#openshift](https://ibm-cloud-success.slack.com/messages/CKCJLJCH4) channel.



2. Review forums such as Red Hat OpenShift help or Stack Overflow to see whether other users ran into the same issue. When you use the forums to ask a question, tag your question so that it is seen by the IBM Cloud development teams.



* If you have technical questions about developing or deploying clusters or apps with Red Hat OpenShift on IBM Cloud, post your question on [Stack Overflow](https://stackoverflow.com/questions/tagged/ibm-cloud+containers) and tag your question with ibm-cloud, openshift, and containers.
* See [Getting help](https://cloud.ibm.com/docs/get-support?topic=get-support-using-avatar) for more details about using the forums.









 Contacting support 

Before you open a support case, gather relevant information about your cluster environment.



1. Get your cluster details.

ibmcloud oc cluster get -c <cluster_name_or_ID>
2. If your issue involves worker nodes, get the worker node details.



1.
4. Infrastructure account: If the user's username is returned as the owner of the infrastructure account, migrate your existing clusters to a different infrastructure account before removing the user. For each cluster that the user created, follow these steps:



1. Check which infrastructure account the user used to provision the cluster.



1. In the Worker Nodes tab, select a worker node and note its ID.
2. Open the menu ![Menu icon](https://cloud.ibm.com/docs-content/v1/content/4d497aff9e27b92fbbc2ee4e343a9ec4549cb1d7/icons/icon_hamburger.svg) and click Classic Infrastructure.
3. From the infrastructure navigation pane, click Devices > Device List.
4. Search for the worker node ID that you previously noted.
5. If you don't find the worker node ID, the worker node is not provisioned into this infrastructure account. Switch to a different infrastructure account and try again.



2. Determine what happens to the infrastructure account that the user used to provision the clusters after the user leaves.



* If the user does not own the infrastructure account, then other users have access to this infrastructure account and it persists after the user leaves. You can continue to work with these clusters in your account. Make sure that at least one other user has the [Administrator platform access role](https://cloud.ibm.com/docs/containers?topic=containers-usersadd_users) for the clusters.
* If the user owns the infrastructure account, then the infrastructure account is deleted when the user leaves. You can't continue to work with these clusters. To prevent the cluster from becoming orphaned, the user must delete the clusters before the user leaves. If the user leaves but the clusters were not deleted, you must use the ibmcloud ks credential set command to change your infrastructure credentials to the account that the cluster worker nodes are provisioned in, and delete the cluster.
Because Portworx runs as a DaemonSet in your cluster, existing worker nodes are automatically inspected for raw block storage and added to the Portworx data layer when you deploy Portworx. If you add or update worker nodes to your cluster and add raw block storage to those workers, restart the Portworx pods on the new or updated worker nodes so that your storage volumes are detected by the DaemonSet. 
 VPC clusters Storage volume reattachment required when updating worker nodes. When you update a worker node in a VPC cluster, the worker node is removed from your cluster and replaced with a new worker node. If Portworx volumes are attached to the worker node that is replaced, you must attach the volumes to the new worker node. You can attach storage volumes with the [API](https://cloud.ibm.com/docs/openshift?topic=openshift-utilitiesvpc_api_attach) or the [CLI](https://cloud.ibm.com/docs/openshift?topic=openshift-kubernetes-service-clics_storage_att_cr). Note this limitation does not apply to Portworx deployments that are using cloud drives. 
 The Portworx experimental InitializerConfiguration feature is not supported. Red Hat OpenShift on IBM Cloud does not support the [Portworx experimental InitializerConfiguration admission controller](https://docs.portworx.com/portworx-install-with-kubernetes/storage-operations/hyperconvergence/initializer-experimental-feature-in-stork-v1-1). 
 Private clusters To install Portworx in a cluster that doesn't have VRF or access to private cloud service endpoints (CSEs), you must create a rule in the default security group to allow inbound and outbound traffic for the following IP addresses: 166.9.24.81, 166.9.22.100, and 166.9.20.178.
* Business impact so IBM Support understands the urgency of the issue and can prioritize it.
* Case information for any related cases or a parent case.
* Software versions of both the Discovery service version and IBM Cloud Pak for Data version.
* Relevant details about configuration choices that were made during installation and deployment.







 Problem description 



* What outcome were you expecting and what happened?
* Message text that is displayed when the error occurs, especially the document ID, if specified.
* Steps to take to reproduce the issue.
* Any screen captures that illustrate the problem.
* When did the problem occur?
* Instance ID. (From the IBM Cloud Pak for Data web client main menu, expand Services, and then click Instances. Find your instance, and open its summary page. Scroll to the Access information section of the page, and then copy the URL. The instance ID is part of the URL. You can provide the full URL to IBM Support.)
* If you are using the API, share example API calls, including the version parameter value that was specified, and the API response body.

Do not share code examples. IBM Support cannot debug custom code.
* Relevant logs, including the Red Hat OpenShift collector logs.

The IBM Support representative can share a script with you that collects relevant logs from your cluster.
* If the problem is related to a particular project or collection, provide the project ID and collection ID.



* Project ID. (You can copy the Project ID from the API Information tab of the Integrate and deploy page in the product user interface.)
* Collection ID, if you were able to create a collection. (To get the ID, open the Manage collections page, and then click the collection to open it. From the web browser location field, scroll to the end of the URL. Look for the collections/ section, and then copy the ID that is displayed after it. For example, in the URL /collections/5a525eb7-b175-3820-0000-017d00f0fcd1/activity, the collection ID is 5a525eb7-b175-3820-0000-017d00f0fcd1.)
(https://cloud.ibm.com/docs-content/v1/content/522c6f62358b063f921283400a601c3f9bc66f08/Cloudant/images/sharding_document.svg)

Figure 2. Document consistent hashing

Occasionally, shards are rebalanced. Rebalancing involves moving replicas to different servers. Rebalancing occurs for several reasons, for example, when server monitoring suggests that a server is more heavily or lightly used than other servers, or when a server must be taken out of service temporarily for maintenance. The number of shards and replicas stays the same, and documents remain assigned to the same shard, but the server storage location for a shard replica changes.

The default value for Q varies for different clusters. The value can be tuned over time.

The number of replicas (copies of a shard) is also configurable. In practice, observation and measurement of many systems suggests that three replicas are a pragmatic number in most cases to achieve a good balance between performance and data safety. It would be exceptional and unusual for an IBM Cloudant system to use a different replica count.





 How does sharding affect performance? 

The number of shards for a database is configurable because it affects database performance in a number of ways.

When a request comes into the database from a client application, one server or 'node' in the cluster is assigned as the coordinator of the request. This coordinator makes internal requests to the nodes that hold the data relevant to the request, determines the response to the request, and returns this response to the client.

The number of shards for a database can affect the performance in two ways:



1. Each document in the database is stored on a single shard.

Therefore, having many shards enables greater parallelism for any single document request. The reason is that the coordinator sends requests only to the nodes that hold the document. Therefore, if the database has many shards, there are likely to be many other nodes that do not need to respond to the request. These nodes can continue to work on other tasks without interruption from the coordinator request.
2.
Your worker node machine is removed, and data is deleted if not [stored outside the pod](https://cloud.ibm.com/docs/openshift?topic=openshift-storage-plan).



* [Access your Red Hat OpenShift cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-access_cluster).
* [Update the master](https://cloud.ibm.com/docs/openshift?topic=openshift-updatemaster). The worker node version can't be higher than the API server version that runs in your Kubernetes master.
* Make any changes that are marked with Update after master in the [Red Hat OpenShift version preparation guide](https://cloud.ibm.com/docs/openshift?topic=openshift-openshift_versions).
* If you want to apply a patch update, review the [Kubernetes clusters](https://cloud.ibm.com/docs/openshift?topic=openshift-openshift_versions) version change log.
* Make sure that you have the [Operator or Administrator IBM Cloud IAM platform access role](https://cloud.ibm.com/docs/openshift?topic=openshift-userschecking-perms).







 Updating VPC worker nodes in the CLI 

Complete the following steps to update your worker nodes by using the CLI.



1. Complete the [prerequisite steps](https://cloud.ibm.com/docs/openshift?topic=openshift-updatevpc_worker_prereqs).
2. Optional: Add capacity to your cluster by [resizing the worker pool](https://cloud.ibm.com/docs/openshift?topic=openshift-add_workersresize_pool). The pods on the worker node can be rescheduled and continue running on the added worker nodes during the update.
3. List the worker nodes in your cluster and note the ID and Primary IP of the worker node that you want to update.
You can find more information about your worker node when you run the ibmcloud ks worker ls --cluster <cluster_name_or_ID> command and look for the worker node with the same Private IP address.
2. Replace the worker node. As part of the replace process, the pods that run on the worker node are drained and rescheduled onto remaining worker nodes in the cluster. The worker node is also cordoned, or marked as unavailable for future pod scheduling. Use the worker node ID that is returned from the ibmcloud ks worker ls --cluster <cluster_name_or_ID> command.

ibmcloud ks worker replace --cluster <cluster_name_or_ID> --worker <worker_node_ID>
3. Verify that the worker node is replaced.

ibmcloud ks worker ls --cluster <cluster_name_or_ID>









 Unable to create or delete worker nodes due to paid account or one time password error 

Classic infrastructure

 What’s happening 

You can't manage worker nodes for your cluster, and you receive an error message similar to one of the following examples.

Unable to connect to the IBM Cloud account. Ensure that you have a paid account.

can't authenticate the infrastructure user: Time-based One Time Password authentication is required to log in with this user.

 Why it’s happening 

Your IBM Cloud account uses its own automatically linked infrastructure through a Pay-as-you-Go account.

However, the account administrator enabled the time-based one-time passcode (TOTP) option so that users are prompted for a time-based one-time passcode (TOTP) at login. This type of [multifactor authentication (MFA)](https://cloud.ibm.com/docs/account?topic=account-typesaccount-based) is account-based, and affects all access to the account. TOTP MFA also affects the access that IBM Cloud Kubernetes Service requires to make calls to IBM Cloud infrastructure. If TOTP is enabled for the account, you can't create and manage clusters and worker nodes in IBM Cloud Kubernetes Service.

 How to fix it