vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Again, for our example, we use the CLI to set up a daily snapshot of a CouchDB volume at 3:00 a.m. and retain seven snapshots.

ibmcloud sl file snapshot-enable 155617812 -s DAILY -c 7 -r 3

Which produces a message similar to the following:

OK
DAILY snapshots have been enabled for volume 155617812.

We can now list the snapshot schedule to verify it's in place by issuing:

ibmcloud sl file snapshot-schedule-list 155617812

Which produces a message similar to the following:

id active type replication date_created minute hour day week day_of_week date_of_month month_of_year maximum_snapshots
542130 * DAILY - 2020-09-02T16:35:58Z 0 3 * - * * * 7

Repeat the following steps for each pod you want to back up, remembering to schedule your backups in the proper order.







 Restoring nodes 

If you have to restore a node, first cancel any scheduled snapshots. Only restore from a snapshot that does not have bad or corrupted data. After a node has been restored, you can reimplement your snapshot schedule.

Here we have two examples. In the first example, we show the steps for restoring peer nodes, while in the second example, we describe the steps for restoring the ordering service, which requires restoring peer nodes as well.

Because the details for finding the persistent volumes that are associated with each ordering node and restoring snapshots are identical to what is documented for the peer, the commands are only listed once, in [the peer instructions](https://cloud.ibm.com/docs/blockchain?topic=blockchain-backup-restorebackup-restore-restore-peer). However, additional considerations when restoring ordering nodes are documented in [Restoring an ordering node](https://cloud.ibm.com/docs/blockchain?topic=blockchain-backup-restorebackup-restore-restore-orderer).



 Sequencing restorations 

Just as when backing up, care must be taken to restore things in the proper order.
Run debug commands to help you gather information and troubleshoot issues. Commands that you might use to debug, such as tcpdump, mtr, curl, ip, ifconfig, nc, ping, and ps, are already available in the shell. You can also install other tools, such as dig, by running apk add <tool>. For example, to add dig, run apk add bind-tools.

Before you can use the tcpdump command, you must first move the binary to a new location that does not conflict with the install path for the binary on the host. You can use the following command to relocate the binary: mv /usr/sbin/tcpdump /usr/local/bin/.
6. Delete the host access pod that you created for debugging.

kubectl delete pod debug-${NODE}







 Debugging by creating a pod with root SSH access 

If you are unable to use the oc debug node or kubectl exec commands, such as if the VPN connection between the cluster master and worker nodes is down, you can create a pod that enables root SSH access and copies a public SSH key to the worker node for SSH access.

Allowing root SSH access is a security risk. Only allow SSH access when it is required and no other option is available to troubleshoot worker node issues. When you finish troubleshooting, be sure to follow the steps in the [Cleaning up after debugging](https://cloud.ibm.com/docs/openshift?topic=openshift-cs_ssh_workerssh-debug-cleanup) section to disable SSH access.



1. Choose an existing or create a new public SSH key.

ssh-keygen -t rsa -b 4096
ls /.ssh

id_rsa id_rsa.pub

cat /.ssh/id_rsa.pub
2. Get the name of the worker node that you want to access. The worker node name is its private IP address.

oc get nodes -o wide
3.
To avoid downtime for your app, you must ensure that you have enough capacity in the cluster to carry the workload, such as by [resizing your worker pools](https://cloud.ibm.com/docs/openshift?topic=openshift-add_workersresize_pool).

What happens to my worker node during an update?
: Your VPC worker node is replaced by removing the old worker node and provisioning a new worker node that runs at the updated patch or major.minor version. The replacement worker node is created in the same zone, same worker pool, and with the same flavor as the deleted worker node. However, the replacement worker node is assigned a new private IP address, and loses any custom labels or taints that you applied to the old worker node (worker pool labels and taints are still applied to the replacement worker node).

What if I replace multiple worker nodes at the same time?
: If you replace multiple worker nodes at the same time, they are deleted and replaced concurrently, not one by one. Make sure that you have enough capacity in your cluster to reschedule your workloads before you replace worker nodes.

What if a replacement worker node is not created?
: A replacement worker node is not created if the worker pool does not have [automatic rebalancing enabled](https://cloud.ibm.com/docs/containers?topic=containers-auto-rebalance-off).



 Prerequisites 

Before you update your VPC infrastructure worker nodes, review the prerequisite steps.

Updates to worker nodes can cause downtime for your apps and services. Your worker node machine is removed, and data is deleted if not [stored outside the pod](https://cloud.ibm.com/docs/openshift?topic=openshift-storage-plan).



* [Access your Red Hat OpenShift cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-access_cluster).
* [Update the master](https://cloud.ibm.com/docs/openshift?topic=openshift-updatemaster).
Why can't I SSH into my worker node? 

Virtual Private Cloud

Classic infrastructure

 What’s happening 

You can't access your worker node by using an SSH connection.

 Why it’s happening 

SSH by password is unavailable on the worker nodes.

 How to fix it 

To run actions on every worker node, use a Kubernetes [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/), or use jobs for one-time actions.

To get host access to worker nodes for debugging and troubleshooting purposes, review the following options.



 Debugging by using oc debug 

Use the oc debug node command to deploy a pod with a privileged securityContext to a worker node that you want to troubleshoot.

The debug pod is deployed with an interactive shell so that you can access the worker node immediately after the pod is created. For more information about how the oc debug node command works, see [this Red Hat blog post](https://www.redhat.com/sysadmin/how-oc-debug-works).



1. Get the name of the worker node that you want to access. The worker node name is its private IP address.

oc get nodes -o wide
2. Create a debug pod that has host access. When the pod is created, the pod's interactive shell is automatically opened. If the oc debug node command fails, continue to option 2. The Docker alpine image here is used as an example. If the worker node doesn't have public network access, you can maintain a copy of the image for debugging in your own ICR repository or build a customized image with other tools to fit your needs.

oc debug node/<NODE_NAME>
3. Run debug commands to help you gather information and troubleshoot issues. Commands that you might use to debug, such as tcpdump, curl, ip, ifconfig, nc, ping, and ps, are already available in the shell.
Follow these troubleshooting steps to determine the cause and resolve the issue.

Some steps are specific to a specialized area, such as networking or automation. Consult with the relevant administrator or team in your organization before completing these steps.



1. Check if there were any recent changes to your cluster, environment, or account that might impact your worker nodes. If so, revert the changes and then check the worker node status to determine if the changes caused the issue.



* For classic clusters, check any firewall or gateway, such as Virtual Router Appliance, Vyatta, or Juniper that manages traffic for cluster workers. Look for changes or issues that might drop or redirect traffic from cluster workers.
* For VPC clusters, check if any changes were made to the default security group and ACLs on the VPC or the worker nodes. If any modifications were made, ensure that you are allowing all necessary traffic from the cluster worker nodes to the cluster master, container registry, and other critical services. For more information, see [Controlling Traffic with VPC Security Groups](https://cloud.ibm.com/docs/containers?topic=containers-vpc-security-group&interface=ui) and [Controlling traffic with ACLs](https://cloud.ibm.com/docs/containers?topic=containers-vpc-acls&interface=ui).
* For VPC clusters, check any custom routing rules for changes that might be blocking traffic from the cluster apiserver.
* Check any Calico or Kubernetes network policies that are applied to the cluster and make sure that they do not block traffic from the worker node to the cluster apiservice, container registry, or other critical services.



2. Check if the applications, security, or monitoring components in your cluster are overloading the cluster apiserver with requests, which may cause disruptions for your worker nodes.
3. If you recently added any components to your cluster, remove them. If you made changes to any existing components in your cluster, revert the changes. Then, check the status of your worker nodes to see if the new components or changes were causing the issue.
4.
Output 

Record the interconnections between your IBM® Cloud Foundry applications, making note of what applications needs to talk to other applications.





 Migration steps 

Communicating directly between running containers or micro-services can be important for operational load, speed, and security. Be sure to understand the needs for the compute type you are moving to to ensure smooth ongoing operations.







 User responsibilities 

What are the responsibilities for users for ongoing IBM® Cloud Foundry applications?



* [Understanding your responsibilities when using IBM Cloud Foundry](https://cloud.ibm.com/docs/cloud-foundry-public?topic=cloud-foundry-public-cfp_responsibilities)





 Commands 

IBM® Cloud Foundry typically requires less user responsibilities than other application compute service models.

In IBM® Cloud Foundry you generally need to ensure that your application has enough instances to run at the availability level you need, and ensure that your application is properly monitored for that availability level.





 Output 

Evaluate how your application development and operations team interact with the IBM® Cloud Foundry system and your applications. Decide what steps need to be taken in the new migration environment, if any, that are different from what you do now.





 Migration steps 

User responsibilities might change with different hosting services. Different cloud hosting methods have different responsibilities for on-going management and monitoring of applications and infrastructure. Be sure to understand the needs for the compute type you are moving to to ensure smooth ongoing operations.
ibmcloud oc worker ls --cluster <cluster_name_or_ID>

When the worker nodes are ready, the worker node State changes to normal and the Status changes to Ready. When the node Status changes to Ready, you can access the cluster. Note that even if the cluster is ready, some parts of the cluster that are used by other services, such as Ingress secrets or registry image pull secrets, might still be in process.

ID Public IP Private IP Flavor State Status Zone Version
kube-blrs3b1d0p0p2f7haq0g-mycluster-default-000001f7 169.xx.xxx.xxx 10.xxx.xx.xxx b3c.4x16.encrypted normal Ready dal10 4.11.42_1544_openshift

Every worker node is assigned a unique worker node ID and domain name that must not be changed manually after the cluster is created. If you change the ID or domain name, the Red Hat OpenShift master cannot manage your cluster.



Your cluster is ready for your workloads! You might also want to [add a tag to your cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-add_workerscluster_tags), such as the team or billing department that uses the cluster, to help manage IBM Cloud resources. For more ideas of what to do with your cluster, review the [Next steps](https://cloud.ibm.com/docs/openshift?topic=openshift-clustersnext_steps).





 Example commands to create VPC clusters 

VPC Gen 2 cluster flavors with instance storage are available for allowlisted accounts. To get added to the allowlist, [open a case](https://cloud.ibm.com/unifiedsupport/cases/form) with support.
Why are the file systems for worker nodes changed to read-only? 

Classic infrastructure

  What’s happening 

You might see one of the following symptoms {: #stuck_creating_state}:



*  When you run kubectl get pods -o wide, you see that multiple pods that are running on the same worker node are stuck in the ContainerCreating state.
*  When you run a kubectl describe command, you see the following error in the Events section: MountVolume.SetUp failed for volume ... read-only file system.



  Why it’s happening 

The file system on the worker node is read-only.

  How to fix it 

Back up your data and reload your worker node.



1.  Back up any data that might be stored on the worker node or in your containers.
2.  For a short-term fix to the existing worker node, reload the worker node.

ibmcloud ks worker reload --cluster <cluster_name> --worker <worker_ID>



For a long-term fix, [update the flavor of your worker pool](https://cloud.ibm.com/docs/containers?topic=containers-updatemachine_type).
Why are the file systems for worker nodes changed to read-only? 

Classic infrastructure

  What’s happening 

You might see one of the following symptoms {: #stuck_creating_state}:



*  When you run oc get pods -o wide, you see that multiple pods that are running on the same worker node are stuck in the ContainerCreating state.
*  When you run a oc describe command, you see the following error in the Events section: MountVolume.SetUp failed for volume ... read-only file system.



  Why it’s happening 

The file system on the worker node is read-only.

  How to fix it 

Back up your data and reload your worker node.



1.  Back up any data that might be stored on the worker node or in your containers.
2.  For a short-term fix to the existing worker node, reload the worker node.

ibmcloud oc worker reload --cluster <cluster_name> --worker <worker_ID>



For a long-term fix, [update the flavor of your worker pool](https://cloud.ibm.com/docs/containers?topic=containers-updatemachine_type).
Why are my nodes failing to register using Azure File or Azure Disk storage templates? 

  What’s happening 

Your worker nodes are failing to register when creating an Azure Disk or Azure File storage configuration.

  How to fix it 

If you manually assigned your Azure hosts to your location and did not use an automated deployment such as deploying from the console or a Terraform template, then you must label your worker nodes before creating your storage configuration. If your nodes are failing to register when creating a Azure Disk or Azure File Satellite storage configuration, verify that your worker nodes are labeled with the proper zone. Follow the steps to verify that your worker nodes have the proper labels:



1.  

1.  Log in to the IBM Cloud CLI.



ibmcloud login
2.  List your Satellite locations and note the Managed from column.

ibmcloud sat location ls
3.  Target the Managed from region of your Satellite location. For example, for wdc target us-east. For more information, see [Satellite regions](https://cloud.ibm.com/docs/satellite?topic=satellite-sat-regions).

ibmcloud target -r us-east
4.  If you use a resource group other than default, target it.

ibmcloud target -g <resource-group>
5.  List your Azure worker nodes and make a note of the name of each node.

oc get nodes
6.  Get the details of each node and make a note of the zone that the node is in. For example: eastus-1.

oc get nodes NODE-NAME -o yaml | grep zone
7.  Label your Azure worker nodes with the zone value that you retrieved earlier. Replace <node_name> and <zone> with the node name and zone of your worker nodes. For example, if you have a worker node in zone: eastus-1, use the following commands to add eastus-1 as a label to the worker node in the eastus-1 zone.

oc label node <node_name> topology.kubernetes.io/zone-
oc label node <node_name> topology.kubernetes.io/zone=<zone> --overwrite
8.  Repeat the previous steps for each worker node.
After deleting all worker nodes, why don't my pods start on new worker nodes? 

Virtual Private Cloud

Classic infrastructure

 What’s happening 

You deleted all worker nodes in your cluster so that zero worker nodes exist. Then, you added one or more worker nodes. When you run the following command, several pods for Kubernetes components are stuck in the ContainerCreating status, and the calico-node pods are stuck in the CrashLoopBackOff status.

kubectl -n kube-system get pods

 Why it’s happening 

When you delete all worker nodes in your cluster, no worker node exists for the calico-kube-controllers pod to run on. The Calico controller pod's data can't be updated to remove the data of the deleted worker nodes. When the Calico controller pod begins to run again on the new worker nodes, its data is not updated for the new worker nodes, and it does not start the calico-node pods.

 How to fix it 

Delete the existing calico-node worker node entries so that new pods can be created.

Before you begin: Install the [Calico CLI](https://cloud.ibm.com/docs/containers?topic=containers-network_policiescli_install).



1. Run the ibmcloud ks cluster config command and copy and paste the output to set the KUBECONFIG environment variable. Include the --admin and --network options with the ibmcloud ks cluster config command. The --admin option downloads the keys to access your infrastructure portfolio and run Calico commands on your worker nodes. The --network option downloads the Calico configuration file to run all Calico commands.

ibmcloud ks cluster config --cluster <cluster_name_or_ID> --admin --network
2. For the calico-node pods that are stuck in the CrashLoopBackOff status, note the NODE IP addresses.

kubectl -n kube-system get pods -o wide

In this example output, the calico-node pod can't start on worker node 10.176.48.106.
After deleting all worker nodes, why don't my pods start on new worker nodes? 

Virtual Private Cloud

Classic infrastructure

 What’s happening 

You deleted all worker nodes in your cluster so that zero worker nodes exist. Then, you added one or more worker nodes. When you run the following command, several pods for Kubernetes components are stuck in the ContainerCreating status, and the calico-node pods are stuck in the CrashLoopBackOff status.

oc -n kube-system get pods

 Why it’s happening 

When you delete all worker nodes in your cluster, no worker node exists for the calico-kube-controllers pod to run on. The Calico controller pod's data can't be updated to remove the data of the deleted worker nodes. When the Calico controller pod begins to run again on the new worker nodes, its data is not updated for the new worker nodes, and it does not start the calico-node pods.

 How to fix it 

Delete the existing calico-node worker node entries so that new pods can be created.

Before you begin: Install the [Calico CLI](https://cloud.ibm.com/docs/openshift?topic=openshift-network_policiescli_install).



1. Run the ibmcloud oc cluster config command and copy and paste the output to set the KUBECONFIG environment variable. Include the --admin and --network options with the ibmcloud oc cluster config command. The --admin option downloads the keys to access your infrastructure portfolio and run Calico commands on your worker nodes. The --network option downloads the Calico configuration file to run all Calico commands.

ibmcloud oc cluster config --cluster <cluster_name_or_ID> --admin --network
2. For the calico-node pods that are stuck in the CrashLoopBackOff status, note the NODE IP addresses.

oc -n kube-system get pods -o wide

In this example output, the calico-node pod can't start on worker node 10.176.48.106.
Because the details for finding the persistent volumes that are associated with each ordering node and restoring snapshots are identical to what is documented for the peer, the commands are only listed once, in [the peer instructions](https://cloud.ibm.com/docs/blockchain?topic=blockchain-backup-restorebackup-restore-restore-peer). However, additional considerations when restoring ordering nodes are documented in [Restoring an ordering node](https://cloud.ibm.com/docs/blockchain?topic=blockchain-backup-restorebackup-restore-restore-orderer).



 Sequencing restorations 

Just as when backing up, care must be taken to restore things in the proper order. When restoring ordering nodes, for example, even if you only restore one node, scale down all of the pods of the ordering nodes by reducing their resources to zero. If every ordering node must be restored, the best practice is to stop them all at once and then restore them all at once. Then, make sure all of them come up and give them a few minutes to elect a leader and establish quorum before driving any transactions.

If you are restoring the ordering service from a backup, you must also either create new peers or restore all peers to a backup older than the ordering service. Peers restored from an older backup can catch up to the latest data by receiving blocks from the ordering service.

When restoring a peer, make sure to restore backups of CouchDB older than the backup of the peer pod. If your state database has transactions your peer does not have, the database cannot sync with the peer.





 Restoring a peer 

To restore a peer, you need to:



1. Find the deployment that you want to restore.
2. Scale it down to zero to stop the node.
3. Restore the node from the backup.



First, get the list of deployments:

kubectl get deployments

As before, this produces a list of nodes and their persistent volumes similar to this example:

NA.M.E READY UP-TO-DATE AVAILABLE AGE
ibp-operator 1/1 1 1 48d
orderer1node1 1/1 1 1 20h
Why can't I SSH into my worker node? 

Virtual Private Cloud

Classic infrastructure

 What’s happening 

You can't access your worker node by using an SSH connection.

 Why it’s happening 

SSH by password is unavailable on the worker nodes.

 How to fix it 

To run actions on every worker node, use a Kubernetes [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/), or use jobs for one-time actions.

To get host access to worker nodes for debugging and troubleshooting purposes, review the following options.



 Debugging by using kubectl debug 

Use the kubectl debug node command to deploy a pod with a privileged securityContext to a worker node that you want to troubleshoot.

The debug pod is deployed with an interactive shell so that you can access the worker node immediately after the pod is created. For more information about how the kubectl debug node command works, see [debug command in Kubernetes reference](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commandsdebug).



1. Get the name of the worker node that you want to access. The worker node name is its private IP address.

kubectl get nodes -o wide
2. Create a debug pod that has host access. When the pod is created, the pod's interactive shell is automatically opened. If the kubectl debug node command fails, continue to option 2. The Docker alpine image here is used as an example. If the worker node doesn't have public network access, you can maintain a copy of the image for debugging in your own ICR repository or build a customized image with other tools to fit your needs.

kubectl debug node/<NODE_NAME> --image=docker.io/library/alpine:latest -it
3. Run debug commands to help you gather information and troubleshoot issues.