vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Debugging persistent storage failures 

Review the options to debug persistent storage and find the root causes for failures.



 Checking whether the pod that mounts your storage instance is successfully deployed 



1. List the pods in your cluster. A pod is successfully deployed if the pod shows a status of Running.

oc get pods
2. Get the details of your pod and check whether errors are displayed in the Events section of your CLI output.

oc describe pod <pod_name>
3. Retrieve the logs for your app and check whether you can see any error messages.

oc logs <pod_name>







 Restarting your app pod 



1. If your pod is part of a deployment, delete the pod and let the deployment rebuild it. If your pod is not part of a deployment, delete the pod and reapply your pod configuration file.

oc delete pod <pod_name>

oc apply -f <app.yaml>
2. If restarting your pod does not resolve the issue, [reload your worker node](https://cloud.ibm.com/docs/openshift?topic=openshift-kubernetes-service-clics_worker_reload).
3. Verify that you use the latest IBM Cloud and IBM Cloud Kubernetes Service plug-in version.

ibmcloud update

ibmcloud plugin repo-plugins







 Verifying that the storage driver and plug-in pods show a status of Running 



1. List the pods in the kube-system project.

oc get pods -n kube-system
2. If the storage driver and plug-in pods don't show a Running status, get more details of the pod to find the root cause. Depending on the status of your pod, you might not be able to execute all the following commands.



1. Get the names of the containers that run in the driver pod.
If every compliance check (for example, unit test, CRA tasks, branch protection, detect secrets) is successful, the change request is approved automatically, and the task runs successfully.

If a compliance check fails, the change request state is not approved.

You can approve the change request manually and add the change-request-id to the environment properties to use the already created change request in the next run.

Another solution is to use the emergency label in the promotion pull requests. For more information, see [Add emergency label](https://cloud.ibm.com/docs/devsecops?topic=devsecops-cd-devsecops-approve-cr).





 Set to implement 

This step sets the status of the change request to implement depending on the success or failure status of the change.





 Close change request 

Details about the deployment are uploaded to the closing summary change task, and the change request is closed. In the close change request task, the close_category is added with these values:



* successful
* successful with issues (if the summary has issues)
This means that if a customer onboards to IBMid federation, this applies to his employees when logging on to IBM Cloud, but also when using other IBM SaaS offerings that are using IBMid. The federation has impact only on the IBM Cloud account in which the IBM Cloud App ID instance is configured to allow logins. It also does not impact any other IBM Cloud accounts or IBM SaaS offerings. 
 Account member onboarding Users need to be invited by their email address. Each user that can successfully log in to the configured IBM Cloud App ID instance will automatically be added to the IBM Cloud account. 
 Cross-account membership IBMids can be members of multiple accounts. Users can be members of one account only. Even if multiple IBM Cloud App ID instances federate to the same IdP, the users are treated as separate users in each account. 
 Use of SAML assertions in access group dynamic rules Any SAML assertion sent by the IdP can be used in access group dynamic rules. Any SAML assertion sent by the IdP can be used in access group dynamic rules. 
 Reliability IBMid is a global service with a dedicated operations team. In case of an outage in a data center, IBMid can failover to a different data center. Any IBM Cloud App ID instance exists in one region. Failures in that region can prevent account members from being able to log in. Users that are already logged in are usually not affected by such a failure. 
 Login behavior Users log in by using the central login page. You need to use a special URL to log in to your IBM Cloud account. Your account administrator gets this URL from the IAM [Identity providers](https://cloud.ibm.com/iam/identity-providers) page in the IBM Cloud console. 







 Scenarios 

For additional help deciding the correct option, review the common scenarios:

Using a single IBM Cloud account
: A customer is using a single IBM Cloud account and no other IBM SaaS offerings. Since the customer is not using any other IBM SaaS offering or any other IBM Cloud account, they have no advantage in the federation scope capabilities or cross-account capabilities of IBMid.
5. View the output of a failed test by looking at the logs of the test pod.

oc logs <test_program>

Some tests have requirements that are optional settings in the VPN configuration. If some tests fail, the failures might be acceptable depending on whether you specified these optional settings. Refer to the following table for information about each test and why it might fail.

vpn-strongswan-check-config
: Validates the syntax of the ipsec.conf file that is generated from the config.yaml file. This test might fail due to incorrect values in the config.yaml file.

vpn-strongswan-check-state
: Checks that the VPN connection has a status of ESTABLISHED. This test might fail for the following reasons.



* Differences between the values in the config.yaml file and the on-premises VPN endpoint settings.
* If the cluster is in "listen" mode (ipsec.auto is set to add), the connection is not established on the on-premises side.



vpn-strongswan-ping-remote-gw
: Pings the remote.gateway public IP address that you configured in the config.yaml file. If the VPN connection has the ESTABLISHED status, you can ignore the result of this test. If the VPN connection does not have the ESTABLISHED status, this test might fail for the following reasons.



* You did not specify an on-premises VPN gateway IP address. If ipsec.auto is set to start, the remote.gateway IP address is required.
* ICMP (ping) packets are being blocked by a firewall.



vpn-strongswan-ping-remote-ip-1
: Pings the remote.privateIPtoPing private IP address of the on-premises VPN gateway from the VPN pod in the cluster. This test might fail for the following reasons. \n - You did not specify a remote.privateIPtoPing IP address. If you intentionally did not specify an IP address, this failure is acceptable.
Monitor the status of a product spun up by your deployable architecture IBM provides the ability for customers to monitor the lifecycle of the instances. Use the resource list, service instance pages, or the [Status](https://cloud.ibm.com/status) page to monitor events that specifically impact your service instance. 







 Change management 

Change management includes tasks such as deployment, configuration, upgrades, patching, configuration changes, and deletion.



Table 2. Responsibilities for change management
The first column describes the task that a customer or IBM might be responsibility for. The second column describes IBM responsibilities for that task. The third column describes your responsibilities as the customer for that task.

 Task IBM Responsibilities Your Responsibilities 

 Creation of the IBM Cloud deployable architectures IBM provides the base pattern as a deployable architecture for instantiation through Terraform. N/A 
 Must use supported version of [IBM Cloud Terraform Provider](https://cloud.ibm.com/docs/secure-enterprise?topic=secure-enterprise-responsibilities-deployable-architectureschange-management-da). IBM publishes Terraform provider of all Terraform enabled services on IBM Cloud. Customers should use the latest major version. Terraform Providers version requirements are documented within the version.tf file for each deployable architecture. 
 Third-party Terraform Providers used within templates N/A Customer is responsible for any use of third-party Terraform code that is used with the deployable architecture. 
 Running default configuration (out of the box) IBM provides the ability for customers to create and deploy configurations of deployable architectures by using projects, Terraform-as-a-service, or the projects CLI. Use projects to [configure and deploy a deployable architecture](https://cloud.ibm.com/docs/secure-enterprise?topic=secure-enterprise-config-project). 
 Running templates locally by using Terraform directly N/A Customer can run the deployable architecture on their local system. 
 Customize modules or deployable architectures with pre-supported modules IBM provides and supports base Terraform modules for services on IBM Cloud, and provides preset JSON configuration overrides for templates.
* If the comparison is a failure, enter the commands to run.



If you use interactive mode, etcdctl txn -i, the session prompts for the parts.

$ etcdctl put key1 1
OK

$ etcdctl txn -i
 compares:
mod("key1") > "0"
 success requests (get, put, delete):
put key1 "overwrote-key1"
>
>
 failure requests (get, put, delete):
>
>
SUCCESS

OK
Show more

To check that the new value got assigned to the key,

etcdctl get key1
> key1
overwrote-key1





 Leases 

etcd3 uses leases to manage temporary keys. A lease is created with a time-to-live (TTL) and then attached to a key at creation. One lease can be used for multiple keys. When the TTL on the lease expires, it deletes all associated keys. To create a lease, use grant with a number of seconds for TTL.

$ etcdctl lease grant 500
lease 694d5cefe052b00d granted with TTL(500s)

Then, create a key/pair with the --lease option.

$ etcdctl put foo1 bar1 --lease=694d5cefe052b00d
OK

You can find out what keys are associated with a granted lease, along with the time remaining on the lease.

$ etcdctl lease timetolive 694d5cefe052b00d --keys
lease 694d5cefe052b00d granted with TTL(500s), remaining(377s), attached keys([foo1])

After the time on a lease expires, requesting the value of a key does not return anything. If you try to provision a key on a lease that has already expired, it returns an error.

$ etcdctl lease grant 10
lease 694d5cefe052b009 granted with TTL(10s)
Click the name of a replica file share to open its details page.
3. From the Actions menu, select Perform failover. In the side panel, you can preview the changes. Before performing the failover, a final sync of the files is performed to ensure that the failover share has the latest content. When the failover completes, the replica file share becomes the new source file share. The former source share becomes the new read-only replica share.
4. To set a timeout value and change the default timeout of 5 minutes, check the box under Timeout (optional) and specify a time value. This value specifies an absolute time limit for the failover to complete. Set a timeout based on how long you can have your file share offline.
5. Under Failover policy, if the failover operation does not succeed or times out, choose to keep the replication relationship or change it:



* Keep replication relationship - No changes are made to the replica file share or source file share.
* [Remove replication relationship](https://cloud.ibm.com/docs/vpc?topic=vpc-file-storage-manage-replicationfs-remove-replication) - This action creates two separate read/write file shares. Because the relationship is broken, changes to one file share do not affect the other.



After you break the relationship, it can't be reestablished.
6. Click Perform failover. Messages display that indicate that the failover was requested and is being performed.



The file share details page is updated, and the replication relationship shows the replica file share as the new source file share.





 Failover from the CLI 

Run the ibmcloud is share-replica-failover command and specify the failback-policy property. You can specify fail or split for this property.

This example specifies fail for the failback-policy property, which indicates that if the failover operation fails or the timeout is reached, the failover operation is unsuccessful.

ibmcloud is share-replica-failover dc1e70af-c3cb-44c6-a96d-2ece09b51ae3 --fallback-policy fail
You can use IBM Cloud public portable IP addresses for the appliance.
4. You can remove or disable the public uplinks on T0. If you do, remove the default route as well.
5. Create an NSX-T overlay segment for a transit network between the T0 and the third-party device. Add T0 uplink and an interface in the third-party device and establish static or dynamic routing between them.
6. Add the default, IBM Cloud private network, and IBM Cloud services networks into the device routing definitions based on your communications needs.
7. If needed, you might reassign more IPs from the public portable subnet or order new public static subnets for your overlay. With public static subnets, order them with a next of a VIP of the third-party device to cater for failure situations.







 Considerations 

When you design or deploy this architecture pattern, consider the following steps:



* Refer to the specific third-party vendor HW and SW specifications for interoperability with VMware ESXi™ and NSX-T.
* This approach supports BYOL only for the third-party solution licensing. Contact your vendor for licensing details.







 Related links 



* [VMware vSphere overview](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-vs_vsphereclusteroverview)
* [About IBM Cloud Gateway Appliance](https://cloud.ibm.com/docs/gateway-appliance?topic=gateway-appliance-about)
--help Show this message and exit.





 Starting a failover from a volume to its replica 

If a failure event is imminent, you can start a Controlled Failover or an Immediate failover your destination, or target, volume.

A Controlled Failover is the best choice when you want to test the failover function. It's also the best choice, when it’s more important to continue operations at the replica location with the most recent data. In a Controlled Failover, a new snapshot is taken and copied over to the replica location. After the data is successfully copied over, the volume is made available for mounting.

When you choose an Immediate Failover, the last successfully replicated snapshot is activated, and the volume is made available for mounting. The target volume becomes active in less time compared to a Controlled Failover. However, any data that was written to the source volume since the previous replication cycle is lost.

When a failover is started, the replication relationship is flipped. Your original target volume becomes your active source volume, and your former source volume becomes your inactive replication target.

Failovers are started under Storage, File Storage for Classic in the [IBM Cloud® console](https://cloud.ibm.com/cloud-storage/file).

Before you proceed with these steps, disconnect the volume. Failure to do so, results in corruption and data loss.





 Fail over to replica in the UI 



1. Click your active volume (“source”).
2. Click Actions![Actions icon](https://cloud.ibm.com/docs-content/v1/content/04e9937a86546143babfc65ea21fdd4ea2d12d13/icons/action-menu-icon.svg).
3. Select Failover. Select Controlled Failover or Immediate Failover.
4. Mark the checkbox to acknowledge pertinent information and click Yes to proceed. A window is displayed to confirm that the failover is in progress. Click Close to return the File shares list.
5. In the list, locate your shares.
Figure 1. Overview of fault domains in a high availability cluster

Potential failure point 1: Container or pod availability
: Containers and pods are, by design, short-lived and can fail unexpectedly. For example, a container or pod might crash if an error occurs in your app. To make your app highly available, you must ensure that you have enough instances of your app to handle the workload plus additional instances in the case of a failure. Ideally, these instances are distributed across multiple worker nodes to protect your app from a worker node failure. For more information, see [Deploying highly available apps](https://cloud.ibm.com/docs/openshift?topic=openshift-plan_deployhighly_available_apps).

Potential failure point 2: Worker node availability
: A worker node is a VM that runs on physical hardware. Worker node failures include hardware outages, such as power, cooling, or networking, and issues on the VM itself. You can account for a worker node failure by setting up multiple worker nodes in your cluster. For more information, see [Creating clusters with multiple worker nodes](https://cloud.ibm.com/docs/openshift?topic=openshift-kubernetes-service-clics_cluster_create).

Worker nodes in one zone are not guaranteed to be on separate physical compute hosts. For example, you might have a cluster with three worker nodes, but all three worker nodes were created on the same physical compute host in the IBM zone. If this physical compute host goes down, all your worker nodes are down. To protect against this failure, you must [set up a multizone cluster to spread worker nodes across three zones, or create multiple single zone clusters](https://cloud.ibm.com/docs/openshift?topic=openshift-ha_clustersha_clusters) in different zones.

Potential failure point 3: Cluster availability
: The Red Hat OpenShift master is the main component that keeps your cluster up and running. The master stores cluster resources and their configurations in the etcd database that serves as the single point of truth for your cluster.
Debugging a blueprint apply failure follows the same approach as is followed for debugging a Terraform command failure for a Schematics Workspace or stand-alone Terraform usage.

The blueprint apply command indicates which module fails. Then, prompt to show a summary of the log for the failed workspace.

Errors:
Install of module basic-cos-storage Failed

Error on Terraform Apply operation. Review log for cause of failure
Review job failure log yes/no [y/N]>

To help problem diagnosis reply y to the prompt to review the failure logs. The blueprint apply CLI command output includes the last few lines of the Terraform Apply job log. As Terraform fails on an error, frequently the last few lines of the Terraform apply log includes sufficient information to identify the cause of the Terraform failure.

If the log summary does not provide sufficient information, the ibmcloud schematics logs command to review the entire Terraform log for the failing workspace is printed summary.

Status FAILED
Workspace Logs CLI ibmcloud schematics logs --id eu-gb.workspace.basic-cos-storage.7378a905 --act-id a28612c58714c908a0d0c111df7cc74c

See troubleshooting Schematics apply errors for additional information on debugging Terraform Apply failures





 Blueprint apply failure due to Terraform config coding error 

 What’s happening 

When you run the blueprint apply command, it fails with message that the install of module fails.

 Why it’s happening 

The analysis of the module logs indicate that the modules Terraform config has a coding error, which caused the Terraform apply failure.

 How to fix it 

Correct the Terraform config error at source and push a new release to its Git source repository.

If explicit version of the blueprint modules is used on specific branches. The blueprint template requires updating in its Git repository to specify the new release tag or branch for the module statement.



* Update the blueprint module statements to specify the new module version.
* Push the new release of the blueprint template to its Git source repository.
If you still need help with the error, contact [support](https://cloud.ibm.com/docs/vpc?topic=vpc-getting-help).





 Error: stopped_for_image_creation 

Error message: Instance was stopped because a new image was created

The virtual server instance can't start instance because the image is being created from boot volume.

To resolve this error, try to start the instance again after the image creation process is complete.

If you still need help with the error, contact [support](https://cloud.ibm.com/docs/vpc?topic=vpc-getting-help).

For more information about creating images, see [Creating virtual server instances by using the UI](https://cloud.ibm.com/docs/vpc?topic=vpc-creating-virtual-servers) or [Creating virtual server instances by using the CLI](https://cloud.ibm.com/docs/vpc?topic=vpc-creating-virtual-servers&interface=cli).





 Error: stopped_by_host_failure 

Error message: Host not responding: instance stopped

The virtual server instance stopped because the host the instance is on has failed. The host failure policy for automatic restart is disabled and the instance has remained stopped.

To address this error, try the follow possible resolutions:

Stop and restart the instance. Delete the instance and try again.

If you still need help with the error, contact [support](https://cloud.ibm.com/docs/vpc?topic=vpc-getting-help).

For morre information about unexpected host failures, see [Host Failure Recovery Policies](https://cloud.ibm.com/docs/vpc?topic=vpc-host-failure-recovery-policies&interface=ui).
: Your location must have at least three zones that are physically separate so that you can spread out hosts evenly across the zones to increase [high availability](https://cloud.ibm.com/docs/satellite?topic=satellite-ha). For example, your cloud provider might have three different zones within the same region, or you might use three racks with three separate networking and power supply systems in an on-prem environment.

Latency between IBM Cloud and the location
: The hosts that you want to attach to the Satellite location control plane must have a low latency connection of less than or equal to 200 milliseconds (<= 200ms) round-trip time (RTT) to the IBM Cloud region that your Satellite location is managed from. As latency increases, you might see impacts to performance, including Satellite Link throughput, Satellite-enabled IBM Cloud service provisioning time, host failure recovery time, and in extreme cases, the availability of resources that run in the Satellite location control plane, such as Red Hat OpenShift cluster masters. For more information, see [Testing the latency between IBM Cloud and the Satellite location control plane hosts](https://cloud.ibm.com/docs/satellite?topic=satellite-host-latency-testhost-latency-mzr).

Latency between hosts in your location
: Your host infrastructure setup must have a low latency connection of less than or equal to 100 milliseconds (<= 100ms) round-trip time (RTT) between the hosts that are used for the Satellite location control plane worker nodes and the hosts that are used for other resources in the location, like clusters or [Satellite-enabled IBM Cloud service](https://cloud.ibm.com/docs/satellite?topic=satellite-managed-services). For example, in cloud providers such as AWS, this setup typically means that all the hosts in the Satellite location are from the same cloud region, like us-east-1. As latency increases, you might see impacts to performance, including provisioning and recovery times, reduced worker nodes in the cluster, Satellite-enabled IBM Cloud service degradation, and in extreme cases, failures in your cluster applications.





 Plan your host systems