vania-janet/multiturn-rag-retrieval-data huggingface dataset:
The [failover status](https://cloud.ibm.com/docs/vpc?topic=vpc-file-storage-managingfile-storage-vpc-status) shows failover_pending while the operation is underway, or while the service is waiting for another operation to complete, such as file share expansion.



 Failover for routine maintenance 

Use failover for routine maintenance on the primary site or when the site is having problems. The process works as follows.



* The file share at the source site begins refusing all read and write operations and the system attempts to pull a final copy of the share's data to the replica file share.
* The data is copied to the replica file share, which becomes read/write and is considered the new source site. (The replication relationship is reversed.)
* The service attempts to replicate data to the original site as scheduled. If the data transfer fails, the system attempts again at the next scheduled replication time.
* You can fall back to the original share when the maintenance is done and the site is stable again. Or, you can keep the replica share as your source share.







 Failover in event of a disaster recovery situation 

Failover is also an option for disaster recovery. If the original site is confirmed to be unavailable and you need your application started as soon as possible on the replica location, choose the remove replication relationship option as the fallback policy when you initiate the failover. Failover for disaster recovery works like this:



* The file share at the source site begins refusing all read and write operations and the system attempts to pull a final copy of the share's data to the replica file share.
* When the last data pull times out and fails, the file service determines that it is a disaster recovery situation. Then, the service breaks the replication relationship. The replica file share becomes read/write and operates as an independent file share.
* Replication relationship cannot be reestablished. However, you can set up a new replica on the original site if and when the site becomes operational again.



Due to the nature of the disaster recovery failover, you might find that the latest data set did not get copied over.
Recovery of logging agents N/A [Reinstall](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-logdna_agentlogdna_agent_configure) the logging agent in the event of any disaster event that impacts the agent runtime. 
 Backup the metadata of a logging instance Backup metadata that is used by the service. [Backup the metadata such as views, dashboards, screens, parsing templates, and alerts for each logging instance.](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-reuse_resource_definitionsexport_config_res) 
 Restore the metadata of a logging instance Restore metadata that is used by the service. [Restore the metadata such as views, dashboards, screens, parsing templates, and alerts for each logging instance.](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-reuse_resource_definitionsimport_config) 
 Backup of the data N/A [Configure archiving to retain a backup copy of the data.](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-archiving) 



[] Recovered and restarted service components will not have customer data reloaded.
Responsibilities for identity and access management

 Task IBM Responsibilities Your Responsibilities 

 Manage permissions for IBM Cloud Metrics Routing Provide the ability to restrict access to resources. <br>IBM is responsible for the security and compliance of the IBM Cloud Metrics Routing feature. Restrict access to IBM Cloud Metrics Routing by using Cloud IAM access policies. Define IAM policies to control which users within your account have access to manage the service and related resources in your account. <br>[Learn more about controlling access through IAM](https://cloud.ibm.com/docs/metrics-router?topic=metrics-router-iam). 
 Configure authorization policies for each target Support service to service authentication between IBM Cloud Metrics Routing and the Monitoring service. [Configure 1 or more targets and corresponding authorization policies](https://cloud.ibm.com/docs/metrics-router?topic=metrics-router-target-manage). 







 Security and regulation compliance 



Table 4. Responsibilities for security and regulation compliance

 Task IBM Responsibilities Your Responsibilities 

 Encrypt data Encrypt data in motion and at rest per compliance specifications. N/A 
 Meet security and compliance objectives Maintain controls that are commensurate to various industry compliance standards such as SOC2, PCI, HIPAA and Privacy Shield. Set up and maintain security and regulation compliance for your apps and data. 







 Disaster recovery 



Table 5. Responsibilities for disaster recovery

 Task IBM Responsibilities Your Responsibilities 

 Restore functionality for IBM Cloud Metrics Routing Automatically recover and restart IBM Cloud Metrics Routing components after any disaster event.` [Complete the disaster recovery (DR) steps for IBM Cloud Metrics Routing](https://cloud.ibm.com/docs/metrics-router?topic=metrics-router-ha_drdr-metrics-router). 
 Backup IBM Cloud Metrics Routing components Daily backup of the IBM Cloud Metrics Routing infrastructure and components. [Configure account settings](https://cloud.ibm.com/docs/metrics-router?topic=metrics-router-settings), specifically, set up a primary metadata location and a backup metadata location.
Be sure to review the “Understanding high availability and disaster recovery” topics to learn about where those services are run.





 High availability for resources 

High availability and disaster recovery aren't universal across all resources, so the type of high availability and disaster recovery that's available depends on the resource that you're using. All IBM Cloud resources are hosted in data center locations around the world. The locations that you deploy your app to can spread workloads across data centers, and you can ensure that a failover design is in place to keep your app up and running.

As an IBM classic infrastructure (non-VPC) customer, you can order bare metal or virtual server instances (VSI) and deploy to over 60 data centers that are located around the world in addition to several multizone regions (MZR) with multiple availability zones. Whether the data centers are MZRs or not, all of them maintain multiple power feeds, fiber links, dedicated generators, and battery backup to avoid a single-point-of-failure (SPOF) between zones and regions.

Even though all the data centers have multiple power feeds, several of the more mature sites, such as DAL05, DAL08, FRA02, MIL01, PAR01, SJC01, SNG01, WDC01, and WDC03, have some 1U single socket server chassis that might not accommodate a dual power feed. If you have a 1U single socket server in one of these sites, you might want to consider a 2U chassis with redundant power supplies.

For more information about the specific options for deploying your workloads, see [locations for resource deployment](https://cloud.ibm.com/docs/overview?topic=overview-locations).

For more information about particular high availability and disaster recovery practices specific to each service or infrastructure option, refer the documentation for that service.





 High availability for the network 

Except for the oldest pod that still has some single points of failure, the IBM Cloud network is designed in such a way that a single point of failure never happens. Diverse, redundant connectivity exists at every point of the network, by using diverse telecommunication providers for the same service connectivity whenever possible within each region. Diverse dark fiber providers are used to connect every edge site to all of the regional compute facilities.
Recovery of a backup server is achieved by restoring a configuration backup.





 Configuration database 

The Veeam Backup and Replication configuration database stores data about the backup infrastructure, jobs, sessions, and other configuration data. In the deployment patterns, the database instance uses the default Microsoft® SQL Server Express Edition on the same virtual server that hosts the backup server.

If you are protecting more than 500 VMs, it is recommended to move to either a Standard or Enterprise Edition. It is still recommended to run the SQL Server locally on the backup server for lowest latency and highest performance. In large deployments, for high availability and fast recovery, you can use a Microsoft SQL cluster with AlwaysOn Availability Group. Fail over to a standby backup server can be simplified by connecting to the configuration database directly without the need to restore from a configuration backup.





 Backup and replication console 

The Veeam Backup and Replication console provides access to the backup server to perform numerous tasks. The console can be only installed on a Microsoft Windows OS. To log in to the Veeam console, you must be in the Local Users group on the backup server or in a domain group that has access to the backup server. The user can perform the scope of operations that are permitted by their defined role in Veeam.

As part of the standard installation, the console is installed on the backup server. To harden the backup server, uninstall the console and install it on a jump server.





 Enterprise manager 

Veeam Backup Enterprise Manager is an optional component that is intended for distributed environments. Veeam Enterprise Manager is typically installed to enable one or more of the following features:



* Centrally manage multiple Veeam Backup and Replication servers in the distributed deployment scenario.
* The use of the Veeam RESTful API to automate data protection tasks.
* Password loss protection for backup or backup copy jobs by using encryption.
* Self-service capabilities, such as the delegation of permissions in a granular way.
* Veeam Cloud Connect Replication services and the need for a stand-alone web tool that allows a tenant to run a cloud failover plan remotely from a web browser.







 VMware backup proxy 

A VMware backup proxy is an architecture component that sits between the backup server and other components of the backup infrastructure.
For example, choose how to set up your [cluster network](https://cloud.ibm.com/docs/containers?topic=containers-plan_clusters), [protect sensitive information](https://cloud.ibm.com/docs/containers?topic=containers-encryption) such as with IBM Key Protect encryption, and configure further [security settings](https://cloud.ibm.com/docs/containers?topic=containers-securitysecurity) to meet your workload's security and compliance needs. If applicable, configure your [firewall](https://cloud.ibm.com/docs/containers?topic=containers-firewallfirewall).<br> * As part of your incident and operations management responsibilities for the worker nodes, apply the provided security patch updates.<br><br><br> 







 Disaster recovery 

IBM is responsible for the recovery of Red Hat OpenShift on IBM Cloud components in case of disaster. You are responsible for the recovery of the workloads that run the cluster and your application data. If you integrate with other IBM Cloud services such as file, block, object, cloud database, logging, or audit event services, consult those services' disaster recovery information.



Table 6. Responsibilities for disaster recovery

 Resource IBM responsibilities Your responsibilities
sudo mkdir /swlib/ansible
2. Create an inventory file called hosts that has two groups: [LHBR] and [VBR], with the IP addresses or FQDN of the Linux® hardened repository and Veeam® backup server:

touch /swlib/ansible/hosts
vi /swlib/ansible/hosts





 Examples 

A hosts file for the immutable backup.

[lhbr]
lhbr01.test.ibmcloud.local ansible_user=root

[vbr]
prodbackup ansible_host=10.38.207.157 ansible_user=sa-ansible ansible_connection=winrm ansible_winrm_server_cert_validation=ignore

A hosts file for the isolated recovery environment.

[lhbr]
lhbr01.test.ibmcloud.local ansible_user=root

[vbr]
prodbackup ansible_host=10.38.207.157 ansible_user=sa-ansible ansible_connection=winrm ansible_winrm_server_cert_validation=ignore

[vsrx]
gateway01 ansible_host=10.5.37.138 ansible_user=sa-ansible







 Using Ansible vault to encrypt files 

Ansible vault allows the encryption of values and data structures to secure sensitive data that is necessary to successfully run Ansible plays, but are not publicly visible, like passwords or private keys. Ansible automatically decrypts vault-encrypted content at run time when the key is provided.

Vault is implemented with file-level granularity, which means the files are either entirely encrypted or unencrypted. It uses the AES256 algorithm to provide symmetric encryption that is keyed to a user-supplied password. The same password is used to encrypt and decrypt content. The ansible-vault command is used to initially encrypt files, and then, is used to view, edit, or decrypt the data.
Building from source <br><br> * Continuously update the build tools, including BuildKit, and Paketo buildpacks to the latest version.<br><br><br> <br><br> * Resubmit builds to pick up fixes in the base image of your Dockerfile-based builds and to pick up operating system and runtime environment fixes in your Buildpacks-based builds.<br><br><br> 







 Disaster recovery 

IBM is responsible for the recovery of Code Engine projects and entities in case of disaster. You are responsible for the recovery of the workloads and your workload data. If you integrate with other IBM Cloud services such as file, block, object, cloud database, logging, or audit event services, consult those services' disaster recovery information.



Table 6. Responsibilities for disaster recovery
The rows are read from left to right. The resource area of comparing responsibilities is in the first column, with the responsibilities of IBM in the second column and your responsibilities in the third column.

 Task IBM responsibilities Your responsibilities 

 General <br><br> * Maintain service availability across [worldwide regions](https://cloud.ibm.com/docs/codeengine?topic=codeengine-regions) so that customers can deploy projects across zones and regions for higher DR tolerance.<br> * Provision projects with three replicas in the same region for high availability.<br> * Continuously monitor Code Engine infrastructure to ensure the reliability and availability of the service environment by site reliability engineers.<br> * Update and recover operational Code Engine entities.<br> * Back up and recover Code Engine infrastructure data, as well as your Code Engine entity configuration files.<br> * Provide integration with other IBM Cloud services such as storage providers so that data can be backed up and restored.<br><br><br> <br><br> * Set up and maintain disaster recovery capabilities for your Code Engine entities and data. For example, to prepare your project for HA/DR scenarios, follow the guidance in [High availability for Code Engine](https://cloud.ibm.com/docs/codeengine?topic=codeengine-ha-dr).
Because the replication relationship is severed, you need to set up replication anew when the original site becomes operational again.



* [Remove the replication relationship](https://cloud.ibm.com/docs/vpc?topic=vpc-file-storage-manage-replication) - In this case, you split the two shares apart and create two independent file shares. Both shares are read/write accessible and data is no longer synchronized between the two. In the [API](https://cloud.ibm.com/docs/vpc?topic=vpc-file-storage-failover&interface=uifs-failover-concepts), this operation is called a replica split operation. Removing the replica relationship is permanent, you cannot reestablish it between the two shares. However, you can create new replicas in the same zone or other zones of the same region.



Removing the replication relationship or failing over to the replica does not occur when another operation is being performed on the source or replica file share. (An example of such an operation is expanding the file share size.) The split or failover operations remains pending until the other operation completes.





 Use cases 

You can use replication to address disaster recovery concerns. Replication addresses these scenarios:



* Disaster Recovery from an application failure.

In this scenario, the application that you are running fails. The data is unaffected, but the application is not operational. You can perform a failover, where the data is quiesced and sent to another zone. The virtual server instances in that zone can be configured to take over the operation of the application while the primary servers are repaired.
* Disaster Recovery due to IBM Cloud infrastructure failure.

In this scenario, the IBM Cloud availability zone in which your application is running becomes unusable. You need to start your application in the replica location as quickly as possible and use the replicated data from the last replication event. You can initiate the Failover with the split option to make the replica volume independent. Replication is stopped.
* Facilitate regular maintenance of your applications.

Use replication to facilitate certain administrative tasks such as upgrades with greater availability. Migrate data between two zones that might be running different levels of application code.
A region could be a specific postal code area, a town, a city, a state, a group of states, or even a group of countries.

A region contains [multiple availability zones](https://www.ibm.com/cloud/data-centers/) to meet local access, low latency, and security requirements for the region.
* N/A means feature that is not applicable to that geography.
* MZR means multi-zone region. [Learn more](https://cloud.ibm.com/docs/overview?topic=overview-locationsmzr-table).







 Availability of a monitoring instance 

When you provision a monitoring instance, you select the MZR (location) where the instance is created. The region determines where the monitoring data is processed and the data is hosted.

A multizone region (MZR) consist of 3 or more availability zones that are independent from each other to ensure that single failure events affect only a single zone.

By default, each monitoring instance consist of 3 zones, one primary zone and two secondary zones:



* Each zone is located in a different data center in the region.
* The data in your primary zone is automatically replicated to the secondary zones with low latency. You don't need to do anything to enable the replication.
* When the primary zone fails, a secondary zone is elected as the primary to prevent your service instance from being affected.
* If 2 zones fail at the same time, the service is down.



The MZR architecture offers automatic failover between 2 zones, and high availability for a monitoring instance withing a region.





 Disaster recovery (DR) of the monitoring service in a region 

Disaster recovery is about surviving a catastrophic failure or loss of availability in a single location.

IBM Cloud Monitoring follows IBM Cloud requirements for [planning and recovering from disaster events](https://cloud.ibm.com/docs/overview?topic=overview-zero-downtimedisaster-recovery).

If a regional disaster occurs, consider the following information:



* The estimated recovery time for rebuilding the regional site and restoring the service at another location is 24 hours.
Understanding high availability and disaster recovery for API Connect 

IBM Cloud provides high-availability and disaster-recovery solutions to ensure that your APIs remain accessible.

API Connect instances are deployed in multi-zone regions (e.g. Dallas, Frankfurt, London, Sydney, Tokyo, and Washington). Each instance is deployed in a highly-available configuration. That is, configuration data is replicated onto more than one server and multiple instances of each component are simultaneously active. This provides resiliency in the event of failures within a datacenter or rack.

In multi-zone regions, instance members are distributed across different data centers, or zones. In single-zone regions, instance members are distributed across different hosts within the same data center.

In the case of a single zone failure in a multi-zone region or a hardware failure in any region, your instance is still accessible as it is replicated across other fully functioning servers. Such issues will be addressed by IBM Cloud Specialists in place.

In addition to the high-availability configuration, your data is snapshotted and backed up daily by the API Connect platform and stored in cross-region Cloud Object Storage buckets.

In the case of a complete region failure, the instance servers and APIS in the region may not be accessible, but the backup data will remain available. IBM will initiate internal disaster recovery procedures to restore your instance into the next nearest available region with enough capacity. IBM will also re-route managed DNS configuration to the recovered instance. This is typically completed within 24 hours of failure. When the original region is restored, IBM will migrate the instance back to the primary region.

For critical API traffic, IBM recommends that you deploy an API in at least two regions and globally load balance the regions. You can configure global load balancing with a technology such as IBM Cloud® Internet Services (Cloud Internet Services (CIS)). Cloud Internet Services (CIS) provides health checking and routing rules to ensure that API traffic is automatically routed to the nearest healthy region. For more information about global load balancing, see the [Global Load Balancer (GLB) documentation](https://cloud.ibm.com/docs/infrastructure/cis?topic=cis-global-load-balancer-glb-concepts).
Learn about the Bare Metal Restore plug-in 

Bare Metal Restore is a disaster recovery solution. You can use BMR to restore your server from a bare metal state after a disaster, such as an operating system or hardware failure, occurred. With BMR, you can quickly restore the system image from a safe, secure location that is managed by IBM Cloud®.

BMR is a Windows®-only product on physical servers. It is not available for virtual servers. Bare Metal Restores for Linux® distributions aren't supported.



  Capabilities provided 



*  Restore your system to a select point-in-time.
*  Restore your system from image or file-based backups.
*  Restore your system from backups that are stored on the IBM Cloud Backup for Classic.
*  Start a recovery transaction that you can use to restore your data without a bootable system.







  Installing the BMR plug-in 

The plug-in is installed during the Windows® Agent installation. The plug-in can be installed at the same time as the Agent or it can be installed later, by rerunning the installation with the Modify selection.





  Configuring BMR backup job 

For more information, see [Configuring BMR backup jobs](https://cloud.ibm.com/docs/Backup?topic=Backup-configureBMR).





  Restoring a BMR system volume image 

For more information, see [Restoring a BMR system volume image](https://cloud.ibm.com/docs/Backup?topic=Backup-restoreBMR).





  Downloading the user guide 

Connect to the IBM Cloud® network with [IBM Cloud® VPN](https://www.ibm.com/cloud/vpn-access) so that you can access and download the user guide from the [Downloadable IBM Cloud Backup for Classic Documentation](http://downloads.service.softlayer.com/evault/Documentation/).