vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Currently, HPCS is available in the following regions: Dallas, Washington DC, Sydney, and Frankfurt. It is possible, but not ideal, to use an HPCS instance in a different region for IBM Cloud for VMware® Regulated Workloads instances in London and Tokyo.

The VMware Regulated Workloads design has two use cases for encryption:



* SaaS Provider - VMware vSphere encryption used to encrypt the management and workload VMs. These keys are managed by the SaaS Provider.
* SaaS Consumer - An optional, application level encryption that is used to encrypt application data in addition to the VM encryption. These keys are managed by the SaaS Consumer.



In the IBM Cloud for VMware® Regulated Workloads dual region design for SaaS Provider key management, two separate HCPS instances are used along with the region-specific KMIP for VMware Service. The keys are not shared between HCPS instances.

Veeam Backup and Replication has access to the unencrypted data, and backup or replication data does not need the source encryption keys. When the backup is restored or the replica is configured, specify the target encrypted storage policy.

In the VMware Regulated Workloads dual-zone design, the following steps describe the backup and restore process between regions:



1. The protected VM is encrypted with VMware vSphere encryption in the VMware datastore.
2. A backup of the protected VM is taken.
3. VM backup files are encrypted on disk in the Protected Region Veeam Repository with Veeam encryption that is protected by a password that is stored in the Veeam database.
4. If a local restore is needed, the VMware datastore encryption storage policy is used to reencrypt the VM by using an encryption key from the protected region HPCS instance through the KMIP for VMware service.
5. Veeam network encryption is used for file and backup copy jobs to the recovery region.
6. VM backup files are encrypted on disk in the recovery region Veeam Repository with Veeam encryption.
7.
Databases <br><br> * [IBM Cloud® Hyper Protect DBaaS for MongoDB](https://cloud.ibm.com/docs/hyper-protect-dbaas-for-mongodb?topic=hyper-protect-dbaas-for-mongodb-high-availability-disaster-recovery)<br> * [IBM Cloud® Hyper Protect DBaaS for PostgreSQL](https://cloud.ibm.com/docs/hyper-protect-dbaas-for-postgresql?topic=hyper-protect-dbaas-for-postgresql-high-availability-disaster-recovery)<br><br><br> 



The references cover the steps that the service has taken to achieve high availability as well as the steps you must take. For example, consider Direct Link. The Direct Link service itself makes use of multiple zones within a region to help achieve HA. However, you Direct Link is not a redundant service at the cross-connect router (XCR), so you have the responsibility for creating redundancy through your border gateway protocol (BGP) schemas. See [High availability and disaster recovery for Direct Link](https://cloud.ibm.com/docs/dl?topic=dl-ha-dr) and [Models for diversity and redundancy in Direct Link](https://cloud.ibm.com/docs/dl?topic=dl-models-for-diversity-and-redundancy-in-direct-link) for more details.

In addition to the Financial Services Validated services in the reference architecture, see the following references for other important information:



* [How IBM Cloud ensures high availability and disaster recovery](https://cloud.ibm.com/docs/overview?topic=overview-zero-downtime)
* [Responsibilities for operating services in your deployment](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-shared-responsibilities)







 Related controls in IBM Cloud Framework for Financial Services 

The following IBM Cloud Framework for Financial Services controls are most related to this guidance.
For more information, see [Managing encryption](https://cloud.ibm.com/docs/event-notifications?topic=event-notifications-en-managing-encryption).





 Protecting your sensitive data in Event Notifications 

The data that you store in the Event Notifications instance is encrypted at rest by using a randomly generated key, which is in turn protected by Key Protect, managed by Event Notifications service.





 Deleting your data in Event Notifications 

The Event Notifications data retention policy describes how long your data is stored after you delete the service. As in IBM Cloud data retention policy you can [restore](https://cloud.ibm.com/docs/account?topic=account-resource-reclamation&interface=clirestore-resource-cli) a resource within 7 days after you delete it.



 Commands to delete data 



1. Log in to your IBM Cloud account by using IBM Cloud CLI from terminal.
2. ibmcloud resource reclamations lists your deleted instance along with the reclamation ID for it.
3. Use ibmcloud resource reclamation-delete <reclamation_id_for_instance> to permanently delete data that is related to a deleted instance.
4. If an instance is not restored, all related data is automatically deleted after the data retention period.







 Deleting Event Notifications instances 

If you no longer need an instance of Event Notifications, you can delete the service instance by using IBM Cloud CLI. You can also choose to delete your service instance by using the console. Any data that is stored related to that instance is also deleted.





 Restoring deleted data for Event Notifications 

To restore a deleted instance or to delete the instance permanently, you can use Resource Reclamations. For more information about resource reclamation, see [Using Resource Reclamations](https://cloud.ibm.com/docs/account?topic=account-resource-reclamation).
High availability and disaster recovery 

All IBM Cloud® general availability (GA) services have a Service Level Agreement of 99.99% availability. The services that provide the IBM Cloud getting started experience are GA services that are offered in these regions: Dallas, London, Frankfurt, Sydney, Tokyo, and Washington, DC. Each region has multiple different data centers for redundancy. The data for each location is kept in the data centers near that location. If all the data centers in a location fail, the services for that location become unavailable.

The services are available only on the public cloud, but the tools that are used with IBM Cloud® Continuous Delivery, which the services use, can run in dedicated or public cloud environments and in on-premises environments.

You can add resources manually before you create your app. If the resource stores data, you must ensure that the resource meets all business continuity, disaster recovery, and regulatory compliance standards and policies for your organization. For more information, see [resource-specific documentation](https://cloud.ibm.com/docs?tab=all-docs).

See [ensure zero downtime](https://cloud.ibm.com/docs/overview?topic=overview-zero-downtime) to learn more about the high availability and disaster recovery standards in IBM Cloud. For more information about high availability and disaster recovery for Continuous Delivery, see [Continuous Delivery high availability and disaster recovery](https://cloud.ibm.com/docs/ContinuousDelivery?topic=ContinuousDelivery-ha-dr). You can also find information about [Service Level Agreements](https://cloud.ibm.com/docs/overview?topic=overview-slas).
The MZR architecture offers automatic failover between zones within the region, and high availability for a auditing instance within a region.





 SZR 

The SZR architecture offers failover across 3 distinct systems within the single datacenter so that you get high availability from a system failure, but not from a datacenter failure.





 Data availability for Activity Tracker hosted event search 

When you provision an auditing instance, you select the location where the instance is created. The region determines where the auditing data is processed and the data is hosted.





 Considerations 

IBM Cloud Activity Tracker hosted event search follows IBM Cloud requirements for [planning and recovering from disaster events](https://cloud.ibm.com/docs/overview?topic=overview-zero-downtimedisaster-recovery).

When a regional disaster occurs, consider the following information:



* Data and the auditing metadata such as dashboards, alerts, views, screens, templates are backed up every 24 hours. In the event of an un-recoverable disaster, up to 24 hours of data and metadata changes to the auditing instance in the failure region can be lost.
* The estimated recovery time for rebuilding the regional site and restoring the service at another location is 24 hours.
* Due to the large volume of data, older data might not be available at the time the service is restored, as this process requires additional time to recover data from the backups.
* When the auditing instance in the DR region is available in the new location, you will be able to use it while the data is restored into the newly constructed region.
<apiKey>APIKEY</apiKey>
</ns1:authenticate>
<ns2:SoftLayer_Network_StorageInitParameters>
<id>primary Volume Id</id>
</ns2:SoftLayer_Network_StorageInitParameters>
</SOAP-ENV:Header>
<SOAP-ENV:Body>
<ns1:disasterRecoveryFailoverToReplicant>
<replicantId xsi:type="int">replica Volume ID</replicantId>
</ns1:disasterRecoveryFailoverToReplicant>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>









 Fail back to the original primary site in the UI 

After a disaster event, IBM Cloud® begins remediation work to return the impacted locations to normal operations. When the site is restored, you can initiate a Failback to the original site by clicking Storage, Block Storage for Classic in the [IBM Cloud® console](https://cloud.ibm.com/cloud-storage/block).



1. Click your active volume ("target").
2. In the upper right, click Replica and click Actions.
3. Select Failback. When the primary location is marked unavailable, the option of Disaster Recovery Failback becomes active.

During the Disaster Recovery Failover, the system is forced to fail over to the replica site and the replication relationship is severed. To be able to fail back to the original site after the site is restored to normal operations, the system must reestablish the replication bond. This operation can take considerable amount of time. Expect a message that shows the failover is in progress. Additionally, an icon appears next to your volume on the Block Storage for Classic that indicates that an active transaction is occurring. Hovering over the icon produces a window that shows the transaction. The icon disappears when the transaction is complete. During the Failback process, configuration-related actions are read-only. You can't edit any snapshot schedule or change snapshot space. The event is logged in replication history.
4.
By including the system files, you can recover from corrupted system files, or if you accidentally remove some service packs, or want to recover with a bare-metal restore. You can return to the state of the backup without having to reinstall the O/S from the installation kit, and then installing each service pack separately.

No user data file is included in System state backup. A system state backup job must be configured as a stand-alone job. There mustn't be any other data source that is included in the System State backup job.





 What happens to open files? 

By default, the base client has a state-of-the-art technology to handle most open files that are running on the OS.





 What does VSS (Volume Shadow Copy Services) do? 

The current version of the SQL Server plug-in uses VSS (Volume Shadow Copy Services) to complete backups. By using VSS, the SQL Server plug-in effectively backs up SQL databases, even SQL databases that span volumes. Backups can be completed while applications continue to write to a volume. The SQL Server plug-in provides data consistency within and across databases. VSS allows multiple backups to run at the same time.





 Where can I find information about pricing? 

For more information, see [Backup storage](https://www.ibm.com/cloud/backup-and-restore) and [IBM Cloud Backup for Classic: Pricing](https://www.ibm.com/cloud/backup/pricing).





 Can the IBM Cloud Backup for Classic capacity be increased or decreased without compromising the backups? 

You can increase or decrease the size of your vault through the [IBM Cloud console](https://cloud.ibm.com/login). The modification to the capacity does not affect the integrity of the data that is stored in the vault. For more information, see [expanding vault capacity](https://cloud.ibm.com/docs/Backup?topic=Backup-expandcapacityexpandcapacity).





 What happens when the IBM Cloud Backup for Classic capacity is exceeded? 

You can still save and retrieve your backups even if you reached the limit of the capacity that you purchased previously.
If the source snapshot is not encrypted with a customer key, the encryption of the copy remains provider-managed. If the source snapshot is protected by a customer-managed key, you must specify the customer-managed key that you want to use to encrypt the new copy with the encryption_key subproperty. See the following example.

The following example creates a backup policy in the us-south region with a copy of the backup in us-east region.

curl -X POST "$vpc_api_endpoint/v1/backup_policies/8758bd18-344b-486a-b606-5b8cb8cdd044/plans?version=2023-05-09&generation=2"
-H "Authorization: $iam_token"
-d '{
"active": true,
"attach_user_tags": [
"hourly-backups"
],
"copy_user_tags": true,
"cron_spec": "0 /2 * * ",
"deletion_trigger": {
"delete_after": 20,
"delete_over_count": 20
},
"remote_region_policies": {
"delete_over_count": 5,
"encryption_key": [
{
"CRN": "crn:v1:bluemix:public:kms:us-south:a/dffc98a0f1f0f95f6613b3b752286b87:e4a29d1a-2ef0-42a6-8fd2-350deb1c647e:key:5437653b-c4b1-447f-9646-b2a2a4cd617"
}
],
"region": [
{
"name":"us-east"
}
]
},
"name": "my-hourly-plan-2"
}'
CP-2 (g)
: Protects the contingency plan from unauthorized disclosure and modification.





 Additional IBM Cloud for Financial Services specifications 



* Documentation includes plan and playbook(s).
* The organization must provide required pre-test documentation prior to test initiation. The customer must provide the organization with what is included in pre-test documentation.







 Implementation guidance 

See the resources that follow to learn more about how to implement this control.



* [Business continuity and disaster recovery overview](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-shared-bcdr)
* [High availability overview](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-shared-high-availability)







 NIST supplemental guidance 

Contingency planning for information systems is part of an overall organizational program for achieving continuity of operations for mission/business functions. Contingency planning addresses both information system restoration and implementation of alternative mission/business processes when systems are compromised. The effectiveness of contingency planning is maximized by considering such planning throughout the phases of the system development life cycle. Performing contingency planning on hardware, software, and firmware development can be an effective means of achieving information system resiliency. Contingency plans reflect the degree of restoration required for organizational information systems since not all systems may need to fully recover to achieve the level of continuity of operations desired. Information system recovery objectives reflect applicable laws, Executive Orders, directives, policies, standards, regulations, and guidelines. In addition to information system availability, contingency plans also address other security-related events resulting in a reduction in mission and/or business effectiveness, such as malicious attacks compromising the confidentiality or integrity of information systems. Actions addressed in contingency plans include, for example, orderly/graceful degradation, information system shutdown, fallback to a manual mode, alternate information flows, and operating in modes reserved for when systems are under attack. By closely coordinating contingency planning with incident handling activities, organizations can ensure that the necessary contingency planning activities are in place and activated in the event of a security incident.
* UI
* CLI
* API







 Disaster Recovery - Fail over from an inaccessible primary volume 

If a catastrophic failure or disaster causes an outage on the primary site, customers can perform the following actions to quickly access their data on the secondary site. When the primary volume is inaccessible, you can force a failover to the remote replica.

Authorized hosts and volumes must be in the same data center. For example, you can't have a replica volume in London and the host in Amsterdam. Both must be in London or both must be in Amsterdam.

This action breaks the replication relationship and restoring the connection between the primary and the replica location can be time-consuming.



 Fail over to the replica volume in the UI 



1. Go to your list of IBM Cloud® Block Storage for Classic. From the Classic Infrastructure![Classic icon](https://cloud.ibm.com/docs-content/v1/content/14cf2f2f32b430cf4ed67ad14b3cecc010f91c45/icons/classic.svg) menu, click Storage > Block Storage for Classic.
2. Locate and click the volume name.
3. Click Actions![Actions icon](https://cloud.ibm.com/docs-content/v1/content/14cf2f2f32b430cf4ed67ad14b3cecc010f91c45/icons/action-menu-icon.svg)> Failover.
4. When the primary location is unavailable, the option of Disaster Recovery Failover becomes active. Check the box to confirm you understand the failover cannot be undone without a support case.
5. Click Yes to proceed.







 Fail over to the replica volume by using the SL CLI 

Use the following command to fail a block volume over to a specific replicant volume.

 slcli block disaster-recovery-failover --help
Usage: slcli block disaster-recovery-failover [OPTIONS] VOLUME_ID

Options:
--replicant-id TEXT ID of the replicant volume
By including the system files, you can recover from corrupted system files, or if you accidentally remove some service packs, or want to recover with a bare-metal restore. You can return to the state of the backup without having to reinstall the O/S from the installation kit, and then installing each service pack separately.

No user data file is included in System state backup. A system state backup job must be configured as a stand-alone job. There mustn't be any other data source that is included in the System State backup job.
* What happens to open files?

By default, the base client has a state-of-the-art technology to handle most open files that are running on the OS.
* What does VSS (Volume Shadow Copy Services) do?

The current version of the SQL Server plug-in uses VSS (Volume Shadow Copy Services) to complete backups. By using VSS, the SQL Server plug-in effectively backs up SQL databases, even SQL databases that span volumes. Backups can be completed while applications continue to write to a volume. The SQL Server plug-in provides data consistency within and across databases. VSS allows multiple backups to run at the same time.
* Where can I find information about pricing?

For more information, see [Backup storage](https://www.ibm.com/cloud/backup-and-restore) and [IBM Cloud Backup for Classic: Pricing](https://www.ibm.com/cloud/backup/pricing).
* Can the IBM Cloud Backup for Classic capacity be increased or decreased without compromising the backups?

You can increase or decrease the size of your vault through the [IBM Cloud console](https://cloud.ibm.com/login). The modification to the capacity does not affect the integrity of the data that is stored in the vault. For more information, see [expanding vault capacity](https://cloud.ibm.com/docs/Backup?topic=Backup-expandcapacityexpandcapacity).
* What happens when the IBM Cloud Backup for Classic capacity is exceeded?

You can still save and retrieve your backups even if you reached the limit of the capacity that you purchased previously.
High availability and disaster recovery 

IBM Watson® Natural Language Understanding is highly available within multiple IBM Cloud locations (for example, Dallas and Washington, DC). However, recovering from potential disasters that affect an entire location requires planning and preparation.

You are responsible for understanding your configuration, customization, and usage of the service. You are also responsible for being ready to re-create an instance of the service in a new location and to restore your data in any location. See [How do I ensure zero downtime? ![External link icon](https://cloud.ibm.com/docs-content/v1/content/icons/launch-glyph.svg)](https://cloud.ibm.com/docs/overview?topic=overview-zero-downtimezero-downtime) for more information.



  High availability 

Natural Language Understanding supports high availability with no single point of failure. The service achieves high availability automatically and transparently by using the multi-zone region (MZR) feature provided by IBM Cloud.

IBM Cloud enables multiple zones that do not share a single point of failure within a single location. It also provides automatic load balancing across the zones within a region.





  Disaster recovery 

In the event of a catastrophic failure in a region, complete the following steps.



*  Create a new Natural Language Understanding service instance.
*  Adjust your application software to use the new service instance URL and credentials.
*  If you use Natural Language Understanding with Knowledge Studio custom models, you will need to redeploy your custom models to the new service instance. See [Backing up and restoring data](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-backup-restorerestoremodels) to learn how to backup and restore Knowledge Studio data.