vania-janet/multiturn-rag-retrieval-data huggingface dataset:
One recovery unit is allocated in the same region as the operational crypto units, and the other is allocated in a backup region. When you run the ibmcloud tke auto-init or ibmcloud tke auto-mk-rotate command, a random master key value is generated in one of the recovery crypto units and then securely exported to the other crypto units (including failover crypto units if any) for the service instance.

The sole purpose of the recovery crypto units is to save a backup copy of the master key value. The recovery crypto units are not used when running operational workloads. If the current master key value is lost or destroyed, you can recover the master key value from either of the recovery crypto units by using the ibmcloud tke auto-mk-recover command. This command copies the value in the current master key register of a recovery crypto unit to the current master key register of other crypto units.

Currently, only the us-south and us-east regions are enabled with the recovery crypto units, which means, when a service instance is provisioned in either region, you are by default enabled with the option to back up your master keys in the recovery crypto units located in both regions. For more information about supported regions, see [Regions and locations](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-regions).

For detailed instructions on recovering the master key, see [Recovering a master key from a recovery crypto unit](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-recover-master-key-recovery-crypto-unit).







 Initializing service instances by using key part files 

You can also initialize your service instance by using master key parts that you create and store in files on your local workstation. You can use this approach regardless of whether or not your service instance includes recovery crypto units. In this case, the workstation key files serve as a backup copy of your master key value. To complete the initialization in this approach, you also need to use the IBM Cloud TKE CLI plug-in.



 Understanding master key parts
* [Your responsibilities when you're using Container Registry](https://cloud.ibm.com/docs/Registry?topic=Registry-bc-drbc-dr-responsibilities)
* [Disaster recovery strategy](https://cloud.ibm.com/docs/Registry?topic=Registry-bc-drbc-dr-strategy)
* [Locations for service availability](https://cloud.ibm.com/docs/Registry?topic=Registry-bc-drbc-dr-locations)
* [Frequently asked questions about disaster recovery](https://cloud.ibm.com/docs/Registry?topic=Registry-bc-drbc-dr_faq)



* [Does the service replicate the data?](https://cloud.ibm.com/docs/Registry?topic=Registry-bc-drbc-dr_replicate_data)
* [What data is backed up or replicated?](https://cloud.ibm.com/docs/Registry?topic=Registry-bc-drbc-dr_backup)
* [Are users required to replicate the data?](https://cloud.ibm.com/docs/Registry?topic=Registry-bc-drbc-dr_client)









 Your responsibilities 

[Your responsibilities](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_responsibilitiesregistry_responsibilities)



* [Incident and operations management](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_responsibilitiesincident-and-ops)
* [Change management](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_responsibilitieschange-management)
* [Identity and access management](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_responsibilitiesiam-responsibilities)
* [Security and regulation compliance](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_responsibilitiessecurity-compliance)
Recovering from a regional disaster 

IBM Cloud Activity Tracker hosted event search is a highly available, multi-tenant, regional service. In this topic, you can learn more about Activity Tracker availability and disaster recovery strategies, what you need to plan for, and what you need to do if you need to recover from a regional disaster.



 Prereqs 



* Learn about your responsibilities in the event of a disaster. See [Disaster and recovery](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-shared-responsibilitiesdisaster-recovery).
* Learn about disaster recovery (DR). See [Overview](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-ha_dr_stepsha_dr_steps_ov).







 Planning for a DR scenario 

The steps in this section assume that you have a list of all IBM Cloud Activity Tracker instances for each resource group and region in the account; access reports for each instance; and a current export of the configuration resources for each instance.

Consider using access groups to manage permissions to work with auditing instances. In the event of a DR situation, you can quickly add permissions for users and service IDs in each access group and accelerate recovery time.



* To get the access report for an instance, go to the Observability UI, and in the Activity Tracker section, select 1 instance. Select the 3 dots icon and click Access report. Then, save it in a version control tool.

The report includes a list of users, access groups, and service IDs that have access to the selected resource. You must have the Administrator role on the selected resource to view the report.

You can download the report as CSV or as JSON.

Generate an access report everytime permissions to work with auditing instances change in the account.

You can use terraform to manage IAM permissions.
We recommend that you verify possible changes or additional features that might apply to your environment.



 General disaster recovery strategy 

The approach to defining your DR strategy needs to be systematic and start with the application or service, which is defined as a set of compute resources, for example Kubernetes apps, virtual machines, services, and more, that make up a business application.

While a holistic approach might be wanted, the reality is that each business application (cloud service) is independent, with its own

recovery time objective(RTO) andrecovery point objective(RPO) requirements, which for many customers is expressed in the form of a set of service classes.

Because each business application has a unique set of compute, services, and other resources, each one needs to be reviewed to make sure that the strategy and requirements for DR for that app are understood, documented, and implemented before going to production. To build the framework that will drive that analysis and work, we outlined profiles for continuous availability and advanced recovery applications with common sets of resources to create the scaffolding that application teams can use for their own applications. For more information, see [Designing an architecture for your application resiliency objectives](https://cloud.ibm.com/docs/overview?topic=overview-bcdr-app-recovery).



 DR strategy options 

There are many options to implement DR solutions. For the sake of simplicity, we can group the different options in three major categories:

Active/Passive
: Active/Passive options are based on keeping the full application stack active in one location, while another application stack is deployed in a different location but kept idle or shut down. In the case of prolonged unavailability of the primary site, the application stack is activated in the backup site. Often that requires the restoring of backups that are taken in the primary site. This approach is not recommended when losing data can be a problem, for example when the RPO is less than a few hours or when the availability of the service is critical and the RTO objective is less than a few hours.

Active/Standby
: In the Active/Standby case, the full application stack is active in both the primary and backup location. However, user's transactions are served by the primary site only.
: A CLI plug-in working with IBM Cloud CLI for you to initialize service instances. Depending on whether recovery crypto units are assigned to your instance, the plug-in provides two ways for instance initialization: by using recovery crypto units and by using key part files.

Operational crypto unit
: Each service instance is composed of multiple operational crypto units. The operational crypto units are located in different availability zones of the same region for high availability. They are used to manage encryption keys and perform cryptographic operations. The number of crypto units that you specify when you create your instance is the number of operational crypto units.

Recovery crypto unit
: The purpose of recovery crypto units is to generate a random master key value and to save a backup copy of the master key value. You can use recovery crypto units to load the master key and restore the master key when it is destroyed or lost.

Currently, recovery crypto units are enabled only in the region of Dallas (us-south) and Washington DC (us-east). If you create your instance in either of the two regions, two recovery crypto units are automatically assigned to your instance without extra costs: one is in the us-south; the other is in the us-east.

If smart cards are used to load the master key, the recovery crypto units are not applicable and can be ignored. The backup of the master key relies on the backup of the smart cards in that case.

Failover crypto unit
: Failover crypto units back up the operational crypto units in another region, which includes keystores that store encryption keys. When a regional disaster occurs, you can use failover crypto units to ensure production workloads and avoid data loss.

Currently, failover crypto units are available only in the region of Dallas (us-south) and Washington DC (us-east). If you create your instance in either of the two regions, you can choose whether to enable the failover crypto units with [extra charges](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-faq-pricing).
You can use and manage the cross-regional snapshot in the target region independently from the parent volume or the original snapshot.

If the source snapshot is not encrypted with a customer key, the encryption of the copy remains provider-managed. If the source snapshot is protected by a customer-managed key, you must specify the customer-managed key that you want to use to encrypt the new copy.

Only one copy of the backup snapshot can exist in each region. You can't create a copy of the backup snapshot in the source (local) region.

Creating a cross-regional copy affects billing. You're charged for the data transfer and the storage consumption in the target region separately.







 User roles for backup policies 

Depending on your assigned role as a backup user, you can create and administer backup policies. Table 1 describes these capabilities.



Table 1. Backup user roles for backup policies

 Backup user role What you can do: 

 Viewer, Operator <br><br> * List all backup policies.<br> * View details of a backup policy.<br> * View a backup plan.<br><br><br> 
 Editor, Administrator <br><br> * Organize and manage snapshots by using tags.<br> * Associate backup policies to volumes by using user tags.<br> * Create, update, and delete backup policies and plans.<br> * Specify CRON format to schedule snapshot creation.<br><br><br> 





Table 2. Volume user roles for backup policies

 Volume user role What you can do: 

 Editor <br><br> * Create data volumes with user tags.<br> * View volume details with user tags.<br> * List volumes filtered by tag.<br> * Update user tags post volume creation.<br><br><br> 





Table 3. Snapshot user roles for backup policies

 Snapshot user role What you can do: 

 Editor <br><br> * Create snapshots with user tags.<br> * View snapshot details with tags.<br> * List snapshots filtered by tag.<br> * Update user tags post snapshot creation.<br><br><br>
High availability and disaster recovery 

VPC+ Cloud Migration is a third-party service that is managed by Wanclouds. This service is highly available, and it has no backup and restore requirements for disaster recovery.



  High availability 

VPC+ Cloud Migration application deployment is based on Kubernetes, which is available both as a single and multi-master. If one master goes down, Kubernetes shifts the control to another master. Furthermore, POD replicas are deployed across different nodes. This service achieves high availability automatically and is apparent to the customer.





  Disaster recovery 

For this service, there is no action that you need to take to prepare for an event of a catastrophic failure in a region. Wanclouds follows best practices for disaster recovery and automatically recovers and restarts after any disaster event.
System and Communications Protection (SC) [SC-8 Transmission Confidentiality and Integrity](https://cloud.ibm.com/docs/framework-financial-services-controls?topic=framework-financial-services-controls-sc-8) <br>[SC-8 (1) Transmission Confidentiality and Integrity | Cryptographic Protection](https://cloud.ibm.com/docs/framework-financial-services-controls?topic=framework-financial-services-controls-sc-8.1) <br>[SC-12 Cryptographic Key Establishment and Management](https://cloud.ibm.com/docs/framework-financial-services-controls?topic=framework-financial-services-controls-sc-12) <br>[SC-12 (2) Cryptographic Key Establishment and Management | Symmetric Keys](https://cloud.ibm.com/docs/framework-financial-services-controls?topic=framework-financial-services-controls-sc-12.2) <br>[SC-12 (3) Cryptographic Key Establishment and Management | Asymmetric Keys](https://cloud.ibm.com/docs/framework-financial-services-controls?topic=framework-financial-services-controls-sc-12.3) <br>[SC-13 Cryptographic Protection](https://cloud.ibm.com/docs/framework-financial-services-controls?topic=framework-financial-services-controls-sc-13) <br>[SC-28 Protection of Information At Rest](https://cloud.ibm.com/docs/framework-financial-services-controls?topic=framework-financial-services-controls-sc-28) <br>[SC-28 (1) Protection of Information at Rest | Cryptographic Protection](https://cloud.ibm.com/docs/framework-financial-services-controls?topic=framework-financial-services-controls-sc-28.1) 







 12. Implement business continuity and disaster recovery 

Requirement: Implement a system for business continuity and disaster recovery (BCDR). Define Recovery Point Objective (RPO), Recovery Time Objective (RTO), and Maximum Tolerable Downtime (MTD) metrics for essential business functions.
Or a large-scale network failure might mean systems in a cluster can't be contacted, even though they continue to work correctly.

Addressing your HA or DR requirements often begins by simplifying the problem into more generic requirements. When you identify your requirements, you can apply the tools and features that help solve the generic needs. Together, the tools and features can address your HA or DR requirements.

Different tools and features provide different levels of protection. The different features might be more or less suitable for your specific HA or DR requirement.

IBM Cloudant provides a number of tools and features that address general requirements:



1. Data redundancy within a single region, also known as [In-region automatic data redundancy](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-disaster-recovery-and-backupin-region-automatic-data-redundancy).
2. Cross-region data redundancy and failover, also known as [Cross-region redundancy for disaster recovery](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-disaster-recovery-and-backupcross-region-redundancy-for-disaster-recovery).
3. Point-in-time backup for point-in-time restores that use "traditional" [database backup and recovery](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-disaster-recovery-and-backupdatabase-backup-and-recovery).







 In-region automatic data redundancy 

Within a single IBM Cloudant account, data is stored in triplicate by using internal and automatic processes. You don't need to do anything to enable this internal data replication.

In-region data redundancy enables high availability protection. Specifically, in-region data redundancy provides protection for your data against hardware failure within the region. When a hardware unit within the region fails, only the copy of your data that is stored on that unit is no longer available. Your applications stay usable because IBM Cloudant automatically routes requests to the copies of your data that is still available on other hardware units within the region.
Disaster recovery 

IBM is responsible for the recovery of IBM Cloud Kubernetes Service components in case of disaster. You are responsible for the recovery of the workloads that run the cluster and your application data. If you integrate with other IBM Cloud services such as file, block, object, cloud database, logging, or audit event services, consult those services' disaster recovery information.



Table 6. Responsibilities for disaster recovery

 Resource IBM responsibilities Your responsibilities 

 General <br><br> * Maintain service availability across [worldwide locations](https://cloud.ibm.com/docs/containers?topic=containers-regions-and-zones) so that customers can deploy clusters across zones and regions for higher DR tolerance.<br> * Provision clusters with three replicas of master components for high availability.<br> * In multizone regions, automatically spread the master replicas across zones.<br> * Continuously monitor to work to ensure the reliability and availability of the service environment by site reliability engineers.<br> * Update and recover operational IBM Cloud Kubernetes Service and Kubernetes components within the cluster, such as the Ingress application load balancer and file storage plug-in.<br> * Back up and recover data in etcd, such as your Kubernetes workload configuration files<br> * Provide the optional [worker node Autorecovery](https://cloud.ibm.com/docs/containers?topic=containers-health-monitorautorecovery).<br> * Provide the ability to integrate with other IBM Cloud services such as storage providers so that data can be backed up and restored.<br><br><br> <br><br> * Set up and maintain disaster recovery capabilities for your apps and data. For example, to prepare your cluster for HA/DR scenarios, follow the guidance in [High availability for IBM Cloud Kubernetes Service](https://cloud.ibm.com/docs/containers?topic=containers-ha). Note that persistent storage of data such as application logs and cluster metrics are not set up by default.<br><br><br> 







 Applications and data 

You are completely responsible for the applications, workloads, and data that you deploy to IBM Cloud.
Recovery of configuration Recovery will be conducted in the original data center after the infrastructure is available. If long-term outage occurs, offsite recovery is conducted. 
 Recovery of workloads Restore capabilities are available in normal operations. For configuration restores, customer restore services will be provided after the infrastructure is available. If an offsite recovery is required, IBM works with the customer to help recover. Restore systems from the configured backup jobs. 







 Disaster recovery for VMware Solutions offerings (other than VMware Shared) 

The following table describes the responsibilities that are related to disaster recovery for all VMware Solutions offerings, other than VMware Shared.



Table 9. Responsibilities for disaster recovery for VMware Solutions offerings (other than VMware Shared)
The rows are read from left to right. The first column describes the task that a customer or IBM might be responsible for. The second column describes IBM responsibilities for that task. The third column describes your responsibilities as the customer for that task.

 Task IBM responsibilities Your responsibilities 

 Business continuity and Disaster Recovery (DR) Provide automated provision and integration for third-party services, such as Veeam and Zerto. Provision third-party solutions such as Veeam and Zerto, or other solutions of your choice, along with the VMware Solutions instance. Configure these solutions to meet your business continuity and DR requirements for your workload. 









 Preserving critical configuration 

Configuration changes can impact the ability to use the VMware Solutions console to view and manage instances. Preserving certain configurations and settings is critical for the VMware Solutions automation to continue to function properly.

You must manage the VMware Solutions components that are created in your IBM Cloud account only from the VMware Solutions console. For more information, see:



* [vCenter Server overview](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-vc_vcenterserveroverview)
* [Expanding and contracting capacity for vCenter Server instances](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-vc_addingservers)
If the source snapshot is not encrypted with a customer key, the encryption of the copy remains provider-managed. If the source snapshot is protected by a customer-managed key, you must specify the customer-managed key that you want to use to encrypt the new copy with the encryption_key subproperty. See the following example.

The following example creates a backup policy in the us-south region with a copy of the backup in us-east region.

curl -X POST "$vpc_api_endpoint/v1/backup_policies/8758bd18-344b-486a-b606-5b8cb8cdd044/plans?version=2023-05-09&generation=2"
-H "Authorization: $iam_token"
-d '{
"active": true,
"attach_user_tags": [
"hourly-backups"
],
"copy_user_tags": true,
"cron_spec": "0 /2 * * ",
"deletion_trigger": {
"delete_after": 20,
"delete_over_count": 20
},
"remote_region_policies": {
"delete_over_count": 5,
"encryption_key": [
{
"CRN": "crn:v1:bluemix:public:kms:us-south:a/dffc98a0f1f0f95f6613b3b752286b87:e4a29d1a-2ef0-42a6-8fd2-350deb1c647e:key:5437653b-c4b1-447f-9646-b2a2a4cd617"
}
],
"region": [
{
"name":"us-east"
}
]
},
"name": "my-hourly-plan-2"
}'
Show more
vSphere encryption operates at the VM disk level, and so it can prevent data exposure if loss of physical disk drives or loss of VM disks occurs. Some backup and replication technologies cannot back up or replicate effectively because the provided data is encrypted.

vSphere encryption is not compatible with VMware HCX or Zerto. vSphere encryption is compatible with cross-vCenter vMotion when you use vSphere 7.0 or later. vSphere encryption is compatible with vSphere replication when you use vSphere 6.7u1 or later. When properly configured, Veeam Backup and Replication is compatible with vSphere encryption.





 More considerations 

When either type of encryption is enabled in your vSphere cluster, VMware creates an extra key to encrypt your ESXi core dumps. These core dumps might contain sensitive data such as key management credentials, encryption keys, or decrypted data. For more information, see [vSphere VM encryption and core dumps](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.security.doc/GUID-63728E8B-810D-418B-B1AA-6A0A2F92AABE.html).

When KMIP for VMware is used together with vSAN encryption or vSphere encryption, several layers of key protection exist.

If you plan to rotate keys, review the following information about the levels at which the keys can be rotated:



* Level 1 - Your customer root key (CRK) protects all VMware keys. These keys can be rotated in the IBM Key Protect or Hyper Protect Crypto Services instance that is associated with your KMIP for VMware instance. When these keys are rotated in the IBM Key Protect instance, KMIP for VMware handles the new CRK automatically and no operation is needed in VMware or vCenter Server.
* Level 2 - KMIP for VMware uses your CRK to protect the keys that it generates and distributes to VMware. VMware considers these keys to be key encrypting keys (KEKs).