vania-janet/multiturn-rag-retrieval-data huggingface dataset:
For more information about when to use it, see [Configuring jobs to use PVC](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-backup-restorepvc). By default, the backup and restore scripts create a tmp directory in the current directory that the script uses for extracting or compressing backup files.

If you run into issues with the backup, rerun the backup command and include the --use-job parameter. This parameter instructs the backup script to use a Kubernetes job to back up ElasticSearch and MinIO in addition to Postgres, which uses a Kubernetes job by default. If the size of the data in ElasticSearch and MinIO is large and ephemeral storage is insufficient, include the --pvc option. When you do so, the script uses the persistent volume claim that is specified with the --pvc option instead of the emptyDir ephemeral storage as the temporary working directory for the job.





 Extracting files from the backup archive file 

The scripts generate an archive file, including the backup files of the services that are listed in Step 1.



1. You can extract files from the archive file by running the following command:

tar xvf <backup_file_name>







 Configuring jobs to use PVC 

The backup and restore process uses Kubernetes jobs. The jobs use ephemeral volumes that use ephemeral storage. It is a temporary storage mount on the pod that uses local storage of a node. In rare cases, the ephemeral storage is not large enough. You can optionally instruct the job to mount a Persistent Volume Claim (PVC) on its pod to use for storing the backup data. To do so, specify the --pvc option when you run the script. The scripts use emptyDir of Kubernetes otherwise.

In most cases, you don't need to use a persistent volume. If you choose to use a persistent volume, the volume must be 3 times as large as the largest backup file in the data store. The size of the data store's backup file depends on usage.
./all-backup-restore.sh restore -f backup_file_name [--pvc] [--mapping]

The --pvc parameter is optional. For more information about when to use it, see [Configuring jobs to use PVC](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-backup-restorebackup-pvc).

The --mapping parameter is optional. For more information about when to use it, see [Mapping multitenant clusters](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-backup-restorebackup-mapping).

By default, the backup and restore scripts create a tmp directory in the current directory that the script uses for extracting or compressing backup files. If you used the --use-job parameter when you backed up the data, specify it again when you restore the data. This parameter instructs the backup script to use a Kubernetes job to back up ElasticSearch and MinIO.

The gateway, ingestion, orchestrator, hadoop worker, and controller pods automatically restart.







 Using OADP to back up a cluster where Discovery is installed 

If you plan to back up and restore an entire IBM Cloud Pak for Data instance by using the IBM Cloud Pak for Data Red Hat OpenShift APIs for Data Protection (OADP) backup and restore utility, you must do some additional steps in the right order for the utility to work properly when Discovery is present.



1. Run the Discovery backup script.
2. Use the OADP backup utility to back up the cluster.
3. Delete the project. This process removes the persistent volume claims and persistent volumes that are associated with Discovery.
4. Use the OADP backup utility to restore the cluster.
5. Uninstall Discovery, and then install Discovery again on the restored cluster.

A repeat of the installation is required because the utility does not always reinstall Discovery correctly.
6. Run the Discovery restore script to restore your data.





 Backing up data manually 

Manually back up data that is not backed up by using the scripts.
Veeam is preferred for extensive feature set and its ability to backup application workloads. However, Veeam does require deploying an agent to backup workloads not hosted on VMWare. The Veeam server that is used for backups should be located in a alterative region and in the backup account as described in [Backup accounts](https://cloud.ibm.com/docs/enterprise-account-architecture?topic=enterprise-account-architecture-bcdraccounts).





 Red Hat OpenShift / Kubernetes 

Red Hat OpenShift and Kubernetes clusters on IBM Cloud can be backed up through [PX-Backup](https://cloud.ibm.com/docs/containers?topic=containers-storage_portworx_backup). The Portworks server that is used for backups should be located in a alterative region and in the backup account as described in [Backup accounts](https://cloud.ibm.com/docs/enterprise-account-architecture?topic=enterprise-account-architecture-bcdraccounts).





 Secrets Manager 

Secrets manager can be [backed up using the IBM Cloud CLI](https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-ha-drauto-backup) and customer-supplied automation. These backups should be stored in a backup instance of secrets manager in an alterative region and in the backup account as described in [Backup accounts](https://cloud.ibm.com/docs/enterprise-account-architecture?topic=enterprise-account-architecture-bcdraccounts).





 Cloud Object Storage 

Cloud Object Storage buckets can be cross regional, regional, or stored in a single data center. In addition, Cloud Object Storage Buckets can be backed up using [bucket replication](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-replication-overview).
Topic names are backed up by Event Streams, although it is recommended good practice for users to back up topic names and the configuration data for those topics.

If you have configured your Event Streams instance in a Multi-Zone Region, a regional disaster is very unlikely. However, we recommend that users do plan for such circumstances. If a user's instance is no longer available because of a disaster (and a remote DR instance is not already set up), the user should consider configuring a new instance in a new region and restoring their topics and data from backup if available. Applications can then be pointed at the new instance.
* [Backup and disaster recovery for IBM Cloud services](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-shared-bcdribm-cloud-services)
* [Related controls in IBM Cloud Framework for Financial Services](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-shared-bcdrrelated-controls)
* [Next steps](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-shared-bcdrnext-steps)



[Business continuity and disaster recovery for VPC reference architecture](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-vpc-architecture-bcdrvpc-architecture-bcdr)



* [Workloads in VPC reference architecture](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-vpc-architecture-bcdryour-workloads)



* [Workloads on virtual server instances](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-vpc-architecture-bcdryour-workloads-snapshots-for-vpc)
* [Workloads on Red Hat OpenShift on IBM Cloud](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-vpc-architecture-bcdryour-workloads-openshift)



* [Backup and disaster recovery for IBM Cloud services](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-vpc-architecture-bcdribm-cloud-services)
* [Related controls in IBM Cloud Framework for Financial Services](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-vpc-architecture-bcdrrelated-controls)
The following [IBM Cloud Framework for Financial Services controls](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-aboutframework-control-requirements) are most related to this guidance. However, in addition to following the guidance here, do your own due diligence to ensure you meet the requirements.



Table 2. Related controls in IBM Cloud Framework for Financial Services

 Family Control 

 Contingency Planning (CP) [CP-2 Contingency Plan](https://cloud.ibm.com/docs/framework-financial-services-controls?topic=framework-financial-services-controls-cp-2) <br>[CP-6 Alternate Storage Site](https://cloud.ibm.com/docs/framework-financial-services-controls?topic=framework-financial-services-controls-cp-6) <br>[CP-7 Alternate Processing Site](https://cloud.ibm.com/docs/framework-financial-services-controls?topic=framework-financial-services-controls-cp-7) <br>[CP-9 Information System Backup](https://cloud.ibm.com/docs/framework-financial-services-controls?topic=framework-financial-services-controls-cp-9) <br>[CP-10 Information System Recovery and Reconstitution](https://cloud.ibm.com/docs/framework-financial-services-controls?topic=framework-financial-services-controls-cp-10) <br>[CP-10 (2) System Recovery and Reconstitution | Transaction Recovery](https://cloud.ibm.com/docs/framework-financial-services-controls?topic=framework-financial-services-controls-cp-10.2) 







 Next steps 



* [Development processes and software integrity](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-shared-development-processes)
Veeam Backup and Replication supports this replication technique. To implement this technique, see the [steps that you must take](https://helpcenter.veeam.com/docs/backup/vsphere/encrypted_vms_backup.html?ver=120) as indicated in the Veeam® documentation.

This technique does not support the replication of virtual machines with a vTPM device.





 Related links 



* [Solution overview](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-kmip-overview)
* [Solution design](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-kmip-design)
* [Implementation and management](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-kmip-implementation)
* [Veeam and encrypted VMs](https://helpcenter.veeam.com/docs/backup/vsphere/encrypted_vms_backup.html?ver=120)
CP-6 (a) <br><br> * Check whether Cloud Object Storage bucket resiliency is set to cross region<br> * Check that any Cloud Object Storage buckets used by Activity Tracker Event Routing are configured as cross-region<br> * Check that Hyper Protect Crypto Services has failover units in at least 2 different regions that are Financial Services Validated<br><br><br> 
 CP-6 (b) <br><br> * Check whether Cloud Object Storage bucket resiliency is set to cross region<br> * Check that any Cloud Object Storage buckets used by Activity Tracker Event Routing are configured as cross-region<br> * Check that Hyper Protect Crypto Services has failover units in at least 2 different regions that are Financial Services Validated<br><br><br> 







 NIST supplemental guidance 

Alternate storage sites are sites that are geographically distinct from primary storage sites. An alternate storage site maintains duplicate copies of information and data in the event that the primary storage site is not available. Items covered by alternate storage site agreements include, for example, environmental conditions at alternate sites, access rules, physical and environmental protection requirements, and coordination of delivery/retrieval of backup media. Alternate storage sites reflect the requirements in contingency plans so that organizations can maintain essential missions/business functions despite disruption, compromise, or failure in organizational information systems.
* [IBM Cloud for Financial Services profile](https://cloud.ibm.com/docs/framework-financial-services-controls?topic=framework-financial-services-controls-cp-8.2scc-fs-cloud-profile)



[CP-9 - Information System Backup](https://cloud.ibm.com/docs/framework-financial-services-controls?topic=framework-financial-services-controls-cp-9cp-9)



* [Control requirements](https://cloud.ibm.com/docs/framework-financial-services-controls?topic=framework-financial-services-controls-cp-9control-requirements)
* [Implementation guidance](https://cloud.ibm.com/docs/framework-financial-services-controls?topic=framework-financial-services-controls-cp-9implementation-guidance)
* [IBM Cloud for Financial Services profile](https://cloud.ibm.com/docs/framework-financial-services-controls?topic=framework-financial-services-controls-cp-9scc-fs-cloud-profile)
* [NIST supplemental guidance](https://cloud.ibm.com/docs/framework-financial-services-controls?topic=framework-financial-services-controls-cp-9nist-supplemental-guidance)



[CP-9 (1) - Testing for Reliability / Integrity](https://cloud.ibm.com/docs/framework-financial-services-controls?topic=framework-financial-services-controls-cp-9.1cp-9.1)



* [Control requirements](https://cloud.ibm.com/docs/framework-financial-services-controls?topic=framework-financial-services-controls-cp-9.1control-requirements)



[CP-10 - Information System Recovery and Reconstitution](https://cloud.ibm.com/docs/framework-financial-services-controls?topic=framework-financial-services-controls-cp-10cp-10)
Configuring Oracle DB backups 

To protect data in Oracle® databases, install the Oracle® plug-in with the IBM Cloud® Backup for Classic agent on the Oracle® database server. Then, you can add and run backup jobs that specify which databases to back up, and where to save the backup data. The plug-in provides ARCHIVELOG-based, non-RMAN backups of whole online database instances. All nontemporary table spaces and instance parameter files are automatically backed up. Full and partial databases can be restored through normal user-managed Oracle recovery mechanisms. Database passwords are encrypted for enhanced security over script-based methods.

The Oracle plug-in performs what the Oracle Corporation deems an “inconsistent” whole database backup that requires that the database is run in ARCHIVELOG mode. During a live backup, any changes to the database are written to archive logs. The database administrator must ensure that the database is in ARCHIVELOG mode.



 Starting Cloud Backup Portal 

You need to be connected to the IBM Cloud® private network to be able to start the Portal.



1. Log in to the [IBM Cloud console](https://cloud.ibm.com/login). From the menu ![Menu icon](https://cloud.ibm.com/docs-content/v1/content/921d024aca5ff45536208840a52f864567eb4068/icons/icon_hamburger.svg), select Classic Infrastructure![Classic icon](https://cloud.ibm.com/docs-content/v1/content/921d024aca5ff45536208840a52f864567eb4068/icons/classic.svg).
2. Click Storage > Cloud Backup to display the backup services.
3. Select the instance name of the IBM Cloud Account.
4. Click View backup portal to start the portal in your browser.

If the Portal doesn't start, you might have a problem with your VPN connection. You might also see a message that says that the form you’re sending isn’t secure.
In the following example, RTO and RPO requirements are homogeneous. This helps in the design and implementation of the application's DR strategy.



Table 1. Example RTO and RPO requirements

 Recovery classes RTO RPO 

 Continuous availability <= 1 hour <= 1 hour 
 Advanced recovery > 1 hour - <= 72 hours <2 hours - <72 hours 
 Standard recovery > 72 Hr Last backup 
 No recovery n/a n/a 



The resiliency options, proposed profiles, and associated tables are presented so that you can define your application's disaster recovery requirement levels. Informaton stated is not a warranty and IBM® will not issue credits for failure to meet an objective. These RTO and RPO [examples](https://cloud.ibm.com/docs/overview?topic=overview-bcdr-app-recovery) are presented as a reference to guide you on additional steps that can be taken to achieve those levels of resiliency. Refer to the Service Level Agreements (SLAs) for any commitments and credits that are issued upon failure to meet any committed SLAs.

When planning for a migration on cloud, it could be the right moment to review the real business needs and the DR requirements (RPO/RTO). Not all of the workloads need the highest service levels. Leveraging the on-demand nature of the cloud, you can fit business requirements of a subset of applications with a lower class.

Because the final solution depends on the combination of cloud services consumed, when selecting services on cloud, you must verify that they are provided with services levels that match your needs. This applies to every aspect of the managed cloud services, such as backup policies and frequency, committed RPO/RTO if you used for example a database as a service option, and so on. This could generate relevant savings in the costs of the DR solution.





 Resources for disaster recovery implementations 

It is important to have a clear view of the ownership or control of the DR implementation. Check [Shared responsibilities](https://cloud.ibm.com/docs/overview?topic=overview-shared-responsibilities) for a detailed outline for the responsibilities and roles for each service offered by IBM Cloud.
Each region has a vCenter Server instance for the management of the ESXi hosts within the region. Each instance is a separate SSO domain. The IBM Cloud design uses a vCenter Server Appliance (VCSA) with an embedded Platform Services Controller (PSC). The use of one vCenter per region with separate SSO domains provides service isolation.

Within each region, vSphere High Availability (HA) provides availability of the appliance. For business resiliency, image level and file level backups of the appliance are recommended. These backups are stored locally in the region to allow for quick restore.

It is recommended to also configure Veeam backup copy jobs. If a failure of the Veeam Repository server occurs, a copy of the appliance backup at the other region is available.

Veeam is used to take an image level backup of the appliance. However, this type of backup does not quiesce the database and in rare occasions the image might not be usable. Therefore, a scheduled file level backup is also recommended.

To restore the VCSA image, configure the Veeam Backup & Replication Manager to attach to an ESXi host in the management cluster first, as the connection through the VCSA is not available. Within the Veeam UI restore wizard, select Entire VM, select the VCSA, then select Restore to a new location and then select the ESXi server.

It is good practice to back up the vDS switch configuration as part of the backup, which can be achieved through a script that is triggered by the backup job. For more information, see [PowerCLI Gather complete Virtual Distributed Switch (VDS) information from VMware vCenter](http://vcloud-lab.com/entries/powercli/powercli-gather-complete-virtual-distributed-switch-vds--information-from-vmware-vcenter).

The VCSA supports a file-based backup and restore mechanism that helps with the recovery of the appliance after a failure. It is only possible to do a file-based backup by using the VCSA management user interface (UI).
* [Backup and disaster recovery for IBM Cloud services](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-vpc-architecture-bcdribm-cloud-services)
* [Related controls in IBM Cloud Framework for Financial Services](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-vpc-architecture-bcdrrelated-controls)
* [Next steps](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-vpc-architecture-bcdrnext-steps)



[Business continuity and disaster recovery for Satellite reference architecture](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-satellite-architecture-bcdrsatellite-architecture-bcdr)



* [BCDR in your Satellite location](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-satellite-architecture-bcdryour-workloads)
* [Backup and disaster recovery for IBM Cloud services](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-satellite-architecture-bcdribm-cloud-services)
* [Related controls in IBM Cloud Framework for Financial Services](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-satellite-architecture-bcdrrelated-controls)
* [Next steps](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-satellite-architecture-bcdrnext-steps)







 High availability 

[High availability overview](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-shared-high-availabilityshared-high-availability)
Floating IP for VPC floatingIpId: <fip-id> 
 Flow Logs for VPC flowLogCollectorId: <flc-id> 
 Image Service for VPC imageId:<image-id> 
 Load Balancer for VPC loadBalancerId: <load-balancer-id> 
 Network ACL networkAclId: <nacl-id> 
 Placement Group for VPC placementGroupId: <placement-group-id> 
 Public Gateway for VPC publicGatewayId: <pgw-id> 
 Security Group for VPC securityGroupId: <default-sec-grp-id> 
 Snapshots snapshotId: <snapshot-id> 
 SSH Key for VPC keyId:<key-id> 
 Subnet subnetId: <subnet-id> 
 Virtual Private Endpoint for VPC endpointGatewayId:<endpoint-gateway-id> 
 Virtual Private Cloud vpcId: <vpc-id> 
 Virtual Server for VPC instanceId: <instance-id> 
 VPN for VPC vpnGatewayID: <vpn-gateway-id>
"href": "https://us-south.iaas.cloud.ibm.com/v1/vpcs/c8b8fa2d-ccf7-4f42-9d38-df6d123c867d",
"id": "c8b8fa2d-ccf7-4f42-9d38-df6d123c867d",
"name": "my-vpc",
"resource_type": "vpc"
}
}
],
"name": "my-share-3",
"profile": {
"family": "defined-performance",
"href": "https://us-south.iaas.cloud.ibm.com/v1/share/profiles/dp2",
"name": "dp2",
"resource_type": "share_profile"
},
"replication_role": "source",
"replication_status": "active",
"replication_status_reasons": [],
"resource_group": {
"crn": "crn:[...]",
"href": "https://resource-controller.cloud.ibm.com/v2/resource_groups/678523bcbe2b4eada915d32640909956",
"id": "678523bcbe2b4eada915d32640909956",
"name": "Default"
},
"resource_type": "share",
"size": 4800,
"user_tags": [],
"zone": {
"href": "https://us-south.iaas.cloud.ibm.com/v1/regions/us-south/zones/us-south-1",
"name": "us-south-1",
"resource_type": "zone"
}
}
"ipAddress": "192.168.0.1",
"networkType": "public"
}
},
"resource": {
"attributes": {
"accountId": "67db3d7ff3f34220b40e2d81480754c9",
"resource": "NEWRESOURCE",
"vpcId": "NEWRESOURCE",
"resourceGroupId": "d272ce832535498e9142b557ff8638ed",
"region": "au-syd",
"serviceName": "is",
"resourceType": "vpc"
}
},
"subject": {
"attributes": {
"scope": "ibm openid",
"token.account.bss": "67db3d7ff3f34220b40e2d81480754c9",
"id": "IBMid-550009FB7N"
}
}
},
"responseData": {
"decision": "Deny",
"isEnforced": true
}
}
Show more

You can then use the same Trace ID or Reference ID to look at the specific VPC Infrastructure service event in your Activity Tracker instance.

{
"action": "is.vpc.vpc.create",
"outcome": "failure",
"message": "Virtual Private Cloud: create vpc -failure",
"severity": "critical",
"dataEvent": false,
"logSourceCRN": "crn:v1:bluemix:public:is:au-syd:a/67db3d7ff3f34220b40e2d81480754c9::vpc:",
"saveServiceCopy": true,
"initiator": {
"id": "IBMid-ABCDEFGH",