vania-janet/multiturn-rag-retrieval-data huggingface dataset:
The "all-in-one" server is installed automatically when the Veeam service instance is ordered. At order time, one of the following deployment options can be selected:



* VM - Veeam components that are installed on a Windows®-based virtual machine (VM) hosted on the vCenter Server® instance.
* VSI - Veeam components installed on a Windows-based VSI.
* Bare metal - Veeam components installed on a Windows-based bare metal server.



The simple deployment scenario is suitable for small virtual environments.

For larger scale-virtual environments, the [Advanced deployment](https://helpcenter.veeam.com/docs/backup/vsphere/advanced.html?ver=120) scenario is recommended. This deployment scenario moves backup workload to dedicated backup infrastructure components. In this scenario, the backup server functions as a manager for backup jobs and for deploying and maintaining backup infrastructure components. The move from a simple to an advanced deployment scenario is a manual process.

The Veeam [distributed deployment](https://helpcenter.veeam.com/docs/backup/vsphere/distributed.html?ver=120) scenario federates multiple Veeam instances and uses Veeam Backup Enterprise Manager for centralized management and reporting for multiple backup servers. This deployment scenario is not used in these solution architectures.



 Backup server 

The backup server is a Windows-based component and is the core component in the backup infrastructure. The backup server performs the administrative activities, such as coordination, scheduling, resource allocation, and management of backup infrastructure components. Each Veeam instance has only one backup server and high availability support is not provided. The VM option provides some high availability functions due to vSphere HA and DRS. Recovery of a backup server is achieved by restoring a configuration backup.





 Configuration database 

The Veeam Backup and Replication configuration database stores data about the backup infrastructure, jobs, sessions, and other configuration data. In the deployment patterns, the database instance uses the default Microsoft® SQL Server Express Edition on the same virtual server that hosts the backup server.

If you are protecting more than 500 VMs, it is recommended to move to either a Standard or Enterprise Edition.
VMware requires the backup location to be an empty folder, so plan for your backup rotation or automation to leave the location empty for each subsequent backup job.





 NSX file-based backup 

Proper backup of all NSX components is crucial to restoring the system to its working state if a failure occurs. The design requires you to configure NSX backup through the NSX Manager backup function. For this purpose, you can configure NSX Manager to regularly perform backups to your file server. Ensure that your file server or its data is backed up correctly, and ensure the rotation of old NSX backups.





 Image-based backup of management VMs 

After you deployed your instance and deployed the Veeam backup service, configure a backup job for your management VMs. Plan to back up the following VMs with at least seven days of daily backups.



* If present, VMware SDDC Manager
* If present, Active Directory servers
* File backup server



Plan to allocate enough Veeam licenses to back up these VMs, and plan for at least 2 TB of backup storage for the VMs.





 Add-on services 

If you deploy add-on solution components into your instance, plan for the backup of these components as part of your management backup strategy.



* Zerto Virtual Replication - The Zerto Virtual Manager (ZVM) system is deployed as an IBM Cloud virtual server instance (VSI) and its backup is not supported by Veeam. If your disaster recovery strategy requires you to recover the ZVM without you perform a site failover, use your preferred Windows® backup solution to back up and restore the ZVM.
* F5® BIG-IP - F5 recommends [file-based backup of the F5 configuration](https://support.f5.com/csp/article/K13132), which you can direct to your file server.
The generated public and private keys are ECDSA with Curve P256. These keys are stored in the browser when they are added to the member's blockchain wallet so that the console can use them to manage blockchain components. However, it is recommended that customers export these keys and import them into their own key management system in case they clear their browser cache or switch browsers. Customers are responsible for the storage, backup, and disaster recovery of all keys that they export.

Because these public and private key pairs are essential to how the IBM Blockchain Platform functions, key management is a critical aspect of security. If a private key is compromised or lost, hostile actors might be able to access your data and functionality. Although you use the IBM Blockchain Platform console to generate the certificates and private keys, they are not permanently stored by the browser or the cloud database. Public and private keys are temporarily stored in the browser and added to the member's wallet so that the console can use the private key to digitally sign transactions. Customers are ultimately responsible for exporting the keys and managing their storage, backup, and disaster recovery.

If a private key is lost and cannot be recovered, you will need to generate a new private key by registering and enrolling a new identity with your Certificate Authority. You should also then remove and replace your signCert in any components or organizations where you had used the lost or corrupted identity. See [Updating an organization MSP definition](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-console-organizationsibp-console-govern-update-msp) for detailed steps.

In order to secure private keys in a production environment, the IBM Blockchain Platform includes the option to configure a

Hardware Security Module(HSM) to generate and store the private keys for a node. An HSM is an optional hardware appliance that performs cryptographic operations and ensures that the cryptographic keys never leave the HSM. You are responsible for configuring an HSM device that implements the
Enabling disaster recovery notifications 

This document contains information about the disaster recovery (DR) process for the MQ on Cloud service.

In the event of a catastrophic failure of a hosting facility, the IBM Operations is responsible for recovering deployed queue managers to an alternate location using a recent configuration backup. The following sections describe how you can configure a callback to be notified when that recovery process has been completed by the IBM Operations team. This is so you can carry out any subsequent actions required to return the restored queue manager to normal operation.

You will be guided through the process of setting up DR notifications and configuring your endpoint handler to receive the notifications. You may use your own endpoint handler; however there is a working example in the final section of this document.

The cold restore of the queue manager from a configuration backup does not retain runtime state, such as message data and channel sequence numbers. Therefore when you receive the notification event from IBM, you may need to take further action. For example, resetting channel sequence numbers to re-establish communication with other queue managers, or validating application behaviour in light of the potential for persistent messages to have been lost.

The following sections will be covered in this document:



* [How to configure notification calls](https://cloud.ibm.com/docs/mqcloud?topic=mqcloud-mqoc_dr_notificationsmqoc_dr_notify)
* [The payload](https://cloud.ibm.com/docs/mqcloud?topic=mqcloud-mqoc_dr_notificationsmqoc_dr_payload)
* [How to implement an endpoint handler](https://cloud.ibm.com/docs/mqcloud?topic=mqcloud-mqoc_dr_notificationsmqoc_dr_handler)





 How to configure notification calls 



* Within the IBM Cloud console for the service instance in which your queue managers are deployed, navigate to your service instance and select the queue manager you wish to configure DR notification calls for.
* Click the tab titled 'Notification settings'.

!
Backup strategies for IBM Power Systems Virtual Servers 

Learn more about different AIX and IBM i backup strategies for IBM® Power Systems™ Virtual Server.



 Image capture 

Image capture produces a storage FlashCopy of the logical partition (LPAR) and works on both AIX, Linux, and IBM i LPARs. You can use image capture to store VM images within your account (locally) as a part of your image catalog, or directly to [IBM Cloud Object Storage](https://cloud.ibm.com/docs/power-iaas?topic=power-iaas-capturing-exporting-vm), or both.

Importing and exporting images requires a considerable amount of processing power and network bandwidth. As a result, you can submit only one import or export request before it is queued. Typically, users import or export system disks (AIX rootvg disks) that are smaller in size (less than 1 TB) to facilitate the transfer to and from Cloud Object Storage. If your image size is greater than 1 TB, your transfer might take a long time and is prone to failure. The maximum image size that you can import or export is 10 TB.





 AIX backup strategies 

Power Systems Virtual Server users can implement any compatible agent-based backup for AIX virtual machines (VM). Veeam for AIX and IBM Spectrum Protect are two commonly used backup strategies.



* Veeam for AIX - See [Additional backup strategies](https://cloud.ibm.com/docs/power-iaas?topic=power-iaas-backup-strategiesadd-backup) for more information.
* IBM Spectrum Protect provides scalable data protection for physical file servers, applications, and virtual environments. Organizations can scale up to manage billions of objects per backup server. They can reduce backup infrastructure costs with built-in data efficiency capabilities and the ability to migrate data to tape, public cloud services, and on-premises object storage. IBM Spectrum Protect can also be a data offload target for IBM Spectrum Protect Plus, for a long-term data retention and disaster recovery. For more information, see [What can IBM Spectrum Protect do for your business?](https://www.ibm.com/us-en/marketplace/data-protection-and-recovery).
Finally, always access the application that should be recoverable by using a URL that is pointing to the virtual server IP address and never access the IP address directly.

In a recovery scenario, you can then adjust the URL to point to the recovery virtual server).





 Recovery 

To recover from a disaster by using the previously described backup environment, perform the following steps:



1. Connect to the recovery virtual server instance.
2. Start the concerned application.
3. Test whether the application starts successfully.
4. Reconfigure the DNS to map the application URL to the recovery virtual server.
5. Test whether the application is reachable from outside as expected.



Test the recovery procedure periodically to ensure its effectiveness.
Review the layout and volume and drive sizes for disks and volumes for MS SQL in all-in-one deployment:



* 2 x 1 TB SATA disks in RAID 1



* VBR database



* Database drive 250 GB
* Log drive 100 GB



* Enterprise Manager database (optional)



* Database drive 250 GB
* Log drive 100 GB







For VMware vCenter Server®, you are responsible for backing up the database. By default, VBR's backup is configured to use the RAID 6 array of the all-in-one server as the backup repository.





 Veeam Backup & Replication console 

Veeam provides a self-service interface through the Veeam Console that is Role Based Access Controls (RBAC) enabled. The solution is deployed with one local administrator by default, and the solution can be integrated post deployment with the Infrastructure Active Directory™ of the vCenter Server instance. The design allows you to define more users on the Active Directory and assign them roles such as: system admin, reporting focal, or dba. Those users can then perform actions such as initiating restores and their scope is limited to the systems or data types for which they are responsible.





 Backup repository and scale-out backup repository 

The backup repository is a storage location where Veeam keeps backup files, VM copies, and metadata for replicated VMs. This design relies on DAS (local SATA disks) to store the operational copy of customer data. Data is written to the disk in deduplicated and compressed format.

Veeam supports various scaling methods and (virtually) unlimited scaling with Scale-out backup repository functions. The scale-out backup repository is a logical entity. It groups several backup repositories (called extents) to create a pool of storage devices and systems, summarizing their capacity. This design uses scale-out backup repository functions in the backup infrastructure and one scale-out backup repository is created for a customer by default. It is recommended that no more than 300 TB is used in the scale-out backup repository with the all-in-one MS SQL Express database.

The customer backup data that is stored in the scale-out repository is encrypted with Veeam's own encryption and the encryption keys are password protected.
Disaster recovery example scenario 

Let's look at an end-to-end disaster recovery scenario. This scenario demonstrates how mirroring can be used to provide increased availability and keep applications working in a major incident that affects a full region. Two clusters were provisioned in different regions and configured for mirroring (following the information in [Mirroring setup guide](https://cloud.ibm.com/docs/EventStreams?topic=EventStreams-mirroring_setup)) by using A and B as cluster aliases.

A producer publishes records to a topic called accounting.invoices and a consumer reads the messages from that topic in cluster A.

Zoom

![Mirroring overview diagram.](https://cloud.ibm.com/docs-content/v1/content/e0c3acba1296ff980e76d5db456d53adefaabba4/EventStreams/disaster1.png)

Figure 1. Mirroring overview



 Source cluster becomes unavailable 

Let's consider what happens if a disaster occurs on the source cluster's region.

Zoom

![Disaster on source cluster diagram.](https://cloud.ibm.com/docs-content/v1/content/e0c3acba1296ff980e76d5db456d53adefaabba4/EventStreams/disaster2.png)

Figure 2. Disaster in source cluster's region

It is the responsibility of the Event Streams instance owner to determine whether the impact of the event is such as to declare a disaster. The service instance owner must coordinate the failover of applications including their reconfiguration, redeployment, and restart if necessary.





 Failing over producers 

Perform the following steps to fail over:



1. Stop the producers that were pointing to cluster A.
2. Restart the producers that were pointing to cluster B's endpoints.
3. If cluster A and the link from A to cluster B is still operational, ensure that all data was mirrored by checking that the lag on those topics on cluster B is zero.
4.
Physical storage The physical storage devices and resources, such as disks and storage devices that are used to host the virtual block, file, or Object Storage buckets. IBM owns the [IKS](https://cloud.ibm.com/docs/containers?topic=containers-responsibilities_iks) used by Schematics Client owns, if cluster provided by Customer. IBM owns if the cluster is provided by [IKS / ROKS / Kubernetes](https://cloud.ibm.com/docs/containers?topic=containers-responsibilities_iks). 
 Physical network and devices The physical network devices and resources, such as switches, routers, gateways, firewalls, and load balancers that are used to host the virtual network resources. IBM owns the [IKS](https://cloud.ibm.com/docs/containers?topic=containers-responsibilities_iks) used by Schematics Client owns, if cluster provided by Customer. IBM owns if the cluster is provided by [IKS / ROKS / Kubernetes](https://cloud.ibm.com/docs/containers?topic=containers-responsibilities_iks). 
 Facilities and data centers The physical data center buildings with power, cooling, and rooms for all the IBM Cloud physical equipment. IBM owns the [IKS](https://cloud.ibm.com/docs/containers?topic=containers-responsibilities_iks) used by Schematics Client owns, if cluster provided by Customer. IBM owns if the cluster is provided by [IKS / ROKS / Kubernetes](https://cloud.ibm.com/docs/containers?topic=containers-responsibilities_iks). 





 Incident and operations management 

Includes tasks such as monitoring, event management, high availability, problem determination, recovery, and full state backup and recovery.





 Change management 

Includes tasks such as deployment, configuration, upgrades, patching, configuration changes, and deletion.





 Identity and access management 

Includes tasks such as authentication, authorization, access control policies, and approving, granting, and revoking access.





 Security and regulation compliance 

Includes tasks such as security controls implementation and compliance certification.





 Disaster recovery
Understanding high availability and disaster recovery for App ID 

App ID is a highly available, regional service that runs in the following regions:



*  Dallas (us-south)
*  Frankfurt (eu-de)
*  London (eu-gb)
*  Osaka (jp-osa)
*  Sao Paulo (br-sao)
*  Sydney (au-syd)
*  Tokyo (jp-tok)
*  Toronto (ca-tor)
*  Washington (us-east)



In each supported region, the service exists in multiple availability zones with no single point of failure. However, because App ID is a regional service, there is no automatic cross-regional failover or cross-regional disaster recovery. If all the availability zones in a region fail, App ID becomes unavailable in that location. To establish cross-region high availability and implement a recovery plan, you need to create and maintain backup instances in multiple regions. To synchronize a service instance in one region with an instance in another region, you can use [the APIs](https://us-south.appid.cloud.ibm.com/swagger-ui//).

To learn more about the high availability and disaster recovery standards in IBM Cloud, see [How do I ensure zero downtime?](https://cloud.ibm.com/docs/overview?topic=overview-zero-downtime) or [Service Level Agreements](https://cloud.ibm.com/docs/overview?topic=overview-slas).
Use other services or tools for supporting functions Apart from using an IBM Log Analysis and IBM Cloud Object Storage instance and depending on your use case, you might want to use other supporting tools and services. For instance, you can use Apache Airflow (managed by you) for orchestrating, scheduling and automating your applications. You can also make use of IBM Secrets Manager to store the secrets required for your applications and use your automation scripts to read the secrets from the Secrets Manager before submitting your applications. You can also get creative with your application arguments, passing a token required to read the required secrets from the Secrets Manager directly from within your application. <br><br> * [Configuring Secrets Manager](https://cloud.ibm.com/docs/ContinuousDelivery?topic=ContinuousDelivery-secretsmanager)<br><br><br> 
 Use instances in alternate regions for backup and disaster recovery Currently, IBM Analytics Engine Serverless instances can be created in two regions, namely Dallas(us-south) and Frankfurt(eu-de). Although it is advisable to create your instances in the same region where your data is located, it is always useful to create a backup instance in an alternate region with the same set of configurations as your primary instance, in case the primary instance becomes unavailable or unusable. Your automations should enable switching application submissions between the two regions if required. NA 
 Use separate buckets and service credentials for application files, data files, and home instance Use the "separation of concerns" principle to distinguish the access between different resources.<br><br><br><br> * Do not store data or application files in the home instance bucket.<br> * Use separate buckets for data and application files.<br> * Use separate access credentials (IAM Key based) with restricted access to the bucket for application files and the bucket that contains your data.<br><br><br> <br><br> * [Assigning access to an individual bucket](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-iam-bucket-permissions&interface=ui)<br><br><br>
[Complete the disaster recovery (DR) steps for IBM Cloud Metrics Routing](https://cloud.ibm.com/docs/metrics-router?topic=metrics-router-ha_drdr-metrics-router). 
 Backup IBM Cloud Metrics Routing components Daily backup of the IBM Cloud Metrics Routing infrastructure and components. [Configure account settings](https://cloud.ibm.com/docs/metrics-router?topic=metrics-router-settings), specifically, set up a primary metadata location and a backup metadata location. These locations keep information about the account routing rules and the target destination data to send data. Keep a copy of the setting configuration, definitions of targets, and definitions of routes in the account. For more information, see [Collecting information about resources](https://cloud.ibm.com/docs/metrics-router?topic=metrics-router-config_report). 



[] Recovered and restarted service components will not have customer data reloaded.
High Availability and Disaster Recovery 

This section describes the High Availability (HA) provided by the IBM MQ on Cloud service, how to create a queue manager HA solution architecture and details of disaster recovery.



 High Availability of a queue manager 

Internally, IBM MQ on Cloud is deployed using a series of components that are built and deployed as containers into a Kubernetes cluster that runs in the particular geographical region you have selected. Each paid queue manager that you deploy exists in its own isolated container and has dedicated CPU, RAM and disk allocated for its use.

A Kubernetes cluster consists of multiple worker nodes (e.g. virtual machines) across which the deployed containers are distributed and each container has a health-check and liveness-check defined so that Kubernetes will automatically cause the container to be moved from one worker to another in the event of certain types of failure.

At the current time MQ on Cloud infrastructure is deployed to a single data center (also referred to as "single Availability Zone") within each Region, so each of the workers in the Kubernetes cluster is within a single data center.

The persistent state of a queue manager such as the defined queues, persistent messages that are contained in those queues and channel sequence state of queue manager channels is stored on a persistent volume that exists outside the container image, so when a queue manager container is restarted on a different worker node it still has exactly the same persistent state as when it was running on the original worker node.

![IBM MQ on Cloud high availability](https://cloud.ibm.com/docs-content/v1/content/3a120b16db5fdc3e70d071af8a63ce5515c1b096/mqcloud/images/mqoc_ha_qmgr.png)

The approach described above forms the basis for High Availability within IBM MQ on Cloud and ensures that queue managers are able to continue running even in the event of the failure of an individual worker in the cluster.





 Required solution architecture for HA configuration