vania-janet/multiturn-rag-retrieval-data huggingface dataset:
[High availability options for persistent storage in a multizone cluster](https://cloud.ibm.com/docs-content/v1/content/bce0f0917a9eea684d1b4b704ac6343a1f65f446/openshift/images/cs_storage_options_multizone.png)

Figure 1. High availability options for persistent storage in a multizone cluster



 IBM Cloud Object Storage 

Virtual Private Cloud

Classic infrastructure

Satellite



 Characteristic Description 

 Ideal data types Semi-structured and unstructured data 
 Data usage pattern Read-intensive workloads. Few or no write operations. 
 Access Via file system on mounted volume (plug-in) or via REST API from your app 
 Performance High for read operations. Predictable due to assigned IOPS and size when you use non-SDS machines. 
 Consistency Eventual 
 Durability Very high as data slices are dispersed across a cluster of storage nodes. Every node stores only a part of the data. 
 Resiliency High as data slices are dispersed across three zones or regions. Medium, when set up in a single zone only. 
 Availability High due to the distribution across zones or regions. 
 Scalability Scales automatically 
 Encryption In transit and at rest 
 Backup and recovery Data is automatically replicated across multiple nodes for high durability. For more information, see the SLA in the [IBM Cloud Object Storage service terms](http://www.ibm.com/support/customer/csol/terms/?id=i126-7857&lc=en). 
 Common use cases Geographically distributed data, static big data, static multimedia content, web apps, backups, archives, stateful sets. 
 Non-ideal use cases Write-intensive workloads, random write operations, incremental data updates, or transaction databases. 







 Portworx 

Virtual Private Cloud

Classic infrastructure



 Characteristics SDS (Portworx) 

 Ideal data types Semi-structured and unstructured data 
 Data usage pattern Read-intensive workloads. Few or no write operations. 
 Access Via file system on mounted volume (plug-in) or via REST API from your app
Performance Predictable due to assigned IOPS and size. IOPS are not shared between pods. 
 Consistency Strong 
 Durability High 
 Resiliency Medium as specific to a data center. Block storage server is clustered by IBM with redundant networking. 
 Availability Medium as specific to a data center. 
 Scalability Difficult to extend beyond the data center. You can't change an existing storage tier. 
 Encryption Encryption in transit with Key Protect 
 Backup and recovery Set up periodic snapshots, replicate snapshots, duplicate storage, back up data to IBM Cloud Object Storage, or copy data to and from pod and containers. 
 Common use cases Stateful sets, backing storage when you run your own database, or high-performance access for single pods. 
 Non-ideal use cases Multizone clusters, geographically distributed data, or sharing data across multiple app instances. 









 Persistent storage options for multizone clusters 

The following sections show the options that you have in IBM Cloud Kubernetes Service to permanently store your data in a multizone cluster and make your data highly available. You can use these options in a single zone cluster, but you might not get the high availability benefits that your app requires.

Looking to connect your cluster to an on-prem database instead? See [Setting up VPN connectivity to your cluster](https://cloud.ibm.com/docs/containers?topic=containers-vpnvpn).

The IBM Cloud Object Storage plug-in uses s3fs to manage your Object Storage buckets and objects. Object Storage, and the plug-in, works best with read-intensive workloads. If your workload is read/write intensive, you might want to try Block or File storage instead.

Zoom

![High availability options for persistent storage in a multizone cluster](https://cloud.ibm.com/docs-content/v1/content/4d497aff9e27b92fbbc2ee4e343a9ec4549cb1d7/containers/images/cs_storage_options_multizone.png)

Figure 1. High availability options for persistent storage in a multizone cluster



 IBM Cloud Object Storage 

Virtual Private Cloud

Classic infrastructure

Satellite
IBM Secure Content Store powered by IBM Cloud® Object Storage provides unparalleled agility in supporting fast, highly consistent application deployment around the world to help customers securely expand their business into new regions, from business-critical data to video archive solutions. It also offers immutable storage, immutable backup, and archive data with industry-leading security and controls for regulatory/compliance requirements.

Object Storage



* 15 minutes
* 2023-06-29



[Limiting access to a single Object Storage bucket](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-limit-access)Limiting access to a single Object Storage bucket

IBM Cloud IAM resource groups and access groups allow administrators to restrict users access to various service instances, but what if a user needs to only access a limited number of buckets within a service instance? This can be accomplished using a custom role and a narrowly tailored IAM policy.

Object Storage



* 15 minutes
* 2022-10-10



[Encrypting a bucket with Key Protect ](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-tutorial-kp-encrypt-bucket)Encrypting a bucket with Key Protect

While all data stored in Cloud Object Storage is automatically encrypted using randomly generated keys, some workloads require that the keys can be rotated, deleted, or otherwise controlled by a key management system (KMS) like Key Protect.

Object Storage



* 10 minutes
* 2022-04-07



[Building a Static Website](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-static-website-tutorial)Building a Static Website

This tutorial shows how to host a static website on IBM Cloud® Object Storage, including creating a bucket, uploading content, and configuring your new website.

Object Storage



* 15 minutes
* 2020-11-07



[Developing a web application](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-web-application)Developing a web application
Eventually, the database copies complete their replication so that all the changes to a document are present in each copy.

This "eventual consistency" model has two characteristics that affect a list of changes:



1. A change that affects a document almost certainly takes place at different times in different copies of the database.
2. The order in which changes affect documents might differ between different copies of the database, depending on when and from where the replication took place.



A consequence of the first characteristic is that, when you ask for a list of changes, it is meaningless to ask for a list of changes after a specific point in time. The reason is that the list of changes might be supplied by a different database copy, which resulted in document updates at different times. However, it is meaningful to ask for a list of changes after a specific change, which is specified by using a sequence identifier.

An extra consequence of the first characteristic is that it might be necessary to "look back" at preceding changes to agree on the list of changes. In other words, to get a list of changes, you start from the most recent change that the database copies agree on. The point of agreement between database copies is identified within IBM Cloudant by using the [checkpoint](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-replication-guidecheckpoints) mechanism that enables replication between database copies to be synchronized.

Finally, when you look at a list of changes, they might be presented in a different order in subsequent requests. The order depends on how documents were changed between different database copies. In other words, an initial list of changes might report changes A, B, then C in that order. But a subsequent list of changes might report changes C, A, then B in that order. All the changes are listed, but in a different order. This difference is because the sequence of changes that are received during replication might vary between two different copies of the database.



 What does "eventual consistency" mean for the list of changes? 

When you request a list of changes, the response you get might vary depending on which database copy supplies the list.

The since option obtains a list of changes after a specific update sequence identifier.
[Block Storage](https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-getting-started) Persistent, high-performance iSCSI storage that is provisioned and managed independently of compute instances. iSCSI-based Block Storage LUNs are connected to authorized devices through redundant multi-path I/O (MPIO) connections. 
 [File Storage](https://cloud.ibm.com/docs/FileStorage?topic=FileStorage-getting-started) Persistent, fast, and flexible network-attached, NFS-based File Storage. In this network-attached storage (NAS) environment, you have total control over your file shares function and performance. File Storage shares can be connected to up to 64 authorized devices over routed TCP/IP connections for resiliency. 
 [IBM Cloud Object Storage - Classic](https://cloud.ibm.com/docs/cloud-object-storage-infrastructure?topic=cloud-object-storage-infrastructure-about-ibm-cloud-object-storage-classic-) Information that is stored with IBM Cloud Object Storage is encrypted and dispersed across multiple geographic locations, and accessed over HTTP by using a REST API. This service uses the distributed storage technologies that are provided by the IBM Cloud Object Storage System (formerly Cleversafe). 
 [IBM Cloud Master Data Management on Cloud](https://cloud.ibm.com/docs/MDMOnCloud?topic=MDMOnCloud-mdmoc_getting_startedmdmoc_getting_started) Orchestrates your enterprise data throughout the complete information lifecycle with a highly configurable framework that supports hybrid cloud environments. Offers the rich features of an on-premises InfoSphere® Master Data Management deployment without the cost, complexity, and risk of managing your own infrastructure. 
 [IBM Cloud® Backup](https://cloud.ibm.com/docs/Backup?topic=Backup-getting-started) An automated agent-based backup system that is managed through a browser-based management utility. You can back up data between servers in one or more data centers on the IBM Cloud network. 







 Next steps
This includes holding multiple resource copies available worldwide, e.g., stored in [replicated Object Storage buckets](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-replication-overview).
* Control costs by sharing more expensive services where possible. Not every development project needs to have all services deployed as dedicated instances. Often, it is enough to share service instances - within accounts or across. Even for production environments, service instances might be shared depending on their cost/value factor and technical feasibility. This can be organized by restricting available services in an account by utilizing [private catalogs and restricting the public catalog](https://cloud.ibm.com/docs/account?topic=account-filter-account), then centrally providing instances of restricted services.
* Central management of resources on a corporate level or for a business unit. This could be assets needed for branding or centrally managed templates, base images (virtual machines, containers), and more. Again, private catalogs and the [Container Registry](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_access) are typical services.
* Make scarce resources available to more users. Sometimes, a resource type is only available in limited quantity. By sharing, more applications can benefit from it. This may require rate limiting.







 Sharing of security resources 

Often, security is managed on a corporate level with company-wide rules in place. Therefore, enforcement is managed centrally, too. This is still true with workloads moving to cloud environments. Resource sharing is at the foundation of centrally managing security as well as assessing and enforcing compliance.

Zoom

![Sharing of security-related resources](https://cloud.ibm.com/docs-content/v1/content/f84e2238288b7f0355715a4c355609ce7ac48fb0/solution-tutorials/images/solution65-resource-sharing/resource-sharing-architecture-security.svg)

Sharing of security-related resources

The above diagram shows the following scenarios:



1.
[High availability options for persistent storage in a multizone cluster](https://cloud.ibm.com/docs-content/v1/content/4d497aff9e27b92fbbc2ee4e343a9ec4549cb1d7/containers/images/cs_storage_options_multizone.png)

Figure 1. High availability options for persistent storage in a multizone cluster



 IBM Cloud Object Storage 

Virtual Private Cloud

Classic infrastructure

Satellite



 Characteristic Description 

 Ideal data types Semi-structured and unstructured data 
 Data usage pattern Read-intensive workloads. Few or no write operations. 
 Access Via file system on mounted volume (plug-in) or via REST API from your app 
 Performance High for read operations. Predictable due to assigned IOPS and size when you use non-SDS machines. 
 Consistency Eventual 
 Durability Very high as data slices are dispersed across a cluster of storage nodes. Every node stores only a part of the data. 
 Resiliency High as data slices are dispersed across three zones or regions. Medium, when set up in a single zone only. 
 Availability High due to the distribution across zones or regions. 
 Scalability Scales automatically 
 Encryption In transit and at rest 
 Backup and recovery Data is automatically replicated across multiple nodes for high durability. For more information, see the SLA in the [IBM Cloud Object Storage service terms](http://www.ibm.com/support/customer/csol/terms/?id=i126-7857&lc=en). 
 Common use cases Geographically distributed data, static big data, static multimedia content, web apps, backups, archives, stateful sets. 
 Non-ideal use cases Write-intensive workloads, random write operations, incremental data updates, or transaction databases. 







 Portworx 

Virtual Private Cloud

Classic infrastructure



 Characteristics SDS (Portworx) 

 Ideal data types Semi-structured and unstructured data 
 Data usage pattern Read-intensive workloads. Few or no write operations. 
 Access Via file system on mounted volume (plug-in) or via REST API from your app
About RAID 

RAID (Redundant Array of Independent Disks) creates a single usable data disk, where several physical disks are combined into an array for better speed and fault tolerance. Following are the three key concepts in RAID:



* Mirroring: copying data to more than one disk
* Striping: splitting data across more than one disk
* Error correction (fault tolerance): redundant data is stored to allow problems to be detected and possibly fixed.



Although many different levels of RAID exist, IBM chooses to support the most common RAID types: 0, 1, 5, 6, and 10. The different RAID levels use one or more of the following techniques, depending on the system requirements. The main purpose of using RAID is to improve reliability by using either 3Ware 9550SX Raid SATA or an Adaptec SA-SCSI RAID controller for all RAID solutions deployed.

RAID is not a backup solution. Rather, RAID creates a single usable data disk, where several physical disks are combined into an array for better speed and fault tolerance.

RAID 0 (Striped set without parity / Non-Redundant Array) Implements data striping, where file blocks are written across multiple disks in fragments that require a minimum of two disks. The advantage of a RAID 0 is that the read/write speed is dramatically increased. The more disks that are in the array, the greater the bandwidth. The disadvantage to a RAID 0 is that it has no fault tolerance. If a single drive fails, the array is broken. Also, RAID 0 does not implement error checking. So, any error is also unrecoverable. A common solution for fault tolerance is to have a drive outside of the array that is used as backup storage in a hardware failure.

RAID 1 (Mirrored set without parity) Implements data mirroring. Data is duplicated on 2 or 4 disks through a hardware raid controller and provides some fault tolerance. The array is recoverable if at least one drive does not fail. It provides faster read performance than a single drive and provides drive redundancy if a drive failure occurs. Write speed is slightly reduced.

RAID 5 (Striped set with dual distributed parity) Implements data striping at a block level and distributes parity among the disks.
* Uniqueness of data per job - does every job requires unique data inputs or are most of the jobs reusing or sharing common data?
* Runtime of the workload - if the ratio of compute time to time taken to get the data to the cloud is low, it is probably not cost-effective to run that workload on the cloud



For data requirements that are static or that have small incremental changes, mirroring the data set between on-premises and the cloud can be cost-effective by using solutions such as IBM Aspera. Data can be moved in bulk when the cluster is created and resynced on a schedule or a per-job-basis to ensure that the latest changes are available.

For jobs that have unique requirements, the data requirements can be specified as part of the job submission (bsub -f) and the files are transferred to the compute node when the job runs and the results are transferred back upon completion. In this case, the data is purely transient and is erased when the compute node is de-provisioned.

As data transfer times increase, the example solution becomes less efficient as the compute node is provisioned before the data is transferred and the results are transferred back before the node is de-provisioned.

LSF's Data Manager component addresses this by scheduling data movement independently of the job. Input files are transferred to cloud storage before any nodes are provisioned, and results are transferred back after the node is de-provisioned. Data Manager also de-duplicates transfers avoiding the same files that are being transferred time and time again. This is important when running conducting design of experiment type analyses or regression and verification runs where most of the data is common between thousands or hundreds of thousands of jobs.

To further reduce data movement, pre- and post-processing can also be conducted in the cloud. LSF's Application Center provides a web portal and restful API, which allows jobs to be submitted directly to the cloud cluster and starts remote visualization tools on the job or data in the cloud. The Application Center built in support for visualization of common output formats. It also has a client component that can be used to upload or download data directly from your laptop to the cloud cluster.
The second option would be to use the managed database-as-a-service (DBaaS) option. This option is usually easier to configure and provides built-in backups and scaling. You can find many different types of databases in the [IBM Cloud catalog](https://cloud.ibm.com/catalog?category=databasesservices). To use this option, you would need to do the following:



* Create a managed database-as-a-service (DBaaS) from the [IBM Cloud catalog](https://cloud.ibm.com/catalog?category=databasesservices).
* Store database credentials inside a secret. You will learn more on secrets in the "Store credentials in Kubernetes secrets" section.
* Use the database-as-a-service (DBaaS) in your application.









 Decide where to store application files 

Kubernetes Service provides several options to store and share data across pods. Not all storage options offer the same level of persistence and availability in disaster situations.



 Non-persistent data storage 

Containers and pods are, by design, short-lived and can fail unexpectedly. You can store data in the local file system of a container. Data inside a container cannot be shared with other containers or pods and is lost when the container crashes or is removed.





 Learn how to create persistent data storage for your app 

You can persist app data and container data on [NFS file storage](https://www.ibm.com/products/file-storage) or [block storage](https://www.ibm.com/products/block-storage) by using native Kubernetes persistent volumes.

To provision NFS file storage or block storage, you must request storage for your pod by creating a persistent volume claim (PVC). In your PVC, you can choose from predefined storage classes that define the type of storage, storage size in gigabytes, IOPS, the data retention policy, and the read and write permissions for your storage. A PVC dynamically provisions a persistent volume (PV) that represents an actual storage device in IBM Cloud. You can mount the PVC to your pod to read from and write to the PV.
You can allocate and mount more file shares across all hosts for your workloads at the time of purchase or later within the console. You can select from the available IBM Cloud Endurance file storage capacity options and performance tiers in the corresponding IBM Cloud data center. All shares are attached by using the NFS v3 protocol. IBM Cloud Endurance file storage includes provider-managed encryption of data at rest (AES-256 encryption), and is backed up by all-flash storage.

The 10 IOPS/GB performance tier is limited to a maximum capacity of 4 TB.

For more information about the shared NAS used in this solution, see [Shared storage architecture](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-storage-benefitsstorage-benefits).







 Related links 



* [vCenter Server BOM](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-vc_bom)
* [Shared storage architecture](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-storage-benefitsstorage-benefits)
: Lifecycle actions on [Key Protect](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-kpkp-lifecycle) and [Hyper Protect Crypto Services](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-hpcs-lifecycle) encryption keys can now generate bucket events in Activity Tracker.
: Buckets can now be encrypted using Hyper Protect Crypto Services in the US East region.





 18 August 2020 

New feature!
: Data can now be [archived](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-archive) using an Accelerated class that allows restoration of archived objects in under two hours for an [additional cost](https://www.ibm.com/cloud/object-storage/pricing).





 30 April 2020 

IAM updates
: Users and Service IDs can now be granted a new ObjectWriter role that allows access to writing objects, but without permissions to download objects or to list the contents of a bucket.
: Public Access can now be granted to a new ObjectReader role that allows anonymous access to reading objects, but without permissions to list the contents of a bucket.





 10 February 2020 

New feature!
: Buckets can now be created in a new [Smart Tier storage class](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-billingsmart-tier-pricing-details) that optimizes costs based on usage patterns.





 6 December 2019 

New location!
: Buckets can now be created in a Single Data Center configuration in Singapore. More information can be found in the [Select Regions and Endpoints](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-endpoints).





 15 November 2019 

Lifecycle update