vania-janet/multiturn-rag-retrieval-data huggingface dataset:
99.999999999% (11 nines) durability means that if you store 10 million files, then you expect to lose one file every 10000 years.

When people hear the word durability, most of them think of hardware failures of storage, compute, and network components that might cause data loss. In VPC storage, your data is protected against drive failures and numerous type of disk errors that otherwise might negatively impact data durability and data integrity. The data is stored redundantly across multiple physical disks in an Availability Zone to prevent data loss due to failure of any single component.

Other than physical failure, a common source of data loss is accidental deletion or modifications of files by users. Block Storage for VPC, Snapshots for VPC, and File Storage for VPC are only accessible to authorized hosts within your virtual private network. You control who can access it. For more information, see [Managing security and compliance](https://cloud.ibm.com/docs/vpc?topic=vpc-manage-security-compliance).

Another measure to protect against accidental deletion and modification of files is a snapshot. If a user accidentally modifies or deletes crucial data from a volume, the data can be easily and quickly restored from a snapshot or a backup. For more information about this feature, see [About Snapshots for VPC](https://cloud.ibm.com/docs/vpc?topic=vpc-snapshots-vpc-about).

The 11 nines durability target applies to a single Availability Zone. To protect against natural or man-made disasters that might destroy an entire Availability Zone, consider storing your most important data in multiple locations. For more information, see [Understanding high availability and disaster recovery](https://cloud.ibm.com/docs/vpc?topic=vpc-ha-dr-vpc).





 High Availability 

VPC storage is built upon best-in-class, proven, enterprise-grade hardware and software to ensure high availability and uptime. The data is stored redundantly across multiple physical disks on HA paired nodes.
Using storage classes 

Not all data feeds active workloads. Archival data might sit untouched for long periods of time. For less active workloads, you can create buckets with different storage classes. Objects that are stored in these buckets incur charges on a different schedule than standard storage.

This feature is not currently supported in Object Storage for Satellite. [Learn more.](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-about-cos-satellite)



 What are the classes? 

You can choose from four storage classes:



* Smart Tier can be used for any workload, especially dynamic workloads where access patterns are unknown or difficult to predict. Smart Tier provides a simplified pricing structure and automatic cost optimization by classifying the data into "hot", "cool", and "cold" tiers based on monthly usage patterns. All data in the bucket is then billed at the lowest applicable rate. There are no threshold object sizes or storage periods, and there are no retrieval fees. For a detailed explanation of how it works, see the [billing topic](https://cloud.ibm.com/docs/services/cloud-object-storage/iam?topic=cloud-object-storage-billingbilling-storage-classes).
* Standard is used for active workloads, with no charge for data retrieved (other than the cost of the operational request itself).
* Vault is used for cool workloads where data is accessed less than once a month - an extra retrieval charge ($/GB) is applied each time data is read. The service includes a minimum threshold for object size and storage period consistent with the intended use of this service for cooler, less-active data.
* Cold Vault is used for cold workloads where data is accessed every 90 days or less - a larger extra retrieval charge ($/GB) is applied each time data is read. The service includes a longer minimum threshold for object size and storage period consistent with the intended use of this service for cold, inactive data.



Flex has been replaced by Smart Tier for dynamic workloads.
For example, you can reuse your logging resources across different environments for your stage, pre-production, and production logging instances. [Learn more](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-reuse_resource_definitions).

Backup logging resource definitions into a version control system such as a git repository where you can control access to the archived files and manage versions.





 Archive log data to a COS bucket 

Enable archiving of your data from a logging instance to an IBM Cloud Object Storage (COS) bucket.

After you provision a logging instance, you can configure archiving to an IBM Cloud Object Storage (COS) bucket. You can create different types of buckets based on your requirements.

When you plan the bucket for a logging instance, consider the following information:



Table 5. COS bucket requirements

 Requirement Question to answer Information 

 Type of workload [1] How often do you need to access the data? [Information about storage classes](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-classesclasses-about) 
 Retention policy [2] Do you need to protect data from being deleted? [Information about Immutable Object Storage](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-immutable) 
 Expiration policy [3] Do you need automatic deletion of files? [Information about expiration rules](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-expiry) 
 Long-term retention policy [4] Do you need to keep data for a long period of time? [Information about archiving COS files for long term storage](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-archive) 
 Encryption of data at-rest with my own key [5] Can I use my own key to encrypt data at-rest?
Working with Object Storage 

IBM Cloud Object Storage is a highly scalable cloud storage service, designed for high durability, resiliency and security. See [IBM Cloud Object Storage](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-about-cloud-object-storage).

You must define IBM Cloud Object Storage as the service instance home in which custom application libraries (library sets) and Spark history events are stored. Additionally, you can use Object Storage to store your data files as well as your application or job binaries, for example Spark Python files.

To work with Object storage:



*  [Getting the Object Storage credentials](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-get-cos-credentials-serverless).
*  [Understanding the Object Storage credentials](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-cos-credentials-in-iae-serverless).
*  [Using Object Storage as the instance home](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-cos-serverless).
But a subsequent list of changes might report changes C, A, then B in that order. All the changes are listed, but in a different order. This difference is because the sequence of changes that are received during replication might vary between two different copies of the database.



 What does "eventual consistency" mean for the list of changes? 

When you request a list of changes, the response you get might vary depending on which database copy supplies the list.

The since option obtains a list of changes after a specific update sequence identifier. The list always includes changes after the update, but changes before the update might also be included. The reason is that the database copy that responds to the list request must ensure that it lists the changes, consistent with all the replicas. To achieve that consistency, the database copy might have to start the list of changes from the point when all the copies agreed. This point is identified by using checkpoints.

Therefore, an application that uses the _changes feed must be ['idempotent'](http://eaipatterns.com/IdempotentReceiver.html). Idempotency means that the application must be able to safely receive the same data multiple times, and potentially in a different order for repeated requests.







 Checkpoints 

Internally, the replication process writes its state in "checkpoint" documents that are stored in both the source and destination databases. Checkpoints allow a replication task to be resumed from where it stopped, without having to start from the beginning. Checkpoint creation can be prevented by supplying the ["use_checkpoints": false](https://cloud.ibm.com/apidocs/cloudantpostreplicate) option when you request replication. It is helpful to leave the feature on if your replication is to resume efficiently from its last known position.





 Permissions 

Admin access is necessary to insert a document into the _replicator database. The login credentials that are supplied in the source and target parameters do not require full admin permissions. It is sufficient if the credentials perform the following tasks:



* Write documents at the destination end.
* Write checkpoint documents at both ends.



IBM Cloudant has a special _replicator user permission.
When IBM decommissions a physical drive, the drive is destroyed before disposal. The decommissioned drives are unusable and any data on them is inaccessible.

Customers with special requirements for compliance such as NIST 800-88 Guidelines for Media Sanitization can perform the data sanitization procedure before they delete their storage.





 I cannot cancel a File Storage for Classic volume because the Cancel action in the Cloud console is unavailable or disabled. What’s happening? 

The cancellation process for this storage device is in progress so the Cancel action is no longer available. The volume remains visible for at least 24 hours until it is reclaimed. An hourglass or clock icon next to the device name in the UI indicates that it’s in a waiting period. The minimum 24-hour waiting period gives you a chance to void the cancel request if needed.





 Which NFS versions are supported? 

Both NFSv3 and NFSv4.1 are supported in the IBM Cloud® environment. NFSv4.2 is not supported.

Use NFSv3 protocol when possible. NFSv3 supports safe asynchronous writes and is more robust at error handling than the previous NFSv2. It supports 64-bit file sizes and offsets, allowing clients to access more than 2 GB of file data.

NFSv3 natively supports no_root_squash that allows root clients to retain root permissions on the NFS share. You can enable this feature in NFSv4.1, by editing the domain information and running the rpcidmapd or a similar service. For more information, see [Implementing no_root_squash for NFS](https://cloud.ibm.com/docs/FileStorage?topic=FileStorage-mountingLinuxnorootsquash).

When File Storage for Classic is used in a VMware® deployment, NFSv4.1 might be the better choice for your implementation.
All replicas are objects themselves, and [contribute usage](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-billing) just like any other data. Successful replication results in billable PUT, GET, and HEAD requests, although any bandwidth consumed in the replication process is not billed.

Replication generates additional metrics for use with IBM Cloud Monitoring:



* ibm_cos_bucket_replication_sync_requests_issued
* ibm_cos_bucket_replication_sync_requests_received







 Interactions 



 Versioning 

Versioning is mandatory in order to enable replication. After you [enable versioning](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-versioning) on both the source and target buckets and configure replication on the source bucket, you may encounter the following issues:



* If you attempt to disable versioning on the source bucket, Object Storage returns an error. You must remove the replication configuration before you can disable versioning on the source bucket.
* If you disable versioning on the target bucket, replication fails.







 Key Protect encryption 

Source objects will be [encrypted using the root key](https://cloud.ibm.com/docs/key-protect?topic=key-protect-about) of the source bucket, and replicas are encrypted using the root key of the target bucket.





 Lifecycle configurations 

If a [lifecycle policy is enabled](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-expiryexpiry-rules-attributes) on a target bucket, the lifecycle actions will be based on the original creation time of the object at the source, not the time that the replica becomes available in the target bucket.





 Immutable Object Storage
Larger disks have higher performance with baseline input/output operations per second (IOPS) of 10 IOPS for each GB. etcd relies on writing data to disk to maintain its consensus algorithm, so exceeding your IOPS deployment's limits can cause cluster instability. Scaling your deployment's disk resources increases the IOPS available to your deployment.





 Memory Usage 

etcd uses memory to cache data and can benefit from increasing the amount of memory to at least the size of the data set. Memory is also used to maintain the watchers, so if your use-case requires thousands of watchers, you can benefit from increasing the amount of RAM available to your deployment ever further.
To prepare for that configuration, review the following information.



1. The performance of the hosts, network latency, and block storage performance can all impact maximum throughput. As you review the following block storage information and select a storage class to achieve a maximum throughput, select block storage that provides a minimum of 4 IOPS per GB of storage and that can support 75 MB/s disk writes and 25 MB/s disk reads concurrently.
2. To configure block storage, Satellite provides configuration templates for several storage infrastructure providers. The [Understanding Satellite storage templates](https://cloud.ibm.com/docs/satellite?topic=satellite-sat-storage-template-ov) information explains how the storage templates work and how to configure them using the IBM Cloud CLI. The Event Streams service instance provision helps simplify the configuration by prompting for the needed configuration. To prepare for the configuration, complete the following steps:



1. Review the supported block storage classes in the [Storage class reference](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-class-ref).
2. Identify the block storage classes available for your storage infrastructure provider.
3. When available, select a block storage class that has a Volume Binding Mode = WaitForFirstConsumer.
4. The following table outlines the amount of block storage that is allocated by Event Streams.





Table 1. Block storage usage

 Usage Amount allocated Total 

 Message data retention 2 TB x 3 replicas/availability zones 6 TB total 
 Message data management 250 GB x 3 replicas/availability zones 750 GB total 
 Service instance monitoring 125 GB x 3 replicas/availability zones 375 GB total 



The information regarding the amount of storage is for a single Event Streams Satellite service instance. If multiple Event Streams Satellite instances are required, the information applies to each instance of Event Streams.





To provision Event Streams, complete the steps in [Provisioning Event Streams for Satellite](https://cloud.ibm.com/docs/EventStreams?topic=EventStreams-satellite-provisioning).
A migration of a VM incurs the following workflow:



* Creation of a new VM on the destination side and its corresponding virtual disks.
* Replication of VM data to the new VM. Replication starts as soon as the wizard is completed regardless of switch over scheduling.
* Power down of the original VM.
* During the power off period, final replication of any data changes occurs.
* Powering on the new VM on the destination side.
* Rename and move of the original VM to the cloud folder to which it was moved.



The following are advantages of bulk migration over vMotion:



* Migration of multiple VMs concurrently.
* More consistent bandwidth use. vMotion can generate fluctuations in bandwidth use that would be visible as peaks and valleys within network monitoring tools or the WAN Opt UI.
* Use bulk migration to obtain an overall use of a network bandwidth capability higher than if you use a single vMotion.
* Schedule bulk migration to switch to the newly migrated VMs during a scheduled outage window.
* Allow migration of VMs that are currently using virtual CPU features, which differ from the cloud side. vMotion migration might fail in these cases.



The following are disadvantages of bulk migration over vMotion:



* Individual VMs migrate much more slowly than with vMotion.
* The VM incurs downtime briefly as the new clone VM is brought up on the destination side.
* VMs that depend on disk ordering and disk UUIDs (Oracle RAC) can have issues and have disks that show up differently as the UUIDs are changed, which might change the OS paths to the virtual disk devices.









 Migration type best practices 



 Shared disk clusters 

Oracle RAC, MS Exchange, and MS-SQL clusters are examples of applications where two or more VMs participate in a cluster that requires shared disk across all VMs or cluster nodes. The VMware® multi-writer flag needs to be enabled on all of the VM nodes for disks that are part of the application cluster (non-OS virtual disks). VMs with the multi-writer flag enabled for any virtual disk are not supported.

Review the following information about migrating a multi-writer virtual disk enabled cluster:
* [Configuring File Storage for Classic for backup with cPanel](https://cloud.ibm.com/docs/FileStorage?topic=FileStorage-cPanelBackups)
* [Configuring File Storage for Classic for backup with Plesk](https://cloud.ibm.com/docs/FileStorage?topic=FileStorage-PleskBackup)







 Disaster recovery considerations 

To avoid data-loss and to ensure business continuity, consider replicating your servers and storage in another data center. Replication keeps your data in sync in two different locations based on your snapshot schedule. For more information, see [Replicating data](https://cloud.ibm.com/docs/FileStorage?topic=FileStorage-replication).

If you want to clone your volume, see [Creating and managing independent duplicate volumes](https://cloud.ibm.com/docs/FileStorage?topic=FileStorage-duplicatevolume). Duplicate volumes can be managed independently and refreshed with data from the original volume on demand.





 Identifying File Storage for Classic volumes on the invoice 

All file shares appear on your invoice as a line item. Endurance volumes appear as “Endurance Storage Service” and Performance volumes appear as "Performance Storage Service". The rate varies based on your storage level. You can expand on Endurance or Performance to see that it's File Storage for Classic.
* UI
* CLI
* Terraform







 Provisioning File Storage for Classic for use as VMware datastore 

This tutorial guides you through how to order and configure IBM Cloud® File Storage for Classic in a vSphere 5.5 or vSphere 6.0 environment at IBM Cloud®. File Storage for Classic is designed to support high I/O applications that require predictable levels of performance. The predictable performance is achieved through the allocation of protocol-level input/output operations per second (IOPS) to individual volumes.

If you require more than eight hosts to access your VMware® datastore, then choosing NFS File Storage for Classic is the best practice.

The File Storage for Classic offering is accessed and mounted through an NFS connection. In a VMware® deployment, a single volume can be mounted to up to 64 ESXi hosts as shared storage. You can also mount multiple volumes to create a storage cluster to use vSphere Storage Distributed Resource Scheduler (DRS).



 Ordering considerations 

Pricing and configuration options for File Storage for Classic are charged based on a combination of the reserved space and the offered IOPS. When you order File Storage for Classic, consider the following information:



* When you decide on the size, consider the size of the workload and throughput needed. Size matters with the Endurance service, which scales performance linearly in relation to capacity (IOPS/GB). Conversely, the Performance service allows the administrator to choose capacity and performance independently. Throughput requirements matter with Performance.

The throughput calculation is IOPS x 16 KB. IOPS is measured based on a 16-KB block size with a 50/50 read/write mix. Increasing block size increases the throughput but decreases IOPS. For example, doubling the block size to 32-KB blocks maintains the maximum throughput but halves the IOPS.
* NFS uses many extra file control operations such as lookup, getattr, and readdir. These operations in addition to read/write operations can count as IOPS and vary by operation type and NFS version.
* File Storage for Classic volumes are exposed to authorized devices, subnets, or IP addresses.
They can reduce backup infrastructure costs with built-in data efficiency capabilities and the ability to migrate data to tape, public cloud services, and on-premises object storage. IBM Spectrum Protect can also be a data offload target for IBM Spectrum Protect Plus, for a long-term data retention and disaster recovery. For more information, see [What can IBM Spectrum Protect do for your business?](https://www.ibm.com/us-en/marketplace/data-protection-and-recovery).



It's the user's responsibility to set up and maintain these environments. Remember to check for any connectivity and bandwidth restrictions to the LPAR server. Your LPAR servers can also use IBM Cloud Object Storage as a repository.

For a complete tutorial on backing up and restoring AIX VM data, see [Backing up and restoring data in an AIX VM](https://cloud.ibm.com/media/docs/downloads/power-iaas-tutorials/PowerVS_AIX_Backups_Tutorial_v1.pdf).

For best practices and guidelines on AIX backup performance on IBM Power Systems Virtual Server, see [AIX Backup Performance Best Practices and Guidelines on IBM Power Systems Virtual Server](https://cloud.ibm.com/media/docs/downloads/power-iaas/PowerVS_AIX_Backup_Performance_Best_Practices_and_Guidelines_v1_0_03012022.pdf).





 IBM i backup strategies 

A common IBM i backup strategy is to use IBM® Backup, Recovery, and Media Services (BRMS) and IBM Cloud Storage Solutions (ICC). Together, these products automatically back up your LPARs to IBM Cloud Object Storage. The ICC product can be integrated with BRMS to move and retrieve objects from remote locations, including Cloud Object Storage. In most cases, this process involves backing up to virtual tapes and image catalogs. Note, you might need extra storage for the LPAR to host the image catalogs until they are moved to Cloud Object Storage.

The typical IBM i customer uses the following flow to back up LPARs and objects:



1. Use the 5733-ICC product to connect to Cloud Object Storage (COS) (2 times the disk capacity to hold the backup images).
2.
* Pack searchable data into your _id field, for example, <customerid><date><orderid> can be used to retrieve data by customer, customer/date, or customer/date/orderid.
* In a partitioned database, the judicious choice of partition key allows an entire database to be winnowed down to a handful of documents for a known partition key. Make sure that your partitioning schema solves your most common use case.
* In a partitioned database, the two parts of the key have to contain your user-supplied data (no auto-generated _ids exist) so it’s best to use it optimally. For example, in an IoT application, <sensorid>:<time-sortable-id> allows data to be sorted by sensor and time without a secondary index. Implement this schema with time-boxed databases for best results.