vania-janet/multiturn-rag-retrieval-data huggingface dataset:
[Information about expiration rules](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-expiry) 
 Long-term retention policy [4] Do you need to keep data for a long period of time? [Information about archiving COS files for long term storage](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-archive) 
 Encryption of data at-rest with my own key [5] Can I use my own key to encrypt data at-rest? [Information about encryption of data at-rest](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-encryption) 
 Data resiliency [6] Do you need to store the data in a specific geographical location? [Information about resiliency](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-endpoints) 



[1]: Data might sit untouched for long periods of time. For less active workloads, you can create buckets with different storage classes. For example, use the standard storage class for active workloads, where you need to access data at any time.

[2]: You can choose Immutable Object Storage to preserve electronic records and maintain data integrity. When you define a retention policy for a bucket, you are specifying that the data is stored in a WORM (Write-Once-Read-Many), non-erasable and non-rewritable manner. This policy is enforced until the end of a retention period and the removal of any legal holds. Notice that Immutable Object Storage is available in certain regions only.

[3]: You can use an expiration rule to delete objects automatically after a defined period from the object creation date.

[4]: You can define a retention policy to retain data for a long period of time and have the data available for downloads and queries.
Continuous replication can be used for backups of data, aggregating data across many databases, or for sharing data.

However, continuous replication means testing continuously for any source database changes. This testing requires continuous internal calls, which might impact performance or the cost of using the database.

Continuous replication can result in many internal calls. These calls might affect costs for multi-tenant users of IBM Cloudant systems. Continuous replication is disabled by default.





 Using the proper tool for the job 

IBM Cloudant is a scalable, durable, highly available, operational JSON document store with an HTTP API. It's suitable for the following purposes:



* Powering your always-on web application.
* Being the server-side data store for mobile applications.
* Storing time-series data in time-boxed databases before you archive to object storage and delete the original.
* Storing application objects as JSON while queries are delivered from secondary indexes.
* Replicating data sets across geographies for disaster recovery, extra capacity, or moving data nearer to your users.



IBM Cloudant doesn't include the following features:



* Low latency, in-memory data store. For more information, see [IBM Cloud® Databases for Redis](https://www.ibm.com/uk-en/cloud/databases-for-redis).
* Limitless object store for archiving data. For more information, see [IBM Cloud Object Storage](https://www.ibm.com/cloud/object-storage).
* Relational database with SQL querying, stored procedures, and constraints and triggers. For more information, see [IBM Cloud Databases for PostgreSQL](https://www.ibm.com/cloud/databases-for-postgresql).
* Data warehouse for ad hoc querying. For more information, see [IBM® Db2® Warehouse on Cloud](https://www.ibm.com/products/db2-warehouse).
* A queue. For more information, see [IBM MQ](https://www.ibm.com/uk-en/products/mq).
For more information, see [Upgrading existing Block Storage for Classic to enhanced Block Storage for Classic](https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-migratestoragemigratestorage).





 Does Block Storage for Classic support SCSI-3 Persistent Reserve to implement I/O fencing for Db2 pureScale? 

Yes, Block Storage for Classic supports both SCSI-2 and SCSI-3 persistent reservations.





 What happens to the data when Block Storage for Classic LUNs are deleted? 

IBM Cloud® Block Storage for Classic presents Block volumes to customers on physical storage that is wiped before any reuse.

When you delete a Block Storage for Classic volume, that data immediately becomes inaccessible. All pointers to the data on the physical disk are removed. If you later create a new volume in the same or another account, a new set of pointers is assigned. The account can't access any data that was on the physical storage because those pointers are deleted. When new data is written to the disk, any inaccessible data from the deleted volume is overwritten.

IBM guarantees that data deleted cannot be accessed and that deleted data is eventually overwritten and eradicated. Further, when you delete a Block Storage for Classic volume, those blocks must be overwritten before that block storage is made available again, either to you or to another customer.

When IBM decommissions a physical drive, the drive is destroyed before disposal. The decommissioned drives are unusable and any data on them is inaccessible.

Customers with special requirements for compliance such as NIST 800-88 Guidelines for Media Sanitization can perform the data sanitization procedure before they delete their storage.





 What happens to the drives that are decommissioned from the cloud data center? 

When drives are decommissioned, IBM destroys them before they are disposed of. The drives become unusable. Any data that was written to that drive becomes inaccessible.





 I cannot cancel a Block Storage for Classic volume because the Cancel action in the Cloud console is disabled. What’s happening? 

The cancellation process for this storage device is in progress so the Cancel action is no longer available. The volume remains visible for at least 24 hours until it is reclaimed.
However, the same LUN with 2 or more sessions would be able to push more I/O's per second of storage throughput to the target LUN. The flip-side of increasing I/O depth is that it adds more latency. To counteract the latency, enable Jumbo Frames. For more information about host queue depth recommendations, see [Adjusting Host Queue settings](https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-hostqueuesettings).
* [Enable Jumbo Frames](https://cloud.ibm.com/docs/BlockStorage?topic=FileStorage-jumboframes) and configure them to be the same on the entire network path from source device <-> switch <-> router <-> switch <-> target device. If the entire chain isn't set the same, it defaults to the lowest setting along the chain. IBM Cloud® has network devices set to 9,000 currently. For best performance, all customer devices need to be set to the same 9,000 value.

Setting MTU to 9000 on your hosts has the following benefits:



* Data can be transmitted in fewer frames.
* Data can be transmitted in faster because fewer packets require fewer bytes of format information that is stored in the packet header.
* Throughput is increased by reducing the number of CPU cycles and instructions for packet processing.
* Jumbo frames provide less opportunity for packets to arrive out of order or to be lost, resulting in fewer retransmissions. Fewer retransmissions mean less time that is spent in TCP recovery. The result is greater throughput.



* Follow VMware specific best practice for teaming. If you plan to use teaming to increase the availability of your network access to the storage array, you must turn off port security on the switch for the two ports on which the virtual IP address is shared. The purpose of this port security setting is to prevent spoofing of IP addresses. Thus, many network administrators enable this setting. If you do not change it, the port security setting prevents failover of the virtual IP from one switch port to another, and teaming cannot fail over from one path to another.
This storage is limited availability and is available only in certain regions. You can learn more about file storage [here](https://cloud.ibm.com/docs/vpc?topic=vpc-file-storage-vpc-about).
* Object storage stores unstructured data as objects in a flat environment. Objects also include metadata (descriptive information) and a unique ID that better handles large amounts of data with limitless scalability. Data that is stored tends to be static. Typical use cases include backups, analytics, or streaming media. You can learn more about object storage [here](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-about-cloud-object-storage).







 Additional Resources 



* [Step by step guide for single-zone](https://cloud.ibm.com/docs/ha-infrastructure?topic=ha-infrastructure-deploy-n-tier-app-saz)
* [Regional snapshots in 3-tier architecture](https://cloud.ibm.com/docs/ha-infrastructure?topic=ha-infrastructure-regional-snapshots-3-tier-arch)
* [Building a highly available infrastructure for 3-tier web applications in VPC](https://cloud.ibm.com/docs/ha-infrastructure?topic=ha-infrastructure-ha-3-tier)
* [Cloud reliability and disaster recovery architecture](https://www.ibm.com/cloud/architecture/architectures/resilience/)
This fact is a price of scalability and data resilience.

It is best to structure your data so that resolving conflicts is quick and does not involve operator assistance. This practice helps your databases to hum along smoothly. The ability to automatically resolve conflicts without user involvement significantly improves their experience and reduces the support burden on your organization.

How you resolve conflicts is application-specific. See the following tips for more ways to improve the process:



* Avoid invariants across document fields if possible. Avoiding invariants makes it more likely that a simple merge operation, if you take the changed field from each conflicted document revision, is suitable. This practice makes simpler and more robust application code.
* Allow documents to be independent. If you have to retrieve other documents to work out the correct resolution, it increases latency in conflict resolution. There's also a chance you get a version of the other documents that aren't consistent with the document you're resolving, making correct resolution difficult.







Heavily conflicted documents exert a heavy toll on the database. Building in the capability to resolve conflicts from the beginning is a great help in avoiding pathologically conflicted documents.





These tips demonstrate how modeling data affects your application’s performance. IBM Cloudant’s data store has some specific characteristics, both to watch out for and to take advantage of, that ensure the database performance scales as your application grows. IBM Cloudant support understands the shift can be confusing, so they are always available to give advice.

For more information, see the [data model for Foundbite](https://cloudant.com/blog/foundbites-data-model-relational-db-vs-nosql-on-cloudant/), or the [example from our friends at Twilio](https://www.twilio.com/blog/2013/01/building-a-real-time-sms-voting-app-part-3-scaling-node-js-and-couchdb.html).



* When must I make my documents immutable?

If you're changing the same piece of state at a rate of once per second or more, consider making your documents immutable. This practice significantly reduces the chance that you create conflicted documents.
Release notes for Object Storage 

News on the latest releases from IBM Cloud® Object Storage provide the updates you need on all things related to IBM Cloud Object Storage.



 14 June 2023 

New location!
: Buckets can now be created in a Regional (MZR) configuration in Madrid, Spain. More information can be found in the [Select Regions and Endpoints](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-endpoints).





 16 March 2023 

New feature!
: It is now possible to [lock objects](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-ol-overview) to ensure individual object versions are stored in a WORM (Write-Once-Read-Many), non-erasable and non-rewritable manner.





 26 September 2022 

New feature!
: It is now possible to create instances [using a One Rate plan](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-onerate) to lower costs and simplify billing for workloads with high levels of egress.





 21 June 2022 

New feature!
: It is now possible to configure buckets for automated [replication of objects to a destination bucket](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-replication-overview).





 7 June 2022 

Encryption update
: Buckets created with [Hyper Protect Crypto Services](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-hpcs-cr) managed encryption can now use [Immutable Object Storage](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-immutable) to create retention policies that prevent object deletion or modification.





 23 May 2022 

Encryption update
* Are ISCSI LUNs thin or thick provisioned?

All File and Block Storage for Classic services are thin-provisioned. This method is not modifiable.
* My billing ID changed, what does this mean?

You might notice that your Storage volumes are now billed as "Endurance Storage Service” or "Performance Storage Service" instead of "Enterprise Storage". You may also have new options in the console, such as the ability to adjust IOPS or increase capacity. IBM Cloud® strives to continuously improve storage capabilities. As hardware gets upgraded in the data centers, storage volumes that reside in those data centers are also upgraded to use all enhanced features. The price that you pay for your Storage volume does not change with this upgrade.
* How durable is Block Storage for Classic?

When you store your data in Block Storage for Classic, it's durable, highly available, and encrypted. The durability target for a single Availability zone is 99.999999999% (11 9's). For more information, see [Availability and Durability of Block Storage for Classic](https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-storageavailability).
* What's the average uptime for Block Storage for Classic?

When you store your data in Block Storage for Classic, it's durable, highly available, and encrypted. Block Storage for Classic is built upon best-in-class, proven, enterprise-grade hardware and software to ensure high availability and uptime. To ensure that the availability target of 99.999% (five 9's) is met, the data is stored redundantly across multiple physical disks on HA paired nodes. Each storage node has multiple paths to its own Solid-State Drives and its partner node's SSDs as well. This configuration protects against path failure, and also controller failure because the node can still access its partner's disks seamlessly. For more information, see [Availability and Durability of Block Storage for Classic](https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-storageavailability).
Snapshots 

Snapshots are a feature of IBM Cloud® File Storage for Classic. A snapshot represents a volume's contents at a particular point in time. With snapshots, you can protect your data with no performance impact and minimal consumption of space. Snapshots are considered your first line of defense for data protection. If a user accidentally modifies or deletes crucial data from a volume, the data can be easily and quickly restored from a snapshot copy.

File Storage for Classic provides you with two ways to take your snapshots.



* First, through a configurable snapshot schedule that creates and deletes snapshot copies automatically for each storage volume. You can also create extra snapshot schedules, manually delete copies, and manage schedules based on your requirements.
* The second way is to take a manual snapshot.



A snapshot copy is a read-only image of a File Storage for Classic volume that captures the state of the volume at a point in time. Snapshot copies are efficient both in the time that is needed to create them and in storage space. A File Storage for Classic snapshot copy takes only a few seconds to create. It's typically less than 1 second, regardless of the size of the volume or the level of activity on the storage. After a snapshot copy is created, changes to data objects are reflected in updates to the current version of the objects, as if Snapshot copies didn't exist. Meanwhile, the copy of the data remains stable.

A Snapshot copy incurs no performance decrease. Users can easily store up to 50 scheduled snapshots and 50 manual snapshots per File Storage for Classic volume, all of which are accessible as read-only and online versions of the data.

With snapshots, you can:



* Nondisruptively create point-in-time recovery points,
* Revert volumes to previous points-in-time.



You must purchase some amount of snapshot space for your volume first so you can take snapshots of it. The snapshot space can be added during the initial order or afterward through the Volume Details page. Scheduled and manual snapshots share the snapshot space, so make sure you order enough Snapshot space.
A database can demonstrate only two of these three attributes for both theoretical and practical reasons. A database prioritizing consistency and availability is simple: a single node stores a single copy of your data. But this model is difficult to scale as you must upgrade the node to get more performance, rather than use extra nodes. And, even a minor system failure can shut down a single-node system, while any message loss means significant data loss. To endure, the system must become more sophisticated.



 Tradeoffs in partition tolerance 

A database that prioritizes consistency and partition tolerance commonly employs a primary-secondary setup, where one node of the many in the system is elected leader. Only the leader approves data writes, while all secondary nodes replicate data from the leader to handle reads. If the leader loses connection to the network, or can't communicate with many of the system's nodes, the remainder elects a new leader. This election process differs between systems, and might be a source of [significant problems](https://aphyr.com/posts/284-call-me-maybe-mongodb).

IBM Cloudant prioritizes availability and partition tolerance by employing a primary-primary setup, such that every node can accept both writes and reads to its portion of your data. Multiple nodes include copies of each portion of your data. Each node copies data with other nodes. If a node becomes inaccessible, others can serve in its place while the network heals. This way, the system returns your data in a timely manner despite arbitrary node failure, and maintains [eventual consistency](https://en.wikipedia.org/wiki/Eventual_consistency). The tradeoff in deprioritizing absolute consistency is that it takes time for all nodes to see the same data. As a result, some responses might include old data while the new data propagates through the system.





 Changing the approach 

Maintaining one consistent view of data is logical and easy to understand because a relational database does this work for you. The expectation is that web-based services that interact with database systems behave in this way. But that expectation doesn't mean that they do work this way. Consistency isn't a given, and it takes a little work to change the approach.
Benefits of using Portworx

 Benefit Description 

 Cloud native storage and data management for stateful apps Portworx aggregates available local storage that is attached to your worker nodes and that can vary in size or type, and creates a unified persistent storage layer for containerized databases or other stateful apps that you want to run in the cluster. By using Kubernetes persistent volume claims (PVC), you can add local persistent storage to your apps to store your data. 
 Highly available data with volume replication Portworx automatically replicates data in your volumes across worker nodes and zones in your cluster so that your data can always be accessed and that your stateful app can be rescheduled to another worker node in case of a worker node failure or reboot. 
 Support to run hyper-converged Portworx can be configured to run [hyper-converged](https://docs.portworx.com/portworx-install-with-kubernetes/storage-operations/hyperconvergence/) to ensure that your compute resources and the storage are always placed onto the same worker node. When your app must be rescheduled, Portworx moves your app to a worker node where one of your volume replicas resides to ensure local-disk access speed and high performance for your stateful app. 
 Encrypt data with IBM Key Protect You can [set up IBM Key Protect encryption keys](https://cloud.ibm.com/docs/openshift?topic=openshift-storage_portworx_encryption) that are secured by FIPS 140-2 Level 2 certified cloud-based hardware security modules (HSMs) to protect the data in your volumes. You can choose between using one encryption key to encrypt all your volumes in a cluster or using one encryption key for each volume. Portworx uses this key to encrypt data at rest and during transit when data is sent to a different worker node.
Archiving and accessing cold data 

IBM Cloud® Object Storage "Archive" and "Accelerated Archive" are [low cost](https://www.ibm.com/cloud/object-storage) options for data that is rarely accessed. You can store data by transitioning from any of the storage tiers (Standard, Vault, Cold Vault and Flex) to long-term offline archive or use the online Cold Vault option. With the new "Accelerated Archive" feature you can quickly access dormant data with restoration occurring in less than two hours.

This feature is not currently supported in Object Storage for Satellite. [Learn more.](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-about-cos-satellite)

You can archive objects using the web console, REST API, and 3rd party tools that are integrated with IBM Cloud Object Storage.

For more information about endpoints, see [Endpoints and storage locations](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-endpointsendpoints)



 Add or manage an archive policy on a bucket 

When creating or modifying an archive policy for a bucket, consider the following:



* An archive policy can be added to a new or existing bucket at any time.
* An existing archive policy can be modified or disabled.
* A newly added or modified archive policy applies to new objects uploaded and does not affect existing objects.



Create a bucket in the console after you've logged in, and you can configure your archive policy using the fields shown in Figure 1.

Zoom

![Create an archive policy](https://cloud.ibm.com/docs-content/v1/content/63322f2f3a72050206eb05ba7f590fdcb741105c/cloud-object-storage/images/bucket-create-ui-archive-rule.png)

Figure 1. Create an archive policy

To immediately archive new objects uploaded to a bucket, enter 0 days on the archive policy.

Archive is available in certain regions only.