vania-janet/multiturn-rag-retrieval-data huggingface dataset:
As always, successful modeling involves achieving a balance between ease of use versus the performance characteristics you're hoping to achieve.

(The FAQ for modeling data to scale is based on a blog article by Mike Rhodes, My top five tips for modeling your data to scale.)



If you're changing the same piece of state at a rate of once per second or more, consider making your documents immutable. This practice significantly reduces the chance that you create conflicted documents.

Conversely, if you're updating a specific document less than once every 10 seconds, an update-in-place data model - that is, updating existing documents - simplifies your application code considerably.

Typically, data models based on immutable data require the use of views to summarize the documents that include the current state. As views are precomputed, this process most likely doesn't adversely affect application performance.





Behind the https://$ACCOUNT.cloudant.com/ interface is a distributed database. Within the cluster, documents are bucketed into a number of shards that collectively form the database. These shards are then distributed across nodes in the cluster. This practice allows the support of databases many terabytes in size.

By default, the database is split into shards. Each shard has three copies, or shard replicas, which reside on a different node of the database cluster. Sharding allows the database to continue serving requests if a node fails, so saving a document involves writing to three nodes. If two updates are made concurrently to the same document, a subset of nodes might accept the first update, and another subset might accept the second update. When the cluster detects this discrepancy, it combines the documents in the same way as normal replication does for concurrent updates by creating a conflict.





Conflicted documents harm performance. A highly concurrent update-in-place pattern also increases the likelihood that writes get rejected. In that situation, the _rev parameter isn’t the expected one, which forces your application to retry and delay processing.

This conflicted-document scenario is significantly more likely to happen for updates that occur more often than once a second. Use immutable documents for updates that occur more than once every 10 seconds to be on the safe side.
However, the primary disadvantages to the approach are if you have an existing application or are unable to use an IBM Cloudant-supported client library.



 Advantages of IAM mode 



* Manage access for many services by using one interface.
* Revoke access to a user globally.
* Account-level API keys that use service IDs.
* Easy-to-rotate credentials.
* Activity Tracker logs capture individual humans and services.
* IAM federates with other identity systems, like enterprise LDAP repositories.
* Fine-grained permissions (for example, Reader, Writer, Monitor, or Checkpointer).







 Disadvantages of IAM mode 



* If you are not using the supported libraries from IBM Cloudant, application changes are likely to be required to use IAM's API keys and access tokens.
* No database-level permissions (yet).
* Some endpoints are not available. For more information, see [unavailable endpoints](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-managing-access-for-cloudantunavailable-endpoints).
* No way to specify a database as "public", that is, not requiring an authorized user to access.







 Advantages of legacy mode 



* No need to change existing applications or client library dependencies.
* Database-level permissions.







 Disadvantages of legacy mode 



* Separate management of IBM Cloudant credentials, so unable to get full overview of all access within centralized interface.











 Create a replication job by using IAM credentials only 

Follow these instructions to generate IAM API keys, generate the bearer token, create the _replicator database, and create the replication job.



 Generating IAM API keys for Source and Target and one for IBM Cloudant API access 

In this exercise, the first two API keys are created so that the two instances can talk to each other during the replication process. The third API key is for the user to access the IBM Cloudant API, create the _replicator database, and then add the replication document to it.

Follow these steps to generate IAM API keys and API access for IBM Cloudant. You must write down the credentials that are requested in the following steps to continue with the example.
IBM Cloud Schematics offers several advantages such as secure storage of sensitive configuration properties.

For more information about IBM Cloud Schematics, see the [IBM Cloud Schematics documentation](https://cloud.ibm.com/docs/schematics).

For more information about sensitive data in Terraform state, see [Sensitive Data in State](https://developer.hashicorp.com/terraform/language/state/sensitive-data).

For more information about how to set input variables in Terraform, see [Assigning Values to Root Module Variables](https://developer.hashicorp.com/terraform/language/values/variablesassigning-values-to-root-module-variables).





 Deleting your data from Continuous Delivery 

When you delete a Continuous Delivery service instance, the related toolchains, tool integrations, tools, and data (including personal data) are not deleted. For more information about how to manage and delete data that is stored with toolchains, tool integrations, and tools, see [Modifying, exploring, and deleting personal data](https://cloud.ibm.com/docs/ContinuousDelivery?topic=ContinuousDelivery-cd_personal_datamanaging_personal_data).
Immutable backup solution architecture 

The immutable backup solution architecture is suitable for clients who want to extend their VMware vCenter Server® instance with the Veeam® service to use immutable storage and minimize costs. The immutable backup solution architecture does not preclude any of the vCenter Server options such as Caveonix, Entrust, and VMware Aria® Operations™.

The solution architecture is enabled by the following key technologies:



* The immutable storage is provided by a Veeam Linux® hardened repository that is hosted on an IBM Cloud® bare metal server. For more information, see [Veeam Linux hardened repository](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-veeam-cr-sa-lhr).
* Optionally, if a sandbox is required, the following technologies are used:



* Veeam vPower NFS service enables a virtual machine (VM) to be started and run directly from the backup file that is hosted in the backup repository.
* Veeam Instant Restore enables a VM to be started directly from the backup files. Veeam vPower NFS service is used to access the backup files.
* The Veeam VM Recovery with the restore to new location option, enables a copy of the VM to be started and connected to an isolated network. The backup file is converted to VMDK files and placed in the designated data store.
* Veeam Secure Restore is only available for Microsoft® Windows® VMs. It is an extra option in the VM Recovery workflow that enables the VM to be scanned by antivirus software before you restore the VM. The VMs disks are connected to a mount server and then the antivirus software on the mount server that is used to scan files from the mounted disks.
* VMware NSX-T™ overlay segments allow the creation of isolated segments onto which copies of the VMs can be attached and isolated from the production VMs.
* NSX-T distributed firewall provides the required isolation so that only required cybertoolsets can access the copies of the VMs.
Modeling data to scale FAQ 

The way you model data on IBM® Cloudant® for IBM Cloud® significantly impacts how your application can scale. The underlying data model differs substantially from a relational model, and ignoring this distinction can be the cause of performance issues down the road.

As always, successful modeling involves achieving a balance between ease of use versus the performance characteristics you're hoping to achieve.

(The FAQ for modeling data to scale is based on a blog article by Mike Rhodes, My top five tips for modeling your data to scale.)



 When must I make my documents immutable? 

If you're changing the same piece of state at a rate of once per second or more, consider making your documents immutable. This practice significantly reduces the chance that you create conflicted documents.

Conversely, if you're updating a specific document less than once every 10 seconds, an update-in-place data model - that is, updating existing documents - simplifies your application code considerably.

Typically, data models based on immutable data require the use of views to summarize the documents that include the current state. As views are precomputed, this process most likely doesn't adversely affect application performance.





 Why does this help me consider immutable data? 

Behind the https://$ACCOUNT.cloudant.com/ interface is a distributed database. Within the cluster, documents are bucketed into a number of shards that collectively form the database. These shards are then distributed across nodes in the cluster. This practice allows the support of databases many terabytes in size.

By default, the database is split into shards. Each shard has three copies, or shard replicas, which reside on a different node of the database cluster. Sharding allows the database to continue serving requests if a node fails, so saving a document involves writing to three nodes. If two updates are made concurrently to the same document, a subset of nodes might accept the first update, and another subset might accept the second update. When the cluster detects this discrepancy, it combines the documents in the same way as normal replication does for concurrent updates by creating a conflict.





 Why do I need to worry about conflicted documents? 

Conflicted documents harm performance.
The immutable backup solution architecture consists of:



* Linux hardened repository - The hardened repository is one or more IBM Cloud bare metal servers that run a supported Linux OS. The hardened repository is configured as an immutable storage repository. The IBM Cloud bare metal servers are ordered with internal disks and a RAID card to present this directly attached storage to the OS to be used as a backup repository.
* Optionally, the immutable backup solution architecture can include one or more sandboxes. For more information, see [Veeam technologies used in the sandbox](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-veeam-cr-sandboxveeam).



The solution architecture does not show the components to adhere to the 3-2-1 backup rule. The 3-2-1 rule describes a backup architecture that:



* 3 - At least three copies of data: production, primary backup, and backup copy
* 2 - Use of two different types of media
* 1 - Keep one backup copy offsite



To adhere to this rule, consider:



* Use a Scale-Out-Repository capacity tier to copy data to Cloud Object Storage. Currently, IBM Cloud Object Storage cannot be used by Veeam as an immutable capacity tier.
* Set up a Veeam backup copy job to transfer the backup to another backup repository hosted in another IBM Cloud data center.





 Related links 



* [Overview of VMware Solutions](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-solution_overview)
* [Veeam on bare metal server introduction](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-veeam-bms-archi-intro)
* [Veeam Backup and Replication 12 overview](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-veeamvm_overview)
For example, WordPress is optimized to store data in a MySQL database. In these cases, the type of storage is selected for you.

Determine the type of data that you want to store.
: Structured data: Data that you can store in a relational database where you have a table with columns and rows. Data in tables can be connected by using keys and is usually easy to access due to the pre-defined data model. Examples are phone numbers, account numbers, Social Security numbers, or postal codes.
: Semi-structured data: Data that does not fit into a relational database, but that comes with some organizational properties that you can use to read and analyze this data more easily. Examples are markup language files such as CSV, XML, or JSON.
: Unstructured data: Data that does not follow an organizational pattern and that is so complex that you can't store it in a relational database with pre-defined data models. To access this data, you need advanced tools and software. Examples are e-mail messages, videos, photos, audio files, presentations, social media data, or web pages.

If you have structured and unstructured data, try to store each data type separately in a storage solution that is designed for this data type. Using an appropriate storage solution for your data type eases up access to your data and gives you the benefits of performance, scalability, durability, and consistency.

Analyze how you want to access your data. Storage solutions are usually designed and optimized to support read or write operations.
: Read-only: You don't want to write or change your data. Your data is read-only.
: Read and write: You want to read, write, and change your data. For data that is read and written, it is important to understand if the operations are read-heavy, write-heavy, or balanced.

Determine the frequency that your data is accessed.
: Hot data: Data that is accessed frequently. Common use cases are web or mobile apps.
: Cool or warm data: Data that is accessed infrequently, such as once a month or less. Common use cases are archives, short-term data retention, or disaster recovery.
Add/edit/remove a bucket's Immutable Storage policy PUT {endpoint}/{bucket}?protection cloud-object-storage.bucket.put_protection cloud-object-storage.bucket-retention.create 
 Read a bucket's Immutable Storage policy GET {endpoint}/{bucket}?protection cloud-object-storage.bucket.get_protection cloud-object-storage.bucket-retention.read 
 Initiate a multipart upload POST {endpoint}/{bucket}/{key}?uploads cloud-object-storage.object.post_initiate_upload cloud-object-storage.object-multipart.start 
 Upload a part PUT {endpoint}/{bucket}/{key}?uploadId={uploadId}&partNumber={partNumber} cloud-object-storage.object.put_part cloud-object-storage.object-multipart.create 
 Complete a multipart upload POST {endpoint}/{bucket}/{key}?uploadID={uploadId} cloud-object-storage.object.post_initiate_upload cloud-object-storage.object-multipart.complete 
 Add a public ACL block configuration PUT {endpoint}/{bucket}?publicAccessBlock cloud-object-storage.bucket.put_public_access_block cloud-object-storage.bucket-public-access-block.create 
 Read a public ACL block configuration GET {endpoint}/{bucket}?publicAccessBlock cloud-object-storage.bucket.get_public_access_block cloud-object-storage.bucket-public-access-block.read 
 Delete a public ACL block configuration DELETE {endpoint}/{bucket}?publicAccessBlock cloud-object-storage.bucket.delete_public_access_block cloud-object-storage.bucket-public-access-block.delete
Deleting a versioned object creates a delete marker. The object may appear to be deleted, but if the object is protected it is not possible to delete the protected version. Delete markers themselves are not protected.





 Replication 

Object Lock cannot be used on the source bucket for replication, only on the destination. Objects will be assigned the default retention period.





 Key Management Systems 

Protected objects will be encrypted using the root key of the bucket. When Object Lock is enabled on a bucket, the root key hosted by Key Protect or Hyper Protect Crypto Services is protected from deletion as long as an associated bucket has Object Lock enabled. This prevents crypto shredding of protected objects.





 Lifecycle configurations 

It is possible to enable lifecycle policies that [archive locked objects](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-archive), but naturally not those that [expire objects](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-expiry) under retention or legal hold (unprotected objects in the bucket can still be expired).





 Immutable Object Storage 

Object Lock is an alternative to the retention policies available when using Immutable Object Storage. As Object Lock requires versioning to be enabled, and Immutable Object Storage is not compatible with versioning, it is not possible to have both WORM solutions enabled on the same bucket. It is possible to have a mix of buckets in a Service Instance, each using either Immutable Object Storage or Object Lock.





 Object Tagging 

There are no restrictions on adding or modifying tags on a protected object.





 Other interactions 

There should be no adverse interactions when using Object Lock with other Object Storage features, such as setting CORS policies, setting IP firewalls or condition based restrictions, bucket quotas, or Code Engine.







 IAM actions 

There are new IAM actions associated with Object Lock.



 IAM Action Role 

 cloud-object-storage.bucket.get_object_lock_configuration Manager, Writer, Reader 
 cloud-object-storage.bucket.put_object_lock_configuration Manager, Writer
[Information about Immutable Object Storage](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-immutable) 
 Expiration policy [3] Do you need automatic deletion of files? [Information about expiration rules](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-expiry) 
 Long-term retention policy [4] Do you need to keep data for a long period of time? [Information about archiving COS files for long term storage](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-archive) 
 Encryption of data at-rest with my own key [5] Can I use my own key to encrypt data at-rest? [Information about encryption of data at-rest](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-encryption) 
 Data resiliency [6] Do you need to store the data in a specific geographical location? [Information about resiliency](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-endpoints) 



[1]: Data might sit untouched for long periods of time. For less active workloads, you can create buckets with different storage classes. For example, use the standard storage class for active workloads, where you need to access data at any time.

[2]: You can choose Immutable Object Storage to preserve electronic records and maintain data integrity. When you define a retention policy for a bucket, you are specifying that the data is stored in a WORM (Write-Once-Read-Many), non-erasable and non-rewritable manner. This policy is enforced until the end of a retention period and the removal of any legal holds. Notice that Immutable Object Storage is available in certain regions only.

[3]: You can use an expiration rule to delete objects automatically after a defined period from the object creation date.
You might also have new options in the console, such as the ability to adjust IOPS or increase capacity. IBM Cloud® strives to continuously improve storage capabilities. As hardware gets upgraded in the data centers, storage volumes that reside in those data centers are also upgraded to use all enhanced features. The price that you pay for your Storage volume does not change with this upgrade.





 How durable is File Storage for Classic? 

When you store your data in File Storage for Classic, it's durable, highly available, and encrypted. The durability target for a single Availability zone is 99.999999999% (11 9's). For more information, see [Availability and Durability of File Storage for Classic](https://cloud.ibm.com/docs/FileStorage?topic=FileStorage-storageavailability).





 What's the average uptime for File Storage for Classic? 

When you store your data in File Storage for Classic, it's durable, highly available, and encrypted. File Storage is built upon best-in-class, proven, enterprise-grade hardware and software to ensure high availability and uptime. To ensure that the availability target of 99.999% (five 9's) is met, the data is stored redundantly across multiple physical disks on HA paired nodes. Each storage node has multiple paths to its own Solid-State Drives and its partner node's SSDs as well. This setup protects against path failure, and also controller failure because the node can still access its partner's disks seamlessly. For more information, see [Availability and Durability of File Storage for Classic](https://cloud.ibm.com/docs/FileStorage?topic=FileStorage-storageavailability).





 Can I get storage performance metrics (IOPS or latency) from the Support teams? 

IBM Cloud® does not provide storage performance IOPS and latency metrics. Customers are expected to monitor their own File Storage for Classic devices by using their choice of third-party monitoring tools.

The following examples are utilities that you might consider to use to check performance statistics.
* What performance latency can be expected from the File Storage for Classic?

Target latency within the storage is less than one ms. The storage is connected to compute instances on a shared network, so the exact performance latency depends on the network traffic during the operation.
* What happens to the data when File Storage for Classic Volumes are deleted?

IBM Cloud® File Storage for Classic presents file shares to customers on physical storage that is wiped before any reuse.

When you delete a File Storage for Classic volume, that data immediately becomes inaccessible. All pointers to the data on the physical disk are removed. If you later create a new volume in the same or another account, a new set of pointers is assigned. The account can't access any data that was on the physical storage because those pointers are deleted. When new data is written to the disk, any inaccessible data from the deleted volume is overwritten.

IBM guarantees that data deleted cannot be accessed and that deleted data is eventually overwritten and eradicated. Further, when you delete a storage volume, the share must be overwritten before the storage is made available again, either to you or to another customer.

When IBM decommissions a physical drive, the drive is destroyed before disposal. The decommissioned drives are unusable and any data on them is inaccessible.

Customers with special requirements for compliance such as NIST 800-88 Guidelines for Media Sanitization can perform the data sanitization procedure before they delete their storage.
* I cannot cancel a File Storage for Classic volume because the Cancel action in the Cloud console is unavailable or disabled. What’s happening?

The cancellation process for this storage device is in progress so the Cancel action is no longer available. The volume remains visible for at least 24 hours until it is reclaimed. An hourglass or clock icon next to the device name in the UI indicates that it’s in a waiting period. The minimum 24-hour waiting period gives you a chance to void the cancel request if needed.
* Which NFS versions are supported?

Both NFSv3 and NFSv4.1 are supported in the IBM Cloud® environment. NFSv4.2 is not supported.

Use NFSv3 protocol when possible.