vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Data is distributed around nodes without the sort of locking that you might see in a relational database.

IBM Cloudant favors availability over consistency: It would rather be up and answering API calls than be down because it can't provide consistency guarantees. (A relational database is often configured in the opposite way: It operates in a consistent manner or not at all.) The upshot of eventual consistency for a developer is that your app must not read its writes in a short time. A small-time window might exist in which it is possible to see an older version of a document than the one you updated. Eventually, the data flows around the cluster, and in most cases, the quorum mechanism provides the illusion of consistency, but it is best not to rely on it.

In CouchDB 4, and in IBM Cloudant services based on that code version, a different consistency model is employed.

If your data model requires you to update a document over and over in a short time window, it's possible that multiple writes for the same revision number are accepted. These writes lead to a branch in the revision tree - known as a conflict. In this example revision 2 was modified in two different ways, causing two revision 3s. It's possible to tidy up conflicts programmatically, but they must be avoided as they can cause performance issues in extreme circumstances.

Conflicts can also happen when you use replication and a document is modified in different ways, and then the conflicting revisions are merged in by using replication. IBM Cloudant does not throw away data in this scenario. A winning revision is chosen, but the nonwinning revisions can be accessed and your application can resolve the conflict by electing a new winner, deleting unwanted revisions, or any action you need. A conflict is not an error condition. It's a side effect of having disconnected copies of data that can be modified without locking. IBM Cloudant chooses to handle conflicts by not discarding clashing changes, but storing them as a conflict.

To check a document for conflicts, simply add ?conflicts=true to a fetch of a single document. Any conflicting revisions are listed in the _conflicts array.

Unwanted revisions can be removed by using the normal DELETE operation, specifying the rev token of the revision you want to delete.
The storage should be configured as “immutable object storage.” Retention policies are should be applied to the storage, so that data is stored in a WORM (Write-Once-Read-Many), non-erasable and non-rewritable manner. The policy is set and enforced for a 12-month (minimum) retention period.



* Authenticating users with MFA using a physical hardware-based security key that generates a six-digit numerical code. A smart card or hardware token designed and operated to FIPS 140-2 level 2 or above or equivalent (e.g., ANSI X9.24, ISO 13491-1:2007) is recommended.
* Locking user accounts after three (3) consecutive failed logon attempts within 15 minutes.
* Locking user accounts for 30 minutes when there have been more than three unsuccessful logon attempts. After the lockout period ends, the user will be able to reset their password. Internal privileged accounts must remain locked until released by an administrator.
* Session timeout after 15 minutes of inactivity.
* Providing a system use notification banner from either the bastion or the target system. The warning banner is displayed before the system grants access to the user and the usage conditions must be approved before proceeding. The banner will provide privacy and security notices consistent with applicable customer policies, regulations, standards, and guidance. The warning banner will state that:



* Users are accessing a financial services information system.
* Information system usage may be monitored, recorded, and subject to audit.
* Unauthorized use of the information system is prohibited and subject to criminal and civil penalties.
* Use of the information system indicates consent to monitoring and recording.







 Set up a bastion host 

You need to install and manage your own bastion solution within your management VPC. There are various ways a bastion solution can be implemented. For one example that uses Teleport Enterprise Edition, see [Setting up a bastion host for secure connectivity](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-vpc-architecture-connectivity-bastion-tutorial-teleport).





 Related controls in IBM Cloud Framework for Financial Services
After identifying the appropriate primitive for your program, you can use Qiskit to prepare inputs, such as circuits, observables (for Estimator), and customizable options to optimize your job. For more information, see the [Primitives](https://qiskit.org/documentation/partners/qiskit_ibm_runtime/primitives.html) topic.



 Benefits of using sessions 

There are several benefits to using sessions:



* Jobs that belong to a single algorithm run are run together without interruptions, increasing efficiency if your program submits multiple sequential jobs.



* The queuing time does not decrease for the first job submitted within a session. Therefore, a session does not provide any benefits if you only need to run a single job.
* Since data from the first session job is cached and used by subsequent jobs, if the first job is cancelled, subsequent session jobs will all fail.



* When using sessions, the uncertainty around queuing time is significantly reduced. This allows better estimation of a workload's total runtime and better resource management.
* In a device characterization context, being able to run experiments closely together helps prevent device drifts and provide more accurate results.
* As long as the session is active, you can submit different jobs, inspect job results, and re-submit new jobs without opening a new session.
* You maintain the flexibility to deploy your programs either remotely (cloud / on-premise) or locally (your laptop).







 The mechanics of sessions (queuing) 

For each backend, the first job in the session waits its turn in the queue normally, but while the session is active, subsequent jobs within the same session take priority over any other queued jobs. If no jobs that are part of the active session are ready, the session is deactivated (paused), and the next job from the regular fair-share queue is run. See [Interactive timeout value](https://cloud.ibm.com/docs/quantum-computing?topic=quantum-computing-sessionsttl) for more information.

A quantum processor still executes one job at a time. Therefore, jobs that belong to a session still need to wait for their turn if one is already running.
First, both maps and reduces are precomputed, so the result of a reduce function is a cheap operation. The operation cost is low even when compared to the significant amounts of IO required to stream hundreds or even thousands of documents from the on-disk storage.

At a deeper level, when a node receives a view request, it asks the nodes that hold the shard replicas of the view's database for the results of the view request from the documents in each shard. As it receives the answers, taking the first answer for each shard replica, the node that services the view request combines the results and streams the final result to the client. As more documents are involved, it takes longer for each replica to stream the results from disk and across the network. The node that services the request also has much more work to do in combining the results from each database shard.

Overall, the goal is for a view request to require the minimum amount of data from each shard. This practice minimizes the time that the data is in transit and being combined to form the final result. Using the power of views to precompute aggregate data is one way to achieve this aim. This practice reduces the time that your application spends waiting for the request to complete.





 How can I de-normalize my data? 

In relational databases, normalizing data is often the most efficient way to store data. This practice makes sense when you can use JOIN to easily combine data from multiple tables. You're more likely to need an HTTP GET request for each piece of data with IBM Cloudant. If you reduce the number of requests you need to build a complete picture of a modeled entity, you can present information to your users more quickly.

By using views, you get many of the benefits of normalized data while you maintain the de-normalized version for efficiency.

As an example, in a relational schema, you'd normally represent tags in a separate table and use a connecting table to join tags with their associated documents. This practice allows quick lookup of all documents with a specific tag.

In IBM Cloudant, you'd store tags in a list in each document.
It is ideal for storage-intensive applications with high I/O needs, such as an OS, database, and application software. This storage is the perfect companion for SAP HANA workloads.

Bare Metals with local disk storage use disks physically inserted into the drive bays; it is not using a SmartNIC to present network block storage as if it were local disk storage.

In addition to the local storage, you might require network storage (to expand beyond the Local SSD Disk total capacity, or for redundancy and backups). These require more network consumption to access Network Block Storage and Network File Storage that are using the same physical network interfaces and the impact should be validated.

Both IBM Cloud Block Storage for Classic or IBM Cloud File Storage for Classic can serve as either backup space or as storage for extra software components that are installed on your server.

All IBM Cloud Block Storage for Classic or IBM Cloud File Storage for Classic is selected based on capacity (GB) and performance (IOPS) measurements.

IOPS are measured based on 16 KB block size with a 50/50 read/write mix. To achieve a maximum I/O throughput, it's advisable to look at the tier and custom profiles available for storage and find the optimal combination of size and IOPS.

Storage volumes differ in performance, depending on their IOPS tier (see [Tiered IOPS profiles](https://cloud.ibm.com/docs/vpc-on-classic-block-storage?topic=vpc-on-classic-block-storage-block-storage-profiles-gen1tiers-gen1) or selecting a [custom size](https://cloud.ibm.com/docs/vpc-on-classic-block-storage?topic=vpc-on-classic-block-storage-block-storage-profiles-gen1custom-gen1) (in GB and IOPS) that is based on the size of the storage.

If you need more than the initially provisioned storage in your virtual server, you can attach extra volumes to a virtual server later. Contact [IBM Cloud Support](https://cloud.ibm.com/docs/get-support?topic=get-support-using-avatargetting-support) for extension options if the attached storage is insufficient for your workload.
All File and Block Storage for Classic services are thin-provisioned. This method is not modifiable.





 My billing ID changed, what does this mean? 

You might notice that your Storage volumes are now billed as "Endurance Storage Service” or "Performance Storage Service" instead of "Enterprise Storage". You may also have new options in the console, such as the ability to adjust IOPS or increase capacity. IBM Cloud® strives to continuously improve storage capabilities. As hardware gets upgraded in the data centers, storage volumes that reside in those data centers are also upgraded to use all enhanced features. The price that you pay for your Storage volume does not change with this upgrade.





 How durable is Block Storage for Classic? 

When you store your data in Block Storage for Classic, it's durable, highly available, and encrypted. The durability target for a single Availability zone is 99.999999999% (11 9's). For more information, see [Availability and Durability of Block Storage for Classic](https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-storageavailability).





 What's the average uptime for Block Storage for Classic? 

When you store your data in Block Storage for Classic, it's durable, highly available, and encrypted. Block Storage for Classic is built upon best-in-class, proven, enterprise-grade hardware and software to ensure high availability and uptime. To ensure that the availability target of 99.999% (five 9's) is met, the data is stored redundantly across multiple physical disks on HA paired nodes. Each storage node has multiple paths to its own Solid-State Drives and its partner node's SSDs as well. This configuration protects against path failure, and also controller failure because the node can still access its partner's disks seamlessly. For more information, see [Availability and Durability of Block Storage for Classic](https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-storageavailability).





 How can I identify a Block Storage for Classic volume from my OS?
: Buckets created using [Key Protect managed encryption](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-kp) can now also use [Immutable Object Storage](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-immutable) to create retention policies that prevent object deletion or modification.





 30 August 2021 

New location!
: Buckets can now be created in a Regional configuration in São Paulo, Brazil. More information can be found in the [Select Regions and Endpoints](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-endpoints).





 12 August 2021 

Encryption update
: New buckets created in us or ap Cross Region configuration can now use [Key Protect managed encryption](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-kp).





 7 July 2021 

New location!
: Buckets can now be created in a Regional configuration in Toronto, Canada. More information can be found in the [Select Regions and Endpoints](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-endpoints).

Metrics update
: In addition to [usage metrics](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-mm-cos-integrationmm-cos-metrics-usage), IBM Cloud Monitoring can now track [request metrics](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-mm-cos-integrationmm-cos-metrics-usage) for buckets.





 31 March 2021 

New feature!
: Buckets can now be [configured to version objects](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-versioning), allowing for non-destructive overwrites and deletes.
Performance Predictable due to assigned IOPS and size. IOPS are not shared between pods. 
 Consistency Strong 
 Durability High 
 Resiliency Medium as specific to a data center. Block storage server is clustered by IBM with redundant networking. 
 Availability Medium as specific to a data center. 
 Scalability Difficult to extend beyond the data center. You can't change an existing storage tier. 
 Encryption Encryption in transit with Key Protect 
 Backup and recovery Set up periodic snapshots, replicate snapshots, duplicate storage, back up data to IBM Cloud Object Storage, or copy data to and from pod and containers. 
 Common use cases Stateful sets, backing storage when you run your own database, or high-performance access for single pods. 
 Non-ideal use cases Multizone clusters, geographically distributed data, or sharing data across multiple app instances. 









 Persistent storage options for multizone clusters 

The following sections show the options that you have in Red Hat OpenShift on IBM Cloud to permanently store your data in a multizone cluster and make your data highly available. You can use these options in a single zone cluster, but you might not get the high availability benefits that your app requires.

Looking to connect your cluster to an on-prem database instead? See [Setting up VPN connectivity to your cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-vpnvpn).

The IBM Cloud Object Storage plug-in uses s3fs to manage your Object Storage buckets and objects. Object Storage, and the plug-in, works best with read-intensive workloads. If your workload is read/write intensive, you might want to try Block or File storage instead.

Zoom

![High availability options for persistent storage in a multizone cluster](https://cloud.ibm.com/docs-content/v1/content/bce0f0917a9eea684d1b4b704ac6343a1f65f446/openshift/images/cs_storage_options_multizone.png)

Figure 1. High availability options for persistent storage in a multizone cluster



 IBM Cloud Object Storage 

Virtual Private Cloud
Inconsistencies that arise due to local or manual changes, or differences in the sequence of automated operations. Changes that make it harder to debug and resolve issues, and increase support costs.

To ensure immutability and eliminate drift, all changes should be made through the Schematics IaC configuration, and resources like VSIs should be redeployed when they need updating.







 Next steps 

Now you understand more about IaC, why not review the use of IaC in Schematics:



* Learn more about the [Open-source tools in Schematics](https://cloud.ibm.com/docs/schematics?topic=schematics-schematics-open-projects)
* Explore these [use cases](https://cloud.ibm.com/docs/schematics?topic=schematics-how-it-works).
These parameters include things like credentials, device information, worker nodes, and more. After you use a template to create a storage configuration, you can reuse that configuration to assign storage across your Satellite clusters.

You can also quickly update the configuration parameters and apply them automatically across clusters, service clusters, and cluster groups in your locations.



Comparison of supported features between using Satellite storage templates and bringing your own storage drivers

 Feature or benefit Satellite storage templates Bring your own drivers 

 Integrated with the Satellite interface. Yes No 
 Quickly and consistently install storage drivers across multiple clusters. Yes No 
 Repeatable, site-specific or storage-specific configurations. Yes No 
 Certified and tested with Satellite. Yes No 







 How do templates work? 

When you create a configuration by using a template, you specify a set of parameters for the storage provider that you want to use. These parameters preset values that are used when you deploy storage drivers, create persistent volumes, provision instances, or use other functions depending on the provider and the type of storage that you want to use. For more information, see the [list of available storage templates](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-template-ovstorage-template-ov-providers).

The following image depicts the workflow for creating a Satellite storage configuration by using a storage template.



1. Select the storage template that you want to use for your configuration.
2. Create a Satellite cluster. Make sure that you select the Enable cluster admin access for Satellite Config option when you create the cluster. If you don't enable Administrator (admin) access for Satellite Config when creating your cluster, you must re-create your cluster and enable admin access before you can deploy storage.
3. Assign your Satellite storage configuration to your clusters.
4. Satellite deploys the storage drivers and any solution-specific resources for the provider that you selected to the clusters that you assigned the storage configuration to.



Zoom

!
Getting started with Block Storage for Classic 

IBM Cloud® Block Storage for Classic is persistent, high-performance iSCSI storage that is provisioned and managed independently of compute instances. iSCSI-based Block Storage for Classic LUNs are connected to authorized devices through redundant multi-path I/O (MPIO) connections.

Block Storage for Classic brings best-in-class levels of durability and availability with an unmatched feature set. It is built by using industry standards and best practices. Block Storage for Classic is designed to protect the integrity of the data and maintain availability through maintenance events and unplanned failures, and provide a consistent performance baseline.

For more information about using Block Storage for Classic with the IBM Cloud® Kubernetes Service, see [Storing data on classic IBM Cloud Block Storage](https://cloud.ibm.com/docs/containers?topic=containers-block_storage).



 Step 1: Before you begin 

Block Storage for Classic LUNs can be provisioned from 20 GB to 12 TB with two options:



* Provision Endurance tiers that feature pre-defined performance levels and other features like snapshots and replication.
* Build a high-powered Performance environment with allocated input/output operations per second (IOPS).



For more information about the Block Storage for Classic offering, see [What is Block Storage for Classic](https://www.ibm.com/cloud/block-storage).





 Step 2: Provisioning considerations 



 Block size 

IOPS for both Endurance and Performance is based on a 16-KB block size with a 50/50 read and write, 50/50 random and sequential workload. A 16-KB block is the equivalent of one write to the volume.

The block size that is used by your application directly impacts the storage performance. If the block size that is used by your application is smaller than 16 KB, the IOPS limit is realized before the throughput limit. Conversely, if the block size that is used by your application is larger than 16 KB, the throughput limit is realized before the IOPS limit.



Table 1 shows examples of how block size and IOPS affect the throughput. Average IO size x IOPS = Throughput in MB/s.
The IBM COS implementation of Immutable Object Storage (i.e. retention policies) is not permitted in buckets with versioning enabled. Attempts to create a retention policy will fail, as will attempts to enable versioning on a bucket with an retention policy. IBM COS does not support AWS S3 APIs for object locking, retention, or legal holds.







 Supported S3 APIs 

The following set of REST APIs can interact with versioning in some way:



* GET Object
* HEAD Object
* DELETE Object
* GET Object ACL
* PUT Object ACL
* Upload Part Copy
* Restore Object
* DELETE Objects
* List Object Versions
* PUT Bucket Versioning
* GET Bucket Versioning
* PUT Object
* POST Object
* Copy Object
* Complete Multipart Upload
* PUT Object Tagging
* GET Object Tagging
* DELETE Object Tagging
* PUT Bucket Lifecycle
* GET Bucket Lifecycle
* DELETE Bucket Lifecycle







 REST API examples 

The following examples are shown using cURL for ease of use. Environment variables are used to represent user specific elements such as $BUCKET, $TOKEN, and $REGION. Note that $REGION would also include any network type specifications, so sending a request to a bucket in us-south using the private network would require setting the variable to private.us-south.



 Enable versioning on a bucket 

=======

curl -X "PUT" "https://$BUCKET.s3.$REGION.cloud-object-storage.appdomain.cloud/?versioning" -H 'Authorization: bearer $TOKEN' -H 'Content-MD5: 8qj8HSeDu3APPMQZVG06WQ==' -H 'Content-Type: text/plain; charset=utf-8' -d $'<VersioningConfiguration>
<Status>Enabled</Status>
</VersioningConfiguration>'

A successful request returns a 200 response.





 Suspend versioning on a bucket 

=======