vania-janet/multiturn-rag-retrieval-data huggingface dataset:
* UI
* CLI





This document outlines the process for using context-based restrictions to protect your Cloud Databases resources. Use this document to prepare your resources for context-based restrictions. Cloud Databases doesn't offer scoping rules to the control plane in this current phase of implementation.



 Protecting Cloud Databases resources with context-based restrictions 

Context-based restrictions give account owners and administrators the ability to define and enforce access restrictions for IBM Cloud® resources based on the context of access requests. Access to Cloud Databases resources can be controlled with context-based restrictions and Identity and Access Management (IAM) policies.

These restrictions work with traditional IAM policies, which are based on identity, to provide an extra layer of protection. Unlike IAM policies, context-based restrictions don't assign access. Context-based restrictions check that an access request comes from an allowed context that you configure. Since both IAM access and context-based restrictions enforce access, context-based restrictions offer protection even in the face of compromised or mismanaged credentials. For more information, see [What are context-based restrictions](https://cloud.ibm.com/docs/account?topic=account-context-restrictions-whatis).

A user must have the Administrator role on the Cloud Databases service to create, update, or delete rules. A user must also have either the Editor or Administrator role on the context-based restrictions service to create, update, or delete network zones. A user with the Viewer role on the context-based restrictions service can add network zones to a rule.

Any IBM Cloud Activity Tracker or audit log events generated come from the context-based restrictions service, not Cloud Databases. Cloud Databases supports audit events only for customer interactions with context-based restrictions-protected platform endpoint calls. Cloud Databases does not support audit events when you enable context-based restrictions rules on the control plane API for your instances. For more information, see [Monitoring context-based restrictions](https://cloud.ibm.com/docs/account?topic=account-cbr-monitor).
* UI
* CLI
* Terraform





This document outlines the process for using context-based restrictions to protect your Cloud Databases resources. Use this document to prepare your resources for context-based restrictions. Cloud Databases doesn't offer scoping rules to the control plane in this current phase of implementation.



 Protecting Cloud Databases resources with context-based restrictions 

Context-based restrictions give account owners and administrators the ability to define and enforce access restrictions for IBM Cloud® resources based on the context of access requests. Access to Cloud Databases resources can be controlled with context-based restrictions and Identity and Access Management (IAM) policies.

These restrictions work with traditional IAM policies, which are based on identity, to provide an extra layer of protection. Unlike IAM policies, context-based restrictions don't assign access. Context-based restrictions check that an access request comes from an allowed context that you configure. Since both IAM access and context-based restrictions enforce access, context-based restrictions offer protection even in the face of compromised or mismanaged credentials. For more information, see [What are context-based restrictions](https://cloud.ibm.com/docs/account?topic=account-context-restrictions-whatis).

A user must have the Administrator role on the Cloud Databases service to create, update, or delete rules. A user must also have either the Editor or Administrator role on the context-based restrictions service to create, update, or delete network zones. A user with the Viewer role on the context-based restrictions service can add network zones to a rule.

Any IBM Cloud Activity Tracker or audit log events generated come from the context-based restrictions service, not Cloud Databases. Cloud Databases supports audit events only for customer interactions with context-based restrictions-protected platform endpoint calls. Cloud Databases does not support audit events when you enable context-based restrictions rules on the control plane API for your instances. For more information, see [Monitoring context-based restrictions](https://cloud.ibm.com/docs/account?topic=account-cbr-monitor).
Documents with similar document IDs are unlikely to be placed onto the same shard.

A non-partitioned database offers only global querying, described in more detail later.





 Partitioned databases 

A partitioned database is the newest type of IBM Cloudant database. Within a partitioned database, documents are formed into logical partitions by use of a partition key. The partition key is part of the document ID for documents within a partitioned database. All documents are assigned to a partition, and many documents are typically given the same partition key. A partition's primary JSON data and its indexes end up colocated, meaning that the database can query data within a partition more efficiently.

A partitioned database offers both partitioned and global querying. Partitioned querying takes advantage of the data layout within the database cluster to deliver improved and more scalable query performance. Partition queries are also often cheaper than global queries.

As partitioned databases offer the advantages of both global and partition querying, IBM Cloudant recommends that new applications take advantage of them.





 What makes a good partition key? 

If you're thinking of using IBM Cloudant's new partitioned database feature, then the choice of a partition key is important. A partition key must have:



* Many values - lots of small partitions are better than a few large ones. A million partitions are perfectly fine, but keep each partition under 10 GB in total size.
* No hot spots - avoid designing a system that makes one partition handle a high proportion of the workload. If the work is evenly distributed around the partitions, the database performs more smoothly.
* Repeating - If each partition key is unique, one document per partition exists. To get the best out of partitioned databases, multiple documents per partition must exist - documents that logically belong together.



Let's look at some use cases and some good and bad choices for a partition key.



Table 1. Good and bad choices for a partition key

 Use Case Description Partition Key Effectiveness 

 E-commerce system - orders One document per order order_id Neutral - one document per partition is fine, but it doesn't provide the benefits of Partition Queries.
[Memory](https://cloud.ibm.com/docs/vpc?topic=vpc-profilesmemory) Memory profiles offer a core to RAM ratio 1 vCPU to 8 GiB of RAM ratio and are best for memory caching and real-time analytics workloads. Memory profiles are best for memory intensive workloads, such as large caching workloads, intensive database applications, or in-memory analytics workloads. 
 [Very High Memory](https://cloud.ibm.com/docs/vpc?topic=vpc-profilesvhmemory) Very High Memory profiles offer a core to RAM ratio of 1 vCPU to 14 GiB of RAM. This family is optimized for running small to medium in-memory databases and OLAP workloads, such as SAP BW/4 HANA. 
 [Ultra High Memory](https://cloud.ibm.com/docs/vpc?topic=vpc-profilesuhmemory) Ultra High Memory profiles offer the most memory per core with 1 vCPU to 28 GiB of RAM. These profiles are optimized to run large in-memory databases and OLTP workloads, such as SAP S/4 HANA. 
 [GPU](https://cloud.ibm.com/docs/vpc?topic=vpc-profilesgpu) GPU enabled profiles provide on-demand access to NVIDIA V100 and A100 GPUs to accelerate AI, high-performance computing, data science, and graphics workloads. 
 [Storage Optimized](https://cloud.ibm.com/docs/vpc?topic=vpc-profilesstorageopt) Storage Optimized profiles offer temporary SSD instance storagedisks at a ratio of 1 vCPU to 300 GB instance storage with a lower price point per GB. These profiles are designed for storage-dense workloads and offer virtio interface type for attached disks.
* API
* Terraform







 MongoDB Enterprise Edition Analytics Add-On 

The Databases for MongoDB EE (Enterprise Edition) Analytics Add-On allows you to run long-running analytical queries or provision a [MongoDB Connector for business intelligence(BI)](https://docs.mongodb.com/bi-connector/current/) to make your query data compatible with BI tools, such as [Tableau](https://www.tableau.com/).

The Databases for MongoDB EE Analytics Add-On is made up of two components:



* [The Analytics node](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-mongodbee-analyticsmongodbee-analytics-node)
* [The connector for Business Intelligence (BI)](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-mongodbee-analyticsmongodbee-analytics-connector-bi)





 What problems does the Databases for MongoDB EE Analytics Add-On solve? 



1. Most BI tools do not work with the MongoDB document data model.

MongoDB's document data model is made up of complex documents with arbitrary, nested data. This schema makes storing data flexible, easy, and scalable. However, most BI tools require data to be in tabular format, which is rigidly defined and stored in tables, not documents. The Databases for MongoDB EE Analytics Add-On converts MongoDB document data into SQL-readable tabular data that can be queried and displayed by BI tools.
2. BI queries are expensive and can degrade database performance.

Long-running queries can negatively impact the operational workflow of your deployment. The Databases for MongoDB EE Analytics Add-On introduces an extra data member, which isolates analytics workloads from operational workloads. The Analytics Node data is kept in sync with the other nodes, so any queries run against it produce the same results.





 The Analytics Node
[Memory](https://cloud.ibm.com/docs/vpc?topic=vpc-profiles&interface=uimemory) Memory profiles offer a core to RAM ratio 1 vCPU to 8 GiB of RAM ratio and are best for memory caching and real-time analytics workloads. Memory profiles are best for memory intensive workloads, such as large caching workloads, intensive database applications, or in-memory analytics workloads. 
 [Very High Memory](https://cloud.ibm.com/docs/vpc?topic=vpc-profiles&interface=uivhmemory) Very High Memory profiles offer a core to RAM ratio of 1 vCPU to 14 GiB of RAM. This family is optimized for running small to medium in-memory databases and OLAP workloads, such as SAP BW/4 HANA. 
 [Ultra High Memory](https://cloud.ibm.com/docs/vpc?topic=vpc-profiles&interface=uiuhmemory) Ultra High Memory profiles offer the most memory per core with 1 vCPU to 28 GiB of RAM. These profiles are optimized to run large in-memory databases and OLTP workloads, such as SAP S/4 HANA. 
 [GPU](https://cloud.ibm.com/docs/vpc?topic=vpc-profiles&interface=uigpu) GPU enabled profiles provide on-demand access to NVIDIA V100 and A100 GPUs to accelerate AI, high-performance computing, data science, and graphics workloads. 
 [Storage Optimized](https://cloud.ibm.com/docs/vpc?topic=vpc-profiles&interface=uistorageopt) Storage Optimized profiles offer temporary SSD instance storagedisks at a ratio of 1 vCPU to 300 GB instance storage with a lower price point per GB. These profiles are designed for storage-dense workloads and offer virtio interface type for attached disks.
If a document needs removal through a right to be forgotten request, follow these steps:



1. File a request with the [IBM DPO](https://w3-03.ibm.com/ibm/privacy/index.html) to request purging of specific document _id values along with the reason.
2. On receipt of a formal request by the IBM DPO, IBM Cloudant operations verifies the request to confirm the id contains PI. IBM Cloudant doesn't purge data that doesn't have PI in the _id.
3. IBM Cloudant triggers the purging action to permanently remove the requested data.



This process is only to be used for emergency deletion requests (for example, right to be forgotten) and must not be relied upon long term. If your application is intentionally using PI in document IDs, then it must be changed to either pseudonymize that PI, or not use PI in document IDs. You cannot rely on regular purging by the IBM Cloudant operations team to avoid this situation. Therefore, IBM Cloudant rejects the following purge requests:



1. The request is for regular purging, for example, every 30 days.
2. The request is for over 100 documents.



Even with purge, PI in the _id field leaks into places you don't want it, such as IBM Cloudant logs, so it must be avoided. IBM Cloudant has a business reason to keep those logs and doesn't remove log lines that include document _id values.





 What about deleting a database? 

Deleting a database adds it to the trash for up to 48 hours. After which time, the database is removed from the file system. The IBM Cloudant team does not make back ups of your databases; this task is the responsibility of the customer. You must ensure that all copies of your database are removed from your system. For more information, see [IBM Cloudant backup and recovery](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-ibm-cloudant-backup-and-recoveryibm-cloudant-backup-and-recovery).
Add "create_target":true to the JSON document that describes the replication if the target database does not exist before replication. For more information, see [Creating a target database during replication](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-replication-apicreating-a-target-database-during-replication).
* Replicator databases must be maintained and looked after, just like any other valuable data store. For more information, see [replication database maintenance](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-advanced-replicationreplication-database-maintenance).



For security purposes, the IBM Cloudant team recommends that you use IAM API keys or IBM Cloudant legacy authentication [API keys](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-work-with-your-accountapi-keys) rather than account-level credentials for replication jobs. For more information, see the [IAM guide](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-managing-access-for-cloudant) or the legacy [Authentication API document](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-work-with-your-accountauthentication) and the legacy [Authorization API document](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-work-with-your-accountauthorization).
The _replicator database is a special database within your account, where you can PUT or POST replication documents to specify the replications you want.

Before you start a replication, you must create the _replicator database. To create a database, send a PUT request to:

https://$ACCOUNT.cloudant.com/_replicator

For more information, see [Databases](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-databases).

To cancel a replication, you DELETE the replication document. The fields that are supplied in the replication document are described in the [Create or modify a replication operation](https://cloud.ibm.com/apidocs/cloudantpostreplicate) description under Request information.

All design documents and _local documents that are added to the /_replicator database are ignored.





 Important notes 



* A new and more powerful [replication scheduler](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-advanced-replicationthe-replication-scheduler) changes the previous behavior of the IBM Cloudant replication mechanisms. Ensure that your applications are updated.
* Replications can severely impact the performance of an IBM Cloudant instance. Performance testing helps you understand the impact on your environment under an increasing number of concurrent replications.
* [Continuous replication](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-replication-apicontinuous-replication) can result in many internal calls. Requiring many calls might affect the costs for multi-tenant users of IBM Cloudant systems. By default, continuous replication is not enabled.
* The target database must exist. It is not automatically created if it does not exist. Add "create_target":true to the JSON document that describes the replication if the target database does not exist before replication. For more information, see [Creating a target database during replication](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-replication-apicreating-a-target-database-during-replication).
* Replicator databases must be maintained and looked after, just like any other valuable data store.
Document versioning and conflicts FAQ 

When you use distributed databases, copies of your data might be stored in multiple locations. Keeping this data in sync is important. However, your work environment might prevent your users from updating documents with their changes immediately, or even replicating to the database.

As a result, the copies of a document might have different updates. "Conflicts" occur because IBM® Cloudant® for IBM Cloud® can't determine which copy is the correct one.

IBM Cloudant uses multi-version concurrency control (MVCC) to ensure that all nodes in each database cluster include only the newest version of a document.



 What is eventual consistency? 

IBM Cloudant databases are [eventually consistent](https://cloud.ibm.com/docs/services/Cloudant?topic=Cloudant-cap-theoremcap-theorem), which means IBM Cloudant must ensure that no differences exist between nodes. These inconsistencies can happen when out-of-date documents are synchronized.

It's important for IBM Cloudant databases to have concurrent read and write access. MVCC enables that capability. MVCC is a form of [optimistic concurrency control](https://en.wikipedia.org/wiki/Optimistic_concurrency_control) that makes read and write operations on IBM Cloudant databases faster because a database lock isn't necessary for read and write operations. At the same time, MVCC enables synchronization between IBM Cloudant database nodes.





 How do I know whether there's a conflict? 

You don't know. Sometimes you request a document that has a conflict. At those times, IBM Cloudant returns the document normally, as though no conflict exists. However, the version that is returned isn't necessarily the most current version. Instead, the version is selected based on an internal algorithm that considers multiple factors. You must not assume that when documents are returned they're always the most current.





 How do I identify a document with a conflict? 

If a conflict with a document exists and you try to update it, IBM Cloudant returns a 409 response.
Performance 

IBM Cloud® Databases for Redis deployments deployments can be both manually [to your usage](https://cloud.ibm.com/docs/databases-for-redis?topic=databases-for-redis-resources-scaling), or configured to [autoscale](https://cloud.ibm.com/docs/databases-for-redis?topic=databases-for-redis-autoscaling) under certain resource conditions. There are a few factors to consider if you are tuning the performance of your deployment.



 Monitoring your deployment 

Databases for Redis deployments offer an integration with the [IBM Cloud® Monitoring service](https://cloud.ibm.com/docs/databases-for-redis?topic=databases-for-redis-monitoring) for basic monitoring of resource usage on your deployment. Many of the available metrics, like disk usage and IOPS, are presented to help you configure [autoscaling](https://cloud.ibm.com/docs/databases-for-redis?topic=databases-for-redis-autoscaling) on your deployment. Observing trends in your usage and configuring the autoscaling to respond to them can help alleviate performance problems before your databases become unstable due to resource exhaustion.





 Memory Policies 

By default, deployments are configured with a noeviction policy. All data is kept in memory until the maxmemory limit is reached and Redis returns an error if the memory limit is exceeded. The maxmemory is set to 80% of a data node's available memory, so your node doesn't run out of system resources.

You can scale the amount of memory to accommodate more data, and you can configure the maxmemory setting to tune memory usage. The [Redis documentation](https://redis.io/topics/memory-optimizationmemory-allocation) has some good information on memory behavior and tuning maxmemory.
"patient": "Joe McIllness",
"surgery": "lumbar puncture"
}

Emitting the "patient" field as the key in your view would then allow querying for all operations for a specific patient. Again, views are used to help knit together a full picture of a specific entity from separate documents. Views help keep the number of HTTP requests low, even though IBM Cloudant splits up the data for a single-modeled entity.





Avoiding conflicted documents helps speed up many operations on your IBM Cloudant databases. There’s a process that works out the current winning revision used each time that the document is read, for example, single document retrievals, calls with include_docs=true, view building, and so on.

The winning revision is a particular revision from the document’s overall tree. Recall that documents on IBM Cloudant are in fact trees of revisions. An arbitrary but deterministic algorithm selects one of the non-deleted leaves of this tree to return when a request is made for the document. Larger trees with a higher branching factor take longer to process than a document tree with no or few branches: each branch needs to be followed to see whether it’s a candidate to be the winning revision. Potential victors then need to be compared against each other to make the final choice.

IBM Cloudant handles small numbers of branches well. After all, replication relies on the fact that documents can branch to avoid discarding data. However, when you reach pathological levels, particularly if you can't resolve the conflicts, it becomes time-consuming and memory-intensive to walk the document tree.





In an eventually consistent system like IBM Cloudant, conflicts eventually happen. This fact is a price of scalability and data resilience.

It is best to structure your data so that resolving conflicts is quick and does not involve operator assistance. This practice helps your databases to hum along smoothly. The ability to automatically resolve conflicts without user involvement significantly improves their experience and reduces the support burden on your organization.

How you resolve conflicts is application-specific. See the following tips for more ways to improve the process:



* Avoid invariants across document fields if possible.