vania-janet/multiturn-rag-retrieval-data huggingface dataset:
This script is unsupported and exists as an example of what you can build to move Databases for Redis data from one instance to another as part of a version upgrade. All source code and/or binaries that are attached to this document are referred to here as "the Program". IBM is not providing program services of any kind for the Program. IBM is providing the Program on an "AS IS" basis without warranty of any kind. IBM WILL NOT BE LIABLE FOR ANY ACTUAL, DIRECT, SPECIAL, INCIDENTAL, OR INDIRECT DAMAGES OR FOR ANY ECONOMIC CONSEQUENTIAL DAMAGES (INCLUDING LOST PROFITS OR SAVINGS), EVEN IF IBM, OR ITS RESELLER, HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Copying data from your old Databases for Redis deployment to a new version involves running a simple Python script [available on GitHub](https://github.com/IBM-Cloud/clouddatabases-migration-examples/blob/master/redis/redis_migration.py). The script copies all the keys from your source database over to your new Databases for Redis deployment. Download the script and make sure that you have Python 3.x installed. If you’re on macOS, you can use homebrew to install it running the command brew install python3, which will give you the latest version.

Next, we recommend creating a migration window to let your users know that you are doing some maintenance. That way you have some time to migrate all your data over to your new Databases for Redis deployment. If you are using Redis as a key-value store with expire times on keys, know that these expiry times are copied over to your new database. Our example does this with 10 million keys in our database, which doesn't take much time to migrate depending on your bandwidth.





 Getting the destination and source database credentials 

Now, you need to have the credentials of both the old source database and your new Databases for Redis deployment. You can get the credentials for both Databases for Redis deployments by clicking the respective databases from the IBM Cloud resources page. Then, click the Service credentials link from the left menu that takes you to the Service credentials view.
Or so that you’re able to rely on the built-in reducers.

Views on partitioned databases do not support custom reduces, which is one factor that contributes to the significant speed-up queries only such views can offer.

For more information, see IBM Cloudant docs on [reduces](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-creating-views-mapreducereduce-functions).





 Use time boxed databases for ever-growing data sets 

It’s generally not a good idea to have an ever-growing database in IBM Cloudant. Large databases can be difficult to back up, require “resharding” to maintain good performance as they grow, and suffer from long index build times.

One way of mitigating this problem is to have several smaller databases instead, with a common pattern that is time-boxed databases: a large data set is split into smaller databases, each representing a time window, for example, a month.



* orders_2019_01
* orders_2019_02
* orders_2019_02



New data is written to this month’s database and queries for historical data can be directed to previous months’ databases. When a month’s data is no longer of interest, it can be archived to Object Storage, the monthly IBM Cloudant database is deleted, and the disk space recovered. For more information, see the following website.



* [Time-series Data Storage blog](https://blog.cloudant.com/2019/04/08/Time-series-data-storage.html)
IBM Cloudant databases are [eventually consistent](https://cloud.ibm.com/docs/services/Cloudant?topic=Cloudant-cap-theoremcap-theorem), which means IBM Cloudant must ensure that no differences exist between nodes. These inconsistencies can happen when out-of-date documents are synchronized.

It's important for IBM Cloudant databases to have concurrent read and write access. MVCC enables that capability. MVCC is a form of [optimistic concurrency control](https://en.wikipedia.org/wiki/Optimistic_concurrency_control) that makes read and write operations on IBM Cloudant databases faster because a database lock isn't necessary for read and write operations. At the same time, MVCC enables synchronization between IBM Cloudant database nodes.
* How do I know whether there's a conflict?

You don't know. Sometimes you request a document that has a conflict. At those times, IBM Cloudant returns the document normally, as though no conflict exists. However, the version that is returned isn't necessarily the most current version. Instead, the version is selected based on an internal algorithm that considers multiple factors. You must not assume that when documents are returned they're always the most current.
* How do I identify a document with a conflict?

If a conflict with a document exists and you try to update it, IBM Cloudant returns a 409 response. If you try to update a document while you're offline, IBM Cloudant can't check for potential conflicts, and you don't receive a 409 response.

When this situation happens, it's best to check for document conflicts when you're back online. If you need to find document conflicts, use the following example map function:

function (doc) {
if (doc._conflicts) {
emit(null, [doc._rev].concat(doc._conflicts));
}
}

If you want to find conflicts within multiple documents in a database, write a [view](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-creating-views-mapreduce).
If changes_pending increases, and revisions_checked also increases, these increases might indicate that the replication can't keep up with the volume of data added to, or updated in, the database.





 What to do about a stuck task? 

To resolve a stalled replication, you might have to [cancel the replication process](https://cloud.ibm.com/apidocs/cloudantdeletereplicationdocument) and start it again.

If that doesn't help, the replication might be stalled because the user who is accessing the source or target databases doesn't have write permissions.

Replication uses [checkpoints](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-replication-guidecheckpoints), which means that content that is already replicated and unchanged doesn't have to be replicated again if the replication is restarted.

If you created the replication process by creating a document in the _replicator database, you can also check the status of the replication there.
terraform output -json > ../import-covid-data/output.json

The Terraform folder contains a number of simple scripts:



* main.tf tells Terraform to use IBM Cloud.
* variables.tf contains the variable definitions whose values are populated from the terraform.tfvars file.
* mongo.tf creates a Databases for MongoDB EE instance with the Analytics Add-On with the following options:



* 6 CPU cores
* 14 GB RAM
* 20 GB memory.





The Terraform script outputs connection strings that are stored in a file named output.json. This file is used in the following step.





 Step 5: Upload data to your Databases for MongoDB EE instance 

In this step, you use a Node.js script to upload data to your Databases for MongoDB EE instance. The [World Health Organization (WHO) Coronavirus (COVID-19) Dashboard](https://covid19.who.int/data) provides a daily update of COVID-19 data (deaths, vaccine utilization, and cases).

Ensure you are in your project directory, then use the following command to import this data into your Databases for MongoDB EE instance.

cd import-covid-data
npm install
node main.js

The npm install command installs all dependencies that are needed by your script. The main.js script does a number of things:



* It reads in the connection strings so that it can connect to your Databases for MongoDB EE instance.
* It reads in your terraform.tfvars credentials file for the password to access your Databases for MongoDB EE instance.
* It reads the COVID data file from the WHO Dashboard. This data file is in CSV format and MongoDB requires JSON format, so the script reads the data and converts it into a JSON document.
* It connects to your Databases for MongoDB EE instance and uploads all data (200k+ rows) into a collection named daily_covid_global_data. If the collection exists, it deletes it and starts again. You can run the script multiple times without importing duplicate data.
cloudantnosqldb.replicator-database.create Create _replicator database. 
 cloudantnosqldb.replicator-database.delete Delete _replicator database. 
 cloudantnosqldb.users-database.create Create _users database. 
 cloudantnosqldb.users-database.delete Delete _users database. 
 cloudantnosqldb.database-security.read Read a security document. 
 cloudantnosqldb.database-security.write A create, update, or delete of a security document. 
 cloudantnosqldb.design-document.write A create, update, or delete of a _design document. 
 cloudantnosqldb.replication.read Read a replication document. 
 cloudantnosqldb.replication.write A create, update, or delete of a replication document. 
 cloudantnosqldb.sapi.apikeys Create a legacy Cloudant API key for an instance. 
 cloudantnosqldb.sapi.db-security Read or write a database's security document. 
 cloudantnosqldb.sapi.usercors Set and get the CORS settings for an instance. 
 cloudantnosqldb.sapi.userplan Get or set plan and plan settings of an instance. 
 cloudantnosqldb.users.write Create, update, or delete a _users document. 
 cloudantnosqldb.volumes.update A change in state of the IBM® Key Protect for IBM Cloud® key protecting dedicated hardware environment. 







 Data events 



Table 2. Data actions

 Action Description 

 cloudantnosqldb.any-document.read Read a JSON document. 
 cloudantnosqldb.account-all-dbs.read Read a list of all databases. 
 cloudantnosqldb.account-dbs-info.read Read metadata about a database. 
 cloudantnosqldb.account-db-updates.read Read _db_updates endpoint. 
 cloudantnosqldb.account-search-analyze.execute Read search index statistics and size.
There’s a process that works out the current winning revision used each time that the document is read, for example, single document retrievals, calls with include_docs=true, view building, and so on.

The winning revision is a particular revision from the document’s overall tree. Recall that documents on IBM Cloudant are in fact trees of revisions. An arbitrary but deterministic algorithm selects one of the non-deleted leaves of this tree to return when a request is made for the document. Larger trees with a higher branching factor take longer to process than a document tree with no or few branches: each branch needs to be followed to see whether it’s a candidate to be the winning revision. Potential victors then need to be compared against each other to make the final choice.

IBM Cloudant handles small numbers of branches well. After all, replication relies on the fact that documents can branch to avoid discarding data. However, when you reach pathological levels, particularly if you can't resolve the conflicts, it becomes time-consuming and memory-intensive to walk the document tree.





 How can I build in conflict resolution? 

In an eventually consistent system like IBM Cloudant, conflicts eventually happen. This fact is a price of scalability and data resilience.

It is best to structure your data so that resolving conflicts is quick and does not involve operator assistance. This practice helps your databases to hum along smoothly. The ability to automatically resolve conflicts without user involvement significantly improves their experience and reduces the support burden on your organization.

How you resolve conflicts is application-specific. See the following tips for more ways to improve the process:



* Avoid invariants across document fields if possible. Avoiding invariants makes it more likely that a simple merge operation, if you take the changed field from each conflicted document revision, is suitable. This practice makes simpler and more robust application code.
* Allow documents to be independent. If you have to retrieve other documents to work out the correct resolution, it increases latency in conflict resolution. There's also a chance you get a version of the other documents that aren't consistent with the document you're resolving, making correct resolution difficult.







 Why do I need to build in conflict resolution
2. Confirm that a client is trying to modify the latest version of a document.



You must specify the previous _rev when you [update a document](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-documentsupdate) or else your request fails and returns a [409 error](https://cloud.ibm.com/apidocs/cloudantlist-of-http-codes).

_rev must not be used to build a version control system because it is an internal value that is used by the server. Therefore, older revisions of a document are transient, and removed regularly.

However, you can query a particular revision by using its _rev, but older revisions are regularly deleted by a process called [compaction](https://en.wikipedia.org/wiki/Data_compaction). You can query a particular document revision by using its _rev in order to obtain a history of revisions to your document. However, a consequence of compaction is that you cannot rely on a successful response. If you need a version history of your documents, a solution is to [create a new document](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-documentscreate-document) for each revision.





 Distributed databases and conflicts 

Distributed databases work without a constant connection to the main database on IBM Cloudant, which is itself distributed, so updates based on the same previous version can still be in conflict.

To find conflicts, add the query parameter [conflicts=true](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-databasesget-changes) when you retrieve a document. The response contains a _conflicts array with all conflicting revisions.

To find conflicts for multiple documents in a database, write a view.

The following map function is an example that emits all conflicting revisions for every document that has a conflict.

See the following example of a map function to find documents with a conflict:

function (doc) {
if (doc._conflicts) {
emit(null, [doc._rev].concat(doc._conflicts));
}
}
There’s a process that works out the current winning revision used each time that the document is read, for example, single document retrievals, calls with include_docs=true, view building, and so on.

The winning revision is a particular revision from the document’s overall tree. Recall that documents on IBM Cloudant are in fact trees of revisions. An arbitrary but deterministic algorithm selects one of the non-deleted leaves of this tree to return when a request is made for the document. Larger trees with a higher branching factor take longer to process than a document tree with no or few branches: each branch needs to be followed to see whether it’s a candidate to be the winning revision. Potential victors then need to be compared against each other to make the final choice.

IBM Cloudant handles small numbers of branches well. After all, replication relies on the fact that documents can branch to avoid discarding data. However, when you reach pathological levels, particularly if you can't resolve the conflicts, it becomes time-consuming and memory-intensive to walk the document tree.
* How can I build in conflict resolution?

In an eventually consistent system like IBM Cloudant, conflicts eventually happen. This fact is a price of scalability and data resilience.

It is best to structure your data so that resolving conflicts is quick and does not involve operator assistance. This practice helps your databases to hum along smoothly. The ability to automatically resolve conflicts without user involvement significantly improves their experience and reduces the support burden on your organization.

How you resolve conflicts is application-specific. See the following tips for more ways to improve the process:



* Avoid invariants across document fields if possible. Avoiding invariants makes it more likely that a simple merge operation, if you take the changed field from each conflicted document revision, is suitable. This practice makes simpler and more robust application code.
* Allow documents to be independent. If you have to retrieve other documents to work out the correct resolution, it increases latency in conflict resolution. There's also a chance you get a version of the other documents that aren't consistent with the document you're resolving, making correct resolution difficult.



* Why do I need to build in conflict resolution
Performance 

IBM Cloud® Databases for MongoDB deployments can be both manually [scaled to your usage](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-resources-scaling), or configured to [autoscale](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-autoscaling) under certain resource conditions. If you are tuning the performance of your deployment, consider a few factors first.



 Monitoring your deployment 

Databases for MongoDB deployments offer an integration with the [IBM Cloud® Monitoring service](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-monitoring) for basic monitoring of resource usage on your deployment. Many of the available metrics, like disk usage and IOPS, are presented to help you configure [autoscaling](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-autoscaling) on your deployment. Observing trends in your usage and configuring the autoscaling to respond to them can help alleviate performance problems before your databases become unstable due to resource exhaustion.





 Disk Usage 

If you are concerned about how much space MongoDB is using to store your data, you can run some native MongoDB [data storage diagnostics](https://docs.mongodb.com/manual/faq/storage/data-storage-diagnostics) to find the sizes of things like databases, collections, and indexes. If the approximate size of your data set is known and fixed, you can manually scale your disk to accommodate your data. If your data set grows at predictable rate over time, you can configure autoscaling to increase disk size when your disk usage hits a certain threshold.





 Disk I/O 

The number of Input-Output Operations per second (IOPS) on Databases for MongoDB deployments is limited by the type of storage volume.
To get the best out of partitioned databases, multiple documents per partition must exist - documents that logically belong together.



Let's look at some use cases and some good and bad choices for a partition key.



Table 1. Good and bad choices for a partition key

 Use Case Description Partition Key Effectiveness 

 E-commerce system - orders One document per order order_id Neutral - one document per partition is fine, but it doesn't provide the benefits of Partition Queries. 
 E-commerce system - orders One document per order user_id Good - all of a user's orders are kept together. 
 E-commerce system - orders One document per order status Bad - grouping orders by a handful of status values (provisional, paid, refunded, cancelled) create too few over-large partitions. 
 Blogging platform One document per blog post author_id Good - if many authors participate. Easy to query each author's posts. 
 IOT - sensor readings One document per reading device_id Good - if many devices exist, make sure that one device isn't producing many more readings than the others. 
 IOT - sensor readings One document per reading date Bad - current readings cause a "hot spot" on the current date's partition. 



Some use cases exist where no viable choice for a partition key exists. In these situations, it's likely a non-partitioned database is the best choice. For example, a database of users that stores email addresses, password hashes, and last-login dates. None of these fields make for a suitable partition key, so a normal non-partitioned database must be used instead.





 Querying 

IBM Cloudant's query types are available for global and partition queries. See also a brief overview of the underlying querying mechanism that you use to select the query mechanism that is best for each query your application needs to make.



 Global querying 

You can make global queries to the following index types:



* IBM Cloudant Query
* Views
* Search



When you make a global query, the database must perform a scatter-gather operation across all data in the database. This action means making requests of many individual database servers.
cloudantnosqldb.any-document.read Read a JSON document. 
 cloudantnosqldb.account-all-dbs.read Read a list of all databases. 
 cloudantnosqldb.account-dbs-info.read Read metadata about a database. 
 cloudantnosqldb.account-db-updates.read Read _db_updates endpoint. 
 cloudantnosqldb.account-search-analyze.execute Read search index statistics and size. 
 cloudantnosqldb.account-uuids.read Read _uuids endpoint. 
 cloudantnosqldb.account-active-tasks.read Read _active_tasks. 
 cloudantnosqldb.current-throughput.read Get the current consumption of the provisioned throughput capacity for an instance. 
 cloudantnosqldb.database-ensure-full-commit.execute Post to _ensure_full_commit endpoint. 
 cloudantnosqldb.database-info.read Read database metadata. 
 cloudantnosqldb.data-document.write Write a JSON document. 
 cloudantnosqldb.iam-session.read Read IAM session. 
 cloudantnosqldb.iam-session.write Write IAM session. 
 cloudantnosqldb.iam-session.delete Delete IAM session. 
 cloudantnosqldb.ibmid-login.authenticate Complete IAM authentication on the Cloudant Dashboard. 
 cloudantnosqldb.ibmid-login.receive Part of the Cloudant Dashboard login with IAM authentication. 
 cloudantnosqldb.ibmid-login.start Initiate Cloudant Dashboard login with IAM authentication. 
 cloudantnosqldb.local-document.write Write a _local document. 
 cloudantnosqldb.replicator-database-info.read Read _replicator database information. 
 cloudantnosqldb,replicator-design-document.write Write a _design document to the _replicator database. 
 cloudantnosqldb.replicator-local-document.write Write a _local document to the _replicator database.
Performance 

IBM Cloud® Databases for MySQL deployments can be manually [scaled to your usage](https://cloud.ibm.com/docs/databases-for-mysql?topic=databases-for-mysql-resources-scalingresources-scaling-consider) or configured to [autoscale](https://cloud.ibm.com/docs/databases-for-mysql?topic=databases-for-mysql-autoscaling-mysql) under certain resource conditions. There are a few factors to consider when you are tuning the performance of your deployment.



 Monitoring your deployment 

Databases for MySQL deployments offer an integration with the [IBM Cloud® Monitoring service](https://cloud.ibm.com/docs/databases-for-mysql?topic=databases-for-mysql-monitoring) for basic monitoring of resource usage on your deployment. Many of the available metrics, like disk usage and IOPS, are presented to help you configure [autoscaling](https://cloud.ibm.com/docs/databases-for-mysql?topic=databases-for-mysql-autoscaling) on your deployment. Observing trends in your usage and configuring the autoscaling to respond to them can help alleviate performance problems before your databases become unstable due to resource exhaustion.





 Disk IOPS 

The number of input/output operations per second (IOPS) is limited by the type of storage volume. Storage volumes for Databases for MySQL deployments are provisioned on [Block Storage Endurance Volumes in the 10 IOPS per GB tier](https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-orderingBlockStorage&interface=ui). If your operational load saturates or exceeds the IOPS limit, database requests and operations are delayed until the disk can catch up. Extended periods of heavy-load can cause your deployment to be unable to process queries and become effectively unavailable. If you experience delayed responses and failing operations, you might be exceeding the disk's IOPS limit.