vania-janet/multiturn-rag-retrieval-data huggingface dataset:
If a design document is updated, IBM Cloudant deletes the indexes from the previous version, and re-creates the index from scratch. If you need to change a design document for a larger database, have a look at the [Design document management guide](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-design-document-managementmanaging-changes-to-a-design-document).

A design document's structure includes the following parts:

_id
: Design document ID. This ID is always prefixed _design and never includes a partition key, regardless of database partitioning type.

_rev
: Design document revision.

Options
: Contains options for this design document.

Partitioned (optional, boolean)
: Determines whether this design document describes partitioned or global indexes. For more information, see [The options.partitioned field](the-options-partitioned-field).

Views (optional)
: An object that describes MapReduce views.

Viewname
: (One for each view) - View Definition.

Map
: Map Function for the view.

Reduce (optional)
: Reduce Function for the view.

Indexes (optional)
: An object that describes search indexes.

Index name
: (One for each index) - Index definition.

Analyzer
: Object that describes the analyzer to be used or an object with the following fields:

Name
: Name of the analyzer. Valid values are standard, email, keyword, simple, whitespace, classic, and perfield.

Stopwords (optional)
: An array of stop words. Stop words are words that must not be indexed. If this array is specified, it overrides the default list of stop words. The default list of stop words depends on the analyzer.
Recall that for a partitioned database the _id field is formed from a partition key part and a document key part.

See an example of using HTTP to create, update, or delete multiple documents:

POST /$DATABASE/_bulk_docs HTTP/1.1
Content-Type: application/json

See an example of using the command line to create, update, or delete multiple documents:



* Curl
* Java
* Python
* Go
* Node



curl -H "Authorization: Bearer $API_BEARER_TOKEN" -X POST "$SERVICE_URL/events/_bulk_docs" -H "Content-Type: application/json" --data '{ "docs": [ { "_id": "0007241142412418284", "type": "event", "userid": "abc123", "eventType": "addedToBasket", "productId": "1000042", "date": "2019-01-28T10:44:22.000Z" }, { "_id": "0008001142412418285", "type": "event", "userid": "def456", "eventType": "addedToBasket", "productId": "1000043", "date": "2019-01-28T12:30:00.000Z" } ], "new_edits": true }'

import com.ibm.cloud.cloudant.v1.Cloudant;
import com.ibm.cloud.cloudant.v1.model.BulkDocs;
import com.ibm.cloud.cloudant.v1.model.Document;
import com.ibm.cloud.cloudant.v1.model.DocumentResult;
import com.ibm.cloud.cloudant.v1.model.PostBulkDocsOptions;

import java.util.Arrays;
import java.util.List;

Cloudant service = Cloudant.newInstance();

Document eventDoc1 = new Document();
After replicating database b to database a, database a also has five documents, one is crossed off.](https://cloud.ibm.com/docs-content/v1/content/522c6f62358b063f921283400a601c3f9bc66f08/Cloudant/images/replication_guide_6.svg)

Figure 5. Two-way replication





 Discussion about continuous replication 

So far, the discussion deals only with one-shot replication, which finishes when all of the source data is written to the target database. With continuous replication, data flows continuously. All subsequent changes to the source database are transmitted to the target database in real time.

Continuous replication is triggered by clicking the Make this replication continuous check box when you define a replication task in the IBM Cloudant Dashboard, or by setting the [continuous](https://cloud.ibm.com/apidocs/cloudantpostreplicate) flag in the IBM Cloudant API.

Two-way replication can be made continuous in one or both of the directions, by setting the continuous flag.

See the following example that uses HTTP to start a continuous replication:

POST /_replicator HTTP/1.1
Content-Type: application/json
Host: $SERVICE_URL
Authorization: ...

See the following example to start a continuous replication:



* Curl
* Java
* Node
* Python
* Go



curl -X POST -H "Content-type: application/json" "$SERVICE_URL/_replicator" -d '{ "_id": "repldoc-example",
"continuous": true,
"create_target": true,
"source": { "url": "'"$SOURCE_SERVICE_URL/source"'" },
"target": {
"auth": { "iam": { "api_key": "'"$API_KEY"'" } },
"url": "'"$TARGET_SERVICE_URL/target"'"
}
}'
GET/HEAD /$DATABASE/_design/$DOCUMENT_ID/_geo/$FURTHER_PATH_PARTS cloudantnosqldb.any-document.read 
 GET/POST /$DATABASE/_design/$DOCUMENT_ID/_search/$FURTHER_PATH_PARTS cloudantnosqldb.any-document.read 
 POST /$DATABASE/_design/$DOCUMENT_ID/_view/$VIEW/queries cloudantnosqldb.any-document.read 
 GET/POST /$DATABASE/_design/$DOCUMENT_ID/_view/$FURTHER_PATH_PARTS cloudantnosqldb.any-document.read 
 POST /$DATABASE/_explain/$FURTHER_PATH_PARTS cloudantnosqldb.any-document.read 
 POST /$DATABASE/_find/$FURTHER_PATH_PARTS cloudantnosqldb.any-document.read 
 GET /$DATABASE/_local/$DOCUMENT_ID cloudantnosqldb.any-document.read 
 POST /$DATABASE/_missing_revs cloudantnosqldb.any-document.read 
 POST /$DATABASE/_revs_diff cloudantnosqldb.any-document.read 





 Unavailable endpoints 

The following endpoints are not available to requests authorized with IAM:



* HTTP rewrite handlers - /db/_design/design-doc/_rewrite/path.



While design documents can contain rewrite handlers, users cannot call them.



* Update handlers - POST /db/_design/ddoc/_update/func.



While design documents can contain update functions, users cannot call them.









 Troubleshooting 

If you can't use IAM to authenticate when you make requests to your IBM Cloudant service instance, verify your account as shown in the next section.



 Ensure that your account is IAM enabled 

On the Overview portion of the IBM Cloudant Dashboard, "authentication method" is listed under deployment details. Your available authentication methods are listed there.
[IBM Cloud Databases for Redis](https://cloud.ibm.com/docs/databases-for-redis) IBM Cloud Databases for Redis is an open source, in-memory key value store designed for the modern application stack. With Databases for Redis, you can use counters, queues, lists, and HyperLogLogs to handle complex data issues simply. [Hyper Protect Crypto Services integration](https://cloud.ibm.com/docs/databases-for-redis?topic=databases-for-redis-hpcs) 
 [IBM Cloud Messages for RabbitMQ](https://cloud.ibm.com/docs/messages-for-rabbitmq) IBM Cloud Messages for RabbitMQ is an enterprise-ready and fully managed RabbitMQ service with native integration into the IBM Cloud. It supports multiple messaging protocols as a broker. [Hyper Protect Crypto Services integration](https://cloud.ibm.com/docs/messages-for-rabbitmq?topic=messages-for-rabbitmq-hpcs) 
 [IBM Db2 on Cloud](https://cloud.ibm.com/docs/Db2onCloud) IBM Db2 on Cloud is an SQL database that is provisioned for you in the cloud. You can use Db2 on Cloud just as you use any database software, but without the time and expense of hardware setup or software installation and maintenance. [Hyper Protect Crypto Services integration](https://cloud.ibm.com/docs/Db2onCloud?topic=Db2onCloud-hpcs) 
 [Hyper Protect DBaaS for PostgreSQL](https://cloud.ibm.com/docs/hyper-protect-dbaas-for-postgresql) - Deprecated Hyper Protect DBaaS for PostgreSQL offers fully managed and highly secure PostgreSQL database environments that have technology-enforced protection and high availability. [Hyper Protect Crypto Services integration](https://cloud.ibm.com/docs/hyper-protect-dbaas-for-postgresql?topic=hyper-protect-dbaas-for-postgresql-hpcs-byok)
The Dashboard Overview 

The Overview page shows you information about your IBM Cloud® Databases for EnterpriseDB deployment. The overview includes essential identifying information.



 Overview 



 Type 

The type of database that is offered by the service, and the database version that your service uses.





 ID 

The ID is a [CRN (Cloud Resource Name)](https://cloud.ibm.com/docs/account?topic=account-crn) which uniquely identifies the database deployment. The CRN is used to refer to the database in the API and can be used with the CLI.The Overview pane shows details of your service.





 Recent Tasks 

Every time that you make administrative changes to your service (such as scaling, or taking a manual backup), a task starts up. The Recent Tasks panel shows the task name and progress bar for any running tasks, and a list of the most recent completed tasks. Depending on how busy your deployment is, successful tasks can be shown for 24-48 hours. Unsuccessful tasks can show for 7-8 days. Tasks can also be retrieved from the [Cloud Databases API](https://cloud.ibm.com/apidocs/cloud-databases-apiget-currently-running-tasks-on-a-deployment) and [CLI plug-in](https://cloud.ibm.com/docs/databases-cli-plugin?topic=databases-cli-plugin-cdb-referencedeployment-tasks-list). A historical record of tasks from any time period is available through the [IBM Cloud® Activity Tracker integration](https://cloud.ibm.com/docs/databases-for-enterprisedb?topic=cloud-databases-activity-tracker-integration).





 Endpoints 

The Endpoints pane within the Overview pane contains connection strings for your deployment. Each tab contains connection information tailored to the type of connection or the protocol that uses it. Basic information includes things like hostname and port, as well as the TLS self-signed certificate, TLS/SSL parameters, and the default database of your deployment.
The UI - Dashboard Overview 

The Overview page shows you information about your IBM Cloud® Databases for MySQL deployment. The overview includes essential identifying information.



 Overview 



 Type 

The type of database that is offered by the service and the database version that your service uses.





 ID 

The ID is a [CRN (Cloud Resource Name)](https://cloud.ibm.com/docs/account?topic=account-crn) that uniquely identifies the database deployment. The CRN is used to refer to the database in the API and can be used with the CLI. The Overview pane shows details of your service.





 Recent Tasks 

Every time that you make administrative changes to your service (such as scaling or taking a manual backup), a task starts up. The Recent Tasks panel shows the task name and progress bar for any running tasks, as well as a list of the most recent completed tasks. Depending on how busy your deployment is, successful tasks can be shown for 24 - 48 hours. Unsuccessful tasks can show for 7 - 8 days. Tasks can also be retrieved from the [Cloud Databases API](https://cloud.ibm.com/apidocs/cloud-databases-apiget-currently-running-tasks-on-a-deployment) and [CLI plug-in](https://cloud.ibm.com/docs/databases-cli-plugin?topic=databases-cli-plugin-cdb-referencedeployment-tasks-list). A historical record of tasks from any time period is available through the [IBM Cloud® Activity Tracker integration](https://cloud.ibm.com/docs/databases-for-mysql?topic=cloud-databases-activity-tracker).





 Endpoints 

The Endpoints pane within the Overview pane contains connection strings for your deployment. Each tab contains connection information that is tailored to the type of connection or the protocol that uses it. Basic information includes things like hostname and port, as well as the TLS self-signed certificate, TLS/SSL parameters, and the default database of your deployment.
"Tombstone" documents 

Tombstone documents are small documents that are kept in place within a database when the original document is deleted. Their purpose is to allow the deletion to be replicated.

When the replication completes, the tombstones are no longer required. Automatic compaction helps ensure that only the minimal amount of data is kept and transferred during replication. Nevertheless, tombstone documents aren't automatically removed.

Over time, as documents are created and deleted, the number of tombstone documents increases. Each tombstone is small, but gradually they add to database disk space usage, and to the query time for the primary index. To reduce these effects, you might want to remove the tombstones.



 Simple removal of "tombstone" documents 

To remove tombstones manually, do the following steps:



1. Create a database to hold the required documents.



The new database is intended to hold all documents except the tombstone documents.



1. Set up a [filtered replication](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-advanced-replicationfiltered-replication-adv-repl) to replicate documents from the original database to the new database.



Configure the filter so that documents with the "_deleted" attribute aren't replicated.



1. When replication is complete, switch your application logic to use the new database.
2. Verify that your applications work correctly with the new database.



When you're satisfied that everything is working correctly, you might want to delete the old database.

In general, try to design and implement your applications to do the minimum necessary amount of deletion.

See an example filter to exclude deleted documents during a replication:

{
"_id": "_design/filters",
"filters": {
"deleted_filter": "function(doc, req) { return !doc._deleted; };"
}
}





 Advanced removal of tombstone documents
MySQL 8 GA 

As part of our commitment to offering a rich and mature cloud database portfolio, IBM Cloud Databases is releasing MySQL version 8.



  New IBM Cloud® Databases for MySQL deployments 

For new IBM Cloud Databases deployments, take a backup of your existing v5.7/8 database instances and import them into version 8. Use mysqldump or mydumper to import to IBM Cloud Databases. For more information, see [Migrating to Databases for MySQL](https://cloud.ibm.com/docs/databases-for-mysql?topic=databases-for-mysql-migrating).





  Existing IBM Cloud® Databases for MySQL v5.7 deployments 

At present, in-place upgrades from MySQL v5.7 are not supported. IBM Cloud Databases is creating additional documentation to provide necessary guidance on a migration path. Until this documentation is released, upgrade existing IBM Cloud® Databases for MySQL v5.7 deployments using the following steps:



1.  Create and export a v5.7x backup.
2.  Migrate and prepare your backup locally to be compatible with 8.0.
3.  Import into ICD MySQL 8.0.



As a best practice, retain your current v5.7 backups until the version 8 import is evaluated. Use mysqldump or mydumper to export your 5.7.x databases. For more information, see [Migrating to Databases for MySQL](https://cloud.ibm.com/docs/databases-for-mysql?topic=databases-for-mysql-migrating).

Once your upgrade is complete, changes cannot be reverted. The changes are incompatible, and you cannot use the data directory from MySQL 8.0 on MySQL 5.7. Retain your MySQL v5.7 backup, as it helps to restore it on a MySQL 5.7 instance should the changes need to be reversed.



  Upgrade Prerequisites 

Before attempting any upgrades, refer to MySQL's [Preparing Your Installation for Upgrade](https://dev.mysql.com/doc/refman/8.0/en/upgrade-prerequisites.html) to ensure upgrade readiness.
High-Availability 

IBM Cloud® Databases for MySQL is a managed cloud database service that is fully integrated into the IBM Cloud environment. The database, storage, and supporting infrastructure all run in IBM Cloud.

Databases for MySQL provides replication, failover, and high-availability features to protect your databases and data from infrastructure maintenance, upgrades, and failures. Deployments contain a cluster with three data members: a leader and two replicas. All members contain a copy of your data by using Orchestrator to handle failovers. If the leader becomes unreachable, the cluster initiates a failover and a replica is promoted to leader. The replica rejoins the cluster and your cluster continues to operate normally.

Databases for MySQL will, at times, do controlled switchovers under normal operation. These switchovers are no-data-loss events that result in resets of active connections. For a short period, reconnections can fail. At times, unplanned failovers might occur due to unforeseen events on the operating environment. These failovers can take a bit longer. In both cases, the potential exists for the downtime to be longer.



 Application-level High-Availability 

Applications that communicate over networks and cloud services are subject to transient connection failures. You want to design your applications to retry connections when errors are caused by a temporary loss in connectivity to your deployment or to IBM Cloud.

Your database might experience downtime for multiple reasons, including



* Network outages
* Storage and volume-related issues
* High CPU usage
* High disk I/O
* Connection overloads



Because Databases for MySQL is a managed service, regular updates and database maintenance occurs as part of normal operations. If both replicas are lost, writes to the leader hang, due to the semisynchronous replication process not having a follower. For more information, see MySQL's [Semisynchronous Replication](https://dev.mysql.com/doc/mysql-replication-excerpt/8.0/en/replication-semisync.html). This scenario occasionally causes short intervals where your database is unavailable.
The Dashboard Overview 

The Overview page shows you information about your IBM Cloud® Databases for PostgreSQL deployment. The overview includes essential identifying information.



 Overview 



 Type 

The type of database that is offered by the service, and the database version that your service uses.





 ID 

The ID is a [CRN (Cloud Resource Name)](https://cloud.ibm.com/docs/account?topic=account-crn) which uniquely identifies the database deployment. The CRN is used to refer to the database in the API and can be used with the CLI. The Overview pane shows details of your service.





 Recent Tasks 

Every time that you make administrative changes to your service (such as scaling, or taking a manual backup), a task starts up. The Recent Tasks panel shows the task name and progress bar for any running tasks, and a list of the most recent completed tasks. Depending on how busy your deployment is, successful tasks can be shown for 24 - 48 hours. Unsuccessful tasks can show for 7 - 8 days. Tasks can also be retrieved from the [Cloud Databases API](https://cloud.ibm.com/apidocs/cloud-databases-apiget-currently-running-tasks-on-a-deployment) and [CLI plug-in](https://cloud.ibm.com/docs/databases-cli-plugin?topic=databases-cli-plugin-cdb-referencedeployment-tasks-list). A historical record of tasks from any time period is available through the [IBM Cloud® Activity Tracker integration](https://cloud.ibm.com/docs/databases-for-postgresql?topic=cloud-databases-activity-tracker).





 Endpoints 

The Endpoints pane within the Overview pane contains connection strings for your deployment. Each tab contains connection information that is tailored to the type of connection or the protocol that uses it. Basic information includes things like hostname and port, as well as the TLS self-signed certificate, TLS/SSL parameters, and the default database of your deployment.
Getting Started 

This tutorial is a short introduction to using an IBM Cloud® Databases for MySQL deployment. To get started, create a connection to [MySQL Workbench](https://www.mysql.com/products/workbench/), an open source administration platform for MySQL that provides many tools for managing your data and databases. [Download and install](https://dev.mysql.com/downloads/workbench/) the version that is appropriate to your environment and then consult MySQL's [documentation](https://dev.mysql.com/doc/workbench/en/wb-mysql-connections.html) to connect and manage your Databases for MySQL deployment.



 IBM Cloud® Databases for MySQL 

IBM Cloud® Databases for MySQL is a serverless, cloud database service that is fully integrated into the IBM Cloud environment. Use Databases for MySQL as a cloud database system without purchasing and setting up your own hardware, installing your own database software, or managing the database yourself.

IBM Cloud® Databases for MySQL requires no software, infrastructure, network, or OS administration. IBM continuously provides fully automated and automatic updates to the service, such as security patches and minor version upgrades. A database instance is deployed by default as highly available across multiple data centers in an IBM Cloud Multi-Zone region with [semisynchronous replication](https://dev.mysql.com/doc/mysql-replication-excerpt/8.0/en/replication-semisync.html). Connect to a single database endpoint and IBM automatically manages the failover between Availability Zones. IBM Cloud® Databases for MySQL allows you to horizontally scale your MySQL instance with Read Replicas in region or cross-regionally. Databases for MySQL Read Replicas can be easily transformed into fully functioning Databases for MySQL instances, an especially useful feature for online cross-regional disaster recovery strategies.

Additionally, Databases for MySQL provides independent scaling of disk, RAM, and vCPU, as well as auto-scaling capabilities and hourly billing.
The Dashboard Overview 

The Overview page shows you information about your IBM Cloud® Databases for Elasticsearch deployment. The overview includes essential identifying information.



 Type 

The type of database that is offered by the service, and the database version that your service uses.





 ID 

The ID is a [CRN (Cloud Resource Name)](https://cloud.ibm.com/docs/account?topic=account-crn) which uniquely identifies the database deployment. The CRN is used to refer to the database in the API and can be used with the CLI. The Overview pane shows details of your service.





 Recent Tasks 

Every time that you make administrative changes to your service (such as scaling, or taking a manual backup), a task starts up. The Recent Tasks panel shows the task name and progress bar for any running tasks, and a list of the most recently completed tasks. Depending on how busy your deployment is, successful tasks can be shown for 24 - 48 hours. Unsuccessful tasks can show for 7 - 8 days. Tasks can also be retrieved from the [Cloud Databases API](https://cloud.ibm.com/apidocs/cloud-databases-apiget-currently-running-tasks-on-a-deployment) and [CLI plug-in](https://cloud.ibm.com/docs/databases-cli-plugin?topic=databases-cli-plugin-cdb-referencedeployment-tasks-list). A historical record of tasks from any time period is available through the [IBM Cloud® Activity Tracker integration](https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=databases-for-elasticsearch-activity-tracker).





 Endpoints 

The Endpoints pane within the Overview pane contains connection strings for your deployment. Each tab contains connection information that is tailored to the type of connection or the protocol that uses it. Basic information includes things like hostname and port, the TLS self-signed certificate, TLS/SSL parameters, and the default database of your deployment.