vania-janet/multiturn-rag-retrieval-data huggingface dataset:
If in doubt, use the [POST /{db}/_explain](https://cloud.ibm.com/apidocs/cloudant?code=javapostexplain) endpoint to check that an index is used, and the execution_stats: true parameter to measure the efficiency of each query.

For a type=json index to be used to support a query, it must match the fields that are used in the selector and sort parameters. Comparison operators might be used on the last element to perform range queries.

For more information, see [Explain plans](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-queryexplain-plans).





 Can I use IBM Cloudant Query with a Lucene index? 

Yes! IBM Cloudant Query supports two types of indexes:



* "type": "json" - a fixed-order index powered by IBM Cloudant's MapReduce API. Good for fixed, boilerplate queries that match every term in the index.
* "type": "text" - a Lucene index powered by IBM Cloudant's Search API. Good for general-purpose queries across one, some, or all indexed fields.



A type=text index is created with an index definition like this:

{
"index": {
"fields": [
{ "name": "firstname", "type": "string"},
{ "name": "surname", "type": "string"},
{ "name": "date", "type": "string"}
]
},
"ddoc": "textindexes",
"name": "byNameAndDate",
"type": "text"
}

Notice that the fields array requires each attribute to be named and typed (unlike type=json indexes).

The resultant index can be used by queries that contain one or more of the indexed fields:

{
"selector": {
"surname": "Dickens"
},
"sort": ["date"],
"limit": 10,
"github.com/IBM/cloudant-go-sdk/cloudantv1"
)

See the following example that uses HTTP to query a global index:

GET /$DATABASE/_design/$DDOC/_search/$INDEX_NAME?include_docs=true&query=":"&limit=1 HTTP/1.1
Content-Type: application/json
Host: $ACCOUNT.cloudant.com

See the following example that uses the command line to query a global index:



* Curl
* Java
* Node
* Python
* Go



curl "https://$ACCOUNT.cloudant.com/$DATABASE/_design/$DDOC/_search/$INDEX_NAME?include_docs=true&query=":"&limit=1"

import com.ibm.cloud.cloudant.v1.Cloudant;
import com.ibm.cloud.cloudant.v1.model.PostSearchOptions;
import com.ibm.cloud.cloudant.v1.model.SearchResult;

Cloudant service = Cloudant.newInstance();

PostSearchOptions searchOptions = new PostSearchOptions.Builder()
.db("<db-name>")
.ddoc("<ddoc>")
.index("<index-name>")
.query(":")
.includeDocs(true)
.limit(1)
.build();

SearchResult response =
service.postSearch(searchOptions).execute()
.getResult();

System.out.println(response);

import { CloudantV1 } from '@ibm-cloud/cloudant';

const service = CloudantV1.newInstance({});

service.postSearch({
db: '<db-name>',
ddoc: '<ddoc>',
index: '<index-name>',
query: ':',
includeDocs: true,
limit: 1
Use either the command-line backup tool, or the library with application code, to enable backup from IBM Cloudant databases as part of more complicated situations. A useful scenario is scheduling backups by using cron, and automatically uploading data to [Cloud Object Storage](https://www.ibm.com/cloud/object-storage/solutions?mhq=object%20storage%20public&mhsrc=ibmsearch_a) for long-term retention.





 Command line scripting examples 

You frequently need to meet the following two requirements:



* Saving disk space by ['zipping' the backup](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-ibm-cloudant-backup-and-recoverycompressing-a-backup-file) file as you create it.
* Creating a backup of a database automatically at [regular intervals](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-ibm-cloudant-backup-and-recoveryhourly-or-daily-backups-that-use-cron-).





 Compressing a backup file 

The couchbackup tool can write a backup file to disk directly, or stream the backup to stdout. Streaming to stdout enables data to be transformed before it is written to disk. This feature is used to compress data within the stream.

couchbackup --url "$SERVICE_URL" --db "animaldb" | gzip > backup.gz

In this example, the gzip tool accepts the backup data directly through its stdin, compresses the data, and emits it through stdout. The resulting compressed data stream is then redirected and written to a file called backup.gz.

If the database requires you to supply access credentials, use $SERVICE_URL with the form https://$USERNAME:$PASSWORD@$ACCOUNT, for example, https://myusername:mypassword@myhost.cloudant.com.

It's straightforward to extend the pipeline if you want to transform the data in other ways.
* Blog post on [replication topology](https://dx13.co.uk/articles/2017/11/07/cloudant-replication-topologies-and-failover/)
* IBM Cloudant [guide to replication](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-replication-guidereplication)







 Use the bulk API 

IBM Cloudant has nice API endpoints for bulk loading (and reading) many documents in a single request. Reading many documents in a single request can be much more efficient than reading and writing many documents one at a time. The write endpoint is shown in the following example:

${database}/_bulk_docs

Its main purpose is to be a central part in the replicator algorithm, but it’s available for your use, too, and it’s awesome.

With _bulk_docs, in addition to creating [PouchDB](https://pouchdb.com/), implement create, update, and delete even for single documents this way for fewer code paths.

The following example creates one new document, updates a second existing, and deletes a third document:

curl -XPOST 'https://ACCT.cloudant.com/DB/_bulk_docs' -H "Content-Type: application/json" -d '{"docs":[{"baz":"boo"}, {"_id":"463bd...","foo":"bar"}, {"_id":"ae52d...","_rev":"1-8147...","_deleted": true}]}'

You can also fetch many documents in a single request by issuing a POST to _all_docs (a relatively new endpoint that is called _bulk_get also exists, but this endpoint is probably not what you want. It’s there for a specific internal purpose).

To fetch a fixed set of docs by using _all_docs, POST with a keys body, run the following command:
postAllDocsOptions.SetStartKey("abc")
postAllDocsOptions.SetLimit(10)

allDocsResult, response, err := service.PostAllDocs(postAllDocsOptions)
if err != nil {
panic(err)
}

b, _ := json.MarshalIndent(allDocsResult, "", " ")
fmt.Println(string(b))

const { CloudantV1 } = require('@ibm-cloud/cloudant');

const service = CloudantV1.newInstance({});

service.postAllDocs({
db: 'orders',
includeDocs: true,
startKey: 'abc',
limit: 10
}).then(response => {
console.log(response.result);
});

The previous Go example requires the following import block:

import (
"encoding/json"
"fmt"
"github.com/IBM/cloudant-go-sdk/cloudantv1"
)

All Go examples require the service object to be initialized. For more information, see the API documentation's [Authentication section](https://cloud.ibm.com/apidocs/cloudant?code=goauthentication-with-external-configuration) for examples.

See the following example that uses HTTP to list all documents in a database that match at least one of the specified keys:

GET /_all_docs?keys=["somekey","someotherkey"] HTTP/1.1

See the following example to list all documents in a database that match at least one of the specified keys:



* Curl
* Java
* Python
* Go
* Node



curl -H "Authorization: Bearer $API_BEARER_TOKEN" -X POST "$SERVICE_URL/orders/_all_docs" -H "Content-Type: application/json" --data '{
"include_docs": true,
date: '2019-01-25T20:00:00.000Z'
}

const bulkDocs = { docs: [eventDoc1, eventDoc2] }

service.postBulkDocs({
db: 'events',
bulkDocs: bulkDocs
}).then(response => {
console.log(response.result);
});

The previous Go example requires the following import block:

import (
"encoding/json"
"fmt"
"github.com/IBM/cloudant-go-sdk/cloudantv1"
"github.com/IBM/go-sdk-core/v5/core"
)

All Go examples require the service object to be initialized. For more information, see the API documentation's [Authentication section](https://cloud.ibm.com/apidocs/cloudant?code=goauthentication-with-external-configuration) for examples.

See an example JSON describing the update, creation, and deletion of three documents in one bulk request:

[
{
"id": "0007241142412418284",
"ok": true,
"rev": "1-5005d65514fe9e90f8eccf174af5dd64"
},
{
"id": "0008001142412418285",
"ok": true,
"rev": "1-2d7810b054babeda4812b3924428d6d6"
}
]





 Bulk request responses 

The HTTP status code that is received in response indicates whether the request was fully or partially successful. In the response body itself, you get an array with detailed information for each document in the request.



Table 3. HTTP status codes

 Code Description 

 201 The request did succeed, but this success doesn't imply all documents were updated.
System.out.println(response);

import { CloudantV1 } from '@ibm-cloud/cloudant';

const service = CloudantV1.newInstance({});

let index: CloudantV1.IndexDefinition = {}

service.postIndex({
db: 'my-movies',
index: index,
type: 'text'
}).then(response => {
console.log(response.result);
});

from ibmcloudant.cloudant_v1 import CloudantV1, IndexDefinition

service = CloudantV1.new_instance()

index = IndexDefinition()

response = service.post_index(
db='my-movies',
index=index,
type='text'
).get_result()

print(response)

postIndexOptions := service.NewPostIndexOptions(
"my-movies",
&cloudantv1.IndexDefinition{},
)
postIndexOptions.SetType("text")

indexResult, _, err := service.PostIndex(postIndexOptions)
if err != nil {
panic(err)
}

b, _ := json.MarshalIndent(indexResult, "", " ")
fmt.Println(string(b))

The previous Go example requires the following import block:

import (
"encoding/json"
"fmt"
"github.com/IBM/cloudant-go-sdk/cloudantv1"
)

See the following example response after a text index is created successfully:

{
"id": "_design/bf936ad6eb87f03c2b42ae153377462844575e40",
"name": "bf936ad6eb87f03c2b42ae153377462844575e40",
IBM Cloudant Geospatial notice 

Support for the IBM® Cloudant® for IBM Cloud® Geospatial capability ends on 31 January 2023. In many cases, existing applications will fail if changes are not made to address the removal of this functionality before the end of support.



  What is IBM Cloudant Geospatial? 

Data is stored as GeoJSON in the IBM Cloudant database to describe point, line, polygon, multi-point, multi-line, and multi-polygon objects. Each object, as well as the geographic information, can have optional properties: Metadata about the object, which is returned in the search results.

Again, an index is defined as a JavaScript function, and then, queries can be used to ask questions of your collection of geographic features. For example, find me the nearest object to this point; find objects within this polygon; find objects along this path; or find objects that intersect with this object.

To summarize, IBM Cloudant Geo is something unique to the IBM Cloudant service and is used to perform advanced geospatial queries against your databases of GeoJSON objects. It cannot be combined with other index types. It is only of use to Geographic Information Systems or use-cases that have a purely geographic purpose.





  What is changing? 

As of 1 February 2023, the following conditions apply:



*  Users cannot query /$DATABASE/_design/$DDOCS/_geo endpoints. Requests to those endpoints return a 404 Not Found response.
*  Users can define indexes by using the st_indexes keyword in design documents, but those indexes are ignored by the service. This ensures that existing design documents can be updated, and replications that contain geospatial indexes do not fail. Existing Geo indexes will be deleted, and customers will no longer be billed for the space they consume.
*  IBM Cloud® support will no longer answer questions or assist with issues that are related to the Geospatial feature of the IBM Cloudant service.







  Alternatives to geospatial 

Many simple geospatial queries can be done without using the Geospatial capability that was removed from the IBM Cloudant service. These alternatives are described in this [IBM Cloudant blog post](https://blog.cloudant.com/2022/06/28/Simple-Geospatial-Queries.html).
"fmt"
"github.com/IBM/cloudant-go-sdk/cloudantv1"
"github.com/IBM/go-sdk-core/core"
)
All Go examples require the service object to be initialized. For more information, see the API documentation's Authentication section]] ! ! ! ! ! ! ! for examples.<-- </section "id="section-option-2-the-u0000-trick" "> --><-- </section "id="section-pagination-options" "> --><-- <section "id="section-pagination-of-views" "> --> Pagination of views MapReduce views, secondary indexes, can be queried in a similar way to the _all_docs endpoint, but with the GET or POST$SERVICE_URL/$DATABASE/_design/$DDOC/_view/$VIEW endpoint instead. MapReduce views are defined by key-value pairs that are produced from user-supplied JavaScript functions. You can define your query in the following ways:<-- <ul> --> * Spool all the data from a view with no parameters. * Include document bodies by supplying include_docs=true. * Choose the range of keys that are required by using startkey/endkey, but in this case, the data type of the keys might not be a string.<-- </ul> -->Another complication is that unlike the primary index, where every _id is unique, the secondary index might have entries with the same key. For example, lots of entries that include the key "herbivore". This situation makes pagination by using only startkey/endkey tricky, so you can use other parameters to help: startkey_docid/endkey_docid.See the following example of a first request:<-- <ul> --> * Curl * Java * Node * Python * Go<-- </ul> --> get first page of animals by diet
What is IBM Cloud Object Storage? 

IBM Cloud® Object Storage is a highly available, durable, and secure platform for storing unstructured data. Unstructured data (sometimes called binary or "blob" data) refers to data that is not highly structured in the manner of a database. Object storage is the most efficient way to store PDFs, media files, database backups, disk images, or even large structured datasets.

The files that are uploaded into IBM Cloud Object Storage are called objects. Objects can be anywhere from very small (a few bytes) [to very large] (up to 10TB). They are organized into buckets that serve as containers for objects, and which can be configured independently from one another in terms of locations, resiliency, billing rates, security, and object lifecycle. Objects themselves have their own metadata in the form of user-defined tags, legal holds, or archive status. Within a bucket, the hierarchy of objects is effectively "flat", although it is possible to add prefixes to object names to provide some organization and to provide flexibility in listing and other operations.

IBM Cloud Object Storage is strongly consistent for all data operations, and eventually consistent for bucket configuration operations. This means that when an object is uploaded, the server responds with a 200 OK after the object is successfully written, and the object is immediately available for listing and reading. All data stored in IBM Cloud Object Storage is encrypted, erasure-coded, and dispersed across three locations (with the distance between locations ranging from within a single data center, across a Multi-Zone Region or MZR, or even across multiple MZRs). This geographic range of dispersal contributes to a bucket's resiliency.

All requests and responses are made over HTTPS and all requests support the use of hash-based integrity checks using a Content-MD5 header. If the provided MD5 hash does not match the checksum computed by the storage service, the object is discarded and an error is returned. All GET and HEAD requests made to objects return an Etag value with the MD5 hash of the object to ensure integrity on the client side.

Developers use APIs to interact with their object storage.
A file contains input JSON data.-i, --instance: Instance name or ID. If not set, the context instance specified by ibmcloud cis instance-set INSTANCE is used.--output: Specify output format, only JSON is supported.<-- </section "id="section-create-glb-pool-options" "> --><-- <section "id="section-create-glb-pool-examples" "> --> Examples Create a GLB pool for instance cis-demo. ibmcloud cis glb-pool-create --json '{"description":"application server pool in US", "name":"us-pool", "enabled":true, "check_regions":"WNAM"], "minimum_origins":1,"monitor":"f1aba936b94213e5b8dca0c0dbf1f9cc", "origins":{"name":"us-app-dal01","address":"1.1.1.1","enabled":true,"header":{"host":"test.com"]}}, {"name":"us-app-dal02","address":"2.2.2.2","enabled":true,"header":{"host":"example.com"]}}] ! ! ! , "notification_email":"someone@example.com"}'-i "cis-demo"
<-- </section "id="section-create-glb-pool-examples" "> --><-- </section "id="section-create-glb-pool" "> --><-- <section "id="section-show-glb-pool" "> --> ibmcloud cis glb-pool Show the details of a given GLB pool. ibmcloud cis glb-pool GLB_POOL_ID -i, --instance INSTANCE] --output FORMAT] ! ! ! ! ! !
* --json: Format output in JSON.



--------------------







 ibmcloud pi instance-snapshot 





 Get the detail of a snapshot 

ibmcloud pi snapshot SNAPSHOT_ID



* SNAPSHOT_ID: The unique identifier of the snapshot.



Options



* --json: Format output in JSON.



--------------------





 ibmcloud pi instance-snapshot-create 



 Create a snapshot 

ibmcloud pi snapshot-create INSTANCE_ID [--volumes] [--name] [--description] [--json]



* INSTANCE_ID: The unique identifier or name of the instance.



Options



* --volumes: Space separated list of volumes to include in the PVM instance snapshot. This parameter is optional. If you do not specify this parameter or if the volumes list is empty, all the volumes that are attached to the PVM instance are included in the snapshot.
* --name: Name of the snapshot.
* --description: Snapshot description.
* --json: Format output in JSON.



Each snapshot that you create is monitored hourly and charged depending on the disk space that is requested for the snapshot. For more information on pricing for the snapshot, see [Metering of snapshot and pricing](https://cloud.ibm.com/docs/power-iaas?topic=power-iaas-volume-snapshot-clonemetering-snapshot)

--------------------







 ibmcloud pi instance-snapshot-delete 



 Delete a snapshot 

ibmcloud pi snapshot-delete SNAPSHOT_ID



* SNAPSHOT_ID: The unique identifier of the snapshot.



Options



* --json: Format output in JSON.



--------------------







 ibmcloud pi instance-snapshot-restore
The $NUMBER_OF_BLOCKS field must be an integer in the range of 1 to 100. Larger capacity sizes can be obtained by contacting IBM Cloudant support.

See the following example request to set the target capacity:



* Curl
* Java
* Node
* Python
* Go



curl -X PUT "$SERVICE_URL/_api/v2/user/capacity/throughput" -H "Content-Type: application/json" --data '{"blocks": 1}'

import com.ibm.cloud.cloudant.v1.Cloudant;
import com.ibm.cloud.cloudant.v1.model.CapacityThroughputInformation;
import com.ibm.cloud.cloudant.v1.model.PutCapacityThroughputConfigurationOptions;

Cloudant service = Cloudant.newInstance();

PutCapacityThroughputConfigurationOptions options =
new PutCapacityThroughputConfigurationOptions.Builder()
.blocks(1)
.build();

CapacityThroughputInformation response =
service.putCapacityThroughputConfiguration(options).execute().getResult();

System.out.println(response);

const { CloudantV1 } = require('@ibm-cloud/cloudant');

const service = CloudantV1.newInstance({});

service.putCapacityThroughputConfiguration({
blocks: 1,
}).then(response => {
console.log(response.result);
});

from ibmcloudant.cloudant_v1 import CloudantV1

service = CloudantV1.new_instance()

response = service.put_capacity_throughput_configuration(
blocks=1
).get_result()

print(response)
Ok response =
service.putSecurity(securityOptions).execute()
.getResult();

System.out.println(response);

from ibmcloudant.cloudant_v1 import CloudantV1, SecurityObject

service = CloudantV1.new_instance()

members = SecurityObject(
names=['user1', 'user2'],
roles=['developers']
)

response = service.put_security(
db='products',
members=members
).get_result()

print(response)

putSecurityOptions := service.NewPutSecurityOptions(
"products",
)
putSecurityOptions.SetMembers(&cloudantv1.SecurityObject{
Names: []string{"user1", "user2"},
Roles: []string{"developers"},
})

ok, response, err := service.PutSecurity(putSecurityOptions)
if err != nil {
panic(err)
}

b, _ := json.MarshalIndent(ok, "", " ")
fmt.Println(string(b))

const { CloudantV1 } = require('@ibm-cloud/cloudant');

const service = CloudantV1.newInstance({});

const members: CloudantV1.SecurityObject = {
names: ['user1', 'user2'],
roles: ['developers']
};

service.putSecurity({
db: 'products',
members: members
}).then(response => {
console.log(response.result);
});

The previous Go example requires the following import block:

import (
"encoding/json"
"fmt"
"github.com/IBM/cloudant-go-sdk/cloudantv1"
)
: IBM Cloud® Functions is used to create processing functions that ingest research data and store them as structured data documents in IBM Cloudant. IBM® Secure Gateway for IBM Cloud® provides an easy way for IBM Cloud® Functions to access on-premises data in a safe and secure manner.
: Web apps in the regional clusters are developed in nodeJS for manual data entry of results, schema definition, and research organizations affiliation. IBM Key Protect helps to secure access to IBM Cloudant data, and IBM Vulnerability Advisor scans app containers and images for security exploits.

ANONYMIZE
: Anytime new data document is stored in IBM Cloudant, an event is triggered, and a Cloud Function anonymizes the data and removes SPI from the data document. These anonymized data documents are stored separate from the "raw" data that is ingested and are the only documents that are shared across regions for analysis.

ANALYZE
: Machine learning frameworks are highly compute-intensive, and thus the nonprofit set up a global processing cluster of bare-metal worker nodes. Associated with this global processing cluster is an aggregated IBM Cloudant database of the anonymized data. A cron job periodically triggers a Cloud Function to push anonymized data documents from the regional centers to the global processing cluster's IBM Cloudant instance.
: The compute cluster runs the PyTorch ML framework, and machine learning apps are written in Python to analyze the aggregated data. In addition to ML apps, researchers in the collective group also develop their own apps that can be published and run on the global cluster.
: The nonprofit also provides apps that run on non-bare metal nodes of the global cluster. The apps view and extract the aggregated data and the ML app output. These apps are accessible by a public endpoint, which is secured by the API Gateway to the world. Then, researchers and data analysts from everywhere can download data sets and do their own analysis.

Developers started by deploying their research-sharing SaaS apps in containers with IBM Cloud Kubernetes Service. They created clusters for a Development environment that allow worldwide Developers to collaboratively deploy app improvements quickly.

Security first: The Development Exec chose bare metal to host the research clusters.