vania-janet/multiturn-rag-retrieval-data huggingface dataset:
_id or other metadata fields aren't automatically included.

See the following example of selective retrieval of fields from matching documents:

{
"selector": {
"Actor_name": "Robert De Niro"
},
"fields": [
"Actor_name",
"Movie_year",
"_id",
"_rev"
]
}





 Pagination 

IBM Cloudant Query supports pagination by the bookmark field. Every _find response contains a bookmark - a token that IBM Cloudant uses to determine where to resume from when later queries are made. To get the next set of query results, add the bookmark that was received in the previous response to your next request. Remember to keep the selector the same, otherwise you receive unexpected results. To paginate backwards, you can use a previous bookmark to return the previous set of results.

The presence of a bookmark doesn’t guarantee more results. You can test whether you are at the end of the result set by comparing the number of results that are returned with the page size requested. If the results returned are less than limit, no more results were returned in the result set.





 Explain plans 

IBM Cloudant Query chooses which index to use for responding to a query, unless you specify an index at query time.

When you specify an index to use, IBM Cloudant Query uses the following logic:



* The query planner looks at the selector section, and finds the index with the closest match to operators and fields that are used in the query. If two or more JSON type indexes match, the index with the smallest number of fields in the index is preferred. If two or more candidate indexes still exist, the index with the first alphabetical name is chosen.
* If a json type index and a text type index might both satisfy a selector, the json index is chosen by default.
* The text type index is chosen when the following conditions are met:



* A json type index and a text type index exist in the same field (for example fieldone).
* The selector can be satisfied only by using a text type index.
All Go examples require the service object to be initialized. For more information, see the API documentation's [Authentication section](https://cloud.ibm.com/apidocs/cloudant?code=goauthentication-with-external-configuration) for examples.

The response is a JSON object that contains all documents in the database that match the parameters. The following table describes the meaning of the individual fields:



Table 2. JSON object fields

 Field Description Type 

 offset Offset where the document list started. Numeric, Null (The type can be null when keys are specified.) 
 rows Array of document objects. Array 
 total_rows Number of documents in the database or view that match the parameters of the query. Numeric 
 pdate_seq Current update sequence for the database. String 



See the following example response after a request for all documents in a database:

{
"total_rows": 3,
"offset": 0,
"rows": [
{
"id": "5a049246-179f-42ad-87ac-8f080426c17c",
"key": "5a049246-179f-42ad-87ac-8f080426c17c",
"value": {
"rev": "2-9d5401898196997853b5ac4163857a29"
}
},
{
"id": "96f898f0-f6ff-4a9b-aac4-503992f31b01",
"key": "96f898f0-f6ff-4a9b-aac4-503992f31b01",
"value": {
"rev": "2-ff7b85665c4c297838963c80ecf481a3"
}
},
{
"github.com/IBM/cloudant-go-sdk/cloudantv1"
)

All Go examples require the service object to be initialized. For more information, see the API documentation's [Authentication section](https://cloud.ibm.com/apidocs/cloudant?code=goauthentication-with-external-configuration) for examples.





 Querying for recent results for a device 

To get the results for a device, you issue a partition query for the device within the bridge-9876 partition. The selector is only slightly more complicated, but still the same as an equivalent global query.



 Query for recent results assuming today is 13 December 2018 

The partition is embedded in the HTTP path when issuing the request to IBM Cloudant:



* Curl
* Java
* Node
* Python
* Go



curl -X POST "$SERVICE_URL/readings/_partition/bridge-9876/_find" -H 'Content-Type: application/json' --data '{
"selector": {
"deviceID": {
"$eq": "device-123456"
},
"ts": {
"$gte": "20181213"
}
}
}'

import com.ibm.cloud.cloudant.v1.Cloudant;
import com.ibm.cloud.cloudant.v1.model.FindResult;
import com.ibm.cloud.cloudant.v1.model.PostPartitionFindOptions;
import com.ibm.cloud.cloudant.v1.model.Selector;

import java.util.HashMap;
import java.util.Map;

Cloudant service = Cloudant.newInstance();

Map equalWithDeviceID = new HashMap<>();
equalWithDeviceID.put("$eq", "device-123456");

Map greaterThanOrEqualWithTs = new HashMap<>();
}

b, _ := json.MarshalIndent(searchResult, "", " ")
fmt.Println(string(b))

The previous Go example requires the following import block:

import (
"encoding/json"
"fmt"
"github.com/IBM/cloudant-go-sdk/cloudantv1"
)



 Query Parameters 

You must enable [faceting](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-cloudant-searchfaceting) before you can use the following parameters: counts and drilldown.



Table 3. Query parameters

 Argument Description Optional Type Supported Values Partition Query 

 bookmark A bookmark that was received from a previous search. This parameter enables paging through the results. If no results exist after the bookmark, you get a response with an empty rows array and the same bookmark, confirming the end of the result list. yes String Yes 
 counts This field defines an array of names of string fields, for which counts are requested. The response includes counts for each unique value of this field name among the documents that match the search query. [Faceting](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-cloudant-searchfaceting) must be enabled for this parameter to function. Yes JSON A JSON array of field names. No 
 drilldown This field can be used several times. Each use defines a pair of a field name and a value. The search matches only documents that include the value that was provided in the named field. It differs from using "fieldname:value" in the q parameter only in that the values aren't analyzed. [Faceting](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-cloudant-searchfaceting) must be enabled for this parameter to function. No JSON A JSON array that includes two elements: the field name and the value. Yes 
 group_field Field by which to group search matches. Yes String A string that includes the name of a string field.
IBM Cloudant JSON documents and requests have the following maximum size limits:



Table 4. Maximum size limits for JSON documents and requests

 Limit Maximum Size 

 Individual Document Size 1 MB 
 Single Attachment Size 10 MB 
 Request Body Size 11 MB 



If you exceed these limits, a [413 response](https://cloud.ibm.com/apidocs/cloudantlist-of-http-codes) alerts you.

The IBM Cloudant team recommends that you store binary attachments, or large JSON blobs, in object storage and save a link to the location in an IBM Cloudant JSON document.

When you replicate, documents or attachments that exceed these limits don't replicate to the target database. For more information about how to detect replication errors, see [Replication errors](https://cloud.ibm.com/apidocs/cloudantgetreplicationdocument).





 Locations and tenancy 

By default, all Lite and Standard plans are deployed on multi-tenant environments. As part of your plan selection, you can choose from the following IBM Cloud locations.



* Chennai (SZR)
* Dallas
* Frankfurt‡
* London
* Osaka
* Sydney
* Seoul (SZR)
* Tokyo
* Washington DC



Single-Zone Region (SZR) means that only one availability zone is available in that location. All other locations are Multi-Zone Regions (MZR) and leverage three separate availability zones for instances that are deployed in those locations. For more information, see the [High availability (HA), disaster recovery (DR), and backup](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-disaster-recovery-and-backup) documentation.

Dedicated Hardware plan instances can be deployed in most [IBM data center locations](https://www.ibm.com/cloud/data-centers/). See the drop-down menu in the IBM Cloud catalog for an up-to-date list of available locations.

‡All IBM Cloudant instances that are deployed from the IBM Cloud Frankfurt region are deployed into EU-managed environments.
"type": "json"
}





 Can I sort in reverse order? 

Yes! IBM Cloudant Query supports sorting the result set in ascending or descending order, but not a combination of the two. For example, a query that sorts some fields in ascending order, and a query where descending is not allowed.

This query returns documents that match firstname and surname and sorts by surname/firstname/date descending:

{
"selector": {
"firstname": "Charles",
"surname": "Dickens"
},
"sort": [
{ "firstname": "desc" },
{ "surname": "desc" },
{ "date": "desc" }
],
"limit": 10
}

A suitable index must be present that contains the selector fields and the sort fields. Otherwise, IBM Cloudant refuses to execute the query. A suitable index definition for the previous query is shown next:

{
"index": {
"fields": [
"firstname",
"surname",
"date"
]
},
"ddoc": "jsonindexes",
"name": "byNameAndDate",
"type": "json"
}

The previous index is suitable for both ascending and descending sort order.





 How can I tell if an index is backing a query? 

The [POST /{db}/_explain](https://cloud.ibm.com/apidocs/cloudant?code=javapostexplain) API endpoint when passed a JSON object that is usually sent to the [POST /{db}/_find](https://cloud.ibm.com/apidocs/cloudantpostfind) endpoint, explains how such a query is handled and which indexes, if any, might be used.

If the index object in the response indicates that "all_docs" is being used, a full database scan is required to service the query.
It differs from using "fieldname:value" in the q parameter only in that the values aren't analyzed. [Faceting](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-cloudant-searchfaceting) must be enabled for this parameter to function. No JSON A JSON array that includes two elements: the field name and the value. Yes 
 group_field Field by which to group search matches. Yes String A string that includes the name of a string field. Fields that include other data such as numbers, objects, or arrays can't be used. No 
 group_limit Maximum group count. This field can be used only if group_field is specified. Yes Numeric No 
 group_sort This field defines the order of the groups in a search that uses group_field. The default sort order is relevance. Yes JSON This field can have the same values as the sort field, so single fields and arrays of fields are supported. No 
 highlight_fields Specifies which fields to highlight. If specified, the result object includes a highlights field with an entry for each specified field. Yes Array of strings Yes 
 highlight_pre_tag A string that is inserted before the highlighted word in the highlights output. Yes, defaults to <em> String Yes 
 highlight_post_tag A string that is inserted after the highlighted word in the highlights output. Yes, defaults to </em> String Yes 
 highlight_number Number of fragments that are returned in highlights. If the search term exceeds the fragment size, then the entire search term is returned. Yes, defaults to 1 Numeric Yes 
 highlight_size Slice up field content into number of characters, so-called fragments, and highlights matches only inside the specified fragments. Yes, defaults to 100 characters Numeric Yes 
 include_docs Include the full content of the documents in the response. Yes Boolean Yes 
 include_fields A JSON array of field names to include in search results. Any fields that are included must be indexed with the store:true option. Yes, the default is all fields. Array of strings Yes
This data is used in other tutorials, like [Using IBM Cloudant Query to find data](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-creating-an-ibm-cloudant-query).



1. Create sample data.

sampleData = [
1, "one", "boiling", 100],
2, "two", "hot", 40],
3, "three", "hot", 75],
4, "four", "hot", 97],
5, "five", "warm", 20],
6, "six", "cold", 10],
7, "seven", "freezing", 0],
8, "eight", "freezing", -5]
]
2. Use a for statement to retrieve the fields in each row by going through each row in the array.

for document in sampleData:
 Retrieve the fields in each row.
number = document[0]
name = document[1]
description = document[2]
temperature = document[3]
3. Create a JSON document that represents all the data in the row.

jsonDocument = {
"numberField": number,
"nameField": name,
"descriptionField": description,
"temperatureField": temperature
}
4. Create a document by using the Database API.

newDocument = myDatabaseDemo.create_document(jsonDocument)
5. Check that the document exists in the database.

if newDocument.exists():
print("Document '{0}' successfully created.".format(number))







 Step 4: Retrieving data 

To perform a minimal retrieval, you first request a list of all documents within the database. This list is returned as an array. You can then show the content of an element in the array.



1. Retrieve a minimal amount of data.
Export them as environment variables, for example,export CLOUDANT_URL=....
2. Store them in a [credentials file](https://cloud.ibm.com/apidocs/cloudant?code=goauthentication-with-external-configuration).



As an alternative to ibmcloud login, you can set the environment variable CLOUDANT_APIKEY to an IAM API key.







 Other information 

The classic and the c subcommands are deprecated since version 0.0.5 and will be unsupported from 31 July 2023. The operations from the classic command are now available on the cloudant command. Track the [changes log](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-cli-change-log) for changes in the interface.





 Server 

Commands for Server resource.



 ibmcloud cloudant capacity 

View the amount of provisioned throughput capacity that is allocated to an IBM Cloudant instance and what is the target provisioned throughput capacity.

ibmcloud cloudant capacity



 Example 

Example request

ibmcloud cloudant capacity





 Example default output 

Example CapacityThroughputInformation response.

blocks	5
query	25
read	500
write	250





 Example full output 

Example CapacityThroughputInformation response.

{
"current" : {
"throughput" : {
"blocks" : 5,
"query" : 25,
"read" : 500,
"write" : 250
}
},
"target" : {
"throughput" : {
"blocks" : 10,
"query" : 50,
"read" : 1000,
"write" : 500
}
}
}





 Default JMESPath 

A JMESPath query is applied to the output of this command by default. The default JMESPath is:

current.throughput

If a custom JMESPath query is provided, it will replace the default JMESPath.







 ibmcloud cloudant capacity-update
Welcome to the Introduction to IBM Cloudant course, an 17-part video series that gives you an overview of the IBM Cloudant database-as-a-service.

This video is part 13 - Dates.

We saw earlier in this course that JSON only natively models strings, numbers, booleans, objects, and arrays. A common use-case is to store a date or date and time value in a database. See some ideas on how that can be achieved with IBM Cloudant.

The ISO-8601 string format for representing a time consists of a y-m-dTh:m:s.msTIMEZONE year, month, day, a 'T' character, hour, minute, second, millisecond, and time zone.

I always recommend storing dates in the UTC time zone even if you collect data from different geographies. Date that is stored in this form can easily be transformed into the local time zone at the front end. It's usually important to store each user's data in the "same units".

This string format sorts into date and time order (because the most significant date units are at the front of the string) and can be easily parsed in MapReduce functions.

Another option is to store the number of milliseconds since 01-January-1970. This option is also a standard, machine-readable way of representing a date and time.

It too can be parsed in MapReduce functions and is handy for comparing two dates: simply take one timestamp from another.

To summarize, no native date format in JSON exists, so you can store dates and times how you like. ISO-8601 is compact, readable, and sorts nicely, as do a timestamp (milliseconds since 1970).

If you need to use IBM Cloudant Query on one of the component parts, then that would need to be broken out explicitly in the document.

That's the end of this part. The next part is called Replication.







 Replication video 

Learn what replication means in IBM Cloudant, as well as different types of replication and how they work.



* Replication video script
(Optional) [Create an acurl alias](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-working-with-curlencode-user-name-and-password).



If you decide not to set up acurl, use the following URL with curl instead of the one provided in the exercises, curl "https://$USERNAME:$PASSWORD@$ACCOUNT.cloudant.com/databasedemo".

The acurl alias is more secure. It prevents someone from reading your password over your shoulder as you type. It also makes sure that your password isn’t sent in plain text over the network by enforcing HTTPS.

Now, we're ready to learn how to run queries against the database you created in step two of [Before you begin](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-creating-an-ibm-cloudant-querybefore-you-begin-qt).





 Step 1: Creating an index 

IBM Cloudant Query uses Mongo-style query syntax to search for documents by using logical operators. IBM Cloudant Query is a combination of a view and a search index.

When you use IBM Cloudant Query, the query planner looks at the selector (your query) to determine the correct index to choose from. In memory, you filter out the documents by the selector, which is why, even without an index, you can still query with various fields.

If no available defined index matches the specified query, then IBM Cloudant uses the _all_docs index, which looks up documents by ID. In the worst case scenario, it returns all the documents by ID (full table scan). Full table scans are expensive to process. It is recommended that you create an index.

To create an index, follow these steps:



1. Copy the following sample JSON data into a file named query-demo-index.json:

{
"index": {
"fields": [
"descriptionField",
"temperatureField"
],
"partial_filter_selector": {
"descriptionField": {
For more information, see the API documentation's [Authentication section](https://cloud.ibm.com/apidocs/cloudant?code=goauthentication-with-external-configuration) for examples.





 Example - Creating a nonpartitioned database 

To create a nonpartitioned database, see the following example:



* Curl
* Java
* Python
* Go
* Node



curl -H "Authorization: Bearer $API_BEARER_TOKEN" -X PUT "$SERVICE_URL/products"

import com.ibm.cloud.cloudant.v1.Cloudant;
import com.ibm.cloud.cloudant.v1.model.Ok;
import com.ibm.cloud.cloudant.v1.model.PutDatabaseOptions;

Cloudant service = Cloudant.newInstance();

PutDatabaseOptions databaseOptions = new PutDatabaseOptions.Builder()
.db("products")
.build();

Ok response =
service.putDatabase(databaseOptions).execute()
.getResult();

System.out.println(response);

from ibmcloudant.cloudant_v1 import CloudantV1

service = CloudantV1.new_instance()

response = service.put_database(db='products').get_result()

print(response)

putDatabaseOptions := service.NewPutDatabaseOptions(
"products",
)

ok, response, err := service.PutDatabase(putDatabaseOptions)
if err != nil {
panic(err)
}

b, _ := json.MarshalIndent(ok, "", " ")
fmt.Println(string(b))

const { CloudantV1 } = require('@ibm-cloud/cloudant');

const service = CloudantV1.newInstance({});

service.putDatabase({
b, _ := json.MarshalIndent(viewResult, "", " ")
fmt.Println(string(b))

The previous Go example requires the following import block:

import (
"encoding/json"
"fmt"
"github.com/IBM/cloudant-go-sdk/cloudantv1"
)

All Go examples require the service object to be initialized. For more information, see the API documentation's [Authentication section](https://cloud.ibm.com/apidocs/cloudant?code=goauthentication-with-external-configuration) for examples.

The response contains the standard view information, but only documents where the keys match.

See the example response after you run a query by using a list of keys:

{
"total_rows": 2,
"offset": 0,
"rows": [
{
"id": "abc125",
"key": "amelie.smith@aol.com",
"value":
"Amelie Smith",
true,
"2020-04-24T10:42:59.000Z"
]
},
{
"id": "abc123",
"key": "bob.smith@aol.com",
"value":
"Bob Smith",
true,
"2019-01-24T10:42:59.000Z"
]
}
]
}
Show more





 Multi-document fetching 

The following section covers a POST request to many documents from a database.

For a client application, this technique is more efficient than using multiple [GET](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-using-viewsquerying-a-view) API requests.

However, include_docs=true might require extra processing time when compared to accessing the view on its own.
Most programmatic applications of Object Storage use an SDK (such as [Java](https://cloud.ibm.com/docs/services/cloud-object-storage/libraries?topic=cloud-object-storage-java), [node.js](https://cloud.ibm.com/docs/services/cloud-object-storage/libraries?topic=cloud-object-storage-node), or [Python](https://cloud.ibm.com/docs/services/cloud-object-storage/libraries?topic=cloud-object-storage-python)) or the [Cloud Object Storage API](https://cloud.ibm.com/docs/services/cloud-object-storage/api-reference?topic=cloud-object-storage-compatibility-api). Typically objects are uploaded in [multiple parts](https://cloud.ibm.com/docs/services/cloud-object-storage/basics?topic=cloud-object-storage-large-objects), with part size and number of parts configured by a Transfer Manager class.





 Conditional requests 

When making a request to read or write data, it is possible to set conditions on that request to avoid unnecessary operations. This is accomplished using the following pre-conditional HTTP headers: If-Match, If-None-Match, If-Modified-Since, and If-Unmodified-Since.

It is generally preferable to use If-Match because the granularity of the Last-Modified value is only in seconds, and may not be sufficient to avoid race conditions in some applications.



 Using If-Match 

On an object PUT, HEAD, or GET request, [the If-Match header](https://tools.ietf.org/html/rfc7232section-3.1) will check to see if a provided Etag (MD5 hash of the object content) matches the provided Etag value. If this value matches, the operation will proceed. If the match fails, the system will return a 412 Precondition Failed error.
service.putSecurity({
db: 'products',
members: members
}).then(response => {
console.log(response.result);
});

The previous Go example requires the following import block:

import (
"encoding/json"
"fmt"
"github.com/IBM/cloudant-go-sdk/cloudantv1"
)

All Go examples require the service object to be initialized. For more information, see the API documentation's [Authentication section](https://cloud.ibm.com/apidocs/cloudant?code=goauthentication-with-external-configuration) for examples.

See the following example modification request in JSON format:

{
"couchdb_auth_only": true,
"members": {
"names": ["member"],"roles":[]
},
"admins": {
"names": ["admin"],"roles":[]
}
}

See the following example response from a modification request:

{
"ok" : true
}