vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Using the IBM Cloudant changes feed FAQ 

An IBM Cloudant database's changes feed's primary use-case is to power the replication of data from a source to a target database. The IBM Cloudant replicator is built to handle the changes feed and runs the necessary checks to ensure data is copied accurately to its destination.

IBM Cloudant has a raw [changes feed API](https://cloud.ibm.com/apidocs/cloudantgetchanges-changes) that can be used to consume a single database's changes but it must be used with care.

The _changes API endpoint can be used in several ways and can output data in various formats. But here we focus on best practice and how to avoid some pitfalls when you develop against the _changes API.



 How do I consume the changes feed? 

Given a single database orders, I can ask the database for a list of changes, in this case, limiting the result set to five changes with ?limit=5:

GET /orders/_changes?limit=5
{
"results": [
{
"seq": "1-g1AAAAB5eJzLYWBg",
"id": "00002Sc12XI8HD0YIBJ92n9ozC0Z7TaO",
"changes":
{
"rev": "1-3ef45fdbb0a5245634dc31be69db35f7"
}
]
},
....
],
"last_seq": "5-g1AAAAB5eJzLYWBg"
}
Show more

The API call returns the following changes:

results
: An array of changes.

last_seq
: A token that can be supplied to the changes endpoint in a subsequent API call to get the next batch of changes.

See how to fetch the next batch of changes in the following example:

GET /orders/_changes?limit=5&since=5-g1AAAAB5eJzLYWBg
{
"results": [ ...],
For an example that uses IBM Cloudant SDK for Go, see the [API and SDK documentation](https://cloud.ibm.com/apidocs/cloudant?code=goauthentication).







 Access by using HTTP client 

IBM Cloud IAM requires that an IAM API key is exchanged for a time-limited access token before you make a request to a resource or service. The access token is then included in the Authorization HTTP header to the service. When the access token expires, the client must handle getting a new one from the IAM token service.

As stated previously, if you use IBM Cloud IAM, it requires that you first exchange an IBM API key for a time-limited access token. After that, you use the token to authenticate against the IBM Cloudant API.

In Python, an example might look like:

import time

import requests

API_KEY = "MxVp86XHkU82Wc97tdvDF8qM8B0Xdit2RqR1mGfVXPWz"
ACCOUNT = "76838001-b883-444d-90d0-46f89e942a15-bluemix"

def get_access_token(api_key):
"""Retrieve an access token from the IAM token service."""
token_response = requests.post(
"https://iam.cloud.ibm.com/identity/token",
data={
"grant_type": "urn:ibm:params:oauth:grant-type:apikey",
"response_type": "cloud_iam",
"apikey": api_key
},
headers={
"Accept": "application/json"
}
)
if token_response.status_code == 200:
print "Got access token from IAM"
return token_response.json()['access_token']
else:
print token_response.status_code, token_response.json()
return None
* [Securing your data in IBM Cloudant](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-securing-your-data-in-cloudant)
* [Auditing events](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-at_events)





* Reference



* [Understanding your responsibilities when you use IBM Cloudant](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-cloudant-responsibilities)
* Observability



* [IBM Cloud Log Analysis integration](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-log-analysis-integration)
* [IBM Cloud Activity Tracker integration](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-auditing_events-for-cloudant)
* [IBM Cloud Monitoring integration](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-monitor-ibm-cloud-pm)



* High availability and disaster recovery



* [Understanding business continuity and disaster recovery for IBM Cloudant](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-bc-dr)
* [Understanding high availability for IBM Cloudant](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-ha)



* Best practices for IBM Cloudant



* [Data modeling](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-data-modeling)
* [Indexing and querying](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-indexing-and-querying)
* [IBM Cloudant in practice](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-cloudant-in-practice)



* Sample apps



* [Offline First shopping list](https://developer.ibm.com/patterns/create-an-offline-first-shopping-list-progressive-web-app/)
<br><br>Accessing IBM Cloudant Programmatically](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-learning-centeraccessing-cloudant-programmatically-video)[![carousel thumbnail](https://cloud.ibm.com/docs-content/v1/content/Cloudant//images/querying.png)<br><br>Querying](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-learning-centerquerying-video)[![carousel thumbnail](https://cloud.ibm.com/docs-content/v1/content/Cloudant//images/indexing.png)<br><br>Indexing](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-learning-centerindexing-video)[![carousel thumbnail](https://cloud.ibm.com/docs-content/v1/content/Cloudant//images/mapreduce.png)<br><br>MapReduce](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-learning-centermapreduce-video)[![carousel thumbnail](https://cloud.ibm.com/docs-content/v1/content/Cloudant//images/dates.png)<br><br>
getActiveTasksOptions := service.NewGetActiveTasksOptions()

activeTask, response, err := service.GetActiveTasks(getActiveTasksOptions)
if err != nil {
panic(err)
}

b, _ := json.MarshalIndent(activeTask, "", " ")
fmt.Println(string(b))

The previous Go example requires the following import block:

import (
"encoding/json"
"fmt"
"github.com/IBM/cloudant-go-sdk/cloudantv1"
)

All Go examples require the service object to be initialized. For more information, see the API documentation's [Authentication section](https://cloud.ibm.com/apidocs/cloudant?code=goauthentication-with-external-configuration) for examples.

Now, you learn how to use the _active_tasks endpoint to monitor long-running tasks. The curl command is used to access the endpoint. The jq command line JSON processor is used to process the JSON response.

This task-focused tutorial covers only what is essential to accomplish this task. For more information, see the [Using IBM® Cloudant® for IBM Cloud®](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-ibm-cloudant-basics) for a complete guide to the available options.



 curl and jq basics 

To get all active tasks and format the output nicely, call your account by using curl, and pipe the output to jq.

With jq, you filter a list of documents by their field values. This filter makes it easier to get all replication documents, or the details of just one particular view indexing task. The API reference has more information about the options.

See an example of obtaining and formatting a list of active tasks:

curl "$SERVICE_URL/_active_tasks" | jq





 Monitoring view builds and search indexes 

View indexes are rebuilt when a design document is updated.
* Database - The name of your IBM Cloudant database: cats.
* iamApiKey- Your IBM Cloudant iam_apikey_name.
* iamURL - Your IBM Cloudant url.



6. Click Deploy.



After the template deploys, you can make further edits to the code to customize it as needed, or go back and check out the catalog of available templates.





 Deploying the IBM Cloudant Events template from the CLI 

Use the CLI to deploy the IBM Cloudant Events template.

Before you begin



* Install the [Cloud Functions](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-cli_install) CLI.
* Specify a namespace to contain your Cloud Functions entities. For more information about namespaces, see [Managing namespaces](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-namespaces).





1. Clone the template repo.

git clone https://github.com/ibm-functions/template-cloudant-trigger.git
2. Navigate to the directory for the action runtime that you want to use. For example, nodejs.

cd template-cloudant-trigger/runtimes/nodejs
3. Deploy the template with the following environment variables.

CLOUDANT_HOSTNAME=<host> CLOUDANT_USERNAME=<username> CLOUDANT_PASSWORD=<password> CLOUDANT_DATABASE=<database> PACKAGE_NAME=<name> RULE_NAME=<name> TRIGGER_NAME=<name> ibmcloud fn deploy -m manifest.yaml



Table 1. Understanding the environment variables

 Variable Description 

 CLOUDANT_HOSTNAME <username>.cloudantnosqldb.appdomain.cloud 
 CLOUDANT_USERNAME Your IBM Cloudant username. 
 CLOUDANT_PASSWORD Your IBM Cloudant password. 
 CLOUDANT_DATABAS The name of your IBM Cloudant database: cats. 
 PACKAGE_NAME A custom name for the package 
 RULE_NAME A custom name for the rule.
r Specify the [read quorum](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-documentsquorum-writing-and-reading-data) value. Yes Numeric 2 
 revs_info Includes detailed information for all known document revisions. Yes Boolean False 
 skip Skip this number of records before returning the results. Yes Numeric 0 
 startkey Return records, starting with the specified key. Yes String 
 startkey_docid Return records, starting with the specified document ID. If startkey isn't set, this argument is ignored. Yes String 





 Notes 



1. Using include_docs=true might have [performance implications](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-using-viewsmulti-document-fetching).
2. When you use the keys argument, it might be easier to send a POST request rather than a GET request if you require multiple strings to list the keys you want.
3. When you use the keys argument and the revision is deleted, the value attribute that is returned is a JSON object with the current _rev of the document and a _deleted attribute. The doc attribute is only populated if you specified include_docs=true in the request and is null if the document is deleted.



See the following example that uses HTTP to list all documents in a database:

GET /_all_docs HTTP/1.1

See the following example to list all documents in a database:



* Curl
* Java
* Python
* Go
* Node



curl -H "Authorization: Bearer $API_BEARER_TOKEN" -X POST "$SERVICE_URL/orders/_all_docs" -H "Content-Type: application/json" --data '{ "include_docs": true, "startkey": "abc", "limit": 10}'

import com.ibm.cloud.cloudant.v1.Cloudant;
import com.ibm.cloud.cloudant.v1.model.AllDocsResult;
See the following example JSON document that lists document IDs to match during filtering:

{
"doc_ids": [
"ExampleID"
]
}

See the following example response (abbreviated) after you filter by _docs_ids:

{
"last_seq":"5-g1A...o5i",
"pending":0,
"results":[
{
"changes":
{
"rev":"13-bcb...29e"
}
],
"id":"ExampleID",
"seq":"5-g1A...HaA"
}
]
}





 The _selector filter 

The _selector filter accepts only changes for documents that match a specified selector, which is defined by using the same [selector syntax](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-queryselector-syntax) that is used for [_find](https://cloud.ibm.com/apidocs/cloudantpostfind).

For more examples that show use of this filter, see the information on [selector syntax](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-queryselector-syntax).

See the following example application of the _selector filter by using HTTP:

POST $SERVICE_URL/$DATABASE/_changes?filter=_selector HTTP/1.1

See the following example applications of the _selector filter:



* Curl
* Java
* Node
* Python
* Go



curl -X POST "$SERVICE_URL/orders/_changes?filter=_selector" -H "Content-Type: application/json" -d '{"selector": {"_id": { "$regex": "^_design/"}}}'

import java.util.HashMap;
import java.util.Map;

import com.ibm.cloud.cloudant.v1.Cloudant;
curl -u $ADMIN_USER "https://$ADMIN_USER.cloudant.com/_api/v2/monitoring/$END_POINT?cluster=$CLUSTER[&format=(json|raw)]"

The fields are described in the following table:



Table 1. Monitoring API request fields

 Field Meaning 

 ADMIN_USER The account name. The account must have administrative privileges. 
 CLUSTER The cluster that you are interested in. 
 DURATION Specifies the duration of the preferred time series query. Select from one of the following time intervals: ["5min", "30min", "1h", "12h", "24h", "1d", "3d", "7d", "1w", "1m", "3m", "6m", "12m", "1y"]. DURATION must be paired with either the START or END request. 
 END UTC timestamp in ISO-8601 or UTC epoch second, which specifies the end of a time series query. The timestamp can't have a query where START and END are the same, or where END is before START, or where START is after END. 
 END_POINT The [aspect](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-monitoring-an-ibm-cloudant-clustermonitoring-endpoints) of the cluster you want to monitor. 
 START UTC timestamp in ISO-8601 or integer seconds where epoch format specifies the starting point of a time series query that is mutually exclusive with END. 



Several of the fields have default values:



Table 2. Default values for monitoring API request fields

 Field Default value 

 DURATION 5 minutes 
 END No default value 
 START The current time 





 Results format 

By default, the monitoring results are returned in [JSON format](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-monitoring-an-ibm-cloudant-clusterwith-format=json-default).
You can also search across all your resources by using Lucene query syntax, with a single command by using the IBM Cloud CLI, starting with version 0.6.7.

You can search for the following attributes:

name
: The user-defined name of the resource.

region
: The geographical location where the resource is created. The allowed values are, for example, us-south, us-east, au-syd, eu-gb, eu-de, and jp-tok.

service_name
: The name of the service as it appears in the Name column of the output of ibmcloud catalog service-marketplace.

family
: The cloud component to which your resource belongs. The allowed values are: cloud_foundry, containers, container-registry, vmware, resource_controller, is, atracker, ims, iam.

organization_id
: The Cloud Foundry organization GUID.

doc.space_id
: The Cloud Foundry space GUID.

doc.resource_group_id
: The ID of the resource group.

type
: The resource type.
When the front end submits a team selection to the team route (see [index.html](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-combine-ibm-servicesindex-html-redis)), the app.route function first checks the cache to see whether it has the data already. If it does, then it returns that. Otherwise, it makes a query to IBM Cloudant to retrieve team data, store it in the cache, and return it to the front end.

The read operation uses an IBM Cloudant design document and a MapReduce view to select documents. This selection is beyond the scope of this tutorial, but you can read more about [views and design documents](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-creating-views-mapreduce) in the documentation.

This script also contains some code that uploads test data (contained in the directorydata.json file) to the database the first time it runs.





 index.html 

The index.html page is the only page of the application that is using the [Vue.js framework](https://vuejs.org/). When it loads, it shows you the available teams.

When you select a team, it makes an HTTP POST request with your choice to the /team route of the application (see [redis.js](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-combine-ibm-servicesserver-js-redis)). A successful return from the application contains all the data for the team members. We are displaying only their names and town for simplicity.





 Summary 

Today, we combined two IBM Cloud services to optimize cost and user experience: IBM Cloudant as a document store and query engine and Databases for Redis as a content cache. Cached documents can be retrieved more quickly and more cheaply, but the tradeoff is that your application might be showing old data to your users for a time.

If you followed this tutorial, you must deprovision your resources to stop incurring charges.
IBM Cloudant in practice 

The IBM Cloudant in practice document is the third best practice document in the series. It shows you the following best practices:



* How to avoid conflicts.
* How deleting documents works.
* What to watch out for with updates.
* How to work in an eventually consistent environment.
* How to set up replication.
* How to use the bulk API.
* Why you must not change Q, R, and N.
* How rate limits work.
* What logging tracks.
* How to compress your HTTP traffic.



For more information, see [Data modeling](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-data-modeling) or [Indexing and querying](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-indexing-and-querying).

The content in this document was originally written by Stefan Kruger as a [Best and worst practice](https://blog.cloudant.com/2019/11/21/Best-and-Worst-Practices.html) blog post on 21 November 2019.



 Avoid conflicts 

IBM Cloudant is designed to treat conflicts as a natural state of data in a distributed system. This feature is a powerful feature that helps an IBM Cloudant cluster always maintain high availability. However, the assumption is that conflicts are still reasonably rare. Tracking conflicts in IBM Cloudant’s core has significant cost that is associated with it.

It is perfectly possible (but a bad idea!) to ignore conflicts. The database merrily carries on operating by choosing a random, but deterministic revision of conflicted documents. However, as the number of unresolved conflicts grows, the performance of the database goes down a black hole, especially when you replicate.

As a developer, it’s your responsibility to check for, and to resolve, conflicts, or even better, employ data models that make conflicts impossible.

If you routinely create conflicts, you must really consider model changes: even if you resolve your conflicts diligently, the conflict branches in the revision tree remain with no easy way to tidy that up. For more information, see the following websites:
Watson Query CLI 

The IBM Cloud® command-line interface (CLI) provides extra capabilities for service offerings. You can use IBM Cloud CLI to manage service instance and virtualizations.



 Prerequisites 



* Install the [IBM Cloud CLI](https://cloud.ibm.com/docs/cli?topic=cli-getting-started).
* Install the <CLI_name> CLI by running the following command:

ibmcloud plugin install watson-query
* Use [ibmcloud login command](https://cloud.ibm.com/docs/cli?topic=cli-ibmcloud_cliibmcloud_login) for logging in to your IBM Cloud account.
* Set IBM Cloudant service URL. For more information, see Service configuration.



You're notified on the command line when updates to the IBM Cloud CLI and plug-ins are available. Be sure to keep your CLI up to date so that you can use the latest commands. You can view the current version of all installed plug-ins by running ibmcloud plugin list.





 Service configuration 

When you make a server resource request, you can either set CRN environment variable or set --instance-id in each sub command to identify instance. To set environment variabel, firstly get CRN of service instance by

ibmcloud resource service-instances --service-name data-virtualization --long

Then set the DATA_VIRTUALIZATION_CRN environment variable. You can define this variable two ways:

Export them as environment variables for example,export DATA_VIRTUALIZATION_CRN=.... Store them in a [credentials file](https://cloud.ibm.com/apidocs/cloudant?code=goauthentication-with-external-configuration). As an alternative to ibmcloud login, you can set the environment variable DATA_VIRTUALIZATION_APIKEY to an IAM API key.

IBM Cloud CLI requires Java&trade 1.8.0. You can download the CLI from IBM Cloud to use on your local system as a complement to the IBM Cloud console.





 Data sources
Getting a list of all databases in the account 

To list all the databases in an account, send a GET request to https://$ACCOUNT.cloudant.com/_all_dbs.

See the following example of using HTTP to list all databases:

GET /_all_dbs HTTP/1.1

See the following example to list all databases:



* Curl
* Java
* Python
* Go
* Node



curl -H "Authorization: Bearer $API_BEARER_TOKEN" -X GET "$SERVICE_URL/_all_dbs"

import com.ibm.cloud.cloudant.v1.Cloudant;
import com.ibm.cloud.cloudant.v1.model.GetAllDbsOptions;

import java.util.List;

Cloudant service = Cloudant.newInstance();

List<String> response =
service.getAllDbs().execute().getResult();

System.out.println(response);

from ibmcloudant.cloudant_v1 import CloudantV1

service = CloudantV1.new_instance()

response = service.get_all_dbs().get_result()

print(response)

getAllDbsOptions := service.NewGetAllDbsOptions()

result, response, err := service.GetAllDbs(getAllDbsOptions)
if err != nil {
panic(err)
}

b, _ := json.MarshalIndent(result, "", " ")
fmt.Println(string(b))

const { CloudantV1 } = require('@ibm-cloud/cloudant');

const service = CloudantV1.newInstance({});

service.getAllDbs().then(response => {
console.log(response.result);
});

The previous Go example requires the following import block:

import (
"encoding/json"
"fmt"
Before you run a curl request, run the following command from the command line to acquire a JWT token: ibmcloud iam oauth-tokens | awk '{print $4}'

See the following example request to retrieve information about event types:



* Curl
* Java
* Node
* Python
* Go



curl -H "Authorization: Bearer $JWT" -X GET "$SERVICE_URL/_api/v2/user/activity_tracker/events"

import com.ibm.cloud.cloudant.v1.Cloudant;
import com.ibm.cloud.cloudant.v1.model.ActivityTrackerEvents;

Cloudant service = Cloudant.newInstance();

ActivityTrackerEvents response =
service.getActivityTrackerEvents().execute().getResult();

System.out.println(response)

const { CloudantV1 } = require('@ibm-cloud/cloudant');

const service = CloudantV1.newInstance({});

service.getActivityTrackerEvents().then(response => {
console.log(response.result);
});

from ibmcloudant.cloudant_v1 import CloudantV1

service = CloudantV1.new_instance()

response = service.get_activity_tracker_events().get_result()

print(response)

getActivityTrackerEventsOptions := service.NewGetActivityTrackerEventsOptions()

activityTrackerEvents, response, err := service.GetActivityTrackerEvents(getActivityTrackerEventsOptions)
if err != nil {
panic(err)
}

b, _ := json.MarshalIndent(activityTrackerEvents, "", " ")
fmt.Println(string(b))

The previous Go example requires the following import block:

import (
"encoding/json"
"fmt"
Using service endpoints to privately connect to IBM Cloudant 

To ensure that you enhance control and security over your data when you use IBM® Cloudant® for IBM Cloud®, you have the option of using private routes to IBM Cloud® service endpoints. Private routes are not accessible or reachable over the internet. By using the IBM Cloud Private service endpoints feature, you can protect your data from threats from the public network and logically extend your private network.

IBM Cloudant sends customer logs to LogDNA by using a private service endpoint.



 Before you begin 

You must first enable virtual routing and forwarding in your account, and then you can enable the use of IBM Cloud Private service endpoints. For more information about setting up your account to support the private connectivity option, see [Enabling VRF and service endpoints](https://cloud.ibm.com/docs/account?topic=account-vrf-service-endpoint).

Only IBM Cloudant users with the dedicated hardware plan can have a cloud service endpoint (CSE). All new dedicated hardware clusters have a CSE. If you are an existing user, and you do not have a CSE, contact the [IBM Cloud Support Center](https://cloud.ibm.com/unifiedsupport/supportcenter).

Multi-tenant users cannot use CSE.





 Setting up service endpoints for IBM Cloudant 

Cloud service endpoints are ready to use when an instance is deployed. Therefore, there is no set up.

You can verify connectivity to your private service endpoint by getting the Cloudant server information from the URL of the private endpoint.



* Curl
* Java
* Node
* Python
* Go



curl https://$ACCOUNT-bluemix.private.cloudantnosqldb.appdomain.cloud

import com.ibm.cloud.cloudant.v1.Cloudant;
import com.ibm.cloud.cloudant.v1.model.ServerInformation;

Cloudant service = Cloudant.newInstance();