vania-janet/multiturn-rag-retrieval-data huggingface dataset:
cloudantConsumer.js: This script creates a trucktracker database in Cloudant if it does not already exist and then polls Event Streams at regular intervals for more data. When data comes in, it gets written in bulk to Cloudant.
5. redisConsumer.js: This script polls Event Streams at regular intervals for data. When data comes in, it gets written into a Redis "hash" where the hash key is the truck ID and the value is the truck data (lat/long). This hash arrangement allows all the truck data to be pulled down by the server.js script in one go. The great thing is that if for some reason one of your consumers crashes, it will pick up where it left off without losing any data, because Event Streams knows the last data point it served to each consumer. Decoupling data-producing systems from data-consuming systems in this way is a good design practice that increases resilience and fault tolerance.







 Summary 

In this tutorial, we have combined a number of IBM services to model a realistic IoT pattern where Internet-enabled devices on trucks are producing data that gets consumed by various applications for different purposes.

Because of the managed nature of all these services, as an application developer, you can focus on solving your business problems and leave us to do the heavy lifting of managing the infrastructure for you, thereby saving potentially months of work on non-core activities like server provisioning, software deployment, patching, security and others.

If you followed this tutorial, remember to de-provision your infrastructure to stop incurring charges. On your terminal, do the following:

cd terraform/
terraform destroy

If you want to take the next step in your developer journey, check out some of our [trial offers](https://www.ibm.com/products?types%5B0%5D=trial).
PUT /_api/v2/user/config/cors <br>[Cloudant](https://cloud.ibm.com/apidocs/cloudantputcorsconfiguration) Modify the CORS configuration. 
 GET /_db_updates <br>[Cloudant](https://cloud.ibm.com/apidocs/cloudantgetdbupdates), [CouchDB](https://docs.couchdb.org/en/latest/api/server/common.htmldb-updates) List all database events in the instance. <br>In IBM Cloudant, the endpoint is only available to Dedicated Hardware plan customers. Its documentation references more query parameters (limit, since, descending) and another feed type value (normal). For CouchDB, its documentation references another feed type value (eventsource). 
 GET /_membership <br>[Cloudant](https://cloud.ibm.com/apidocs/cloudantgetmembershipinformation), [CouchDB](https://docs.couchdb.org/en/latest/api/server/common.htmlmembership) Retrieve cluster membership information. Active clusters are indicated in the cluster_nodes field, while all_nodes has all nodes. 
 POST /_replicate <br>[Cloudant](https://cloud.ibm.com/apidocs/cloudantpostreplicate), [CouchDB](https://docs.couchdb.org/en/latest/api/server/common.htmlreplicate) Request, configure, or stop a replication operation. <br>IBM Cloudant documentation references more request body fields (selector, since_seq). 
 PUT /_replicator/{doc_id} <br>[Cloudant](https://cloud.ibm.com/apidocs/cloudantputreplicationdocument), [CouchDB](https://docs.couchdb.org/en/latest/replication/replicator.html) Trigger a replication. <br>IBM Cloudant documentation references more request body fields (selector, since_seq).
youyou@cloudshell:$ ibmcloud resource service-instances --service-name cloudantnosqldb
Retrieving instances with type service_instance in all resource groups in all locations under ..
OK
Name Location State Type
<your-initials>-cloudant-service us-south active service_instance
youyou@cloudshell:$ ibmcloud resource service-instance <your-initials>-cloudant-service
Retrieving service instance <your-initials>-cloudant-service in all resource groups under ...
OK

Name: <your-initials>-cloudant-service
ID: crn:v1:bluemix:public:cloudantnosqldb:us-south:a/0123456789507a53135fe6793c37cc74:SECRET
GUID: SECRET
Location: us-south
Service Name: cloudantnosqldb
Service Plan Name: standard
Resource Group Name: Default
State: active
Type: service_instance
Sub Type:
Created at: 2020-05-06T22:39:25Z
Created by: youyou@us.ibm.com
Updated at: 2020-05-06T22:40:03Z
Last Operation:
Status create succeeded
Message Provisioning is complete
Updated At 2020-05-06 22:40:03.04469305 +0000 UTC

youyou@cloudshell:$ ibmcloud resource service-keys --instance-name $YOURINITIALS-cloudant-service --output json
[
{
"guid": "01234560-902d-4078-9a7f-20446a639aeb",
"id": "crn:v1:bluemix:public:cloudantnosqldb:us-south:a/0123456789507a53135fe6793c37cc74:SECRET",
const { CloudantV1 } = require('@ibm-cloud/cloudant');

const service = CloudantV1.newInstance({});

service.deleteAttachment({
db: 'products',
docId: 'small-appliances:100001',
attachmentName: 'product_details.txt',
rev: '4-1a0d1cd6f40472509e9aac646183736a'
}).then(response => {
console.log(response.result);
});

The previous Go example requires the following import block:

import (
"encoding/json"
"fmt"
"github.com/IBM/cloudant-go-sdk/cloudantv1"
)

All Go examples require the service object to be initialized. For more information, see the API documentation's [Authentication section](https://cloud.ibm.com/apidocs/cloudant?code=goauthentication-with-external-configuration) for examples.

If the deletion is successful, the response includes "ok": true, and the ID and new revision of the document.

See the following example response after a successful delete of an attachment:

{
"ok": true,
"id": "DocID",
"rev": "3-aedfb06537c1d77a087eb295571f7fc9"
}





 Inline 

Inline attachments are attachments that are included as part of the JSON content. The content must be provided by using [BASE64](https://en.wikipedia.org/wiki/Base64) representation, as shown in the example.

A full list of media types is available in the [media types](https://en.wikipedia.org/wiki/Internet_media_typeList_of_common_media_types) article.

See the following example JSON document that includes an inline attachment of a jpeg image:

{
servicePassword = "49c0c343d225623956157d94b25d574586f26d1211e8e589646b4713d5de4801"
serviceURL = "https://353466e8-47eb-45ce-b125-4a4e1b5a4f7e-bluemix.cloudant.com"
3. Establish a connection to the service instance.

client = Cloudant(serviceUsername, servicePassword, url=serviceURL)
client.connect()
4. Or replace ACCOUNT_NAME and API_KEY with the values from your IAM API service credentials.

client = Cloudant.iam(ACCOUNT_NAME, API_KEY, connect=True)



Now, your Python application can access the service instance on IBM Cloud.





 Step 2: Creating a database within the service instance 

Next, you create a database within the service instance, called databasedemo.



1. Create this instance by defining a variable in the Python application.

databaseName = "databasedemo"
2. Create the database.

myDatabaseDemo = client.create_database(databaseName)
3. Verify that the database was created successfully.

if myDatabaseDemo.exists():
print("'{0}' successfully created.n".format(databaseName))







 Step 3: Storing a small collection of data as documents within the database 

You want to store a small, simple collection of data in the database. This data is used in other tutorials, like [Using IBM Cloudant Query to find data](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-creating-an-ibm-cloudant-query).



1. Create sample data.

sampleData = [
1, "one", "boiling", 100],
2, "two", "hot", 40],
3, "three", "hot", 75],
4, "four", "hot", 97],
}).then(response => {
console.log(response.result);
});

The previous Go example requires the following import block:

import (
"encoding/json"
"fmt"
"io/ioutil"
"github.com/IBM/cloudant-go-sdk/cloudantv1"
)

All Go examples require the service object to be initialized. For more information, see the API documentation [Authentication section](https://cloud.ibm.com/apidocs/cloudant?code=goauthentication-with-external-configuration) for examples.

The response includes the document ID and the new document revision.

Attachments don't have their own revisions. Instead, when you update or create an attachment, the revision of the document it's attached to changes.

See the following example response with the document ID and new revision:

{
"id" : "FishStew",
"ok" : true,
"rev" : "9-247bb19a41bfd9bfdaf5ee6e2e05be74"
}





 Read 

To retrieve an attachment, make a GET request to https://$ACCOUNT.cloudant.com/$DATABASE/$DOCUMENT_ID/$ATTACHMENT. The body of the response is the raw content of the attachment.

See the following example of reading an attachment by using HTTP:

GET /$DATABASE/$DOCUMENT_ID/$ATTACHMENT HTTP/1.1

See the following example of reading an attachment:



* Curl
* Java
* Python
* Go
* Node



curl -H "Authorization: Bearer $API_BEARER_TOKEN" -X GET "$SERVICE_URL/products/small-appliances:100001/product_details.txt"

import com.ibm.cloud.cloudant.v1.Cloudant;
import com.ibm.cloud.cloudant.v1.model.GetAttachmentOptions;

import java.io.BufferedReader;
import java.io.InputStream;
The maximum document size is 1 MB, but documents must be much smaller than that size, typically a few KB.

For more information, see the following blog posts:



* [Optimal IBM Cloudant Indexing](https://blog.cloudant.com/2019/05/10/Optimal-Cloudant-Indexing.html)
* [Time-series Data Storage](https://blog.cloudant.com/2019/04/08/Time-series-data-storage.html)
* [Partitioned databases - data design](https://blog.cloudant.com/2019/03/05/Partition-Databases-Data-Design.html)







 Making the most of the primary index 

IBM Cloudant has a primary index on the document's _id attribute. This index allows documents to be retrieved by _id (GET /db/id) or a range of _ids (GET /db/_all_docs?startkey="a"&endkey="z"). By storing data in the primary key and ensuring that each _id is unique, the primary index can be used to fetch documents and ranges of documents without secondary indexing. See the following list of ideas:



* If you have something unique in your object that would be useful to query against, use it as your _id field, for example, bob.smith@gmail.com, isbn9780241265543, or oakland,ca.
* If your objects contain a hierarchy, model that in your _id: usa:ca:oakland or books:fiction:9780241265543. The hierarchy goes from largest to smallest, so you can use the primary index to find all the cities in usa or all the cities in usa:ca, without secondary indexing.
* If you're storing time-series data, encoding time at the start of your _id sorts the primary index by time, for example, 001j40Ox1b2c1B2ubbtm4CsuLB4L35wQ.
* Partitioned databases group documents that share a partition key together. A partition key must have many values and must not include hot spots to avoid directing a large proportion of your application's traffic to a few partitions.
Learn how to use two API calls to perform all the basic IBM Cloudant operations while also acting on more than one document per API call.



* The Bulk API video script

Welcome to the Introduction to IBM Cloudant course, an 17-part video series that gives you an overview of the IBM Cloudant database-as-a-service.

This video is part 8 - The Bulk API.

In the previous part, we saw how documents could be easily added, updated, and deleted singly by using the IBM Cloudant HTTP API. In this part, we see how two API calls can be used to achieve all the basic IBM Cloudant operations. The added benefit of that is being able to act on more than one document per API call.

We already discussed the _all_docs endpoint. We used it to fetch a list of all the documents in a database, but it has other features too.

The key parameter can be used to specify a single document to fetch, making it equivalent to the GET /db/id API call. Similarly, the keys parameter takes an array of document IDs and returns them all. The startkey and endkey parameters fetch a slice of the primary index between the supplied limits. Adding include_docs=true instructs IBM Cloudant to supply the document bodies too. And limit specifies how many documents to return in one API call.

The _bulk_docs endpoint allows multiple inserts, updates, and deletes operations to be performed in one API call. It expects an object that contains a docs array - each element of that array is an operation to perform on a single document. The request body is posted to IBM Cloudant, allowing many operations to be packed into a single API call.

In this example, the first document is an insert because no revision token is supplied. The second document is an update to a document because a revision token is supplied with a new document body. The third document is a deletion. A revision token is supplied, but the body is simply _deleted: true, which tells IBM Cloudant to mark the document as deleted.

It's important to note that this scenario isn't like a transaction in a relational database - all or none of these operations could succeed or fail individually.
"iam_apikey_name": "Service apikey name]",
"iam_role_crn": "Service role crn]",
"iam_serviceid_crn": "Service id crn]",
"instance": "Cloudant-gm",
"url": "Service url]",
"username": "Service username]"
}
}
}
],
}



In this example, the credentials for the IBM Cloudant service belong to a parameter named __bx_creds.





 Working with documents in an IBM Cloudant database 

You can use an action to read, write, update, delete a document from an IBM Cloudant database.



 Reading a document 

You can use the /whisk.system/cloudant/read action to read a document from your IBM Cloudant database.

Before you begin

If you do not have a document in your IBM Cloudant database, you can create one by using the IBM Cloudant dashboard. The URL for the dashboard is https://<mycloudantaccount>.cloudant.com/dashboard.htmldatabase/<database_name>/_all_docs?limit=100.

Fetch a document by using the read action. Replace /_/myCloudant with your package name, <database_name> with your database name, and <document_id> with the file ID. Invoke the action to test fetching a document.

The following example shows the command syntax for the action invoke command.

ibmcloud fn action invoke /_/myCloudant/read --blocking --result --param dbname <database_name> --param id <document_id>

Invoke the action to test reading a file. This example reads a file with the id of 9f86f4955e7a38ab0169462e6ac0f476, which is an empty file.
* Curl
* Java
* Node
* Python
* Go



curl https://$ACCOUNT-bluemix.private.cloudantnosqldb.appdomain.cloud

import com.ibm.cloud.cloudant.v1.Cloudant;
import com.ibm.cloud.cloudant.v1.model.ServerInformation;

Cloudant service = Cloudant.newInstance();

service.setServiceUrl("https://$ACCOUNT-bluemix.private.cloudantnosqldb.appdomain.cloud");
...

const { CloudantV1 } = require('@ibm-cloud/cloudant');

const service = CloudantV1.newInstance({});

service.setServiceUrl('https://$ACCOUNT-bluemix.private.cloudantnosqldb.appdomain.cloud');

from ibmcloudant.cloudant_v1 import CloudantV1

service = CloudantV1.new_instance()

service.set_service_url('https://$ACCOUNT-bluemix.private.cloudantnosqldb.appdomain.cloud')

import (
"github.com/IBM/cloudant-go-sdk/cloudantv1"
)

service, _ := cloudantv1.NewCloudantV1UsingExternalConfig(
&cloudantv1.CloudantV1Options{},
)
service.SetServiceURL("https://$ACCOUNT-bluemix.private.cloudantnosqldb.appdomain.cloud")

if err != nil {
panic(err)
}

If it succeeds, you are ready to go. Otherwise, you might want to check a few things:



* Are your CSEs correctly enabled?
* Can you reach the CSE endpoints from where you're running the command?
* Do other CSE endpoints in the same account work correctly?
: Only supported value for <TARGET_TYPE> is cloud-object-storage.

--file <@COS_ENDPOINT_DEFINITION_JSON_FILE>
: A file containing an endpoint definition in the following format:

{
"endpoint": "aaaaa",
"target_crn": "yyyyy",
"bucket": "zzzzzz",
"api_key": "xxxxxx"
}

--endpoint <COS_ENDPOINT>
: The IBM Cloud Object Storage endpoint to be associated with the IBM Cloud Object Storage bucket.

--bucket <BUCKET>
: The name of the IBM Cloud Object Storage bucket to be associated with the target.

--target-crn <COS_TARGET_CRN>
: The CRN of the IBM Cloud Object Storage instance.

--api-key <COS_API_KEY> | <@COS_API_KEY_FILE>
: Your [API key](https://cloud.ibm.com/docs/account?topic=account-manapikey) value or a reference to the API Key file used when logging in. For example, ibmcloud login --apikey $KEYFILE

--output JSON
: If specified, output will be returned in JSON format. If JSON is not specified, output will be returned in a tabular format.

help | --help | -h
: List options available for the command.





 Example
For developers of IBM Cloud Object Storage 

The powerful tools of IBM Cloud® Object Storage are available to a developer directly from the command line.

First, ensure that you have the [IBM Cloud® Platform CLI](https://cloud.ibm.com/docs/cli/index.html) and [IBM Developer Tools](https://cloud.ibm.com/docs/cloudnative/idt/index.html) installed.



 Create an instance of IBM Cloud Object Storage 



1. First, make sure that you have an API key. Get it from [IBM Cloud Identity and Access Management](https://cloud.ibm.com/iam/apikeys).
2. Log in to IBM Cloud Platform by using the CLI. It's also possible to store the API key in a file or set it as an environment variable.

ibmcloud login --apikey <value>
3. Next, create an instance of IBM Cloud Object Storage specifying the name for the instance, the ID, and the wanted plan (lite or standard). Now we have a CRN for the instance. If you have an upgraded account, specify the standard plan. Otherwise, specify lite.

ibmcloud resource service-instance-create <instance-name> cloud-object-storage <plan> global



Some users may encounter the error No resource group targeted when trying to create a new instance. This indicates that the default resource group is not available and that a resource group must be explicitly set. A list of available resource groups can be found using ibmcloud resource groups and the target can be set with ibmcloud target -g <resource-group>.

The [Getting Started guide](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-getting-started-cloud-object-storage) walks through the basic steps of creating buckets and objects, as well as inviting users and creating policies. A list of basic 'curl' commands can be found [here](https://cloud.ibm.com/docs/cloud-object-storage/cli?topic=cloud-object-storage-curl).
You can troubleshoot errors in the deployment process by using the ibmcloud cf logs GetStartedJava --recent command.





 Step 5: Add a database 

Next, we'll add an IBM Cloudant NoSQL database to this app and set up the app so that it can run locally and on IBM Cloud.



1. In your browser, log in to IBM Cloud and go to the Dashboard. Select Create resource.
2. Search for IBM Cloudant, and select the service.
3. For Available authentication methods, select Use both legacy credentials and IAM. You can leave the default settings for the other fields. Click Create to create the service.
4. In the navigation, go to Connections, then click Create connection. Select your app, and click Connect.
5. Using the default values, click Connect & restage app to connect the database to your app. Click Restage when prompted.



IBM Cloud will restart your app and provide the database credentials to your app using the VCAP_SERVICES environment variable. This environment variable is available to the app only when it is running on IBM Cloud.

Environment variables enable you to separate deployment settings from your source code. For example, instead of specifying a database password in your source code, you can store it in an environment variable that you reference in your source code.





 Step 6: Use the database 

We're now going to update your local code to point to this database. We'll store the credentials for the services in a properties file. This file will get used ONLY when the app is running locally. When running in IBM Cloud, the credentials will be read from the VCAP_SERVICES environment variable.



1. Find your app in the IBM Cloud [resource list](https://cloud.ibm.com/resources). On the Service Details page for your app, click Connections in the sidebar. Click the IBM Cloudant menu icon (…) and select View credentials.
2. Copy and paste just the url from the credentials to the url field of the src/main/resources/cloudant.properties file, and save the changes.
"type": "text"
}

Notice that the fields array requires each attribute to be named and typed (unlike type=json indexes).

The resultant index can be used by queries that contain one or more of the indexed fields:

{
"selector": {
"surname": "Dickens"
},
"sort": ["date"],
"limit": 10,
"execution_stats": true
}

The Lucene-backed indexes allow some extra flexibility over type=json indexes. You can use one, some, or all of your indexed fields in any order, and the index supports the query. With type=json indexes, the query must match all of the indexed fields to be useful.

For more information, see [Creating a type=text index](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-querycreating-a-type-text-index).





 Can I create an IBM Cloudant Query index for a subset of a database? 

Yes! If your use case requires that queries are performed within the confines of a subset of the database (for example, completed orders, or confirmed users, or in-stock products), then a partial index is best. A partial index allows an IBM Cloudant Query selector to be used at index-build time to winnow the data set that makes it into the index, keeping the index small and performant. Then, at query-time, a standard IBM Cloudant Query selector is used to further filter the partial index to get your result set.

A partial index is created by passing a partial_filter_selector to the POST /{db}/_index method. In this example, only hardback books that have a status of "published" or "reprint" make it to the index. See the following example:

{
"index": {
"partial_filter_selector": {
"$or": [
{ "status": "published" },
{ "status": "reprint" }
],
"hardback": true
},
);
response =
service.postAllDocs(docsOptionsBuilder.build())
.execute()
.getResult();
}
const { CloudantV1 } = require('@ibm-cloud/cloudant');

const service = CloudantV1.newInstance({});

async function paginate(n) {
let allDocsResult = (await service.postAllDocs({
db: 'orders',
limit: n
})).result;
while (allDocsResult.rows.length > 0) {
let documents = allDocsResult.rows;
console.log(documents);
allDocsResult = (await service.postAllDocs({
db: 'orders',
limit: n,
startKey: documentsdocuments.length - 1] .id + 'u0000'
})).result;
}
}

paginate(5)
import json
from ibmcloudant.cloudant_v1 import CloudantV1

service = CloudantV1.new_instance()
page_size = 5

response = service.post_all_docs(
db='orders',
limit=page_size,
).get_result()

while len(response"rows"]) > 0:
documents = response"rows"]
print(json.dumps(documents, indent=2))
response = service.post_all_docs(
db='orders',
limit=page_size+1, Fetch page_size + 1 documents
start_key=documents-1]"id"] !! + 'u0000'
).get_result()
pageSize := core.Int64Ptr(5)
postAllDocsOptions := service.NewPostAllDocsOptions(
"orders",
)
postAllDocsOptions.SetLimit(pageSize)