vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Summary 

Today, we combined two IBM Cloud services to optimize cost and user experience: IBM Cloudant as a document store and query engine and Databases for Redis as a content cache. Cached documents can be retrieved more quickly and more cheaply, but the tradeoff is that your application might be showing old data to your users for a time.

If you followed this tutorial, you must deprovision your resources to stop incurring charges. You can deprovision from the terraform directory on your terminal by typing the following command, terraform destroy --auto-approve.
A text file that contains instructions to build a Docker image.





 document set 

A collection of documents. Documents that are imported together become a document set. Annotated documents that are grouped together for training purposes are generated as document sets.





 domain 

Part of a naming hierarchy that specifies the route. For example, example.com. In IBM Cloud, domains are associated with orgs. Domain objects are not directly bound to apps. See also [host](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx2002243), [subdomain](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx2040080), [Uniform Resource Locator](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx2042491), [custom domain](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx5728384), [organization](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx2032585), [route](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx2037338).





 DR 

See [disaster recovery](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx2113280).





 dynamic secret 

A unique value, such as a password or an API key, that is created dynamically and leased to an application that requires access to a protected resource. After a dynamic secret reaches the end of its lease, access to the protected resource is revoked and the secret is deleted automatically. See also [secret](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx2789492), [secrets engine](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx9968967).







 E 



 endorse 

To validate a chaincode transaction that was made by another member of a blockchain network.
Release notes for IBM Cloud® Databases for EnterpriseDB 

Use these release notes to learn about the latest updates to IBM Cloud® Databases for EnterpriseDB that are grouped by date or build number.



 23 May 2023 

Setting up disk alerts for disk utilization tutorial
: In this tutorial, you use the IBM Cloud API and the [IBM Cloud CLI](https://cloud.ibm.com/docs/cli?topic=cli-getting-started) to set up an alert that emails you whenever the disk utilization of your database exceeds 90%. This specific example creates an alert on a Databases for Elasticsearch deployment, but it is applicable to all the databases in the IBM Cloud Databases catalog. For more information, see [Setting up disk alerts for disk utilization](https://cloud.ibm.com/docs/databases-for-enterprisedb?topic=databases-for-enterprisedb-disk-util-alert-tutorial).





 19 October 2022 

Deploying and Connecting a Cloud Databases Instance Tutorial
: This tutorial guides you through the process of deploying a Cloud Databases instance and connecting it to a web front end by creating a webpage that allows visitors to input a word and its definition. These values are then stored in a database running on Cloud Databases. You install the database infrastructure by using Terraform and your web application uses the popular Express framework. The application can then be run locally, or by using Docker. For more information, see [Deploying and Connecting a Cloud Databases Instance](https://cloud.ibm.com/docs/databases-for-enterprisedb?topic=cloud-databases-create-instance-tutorial).





 11 October 2022 

Protecting IBM Cloud® Databases for EnterpriseDB resources with context-based restrictions
: Context-based restrictions (CBR) give account owners and administrators the ability to define and enforce access restrictions for IBM Cloud® resources based on the context of access requests. Access to Cloud Databases resources can be controlled with CBR and identity and access management (IAM) policies.
* [Local File System](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-lfs-cp4d)
* [Notes](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-notes-cp4d)
* [Salesforce](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-salesforce-cp4d)
* [SharePoint Online](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-sharepoint-online-cp4d)
* [SharePoint On Prem](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-sharepoint-onprem-cp4d)
* [Web crawl](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-web-cp4d)
* [Windows File System](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-wfs-cp4d)



Your data source isn't listed? You can work with a developer to create a custom connector. For more information, see [Building a Cloud Pak for Data custom connector](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-build-connector).

If you have special requirements when you add source documents, such as a need to exclude certain files, you can work with a developer to create a custom crawler plug-in. The crawler plug-in can apply more nuanced rules to what documents and what fields in the documents get added. For more information, see [Building a Cloud Pak for Data custom crawler plug-in](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-crawler-plugin-build).



 Data source requirements 

The following requirements and limitations are specific to IBM Watson® Discovery:
[query](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-parametersquery) A ranked query language search for matching documents. query=bees 





Structure parameters

 Parameter Description Example 

 [count](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-parameterscount) The number of result documents to return. count=15 
 [highlight](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-parametershighlight) Highlight query matches highlight=true 
 [offset](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-parametersoffset) The number of results to ignore before returning result documents from the results set offset=100 
 [return](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-parametersreturn) List of fields to return return=title,url 
 [sort](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-parameterssort) Field to sort results set by sort=enriched_text.sentiment.document.score 
 [spelling suggestions](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-parametersspell) Spelling suggestions returned for natural language queries spelling_suggestions=true 





 Query limitations 

You cannot query on field names that contain the following characters:



* Numbers (0 - 9) in the suffix of the field name. For example, extracted-content2.
* The characters _, +, and - in the prefix of the field_name. For example, +extracted-content.
* The characters ., ,, and : in the field_name. For example, new:extracted-content.









 Operators 

Operators are the separators between different parts of a query. The following table lists the available operators.



Query operators

 Operator Description Example 

 [.]
Examine each enrichment to learn about the JSON fields you can query on.

![Default configuration fields](https://cloud.ibm.com/docs-content/v1/content/ded4adc3ea0bd2a81b113c579f2b1183926da211/discovery/images/JSON.png)
* entities - Start by finding the text field, and examine the other enrichment information.
* sentiment - Start by finding the label field, and examine the other enrichment information.
* concepts - Start by finding the text field, and examine the other enrichment information.
* categories - Start by finding the document field, and examine the other enrichment information.



After you examine the insights in the first document, you can look at the other three documents.
3. View the available fields in the Visual Query Builder. On the Build queries screen, click Search for documents, then Use the Discovery Query Language. Click the Field drop-down to view the fields available in your data. Click Edit in query language to build queries manually using the Discovery Query Language.





 How to structure a basic query 

As you can see, the JSON is hierarchical, so queries must follow that same hierarchy. For example, if the JSON has this hierarchy,

"enriched_text": {
"concepts": [
{
"text": "Cloud computing",
"relevance": 0.610029}
]
}

Your query would be structured like this:

![Example query structure](https://cloud.ibm.com/docs-content/v1/content/ded4adc3ea0bd2a81b113c579f2b1183926da211/discovery/images/query_structure2.png)

Operators that evaluate a field (<= , >=, <, >) require a number or date as the value. Using quotations around a value always makes it a string. Therefore score>=0.5 is a valid query and score>="0.5" is not.
An example for a CRN table is: crn:v1:bluemix:public:dashdb-for-transactions:us-south:s/c3882b7e-00c4-4e7c-a63b-cded1c298f25:23eb50c5-723d-41e0-b7d8-603feaa79ccc:cf-service-instance:/RWS46052.QUERY_RESULT





 Db2 table URI location 

If the SQL user cannot access the service credentials for the Db2 on Cloud instance (because the user does not have access to the account that contains the database instance, or is not granted Operator privilege on the instance), the user can specify the database location by using a URI with the Db2 database hostname.

By default the access to this database is performed with the IAM identity of the user who submitted the query. This default requires that the database is enabled for IAM authentication. Also, before you use this option, make sure that the IBMid of the user was added as a database user. For more information, see [Console user experience](https://www.ibm.com/support/knowledgecenter/en/SS6NHC/com.ibm.swg.im.dashdb.security.doc/doc/iam.html) in the Identity and access management (IAM) on IBM Cloud documentation. This option is not available for Db2 Lite plans because they don't support IAM authentication in the database.

If you cannot or do not want to use the default mechanism of IAM user authentication, you can instead specify a custom user and password or a custom API key. To do so, store the password or key into IBM Cloud® Secrets Manager and specify an [access secret clause](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceaccessSecrets) in your query. For more information, see the [security documentation](https://cloud.ibm.com/docs/sql-query?topic=sql-query-authenticationaccessauthentication).
Once deployed, Databases for Elasticsearch Enterprise Plan clusters make full use of other Elastic Stack components such as [Kibana](https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=databases-for-elasticsearch-getting-startedkibana), [Logstash](https://www.elastic.co/logstash/), and [Beats](https://www.elastic.co/beats/).





 IBM Cloud® Databases for Elasticsearch Standard Plan 

Versions 7.9 and 7.10 are available on Cloud Databases under the Standard Plan, but will be End of Life (EOL) by 30 November 2023. To ensure access to the latest features and a longer timescale of version support, create new Databases for Elasticsearch deployments under the Enterprise Plan.





 Using the Pricing Calculator 

Prices for the different plans reflect different levels of available functionality. Be aware that prices for the Standard Plan will increase starting July 1 to bring them into line with the prices for the Enterprise Plan. Templates are provided for ease of use and provide balanced resource allocations appropriate for general-purpose workloads. The Custom tab can be used to configure Disk, RAM, and vCPU, as wanted.

For pricing estimation, use the Add to Estimate button on the [Databases for Elasticsearch catalog page](https://cloud.ibm.com/catalog/databases-for-elasticsearch). Input your total consumption across three data members into the calculator. For example, 5 GB of disk and 1 GB of RAM across three data members would be priced at 15 GB of disk and 3 GB of RAM.





 Dedicated Cores Pricing 

You have the option of selecting the CPU allocation for your deployment. With dedicated cores, your resource group is given a single-tenant host with a guaranteed minimum reserve of cpu shares. Your deployments are then allocated the number of CPUs you specify. The cost of dedicated cores is $30 per core per month, and each member gets the selected number of cores. For example, if you provision a deployment with 3 dedicated cores per member, that is a total of 9 cores and is billed at $270 per month.

Dedicated cores are an optional feature.
Using IBM Cloudant 

If you never use IBM Cloudant or NoSQL databases in general, scan this introduction and some best practices before you read further. It describes the most important things you need to know about IBM Cloudant and how to use it best. The rest of the documentation assumes that you know these basics.

You can find more information about IBM Cloudant in the following sections:



* [Client Libraries](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-client-librariesclient-libraries)
* [API and SDK reference](https://cloud.ibm.com/apidocs/cloudantintroduction)





 Connecting to IBM Cloudant 

To access IBM Cloudant, you must have an [IBM Cloud® account](https://cloud.ibm.com/login).





 HTTP API 

All requests to IBM Cloudant go over the web. This statement means that any system that can speak to the web can speak to IBM Cloudant. All language-specific libraries for IBM Cloudant are just wrappers that provide some convenience and linguistic niceties to help you work with a simple API. Many users choose to use raw HTTP libraries for working with IBM Cloudant.

For more information about how IBM Cloudant uses HTTP, see [HTTP](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-how-http-works-with-cloudant) in the API reference.

IBM Cloudant supports the following HTTP request methods:

GET
: Request the specified item. As with normal HTTP requests, the format of the URL defines what is returned. With IBM Cloudant, this definition can include static items, database documents, and configuration and statistical information. In most cases, the information is returned in the form of a JSON document.

HEAD
: The HEAD method retrieves the HTTP header of a GET request without the body of the response.

POST
: Upload data. In IBM Cloudant's API, the POST method sets values, uploads documents, sets document values, and starts some administration commands.

PUT
: Used to "store" a specific resource.
Working with IBM Cloudant Query 

IBM® Cloudant® for IBM Cloud® Query is a declarative JSON querying syntax for IBM Cloudant databases. You can use a json or text type of index with IBM Cloudant.

In the following cases, you can specify how the index is created by making it of type json:



* You know exactly what data you want to look for.
* You want to keep storage and processing requirements to a minimum.



But for maximum flexibility when you search for data, you typically create an index of type text. Indexes of type text have a simple mechanism for automatically indexing all the fields in the documents.

While more flexible, text indexes might take longer to create and require more storage resources than json indexes.



 Creating an index 

You can create an index with one of the following types:



* "type": "json"
* "type": "text"





 Creating a type=json index 

To create a JSON index in the database $DATABASE, make a POST request to /$DATABASE/_index with a JSON object that describes the index in the request body. The type field of the JSON object must be set to json. A JSON index can be partitioned or global; this option is set by using the partitioned field.

See the following example that uses HTTP to request an index of type JSON:

POST /$DATABASE/_index HTTP/1.1
Content-Type: application/json

See the following example of a JSON object that creates a partitioned index that is called foo-partitioned-index for the field called foo:

{
"index": {
"fields": ["foo"]
},
"name" : "foo-partitioned-index",
"type" : "json",
"partitioned": true
}

See the following example of a JSON object that creates a global index that is called bar-global-index for the field called bar:

{
"index": {
"fields": ["bar"]
},
"name" : "bar-global-index",
In most cases, the information is returned in the form of a JSON document.

HEAD
: The HEAD method retrieves the HTTP header of a GET request without the body of the response.

POST
: Upload data. In IBM Cloudant's API, the POST method sets values, uploads documents, sets document values, and starts some administration commands.

PUT
: Used to "store" a specific resource. In IBM Cloudant's API, PUT creates new objects, including databases, documents, views, and design documents.

DELETE
: Deletes the specified resource, including documents, views, and design documents.

COPY
: A special method that copies documents and objects.

If the client (such as some web browsers) doesn't support the use of HTTP methods, POST can be used instead with the X-HTTP-Method-Override request header set to the actual HTTP method.



 Method not allowed error 

If you use an unsupported HTTP request type with a URL that doesn't support the specified type, a [405](https://cloud.ibm.com/apidocs/cloudantlist-of-http-codes) error is returned. The error that lists the supported HTTP methods, as shown in the following example.



 Example error message in response to an unsupported request 

{
"error":"method_not_allowed",
"reason":"Only GET,HEAD allowed"
}









 JSON 

IBM Cloudant stores documents that use JSON (JavaScript Object Notation) encoding, so anything encoded into JSON can be stored as a document. Files that include media, such as images, videos, and audio, are called BLOBs (Binary Large Objects). BLOBs can be stored as attachments associated with documents.

More information about JSON can be found in the [JSON Guide](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-how-json-works).





 Distributed systems 

By using IBM Cloudant's API, you can interact with a collaboration of numerous machines, called a cluster.
How JavaScript Object Notation (JSON) works 

Most requests and responses to and from IBM® Cloudant® for IBM Cloud® use [JSON](https://en.wikipedia.org/wiki/JSON) for formatting the content and structure of the data and responses.

In IBM Cloudant databases, the JSON object is used to represent various structures, including all documents in a database.

Parsing JSON into a JavaScript object is supported through the JSON.parse() function in JavaScript, or through various [libraries](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-client-librariesclient-libraries) that perform the parsing of the content into a JavaScript object for you. Libraries for parsing and generating JSON are available for many major programming languages.

JSON is used because it's the simplest and easiest solution for working with data that uses a web browser. As a result, JSON structures can be evaluated and used as JavaScript objects within the web browser environment. JSON also integrates with the server-side JavaScript used within IBM Cloudant. JSON documents are always UTF-8 encoded.

Be careful to follow these guidelines:



* Your JSON structures are valid.
* You normalize strings in JSON documents retrieved from IBM Cloudant.



JSON supports the same basic types that are supported by JavaScript: numbers, strings, Booleans, arrays, and objects.



 Numbers 

Numbers can be integer or floating point values.



 Example of a number in JSON format 

123







 Strings 

Strings must be enclosed by double quotation marks. Strings support Unicode characters and escaping a backslash.



 Example of a string in JSON format 

"A String"







 Booleans 

A true or false value.



 Example of a boolean in JSON format 

{
"value": true
}







 Arrays 

A list of values enclosed in brackets. The values that are enclosed can be any valid JSON.



 Example of an array in JSON format 

[
"one",
2,
"three",
],
true,
{
If it exists, the app can use the IBM Cloudant variable's properties to access the database. However, if the <VCAP_SERVICES> environment variable is not present, the local IBM Cloudant service instance is used with the provided default values.
4. Interact with the service instance.

You can interact with the service instance by using the credential information. The actions that you can take include read, write, and update. The following example demonstrates how to insert a JSON object into the IBM Cloudant service instance:

// create a new message
var create_message = function(req, res) {
require('cloudantdb').connect(cloudant.url, function(err, conn) {
var collection = conn.collection('messages');

// create message record
var parsedUrl = require('url').parse(req.url, true);
var queryObject = parsedUrl.query;
var name = (queryObject["name"] || 'Bluemix');
var message = { 'message': 'Hello, ' + name, 'ts': new Date()
};
collection.insert(message, {safe:true}, function(err){
if (err) { console.log(err.stack); }
res.writeHead(200, {'Content-Type': 'text/plain'});
res.write(JSON.stringify(message));
res.end('n');
});
});
}
Show more
5. Optional: Unbind or delete a service instance.

You might want to unbind or delete a service instance when it is no longer used or when you want to free up some spaces. To unbind a service instance from your app, use the ibmcloud cf unbind-service command. To delete a service instance, use the ibmcloud cf delete-service command.

For more information about services, see Services.
panic(err)
}

b, _ := json.MarshalIndent(findResult, "", " ")
fmt.Println(string(b))

The previous Go example requires the following import block:

import (
"encoding/json"
"fmt"
"github.com/IBM/cloudant-go-sdk/cloudantv1"
)

All Go examples require the service object to be initialized. For more information, see the API documentation's [Authentication section](https://cloud.ibm.com/apidocs/cloudant?code=goauthentication-with-external-configuration) for examples.





 Finding the infrastructure ID for a device 

The two queries we've yet to perform are shown in the following list:



1. Readings for all time for a specific device.
2. Readings for today for a specific device.



For these two queries, you need to find the partition for the devices by using the global by-device index. Then, you can query the individual partition for readings. While you might use a global index to query for the readings for individual devices, the mapping from device to infrastructure ID is highly cache-able. It never changes! With this approach, you can mostly use the cheaper and more efficient partitioned query for most requests.

Using a global index to query directly for device readings might be more efficient if caching the device to infrastructure mapping doesn't work well for a specific application.

To find the relevant partition for a device, you query the by-device view, sending the device ID as the key:

curl -X POST "$SERVICE_URL/readings/_design/infrastructure-mapping/_view/by-device' -H 'Content-Type: application/json' --data '{
"keys": ["device-123456"], "limit": 1
}'

For more language examples that show querying a global view, see the [Query a MapReduce view in API Docs](https://cloud.ibm.com/apidocs/cloudantpostview).