vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Country codes like "UK" must also use the Keyword analyzer to prevent the removal of "stopwords" that match the country codes, for example, "IN" for India. The Keyword analyzer is case-sensitive.





 Text 

It is best to process a block of free-form text with a language-specific analyzer, such as the English analyzer, or in a more general case, the Standard analyzer.







 Which option is best? 

When IBM Cloudant returns data from a search, you can choose between the following options: store: true or include_docs=true. See the following descriptions:



1. At index-time, choose the {store: true} option. This option indicates that the field you're dealing with needs to be stored inside the index. A field can be "stored" even if it isn't used for indexing itself. For example, you might want to "store" a telephone number, even if your search algorithm doesn't include searching by phone number.
2. At query-time, pass ?include_docs=true to indicate to IBM Cloud that you want the entire body of each matching document to be returned.



The first option means you have a larger index, but it's the fastest way of retrieving data. The second option keeps the index small, but adds extra query-time work for IBM Cloud as it must fetch document bodies after the search result set is calculated. This process can be slower to run and adds a further burden to the IBM Cloud cluster.

If possible, choose the first option use the following guidelines:



* Index only the fields that you want to be searchable.
* Store only the fields that you need to retrieve at query-time.







 Entity extraction 

Providing a good search experience depends on the alignment of your users' search needs with structure in the data. Running lots of unstructured data with an indexing engine gets you only so far. If you can add further structure to unstructured data, then the search experience benefits since fewer "false positives" are returned. Look at this example:

"Edinson Cavani scored two superb goals as Uruguay beat Portugal to set up a World Cup quarter-final meeting with France.
"fmt"
"github.com/IBM/cloudant-go-sdk/cloudantv1"
)

All Go examples require the service object to be initialized. For more information, see the API documentation's [Authentication section](https://cloud.ibm.com/apidocs/cloudant?code=goauthentication-with-external-configuration) for examples.







 Many simultaneous replications 

It is easy to forget that you previously set up replication between two databases, and so create extra replication processes in error. Each replication job is independent of the other, so IBM Cloudant does not prevent you from doing creating extra replication processes. However, each replication task uses up system resources.

You can check your "active replications" in the IBM Cloudant Dashboard to ensure that no unwanted replication tasks are in progress. Delete any _replicator documents that are no longer needed.







 Tuning replication speed 

By default, IBM Cloudant replication runs at an appropriate rate to get the data from the source to the target without adversely affecting performance. Choosing between replication rate and cluster performance for other tasks is a tradeoff. Your use case might require faster replication at the expense of other IBM Cloudant services. Alternatively, you might require cluster performance to take priority, with replication treated as a background process.

[Advanced replication API options](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-advanced-replicationadvanced-replication) are available. These options enable an increase or decrease in the amount of computing power that is used during replication, as shown in the following examples:



* If your documents contain attachments, you might want to consider reducing the batch_size and increasing the worker_processes, to accommodate larger documents in smaller batches.
* If you have many tiny documents, then you might consider increasing the [worker_process](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-advanced-replicationperformance-related-options) and [http_connections](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-advanced-replicationperformance-related-options) values.
Truck Tracker: An IoT Pattern Using OpenShift and other IBM Cloud Services 



 Combining IBM Cloud services to produce a variety of insights. 

Truck Tracker is a service that receives data from a fleet of trucks as they travel across the country. It could be any data (e.g., fuel consumption, driver details, ambient temperature, or anything else that can be measured as you cruise). For the purposes of this post, it will be the truck's ID and position (lat/long coordinates). Truck Tracker will store the data in various data stores and also show you the current position of the trucks on a map.

Some app users need this data to run analytics. For example, to find out how many miles the fleet has traveled in the last 24 hours or how much fuel it has consumed. For that, all data points can be stored in a reliable document store like [IBM Cloudant](https://www.ibm.com/cloud/cloudant), as well as searched and aggregated at will. IBM Cloudant is a fully managed, distributed database optimized for heavy workloads and fast-growing web and mobile apps.

Other users might need to know where the trucks currently are. For this, you only need the last known position of each truck. In this case, [Redis](https://www.ibm.com/cloud/learn/redis) is a perfect solution — a fast key-value store where a key is the unique truck ID and the value is its last known position.

[IBM Cloud® Databases for Redis](https://www.ibm.com/cloud/databases-for-redis) is a fully managed in-memory data structure store that can be used as a database, cache and message broker.

There may be other, yet unknown, uses for the data that require it to be accessed or stored in different ways.

So your trucks are flying around the country and data is pouring out of them. How do you make sure everyone gets the data they need? Enter [IBM Event Streams](https://www.ibm.com/cloud/event-streams), an event-streaming platform that helps you build smart apps that can react to events as they happen.
For added fun, it picks a random starting point along the route and turns around and comes back when it reaches the end of the location file.
4. cloudantConsumer.js: This script creates a trucktracker database in Cloudant if it does not already exist and then polls Event Streams at regular intervals for more data. When data comes in, it gets written in bulk to Cloudant.
5. redisConsumer.js: This script polls Event Streams at regular intervals for data. When data comes in, it gets written into a Redis "hash" where the hash key is the truck ID and the value is the truck data (lat/long). This hash arrangement allows all the truck data to be pulled down by the server.js script in one go. The great thing is that if for some reason one of your consumers crashes, it will pick up where it left off without losing any data, because Event Streams knows the last data point it served to each consumer. Decoupling data-producing systems from data-consuming systems in this way is a good design practice that increases resilience and fault tolerance.







 Summary 

In this tutorial, we have combined a number of IBM services to model a realistic IoT pattern where Internet-enabled devices on trucks are producing data that gets consumed by various applications for different purposes.

Because of the managed nature of all these services, as an application developer, you can focus on solving your business problems and leave us to do the heavy lifting of managing the infrastructure for you, thereby saving potentially months of work on non-core activities like server provisioning, software deployment, patching, security and others.

If you followed this tutorial, remember to de-provision your infrastructure to stop incurring charges. On your terminal, do the following:

cd terraform/
terraform destroy

If you want to take the next step in your developer journey, check out some of our [trial offers](https://www.ibm.com/products?types%5B0%5D=trial).
Relations can be joined by using several types of joins that are described in detail in section [joinType](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencejoinType).

Apart from the join type, the following two different types of joins exist:



* Joins that specify a join condition by using a booleanExpression or a USING clause.
* NATURAL joins that make an implicit guess on which columns to use for joining relations. Use natural joins carefully.







 relationPrimary 





 externalTableSpec 

An external table specification represents an URI for an object that is stored on Cloud Object Storage combined with a specification of the object type. Valid values for object type identifier are AVRO, CSV, JSON, ORC, PARQUET, and TEXT.

If the file format is CSV, with the optional FIELDS TERMINATED BY clause you can specify a field separator other than the default , (comma). The following example shows a query for parsing a CSV with | (vertical bar) as the delimiter:

SELECT 
FROM cos://us-geo/sql/BlackFriday.csv
STORED AS CSV FIELDS TERMINATED BY '|'
LIMIT 3

All single Unicode characters are allowed as delimiters.

By default, it is assumed that CSV input objects have a header line that specifies the names of the input columns. If the objects don't have a header line, you must specify the option NOHEADER in the STORED AS CSV clause. In this case, the names _C0, _C1, ... are used for the input columns. For more information, see [COS URI](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceCOSURI).

By default, if the format of the input data is JSON, each line must contain a separate, self-contained, and valid JSON object, also called newline-delimited JSON.
See your system documentation for specific details on the 'cron' options.

A cron entry that runs a daily backup looks similar to the following example:

0 5 * * * couchbackup --url "https://$USERNAME:$PASSWORD@$ACCOUNT.cloudant.com" --db "animaldb" > /path/to/folder/animaldb-backup-date -u "+%Y-%m-%dT%H:%M:%SZ".bak

This cron entry creates a daily backup at 05:00. You can modify the cron pattern to run hourly, daily, weekly, or monthly backups as needed.







 Using CouchBackup as a library 

The couchbackup and couchrestore command-line tools are wrappers around a library that can be used in your own Node.js applications.

The library is useful for more complicated scenarios, for example:



* Backing up several databases in one task. You might do this backup by identifying all the databases by using the [_all_dbs](https://cloud.ibm.com/apidocs/cloudantgetalldbs) call, then doing a backup of each database individually.
* Longer pipelines increase the risk of errors. By using the CouchBackup library, your application can detect and address any error at the earliest opportunity.



For more information, see the [NPM page](https://www.npmjs.com/package/@cloudant/couchbackup).

The following script sample shows how to combine the couchbackup library with use of IBM® Cloud Object Storage. This code illustrates how you might use Cross Region S3 API to back up a database to an object store.

A prerequisite for the code is that you initialize the S3 client object for IBM Cloud Object Storage by following the instructions in [IBM Cloud Object Storage - S3 API Intro](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-compatibility-api).

/
Backup directly from Cloudant to an S3 bucket via a stream.
@param {string} couchHost - URL of database root
Migrating your service binding 

Let's take a common cloud-native app that follows the 12-factor app principles, simplify and update the code, and create both a Cloud Foundry and a Code Engine version. These versions serve as an example for the discussion of migration steps and deployment aspects.

The sample app is a typical web app that is written in Node.js (JavaScript) and uses the Express web framework. An IBM Cloudant NoSQL database serves as the backing service to store data displayed by the app. As typical for cloud-native, 12-factor apps, the sample app is based on discrete, reusable components that act as microservices to make up the overall app. Both the deployed Node.js program and the database can be scaled, improved, or even replaced independently. The components work together because of how they are configured and by using well-defined APIs.

Zoom

![Simple cloud-native app with backend database service.](https://cloud.ibm.com/docs-content/v1/content/ad5f41087138b4b1bb795e3b5b0661ac5a105cbd/codeengine/images/getting-started-CF2CE.png)

Figure 1. Simple cloud-native app with backend database service.

The IBM Cloudant database can be configured to work as an attached resource with the app versions that are deployed to Cloud Foundry and Code Engine. For simplicity, the solution is kept to these two components.



 Step 1: Creating the service bind 

This sample application requires a Cloudant database to store content. You can configure access to Cloudant by manually injecting environment variables, but the typical process is through [service binding](https://cloud.ibm.com/docs/codeengine?topic=codeengine-service-binding). The relationship between the app and its backing services is explicitly stated, so credentials are created and automatically injected into the runtime environment. Cloud Foundry provides the credentials as part of the VCAP_SERVICES object.
Customers are responsible for the storing, processing, and transmission of their cardholder data, and can create cardholder data environments (CDEs) that can store, transmit, or process cardholder data by using IBM Cloudant. Customers can use the IBM Cloud AOCs and SRM guides when they seek their own PCI DSS certifications. It is the responsibility of the customer to document and operate CDEs and applications that are built by using IBM Cloud Platform services in a PCI DSS-compliant manner.

IBM Cloudant documentation on [service security](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-general-data-protection-regulation-gdpr-service-security) and [deletion of data](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-general-data-protection-regulation-gdpr-deletion-of-data) covers methods to manage cardholder data within the environment in accordance with PCI requirements. It is the customer’s responsibility to familiarize themselves with these processes and to manage data retention and removal from the service according to the customer’s policies. To facilitate this process, no cardholder data can be used in an IBM Cloudant document ID. If PAN data is to be stored in IBM Cloudant, they must be rendered unreadable (in accordance with PCI requirement 3.5) before transmission to the IBM Cloudant service.

A full list of PCI DSS-ready IBM Cloud Platform services and options to request that a PCI DSS AOC and SRM guide can be found at the [IBM Cloud compliance page](https://www.ibm.com/cloud/complianceIndustry%20programs).





 SOC 2 Type 2 Certification 

IBM provides a Service Organization Controls (SOC) 2 Type 2 report for IBM Cloudant. The reports evaluate IBM's operational controls according to the criteria set by the American Institute of Certified Public Accountants (AICPA) Trust Services Principles. The Trust Services Principles define adequate control systems and establish industry standards for service providers such as IBM Cloud to safeguard their customers' data and information.

You can request an SOC 2 Type 2 report from the Customer portal or contact your sales representative.
For instance, the $eq operator matches when the specified field contains a value that is equal to the supplied argument.

The basic equality and inequality operators common to most programming languages are supported. Some "meta" condition operators are also available.

Some condition operators accept any valid JSON content as the argument. Other condition operators require the argument to be in a specific JSON format.



Table 2. Condition operator argument requirements

 Operator type Operator Argument Purpose 

 (In) equality $lt Any JSON The field is less than the argument. 
 $lte Any JSON The field is less than or equal to the argument. 
 $eq Any JSON The field is equal to the argument. 
 $ne Any JSON The field isn't equal to the argument. 
 $gte Any JSON The field is greater than or equal to the argument. 
 $gt Any JSON The field is greater than the argument. 
 Object $exists Boolean Check whether the field exists or not, no matter what its value is. 
 $type String Check the document field's type. Accepted values are null, boolean, number, string, array, and object. 
 Array $in Array of JSON values The document field must exist in the list provided. 
 $nin Array of JSON values The document field must not exist in the list provided. 
 $size Integer Special condition to match the length of an array field in a document. Non-array fields can't match this condition. 
 Miscellaneous $mod [Divisor, Remainder] Divisor and Remainder are both positive or negative integers. Non-integer values result in a [404 status](https://cloud.ibm.com/apidocs/cloudantlist-of-http-codes). Matches documents where the expression (field % Divisor == Remainder) is true, and only when the document field is an integer. 
 $regex String A regular expression pattern to match against the document field. Matches only when the field is a string value and matches the supplied regular expression. 



Regular expressions don't work with indexes, so they must not be used to filter large data sets. However, they can be used to restrict a partial index <find/partial_indexes>.
A nonce is an arbitrary number that can be used just once in a cryptographic communication.

 create an import token that expires in 5 minutes and allows 10 retrievals
$ ibmcloud kp import-token create -e 300 -m 10

 create a random, base64-encoded, 32-byte key material
$ KEY_MATERIAL=$(openssl rand -base64 32)

 extract the nonce that was created by the "kp import-token create" command
$ NONCE=$(ibmcloud kp import-token show | jq -r '.["nonce"]')

 encrypt the nonce
$ ibmcloud kp import-token nonce-encrypt -k $KEY_MATERIAL -n $NONCE

Encrypted Nonce IV
mWQad1RHdWoFXFw/D9h8z43t/+0vIZc55VBBQg== 6nvOwUvQdowoD+3v





 Required parameters 



* -i, --instance-id

The IBM Cloud instance ID that identifies your Key Protect instance.

You can set an environment variable instead of specifying -i with the following command: $ export KP_INSTANCE_ID=<INSTANCE_ID>.
* -k, --key

The base64-encoded key material that you store and manage in the service.

-n, --nonce

The nonce that is used to verify a request to import a key. The value is base64-encoded. To retrieve a nonce, use ibmcloud kp import-token show.







 Optional parameters 



* -c, --cbc

Encrypt the nonce using the AES-CBC encryption algorithm. This algorithm method is only supported for HPCS. The default encryption algorithm is AES-GCM.

This option is used to generate an encrypted nonce that can only be used in a request with the HPCS API. Do not use this option to create Key Protect root keys.

Hyper Protect Crypto Services (HPCS) is a dedicated key management service and hardware security module (HSM).
In this example, only hardback books that have a status of "published" or "reprint" make it to the index. See the following example:

{
"index": {
"partial_filter_selector": {
"$or": [
{ "status": "published" },
{ "status": "reprint" }
],
"hardback": true
},
"fields": ["firstname","surname","date"]
},
"ddoc" : "partialindexes",
"name": "byNameAndDate"
"type" : "json"
}

At query time, the use_index field must be supplied to tell IBM Cloudant that you want it to use the specified index:

{
"selector": {
"$or": [
{ "status": "published" },
{ "status": "reprint" }
],
"hardback": true,
"firstname": "Charles",
"surname": "Dickens"
},
"sort": ["date"],
"limit": 10,
"execution_stats": true,
"use_index": "partialindexes/byNameAndDate"
}

The partial_filter_selector fields are repeated in the query-time selector.

For more information, see [Creating a partial index](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-querycreating-a-partial-index).





 Can I use IBM Cloudant Query on partitioned databases? 

Yes! By default, an IBM Cloudant Query index is global (for example, not partitioned) but on a partitioned database, a partitioned index can be created by adding partitioned: true when the index is created: e To create a partitioned index on firstname and surname, the POST /<dbname>/_index is used:

{
"index": {
"fields": ["firstname", "surname"]
},
* If the CSV file has headers, the header names are used to name the fields in which the content from the corresponding column is stored. Do not use names that have special meaning in Discovery. Be sure that the field names conform to the naming rules, such as having no spaces and no appended numbers. For example, you can rename the start date header to start_date and label1 to label-one before you add the file. For more information, see [How fields are handled](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-index-overviewfield-name-limits).
* When a CSV file header name contains restricted characters, the document converter automatically removes the restricted characters from the field name when it adds the resulting field to the index.



Note about enhancing data:



* You cannot apply prebuilt or user-trained Smart Document Understanding models to CSV files.







 HTML files 

If you upload an HTML file or crawl a data source with HTML files, such as a website, an html field is generated along with the text field. For more information, see [HTML fields](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-index-overviewfield-html).





 JSON files 

Notes about adding data:



* Object names from the source JSON file are used to name the fields in which the content is stored. Do not use names that have special meaning in Discovery. Be sure that the names conform to the naming rules, such as having no spaces and no appended numbers. For example, you can rename the updated on object to updated_on and answer2 to answer-two before you add the file. For more information, see [How fields are handled](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-index-overviewfield-name-limits).
* If a root-level field is an array but contains no items, the field is omitted from the index.
* If a root-level field is an array and contains only one item, the array is indexed as the data type of the one item.
For more information, see [Delete a document](https://cloud.ibm.com/apidocs/cloudantdeletedocument) in the API Reference documentation.



 Deleting IBM Cloudant instances 

You can delete a database instance in the IBM Cloudant Dashboard or by using an API.

Once an instance is deleted, all data within the database, as well as the account-level information, such as authentication data, is deleted automatically after the 7-day grace period ends. IBM Cloudant doesn’t hold any contact details for the instances that are created by using the platform. If you have support tickets with IBM Cloudant where you shared information, such as email addresses, that information isn’t removed by this process.

To delete a database, follow these steps:



1. Go to IBM Cloudant Dashboard.
2. On the Databases page, click Delete next to the database you want to delete.
3. Type in the name of the database you want to delete.
4. Click Delete Database.



The database is removed from the list of databases.

A database deletion cannot be undone.

For more information, see [Delete a database](https://cloud.ibm.com/apidocs/cloudantdeletedatabase) in the API Reference documentation.

The IBM Cloudant data retention policy describes how long your data is stored after you delete the service. The data retention policy is included in the IBM Cloudant service description, which you can find in the IBM Cloud Terms and Notices.





 Restoring deleted data for IBM Cloudant 

If you delete your account, you have a 7-day grace period during which you can cancel the request to delete it.

A database deletion cannot be undone.