vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Getting Started 

With IBM Cloud Databases (ICD) enabled by IBM Cloud Satellite, you can deploy ICD instances into a Satellite location. (ICD) enabled by IBM Cloud Satellite currently supports the following ICD managed database services:



* [Databases for etcd](https://cloud.ibm.com/docs/databases-for-etcd)
* [Databases for PostgreSQL](https://cloud.ibm.com/docs/databases-for-postgresql)
* [Databases for Redis](https://cloud.ibm.com/docs/databases-for-redis)
* [Messages for RabbitMQ](https://cloud.ibm.com/docs/messages-for-rabbitmq)



After your Satellite location instance is deployed, ICD will install an ICD Satellite service cluster in your Satellite location into which your database instances will be deployed.

Your ICD service cluster can operate multiple database instances, even with different database types. For instance, you might operate a Databases for etcd and a Databases for PostgreSQL instance on the same service cluster.

The service cluster name is based on the first database instance, even if multiple instances exist. For example, if your service cluster has both a Databases for etcd and a Databases for PostgreSQL instance on the same service cluster, the name is based on your Databases for etcd instance.

ICD enabled by IBM Cloud Satellite supports Satellite locations on [Amazon Web Services (AWS)](https://cloud.ibm.com/docs/satellite?topic=satellite-aws) and [on-premises](https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-satellite-on-prem).



 An Overview 

Zoom

![An Overview of ICD enabled by IBM Cloud Satellite](https://cloud.ibm.com/docs-content/v1/content/4dfeb7bbbdd817dfbdb17402735fb55d9cfbc7fb/messages-for-rabbitmq/includes/cloud-databases/includes/cloud-databases/images/sat_on_icd.svg)
Learning Center 

The IBM® Cloudant® for IBM Cloud® Learning Center offers a video series to help you learn to use IBM Cloudant. The videos start with the basics of using IBM Cloudant. Then the videos walk you through document structure, the API, indexing and querying, and include an Under the Hood topic that highlights the architecture that powers the service.

You can use the [playlist](https://www.youtube.com/embed/playlist?list=PLzpeuWUENMK3F93hGaS4ezGmlX4Bipt4S) to go through the courses, or navigate directly to the topic of your choosing.



 Introduction to IBM Cloudant video 

Learn about the IBM Cloudant 17-part video series that provides an overview of the IBM Cloudant database-as-a-service.



* Introduction to IBM Cloudant video script

Welcome to the Introduction to IBM Cloudant course, an 17-part video series that gives you an overview of the IBM Cloudant database-as-a-service.

This video is part 1 - What is IBM Cloudant?

IBM Cloudant is a database, run as a service in the IBM Cloud®. Its job is to store your application's data securely and make it possible for you to retrieve it quickly and efficiently. IBM Cloudant's key features are shown in the following list:

Database
: Stores and retrieves data. More specifically, it is a JSON document store. JSON comes from JavaScript and represents simple objects in a universal file format.

Document
: The unit of storage in IBM Cloudant. Documents are added, updated, and deleted in their entirety.

HTTP API
: Any IBM Cloudant operation can be achieved by using HTTPS. HTTP is the protocol that powers the World Wide Web and IBM Cloudant is a database that is built for the web. Most databases are hidden in a private network, inaccessible but to a handful of machines. The IBM Cloudant service sits (mainly) on the public internet where it can be accessed by anyone with an internet connection (and permission to do so).

IBM Cloudant wasn't written entirely by IBM.
Deploying the built-in UI components 

For Document Retrieval and custom projects, a set of user interface components are available for your use.

Work with a developer to use the pre-built UI components that are provided by IBM to deploy an application.

For more information about building your own app, see the [Building custom applications with the API](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-api-use).

Several built-in UI components are available.



 Search bar 

A search box that uses a natural language understanding query to fetch the most relevant results.

Zoom

![Search bar type ahead](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/search-bar-example.png)

Figure 1. Search bar type ahead

[Try it](https://watson-developer-cloud.github.io/discovery-components/storybook/?path=/story/searchinput--default)





 Search results 

A set of results that rank the most relevant passages and tables to a query.

Zoom

![Search results list](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/search-results-example.png)

Figure 2. Search results list

[Try it](https://watson-developer-cloud.github.io/discovery-components/storybook/?path=/story/searchresults--default)





 Facets 

Refine your results with facets that help users filter the search results by specific categories and domains.

Zoom

!
<br>Database names must start with a lowercase letter and contain only the following characters: lowercase characters (a-z), digits (0-9), any of the characters _, $, (, ), +, -, and /. 
 DELETE /{db} <br>[Cloudant](https://cloud.ibm.com/apidocs/cloudantdeletedatabase), [CouchDB](https://docs.couchdb.org/en/latest/api/database/common.html?highlight=head--dbdelete--db) Delete the specified database, and all the documents and attachments contained within it. 
 GET /{db}/_all_docs <br>[Cloudant](https://cloud.ibm.com/apidocs/cloudantgetalldbs), [CouchDB](https://docs.couchdb.org/en/latest/api/database/bulk-api.html?highlight=_all_docsget--db-_all_docs) Query a list of all documents in a database. <br>CouchDB documentation references more query parameters (end_key, endkey_docid, end_key_doc_id, stale, start_key, startkey_docid, start_key_doc_id, update_seq). For IBM Cloudant, use GET _all_docs. 
 POST /{db}/_all_docs <br>[Cloudant](https://cloud.ibm.com/apidocs/cloudantpostalldocs), [CouchDB](https://docs.couchdb.org/en/latest/api/database/bulk-api.html?highlight=_all_docspost--db-_all_docs) Query a list of all documents in a database by using the HTTP POST method where the query is submitted as a JSON object in the request body. <br>CouchDB documentation references more query parameters (end_key, endkey_docid, end_key_doc_id, stale, start_key, startkey_docid, start_key_doc_id, update_seq).
IBM Cloudant doesn't have the concept of joins like a relational database, so data isn't normalized. However, data can repeat across objects. For example, an order document can include a subset of the product documents that were purchased.

It's common to store several object types in the same database: a convention is that a type attribute is used to denote the object type. This option is a good one if you need to perform queries that return several object types or if a database needs to be replicated to another location altogether. Otherwise, separate databases, for example, users, orders, products, might be better so that secondary indexes are specific to each object type.

If you're storing arrays of objects within a document, consider whether the array items must really be their own document. For example, a product and each product review must be stored in separate documents, but a user and each of that user's orders must have their own document.

If you have an ever-growing data set, then you probably don't want to store data in a single, ever-growing database. Data is best stored in time-boxed databases that allow older data to be archived and deleted cleanly. Deleting an IBM Cloudant document leaves a tombstone document behind, so don't rely on deleting documents to recover disk space. Instead, you must rely on deleting whole databases.

JSON doesn't offer a native way to store dates or timestamps. Choose your [date format](https://blog.cloudant.com/2018/05/24/Date-formats.html) carefully if you intend to query it later.

The maximum document size is 1 MB, but documents must be much smaller than that size, typically a few KB.

For more information, see the following blog posts:



* [Optimal IBM Cloudant Indexing](https://blog.cloudant.com/2019/05/10/Optimal-Cloudant-Indexing.html)
* [Time-series Data Storage](https://blog.cloudant.com/2019/04/08/Time-series-data-storage.html)
* The similar parameter of the Query method
* The smart_document_understanding field in the Get collection method response







 15 August 2022 

SDKs were updated to reflect the latest API changes.
: The following [Discovery v2 API](https://cloud.ibm.com/apidocs/discovery-data) changes are now reflected in the SDKs:



* Use the new document classifier API to get, add, update, or delete a document classifier.
* A new document status API is available. You can use it to get a list of the documents in a collection and to get details about a single document.
* You can now get, add, and remove a stop words or expansion list for a collection.
* The suggested_refinements parameter of the Query method is deprecated.







 4.5.1 release, 3 August 2022 

IBM Watson® Discovery Cartridge for IBM Cloud Pak® for Data 4.5.1 is available.

There are no new features in this release. For a list of bug fixes, see [What's new and changed in Watson Discovery](https://www.ibm.com/docs/SSQNUZ_4.5.x/fixlist/discovery-fixlist.html)

Features that are not available in this release
: The following features are generally available from managed IBM Cloud deployments at the time of this release, but not from installed deployments:



* Answer finding
* Manage data page
* Advanced document view for search results
* The similar parameter of the Query method
* The smart_document_understanding field in the Get collection method response







 4.5 release, 29 June 2022 

IBM Watson® Discovery Cartridge for IBM Cloud Pak® for Data 4.5 is available.

For a list of new features and bug fixes, see [What's new and changed in Watson Discovery](https://www.ibm.com/docs/SSQNUZ_4.5.x/fixlist/discovery-fixlist.html)

Features that are not available in this release
: The following features are generally available from managed IBM Cloud deployments at the time of this release, but not from installed deployments:



* Answer finding
[Help icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/help.svg) from the header of any page in the product user interface to open the Discovery documentation.





 Browser support 

IBM Cloud Pak for Data



* The minimum required browser software for the product user interface includes the following browsers:

Google Chrome
: Latest version -1 for your operating system

Mozilla Firefox
: Latest regular -1 and Extended Support Release (ESR) version for your operating system

Microsoft Edge
: Latest version -1 for Windows

Apple Safari
: Latest version -1 for Mac
* The IBM Cloud Pak for Data web client where you create service instances supports the IBM Cloud Pak for Data requirements. For more information, see [Supported web browsers](https://www.ibm.com/docs/en/cloud-paks/cp-data/4.6.x?topic=requirements-softwaresoftware-reqs__web)



IBM Cloud



* Deployments of Discovery that are managed by IBM Cloud follow the IBM Cloud requirements. For more information, see [Prerequisites](https://cloud.ibm.com/docs/overview?topic=overview-prereqs-platform)
* For more information about browser support for deployments that are provisioned with Cloud Pak for Data as a Service, see [Which web browsers are supported for Cloud Pak for Data as a Service](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/browser-support.html).







 Language support 

Language support by feature is detailed in the [Supported languages](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-language-support) topic.





 Beta features 

IBM releases services, features, and language support for your evaluation that are classified as beta. These features might be unstable, might change frequently, and might be discontinued with short notice.
Digging deeper into IBM Cloudant Dashboard 

The IBM® Cloudant® for IBM Cloud® Dashboard gives new and experienced IBM Cloudant users the opportunity to add, edit, and delete documents. The IBM Cloudant users can refine the indexing and querying options that best suit their application's use-cases.



 Objectives 

Set up some basic indexes using the Dashboard to see how each of IBM Cloudant's querying mechanisms works.





 Before you begin 

You need to create a service instance in IBM Cloudant before you start this tutorial. You can follow the instructions in the [Getting started](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-getting-started-with-cloudant) tutorial to create one.





 Step 1. The data set 



1. Create a database called books.
2. Create some sample data that represents a book in a library as shown in the following example:

{
"_id": "BXP9G5ZQY9Q4EA13",
"author": "Dickens",
"title": "David Copperfield",
"year": 1840,
"pages": 723,
"publisher": "Penguin",
"url": "https://www.somurl.com/dc"
}
3. Continue to add some documents that match the pattern in the previous step by using the IBM Cloudant Dashboard.

The documents store simple key/value pairs that hold metadata about each book: its author and its publisher. In this example, we address the following three use-cases:



1. A query facility that allows a user to find a book by a known publisher and year.
2. A general-purpose search engine that allows a user to find books by a combination of one or more of the following descriptors: author, title, year, and publisher.
3. A report that details the number of books that are published by year.









 Step 2. Querying books by publisher and year - IBM Cloudant Query
Adding content with the Data Crawler 

The Data Crawler is no longer supported or available for download beginning [17 April 2019](https://cloud.ibm.com/docs/discovery?topic=discovery-release-notes17apr19). This content is provided for existing installations only. See [Connecting to Data Sources](https://cloud.ibm.com/docs/discovery?topic=discovery-sources) for other available connectivity options.

The data crawler lets you automate the upload of content to the Discovery Service.



 Crawling data with the Data Crawler 

The Data Crawler is a command line tool that helps you take your documents from the repositories where they reside (for example: file shares, databases) and push them to the cloud, to be used by Discovery.





 When to use the Data Crawler 

Use the Data Crawler if you want to have a managed upload of a significant number of files from a remote system or if you want to extract content from a supported repository, such as a DB2 database.

The Data Crawler is not intended to be a solution for uploading files from your local drive. If you upload files from a local drive, use the tooling or direct API calls. Another option to upload large numbers of files into Discovery is [discovery-files](https://github.com/IBM/discovery-files) on GitHub.





 Using the Data Crawler 



1. [Configure Discovery](https://cloud.ibm.com/docs/discovery?topic=discovery-configservice).
2. [Download and install the Data Crawler](https://cloud.ibm.com/docs/discovery?topic=discovery-downloading-and-installing-the-data-crawler) on a supported Linux system that has access to the content that you want to crawl.
3. [Connect the Data Crawler](https://cloud.ibm.com/docs/discovery?topic=discovery-configuring-connector-and-seed-options) to your content.
4.
The number of documents that are allowed per service instance depends on your Discovery plan type.

The document limit applies to the number of documents in the index. Upload fewer documents at the start if the enrichments that you plan to apply might increase the number of documents later. For example, the following configurations generate more documents:



* Splitting documents segments one document into multiple documents
* CSV files that you upload generate one document per line
* Database data sources that you crawl produce one document per database row
* Each object that is defined in an array in a JSON file results in a separate document





Number of documents per service instance

 Plan Documents per service instance 

 Cloud Pak for Data Unlimited 
 Premium Unlimited 
 Enterprise Unlimited 
 Plus (includes Trial) 500,000 



The maximum allowed number can vary slightly depending on the size of the documents. Use these values as a general guideline.





 File size limits 



 Crawled documents 

The maximum size of each file that you can crawl by using a connector differs by deployment type.

IBM Cloud

Managed deployments on IBM Cloud



* Premium plans only:



* Box: 50 MB
* IBM Cloud Object Store: 50 MB
* Salesforce Files objects: 50 MB
* All other data sources: 10 MB



* All other plans: 10 MB



IBM Cloud Pak for Data

Installed deployments on IBM Cloud Pak for Data



* All data sources: 32 MB







 Uploaded documents 

The size of each file that you can upload depends on your Discovery plan type. See the *Maximum document size table for details.



Maximum document size

 Plan File size per document 

 Cloud Pak for Data 50 MB 
 Premium 50 MB 
 Enterprise 10 MB 
 Plus (includes Trial) 10 MB 









 Field limits 

When a document is added to a collection, content from the document is evaluated and added to the appropriate fields in an internal index.

For structured data, such as uploaded CSV or JSON files, or data from crawled databases, each column or object is stored as a root-level field. For example, if you add a CSV file to collection, each column in the CSV file is stored as a separate field in the index.
[:](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-operatorsexists) Exists title:* 
 [!](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-operatorsdnexist) Does not exist title!* 







 Aggregations 

Aggregations return a set of data values. The following table lists the available aggregations.



Query aggregations

 Aggregation Description Example 

 [average](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-aggregationsaverage) Mean value for the specified field in the results set. average(product.price) 
 [filter](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-aggregationsaggfilter) Filter results based on the specified pattern filter(enriched_text.concepts.text:cloud computing) 
 [histogram](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-aggregationshistogram) Interval-based distribution histogram(product.price,interval:1) 
 [max](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-aggregationsmax) Maximum value for the specified field in the results set. max(product.price) 
 [min](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-aggregationsmin) Minimum value for the specified field in the results set. min(product.price) 
 [nested](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-aggregationsnested) Restrict aggregation nested(enriched_text.entities) 
 [sum](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-aggregationssum) Sum of all fields in the results set. sum(product.price)
[Architecture](https://cloud.ibm.com/docs-content/v1/content/f84e2238288b7f0355715a4c355609ce7ac48fb0/solution-tutorials/images/solution64-serverless-webapp/architecture-serverless-api-webapp.svg)

Figure 1. Architecture diagram of the tutorial



1. The user accesses the application hosted on the bucket in Object Storage
2. The web application calls a backend API.
3. The app with the backend API is deployed to Code Engine.
4. The backend uses IBM Cloudant to store and retrieve guestbook entries.







 Step 1: Create the Guestbook database 

Let's start by creating a [IBM Cloudant](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-getting-started-with-cloudant) service instance. IBM Cloudant is a fully managed JSON document database. It is built upon and compatible with Apache CouchDB.



1. In the [Catalog](https://cloud.ibm.com/catalog?category=databasesservices), under Services, go to the Databases category. Click on the IBM Cloudant tile. In the new dialog:



1. Under Multitenant select a region.
2. Under Configure Cloudant instance pick a unique name for the service, such as <yourinitials>-guestbook-db.
3. Select a resource group.
4. Select IAM as authentication method.
5. Select the Lite plan. If you already have a Lite plan in your account, select another service plan.
6. Click Create.



2. Back in the [IBM Cloud Resource List](https://cloud.ibm.com/resources/), under Services, click on the IBM Cloudant instance you created to open the instance full details page. Note: You may be required to wait until the status of the service changes to Active.
3.
For Redis](https://cloud.ibm.com/docs/databases-for-redis?topic=databases-for-redis-getting-started)Databases For Redis Service[Databases For etcd](https://cloud.ibm.com/docs/databases-for-etcd?topic=databases-for-etcd-getting-started)Databases For etcd Service[Db2 Warehouse on Cloud](https://cloud.ibm.com/docs/Db2whc?topic=Db2whc-getting-started)Db2 Warehouse on Cloud Service[Db2 on Cloud](https://cloud.ibm.com/docs/Db2onCloud)Db2 on Cloud Service[IBM Cloud Databases](https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-about)IBM Cloud Databases Service[IBM Watson Query](https://cloud.ibm.com/docs/data-virtualization?topic=data-virtualization-getting-started)IBM Watson Query Service[IBM watsonx.data](https://cloud.ibm.com/docs/watsonxdata)IBM watsonx.data Service[Informix on Cloud](https://cloud.ibm.com/docs/services/InformixOnCloud/InformixOnCloud.html)Informix on Cloud Service[Messages For RabbitMQ](https://cloud.ibm.com/docs/messages-for-rabbitmq?topic=messages-for-rabbitmq-getting-started)Messages For RabbitMQ Service[Watson Query CLI](
[IBM Developer<br><br>Visit IBM Developer for technical articles, code patterns, tutorials, and more.<br><br>![112-Developer-tools.svg](https://cloud.ibm.com/media/docs/images/homepage/112-Developer-tools.svg)](https://developer.ibm.com/depmodels/cloud/)[Architecture Center<br><br>Discover the architecture references available for this product.<br><br>![190-Application-Platform.svg](https://cloud.ibm.com/media/docs/images/homepage/190-Application-Platform.svg)](https://www.ibm.com/cloud/architecture)[IBM Training<br><br>Build your skills with IBM Cloud training.<br><br>![83-Elements-of-the-cloud.svg](https://cloud.ibm.com/media/docs/images/homepage/83-Elements-of-the-cloud.svg)](https://www.ibm.com/training/cloud)[Community<br><br>Join the IBM Cloud community and collaborate with others.<br><br>![35-Connectivity.svg](https://cloud.ibm.com/media/docs/images/homepage/35-Connectivity.svg)](https://community.ibm.com/community/user/publiccloud/home)

 Latest updates 

[View more](https://cloud.ibm.com/docs/databases-for-postgresql?topic=databases-for-postgresql-postgresql-relnotes)28 October 2022 | IBM Cloud® Databases for PostgreSQL v10: End-of-Life Extension

To ensure additional time for you to upgrade, Cloud Databases is extending the EOL support for PostgreSQL v10 until February 22, 2023.

19 October 2022 | Deploying and Connecting an IBM Cloud® Databases Instance Tutorial