vania-janet/multiturn-rag-retrieval-data huggingface dataset:
* Crawl, convert, enrich and normalize data.
* Securely explore your proprietary content as well as free and licensed public content.
* Apply additional enrichments such as concepts, relations, and sentiment through Natural Language Understanding (NLU).
* Simplify development while still providing direct access to APIs.



To try out Discovery, see the [IBM Watson™ Discovery Query Demo](https://www.ibm.com/demos/live/watson-discovery/self-service/home).

You can use Smart Document Understanding (SDU) to train IBM Watson™ Discovery to extract custom fields in your documents. See [Smart Document Understanding](https://cloud.ibm.com/docs/discovery?topic=discovery-sdu) for more information.

If you create a chatbot in IBM Watson™ Assistant, you can route complex customer inquiries to Discovery using a search skill. When a customer asks a question that the dialog is not designed to answer, your assistant can search for relevant information from Discovery, extract the information, and return it as the response. See [Creating a search skill](https://cloud.ibm.com/docs/services/assistant?topic=assistant-skill-search-add) for details. (This feature is available only to Watson Assistant Plus or Premium plan users.)

For information about language support, see [Discovery language support](https://cloud.ibm.com/docs/discovery?topic=discovery-language-support).

For information about IBM Cloud security, see the [IBM Cloud Service Description](https://www.ibm.com/software/sla/sladb.nsf/searchsaas/?searchview&searchorder=4&searchmax=0&query=%28IBM+Cloud+Service+description%29)

US Health Insurance Portability and Accountability Act (HIPAA) support is available for Premium plans in the Washington, DC location created on or after 1 April 2019. See [Enabling EU and HIPAA supported settings](https://cloud.ibm.com/docs/account?topic=account-eu-hipaa-supportedeu-hipaa-supported) for more information.



 Browser support and prerequisites
* If you plan to use Databases for MongoDB for your applications, check out [Connecting an external application](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-mongodb-external-app), [Connecting an IBM Cloud application](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-mongodb-connecting-ibmcloud-app), and [MongoDB Node.js Driver Connection Guide](https://docs.mongodb.com/drivers/node/current/fundamentals/connection/).
* For information on TLS/SSL certificate configuration in the API, review [Driver TLS and self-signed certificate support](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-mongodb-external-appmongodb-tls-certificate-support), [Using the self-signed certificate in mongo Shell](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-connecting-cli-clientconnecting-cli-client-cert), and [MongoDB TLS/SSL Configuration for Clients](https://docs.mongodb.com/manual/tutorial/configure-ssl-clients/).
* To ensure the stability of your applications and databases, check out[High-Availability](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-high-availability) and [Performance](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-performance).
Documentation![sorry, search content isn't available](https://cloud.ibm.com/media/docs/images/icons/search.svg)Sorry, this content isn’t available.It seems we can’t find what you’re looking for. Try the IBM Cloud Docs search to find information.
The Query is a standard JavaScript object, which is passed to the db.find function which it POSTs to the _find endpoint on your behalf.

Now, time for a practical exercise. Devise your own IBM Cloudant Query that finds the titles of books that are written in the 20th Century. The IBM Cloudant Query documentation is at the on-screen URL if you need it.

Pause the presentation here if you don't want to know the answer...

See one solution:

I use the $and operator to combine two clauses on the date attribute. One clause to locate documents whose date >= 1900, the other to find documents whose date is < the year 2000. Both clauses have to be true to select a document. As we need only the title of the matching books, we can supply a fields attribute instead of being returned the entire document.

To summarize, IBM Cloudant Query is a query language that is inspired by MongoDB where the syntax is expressed in JSON form.

Queries select subsets of documents from the database by using clauses that operate on data inside the document - not just the document's _id.

Queries are sent to the database's _find endpoint, either programmatically, by using curl, or by using the Dashboard.

The query's selector decides which cut of data is required,

That's the end of this part. The next part is called Indexing.







 Indexing video 

Learn how indexing can speed up your query process.



* Indexing video script

Welcome to the Introduction to IBM Cloudant course, an 17-part video series that gives you an overview of the IBM Cloudant database-as-a-service.

This video is part 11 - Indexing.

The queries that we executed in the previous part were not optimal: to get the answer, IBM Cloudant had to spool through every document in the database in turn to see whether it met with the search criteria.

To make queries that are run in a performant and scalable way, we need Indexing.

With IBM Cloudant, you can specify any number of Indexes (or indices).

An index is a secondary data structure that is built from the document list.
Getting documents 

To list all the documents in a database, send a GET request to https://$ACCOUNT.cloudant.com/$DATABASE/_all_docs.

The _all_docs endpoint accepts the following query string and JSON body arguments:



Table 1. Query string and JSON body arguments

 Argument Description Optional Type Default 

 conflicts Can be set only if include_docs is true. Adds information about conflicts to each document. Yes Boolean False 
 deleted_conflicts Returns information about deleted conflicted revisions. Yes Boolean False 
 descending Return the documents in descending key order. Yes Boolean False 
 endkey Stop returning records when the specified key is reached. Yes String 
 endkey_docid Stop returning records when the specified document ID is reached. If endkey isn't set, this argument is ignored. Yes String 
 include_docs Include the full content of the documents in the return. Yes Boolean False 
 inclusive_end Include rows whose key equals the "endkey" value. Yes Boolean True 
 key Return only documents with IDs that match the specified key. Yes String 
 keys Return only documents with IDs that match one of the specified keys. Yes List of strings 
 limit Limit the number of returned documents to the specified number. Yes Numeric 
 meta Short-hand combination of the following three arguments: conflicts, deleted_conflicts, and revs_info. Using meta=true is the same as using conflicts=true&deleted_conflicts=true&revs_info=true. Yes Boolean False 
 r Specify the [read quorum](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-documentsquorum-writing-and-reading-data) value. Yes Numeric 2 
 revs_info Includes detailed information for all known document revisions. Yes Boolean False 
 skip Skip this number of records before returning the results. Yes Numeric 0 
 startkey Return records, starting with the specified key. Yes String 
 startkey_docid Return records, starting with the specified document ID. If startkey isn't set, this argument is ignored.
You can use an IBM App Connect default connector to send data from a large set of popular data sources to Discovery by creating flows within the App Connect tooling. Note that creating a separate App Connect instance is required to use this App Connect default connector and that any costs that you incur when you use a paid App Connect instance are not included with the cost for using Discovery. Additionally, except for indexing, Discovery does not support any integration with App Connect that you perform on your own. For information about integrating App Connect with Discovery or for integration support or questions, see [How to use IBM App Connect with IBM Watson™ Discovery](https://www.ibm.com/support/knowledgecenter/SS6KM6/com.ibm.appconnect.dev.doc/how-to-guides-for-apps/watson-discovery.html). For the available data sources that you can use with the App Connect default connector to send data to Discovery, see [Connectors A-Z](https://www.ibm.com/cloud/app-connect/connectors/).



 Adding content with the API or tooling 

Consider the following when you are ready to add documents to your collection:



* The maximum file size that can be uploaded to Discovery is 50MB.
* Only the first 50,000 characters of each JSON field selected for enrichment are enriched.
* A maximum of 1,000 fields can be added to the document index.
* When creating a collection, you select the document language (English is the default). See [Language support](https://cloud.ibm.com/docs/discovery?topic=discovery-language-support) for the list of languages. Your documents are enriched in the selected language. Do not mix languages within the same collection.
* The following file types can be ingested by Discovery, all other document types are ignored:





 Lite plans Advanced plans 

 PDF, Word, PowerPoint, Excel, JSON*, HTML* PDF, Word, PowerPoint, Excel, PNG**, TIFF**, JPG**, JSON*, HTML* 



* JSON and HTML documents are supported by IBM Watson™ Discovery, but can not be edited using the SDU editor.
GET /{db}/_changes <br>[Cloudant](https://cloud.ibm.com/apidocs/cloudantgetchanges-databases), [CouchDB](https://docs.couchdb.org/en/latest/api/database/changes.htmlget--db-_changes) List of changes that are made to documents in the database, including insertions, updates, and deletions. <br>CouchDB includes query parameters (attachments, att_encoding_info, last-event-id, view). 
 POST /{db}/_changes <br>[Cloudant](https://cloud.ibm.com/apidocs/cloudantpostchanges-databases), [CouchDB](https://docs.couchdb.org/en/latest/api/database/changes.htmlpost--db-_changes) List of changes that are made to documents in the database, including insertions, updates, and deletions. 
 POST /{db}/_compact <br>[CouchDB](https://docs.couchdb.org/en/latest/api/database/compact.htmlpost--db-_compact) Request compaction of the specified database. 
 POST /{db}/_compact/{ddoc} <br>[CouchDB](https://docs.couchdb.org/en/latest/api/database/compact.htmlpost--db-_compact-ddoc) Compact the view indexes associated with the specified design document. 
 HEAD /{db}/_design/{ddoc} <br>[Cloudant](https://cloud.ibm.com/apidocs/cloudantheaddesigndocument), [CouchDB](https://docs.couchdb.org/en/latest/api/ddoc/common.html?highlight=_designhead--db-_design-ddoc) Retrieve the HTTP headers that contain a minimal amount of information about the specified design document.
Getting Started 

With IBM Cloud Databases (ICD) enabled by IBM Cloud Satellite, you can deploy ICD instances into a Satellite location. (ICD) enabled by IBM Cloud Satellite currently supports the following ICD managed database services:



* [Databases for etcd](https://cloud.ibm.com/docs/databases-for-etcd)
* [Databases for PostgreSQL](https://cloud.ibm.com/docs/databases-for-postgresql)
* [Databases for Redis](https://cloud.ibm.com/docs/databases-for-redis)
* [Messages for RabbitMQ](https://cloud.ibm.com/docs/messages-for-rabbitmq)



After your Satellite location instance is deployed, ICD will install an ICD Satellite service cluster in your Satellite location into which your database instances will be deployed.

Your ICD service cluster can operate multiple database instances, even with different database types. For instance, you might operate a Databases for etcd and a Databases for PostgreSQL instance on the same service cluster.

The service cluster name is based on the first database instance, even if multiple instances exist. For example, if your service cluster has both a Databases for etcd and a Databases for PostgreSQL instance on the same service cluster, the name is based on your Databases for etcd instance.

ICD enabled by IBM Cloud Satellite supports Satellite locations on [Amazon Web Services (AWS)](https://cloud.ibm.com/docs/satellite?topic=satellite-aws) and [on-premises](https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-satellite-on-prem).



 An Overview 

Zoom

![An Overview of ICD enabled by IBM Cloud Satellite](https://cloud.ibm.com/docs-content/v1/content/fcee192e52fce225776143bc6ccc3a023d1d193a/databases-for-etcd/includes/cloud-databases/includes/cloud-databases/images/sat_on_icd.svg)
* URL



Click Next.
5. Name the collection.
6. If the language of the documents on the site is not English, select the appropriate language.

For a list of supported languages, see [Language support](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-language-support).
7. Optional: Change the synchronization schedule.

For more information, see [Crawl schedule options](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionscrawlschedule).
8. Select the objects that you want to crawl.

The more objects that you select, the longer the processing of the documents takes.
9. If you want to limit the types of files to add to the collection, you can list the file extensions for file types to either include or exclude.

When you choose to list extensions for file types to exclude, you must add at least one file extension.

For a list of supported file types, see [Supported file types](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionssupportedfiletypes).
10. If you want the crawler to extract text from images on the site, expand More processing settings, and set Apply optical character recognition (OCR) to On.

When OCR is enabled and your documents contain images, processing takes longer. For more information, see [Optical character recognition](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionsocr).
11. Click Finish.



The collection is created quickly. It takes more time for the data to be processed as it is added to the collection.

If you want to check the progress, go to the Activity page. From the navigation pane, click Manage collections, and then click to open the collection.
* If no documents are listed after a few minutes and a page refresh, then make sure that the content you want to ingest is available from the URL's page source. Some web page content is dynamically generated and therefore cannot be crawled.



* Configuring search results for uploaded documents: If you are using a collection of uploaded documents and cannot get the correct search results or the results are not concise enough, consider using Smart Document Understanding when you create the data collection.

This feature enables you to annotate documents based on text formatting. For example, you can teach Discovery that any text in 28-point bold font is a document title. If you apply this information to the collection when you ingest it, you can later use the title field as the source for the title section of your search result.

You can also use Smart Document Understanding to split up large documents into segments that are easier to search. For more information, see the the [Using Smart Document Understanding](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-configuring-fields) topic in the Discovery documentation.
* Improve search results: If you don't like the results you are seeing, call the search skill from a dialog node, and specify filter details.

From a dialog node search skill response, you can specify a full Discovery query syntax filter to help narrow the results.

For example, you can define a filter that filters out any documents in the data collection that do not mention an intent in the document title or some other metadata field. Or the filter can filter out documents that do not identify an entity as a known entity in the data collection's metadata or that don't mention the entity anywhere in the full text of the document. For details about how to add a search skill response type, see [Adding a Search skill response type](https://cloud.ibm.com/docs/assistant?topic=assistant-dialog-overviewdialog-overview-add-search-skill).

For more tips about improving results, read the [Improve your natural language query results from Watson Discovery](https://developer.ibm.com/blogs/improving-your-natural-language-query-results-from-watson-discovery/) blog post.
Databases for MongoDB Enterprise Edition also offers both private and public cloud service endpoints. If you want to access the Management UI from a browser that is not on the private network, you must take these additional steps as listed in the [Connecting Through Private Endpoints](https://cloud.ibm.com/docs/databases-for-mongodb?topic=cloud-databases-service-endpointsprivate-endpoints) documentation for IBM Cloud® Databases.

After you configure your environment for private endpoint access, you can go to to the Databases for MongoDB Enterprise management endpoint URL from your browser. For example, https://bfdb-4263-8ad2-c9a4beaf4591.8f7bfc8f3faa4218afd56e0.databases.appdomain.cloud:323232





 Ops Manager API key creation and usage 

Databases for MongoDB Enterprise Edition provides access to the Ops Manager API through generation of API keys. To do this:



* Navigate to your Account page
* From the Public API Access tab, generate up to ten API keys by using the Generate button.



Using the Ops Manager connection string (also found in the [Getting connection strings](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-connection-strings) section of your deployment dashboard), you can access API commands with the credentials that you created for Ops Manager user.



 Example connection details 

Use the following example credentials to query the Ops Manager user API:



* Example username that you created by using the API:



* opsmanager-123



* Example API key that is generated by using the prior steps:



* d043b2ae-bbf2-4f55-8b09-e1ce0906126f



* Example connection string obtained from the deployment dashboard Endpoints section:
Previously, you were required to open the Integrate and deploy page, select the Launch application tab, and click the Launch button.

Improved segment limit
: The segment limit when splitting documents has been increased to 1,000. For details, see [Split documents to make query results more succinct](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-split-documents).

Improved Filenet connector
: The [Filenet connector](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collection-typesfilenet-connect) has document level security.

New beta Curations feature
: You can specify up to 1,000 curations. For details about this beta feature, see [Curations](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-traincurations).

Fixed defects in the 2.1.3 release
: In versions 2.1.2, 2.1.1, and 2.1.0, PNG, TIFF, and JPG individual image files are not scanned, and no text is extracted from those files. PNG, TIFF, and JPEG images embedded in PDF, Word, PowerPoint, and Excel files are also not scanned, and no text is extracted from those image files.





 2.1.2 release, 31 March 2020 

New release now available
: IBM Watson® Discovery for IBM Cloud Pak® for Data version 2.1.2 is available.

New IBM FileNet connector
: You can now crawl IBM FileNet systems. For more information, see [FileNet connector](https://cloud.ibm.com/docs/services/discovery-data?topic=discovery-data-collection-typesfilenet-connect).

New Swedish, Norwegian, and Danish language support
: Added basic support for Swedish, Norwegian (Bokmål and Nynorsk), and Danish. For more information, see [Language support](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-language-support).

Change to Advanced rules models enrichment
Some web page content is dynamically generated and therefore cannot be crawled.



* Configuring search results for uploaded documents: If you are using a collection of uploaded documents and cannot get the correct search results or the results are not concise enough, consider using Smart Document Understanding when you create the data collection.

This feature enables you to annotate documents based on text formatting. For example, you can teach Discovery that any text in 28-point bold font is a document title. If you apply this information to the collection when you ingest it, you can later use the title field as the source for the title section of your search result.

You can also use Smart Document Understanding to split up large documents into segments that are easier to search. For more information, see the the [Configuring your collection with Smart Document Understanding](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-configuring-fields) topic in the Discovery documentation.
* Improve search results: If you don't like the results you are seeing, review this information for help.



* Call the search skill from a dialog node, and specify filter details.



From a dialog node search skill response, you can specify a full Discovery query syntax filter to help narrow the results.

For example, you can define a filter that filters out any documents in the data collection that do not mention an intent in the document title or some other metadata field. Or the filter can filter out documents that do not identify an entity as a known entity in the data collection's metadata or that don't mention the entity anywhere in the full text of the document. For details about how to add a search skill response type, see [Adding rich responses](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-dialog-overviewdialog-overview-multimedia-add).

For more tips about improving results, read the [Improve your natural language query results from Watson Discovery](https://developer.ibm.com/blogs/improving-your-natural-language-query-results-from-watson-discovery/) blog post.







 Next steps 

After you create the skill, it appears as a tile on the Skills page.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE OF CONTENTS

THE REMAINDER OF THIS IBM NOTICES FILE CONSISTS OF THE FOLLOWING
SECTIONS:

CC-BY-2.5
CC-BY-3.0
CC-BY-4.0
CC-BY-SA-3.0
CC-BY-SA-4.0
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

CREATIVE COMMONS ATTRIBUTION 2.5 GENERIC

The Program includes some or all of the following works licensed under
the Creative Commons.
The url to the license is https://creativecommons.org/licenses/by/2.5/legalcode

GUARDEDBY.JAVA [checker-qual]
(https://mvnrepository.com/artifact/org.checkerframework/checker-qual/2.5.2),
CONCURRENT [JSR305]
(https://search.maven.org/remotecontent?filepath=com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2-sources.jar),
ANNOTATIONS [MongoDB-Driver]
(https://search.maven.org/remotecontent?filepath=org/mongodb/mongodb-driver-core/4.0.5/mongodb-driver-core-4.0.5-sources.jar)

END OF CREATIVE COMMONS ATTRIBUTION 2.5 GENERIC NOTICES AND INFORMATION