vania-janet/multiturn-rag-retrieval-data huggingface dataset:
* Up to 20 custom dictionaries
* Up to 20 custom pattern extraction models
* Up to 20 custom regular expression models
* Up to 20 custom text classification models









 Enterprise 

Scale and secure your Discovery application with enterprise-grade support and performance, and address more use cases including contract analysis and content mining to explore insights across documents.



 What's included 

For subscription accounts, the monthly fee per instance is $5,000.

The plan includes the following features:



* 100,000 documents per month ($5 for every additional 1,000 documents per month)
* 100,000 query or analyze API calls per month ($5 for every additional 1,000 calls per month)
* Up to 3 custom models [Learn more](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-pricing-planscustom-models)
* Up to 10 entity extractor models
* Up to 5 queries per second
* Everything that's available in Plus
* Analyze API
* Content Intelligence
* Content Mining
* Document classification (Text classification is available in all plans. Document classification is available with Enterprise and higher-level plans only.)



Your bill labels requests that are generated from both query searches and analyze API calls as "Queries".





 Artifact limits 



* Unlimited queries per month
* Unlimited documents
* Up to 100 projects
* Up to 300 collections (Up to 5 collections for all project types except Content Mining, which supports 1)
* Up to 10 MB document size
* Up to 10 custom Knowledge Studio models
* Up to 10 entity extractor models
* Up to 100 custom fields for Smart Document Understanding model and model import and export
* Up to 100 custom dictionaries
* Up to 100 custom pattern extraction models
* Up to 100 custom regular expression models
* Up to 20 custom text classification models
* Up to 20 custom document classification models



Autocompletion, curation, and notice requests are not billed in any plan type.







 How is document pricing calculated? 

IBM uses the maximum number of documents per day and then prorates the document price to calculate the cost for a month.
Database partitioning 

IBM® Cloudant® for IBM Cloud® supports two types of databases:



* Partitioned
* Non-partitioned



A partitioned database offers significant performance and cost advantages but requires you to specify a logical partitioning of your data. This process is described more in the following text.

Alternatively, you can create a non-partitioned database. This type of database might be easier to work with as no partitioning scheme needs to be defined, but only global secondary indexes can be created.

IBM Cloudant strongly recommends that you use a partitioned database for best long-term database performance where the data model allows for logical partitioning of documents.

You can decide whether to partition at database creation time. When you create a database, use the partitioned query string parameter to set whether the database is partitioned. The default for partitioned is false, maintaining compatibility with an earlier version.

The partitioning type can't be changed for an existing database.



 Database shards 

Before you read this document, you must understand the [sharding concept](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-how-is-data-stored-in-ibm-cloudant-how-is-data-stored-in-ibm-cloudant-) within IBM Cloudant.





 Non-partitioned databases 

A non-partitioned database is the older type of IBM Cloudant database, and the one that is familiar if you used CouchDB or IBM Cloudant previously.

Within a non-partitioned database, documents are distributed to shards in an arbitrary manner based on a transformation of their document ID. Therefore, no real relation exists between a document's ID and the shard it ends up on. Documents with similar document IDs are unlikely to be placed onto the same shard.

A non-partitioned database offers only global querying, described in more detail later.





 Partitioned databases 

A partitioned database is the newest type of IBM Cloudant database. Within a partitioned database, documents are formed into logical partitions by use of a partition key. The partition key is part of the document ID for documents within a partitioned database.
[Resolved](https://cloud.ibm.com/docs/discovery?topic=discovery-release-notes5jun18)

New Simplified Chinese and Dutch language support : Added Basic language support for Simplified Chinese and Dutch. See [Language Support](https://cloud.ibm.com/docs/discovery?topic=discovery-language-support) for more information. Currently, Simplified Chinese and Dutch collections must be created with the API.

New parameters for Data Crawler : Added two new parameters for the Data Crawler: proxy_host_port and read-timeout. See [Configuring the Data Crawler](https://cloud.ibm.com/docs/discovery?topic=discovery-configuring-the-data-crawler) for details.

Known issues during PDF ingestion : When ingestion notices are queried, the field file_type for pdf documents is returned as html. : The field file_type in the extracted_metadata object of results for pdf documents is set to html. : The document details API also returns the field file_type for pdf documents as html. : If you are ingesting JSON, mixed-type arrays are not supported.
: Update: [Resolved](https://cloud.ibm.com/docs/discovery?topic=discovery-release-notes21feb18)

New visual query builder in beta Knowledge Graph : Added a visual query builder for the beta version of IBM Watson™ Discovery Knowledge Graph. : Update: As of 30 September 2019, the IBM Watson™ Discovery Knowledge Graph Beta APIs are no longer accessible. See the [3 September 2019 release notes](https://cloud.ibm.com/docs/discovery?topic=discovery-release-notesdiscovery-3september2019) for more information.





 30 November 2017 

New beta Knowledge Graph : Released the beta version of IBM Watson™ Discovery Knowledge Graph, which provides new end-points for querying entities and relations across documents. This includes context-based searches and relevance ranking. : Update: As of 30 September 2019, the IBM Watson™ Discovery Knowledge Graph Beta APIs are no longer accessible.
: Discovery includes autocomplete (type-ahead) for searches, as well as a reusable component for providing this feature to your end users.

New support for 12 languages
: Language support for Discovery is now available in 12 additional languages. For the complete list, see [Language support](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-language-support).

Cloud Object Storage connector limitation
: When connecting to an IBM Cloud® Object Storage data source, only the first 75 buckets for a given credential are displayed.

Current API version
: The API version (v2) is 2019-11-29.

Change to features in this release
: Deduplication is not available in this release.
: Anomaly Detection is not offered.
: IBM Watson® Discovery News is no longer included.
: Several Watson Natural Language Understanding enrichments are not available at this time (Entity extraction, Relation extraction, Keyword extraction, Category classification, Concept tagging, Semantic Role extraction, Sentiment analysis, Emotion analysis)
: The SharePoint 2016 On-Premises and Box data sources are not available at this time.
IBM Cloud® Databases for etcd docs 

etcd is a distributed, reliable key-value store for the most critical data of a distributed system.

 Developer tools 

[API & SDK reference](https://cloud.ibm.com/apidocs/cloud-databases-api/cloud-databases-api-v5introduction)[CLI reference](https://cloud.ibm.com/docs/databases-cli-plugin)[Terraform reference](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/resources/database)

 Recommended content 

[Getting Started A brief tutorial to introduce using an IBM Cloud® Databases for etcd deployment.](https://cloud.ibm.com/docs/databases-for-etcd?topic=databases-for-etcd-getting-started)[Setting the Root Password The Databases for etcd service is provisioned with a root user. You must set the root password before you use it to connect.](https://cloud.ibm.com/docs/databases-for-etcd?topic=databases-for-etcd-root-password)[Creating Users and Getting Connection Strings To connect to IBM Cloud® Databases for etcd, you need some users and connection strings. Connection Strings for your deployment are displayed on the Dashboard Overview, in the Endpoints panel. These strings can be used with any set of credentials that you generate.](https://cloud.ibm.com/docs/databases-for-etcd?topic=databases-for-etcd-connection-strings&interface=ui)

 Learn more 

[IBM Developer<br><br>Visit IBM Developer for technical articles, code patterns, tutorials, and more.<br><br>![112-Developer-tools.svg](https://cloud.ibm.com/media/docs/images/homepage/112-Developer-tools.svg)](https://developer.ibm.com/depmodels/cloud/)[Architecture Center<br><br>Discover the architecture references available for this product.<br><br>!
: We are excited to announce that autoscaling of your deployments based on disk capacity and disk I/O utilization is now available for IBM Cloud® Databases for etcd via the UI, API, and CLI. See blog post announcement [here](https://www.ibm.com/cloud/blog/announcements/ibm-cloud-databases-portfolio-introduces-autoscaling).





 6 August 2019 

New Regions Available for IBM Cloud Database Services
: IBM Cloud® Databases for etcd is now available to be deployed in Seoul; South Korea; and Chennai, India. See blog post announcement [here](https://www.ibm.com/cloud/blog/announcements/new-regions-available-for-ibm-cloud-database-services).





 19 December 2018 

General Availability of Databases for etcd
: Databases for etcd added to the [IBM Cloud Databases](https://www.ibm.com/cloud/databases) family. See blog post announcement [here](https://www.ibm.com/cloud/blog/ibm-cloud-databases-for-etcd-elasticsearch-and-messages-for-rabbitmq-are-now-generally-available).
[IBM Cloud® Databases for DataStax](https://cloud.ibm.com/docs/databases-for-cassandra?topic=databases-for-cassandra-getting-started) IBM Cloud® Databases for DataStax is a scale-out NoSQL database that is built on Apache Cassandra. It’s designed to power real-time applications with high availability and massive scalability. [More info](https://cloud.ibm.com/docs/databases-for-cassandra?topic=cloud-databases-logging) 
 [IBM Cloud Databases for etcd](https://cloud.ibm.com/docs/databases-for-etcd?topic=databases-for-etcd-getting-started) IBM Cloud Databases for etcd is a managed etcd service that is hosted in the IBM Cloud and integrated with other IBM Cloud services. [More info](https://cloud.ibm.com/docs/databases-for-etcd?topic=cloud-databases-logging) 
 [Databases for MongoDB](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-getting-started) Databases for MongoDB is a managed MongoDB service that is hosted in the IBM Cloud and integrated with other IBM Cloud services. [More info](https://cloud.ibm.com/docs/databases-for-mongodb?topic=cloud-databases-logging) 
 [IBM Cloud Databases for PostgreSQL](https://cloud.ibm.com/docs/databases-for-postgresql?topic=databases-for-postgresql-getting-started) IBM Cloud Databases for PostgreSQL is a managed PostgreSQL service that is hosted in the IBM Cloud and integrated with other IBM Cloud services. [More info](https://cloud.ibm.com/docs/databases-for-postgresql?topic=cloud-databases-logging)
[IBM Developer<br><br>Visit IBM Developer for technical articles, code patterns, tutorials, and more.<br><br>![112-Developer-tools.svg](https://cloud.ibm.com/media/docs/images/homepage/112-Developer-tools.svg)](https://developer.ibm.com/depmodels/cloud/)[Architecture Center<br><br>Discover the architecture references available for this product.<br><br>![190-Application-Platform.svg](https://cloud.ibm.com/media/docs/images/homepage/190-Application-Platform.svg)](https://www.ibm.com/cloud/architecture)[IBM Training<br><br>Build your skills with IBM Cloud training.<br><br>![83-Elements-of-the-cloud.svg](https://cloud.ibm.com/media/docs/images/homepage/83-Elements-of-the-cloud.svg)](https://www.ibm.com/training/cloud)[Community<br><br>Join the IBM Cloud community and collaborate with others.<br><br>![35-Connectivity.svg](https://cloud.ibm.com/media/docs/images/homepage/35-Connectivity.svg)](https://community.ibm.com/community/user/publiccloud/home)

 Latest updates 

[View more](https://cloud.ibm.com/docs/databases-for-etcd?topic=databases-for-etcd-etcd-relnotes)19 October 2022 | Deploying and Connecting an IBM Cloud® Databases Instance Tutorial

This tutorial guides you through the process of deploying a Cloud Databases instance and connecting it to a web front end by creating a webpage that allows visitors to input a word and its definition. These values are then stored in a database running on Cloud Databases. You install the database infrastructure by using Terraform and your web application uses the popular Express framework. The application can then be run locally, or by using Docker.
Query reference 

IBM Watson™ Discovery offers powerful content search capabilities through queries. After your content is uploaded and enriched by Discovery, you can build queries, integrate Discovery into your own projects, or create a custom application by using the Watson Explorer Application Builder.

For more information about writing queries, see:



* [Getting started with querying](https://cloud.ibm.com/docs/discovery?topic=discovery-getting-started-with-querying)
* [Query concepts](https://cloud.ibm.com/docs/discovery?topic=discovery-query-concepts)





 Parameters descriptions 

Query parameters enable you to search your collection, identify a result set, and perform analysis on the result set.



 Parameter Description Example 

 Search parameters 
 [query](https://cloud.ibm.com/docs/discovery?topic=discovery-query-parametersquery) A ranked query language search for matching documents. query=bees 
 [filter](https://cloud.ibm.com/docs/discovery?topic=discovery-query-parametersfilter) An unranked query language search for matching documents. filter=bees 
 [natural_language_query](https://cloud.ibm.com/docs/discovery?topic=discovery-query-parametersnlq) A ranked natural language search for matching documents natural_language_query="How do bees fly" 
 [aggregation](https://cloud.ibm.com/docs/discovery?topic=discovery-query-parametersaggregation) A statistical query of the results set aggregation=term(enriched_text.entities.type) 
 Structure parameters 
 [count](https://cloud.ibm.com/docs/discovery?topic=discovery-query-parameterscount) The number of result documents to return. count=15 
 [offset](https://cloud.ibm.com/docs/discovery?topic=discovery-query-parametersoffset) The number of results to ignore before returning result documents from the results set offset=100 
 [return](https://cloud.ibm.com/docs/discovery?topic=discovery-query-parametersreturn) List of fields to return return=title,url
About Cloud Databases 

IBM Cloud Databases are managed databases (and a messaging queue) that are hosted in IBM Cloud and integrated with other IBM Cloud services.

Cloud Databases covers the following services:



*  IBM Cloud® Databases for DataStax
*  IBM Cloud® Databases for Elasticsearch
*  IBM Cloud® Databases for EnterpriseDB
*  IBM Cloud® Databases for etcd
*  IBM Cloud® Databases for MongoDB
*  IBM Cloud® Databases for PostgreSQL
*  IBM Cloud® Databases for Redis
*  IBM Cloud® Databases for MySQL
*  IBM Cloud® Messages for RabbitMQ
* You can specify multiple crawler spaces by using both the Class and Folder types and crawl the documents belonging to the folder name and class name.
* You cannot specify a class outside the object store that you defined.
* No support is available for specifying a class that is a subclass of a Custom Object and Folder.



11. After you enter one or more paths, click Add.
12. Optional: In the Security section, if you want to enable document-level security, set the Enable Document Level Security switch to On.

When set to On, your users can crawl the same content that they have access to in FileNet.
13. If you want to limit the types of files to add to the collection, you can list the file extensions for file types to either include or exclude.

For a list of supported file types, see [Supported file types](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionssupportedfiletypes).

Support for this option was added with the 4.7.0 release.
14. If you want the crawler to extract text from images in documents, expand More processing settings, and set Apply optical character recognition (OCR) to On.

When OCR is enabled and your documents contain images, processing takes longer. For more information, see [Optical character recognition](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionsocr).
15. Click Finish.



The collection is created quickly. It takes more time for the data to be processed as it is added to the collection.

If you want to check the progress, go to the Activity page. From the navigation pane, click Manage collections, and then click to open the collection.
Learn more 

[IBM Developer<br><br>Visit IBM Developer for technical articles, code patterns, tutorials, and more.<br><br>![112-Developer-tools.svg](https://cloud.ibm.com/media/docs/images/homepage/112-Developer-tools.svg)](https://developer.ibm.com/components/hyper-protect/)

 Latest updates 

[View more](https://cloud.ibm.com/docs/hyper-protect-dbaas-for-mongodb?topic=hyper-protect-dbaas-for-mongodb-what-new)15 January 2023 | Updated: Deprecation

Hyper Protect DBaaS for MongoDB is deprecated. As of 15 April 2023, you can't create new instances, and access to free instances will be removed. Existing paid instances are supported until 30 November 2023. Existing instances will be deleted on 1 March 2024.

26 September 2022 | Added: Restoring local backups in the UI

You can use the dashboard to view the list of local backups and choose one to restore.

23 September 2022 | Added: Self-service restore of local backups

You can use the CLI or API to list and restore local backups.

23 September 2022 | Added: Enhancement to service-managed cross-region backup and restore

You can change the configuration of backup to IBM Cloud Object Storage.

23 September 2022 | Added: Backup & restore page in the dashboard

You can use the dashboard for cross-region backup and restore.

22 April 2022 | Added: Service-managed cross-region backup and restore

Hyper Protect DBaaS for MongoDB supports service-managed cross-region backup and restore through integration with IBM Cloud Object Storage.

22 April 2022 | Updated: Frequency of automatic local backups

Hyper Protect DBaaS for MongoDB automatically triggers a backup of your complete databases on local storage once every 8 hours (previously every 24 hours).

28 January 2022 | Updated: Supported cipher suites

Hyper Protect DBaaS for MongoDB stops supporting some weaker cipher suites in TLS connections to the service to enhance data security.
: Any IBM Cloudant operation can be achieved by using HTTPS. HTTP is the protocol that powers the World Wide Web and IBM Cloudant is a database that is built for the web. Most databases are hidden in a private network, inaccessible but to a handful of machines. The IBM Cloudant service sits (mainly) on the public internet where it can be accessed by anyone with an internet connection (and permission to do so).

IBM Cloudant wasn't written entirely by IBM. It is based on Apache CouchDB, an open source project run by the Apache Foundation. IBM Cloudant employs a number of CouchDB contributors but by the rules of Apache, they cannot monopolize its development.

Much of what you see that is in this course is as applicable to Apache CouchDB as it is to IBM Cloudant. Their APIs are 99% the same - I point out where they diverge.

IBM Cloudant can be thought of as CouchDB run "as-a-service". An IBM Cloudant service is easily deployed and is managed by IBM engineers 24-7. The service has no software to install, no servers to manage, no configuration to understand. The user doesn't need to be a CouchDB expert to use and manage it.

You can be sure that your data layer is not locked in to a particular platform, cloud, or vendor because IBM Cloudant is being built on truly open source foundations. IBM Cloudant can be used in concert with CouchDB to create hybrid applications that share data through replication, as we see.

Later on in the course we look under the hood to see how IBM Cloudant works, but initially we treat IBM Cloudant as a "black box".

To summarize, Cloudant is based on Apache CouchDB, an open source project. It stores JSON documents. It is accessed with an HTTP API and can therefore be accessed by any device on the internet that speaks HTTP: application code, web browser, IoT device, or mobile phone. IBM Cloudant is a highly available managed service able to continue to operate with multiple hardware failures.

That's the end of this part. The next part is called The Document.







 The Document video 

Learn about IBM Cloudant databases and documents work.



* The Document video script
IBM Cloud® Databases for Elasticsearch autoscaling
: We are excited to announce that autoscaling of your deployments based on disk capacity and disk I/O utilization is now available for IBM Cloud® Databases for Elasticsearch via the UI, API, and CLI. See blog post announcement [here](https://www.ibm.com/cloud/blog/announcements/ibm-cloud-databases-portfolio-introduces-autoscaling).





 20 February 2020 

IBM Cloud® Databases for Elasticsearch customers are now able to scale their IBM Cloud® Databases for Elasticsearch instances up to 20 members.
: This new feature allows Elasticsearch instances to now grow to 2.2TB RAM, 80 TB of Disk, and 560 vCPUs. See blog post announcement [here](https://www.ibm.com/cloud/blog/announcements/databases-for-elasticsearch-introduces-horizontal-scaling).





 6 August 2019 

New Regions Available for IBM Cloud Database Services
: IBM Cloud® Databases for Elasticsearch is now available to be deployed in Seoul; South Korea; and Chennai, India. See blog post announcement [here](https://www.ibm.com/cloud/blog/announcements/new-regions-available-for-ibm-cloud-database-services).





 19 December 2018 

General Availability of IBM Cloud® Databases for Elasticsearch
: IBM Cloud® Databases for Elasticsearch added to the [IBM Cloud Databases](https://www.ibm.com/cloud/databases) family. See blog post announcement [here](https://www.ibm.com/cloud/blog/ibm-cloud-databases-for-etcd-elasticsearch-and-messages-for-rabbitmq-are-now-generally-available).
* [Content Mining](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-resultsquery-results-cm)







 Document Retrieval 



1. Do one of the following things:



* Click Run search for one of the keywords that Discovery calculated to have special meaning in your collection.
* Submit your own phrase or keyword from the search bar.



You can see that the query results that are returned consist of passages.

Entities that are recognized in your documents (based on the Entities enrichment that is applied to the project by default) are displayed as facets by which you can filter the query results.
2. To explore a query result in more detail, click View passage in document.
3. Click Open advanced view to explore the entity mentions that are recognized by Discovery.





 Excerpt unavailable 

If the result shows “Excerpt unavailable”, you might need to adjust where the content of the response is taken from. By default, a passage from the document is returned as the response. You can change what is returned.



1. From the Improvement tools panel, expand Customize display.
2. Click Search results.
3. For the source of result content, select Field, and then choose the field from which you want to extract the response.

Zoom

![Shows the Search results dialog](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/search-result-by-field.png)

Figure 1. Search results dialog
4. Click Apply.









 Conversational Search 

A single search field is displayed that mimics the user interface of a virtual assistant.



1. Submit a phrase or keyword.



The query results are returned as passages by default.