vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Overview 

Documents are [JSON objects](https://en.wikipedia.org/wiki/JSONData_types.2C_syntax_and_example). Documents are also containers for your data, and are the basis of the IBM® Cloudant® for IBM Cloud® database.

If you're using an [IBM Cloudant service on IBM Cloud®](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-ibm-cloud-publicibm-cloud-public), documents are limited to a maximum size of 1 MB. Exceeding this limit causes a [413 error](https://cloud.ibm.com/apidocs/cloudantlist-of-http-codes).

IBM Cloudant uses an [eventually consistent](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-cap-theoremcap-theorem) model for data. If you use the eventually consistent model, it's possible, under some conditions, to retrieve older document content. For example, older content is retrieved when your application writes or updates a document that is followed immediately by a read of the same document.

In other words, your application would see the document content as it was before the write or update occurred. For more information about this model, see the topic on [Consistency](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-cap-theoremcap-theorem).



 Document fields 

All documents must have two fields:



* A unique _id field. The _id field is detailed in the next section.
* A _rev field. The _rev field is a revision identifier, and is [essential to the IBM Cloudant replication protocol](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-document-versioning-and-mvccdocument-versioning-and-mvcc).



In addition to these two mandatory fields, documents can generally contain any other content that can be described by using JSON, subject to some caveats detailed in the following sections.



 Document IDs
* A smart_document_understanding field is returned with the Get collection method. This new field specifies whether an SDU model is enabled for the collection and indicates the model type.
* A similar parameter is available from the Query method. Use it to find documents that are similar to documents of interest to you.
* The suggested_refinements parameter of the Query method is deprecated. The suggested_refinements parameter was used to identify dynamic facets from Premium plan data.







 8 August 2022 

Larger documents can be crawled
: The maximum file sizes that are allowed for crawled documents increased for Premium plans. It also increased for the Box, IBM Cloud Object Storage, and Salesforce connectors. For more information, see [File size limits](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionscollections-file-size-limits).





 2 August 2022 

IAM authentication support was added to the IBM Cloud Object Storage connector
: You can now choose to authenticate with the IBM Cloud Identity and Access Management (IAM) service. For more information, see [IBM Cloud Object Storage](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-cos-cloud).





 28 July 2022 

API updates
: The following changes were made to the [Discovery v2 API](https://cloud.ibm.com/apidocs/discovery-data).

New fields are available:



* A smart_document_understanding field is returned with the Get collection method. This new field specifies whether an SDU model is enabled for the collection and indicates the model type.
* A similar parameter is available from the Query method. Use it to find documents that are similar to documents of interest to you.



The suggested_refinements parameter of the Query method is deprecated. The suggested_refinements parameter was used to identify dynamic facets from Premium plan data.





 Discovery v1 deprecation announcement 

Watson Discovery v1 is being deprecated.
For more information, see [IBM Cloud Databases for PostgreSQL](https://www.ibm.com/cloud/databases-for-postgresql).
* Data warehouse for ad hoc querying. For more information, see [IBM® Db2® Warehouse on Cloud](https://www.ibm.com/products/db2-warehouse).
* A queue. For more information, see [IBM MQ](https://www.ibm.com/uk-en/products/mq).



For more information, see the [Best and worst practice](https://blog.cloudant.com/2019/11/21/Best-and-Worst-Practices.html) blog.





 Organizing documents and databases 

IBM Cloudant data is organized in a hierarchy of databases and documents. A document is a JSON object with a unique identifier: its _id. A database is a collection of documents with a primary index that allows documents to be retrieved by _id. It also has optional secondary indexes that allow documents to be queried by other attributes in the object.

When developers start a project, they sometimes struggle with the following questions:



* How much data can I put into a single object?
* Must I store different document types in the same collection or one database per document type?



It is important for a document to include all the data about an object that is modeled by your application, for example, a user, an order, or a product. This practice ensures you fetch the entire object from the database in one API call. IBM Cloudant doesn't have the concept of joins like a relational database, so data isn't normalized. However, data can repeat across objects. For example, an order document can include a subset of the product documents that were purchased.

It's common to store several object types in the same database: a convention is that a type attribute is used to denote the object type. This option is a good one if you need to perform queries that return several object types or if a database needs to be replicated to another location altogether.
: Our strategic partnership with [Elastic](https://www.elastic.co/about/) since January 2023 means that we are able to offer more and richer functionality, as well as world-class levels of support. IBM Cloud® Databases for Elasticsearch Enterprise Plan Version 7.17 is the current supported version of Databases for Elasticsearch and is offered under our Enterprise Plan. This version includes functionality that is not available in our Standard plan. IBM Cloud® Databases for Elasticsearch Standard Plan Versions 7.9 and 7.10 are available on Cloud Databases under the Standard Plan, but will be End of Life (EOL) by 30 November 2023. For more information, see [Databases for Elasticsearch Pricing](https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=databases-for-elasticsearch-pricing).





 19 October 2022 

Deploying and Connecting a Cloud Databases Instance Tutorial
: This tutorial guides you through the process of deploying a Cloud Databases instance and connecting it to a web front end by creating a webpage that allows visitors to input a word and its definition. These values are then stored in a database running on Cloud Databases. You install the database infrastructure by using Terraform and your web application uses the popular Express framework. The application can then be run locally, or by using Docker. For more information, see [Deploying and Connecting a Cloud Databases Instance](https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=cloud-databases-create-instance-tutorial).





 11 October 2022 

Protecting IBM Cloud® Databases for Elasticsearch resources with context-based restrictions
: Context-based restrictions (CBR) give account owners and administrators the ability to define and enforce access restrictions for IBM Cloud® resources based on the context of access requests. Access to Cloud Databases resources can be controlled with CBR and identity and access management (IAM) policies. For more information, see [Protecting Cloud Databases resources with context-based restrictions](https://cloud.ibm.com/docs/databases-for-cassandra?topic=cloud-databases-cbr&interface=ui).





 25 March 2022
[Databases for MySQL](https://cloud.ibm.com/docs/databases-for-mysql) IBM Cloud Databases for MongoDB is a managed MySQL service that is hosted in the IBM Cloud and integrated with other IBM Cloud services. [View docs](https://cloud.ibm.com/docs/databases-for-mysql?topic=cloud-databases-key-protect) 
 [IBM Cloud Databases for PostgreSQL](https://cloud.ibm.com/docs/databases-for-postgresql) IBM Cloud Databases for PostgreSQL is a managed PostgreSQL service that is hosted in the IBM Cloud and integrated with other IBM Cloud services. [View docs](https://cloud.ibm.com/docs/databases-for-postgresql?topic=cloud-databases-key-protect) 
 [IBM Cloud Databases for Redis](https://cloud.ibm.com/docs/databases-for-redis) IBM Cloud Databases for Redis is a managed service that is hosted in the IBM Cloud and integrated with other IBM Cloud services. [View docs](https://cloud.ibm.com/docs/databases-for-redis?topic=cloud-databases-key-protect) 
 [IBM Cloud Hyper Protect DBaaS for MongoDB](https://cloud.ibm.com/docs/hyper-protect-dbaas-for-mongodb) Hyper Protect DBaaS for MongoDB offers fully managed and highly secure MongoDB databases with a high level of data confidentiality for your sensitive data. [View docs](https://cloud.ibm.com/docs/hyper-protect-dbaas-for-mongodb?topic=hyper-protect-dbaas-for-mongodb-key-protect-byok)
IBM Cloud® Databases for MongoDB 4.0 End of Life in April, 2022
: On April 27, 2022, all IBM Cloud® Databases for MongoDB instances on version 4.0 that are still active will be upgraded in-place to the next major version. See blog post announcement [here](https://www.ibm.com/cloud/blog/announcements/databases-for-mongodb-40-end-of-life-in-april-2022).





 1 December 2020 

IBM Cloud® Databases for MongoDB 3.6 End of Life in April, 2021
: On April 27, 2021, all IBM Cloud® Databases for MongoDB instances on version 3.6 that are still active will be upgraded in-place to the next major version. See blog post announcement [here](https://www.ibm.com/cloud/blog/announcements/databases-for-mongodb-36-end-of-life-in-april-2021).





 31 August 2020 

General Availability of IBM Cloud® Databases for MongoDB Enterprise
: Introducing IBM Cloud Databases for MongoDB Enterprise. See blog post announcement [here](https://www.ibm.com/cloud/blog/announcements/powering-up-databases-for-mongodb).





 26 May 2020 

IBM Cloud® Databases for MongoDB's Improved High Availability Architecture
: At IBM Cloud Data Services, we are tirelessly investigating improvements to our service in order to offer our customers the best possible price-to-performance ratio balanced with data availability and safety. As part of that work, we are changing the default number of data members for IBM Cloud® Databases for MongoDB from two to three. See blog post announcement [here](https://www.ibm.com/cloud/blog/databases-for-mongodb-improved-high-availability-architecture).





 13 April 2020 

IBM Cloud® Databases for MongoDB autoscaling
: We are excited to announce that autoscaling of your deployments based on disk capacity and disk I/O utilization is now available for IBM Cloud® Databases for MongoDB via the UI, API, and CLI.
See blog post announcement [here](https://www.ibm.com/cloud/blog/databases-for-mongodb-improved-high-availability-architecture).





 13 April 2020 

IBM Cloud® Databases for MongoDB autoscaling
: We are excited to announce that autoscaling of your deployments based on disk capacity and disk I/O utilization is now available for IBM Cloud® Databases for MongoDB via the UI, API, and CLI. See blog post announcement [here](https://www.ibm.com/cloud/blog/announcements/ibm-cloud-databases-portfolio-introduces-autoscaling).





 6 August 2019 

New Regions Available for IBM Cloud Database Services
: IBM Cloud® Databases for MongoDB is now available to be deployed in Seoul; South Korea; and Chennai, India. See blog post announcement [here](https://www.ibm.com/cloud/blog/announcements/new-regions-available-for-ibm-cloud-database-services).





 27 February 2019 

General Availability of IBM Cloud® Databases for MongoDB
: IBM Cloud® Databases for MongoDB added to the [IBM Cloud Databases](https://www.ibm.com/cloud/databases) family. See blog post announcement [here](https://www.ibm.com/cloud/blog/ibm-cloud-databases-for-mongodb-is-generally-available::text=IBM%20Cloud%20is%20announcing%20the,with%20enterprise%20security%20in%20mind.).
Known issue in syndicated environments : Discovery Tooling does not support syndicated environments. [Resolved](https://cloud.ibm.com/docs/discovery?topic=discovery-release-notes15nov17)





 14 September 2017 

New Data Schema Explorer : Added the Data Schema Explorer, which displays the fields and values in your transformed documents. This information can be used to understand the data structure of your collection before building queries using the Discovery Query Language. The data schema can be viewed two ways: by document (Document view), or by field (Collection view). To access the Data Schema Explorer: on the My Data Insights screen, click the View data schema button, or click the View Data Schema icon on the left.





 6 September 2017 

New beta deduplicate documents function in Discovery News : Added the beta ability to deduplicate documents returned from your query. This beta feature works for both private and Watson Discovery News collections. See [Excluding duplicate documents from query results](https://cloud.ibm.com/docs/discovery?topic=discovery-query-parametersdeduplication) for details. : Document deduplication is currently supported only as a beta capability.





 31 August 2017 

Update to version string : The version string for all API calls changed to 2017-09-01 from 2017-08-01. This version includes updates that filter out the following invalid JSON fields during preview and ingestion so that only valid JSON fields are ingested. Update your version string to 2017-09-01 to avoid conflicts and possible errors.

Improved query and add functions at top level : id, score, and highlight at the top level (You can continue to add documents to your collection using document IDs with the add a document function. See the [API Reference](https://cloud.ibm.com/apidocs/discoveryadd-a-document) for details. : _ prefixed field names at the top level (as a result, when querying for a document by ID, you can query for id instead of _id.)
Access to Cloud Databases resources can be controlled with CBR and identity and access management (IAM) policies. For more information, see [Protecting Cloud Databases resources with context-based restrictions](https://cloud.ibm.com/docs/databases-for-mongodb?topic=cloud-databases-cbr&interface=ui).





 22 June 2022 

Mapping Global COVID-19 cases with the Databases for MongoDB EE (Enterprise Edition) Analytics Add-On and Tableau tutorial published
: The Databases for MongoDB EE (Enterprise Edition) Analytics Add-On allows you to run long-running analytical queries or provision a [MongoDB Connector for business intelligence(BI)](https://docs.mongodb.com/bi-connector/current/) to make your query data compatible with BI tools, such as [Tableau](https://www.tableau.com/). For more information, see [Databases for MongoDB EE (Enterprise Edition) Analytics Add-On](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-mongodbee-analytics). This tutorial familiarizes you with the Analytics Add-On using Tableau to visualize data in your MongoDB instance. For more information, see [Mapping Global COVID-19 cases with the Databases for MongoDB EE (Enterprise Edition) Analytics Add-On and Tableau](https://cloud.ibm.com/docs/databases-for-mongodb?topic=cloud-databases-bi-connector-tutorial-description).





 31 May 2022 

Provision a IBM Cloud® Databases for MongoDB Enterprise Edition instance with Terraform tutorial published
: In this tutorial, you learn how to use Terraform to provision a Databases for IBM Cloud® Databases for MongoDB Enterprise Edition instance that includes the IBM Cloud® Databases for MongoDB Enterprise Edition Analytics Add-On.
Your database's schema can evolve over time to match your application's needs. You don't (necessarily) need to tell the database about the schema change - write new documents in the new format.

We can even store multiple document "types" in the same database. In this case, people, books, and places reside in the same database. We know which is which because of the "type" field (this field is a convention and not something that means anything to IBM Cloudant).

An alternative to this configuration is to have three databases people, books, and places and keep each data type in its own database. Both approaches are fine. You can choose to have multiple types together in the same database if you need to perform queries across types or if you need to replicate all data types together. Otherwise, the separate databases approach might be better.

To summarize, although IBM Cloudant is "schemaless", this fact doesn't absolve you of the need to do detailed data design to get the best performance.

These tips are especially relevant if you have some relational database experience.



* Avoid thinking in joins - an IBM Cloudant document must contain everything that you need about that object so that it can be retrieved in one API call.
* Normalization goes out of the window in JSON store, some repeated values can be tolerated if it makes data retrieval more efficient.
* Although we have a 1 MB limit on document size, your documents must be much smaller - a few KB is typical.
* If your application can embrace a "write only"* design pattern, where data is only ever added to a database, then it can make your life easier. You must definitely avoid patterns that rely on updating the same document over and over in a small time window.



That's the end of this part. The next part is called The Document ID.







 The _id video 

Learn how _ids work in IBM Cloudant, how they are different from relational databases, and how you can define your own _id.



* The _id video script

Welcome to the Introduction to IBM Cloudant course, an 17-part video series that gives you an overview of the IBM Cloudant database-as-a-service.
An IBM Cloudant document must be a JSON object, starting and ending with curly braces and containing a number of key-value attributes.

JSON objects must be less that 1 megabyte in size and contain any number of strings, numbers, booleans, arrays, and objects. The nesting of objects within objects can continue to any depth.

The keys that are used can be as brief or verbose as you like.

The following list includes some simple example documents that show how each data type is used.



* The first example shows a person object, storing strings, booleans, and an array of tags.
* The second example shows brief attribute names to save on storage and represents a web event such as click a website.
* The last example shows how the document may itself contain subjects.



A note on dates. JSON has no native Date type, so dates are usually stored in 30-October-2018 or similar formats. We return to dates later.

Now, for your first practical exercise, visit www.ibm.com/cloud. Register an account with the IBM Cloud, if you don't have one already.

Once registered, you can click services, search for the Cloudant database, and provision a new service.

The IBM Cloudant Lite service provides a free plan to allow users to try IBM Cloudant in a limited capacity while in development. Its bigger brother, the Standard Plan, is a paid-for service where you specify the number of reads, writes, and queries per second for your application and that capacity is reserved for you. You pay for the capacity you provision and your data storage usage.

The Lite plan operates in a similar way. It has only a small provisioned capacity and a fixed storage size, but it's fine for testing the IBM Cloudant service.

IBM Cloudant is often referred to as a "schemaless" database - but we have to be careful how we define that term.

It's true to say that you don't need to define your schema (field names, types, constraints, and relationships) ahead of time in an IBM Cloudant database. You can simply write a JSON document of your own design to a database.
HCL Notes 

Crawl an HCL Notes (formerly Lotus Notes) database.

IBM Cloud Pak for Data

This information applies only to installed deployments.



 What documents are crawled 



* Each document in the HCL Notes database is crawled and added to the collection as a document.
* If an HCL Notes document has a file attachment, and you choose to process file attachments, only documents that are supported by Discovery are crawled; all others are ignored. For more information, see [Supported file types](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionssupportedfiletypes).
* If you choose to process attachments, the crawler attempts to crawl and index files that are attached to HCL Notes documents. File types that are supported by Discovery are indexed. For more information, see [Supported file types](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionssupportedfiletypes).
* Document-level security is supported. When this option is enabled, your users can crawl and query the same content that they can access when they are logged in to HCL Notes. For more information, see [Supporting document-level security](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collection-typesconfiguredls).
* When a source is recrawled, new documents are added, updated documents are modified to the current version, and deleted documents are deleted from the collection's index.
* All Discovery data source connectors are read-only. Regardless of the permissions that are granted to the crawl account, Discovery never writes, updates, or deletes any content in the original data source.







 Data source requirements 

In addition to the [data source requirements](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collection-typesrequirements) for all installed deployments, your HCL Notes data source must meet the following requirements:



* The data source can crawl HCL Notes 9.0.1 databases.
[IBM Cloud Databases for etcd](https://cloud.ibm.com/docs/databases-for-etcd?topic=databases-for-etcd-getting-started) IBM Cloud Databases for etcd is a managed etcd service that is hosted in the IBM Cloud and integrated with other IBM Cloud services. [Platform metrics](https://cloud.ibm.com/docs/databases-for-etcd?topic=databases-for-etcd-monitoring) 
 [IBM Cloud Databases for Elasticsearch](https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=databases-for-elasticsearch-getting-started) IBM Cloud Databases for Elasticsearch is a managed Elasticsearch service that is hosted in the IBM Cloud and integrated with other IBM Cloud services. [Platform metrics](https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=databases-for-elasticsearch-monitoring) 
 [IBM Cloud Databases for MongoDB](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-getting-started) IBM Cloud Databases for MongoDB is a managed MongoDB service that is hosted in the IBM Cloud and integrated with other IBM Cloud services. [Platform metrics](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-monitoring) 
 [IBM Cloud Messages for RabbitMQ](https://cloud.ibm.com/docs/messages-for-rabbitmq?topic=messages-for-rabbitmq-getting-started) IBM Cloud Messages for RabbitMQ is a managed RabbitMQ service that is hosted in the IBM Cloud and integrated with other IBM Cloud services. [Platform metrics](https://cloud.ibm.com/docs/messages-for-rabbitmq?topic=messages-for-rabbitmq-monitoring)