vania-janet/multiturn-rag-retrieval-data huggingface dataset:
* Low quality images due to incorrect scanner settings, insufficient resolution, bad lighting (such as with mobile capture), loss of focus, unaligned pages, and badly printed documents
* Documents with irregular fonts or a variety of colors, font sizes, and backgrounds







 1 November 2022 

Entity extractor loads the first 40,000 characters from training data documents
: Even extra long documents from the collection that you use to define custom entity examples are loaded into the document view of the tool. However, only the first 40,000 characters, which is approximately 15-20 pages, are displayed. The rest of the file content is truncated. You'll know if your document is truncated because a notification is displayed in the document view. For more information, see [Entity extractor](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-entity-extractor).

You can set the passages per document setting to be higher than one
: A bug was fixed that prevented you from using the search bar settings in the product user interface to increase the maximum number of passages to return per document. For more information, see [How passages are derived](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-index-overviewquery-results-passages).

Improved query aggregation documentation
: The documentation that describes the aggregation types that you can specify in the query aggregation parameter was updated. For more information, see [Query aggregations](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-aggregations).





 30 September 2022 

Lite plans are no longer available from the London data center
: Lite plans are discontinued. You cannot create new service instances that use the Lite plan type in any location, including London. Use the new Plus plan and its associated 30-day free trial to explore new features and a simpler way to build that is available with the latest version of the product.





 22 September 2022 

Plus plan supports more entity extractors
: The maximum number of entity extractors that you can create with a Plus plan increased from 3 to 6.

You cannot apply a Smart Document Understanding model to Microsoft Excel files
For more information, see [Web crawl](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-web-cloud).





 16 May 2022 

Added API methods for working with stop words and expansion lists
: You can now get, add, and remove a stop words or expansion list for a collection programmatically. For more information, see the [Query modifications](https://cloud.ibm.com/apidocs/discovery-datagetstopwordlist) methods.





 13 May 2022 

An improved JSON view is available
: You can now use keyboard keys to tab through elements in the view. The new JSON view also numbers the occurrences of elements in each JSON object, which makes it easier to keep track of information and to read totals at a glance.





 20 April 2022 

Analyze API is supported in Enterprise plan deployments
: Use the Analyze API to process a JSON file according to a collection's configuration settings, and then return the file for realtime use without storing it in the collection. The Analyze API was supported only in installed deployments previously. For more information, see [Analyze API](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-analyzeapi).

A new document status API is available
: Use the new document status API to programmatically get a list of the documents in a collection and to get details about a single document. The following notes apply to this release:



* The API is supported for collections that are created after 23 March 2022.

If you want to get status information about a collection that was created earlier, trigger a process that runs the conversion step of ingestion on the documents. For example, you can enable the API by making changes in the Identify fields, Manage fields, CSV settings, or Processing settings (such as OCR or FAQ extraction settings) pages, or by applying a Smart Document Understanding model to the older collection.
* The API is available only from Plus and Enterprise plan instances.



For more information about the new API, see the [API reference documentation](https://cloud.ibm.com/apidocs/discovery-datalistdocuments).
PNG, TIFF, and JPEG images embedded in PDF, Word, PowerPoint, and Excel files are also not scanned, and no text is extracted from those image files. * Smart Document Understanding does not support .doc, .docx, .odf, .xls, .xlsx, .ods, .ppt, .pptx, and .odp conversion when FIPS (Federal Information Processing Standards) is enabled. * In a Content Mining application, any document flags set will disappear if the index is rebuilt for that collection. * Beginning with the 2.1.2 release, uploading and managing relevancy training data using the v1 APIs will not train a relevancy training model. The v1 APIs have been superseded by the Projects relevancy training v2 APIs]] . If your training data needs to be preserved, it can be listed using the v1 API, then added to a project with the v2 API. * Multiple Regular expressions]] cannot be applied to a collection at the same time. IBM Cloud Pak for Data

There were two small changes to the installation instructions README included with the download of IBM Watson® Discovery for IBM Cloud Pak® for Data. For the updated version of the README, see the Discovery Helm chart README.md]Discovery Helm chart README_.md]!!! ! ! ! .

<-- <ul> -->

* A change to the description of the --cluster-pull-prefix PREFIX argument.
* The language extension pack name has been updated from ibm-watson-discovery-pack1-2.1.2.tar.xz. to ibm-wat-dis-pack1-prod-2.1.2.tar.xz.

<-- </ul> -->
For more information, see [Mapping Global COVID-19 cases with the Databases for MongoDB EE (Enterprise Edition) Analytics Add-On and Tableau](https://cloud.ibm.com/docs/databases-for-mongodb?topic=cloud-databases-bi-connector-tutorial-description).





 31 May 2022 

Provision a IBM Cloud® Databases for MongoDB Enterprise Edition instance with Terraform tutorial published
: In this tutorial, you learn how to use Terraform to provision a Databases for IBM Cloud® Databases for MongoDB Enterprise Edition instance that includes the IBM Cloud® Databases for MongoDB Enterprise Edition Analytics Add-On. For more information, see [Provision a Databases for IBM Cloud® Databases for MongoDB Enterprise Edition instance with Terraform](https://cloud.ibm.com/docs/databases-for-postgresql?topic=cloud-databases-tutorial-provision-mongodbee-tf).





 26 April 2022 

IBM Cloud® Databases for MongoDB Enterprise Edition Analytics Add-On Release
: The IBM Cloud® Databases for MongoDB Enterprise Edition Analytics Add-On allows you to execute long-running analytical queries and/or provision a [MongoDB Connector for business intelligence(BI)](https://docs.mongodb.com/bi-connector/current/) to make your query data compatible with BI tools, such as [Tableau](https://www.tableau.com/). For more information, see the documentation [here](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-mongodbee-analytics).





 17 September 2021 

IBM Cloud® Databases for MongoDB 4.0 End of Life in April, 2022
: On April 27, 2022, all IBM Cloud® Databases for MongoDB instances on version 4.0 that are still active will be upgraded in-place to the next major version. See blog post announcement [here](https://www.ibm.com/cloud/blog/announcements/databases-for-mongodb-40-end-of-life-in-april-2022).
IBM Cloud® Databases for Elasticsearch docs 

Elasticsearch combines the power of a full text search engine with the indexing strengths of a JSON document database.

 Developer tools 

[API & SDK reference](https://cloud.ibm.com/apidocs/cloud-databases-api/cloud-databases-api-v5introduction)[CLI reference](https://cloud.ibm.com/docs/databases-cli-plugin)[Terraform reference](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/resources/database)

 Recommended content 

[Getting Started A brief tutorial to introduce using an IBM Cloud® Databases for Elasticsearch deployment.](https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=databases-for-elasticsearch-getting-started)[Setting the Admin Password The Databases for Elasticsearch service is provisioned with an admin user. You must set the admin password before you use it to connect.](https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=databases-for-elasticsearch-admin-password&interface=cli)[Getting Connection Strings To connect to IBM Cloud® Databases for Elasticsearch, you need some users and connection strings. Connection Strings for your deployment are displayed on the Dashboard Overview, in the Endpoints panel. These strings can be used with any set of credentials that you generate.](https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=databases-for-elasticsearch-connection-strings&interface=cli)

 Learn more 

[IBM Developer<br><br>Visit IBM Developer for technical articles, code patterns, tutorials, and more.<br><br>!
cloudant.com 

IBM® Cloudant® for IBM Cloud® is a hosted and fully managed database-as-a-service (DBaaS). It was built to scale globally, run non-stop, and handle a wide variety of data types like [JSON](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-ibm-cloudant-basicsjson-overview-basics) and [full-text](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-querycreating-an-index).

IBM Cloudant Shared Plan is being retired on 31 March 2018. Starting 24 July 2017, no new Shared Plan accounts can be created from the [IBM Cloudant product page](https://www.ibm.com/cloud/cloudant).

IBM Cloudant is an operational data store optimized to handle concurrent reads and writes, and enable high availability and data durability.

It provides an [HTTP API](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-ibm-cloudant-basicshttp-api-basics) for working with your [JSON](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-ibm-cloudant-basicsjson-overview-basics) data, and 24-hour operational support and maintenance. IBM Cloudant is based on [Apache CouchDB](https://couchdb.apache.org/), and is delivered as various multi-tenant, dedicated, and installed services.

All currency values in this document are in US dollars ($).

IBM Cloudant is offered as either a [Shared Plan](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-ibm-cloudant.comshared-plan) or an [Enterprise (Dedicated) Plan](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-ibm-cloudant.comenterprise-plan), and as an [IBM Cloud® service](https://www.ibm.com/cloud/).
This data source is not available in Dedicated environments. See [IBM Cloud Object Storage](https://cloud.ibm.com/docs/discovery?topic=discovery-sourcesconnectcos) for more information.





 4 February 2019 

Updates to Smart Document Understanding : [Smart Document Understanding (SDU)](https://cloud.ibm.com/docs/discovery?topic=discovery-sdu) moved from beta status to GA status. The beta of SDU was announced [January 22, 2019](https://cloud.ibm.com/docs/discovery?topic=discovery-release-notes22jan19).
: Table annotation remains in beta status. For a statement explaining beta features, see [Beta features](https://cloud.ibm.com/docs/discovery?topic=discovery-release-notesbeta-features). : The Data settings button on the upper right is renamed as Configure data. : Uploading a document is no longer required to access the Configure data (formerly Data settings) button.





 28 January 2019 

Deprecated support for Data Crawler : Assistance is no longer provided for the Data Crawler if you are using it with a data source supported by the Discovery connectors. The Discovery connectors support crawling Box, SharePoint, Salesforce, and more. See [Connecting to data sources](https://cloud.ibm.com/docs/discovery?topic=discovery-sources) for details. It is recommended that the Data Crawler only be used to crawl file shares or databases. In all other cases, use the Discovery connector. Another option to upload large numbers of files into Discovery is [discovery-files](https://github.com/IBM/discovery-files) on GitHub.





 22 January 2019 

New beta Smart Document Understanding : Released the beta version of [Smart Document Understanding (SDU)](https://cloud.ibm.com/docs/discovery?topic=discovery-sdu), a new way to train IBM Watson™ Discovery to extract custom fields in your documents. With Smart Document Understanding, you annotate fields within your documents to train custom conversion models.
Go there to see a list of the documents in your collection and get a quick view of information about the documents. You can also delete documents from a collection with just a few clicks. For more information, see [Excluding content from query results](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-hide-data).





 15 February 2022 

An alternative authentication mechanism is available for Microsoft Sharepoint Online connectors
: You can now use Open Authentication to sign in to Microsoft SharePoint directly when you configure a new IBM Cloud connector. The Sign in with Microsoft option that uses Open Authentication to authenticate with the external data source is a beta feature. For more information, see [Microsoft SharePoint Online](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-sharepoint-online-cloud).





 7 January 2022 

Upgrade from Plus to Enterprise without help
: You can perform an in-place upgrade from a Plus plan to an Enterprise plan. For more information, see [Upgrading](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-upgrade).





 6 December 2021 

Crawling web pages with dynamic content is now generally available
: The Execute JavaScript during crawl feature was introduced as a beta feature, but is now generally available. For more information, see [Web crawl](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-web-cloud).

Capturing the SharePoint ACL information from crawled documents
: You can now configure the data source crawl to store ACL information as metadata in the documents that are added to your SharePoint Online collection. For more information, see [Microsoft SharePoint Online](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-sharepoint-online-cloud).

You can add more documents to the training data of the beta entity extractor model
: If you added and labeled 20 documents to train a model, and now want to continue to improve the model's performance, you can add more documents.
* [Databases for MongoDB](https://cloud.ibm.com/catalog/services/databases-for-mongodb)
* [Messages for RabbitMQ](https://cloud.ibm.com/catalog/services/messages-for-rabbitmq)



All of these services share the same characteristics:



* For high availability, they are deployed in clusters. Details can be found in the documentation of each service:



* [PostgreSQL](https://cloud.ibm.com/docs/databases-for-postgresql?topic=databases-for-postgresql-high-availabilityhigh-availability)
* [EnterpriseDB](https://cloud.ibm.com/docs/databases-for-enterprisedb?topic=databases-for-enterprisedb-high-availability)
* [DataStax](https://cloud.ibm.com/docs/databases-for-cassandra?topic=databases-for-cassandra-high-availability)
* [Redis](https://cloud.ibm.com/docs/databases-for-redis?topic=databases-for-redis-high-availabilityhigh-availability)
* [ElasticSearch](https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=databases-for-elasticsearch-high-availabilityhigh-availability)
* [etcd](https://cloud.ibm.com/docs/databases-for-etcd?topic=databases-for-etcd-high-availabilityhigh-availability)
* [MongoDB](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-high-availabilityhigh-availability)
* [RabbitMQ](https://cloud.ibm.com/docs/messages-for-rabbitmq?topic=messages-for-rabbitmq-high-availability)



* Each cluster is spread over multiple zones.
* Data is replicated across the zones.
* Users can scale up storage and memory resources for an instance.
IBM Cloudant docs 

The IBM Cloudant portfolio includes two complementary, document-oriented NoSQL database-as-a-service offerings on the IBM Cloud. In addition to the comprehensive information in the IBM Cloudant docs, links to additional sources of information are provided below.

 Developer tools 

[API & SDK reference](https://cloud.ibm.com/apidocs/cloudantintroduction)[CLI reference](https://cloud.ibm.com/docs/Cloudant-cli-plugin/Cloudant-cli-plugin-cloudant-cli)[Terraform reference](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/resources/cloudant)

 Recommended content 

[IBM Cloudant plans Describes the plans you can use with IBM Cloudant and how IBM Cloudant calculates usage.](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-ibm-cloud-public)[IBM Cloudant security Describes IAM, security compliance, and securing your connection and data.](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-managing-access-for-cloudant)[Try IBM Cloudant for free Describes the steps to get started with IBM Cloudant for free.](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-getting-started-with-cloudant)

 Video library 

[!
For more information about all file formats, their file extensions and content types, and how and when to specify the file extension or content type, see [Supported file formats](https://cloud.ibm.com/docs/language-translator?topic=language-translator-document-translator-tutorialsupported-file-formats).

Documentation updates for bidirectional translation
: The documentation now states that the service correctly translates from and to bidirectional languages that are written left-to-right and right-to-left (for example, Arabic, Hebrew, and Urdu). This is not new functionality, but the documentation had failed to emphasize the information.





 5 August 2022 

The following changes were made available on 1 August 2021. They are now documented.

Most document formats for translation are now generally available
: Most supported file formats for document translation are now generally available (GA). In addition, subtitle formats for documentation translation are also GA. The PDF format remains experimental. For more information, see [Supported file formats](https://cloud.ibm.com/docs/language-translator?topic=language-translator-document-translator-tutorialsupported-file-formats).





 15 June 2022 

The following changes were made available on 1 November 2021. They are now documented.

New customizable translation models
: Added new customizable translation models for the following languages:



* English to Kannada (en-kn)
* English to Marathi (en-mr)
* English to Punjabi (Indian) (en-pa)
* English to Punjabi (Pakistani) (en-pa-PK)
* Kannada to English (kn-en)
* Marathi to English (mr-en)
* Punjabi (Indian) to English (pa-en)
* Punjabi (Pakistani) to English (pa-PK-en)



The Kannada language is new; it is not identifiable. All of the languages except for Kannada already existed and are identifiable. For more information, see [Supported languages for translation](https://cloud.ibm.com/docs/language-translator?topic=language-translator-translation-models).

Improved customizable translation models
The following table lists services that send logs to IBM Log Analysis:



Table 4. List of database services

 Service Description More info 

 [IBM Cloudant](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-getting-started-with-cloudant) IBM Cloudant is a document-oriented database as a service (DBaaS). It stores data as documents in JSON format. [More info](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-log-analysis-integration) 
 [IBM Cloud Databases for Elasticsearch](https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=databases-for-elasticsearch-getting-started) IBM Cloud Databases for Elasticsearch is a managed Elasticsearch service that is hosted in the IBM Cloud and integrated with other IBM Cloud services. [More info](https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=cloud-databases-logging) 
 [IBM Cloud® Databases for EnterpriseDB](https://cloud.ibm.com/docs/databases-for-enterprisedb?topic=databases-for-enterprisedb-getting-started) Databases for EnterpriseDB is a database engine that optimizes the built-in features of PostgreSQL. [More info](https://cloud.ibm.com/docs/databases-for-enterprisedb?topic=cloud-databases-logging) 
 [IBM Cloud® Databases for DataStax](https://cloud.ibm.com/docs/databases-for-cassandra?topic=databases-for-cassandra-getting-started) IBM Cloud® Databases for DataStax is a scale-out NoSQL database that is built on Apache Cassandra. It’s designed to power real-time applications with high availability and massive scalability. [More info](https://cloud.ibm.com/docs/databases-for-cassandra?topic=cloud-databases-logging)
New support for Notes attachments
: Added support for attachments in the Notes data source. For more information, see [Notes](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collection-typesconnectnotes)

New web crawl scheduling option
: You can specify the exact time that you would like your crawls to run for any data source, giving you the flexibility to run them at the times you prefer. For more information, see [Configuring Cloud Pak for Data data sources](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collection-types).

New Facet creation in Content Miner
: You can now create Facet groups in a Content Miner application.

New custom crawler creation
: Added the option to create your own custom crawler plug-in. For more information, see [Building a Cloud Pak for Data crawler plug-in](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-crawler-plugin-build)Note: Any custom code used with Watson Discovery is the responsibility of the developer and is not covered by IBM support.

Change to Dynamic Facets
: Dynamic Facets are no longer enabled by default in Document Retrieval projects.





 2.1.4 release, 2 September 2020 

New release now available
: IBM Watson® Discovery for IBM Cloud Pak® for Data version 2.1.4 is available.

New Notes connector
: Crawl Notes version 9.0.1 systems. For more information, see [Notes connector](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collection-typesconnectnotes).

New Enable proxy settings in multiple connectors
Release notes for IBM Cloud® Databases for MongoDB 

Use these release notes to learn about the latest updates to IBM Cloud® Databases for MongoDB that are grouped by date or build number.



 30 May 2023 

Databases for MongoDB version 5 Preferred
: Databases for MongoDB version 5 is the preferred version. For more information, see [Upgrading to a new Major Version](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-upgrading&interface=ui).





 22 May 2023 

Setting up disk alerts for disk utilization tutorial
: In this tutorial, you use the IBM Cloud API and the [IBM Cloud CLI](https://cloud.ibm.com/docs/cli?topic=cli-getting-started) to set up an alert that emails you whenever the disk utilization of your database exceeds 90%. This specific example creates an alert on a Databases for Elasticsearch deployment, but it is applicable to all the databases in the IBM Cloud Databases catalog. For more information, see [Setting up disk alerts for disk utilization](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-disk-util-alert-tutorial).





 12 December 2022 

Point-in-time Recovery
: IBM Cloud® Databases for MongoDB Enterprise Edition offers PITR using any timestamp greater than the earliest available recovery point. To discover the earliest recovery point through the API, use the [point-in-time-recovery timestamp endpoint](https://cloud.ibm.com/apidocs/cloud-databases-api/cloud-databases-api-v5capability). For more information, see [Point-in-time Recovery (PITR)](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-pitr&interface=ui).





 19 October 2022 

Deploying and Connecting a Cloud Databases Instance Tutorial