vania-janet/multiturn-rag-retrieval-data huggingface dataset:
You can get status information for a specific document by using the [Get document details](https://cloud.ibm.com/apidocs/discovery-datagetdocument) method. <br>The objects document_counts, disk_usage, training_status, and crawl_status are not present in the response body in v2. Training status is returned in the Get project method response. The other information is not available in v2. For example, you cannot get the document count for a collection and cannot get the crawl status for a collection that connects to an external data source in v2. In v2, you can get information about the enrichments that are applied to the collection. 
 Update a collection v2 uses POST instead of PUT. In v2, you can update the enrichments that are applied to the documents in the collection by specifying an optional enrichments object. <br>The v2 response doesn't include the status and configuration_id fields. 









 Query modifications 

The method that was available in v1 for configuring tokenization programmatically is not supported in the v2 API.



Query modifications API support details

 v1 API v2 API 

 [Tokenization dictionary API](https://cloud.ibm.com/apidocs/discoverycreatetokenizationdictionary) Not available. 
 [Expansions v1 API](https://cloud.ibm.com/apidocs/discoverylistexpansions) [Expansions v2 API](https://cloud.ibm.com/apidocs/discovery-datalistexpansions) 
 [Stopwords v1 API](https://cloud.ibm.com/apidocs/discoverygetstopwordliststatus) [Stopwords v2 API](https://cloud.ibm.com/apidocs/discovery-datagetstopwordlist) 







 Documents 



Documents API support details

 Action v1 API v2 API 

 List documents Not available from the v1 API [GET /v2/projects/{project_id}/collections/{collection_id}/documents](https://cloud.ibm.com/apidocs/discovery-datalistdocuments)
[""](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-operatorsphrase) Phrase query enriched_text.concepts.text:"IBM Watson" 
 [(), ]](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-operatorsnestedquery) Nested groups filter-entities:(text:Turkey,type:Location) 
 [|](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-operatorsor) or query-enriched.entities.text:Google IBM 
 [,](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-operatorsand) and query-enriched.entities.text:Google,IBM 
 [<=, >=, >, <](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-operatorscomparisons) Numerical comparisons enriched_text.sentiment.document.score>0.679 
 [^x](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-operatorsmultiplier) Score multiplier text:IBM^3 
 [](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-operatorswildcard) Wildcard query-enriched_text.concepts.text:pre* 
 [n](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-operatorsvariation) String variation query-enriched_text.entities.text:cat1 
 [:](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-operatorsexists) Exists title:* 
 [!](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-operatorsdnexist) Does not exist title!* 







 Aggregations 

Aggregations return a set of data values. The following table lists the available aggregations.
A goal to aim for is to eventually have at least 50 annotations for each entity type and 50 for each relation type in the document collection.
* Again, documents should represent the breadth of the subject matter that the application will cover, but in the case of skewed frequency-of-occurrence of entity types and relation types, try to get at least 50 exemplars of each type, more for entity types that have mentions which tend to be phrases.
* The set that you create for training must contain at least 10 annotated documents.



When you are ready to create and train the model, documents that you add to the workspace can be divided into sets that are used as training data, test data, and blind data. The separate data sets are important for assessing model performance.

You can add documents in the following ways. For more information about the supported document types, size limits, and other information, see [Creating a workspace > Summary of inputs, outputs, and limitations](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-create-projectwks_formats).



* A two-column CSV file in UTF-8 format
* Text files in UTF-8 format
* HTML files
* PDF files (scanned and password-protected files are not supported)
* Microsoft Word DOC or DOCX files (password-protected files are not supported)
* A .zip file that contains documents downloaded from a Knowledge Studio workspace
* A .zip file that contains files in UIMA CAS XMI format





 CSV files 

You can upload a two-column CSV file that contains sample text from your local machine. Upload one CSV file at a time. The first column in the CSV file specifies the file name of the document. The second column in the file contains the document text. For an example of the required format, see the [documents-new.csv!
[Support plans ](https://cloud.ibm.com/docs/get-support?topic=get-support-support-plans)

You can choose a Basic, Advanced, or Premium support plan to customize your IBM Cloud support experience for your business needs.

[Viewing IBM Cloud status ](https://cloud.ibm.com/docs/get-support?topic=get-support-viewing-cloud-status)

The Status page is the central place to find details about major incidents that affect the IBM Cloud platform and services.

[Working with support cases ](https://cloud.ibm.com/docs/get-support?topic=get-support-open-case)

You can choose a Basic, Advanced, or Premium support plan to customize your IBM Cloud support experience for your business needs.

[Platform prerequisites ](https://cloud.ibm.com/docs/overview?topic=overview-prereqs-platform)

The prerequisites for using the IBM Cloud platform are limited, but we do have a few.

![Side card image](https://cloud.ibm.com/media/docs/images/homepage/data-fabric-800x450.Original.jpg)

 Brand new to IBM Cloud? 

[Getting started on IBM Cloud checklist](https://cloud.ibm.com/docs/overview?topic=overview-get-started-checklist)[How does the platform work?](https://cloud.ibm.com/docs/overview?topic=overview-whatis-platform)[Navigating the console](https://cloud.ibm.com/docs/overview?topic=overview-ui)[Estimating your costs](https://cloud.ibm.com/docs/billing-usage?topic=billing-usage-cost)[IBM Cloud training](https://www.ibm.com/training/cloud)

 Explore popular documentation 

Read the docs from our most popular services.
Release notes for IBM Cloud® Databases for Elasticsearch 

Use these release notes to learn about the latest updates to IBM Cloud® Databases for Elasticsearch that are grouped by date or build number.



 22 June 2023 

Elasticsearch 8.7.0 (preview)
: For more information, see [IBM Cloud® Databases for Elasticsearch Enterprise Plan](https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=databases-for-elasticsearch-pricingelastic-enterprise-pricing).





 18 May 2023 

Setting up disk alerts for disk utilization tutorial
: In this tutorial, you use the IBM Cloud API and the [IBM Cloud CLI](https://cloud.ibm.com/docs/cli?topic=cli-getting-started) to set up an alert that emails you whenever the disk utilization of your database exceeds 90%. This specific example creates an alert on a Databases for Elasticsearch deployment, but it is applicable to all the databases in the IBM Cloud Databases catalog. For more information, see [Setting up disk alerts for disk utilization](https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=databases-for-elasticsearch-disk-util-alert-tutorial).





 03 May 2023 

IBM Cloud® Databases for Elasticsearch Enterprise Plan
: Our strategic partnership with [Elastic](https://www.elastic.co/about/) since January 2023 means that we are able to offer more and richer functionality, as well as world-class levels of support. IBM Cloud® Databases for Elasticsearch Enterprise Plan Version 7.17 is the current supported version of Databases for Elasticsearch and is offered under our Enterprise Plan. This version includes functionality that is not available in our Standard plan. IBM Cloud® Databases for Elasticsearch Standard Plan Versions 7.9 and 7.10 are available on Cloud Databases under the Standard Plan, but will be End of Life (EOL) by 30 November 2023.
A historical record of tasks from any time period is available through the [IBM Cloud® Activity Tracker integration](https://cloud.ibm.com/docs/databases-for-mongodb?topic=cloud-databases-activity-tracker).





 Endpoints 

The Endpoints pane within the Overview pane contains connection strings for your deployment. Each tab contains connection information tailored to the type of connection or the protocol that uses it. Basic information includes things like hostname and port, as well as the TLS self-signed certificate, TLS/SSL parameters, and the default database of your deployment.

Reference tables for the different connection types are available on the [Getting Credentials and Connection Strings](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-connection-strings) page.

Connection strings reflect whether your deployment uses public endpoints, private endpoints or both. You can configure which endpoints are available on your deployment. For more information, see the [Service Endpoints Integration](https://cloud.ibm.com/docs/databases-for-mongodb?topic=cloud-databases-service-endpoints) page.

You can manage your Databases for MongoDB service through the Cloud Databases API. This panel provides the essential information for using the API. For more information about the Cloud Databases API, see the [API reference](https://cloud.ibm.com/apidocs/cloud-databases-api) page.





 Resources 

The resources tab contains information and configuration options on the size and resource usage of your deployment. You can



* [Scale disk, memory, and CPU](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-resources-scaling)
* [Configure Autoscaling](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-autoscaling)







 Backups 

The Backups tab is the UI for managing your deployments backups. All of the available backups are listed with their timestamps.
Cloud Databases CLI 

The Cloud Databases CLI plug-in offers extra methods of accessing the capabilities of the Cloud Databases services. You can use the Cloud Databases CLI to manage and connect to the following services:



* [IBM Cloud® Databases for PostgreSQL](https://cloud.ibm.com/docs/databases-for-postgresql)
* [IBM Cloud® Databases for MySQL](https://cloud.ibm.com/docs/databases-for-mysql)
* [IBM Cloud® Databases for EnterpriseDB](https://cloud.ibm.com/docs/databases-for-enterprisedb)
* [IBM Cloud® Databases for Redis](https://cloud.ibm.com/docs/databases-for-redis)
* [IBM Cloud® Databases for Elasticsearch](https://cloud.ibm.com/docs/databases-for-elasticsearch)
* [IBM Cloud® Databases for etcd](https://cloud.ibm.com/docs/databases-for-etcd)
* [IBM Cloud® Messages for RabbitMQ](https://cloud.ibm.com/docs/messages-for-rabbitmq)
* [IBM Cloud® Databases for MongoDB](https://cloud.ibm.com/docs/databases-for-mongodb)
* [IBM Cloud® Databases for DataStax](https://cloud.ibm.com/docs/databases-for-cassandra)



The Cloud Databases CLI plug-in requires [IBM Cloud CLI](https://cloud.ibm.com/docs/cli?topic=cli-getting-started) to be installed.



 The IBM Cloud CLI 

The IBM Cloud CLI is a general-purpose developer tool that provides access to your IBM Cloud account and services through a command-line interface.

An introduction and installation instructions are available on the [IBM Cloud CLI Getting Started page](https://cloud.ibm.com/docs/cli?topic=cli-getting-started).
[IBM Cloud Databases for PostgreSQL](https://cloud.ibm.com/docs/databases-for-postgresql?topic=databases-for-postgresql-getting-started) IBM Cloud Databases for PostgreSQL is a managed PostgreSQL service that is hosted in the IBM Cloud and integrated with other IBM Cloud services. [More info](https://cloud.ibm.com/docs/databases-for-postgresql?topic=cloud-databases-logging) 
 [IBM Cloud Messages for RabbitMQ](https://cloud.ibm.com/docs/messages-for-rabbitmq?topic=messages-for-rabbitmq-getting-started) IBM Cloud Messages for RabbitMQ is a managed RabbitMQ service that is hosted in the IBM Cloud and integrated with other IBM Cloud services. [More info](https://cloud.ibm.com/docs/messages-for-rabbitmq?topic=cloud-databases-logging) 
 [IBM Cloud Databases for Redis](https://cloud.ibm.com/docs/databases-for-redis?topic=databases-for-redis-getting-started) IBM Cloud Databases for Redis is a managed service that is hosted in the IBM Cloud and integrated with other IBM Cloud services. [More info](https://cloud.ibm.com/docs/databases-for-redis?topic=cloud-databases-logging) 
 [IBM Cloud Hyper Protect DBaaS for MongoDB](https://cloud.ibm.com/docs/hyper-protect-dbaas-for-mongodb?topic=hyper-protect-dbaas-for-mongodb-gettingstarted) IBM Cloud Hyper Protect DBaaS for MongoDB is a managed highly secure MongoDB service that is hosted in the IBM Cloud and integrated with other IBM Cloud services. [More info](https://cloud.ibm.com/docs/hyper-protect-dbaas-for-mongodb?topic=hyper-protect-dbaas-for-mongodb-sendlogs)
* If your deployment is not using public endpoints, take [these additional steps](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-service-endpointsprivate-endpoints) to configure private endpoint access.

MongoDB cannot support both [public and private endpoints simultaneously](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-service-endpoints&interface=uiprovisioning-service-endpoints). This cannot be changed after provisioning.
* Finally, review the [Getting to production](https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-best-practices) documentation for general guidance on setting up a basic IBM Cloud® Databases for MongoDB deployment.







 IBM Cloud® Databases for MongoDB Enterprise Edition 

IBM Cloud® Databases for MongoDB Enterprise Edition (EE) offers more functionality, including:



* Automatic, client-side encryption
* [Audit logging](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-auditlogging)
* Federal Information Processing Standard (FIPS) approved encryption
* [Ops Manager](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-ops-manager), which allows you to manage, monitor, and back up MongoDB deployments
* [MongoDB EE Analytics Add-On](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-mongodbee-analytics), which allows you to make your query data compatible with business intelligence (BI) tools





 Provisioning IBM Cloud® Databases for MongoDB EE 

To take advantage of the additional functions of IBM Cloud® Databases for MongoDB EE, choose Enterprise during your IBM Cloud® Databases for MongoDB provisioning process.







 Step 2: Connect with MongoDB Compass
Instead, keep orders separate as their own document type, referencing the customer ID. Now the model is immutable. To add an order, I create a new order document in the database, which cannot generate conflicts.

To be able to retrieve all orders for a specific customer, we can employ a view, which we cover later.

Avoid constructs that rely on updates to parts of existing documents, where possible. Bad data models are often hard to change after you’re in production.

The previous pattern can be solved efficiently by using partitioned databases, which are covered in greater detailed later.

For more information, see the following documentation:



* IBM Cloudant guide to [data modeling](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-five-tips-for-modeling-your-data-to-scale-faq)
* [Database partitions](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-database-partitioning)







 Keep documents small 

IBM Cloudant imposes a max doc size of 1 MB. This limit does not mean that a close-to-1-MB document size is a good idea. On the contrary, if you find you are creating documents that exceed single-digit KB, you probably need to revisit your model. Several things in IBM Cloudant become less performant as documents grow. JSON decoding is costly, for example.

Let's look at the following sections: Documents must group data that mostly changes together and Keep documents small. It’s worth stressing that models that rely on updates have a maximum volume limit of 1 MB, the cut-off for document size. This size isn’t what you want.





 Avoid using attachments 

IBM Cloudant has support for storing attachments alongside documents, a long-standing feature it inherits from CouchDB. If you use IBM Cloudant as a backend for a web application, you can also store small icons and other static assets such as CSS and JavaScript files with the data.

You must consider a few things before you use attachments in IBM Cloudant today, especially if you’re looking at larger assets such as images and videos:



1.
For more information about adding a document classifier by using the product user interface, see [Using the Content Mining application](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-doc-classifier).

The document classifier endpoints are not supported in the SDKs currently.





 4.0.6 release, 1 March 2022 

IBM Watson® Discovery Cartridge for IBM Cloud Pak® for Data 4.0.6 is available.

Multitenancy is now supported
: An administrator can now create up to 10 instances of the Discovery service per deployment, which means that more teams can work on discrete Discovery projects at the same time.

Simpler installation and management of custom connectors
: The manage_custom_crawler.sh script was improved to make it easier for you to install and manage your custom connectors in a multitenant environment. For more information, see [Installing a custom crawler](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-install-connector).

Security vulnerabilities were addressed
: The following security patches were applied:



* [Security Bulletin: IBM Watson Discovery for IBM Cloud Pak for Data affected by vulnerability in Java](https://www.ibm.com/support/pages/node/6556970)
* [Security Bulletin: IBM Watson Discovery for IBM Cloud Pak for Data affected by vulnerability in Logback](https://www.ibm.com/support/pages/node/6556972)
* [Security Bulletin: IBM Watson Discovery for IBM Cloud Pak for Data affected by vulnerability in Apache Log4j](https://www.ibm.com/support/pages/node/6556974)



Features that are not available in this release
: The following features are generally available from managed IBM Cloud deployments at the time of this release, but not from installed deployments:



* Home page updates
* Answer finding
* Access to guided tours from the page header







 4.0.5 release, 26 January 2022 

IBM Watson® Discovery Cartridge for IBM Cloud Pak® for Data 4.0.5 is available.

A security vulnerability was addressed
: Support for Parquet schema evolution through the [MERGE SCHEMA](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceexternalTableSpec) sub clause for STORED AS PARQUET input data. Check out the new samples in the UI.

New table transformation functions
: Support for [CLEANCOLS](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencetableTransformer) table transformation function that generically cleanses all input column names from characters that are not supported by the Parquet target format. Check out the new Samples.
: Support for [FLATTEN](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencetableTransformer) table transformation function that generically flattens all nested input columns into a flat hierarchy, allowing to easily work with, for example, JSON input data and write the results out to flat CSV files. Check out the new samples in the UI.





 June 2019 

Data Engine now available in Frankfurt
: Data Engine is available in Frankfurt, Germany. When you provision new instances, you can select whether it is being provisioned in Dallas or in Frankfurt.

Support for time series SQL functions
: Support for [time series SQL functions](https://cloud.ibm.com/docs/services/sql-query?topic=sql-query-ts_intro) to process time series data, for example, to identify trends and to predict future values based on these trends.





 May 2019 

Updates to the [SQL reference](https://cloud.ibm.com/docs/services/sql-query?topic=sql-query-sql-reference)
: [JOBPREFIX JOBID/NONE](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencecosResultClause), you can specify whether you want the job ID to be appended to the target prefix, or not.
* Users are responsible for downloading files to EU-supported locations.



To learn how to configure archiving for your auditing instance, see [Archiving logs](https://cloud.ibm.com/docs/services/activity-tracker?topic=activity-tracker-archiving).





 Step 8. Querying archived events with SQL Query service 

Data Engine provides a serverless, no-ETL solution to easily query data stored in COS. [Learn more](https://cloud.ibm.com/docs/services/sql-query?topic=sql-query-overview).

You can use this service to analyze data from archived files in COS.

Once you have SQL Query running on IBM Cloud, you can immediately start querying your data using the SQL Query user interface, programmatically by using either the REST API or the Python ibmcloudsql library, or write a serverless function by using Cloud Functions.

When you query events, consider the following information:



* You must provision an instance of the Data Engine service in Frankfurt.
* You must restrict user access to work with that instance. Users need the platform viewer role to launch the UI, and the service writer role to run queries.
* When you open the UI, the Data Engine service automatically generates a unique COS bucket that will store all of the results as CSV files from your SQL queries. To make sure that you are using an EU-supported bucket, create one. You can specify your custom bucket to store results.
[checkbox](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/checkbox.png) Chunk large documents into many smaller documents so you can apply more targeted enrichments to the content. [Using Smart Document Understanding](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-configuring-fields) 
 ![checkbox](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/checkbox.png) Enhance your data by applying built-in NLU enrichments. [Applying prebuilt enrichments](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-nlu) 
 ![checkbox](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/checkbox.png) Identify and promote terms and patterns from your data with special significance to your use case. [Adding domain-specific resources](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain) 
 ![checkbox](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/checkbox.png) Submit test queries to assess the results. [Testing your project](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-test) 
 !