vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Since the index does not store the document, you can access any field without thinking ahead about what to store in the index.
* The disadvantage is that getting the document back is more costly than the alternative of emitting data into the index itself. First, the database has to look up the requested key in the index and then read the associated document. Also, if you’re reading the whole document, but need only a single field, you’re making the database read and transmit data that you don’t need.



This example also means that a potential race condition exists here. The document might change, or be deleted, between the index and document read (although unlikely in practice).

Emitting data into the index (a so-called “projection” in relational algebra terms) means that you can fine-tune the exact subset of the document that you need. In other words, you don’t need to emit the whole document. Emit a value that represents only the data you need in the app that is a cut-down object with minimal details, for example:

emit(doc.indexed_field, {name: doc.name, dob: doc.dob});

If you change your mind on what fields you want to emit, the index needs rebuilding.

IBM Cloudant Query’s JSON indexes use views this way under the hood. IBM Cloudant Query can be a convenient replacement for some types of view queries, but not all. Do take the time to understand when to use one or the other.



* IBM Cloudant Query [docs](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-query)
* IBM Cloudant guide to using [views](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-using-viewsusing-views)
* Performance implications of using [include_docs](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-using-viewsmulti-document-fetching)







 Never rely on the default behavior of IBM Cloudant Query’s no-indexing 

It’s tempting to rely on IBM Cloudant Query's ability to query without creating explicit indexes.
IBM Cloudant has support for storing attachments alongside documents, a long-standing feature it inherits from CouchDB. If you use IBM Cloudant as a backend for a web application, you can also store small icons and other static assets such as CSS and JavaScript files with the data.

You must consider a few things before you use attachments in IBM Cloudant today, especially if you’re looking at larger assets such as images and videos:



1. IBM Cloudant is expensive as a block store.
2. IBM Cloudant’s internal implementation is not efficient in handling large amounts of binary data.



So, slow and expensive.

IBM Cloudant is acceptable for small assets and occasional use. As a rule, if you need to store binary data alongside IBM Cloudant documents, it’s better to use a separate solution more suited for this purpose. You need store only the attachment metadata in the IBM Cloudant document. Yes, that means you need to write some extra code to upload the attachment to a suitable block store of your choice. Verify that it succeeded before you store the token or URL to the attachment in the IBM Cloudant document.

Your databases are smaller, cheaper, faster, and easier to replicate. For more information, see the following websites:



* IBM Cloudant docs on [attachments](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-how-to-use-attachments)
* Detaching IBM Cloudant attachments to [Object Storage](https://medium.com/codait/detaching-cloudant-attachments-to-object-storage-with-serverless-functions-99b8c3c77925)







 Fewer databases are better than many 

If you can, limit the number of databases per IBM Cloudant account to 500 or fewer. While this particular number is not magic (IBM Cloudant can safely handle more), several use cases exist that are adversely affected by large numbers of databases in an account.

The replicator scheduler has a limited number of simultaneous replication jobs that it is prepared to run. As the number of databases grows, the replication latency is likely to increase if you try to replicate everything contained in an account.
: Enrichments from IBM Watson™ Knowledge Studio custom models are not limited, but split documents into 10-kB chunks. No relationships are annotated across chunk boundaries.





 Query limitations 

Update : Query concepts [Update](https://cloud.ibm.com/docs/discovery?topic=discovery-query-concepts)

Query limitations : Excessive query load can cause the search-index process to restart automatically. : Applications that issue queries must enforce reasonable limits on the number of concurrent queries.







 Known issues 

Updates : API Reference [Update: API reference](https://cloud.ibm.com/apidocs/discovery) : Tooling [Update: tooling](https://cloud.ibm.com/docs/discovery?topic=discovery-getting-started) : Data Crawler [Update: Data Crawler](https://cloud.ibm.com/docs/discovery?topic=discovery-adding-content-with-data-crawler) : Enrichments [Update: Enrichments](https://cloud.ibm.com/docs/discovery?topic=discovery-configserviceadding-enrichments) : Adding content [Update: Adding content](https://cloud.ibm.com/docs/discovery?topic=discovery-addcontent)

Known issues : You cannot delete a document by using the tooling. If you need to delete a document, you must use the API's [Delete a document](https://cloud.ibm.com/apidocs/discoverydelete-a-document) method as described in the API reference. : The API does not currently support getting a list of notices (warnings and error) that are generated during document ingestion. The tooling is therefore unable to show a list of ingestion notices, and there is no easy way to determine which, if any, documents crawled by the Data Crawler failed to be ingested. : Document status information is not always accurate. : If an ingestion operation takes longer than the configured timeout of 10 minutes, the service reports that the document is not known to the service until the ingestion operation completes. After the operation completes, the document status is available and accurate.
A goal to aim for is to eventually have at least 50 annotations for each entity type and 50 for each relation type in the document collection.
* Again, documents should represent the breadth of the subject matter that the application will cover, but in the case of skewed frequency-of-occurrence of entity types and relation types, try to get at least 50 exemplars of each type, more for entity types that have mentions which tend to be phrases.
* The set that you create for training must contain at least 10 annotated documents.



When you are ready to create and train the model, documents that you add to the workspace can be divided into sets that are used as training data, test data, and blind data. The separate data sets are important for assessing model performance.

You can add documents in the following ways. For information about the supported document types, size limits, and other information, see [Creating a workspace > Summary of inputs, outputs, and limitations](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-create-projectwks_formats).



* A two-column CSV file in UTF-8 format
* Text files in UTF-8 format
* HTML files
* PDF files (scanned and password-protected files are not supported)
* Microsoft Word DOC or DOCX files (password-protected files are not supported)
* A ZIP file that contains documents downloaded from a Knowledge Studio workspace
* A ZIP file that contains files in UIMA CAS XMI format





 CSV files 

You can upload a two-column CSV file that contains sample text from your local machine. Upload one CSV file at a time. The first column in the CSV file specifies the file name of the document. The second column in the file contains the document text. For an example of the required format, see the [documents-new.csv](https://watson-developer-cloud.github.io/doc-tutorial-downloads/knowledge-studio/documents-new.csv) file in the tutorial sample files.





 PDF files
* [Query reference](https://cloud.ibm.com/docs/discovery?topic=discovery-query-reference) (includes the list of parameters, operators, and aggregations available in the Discovery Query Language)









 The Discovery data schema 

Start out by getting to know the Discovery JSON. To understand how to build a query using the Discovery Query Language, you need to be familiar with the JSON produced by Discovery after it enriches the documents in your collection. After you are familiar with the data schema of your documents, it is easier to write queries in the Discovery Query Language. Review the following three ways to write queries in the Discovery Query Language:



1. In the Discovery Tooling, open the Manage data screen, choose the collection that contains the IBM Press Releases. Click the View Data Schema button. The View data schema screen displays the fields and values in your transformed documents two ways: by document (Document view), or by field (Collection view). A maximum of 50 documents displays in Document view. Collection view displays the fields in the entire collection.

In the Collection view, under enriched_text, you can examine the enrichments you applied to your collection. Click on categories, concepts, entities, and sentiment to see how your collection was enriched with Watson insights.
2. Run an "empty" query to view the JSON. On the View data schema screen, click the Build queries button, then click Run Query. The results display on the right, under two tabs, Summary (an overview of the query results) and JSON. Start by opening the JSON tab.



* Each of the four documents is preceded by an id number.
* Scroll down to the enriched_text field. Examine each enrichment to learn about the JSON fields you can query on.

![Default configuration fields](https://cloud.ibm.com/docs-content/v1/content/ded4adc3ea0bd2a81b113c579f2b1183926da211/discovery/images/JSON.png)
* entities - Start by finding the text field, and examine the other enrichment information.
* [Scale disk, memory, and CPU](https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=databases-for-elasticsearch-resources-scaling)
* [Configure Autoscaling](https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=databases-for-elasticsearch-autoscaling)







 Backups 

The Backups tab is the UI for managing your deployments backups. All of the available backups are listed with their timestamps. Click a backup to grab its ID or to restore it into a new deployment. More information is on the [Managing Backups](https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=databases-for-elasticsearch-dashboard-backups) page.





 Observability 

The Observability tab provides access to the IBM Cloud Monitoring, logging, and event tracking integrations available for your deployment.



* [IBM Cloud® Activity Tracker](https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=databases-for-elasticsearch-activity-tracker)
* [IBM® Log Analysis](https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=databases-for-elasticsearch-logging)
* [IBM Cloud® Monitoring](https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=databases-for-elasticsearch-monitoring)







 Settings 

The Settings tab contains the UI for many of the tunable settings for your deployment. You can



* view encryption details. Encryption at rest is enabled for all Databases for Elasticsearch deployments. If you brought your own encryption key from [Key Protect](https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=databases-for-elasticsearch-key-protect), the panel provides a link to your Key Protect instance and the Encryption Key field has the name of the key.
Microsoft SharePoint On Prem 

Crawl documents that are stored in an on-premises Microsoft SharePoint data source.

IBM Cloud Pak for Data

IBM Cloud Pak for Data only

This information applies only to installed deployments. For more information about connecting to an on-premises SharePoint site from a managed deployment, see [SharePoint On Prem](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-sharepoint-onprem-cloud).



 What documents are crawled 



* Only documents that are supported by Discovery are crawled; all others are ignored. For more information, see [Supported file types](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionssupportedfiletypes).
* Document-level security is supported. When this option is enabled, your users can crawl and query the same content that they can access when they are logged in to SharePoint. For more information, see [Supporting document-level security](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collection-typesconfiguredls).
* When a source is recrawled, new documents are added, updated documents are modified to the current version, and deleted documents are deleted from the collection's index.
* All Discovery data source connectors are read-only. Regardless of the permissions that are granted to the crawl account, Discovery never writes, updates, or deletes any content in the original data source.







 Data source requirements 

In addition to the [data source requirements](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collection-typesrequirements) for all installed deployments, your SharePoint On Prem data source must meet the following requirements:



* The data source connection supports SharePoint 2013, 2016, or 2019.
* You must obtain any required service licenses for the data source that you want to connect to. For more information about licenses, contact the system administrator of the data source.
[Learn more](https://cloud.ibm.com/docs/sql-query?topic=sql-query-overview).

You can use this service to analyze data from archived files in COS.

Once you have SQL Query running on IBM Cloud, you can immediately start querying your data using the SQL Query user interface, programmatically by using either the REST API or the Python ibmcloudsql library, or write a serverless function by using Cloud Functions.

When you query events:



* You must provision an instance of the Data Engine service.
* You must restrict user access to work with that instance. Users need the platform viewer role to launch the UI, and the service writer role to run queries.
* When you open the UI, the Data Engine service automatically generates a unique COS bucket that will store all of the results as CSV files from your SQL queries. To make sure that you are using a custom bucket, create one. You can specify your custom bucket as the place to to store results.







 IAM permissions to configure archiving 

To configure archiving, you need the following permissions:



 IBM Cloud Activity Tracker service 

The following table lists the minimum roles that a user must have to be able to launch the IBM Cloud Activity Tracker web UI, and configure archiving through the UI or by using the API:



Table 1. IAM roles

 Role Permission granted 

 Platform role: Viewer Allows the user to view the list of service instances in the Observability dashboard. 
 Service role: Manager Allows the user to launch the web UI and configure archiving through the web UI or by using the API. 



For more information on how to configure policies for a user, see [Granting user permissions to a user or service ID](https://cloud.ibm.com/docs/services/activity-tracker?topic=activity-tracker-iam_view_eventsiam_view_events).





 IBM Cloud Object Storage service 

The following table lists the roles that a user can have to complete the actions required to configure the IBM Cloud Object Storage service:



Table 2. Roles and actions

 Service Roles Action
<br>IBM Cloudant documentation references more request body fields (r, bookmark). 
 GET /{db}/_index <br>[Cloudant](https://cloud.ibm.com/apidocs/cloudantgetindexesinformation), [CouchDB](https://docs.couchdb.org/en/latest/api/database/find.htmlget--db-_index) List indexes. 
 POST /{db}/_index <br>[Cloudant](https://cloud.ibm.com/apidocs/cloudantpostindex), [CouchDB](https://docs.couchdb.org/en/latest/api/database/find.htmlpost--db-_index) Create an index on a database. 
 DELETE /{db}/_index/_design/{ddoc}/{type}/{index} <br>[Cloudant](https://cloud.ibm.com/apidocs/cloudantdeleteindex), [CouchDB](https://docs.couchdb.org/en/latest/api/database/find.htmldelete--db-_index-designdoc-json-name) Delete an index. 
 GET /{db}/_local/{doc_id} <br>[Cloudant](https://cloud.ibm.com/apidocs/cloudantgetlocaldocument), [CouchDB](https://docs.couchdb.org/en/latest/api/local.htmlget--db-_local-docid) Get the specified local document. 
 PUT /{db}/_local/{doc_id} <br>[Cloudant](https://cloud.ibm.com/apidocs/cloudantputlocaldocument), [CouchDB](https://docs.couchdb.org/en/latest/api/local.htmlput--db-_local-docid) Store the specified local document. <br>Local documents aren't replicated to other databases.
How do I access and query data that has been archived for long term storage in COS? 

To access data, you can download the archived file locally.

To query the data, you can also use a service like SQL Query to query your COS archives and get information based on queries that you define. [Learn more](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-sqlquery).





 How do I configure archiving for my instance? 

To configure archiving see [Archiving events to IBM Cloud Object Storage](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-archiving).





 I get an error when I try to provision an Activity Tracker instance 

You can only have 1 instance of the IBM Cloud Activity Tracker service per region and an instance already exists in the region.

When an auditing instance already exists in a region, you get a message when you try to provision an auditing instance a second time.

Most likely, your account administrator has already provisioned the auditing instances and has not given you permisisons to see or work with them.

To see auditing instances, you must have IAM platform permissions for the IBM Cloud Activity Tracker service.

Therefore, if you cannot see any Auditing instances when you [launch the Activity Tracker observability dashboard](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-launch), check that you have permissions to at least view them. You need at least the viewer platform role to see the auditing instances. To learn more about IAM permissions, see [Managing access with IAM](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-iam).





 How do I create an API key? 

Complete the following steps:



1. Log in to IBM Cloud with the [IBM Cloud CLI](https://cloud.ibm.com/docs/cli?topic=cli-getting-started).

ibmcloud login
* The following types of documents will not be processed if they do not have the proper file extension: .docx, .pptx, .xlsx.<-- </ul> -->Also see the issues identified in the previous release.<-- <section "id="section-30aug2019ki-fixed" "> --> 2.0.1 issues that were fixed in subsequent releases <-- <ul> --> * [Fixed in version 2.1.2 When you upload documents to a collection with existing documents, a Documents uploaded message displays on the Activity page, but no further processing status displays until the number of documents increases.<-- </ul> --><-- </section "id="section-30aug2019ki-fixed" "> --><-- </section "id="section-30aug2019ki" "> --><-- <section "id="section-known-issues-ga" "> --> General Availability (GA) release, 28 June 2019 <-- <ul> --> * If you are working in the Discovery for Cloud Pak for Data tooling, and your IBM Cloud Pak® for Data session expires, you will receive a blank page. To return to the tooling, refresh the browser and log back in. * All JSON files ingested into Discovery should include the .json file extension. * When querying on the collection_id of a trained collection, the training_status.notices value may occasionally display as 0 instead of the correct value. * Not all query limitations are enforced in this release. See query limitations]] ! ! ! ! !
: IBM Watson™ Discovery supports only English language collections when connecting and syncing to Box, Salesforce, and SharePoint Online with the Discovery tooling. [Resolved](https://cloud.ibm.com/docs/discovery?topic=discovery-release-notes2aug18) : The individual document file size limit for Box, Salesforce, and SharePoint Online is 10MB.

New Performance Dashboard : Added a new Performance Dashboard in Discovery tooling. See [Viewing metrics and improving query results with the Performance dashboard](https://cloud.ibm.com/docs/discovery?topic=discovery-performance-dashboard). The new dashboard is not available in Premium or Dedicated environments.

New full support for Japanese language : Added full support for Japanese. For more information, see [Language support](https://cloud.ibm.com/docs/discovery?topic=discovery-language-support).





 22 June 2018 

New Events and Feedback API : Released the Events and Feedback API. See the [API reference](https://cloud.ibm.com/apidocs/discoverycreate-event) for more information.





 11 June 2018 

New IAM authentication support for Washington, DC location : For applications that are hosted in Washington, DC (US East), the service now supports token-based Identity and Access Management (IAM) authentication. IAM uses access tokens rather than service credentials for authentication with a service. For more information about using IAM tokens with existing and new applications, see the [17 May 2018](https://cloud.ibm.com/docs/discovery?topic=discovery-release-notesdiscovery-17may2018) release update.

New Element Classification contract element : An additional contract element is now supported in Element Classification: Safety and Security. See [Understanding Contract Elements](https://cloud.ibm.com/docs/discovery?topic=discovery-element-classification) for details.





 6 June 2018 

Improved query text display : IBM Watson™ Discovery News queries now display the first 50 words of each article in the text JSON field.
* [Document Retrieval for Contracts](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-projectsdoc-retrieval-contracts)
* [Conversational Search](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-projectsconversational)
* [Content Mining](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-projectsmining)
* [Custom](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-projectscustom)



For more information about the different settings that are applied to each project type, see [Default project settings](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-project-defaults).



 Document Retrieval 

Use this project type to search and find the most relevant answers from your data. Projects of this type are typically deployed as search field components that are added to websites or other applications.

Documents that you add to a project of this type are automatically enriched in the following ways:



* Entities, such as proper nouns, are identified and tagged.
* Parts of speech are identified and tagged.



This tagged information is used later when a natural language phrase is submitted as a search query to return a smarter response.

A sample Document Retrieval project is available for you to explore. For more information, see [Getting started with Watson Discovery](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-getting-started).





 Document Retrieval for Contracts 

If you are working with English-language legal contracts, enable the Content Intelligence feature to apply a contracts enrichment that can recognize and tag contract-related concepts in your data. Use this project type to automate complex business processes, such as contract review and negotiation. This project type can help to increase productivity, minimize costs, and reduce your legal exposure.

Only users of installed deployments (IBM Cloud Pak for Data) or Premium or Enterprise plan managed deployments can create this type of project.

In addition to the enrichments that are applied to a typical document retrieval project, the following enrichments are made automatically: