vania-janet/multiturn-rag-retrieval-data huggingface dataset:
* How do I access and query data that has been archived for long term storage in COS?

To access data, you can download the archived file locally.

To query the data, you can also use a service like SQL Query to query your COS archives and get information based on queries that you define. [Learn more](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-sqlquery).
* How do I configure archiving for my instance?

To configure archiving see [Archiving events to IBM Cloud Object Storage](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-archiving).
* I get an error when I try to provision an Activity Tracker instance

You can only have 1 instance of the IBM Cloud Activity Tracker service per region and an instance already exists in the region.

When an auditing instance already exists in a region, you get a message when you try to provision an auditing instance a second time.

Most likely, your account administrator has already provisioned the auditing instances and has not given you permisisons to see or work with them.

To see auditing instances, you must have IAM platform permissions for the IBM Cloud Activity Tracker service.

Therefore, if you cannot see any Auditing instances when you [launch the Activity Tracker observability dashboard](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-launch), check that you have permissions to at least view them. You need at least the viewer platform role to see the auditing instances. To learn more about IAM permissions, see [Managing access with IAM](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-iam).
* How do I create an API key?

Complete the following steps:



1. Log in to IBM Cloud with the [IBM Cloud CLI](https://cloud.ibm.com/docs/cli?topic=cli-getting-started).

ibmcloud login
: [Translating documents](https://cloud.ibm.com/docs/language-translator?topic=language-translator-document-translator-tutorial) is now available through new API endpoints. Submit a Microsoft Office document, PDF, or other document with a supported file format, and Language Translator will provide a translated copy that preserves the original formatting. [Supported file formats](https://cloud.ibm.com/docs/language-translator?topic=language-translator-document-translator-tutorialsupported-file-formats) include .doc, .ppt, .pdf, and more.

New translation models
: New translation models for Hungarian are now available:



* Hungarian to English (hu-en)
* English to Hungarian (en-hu)







 8 November 2018 

New Tokyo location
: You can now create Language Translator service instances in the IBM Cloud Tokyo location.





 9 August 2018 

New translation models
: New translation models for Norwegian Bokmål are now available:



* Norwegian Bokmål to English (nb-en)
* English to Norwegian Bokmål (en-nb)







 27 June 2018 

New translation models
: New translation models that feature six new languages are now available:



* Catalan



* Catalan to Spanish (ca-es)
* Spanish to Catalan (es-ca)



* Czech



* Czech to English (cs-en)
* English to Czech (en-cs)



* Danish



* Danish to English (da-en)
* English to Danish (en-da)



* Finnish



* Finnish to English (fi-en)
* English to Finnish (en-fi)



* Hindi



* Hindi to English (hi-en)
* English to Hindi (en-hi)



* Swedish



* Swedish to English (sv-en)
* English to Swedish (en-sv)









 15 June 2018 

New support for IBM Cloud IAM
Box 

Crawl documents that are stored in a Box data source.

IBM Cloud Pak for Data

IBM Cloud Pak for Data only

This information applies only to installed deployments. For more information about connecting to Box from a managed deployment, see [Box](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-box-cloud).



 What documents are crawled 



* Only documents that are supported by Discovery in your Box folders are crawled; all others are ignored. For more information, see [Supported file types](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionssupportedfiletypes).
* Document-level security is supported only for connectors that are configured to use the App + Enterprise Access as the Box application's app access level. When this option is enabled, your users can crawl and query the same content that they can access when they are logged in to Box. For more information, see [Supporting document-level security](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collection-typesconfiguredls).
* When a source is recrawled, new documents are added, updated documents are modified to the current version, and deleted documents are deleted from the collection's index.
* All Discovery data source connectors are read-only. Regardless of the permissions that are granted to the crawl account, Discovery never writes, updates, or deletes any content in the original data source.
* Box notes are stored in JSON format, so Discovery also ingests any Box notes in the specified folders.







 Data source requirements 

In addition to the [data source requirements](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collection-typesrequirements) for all installed deployments, your Box data source must meet the following requirement:



* You must obtain any required service licenses for the data source that you want to connect to. For more information about licenses, contact the system administrator of the data source.







 Prerequisite steps
These queries can be used as a starting point for writing your own queries. Sample queries are available for both IBM Watson™ Discovery News and private collections.

New beta visual aggregation builder : Added the beta ability to write aggregations with a visual builder. Click Build in visual mode above the Write an aggregation query using the Discovery Query Language field to try it out. As you build your aggregation visually, the query displays in the Discovery Query Language below it.
: The visual aggregation builder is currently supported only as a beta capability.





 31 July 2017 

New release of Discovery News : A new version of IBM Watson™ Discovery News was released. The original version is renamed as IBM Watson™ Discovery News Original and is retired, with a removal from service date of 15 January 2018. : If you create a new instance of Discovery, you only have access to the new version of IBM Watson™ Discovery News.

New pricing structure : A new pricing plan for IBM Watson™ Discovery was released. See [Discovery pricing plans](https://cloud.ibm.com/docs/discovery?topic=discovery-discovery-pricing-plans) for details.

Update to version string : The version string for all API calls changed to 2017-08-01 from 2017-07-19. This version includes updates for the new pricing plan and the new version of Watson Discovery News. Update the version string to avoid conflicts and possible errors.





 19 July 2017 

Change to Pricing plans : Pricing changes will take effect on 1 August 2017. : Users that are currently on the deprecated 30 day free trial plan will be automatically migrated to the Lite plan. As a result of this transition, existing users might meet or exceed the lite plan limit on documents (2000), storage (200Mb), or number of collections (2). If you exceed the limit of the Lite plan, you cannot add any additional content into the service, but you can still query collections. : View the current status of all these limits by using the Discovery tooling or API. To resume adding content to the Discovery instance, you must complete one of the following actions: : Remove collections or documents so that limits of the Lite plan are not exceeded.
[Custom Models](https://cloud.ibm.com/docs/discovery?topic=discovery-integrating-with-wks):
One Knowledge Studio model included. Additional models: $800 each
[Element Classification](https://cloud.ibm.com/docs/discovery?topic=discovery-element-classification)***: 500 pages included per month. Additional pages: $0.40 each
[News Queries](https://cloud.ibm.com/docs/discovery?topic=discovery-watson-discovery-news): 200 News queries included per month 10,000 additional queries (per month): $0.10 per query
10,001 - 100,000 additional queries (per month): $0.05 per query
Over 100,000 queries (per month): $0.03 per query
[Query Expansions](https://cloud.ibm.com/docs/discovery?topic=discovery-query-conceptsquery-expansion): 5,000 query expansions with 25,000 total terms
[Document splitting](https://cloud.ibm.com/docs/discovery?topic=discovery-configservicedoc-segmentation): 1,000 segments per plan. Additional segments: Not available

The Element Classification enrichment is deprecated and will no longer be available, effective 10 July 2020.

-----
* The document limit assumes an average document size of 100KB on disk. Document size is calculated after it goes through conversion and enrichment, so document size might change significantly from the original input. You can view the number of documents stored and the total amount of storage used by either using the [Environments](https://cloud.ibm.com/apidocs/discoveryget-environment-info) or [Collections](https://cloud.ibm.com/apidocs/discoveryget-collection-details) API, or by using the tooling. If your documents are, on average, larger than 100KB on disk, you will hit the storage limit of a plan before the maximum document limit.
Use the [Delete method](https://cloud.ibm.com/apidocs/discoverydelete-a-document) in the Discovery API to delete documents. There is no delete option within the user interface.



Discovery v2



* How do I search the product documentation?

To search the entire IBM Cloud Docs site, enter your search term into the search field in the IBM Cloud website banner. To search for information about the Discovery service only, scroll to the start of the page and enter your search term into the search field in the page header.

![Shows where the documentation search field is situated at the start of the page](https://cloud.ibm.com/docs/images/search-doc.png)

Figure 1. Search bar for this product documentation versus all IBM Cloud docs
* How does Watson Discovery access my data?

Discovery has built-in connectors that can crawl various data sources, including websites, IBM Cloud Object Storage, Box, Microsoft SharePoint, and Salesforce sites. It even has support for you to build custom connectors. You can schedule crawls so that as the source data changes, the latest version is picked up by your collection automatically. Discovery only ever reads from external data sources; it never writes, updates, or deletes any content in the original data source. For more information, see [Creating collections](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collections)
* Can I upload documents?

Yes, you can upload documents directly to a collection in your project. An upload is a one-time operation that you can use to get started. An alternative approach is to connect to a data source and crawl the source for information. When you crawl data sources, the data can stay where it is and you can set up a schedule by which to crawl the external source to find new and changed information. When you crawl the data, you know that the information in your collection is always up to date. For more information, see [Creating collections](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collections).
[Provision a Databases for MongoDB Enterprise Edition instance with Terraform](https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-tutorial-provision-mongodbee-tf)Provision a Databases for MongoDB Enterprise Edition instance with Terraform

In this tutorial, you learn how to use Terraform to provision a Databases for MongoDB Enterprise Edition instance that includes the Databases for MongoDB Enterprise Edition Analytics Add-On. The Databases for MongoDB EE Analytics Add-On allows you to run long-running analytical queries and/or provision a MongoDB Connector for business intelligence(BI) to make your query data compatible with BI tools, such as Tableau.

IBM Cloud Databases



* 1 hour
* 2022-12-09



[Provision a Databases for PostgreSQL instance with Terraform](https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-tutorial-provision-postgres-tf)Provision a Databases for PostgreSQL instance with Terraform

In this tutorial, you learn how to use Terraform to provision a Databases for PostgreSQL instance.

IBM Cloud Databases



* 1 hour
* 2022-12-09



[Truck Tracker: An IoT Pattern Using OpenShift and other IBM Cloud Services](https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-truck-tracker-ibmcloud)Truck Tracker: An IoT Pattern Using OpenShift and other IBM Cloud Services

Truck Tracker is a service that receives data from a fleet of trucks as they travel across the country. It could be any data (e.g., fuel consumption, driver details, ambient temperature, or anything else that can be measured as you cruise). For the purposes of this post, it will be the truck's ID and position (lat/long coordinates). Truck Tracker will store the data in various data stores and also show you the current position of the trucks on a map.

IBM Cloud Databases



* 2 hours
* 2022-03-03
A subset of documents will be available for annotation purposes, between 20 - 50 documents will appear in the dropdown list. The number will depend on several factors, including the number of documents in your collection in the supported file formats.



You can use the toolbar at the top to complete the following actions:



* Choose a document to annotate
* Navigate the document displayed
* Adjust the page view (single page view, zoom in, zoom out), clear changes, and export/import models. Click on single page view to toggle the display - you can view your annotations and document separately, or together.



You can also crawl various data sources, or do a web crawl with the Discovery tooling. Click the Connect a data source button and see [Connecting to data sources](https://cloud.ibm.com/docs/discovery?topic=discovery-sources) for more information.





 Managing enrichments 

You can enrich fields, including custom fields that Smart Document Understanding identifies, in your collection with cognitive metadata.

Many enrichments are available in Discovery. You must create some of the enrichments before you can apply them. To access the Enrichments tab, click Manage data in your collection and then Configure data. By default, the Identify fields tab opens. Click Enrich fields. For more information about collections, see [Connecting to data sources](https://cloud.ibm.com/docs/discovery?topic=discovery-sources).

To apply an enrichment to a field, complete the following steps:



1. In the Enrich fields tab, look for the fields that you want to enrich in Fields to be enriched, and delete or Add enrichments from or to the field of your choice.
2. If there are any other fields that you want to enrich that are not yet listed in Fields to be enriched, enter them in Add a field to enrich, and add the enrichments.
3. Click Apply changes to collection.







 How to annotate a document 

See [Best practices for annotating documents](https://cloud.ibm.com/docs/discovery?topic=discovery-sdubestpractices) before you begin annotating.
[Query Expansions](https://cloud.ibm.com/docs/discovery?topic=discovery-query-conceptsquery-expansion): 500 query expansions with 1,000 total terms. Additional expansions: Not available
[Document splitting](https://cloud.ibm.com/docs/discovery?topic=discovery-configservicedoc-segmentation): 250 segments per plan. Additional segments: Not available

For information about upgrading from Lite to Advanced, see [Upgrading your service](https://cloud.ibm.com/docs/discovery?topic=discovery-upgrading-your-planservice)

For query performance information, see [Query performance](https://cloud.ibm.com/docs/discovery?topic=discovery-qp). Queries are limited based on plan. Estimated Average Query rates per Lite plan: 1 QPS for reranked queries with two top-level text fields.

The Element Classification enrichment is deprecated and will no longer be available, effective 10 July 2020.





 Advanced 

When choosing an Advanced Plan size, note that resources are required for both document storage and querying. Therefore, you might require a larger environment if you are close to the maximum document limit for a plan size and also have the following requirements:



* Higher query performance requirements (for example, using relevancy training)
* You anticipate a large number of concurrent users
* Complex query needs



For additional details on factors that influence query performance, see [Query performance](https://cloud.ibm.com/docs/discovery?topic=discovery-qp). Queries are limited based on plan. Estimated Average Query rates per Advanced (S, MS) plans: 10 QPS for reranked queries with two top-level text fields.



 Size Number of Docs* Price 

 X-Small**** Up to 50,000 docs total Starting at $500 per month 
 Small Up to 1M docs total Starting at $1,500 per month 
 Medium-Small Up to 2M docs total Starting at $3,000 per month 
 Medium Up to 4M docs total Starting at $5,000 per month
: Support for connecting to Postgres 9.4 was removed because the vendor ended version support was ended by the vendor on 13 February 2020.

Fixed the following defects in the 2.1.2 release
: When installing Discovery for Cloud Pak for Data on OpenShift, the ranker-rest service might intermittently fail to startup, due to an incompatible jar in the classpath.
: When you upload documents to a collection with existing documents, a Documents uploaded! message displays on the Activity page, but no further processing status displays until the number of documents increases.
: Running a query with an empty aggregations parameter returns an empty aggregations array.
: Deprovisioning a IBM Watson® Discovery for IBM Cloud Pak® for Data Instance will not delete the underlying data. Delete the collections and documents manually.





 2.1.1 release, 24 January 2020 

New release now available
: IBM Watson® Discovery for IBM Cloud Pak® for Data version 2.1.1 is available.

Fixed the following defects in the 2.1.1 release:
: In Document Retrieval project types, when you perform an empty search, and the search results source is set to passages, the query results will display excerpt unavailable in the Project workspace.
: When visiting the Storybook links on the Integrate and deploy page, the links do not go to the correct location. Please visit [Storybook](https://watson-developer-cloud.github.io/discovery-components/storybook) instead to view documentation.
: If you are using Smart Document Understanding, two variables no longer need to be set during installation or reinstallation. For more information, see [Environment variable settings for Smart Document Understanding](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-troubleshoottroubleshoot-sdu).
: Discovery for Content Intelligence and Table Understanding enrichments are configured out of the box to be applied on a field named html. When a user uploads a JSON document without a root-level field named html, these enrichments will not yield results in the index.
Salesforce 

Crawl documents that are stored in Salesforce.

IBM Cloud Pak for Data

IBM Cloud Pak for Data only

This information applies only to installed deployments. For more information about connecting to Salesforce from a managed deployment, see [Salesforce](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-salesforce-cloud).



 What documents are crawled 



* Knowledge Articles are crawled only if their version is published and their languages is en-us.
* Only documents that are supported by Discovery are crawled; all others are ignored. For more information, see [Supported file types](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionssupportedfiletypes).
* When a source is recrawled, new documents are added, updated documents are modified to the current version, and deleted documents are deleted from the collection's index during refresh.
* All Discovery data source connectors are read-only. Regardless of the permissions that are granted to the crawl account, Discovery never writes, updates, or deletes any content in the original data source.



Discovery can crawl the following objects:



* Any default and custom objects that you have access to
* Accounts
* Contacts
* Cases
* Contracts
* Knowledge articles
* Attachments







 Data source requirements 

In addition to the [data source requirements](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collection-typesrequirements) for all installed deployments, your Salesforce data source must meet the following requirements:



* The instance that you plan to connect to must be part of an Enterprise plan or higher.
* You must obtain any required service licenses for the data source that you want to connect to. For more information about licenses, contact the system administrator of the data source.



For more information about Salesforce, see the [Salesforce developer documentation](https://developer.salesforce.com/docs/).





 Prerequisite step 

To crawl documents in Salesforce, Discovery uses a Web Service Description Language (WSDL) file.
To get familiar with the data schema of your documents, let's look at the View data schema screen. It displays the fields and values in your transformed documents two ways: by document (Document view), or by field (Collection view). Collection view displays all fields in your collection.

Click the View data schema icon. In the Collection view, under enriched_text, you can examine the enrichments you applied to your collection. Click on categories, concepts, entities, and sentiment to see how your collection was enriched with Watson insights.



If your query does not return the expected results, try swapping the field or value that your query is using for one that you can verify in the data schema.





 Step 2: Build a basic query 

Let's start out by writing a query that searches for the concept Cloud computing in your collection:



1. Click on the Build queries icon ![Query icon](https://cloud.ibm.com/docs-content/v1/content/ded4adc3ea0bd2a81b113c579f2b1183926da211/discovery/images/icon_queryBuilder.png) to open the query page. Select the collection that contains the IBM Press Releases and click Get started.
2. On the Build queries screen, click Search for Documents, then Use the Discovery Query Language then:



* Click the Field drop-down and choose enriched_text.concepts.text, for the Operator choose contains, then enter the Value of Cloud computing. The query enriched_text.concepts.text:Cloud computing is displayed under the Visual Query Builder.
* Alternately, you could click Edit in query language, then Use the Discovery Query Language. Enter enriched_text.concepts.text:"Cloud computing" into the Enter query here field.



3. Click Run query. There is one match ("matching_results": 1). Copy the Query URL at the top of the Summary or JSON tab to use in your application.



Under More options, you have the option to turn on passage retrieval with the Include relevant passages radio button. Passages are short, relevant excerpts extracted from the full documents that your query returns.
Database tools 

IBM Cloud® offers optional at-cost and no-cost database tool products for both Virtual Servers and Bare Metal Servers deployments. However, database tool products are only available for monthly billing cycles of Virtual Servers and Bare Metal Servers.

For more information about Database tools, see [Getting started with database tools.](https://cloud.ibm.com/docs/database-tools?topic=database-tools-getting-started)