vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Searching archive data by using the Data Engine service 

You can use the Data Engine service to query Log Analysis archive files that are stored in an Object Storage (COS) bucket in IBM Cloud.

Zoom

![IBM Cloud services integration scenario](https://cloud.ibm.com/docs-content/v1/content/f61dbc5d84bf0c89e477a05cab04c72f429e1b7a/log-analysis/images/Log-Analysis-08-Search-Events.svg)

Figure 1. Example services integration scenario.

Each Log Analysis instance has a service plan associated that indicates the number of days that you can query data through the web UI. To have access to the data after this period, you must enable archiving in the Log Analysis instance.



* When you enable archiving, you get an archive file hourly. This file contains information for that hour. Notice that UTC timestamps are used to determine which records are included in each file.
* The file name includes the ID of your Log Analysis instance and the date of the records that are included in the file.
* You can archive logging data to a COS bucket that is available in the same account or in a different account in IBM Cloud.



Once you have the Data Engine service running on IBM Cloud, you can immediately start querying your data using the SQL Query user interface, programmatically by using either the REST API or the Python ibmcloudsql library, or write a serverless function by using Cloud Functions.

When you query logs, consider the following information:



* You must provision an instance of the Data Engine service.
* You must restrict user access to work with that instance. Users need the platform viewer role to launch the UI, and the service writer role to run queries. For more information about roles, see [Required user roles](https://cloud.ibm.com/docs/sql-query?topic=sql-query-overviewuser-roles).
* When you open the UI, the Data Engine service automatically generates a unique COS bucket that will store all of the results as CSV files from your SQL queries.
* HMAC



Click Next.
5. Name the collection.
6. If the language of the documents in storage is not English, select the appropriate language.

For a list of supported languages, see [Language support](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-language-support).
7. Optional: Change the synchronization schedule.

For more information, see [Crawl schedule options](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionscrawlschedule).
8. Choose the buckets that you want to crawl.

The more buckets that you select, the longer the processing of the documents takes.
9. If you want to limit the types of files to add to the collection, you can list the file extensions for file types to either include or exclude.

When you choose to list extensions for file types to exclude, you must add at least one file extension.

For a list of supported file types, see [Supported file types](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionssupportedfiletypes).
10. If you want the crawler to extract text from images on the site, expand More processing settings, and set Apply optical character recognition (OCR) to On.

When OCR is enabled and your documents contain images, processing takes longer. For more information, see [Optical character recognition](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionsocr).
11. Click Finish.



The collection is created quickly. It takes more time for the data to be processed as it is added to the collection.

If you want to check the progress, go to the Activity page. From the navigation pane, click Manage collections, and then click to open the collection.
The Settings tab contains the UI for many of the tunable settings for your deployment. You can



* view encryption details. Encryption at rest is enabled for all Databases for MongoDB deployments. If you brought your own encryption key from [Key Protect](https://cloud.ibm.com/docs/databases-for-mongodb?topic=cloud-databases-key-protect), the panel provides a link to your Key Protect instance and the Encryption Key field has the name of the key.
* [Change the admin password](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-admin-password)
* [Implement or modify an IP allowlist](https://cloud.ibm.com/docs/databases-for-mongodb?topic=cloud-databases-allowlisting)







 Service Credentials 

You can generate a new set of credentials for cases where you want to manually [connect an app](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-ibmcloud-app) or [external consumer](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-external-app) to an IBM Cloud service. [Learn more](https://cloud.ibm.com/docs/account?topic=account-service_credentials).





 Connections 

Shows connected resources. You can use the Create connection button to to bind this service to another resource.





 View docs 

The View docs link from the Actions drop list opens the main documentation page for Databases for MongoDB in a new tab.
(https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/query_structure.png)

Figure 1. Example query structure

This basic query contains a nested path expression before the : operator. Each path element is the name of a field in the document separated by a period (.). The : operator indicates that the text that follows the operator must be included in the result.

The :: operator indicates that the text must be matched exactly in the result. For more information, see [Query operators](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-operators). You can see how the two operators are used in the following examples.



* To return matching documents in order of relevance, pass the following data object in the POST request:

{
"query":"enriched_text.entities.text:IBM"
}
* To return matching documents in any order, pass the following data object in the POST request as the query body:

{
"filter":"enriched_text.entities.text::IBM"
}







 Using the filter and query parameters together 

The filter parameter returns faster than the query parameter and its results are cached. If you submit queries that use the filter and query parameters separately on a small data set, each request returns similar (if not identical) results.

In large data sets, if you need results to be returned in order of relevance, combine the filter and query parameters. Using the parameters together improves performance because the filter parameter is applied first. It filters the documents and caches the results. The query parameter then ranks the cached results.



 Filter example: Get a document by its ID 

Query body:

{
"filter": "document_id::b6d8c6e3-1097-421b-9e39-75717d2554aa"
}

If the document exists, the query returns 1 matching result.
FileNet P8 

Crawl documents that are stored in FileNet P8.

IBM Cloud Pak for Data

IBM Cloud Pak for Data only

This information applies only to installed deployments.



 What documents are crawled 



* Only file types that are supported by Discovery are crawled; all others are ignored. For more information, see [Supported file types](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionssupportedfiletypes).
* Document-level security is supported. When this option is enabled, your users can crawl and query the same content that they can access when they are logged in to FileNet. Discovery does not support role-based security when you crawl FileNet P8.

For more information about document-level security, see [Supporting document-level security](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collection-typesconfiguredls).
* Only files with file extensions that match the file extension filter rules that you specify are crawled. Added with the 4.7.0 release.
* When a source is recrawled, new documents are added, updated documents are modified to the current version, and deleted documents are deleted from the collection's index.
* All Discovery data source connectors are read-only. Regardless of the permissions that are granted to the crawl account, Discovery never writes, updates, or deletes any content in the original data source.







 Data source requirements 

In addition to the [data source requirements](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collection-typesrequirements) for all installed deployments, your FileNet P8 data source must meet the following requirements:



* The data source can crawl FileNet P8 5.5.0 and the Content Engine Web Services (CEWS) of a FileNet server that is installed on IBM Cloud Pak for Automation.
* FileNet P8 5.5.0 and FileNet on Cloud Pak for Automation support the HTTP and HTTPS protocols.







 Prerequisite steps 

If you want to enable document-level security, you must take some steps to set it up.
MongoDB Enterprise Audit logging 

Audit logging capabilities for IBM Cloud® Databases for MongoDB Enterprise Edition are provided with IBM® Log Analysis.



  Event Types 

Databases for MongoDB Enterprise Edition provides audit logging capabilities based on the following event types:



*  authenticate
*  createCollection
*  createDatabase
*  createIndex
*  renameCollection
*  dropCollection
*  dropDatabase
*  dropIndex
*  createUser
*  dropUser
*  dropAllUsersFromDatabase
*  updateUser
*  grantRolesToUser
*  revokeRolesFromUser
*  createRole
*  updateRole
*  dropRole
*  dropAllRolesFromDatabase
*  grantRolesToRole
*  revokeRolesFromRole
*  grantPrivilegesToRole
*  revokePrivilegesFromRole



The event types for Databases for MongoDB Enterprise Edition are fixed and not configurable.





  Audit Logs 

Audit events appear in IBM® Log Analysis.

For more information on Databases for MongoDB Enterprise Edition audit logging, review the [Log Analysis Integration documentation](https://cloud.ibm.com/docs/databases-for-mongodb?topic=cloud-databases-logging) along with the [IBM® Log Analysis](https://cloud.ibm.com/docs/log-analysis) getting started tutorial.
Use the [Delete method](https://cloud.ibm.com/apidocs/discoverydelete-a-document) in the Discovery API to delete documents. There is no delete option within the user interface.



Discovery v2



* How do I search the product documentation?

To search the entire IBM Cloud Docs site, enter your search term into the search field in the IBM Cloud website banner. To search for information about the Discovery service only, scroll to the start of the page and enter your search term into the search field in the page header.

![Shows where the documentation search field is situated at the start of the page](https://cloud.ibm.com/images/search-doc.png)

Figure 1. Search bar for this product documentation versus all IBM Cloud docs
* How does Watson Discovery access my data?

Discovery has built-in connectors that can crawl various data sources, including websites, IBM Cloud Object Storage, Box, Microsoft SharePoint, and Salesforce sites. It even has support for you to build custom connectors. You can schedule crawls so that as the source data changes, the latest version is picked up by your collection automatically. Discovery only ever reads from external data sources; it never writes, updates, or deletes any content in the original data source. For more information, see [Creating collections](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collections)
* Can I upload documents?

Yes, you can upload documents directly to a collection in your project. An upload is a one-time operation that you can use to get started. An alternative approach is to connect to a data source and crawl the source for information. When you crawl data sources, the data can stay where it is and you can set up a schedule by which to crawl the external source to find new and changed information. When you crawl the data, you know that the information in your collection is always up to date. For more information, see [Creating collections](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collections).
* Must all my documents be English?
Release notes for Discovery for IBM Cloud 

Learn about features and changes that were included for each release and update of the product software.

IBM Cloud

This information applies only to managed instances of IBM Watson® Discovery that are hosted on IBM Cloud or that were provisioned with [IBM Cloud Pak for Data as a Service](https://dataplatform.cloud.ibm.com/docs/content/wsj/landings/watsondisc.html). For information about releases and updates for installed deployments, see [Release notes for IBM Watson® Discovery Cartridge for IBM Cloud Pak® for Data](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-release-notes-data).



 10 June 2023 

All Entities enrichments use the Entities v2 type system
: Natural Language Understanding Entities v1 is no longer supported. IBM Cloud instances that were created before 2 June 2021 and Discovery for IBM Cloud Pak for Data 2.x deployments used version 1 of the Natural Language Understanding Entities type system for English and Korean collections. Now, all collections use only version 2 of the Natural Language Understanding Entities type system.

Classifiers are identified more clearly
: The Enrichments page lists classifier enrichments as either text classifier or document classifier enrichments.





 16 May 2023 

Improved tool for creating Smart Document Understanding (SDU) user-trained models
: The SDU tool that you use to annotate documents when you create a user-trained SDU model now uses the React UI framework. This update does not change the behavior of the tool, but does make it more responsive.

You can now define JSON normalizations by using the Collections API
: The Create a collection and Update a collection methods now support the addition of conversions and normalizations objects that you can specify to apply normalization operations to the documents in the collection. For example, you can define an operation to copy or merge one field to another in the JSON representation of the documents. The conversions object defines normalization operations that occur during ingestion and the normalizations object defines normalization operations that occur after enrichments are applied.
When writing a query in Discovery, if any query field contains whitespace (for example, body.additional reading), you receive a 400: Invalid query syntax error. [Resolved](https://cloud.ibm.com/docs/discovery?topic=discovery-release-notes8nov17)





 25 September 2017 

New Premium pricing plan : A Premium pricing plan is now available. For more information, see [Discovery pricing plans](https://cloud.ibm.com/docs/discovery?topic=discovery-discovery-pricing-plans).

New feature for querying multiple collections : The ability to query, list fields, and query notices across collections within the same environment is added. For details, see [Querying multiple collections](https://cloud.ibm.com/docs/discovery?topic=discovery-query-conceptsmultiple-collections).

New language support : Language support information for Discovery is available at [Language support](https://cloud.ibm.com/docs/discovery?topic=discovery-language-support).

Change to Visual Query Builder : The Visual Query Builder moved from beta status to GA status. Filter, Timeslice, and Histogram aggregations are not currently supported with the Visual Query Builder. Click Include analysis of your results, then Edit in Query Language on the Build queries screen to write those aggregations.

New beta deduplicate queries in Discovery News : Added the beta capability to deduplicate on IBM Watson™ Discovery News queries.

New multiple language support for collections : In addition to English, German, and Spanish language collections, you can now create Arabic, French, Italian, Korean, Japanese, and Brazilian Portuguese collections.

Known issue in syndicated environments : Discovery Tooling does not support syndicated environments. [Resolved](https://cloud.ibm.com/docs/discovery?topic=discovery-release-notes15nov17)





 14 September 2017 

New Data Schema Explorer : Added the Data Schema Explorer, which displays the fields and values in your transformed documents. This information can be used to understand the data structure of your collection before building queries using the Discovery Query Language.
If you crawl Box or Salesforce, a list of available resources is presented when you configure a source, using the Discovery tooling.
* If you are using the Discovery tooling, you can configure a collection with a single data source.
* Crawling a data source uses resources, namely API calls, of the data source. The number of API calls depends on the number of documents that need to be crawled. You must obtain an appropriate level of service license, for example Enterprise, for the data source. For information about the appropriate service level license that you need, contact the source system administrator.
* Discovery source crawls do not delete documents that are stored in a collection, but you can manually delete them using the API. When a source is re-crawled, new documents are added, updated documents are modified to the current version, and deleted documents remain as the version last stored.
* Discovery can only ingest the following file types, and it ignores all other document types:





 Lite plans Advanced plans 

 PDF, Word, PowerPoint, Excel, JSON*, HTML* PDF, Word, PowerPoint, Excel, PNG**, TIFF**, JPG**, JSON*, HTML* 



* IBM Watson™ Discovery supports crawling JSON and HTML documents, but you cannot edit these documents using the SDU editor. To change the configuration of HTML and JSON documents, you must use the API. For more information, see the [API reference](https://cloud.ibm.com/apidocs/discovery/).

** Individual image files, such as PNG, TIFF, and JPG, are scanned, and any text is extracted. PNG, TIFF, and JPEG images embedded in PDF, Word, PowerPoint, and Excel files are also scanned, and any text is extracted.

View the following table to see the objects that a data source can crawl and which data sources support crawling new and modified documents during a refresh:



Table 1. Data sources that support crawling new and modified documents during refresh and objects that can be crawled

 Data source Crawls new and modified documents during refresh? Compatible objects that can be crawled 

 Box (Application level access) No Files, folders
Use the Data Engine user interface (UI) to [develop your queries](https://cloud.ibm.com/docs/sql-query?topic=sql-query-running) and the [Data EngineREST API](https://cloud.ibm.com/docs/services/sql-query?topic=sql-query-overviewrestapi) to automate them.



 Input and output of queries 

Before you can use the Data Engine service to run SQL queries, the input data must be uploaded to one or more Cloud Object Storage instances. You must also have at least 'Writer' access to at least one Cloud Object Storage bucket, so that result objects (that is, the objects that contain output data) can be written there. For more information about Cloud Object Storage, including how to provision an instance, create buckets, and upload data, see the [Cloud Object Storage Getting Started Guide](https://cloud.ibm.com/docs/cloud-object-storage/getting-started.htmlgetting-started-console) and [Reading and writing to Cloud Object Storage](https://cloud.ibm.com/docs/sql-query/blob/draft/reading_cos.md). You can also [write to databases](https://cloud.ibm.com/docs/sql-query/blob/draft/writing_databases.md) and take advantage of [index management](https://cloud.ibm.com/docs/sql-query?topic=sql-query-index_management).





 Programmatic access 



 REST API 

You can use the [Data Engine service REST API](https://cloud.ibm.com/apidocs/sql-query/sql-query-v3) to run queries and retrieve information about their status. This is especially helpful when you write code that automatically queries data.

Note: The Cloud Resource Name (CRN) is a mandatory part of a Data Engine REST endpoint call. The CRN Copy button copies your CRN to clipboard and you can paste it into your API call.





 Python applications and Notebooks
: Context-based restrictions (CBR) give account owners and administrators the ability to define and enforce access restrictions for IBM Cloud® resources based on the context of access requests. Access to Cloud Databases resources can be controlled with CBR and identity and access management (IAM) policies. For more information, see [Protecting Cloud Databases resources with context-based restrictions](https://cloud.ibm.com/docs/databases-for-cassandra?topic=cloud-databases-cbr&interface=ui).





 25 March 2022 

IBM Cloud® Databases for Elasticsearch Security Goals Updated
: Available security and compliance goals updated. See updated goals [here](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-elasticsearch-manage-security-compliance).





 08 December 2020 

IBM Cloud® Databases for Elasticsearch 6 End of Life
: On April 29, 2021, all IBM Cloud® Databases for Elasticsearch instances on version 6.x that are still active will be disabled. See blog post announcement [here](https://www.ibm.com/cloud/blog/announcements/databases-for-elasticsearch-6-end-of-life).





 09 November 2020 

IBM Cloud® Databases for Elasticsearch v7.9 is now available
: We are pleased to announce the release of version 7.9 on IBM Cloud® Databases for Elasticsearch. See blog post announcement [here](https://www.ibm.com/cloud/blog/announcements/databases-for-elasticsearch-v7-9-is-now-available).





 13 April 2020 

IBM Cloud® Databases for Elasticsearch autoscaling
: We are excited to announce that autoscaling of your deployments based on disk capacity and disk I/O utilization is now available for IBM Cloud® Databases for Elasticsearch via the UI, API, and CLI. See blog post announcement [here](https://www.ibm.com/cloud/blog/announcements/ibm-cloud-databases-portfolio-introduces-autoscaling).





 20 February 2020
For more information, see [Adding domain-specific resources](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain).





 How does the Smart Document Understanding tool work? 

You can use the Smart Document Understanding tool to teach Discovery about sections in your documents with distinct format and structure that you want Discovery to index. You can define a new field, and then annotate documents to train Discovery to understand what type of information is typically stored in the field. For more information, see [Using Smart Document Understanding](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-configuring-fields).





 What's the best way to add synonyms? 

You can use two different methods to define synonyms.



* To define synonyms that are recognized and tagged when a document is ingested and that can be retrieved by search, create a dictionary and add synonyms for the dictionary term entry. A dictionary defines special terms that you want to tag in your documents, such as product names or industry-specific terminology. You can use the dictionary terms later to create facets and to filter documents. For more information, see [Dictionary](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain-dictionary).
* To define synonyms that are applied to the query text that is submitted by users to expand the meaning of the query, add synonyms by using the Synonyms tool on the Improve relevance section of the Improve and customize page. For more information, see [Expanding the meaning of queries](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-search-settingsquery-expansion).







 Can I use Discovery to detect sentiment? 

You can use Discovery to detect both phrase and document sentiment. Document sentiment is a built-in Natural Language Processing enrichment that is available for all project types. Document sentiment evaluates the overall sentiment that is expressed in a document to determine whether it is positive, neutral, or negative. Phrase sentiment does the same. However, phrase sentiment can detect and assess multiple opinions in a single document and, in English and Japanese documents, can find specific phrases.