vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Use the Data Engine user interface (UI) to develop and test your queries, and the [SQL Query REST API](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-sqlqueryrestapi) to automate them.

The Data Engine service provides a serverless, no-ETL solution to easily query data stored in Object Storage. Underneath, SQL Query uses Apache Spark SQL as its underlying query engine.

You can use the Data Engine to run SQL queries (that is, SELECT statements) to analyze, transform structured and semi-structured data, or clean up rectangular data. You cannot run actions such as CREATE, DELETE, INSERT, and UPDATE.

The Data Engine service can process input data that is read from CSV, JSON, ORC, Parquet, or AVRO files. The archive files from an IBM Cloud Activity Tracker instance contain data in JSON format.

Each query result can be written to a CSV, JSON, ORC, PARQUET, or AVRO file in a Object Storage instance of your choice.

When you query an IBM Cloud Activity Tracker archive file, you must convert the JSON formatted file into PARQUET format to be able to query the contents successfully.



 Prerequisites 

To be able to use the Data Engine service to query archived event files, check the following prerequites:



* You must have access to a COS instance in your account.

You must have access to a bucket that contains the IBM Cloud Activity Tracker archive files and a bucket to use to store results from your queries.
* You must have an IBM Cloud Activity Tracker instance provisioned in your account that has [archiving configured to a bucket in the COS instance in your account](https://cloud.ibm.com/docs/services/activity-tracker?topic=activity-tracker-archiving).

Events are archived daily to a file in a COS bucket.

Notice that if archiving is not configured, you must wait at least 24 hours before an archive file is available after archiving is configured.
* You must have 1 or more archive files uploaded in the bucket.
Change to sample document deletion : Sample documents are now automatically deleted after 24 hours, instead of 1 month.





 16 March 2018 

New full support for German language : Added full support for German. For more information, see [Language support](https://cloud.ibm.com/docs/discovery?topic=discovery-language-support).

New Element Classification configuration : A new Discovery tooling configuration named Default Contract Configuration is added to support Element Classification, which can be used to extract party, nature, and category from elements in PDFs. See [Element Classification](https://cloud.ibm.com/docs/discovery?topic=discovery-element-classificationelement-collection) for details.

Update to documentation segmentation : Documentation segmentation moved from beta status to GA status. The segmentation limit is increased to 250 segments. It is no longer limited to 50 segments per document. See [Documentation segmentation](https://cloud.ibm.com/docs/discovery?topic=discovery-configservicedoc-segmentation) for details.

Known issue with wildcards : Wildcards do not work with queries that contain capital letters. For example, given the key/field pair {"borrower": "GOVERNMENT OF INDIA"}, query-borrower:ndia returns results, but query-borrower:NDIA does not.





 8 March 2018 

New beta Knowledge Graph features : The beta version of IBM Watson™ Discovery Knowledge Graph added several features. During the beta release, Knowledge Graph functionality and the methods associated with it are only available for service instances that are subscribed to Advanced plans, Premium plans, and all Dedicated environments. The new features are: entity similarity, evidence, and canonicalization and filtering : Update: As of 30 September 2019, the IBM Watson™ Discovery Knowledge Graph Beta APIs are no longer accessible. See the [3 September 2019 release notes](https://cloud.ibm.com/docs/discovery?topic=discovery-release-notesdiscovery-3september2019) for more information.





 7 March 2018
[Checkmark](https://cloud.ibm.com/docs-content/v1/content/6f3c07c34058f637b1c86044ca4f9b24214330a7/activity-tracker//images/checkmark-icon.svg) [Location-based events](https://cloud.ibm.com/docs/messages-for-rabbitmq?topic=cloud-databases-activity-tracker) 
 [IBM Cloud Databases for MongoDB](https://cloud.ibm.com/docs/databases-for-mongodb) databases-for-mongodb-group ![Checkmark](https://cloud.ibm.com/docs-content/v1/content/6f3c07c34058f637b1c86044ca4f9b24214330a7/activity-tracker//images/checkmark-icon.svg) [Location-based events](https://cloud.ibm.com/docs/databases-for-mongodb?topic=cloud-databases-activity-tracker-integration) 
 [IBM Cloud® Databases for MySQL](https://cloud.ibm.com/docs/databases-for-mysql) databases-for-mysql-group ![Checkmark](https://cloud.ibm.com/docs-content/v1/content/6f3c07c34058f637b1c86044ca4f9b24214330a7/activity-tracker//images/checkmark-icon.svg) [Location-based events](https://cloud.ibm.com/docs/databases-for-mysql?topic=cloud-databases-activity-tracker-integration) 
 [IBM® Db2® on Cloud](https://cloud.ibm.com/docs/Db2onCloud?topic=Db2onCloud-about) db2oncloud !
Microsoft SharePoint Online 

Crawl documents that are stored in an online Microsoft SharePoint data source.

IBM Cloud Pak for Data

IBM Cloud Pak for Data only

This information applies only to installed deployments. For more information about connecting to an online SharePoint site from a managed deployment, see [SharePoint Online](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-sharepoint-online-cloud).



 What documents are crawled 



* During the initial crawl of the content, documents from all of the objects that can be accessed from the site collection path that you specify are crawled and added to your collection. Custom metadata that is associated with the SharePoint content is crawled also.
* You can crawl one site collection path per collection.
* Only documents that are supported by Discovery are crawled; all others are ignored. For more information, see [Supported file types](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionssupportedfiletypes).
* Document-level security is supported. When this option is enabled, your users can crawl and query the same content that they can access when they are logged in to SharePoint. For more information, see [Supporting document-level security](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collection-typesconfiguredls).
* When a source is recrawled, new documents are added, updated documents are modified to the current version, and deleted documents are deleted from the collection's index.
* All Discovery data source connectors are read-only. Regardless of the permissions that are granted to the crawl account, Discovery never writes, updates, or deletes any content in the original data source.



Discovery can crawl the following objects:



* Site Collections
* Sites
* SubSites
* Lists
* List Items
* Document Libraries
* List Item Attachments







 Data source requirements
Provision a Databases for MongoDB Enterprise Edition instance with Terraform 

In this tutorial, you learn how to use Terraform to provision a Databases for MongoDB Enterprise Edition instance that includes the Databases for MongoDB Enterprise Edition Analytics Add-On. The Databases for MongoDB EE Analytics Add-On allows you to run long-running analytical queries and/or provision a [MongoDB Connector for business intelligence(BI)](https://docs.mongodb.com/bi-connector/current/) to make your query data compatible with BI tools, such as [Tableau](https://www.tableau.com/).



 Overview of the available tools 

Before beginning the process of provisioning a database with Terraform, [you need to have an IBM Cloud account.](https://cloud.ibm.com/registration)

In this tutorial, you provision your database by using Terraform, a service that enables you to safely and predictably create, change, and improve infrastructure. It is an open source tool that codifies APIs into declarative configuration files that can be shared among team members, which are treated as code, edited, reviewed, and versioned. It is infrastructure as code. You write down what your infrastructure should look like and Terraform will create, update, and remove cloud resources as needed. For more information, see [Understand the basics of Terraform.](https://www.terraform.io/intro)

To support a multi-cloud approach, Terraform works with providers. A provider is responsible for understanding API interactions and exposing resources. IBM Cloud® has its provider for Terraform, enabling users of IBM Cloud® to manage resources with Terraform. Although Terraform is categorized as infrastructure as code, it is not limited to Infrastructure-As-A-Service resources. For more information, see [ibm_database](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/resources/database).





 Step 1: Install Terraform
* Fetch document by _id.
* Update document whose _id = 'x'.
* Delete document whose _id = 'x'.
* Get documents in the _id range 'a' to 'z'.



These operations are the building blocks of a database, but they get you only so far. What if you need to return a subset of documents that match on fields within the document? A person's birth date? A book's title? An order's value?

Querying comes in here.

IBM Cloudant has several methods for querying data. The first we look at is called IBM Cloudant Query.

IBM Cloudant Query's language was inspired by the MongoDB query language. Queries are expressed in JSON, where the selector attribute describes the subset of data to return. The query JSON is posted to the database's _find endpoint to perform a query.

The simplest form of query is finding documents where an attribute has a fixed value, for example, where author == J Smith.

The second example shows two clauses in the query. Both clauses must be satisfied for a document to make it into the search results, for example, where isbn === 6725252 AND date = 2018-01-01.

The third example shows how logical operators can be added. The $gt operation means greater than (you can also use gte for greater than or equal to, and lt/lte for the equivalent less than comparators). The $or operator is an OR operation, so a matching document must have a date greater than the one in the query either an author of J Smith OR title of Murder.

If you need to access objects within documents, you can use standard dot notation for example, address.zipcode to access a postal code string inside an address object.

We can also add the following parameters:

Fields
: Specifies the document attributes that we want returned (the default is the entire document).

Sort
: Defines how the data is to be sorted. Sort is an array, allowing the sort to be calculated on multiple attributes.

Limit
: The number of documents to return.
[Stopwords v1 API](https://cloud.ibm.com/apidocs/discoverygetstopwordliststatus) [Stopwords v2 API](https://cloud.ibm.com/apidocs/discovery-datagetstopwordlist) 







 Documents 



Documents API support details

 Action v1 API v2 API 

 List documents Not available from the v1 API [GET /v2/projects/{project_id}/collections/{collection_id}/documents](https://cloud.ibm.com/apidocs/discovery-datalistdocuments) 
 Create a document [POST /v1/environments/{environment_id}/collections/{collection_id}/documents](https://cloud.ibm.com/apidocs/discoveryadddocument) [POST /v2/projects/{project_id}/collections/{collection_id}/documents](https://cloud.ibm.com/apidocs/discovery-dataadddocument) <br>Unlike v1, the v2 response does not include a notices object. However, you can get notices information by using the Get document details method in v2. 
 Update a document [POST /v1/environments/{environment_id}/collections /{collection_id}/documents/{document_id}](https://cloud.ibm.com/apidocs/discoveryupdatedocument) [POST /v2/projects/{project_id}/collections/{collection_id}/documents/{document_id}](https://cloud.ibm.com/apidocs/discovery-dataupdatedocument) <br>When you update a document that was split, all of the document segments are overwritten.
Quick Facts for IBM Cloud Object Storage 

Finding what you need shouldn't be a chore. Here are some links to help you get to select points of interest in IBM Cloud® Object Storage.

The information you store with IBM Cloud Object Storage is encrypted and dispersed across multiple geographic locations. Your data is accessed over popular protocols like HTTPS using a modern RESTful API.

For your first steps at IBM Cloud Object Storage, you will need to [order storage](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-provision). Storage can be ordered through the console once you've [logged in](https://cloud.ibm.com/login) and further configured using the command line or a GUI tool.

Many helpful tips for [administrators](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-administrators) and [developers](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-gs-dev) are available in the documentation.

Get to know the options for replicating and accessing your data by learning about [regions](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-endpoints) used in IBM Cloud Object Storage. The resiliency and availability of your information will be a factor in your choice.

After provisioning your instance of Object Storage, you will want to learn how to control access using [Identity and Access Management](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-iam). Other options for controlling access are available, and your needs will determine how best to proceed.

While there are many tools for working with IBM Cloud Object Storage developers using one of the supported languages will want to check the [libraries and SDKs](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-sdk-gs) that are available.



 Next Steps
(https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/gs-json-tab.png)

Figure 13. JSON representation of the document

You can explore the JSON representation to see information that Discovery captured from the document. For example, if you expand the enriched_text section, and then expand the entities section, you can see mentions of entities that were recognized and tagged by the Entities enrichment.

Zoom

![Shows the entriched_text.entities section of the JSON.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/gs-json-entities.png)

Figure 14. Shows the enrichment_text.entities section of the JSON representation









 Step 5: Customize the sample project 

Now, let's customize the search result view a bit by adding a facet. A facet is a way to organize and classify documents that share similar patterns or content.



1. From the Improve and customize page, submit the following natural language query:

How do I install Discovery?
2. Review the query results that are displayed.

Zoom

![Shows the query results for a natural language query with the Top Entities facet expanded.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/gs-sample-entities-facet.png)

Figure 15. Top Entities facet results

Notice that a Top Entities section is displayed. You can expand the entities and click one of them to filter the query results to show only those results in which the entity is mentioned.
For more information about how to create a collection programmatically, see the [API reference documentation](https://cloud.ibm.com/apidocs/discovery-datacreatecollection).





 Optical character recognition 

One of the optional features that you can apply to a collection when you create it is optical character recognition. The optical character recognition (OCR) feature extracts text from images. This capability is useful for preserving information that is depicted in diagrams or graphs, or in text that is embedded in files such as scanned PDFs. By converting the visual information into text, it can later be searched.

IBM Cloud

A new version of the technology was introduced in cloud-managed instances. OCR v2 was developed by IBM Research to be better at extracting text from scanned documents and other images that have the following limitations:



* Low-quality images due to incorrect scanner settings, insufficient resolution, bad lighting (such as with mobile capture), loss of focus, misaligned pages, and badly printed documents
* Documents with irregular fonts or various colors, font sizes, and backgrounds



Things to keep in mind when you enable OCR:



* The time that it takes to ingest a document with images increases when OCR is enabled.
* OCR can read both clear and noisy images. It can convert noisy images to gray scale, and smooth and de-skew them. However, the image quality must meet the minimum requirement of 80 DPI (dots per inch).
* OCR can recognize many languages, but the language of the text in the image must be the same as the language that is specified for the collection where the file is added.



For more information about languages for which OCR v1 and OCR v2 are supported, see [Language support](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-language-support).

For a list of files types where you can apply OCR, see the [Supported file types](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionssupportedfiletypes) table.





 Enabling stemming for uncurated data IBM Cloud Pak for Data
Databases <br><br> * [Hyper Protect DBaaS for MongoDB](https://cloud.ibm.com/docs/hyper-protect-dbaas-for-mongodb?topic=hyper-protect-dbaas-for-mongodb-hpcs-byok)<br> * [Hyper Protect DBaaS for PostgreSQL](https://cloud.ibm.com/docs/hyper-protect-dbaas-for-postgresql?topic=hyper-protect-dbaas-for-postgresql-hpcs-byok)<br><br><br> 







 Related controls in IBM Cloud Framework for Financial Services 

The following [IBM Cloud Framework for Financial Services controls](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-aboutframework-control-requirements) are most related to this guidance. However, in addition to following the guidance here, do your own due diligence to ensure you meet the requirements.



Table 2. Related controls in IBM Cloud Framework for Financial Services

 Family Control
Using [Databases for MongoDB](https://cloud.ibm.com/docs/databases-for-mongodb), [create a Modern web application with MEAN stack](https://cloud.ibm.com/docs/solution-tutorials?topic=solution-tutorials-mean-stack). You learn how to run a MEAN starter locally, create and use a managed database-as-a-service (DBasS), deploy the app to IBM Cloud, and scale the database resources.





 Scale workloads in shared and dedicated VPC environments 

Use [Databases for PostgreSQL](https://cloud.ibm.com/docs/databases-for-postgresql) to set up isolated workloads in a shared (multi-tenant) environment and a dedicated (single-tenant) environment in the [Scale workloads in shared and dedicated VPC environments](https://cloud.ibm.com/docs/solution-tutorials?topic=solution-tutorials-vpc-scaling-dedicated-compute) tutorial.







 Cloud Databases Tutorials 



 Deploying and Connecting a Cloud Databases Instance 

This [tutorial](https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-create-instance-tutorial) guides you through the process of deploying a Cloud Databases instance and connecting it to a web front end. These values are then stored in a database running on Cloud Databases. You install the database infrastructure by using Terraform and your web application uses the popular Express framework. The application can then be run locally, or by using Docker.





 An example Kubernetes Service Application 

The [Cloud Databases "Hello World" Kubernetes tutorial](https://github.com/IBM-Cloud/clouddatabases-helloworld-kubernetes-examples) repository holds sample IBM Cloud® applications that show you how to connect to a Cloud Databases deployment to an IBM Cloud Kubernetes Service application written in various programming languages.

Each Git branch of the examples repository corresponds to samples in a particular programming language. For example, when you click Branch, a Node branch exists where all examples are written in JavaScript that uses Node.js.
AnyDB - SAP MaxDB 

The SAP systems in a landscape have specific requirements for servers, operating systems, network setup, and supported storage.

Deploying SAP AnyDB on IBM Cloud is similar to deploying the infrastructure with on-premises data centers. Therefore, use the information that is provided from SAP and the RDBMS providers.

To assist your project's planning phase, extra design considerations are listed for SAP AnyDB - SAP Max DB with IBM Cloud® for SAP.



 Overview of SAP MaxDB with IBM Cloud® 

Before you start deploying SAP MaxDB software, be sure to:



* Check all relevant SAP MaxDB information and prerequisites (for example, SAP Notes)
* Verify that all required packages are installed for the relevant OS that that is used for SAP MaxDB





 Documentation of SAP MaxDB 

A good entry point into the documentation is the [SAP community page for SAP MaxDB](https://community.sap.com/topics/maxdb).

SAP MaxDB documentation:



* [SAP Help Portal - SAP MaxDB](https://help.sap.com/viewer/product/SAP_MAXDB/latest/en-US)
* [SAP Note 767598 - Available SAP MaxDB documentation](https://launchpad.support.sap.com//notes/767598)
* [SAP Note 1020175 - FAQ: SAP MaxDB installation, upgrade, or applying a patch](https://launchpad.support.sap.com//notes/1020175)



For a current overview of the combinations of SAP MaxDB, SAP NetWeaver (or other SAP components), and operating systems, see the [Product Availability Matrix (PAM)](http://support.sap.com/pam).





 SAP MaxDB using Intel Bare Metal 

See [SAP Note 2414097 - SAP Applications on IBM Cloud Classic Infrastructure environment](https://launchpad.support.sap.com//notes/2414097) for supported SAP MaxDB versions.





 SAP MaxDB using Intel Virtual Servers
Log Analysis Integration 

Cloud Databases are integrated with [IBM Cloud® Log Analysis](https://cloud.ibm.com/docs/log-analysis), so you can view database platform logs.

This document covers the integration of Log Analysis with Cloud Databases, which includes Databases for DataStax, Databases for Elasticsearch, Databases for EnterpriseDB, Databases for etcd, Databases for MongoDB, Databases for PostgreSQL, Databases for Redis, IBM Cloud® Databases for MySQL, and Messages for RabbitMQ.

Currently, IBM Cloud® Log Analysis integration is available for Cloud Databases deployments according to the following table.



Table 1. Log Analysis regions

 Deployment Region Log Analysis Region 

 us-south us-south 
 jp-tok jp-tok 
 eu-gb eu-gb 
 che01 che01 
 eu-de eu-de 
 au-syd au-syd 
 us-east us-east 
 ca-tor ca-tor 
 jp-osa jp-tok 
 br-sao br-sao 
 par01 eu-de 



Platform logs from your deployments appear in a Log Analysis instance in the same region, except for jp-osa. Deployments in jp-osa have their logs forwarded to jp-tok. If you have deployments in multiple regions, you must set up Log Analysis in multiple regions.



 Provisioning IBM Cloud® Log Analysis 

Platform log information from your databases is automatically forwarded to IBM Log Analysis, but to access it you must enable platform logging by [provisioning a Log Analysis service](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-provision) in your IBM Cloud account and [configuring the service to receive IBM Cloud platform logs](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-config_svc_logs).

This setting enables platform logs from ALL IBM Cloud services on your account that have IBM Log Analysis integration to send logs to your IBM Log Analysis service.