vania-janet/multiturn-rag-retrieval-data huggingface dataset:
IBM Cloud Log Analysis integration 

IBM® Cloudant® for IBM Cloud® is integrated with [IBM Cloud® Log Analysis](https://cloud.ibm.com/catalog/services/ibm-log-analysis-with-logdna), so you can view database logs.

Currently, IBM Cloud Log Analysis integration is available for IBM Cloudant deployments according to the following table:



Table 1. IBM Cloud Log Analysis regions

 Deployment Region IBM Cloud Log Analysis Region 

 Chennai Chennai 
 Dallas Dallas 
 Frankfurt Frankfurt 
 London London 
 Osaka Tokyo 
 São Paulo São Paulo 
 Sydney Sydney 
 Tokyo Tokyo 
 Toronto Toronto 
 Washington DC Washington DC 





 Provisioning IBM Cloud Log Analysis 

Log information from your databases is automatically forwarded to IBM Cloud Log Analysis. In order to access it, you must [provision IBM Cloud Log Analysis](https://cloud.ibm.com/catalog/services/ibm-log-analysis-with-logdna) in your IBM Cloud account and [configure the service](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-config_svc_logs) to receive IBM Cloud service logs.

This setting enables logs from all IBM Cloud services on your account with IBM Cloud Log Analysis integration sending logs to your IBM Cloud Log Analysis service. For more information, see the list of [integrated services](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-cloud_servicescloud_services).

IBM Cloud Log Analysis has a Lite plan that is free to use, but it offers only streaming events. To take advantage of the tagging, export, retention, and other features, you need to use one of the [paid plans](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-service_plans).



 HIPAA 

IBM Cloud Log Analysis does not currently offer a HIPAA-compliant plan for the service.

Use caution when you configure the platform service logs, since this setting can impact other services that require HIPAA compliance.







 Using IBM Cloud Log Analysis
* [Scale disk, memory, and CPU](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-resources-scaling)
* [Configure Autoscaling](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-autoscaling)







 Backups 

The Backups tab is the UI for managing your deployments backups. All of the available backups are listed with their timestamps. Click a backup to grab its ID or to restore it into a new deployment. More information is on the [Managing Backups](https://cloud.ibm.com/docs/databases-for-mongodb?topic=cloud-databases-dashboard-backups) page.





 Observability 

The Observability tab provides access to the IBM Cloud monitoring, logging, and event tracking integrations available for your deployment.



* [IBM Cloud® Activity Tracker](https://cloud.ibm.com/docs/databases-for-mongodb?topic=cloud-databases-activity-tracker)
* [IBM® Log Analysis](https://cloud.ibm.com/docs/databases-for-mongodb?topic=cloud-databases-logging)
* [IBM Cloud® Monitoring](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-monitoring)







 Settings 

The Settings tab contains the UI for many of the tunable settings for your deployment. You can



* view encryption details. Encryption at rest is enabled for all Databases for MongoDB deployments. If you brought your own encryption key from [Key Protect](https://cloud.ibm.com/docs/databases-for-mongodb?topic=cloud-databases-key-protect), the panel provides a link to your Key Protect instance and the Encryption Key field has the name of the key.
[IBM Cloudant](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-getting-started-with-cloudant) IBM Cloudant is a document-oriented database as a service (DBaaS). It stores data as documents in JSON format. [Platform metrics](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-monitor-ibm-cloud-pm) 
 [IBM Cloud Databases for PostgreSQL](https://cloud.ibm.com/docs/databases-for-postgresql?topic=databases-for-postgresql-getting-started) IBM Cloud Databases for PostgreSQL is a managed PostgreSQL service that is hosted in the IBM Cloud and integrated with other IBM Cloud services. [Platform metrics](https://cloud.ibm.com/docs/databases-for-postgresql?topic=databases-for-postgresql-monitoring) 
 [IBM Cloud Databases for Redis](https://cloud.ibm.com/docs/databases-for-redis?topic=databases-for-redis-getting-started) IBM Cloud Databases for Redis is a managed service that is hosted in the IBM Cloud and integrated with other IBM Cloud services. [Platform metrics](https://cloud.ibm.com/docs/databases-for-redis?topic=databases-for-redis-monitoring) 
 [IBM Cloud Databases for etcd](https://cloud.ibm.com/docs/databases-for-etcd?topic=databases-for-etcd-getting-started) IBM Cloud Databases for etcd is a managed etcd service that is hosted in the IBM Cloud and integrated with other IBM Cloud services. [Platform metrics](https://cloud.ibm.com/docs/databases-for-etcd?topic=databases-for-etcd-monitoring)
The queries that we executed in the previous part were not optimal: to get the answer, IBM Cloudant had to spool through every document in the database in turn to see whether it met with the search criteria.

To make queries that are run in a performant and scalable way, we need Indexing.

With IBM Cloudant, you can specify any number of Indexes (or indices).

An index is a secondary data structure that is built from the document list. It contains data that is sorted by the fields you specify, for example, books that are sorted by date and title. If you perform a query that asks for data that matches a document's date and title, the indexed data structure can be used to speed up the query process. Instead of scanning through every document in turn, IBM Cloudant can jump to the relevant part of the index (say, the section on 20th century books) and retrieve the data much more quickly.

IBM Cloudant Query indexes include two types of indexes: type=json and type=text. These indexes are backed by two underlying indexing technologies that we meet in subsequent parts of this course.

An index is defined when you POST some JSON to a database's _index endpoint.

The index object contains a fields array, which specifies which document attributes to index. Usually, the fields that need indexing are equivalent to the attributes used in the selector of a query you're going to use to retrieve the data. That is, if you need to query by the date field, we need to index the date field.

Although the name of an index is optional, it's good practice and we follow this convention. It's good to ask IBM Cloudant a question and specify the name of the index you intend it to use. This practice saves IBM Cloudant from having to choose which index to use from the available ones, and it makes it easy for you to remember which index is which.

Let's create an index on our books database from the dashboard. Select the database, then choose the Design Documents tab and Query Indexes from the menu.

Any existing indexes are listed on the side: A special index must exist that represents the primary index, based on the document's _id.
Why isn't an IBM Cloud docs page available? 

If you go to an IBM Cloud® docs page and the content doesn't display, try searching for the content instead to get you to the right place.

  What’s happening 

You click a docs link, select a docs search result, or follow an existing bookmark to the IBM Cloud docs, and the following error is displayed.

> Sorry, this content isn't available.
>
> The information you're looking for might have moved. Try searching the IBM Cloud Docs to find related content.

  Why it’s happening 

The page might have been moved or removed entirely, or the link itself might be incorrect.

  How to fix it 

Search the IBM Cloud docs to find related information.



1.  On the error page, click Search in Docs to go directly to the [IBM Cloud docs search](https://cloud.ibm.com/docs/search).
2.  Enter search terms for the topic that you're trying to find, and press Enter. For example, if the page was about configuring Kubernetes clusters, you might enter config Kubernetes cluster.
3.  To search in the docs for a product or category of products, refine your search results by clicking the Filter icon ![Filter icon "Filter"](https://cloud.ibm.com/docs-content/v1/content/adcecdbe4e86955f88aac7f3c1e7ec3ff44992ef/icons/filter.svg), selecting the category or product, and clicking Filter.



If you still can't find the page that you were looking for, let the IBM Cloud docs team know about the missing page by [submitting feedback](https://cloud.ibm.com/docs/overview?topic=overview-feedback). In your feedback, include information about where you found the link to the missing page.
This tutorial shows how to provision a SQL (relational) database service. As administrator, you create a table and load a large data set (city information) into the database. Then, you deploy a web app "worldcities" to Code Engine. The app allows regular users to look up records from the cloud database. The app is written in Python using the Flask framework.

Code Engine Db2 Warehouse on Cloud



* 2 hours
* 2023-05-05



[Mapping Global COVID-19 cases with the Databases for MongoDB EE (Enterprise Edition) Analytics Add-On and Tableau](https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-bi-connector-tutorial-description)Mapping Global COVID-19 cases with the Databases for MongoDB EE (Enterprise Edition) Analytics Add-On and Tableau

The Databases for MongoDB EE (Enterprise Edition) Analytics Add-On allows you to run long-running analytical queries or provision a MongoDB Connector for business intelligence(BI) to make your query data compatible with BI tools, such as Tableau.

IBM Cloud Databases



* 2 hours
* 2022-06-22



[Example context-based restrictions scenarios](https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-cbr-tutorial)Example context-based restrictions scenarios

With context-based restrictions, account owners and administrators can define and enforce access restrictions for IBM Cloud® resources, based on the context of access requests. Access to Cloud Databases resources can be controlled with context-based restrictions and identity and access management policies. For more information, see Protecting Cloud Databases resources with context-based restrictions.

IBM Cloud Databases



* 30 minutes
* 2022-10-26



[Setting up disk alerts for disk utilization](https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-disk-util-alert-tutorial)Setting up disk alerts for disk utilization

Getting timely alerts about resource utilization is key to managing your database, avoiding problems, and mitigating downtime.
For more information, see [Creating collections](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collections)





 Can I upload documents? 

Yes, you can upload documents directly to a collection in your project. An upload is a one-time operation that you can use to get started. An alternative approach is to connect to a data source and crawl the source for information. When you crawl data sources, the data can stay where it is and you can set up a schedule by which to crawl the external source to find new and changed information. When you crawl the data, you know that the information in your collection is always up to date. For more information, see [Creating collections](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collections).





 Must all my documents be English? 

No. Discovery support multiple languages. For more information about language support per feature, see [Language support](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-language-support).





 What types of files can Discovery ingest? 

Discovery can ingest most standard business file types, including PDF, Microsoft Word documents, spreadsheets, and presentations. For a complete list, see [Supported file types](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionssupportedfiletypes).





 How do I know whether I have Discovery v1 or v2? 

If you're using Discovery on IBM Cloud Pak® for Data, then you're using Discovery v2.

If you have a service instance that is managed by IBM Cloud, then check what you see when you launch the product. When you open the product user interface in v2, the following page is displayed:

Zoom

![Shows the main My Projects page with a single Sample Project tile.]
Overview of IBM Cloud data sources 

You can use IBM Watson® Discovery on the IBM Cloud® to connect to and crawl documents from remote sources.

IBM Cloud

IBM Cloud only

This information applies only to managed deployments. For more information about IBM Cloud Pak for Data data sources, see [Overview of Cloud Pak for Data data sources](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collection-types).

Connect to an external data source so that you can pull documents into Discovery on a schedule. Discovery pulls documents from the data source by crawling the data source. Crawling is the process of systematically browsing and retrieving documents from a starting location that you specify. When the crawler first processes a data source, it performs a full crawl. Each time the crawler runs after the initial crawl, it performs a refresh, where it checks for new and changed files only.

All Discovery data source connectors are read-only. Regardless of the permissions that are granted to the crawl account, Discovery never writes, updates, or deletes any content in the original data source.

You can use Discovery to crawl from the following data sources:



* [Box](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-box-cloud)
* [IBM Cloud Object Storage](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-cos-cloud)
* [Microsoft SharePoint Online](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-sharepoint-online-cloud)
* [Microsoft SharePoint On Prem](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-sharepoint-onprem-cloud)
* [Salesforce](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-salesforce-cloud)
* [Web crawl](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-web-cloud)
4. Do one of the following things to explore the search result.

IBM Cloud



1. Click Open advanced view.

Useful summary information is displayed, such as the number of occurrences of any enrichments that are detected in the document.
2. Select the URL entity to highlight mentions of URLs within the text.

Zoom

![Shows the advanced text view with URL entities highlighted.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/gs-doc-advanced-view.png)

Figure 11. Advanced view that shows entities that were recognized
3. To see how the information from the document is stored in JSON format, click the View as menu from the view header, and select JSON.

Zoom

![Shows the overflow menu options from the advanced text view.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/gs-display-options-menu.png)

Figure 12. Overflow menu options from the Advanced view



IBM Cloud Pak for Data



1. Click JSON.

A JSON representation of the document is displayed.

Zoom

![Shows the JSON representation of the document.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/gs-json-tab.png)

Figure 13. JSON representation of the document

You can explore the JSON representation to see information that Discovery captured from the document.
Comparing Apache CouchDB and IBM Cloudant 

The differences between the fully managed cloud service IBM® Cloudant® for IBM Cloud® and self-managed open source Apache CouchDB still exist and are discussed here.

The foundation of the IBM Cloudant managed database service is the Apache CouchDB database. IBM is active in the Apache CouchDB committee, employs members of the PMC, and commits most of its feature, functions, and enhancements back to the open source project. Over the last few years, IBM made significant effort to align the core feature set of IBM Cloudant and CouchDB. The IBM Cloudant team contributed key features like IBM Cloudant Query and Mango query language, full-text search, and partition queries to CouchDB.

Apache CouchDB and IBM Cloudant are nearly fully API compatible, which means they can serve as drop-in replacements for each other in your application. They speak the same replication protocol so that you can seamlessly synchronize data between IBM Cloudant and CouchDB instances.

For more information, see the [API comparison guide](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-comparison-of-ibm-cloudant-and-couchdb-api-endpoints) for a detailed breakdown of the API endpoints.

The following table shows the feature and function differences that you must be cognizant of when you use the Apache CouchDB and IBM Cloudant data layer ecosystem.



Table 1. Feature and function differences between IBM Cloudant and Apache CouchDB

 Feature CouchDB 1.6 CouchDB 2.3.1 CouchDB 3.x IBM Cloudant 

 Clustering No Yes Yes Yes 
 Fauxton Dashboard UI No Yes Yes Yes 
 MapReduce view Yes Yes Yes Yes 
 Mango and IBM Cloudant Query No Yes Yes Yes 
 Full-text search No No Yes, requires separate installer or container. Yes 
 Partition queries No No Yes Yes 
 Shard splitting No No Yes Available as tool for IBM Ops. 
 Selector on changes feed No Yes Yes Yes 
 Rate limits No No No User-defined [provisioned throughput capacity](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-ibm-cloud-publicprovisioned-throughput-capacity) settings
Databases for EnterpriseDB [Scaling Disk, RAM, and CPU](https://cloud.ibm.com/docs/databases-for-enterprisedb?topic=databases-for-enterprisedb-resources-scaling&interface=ui) [Autoscaling](https://cloud.ibm.com/docs/databases-for-enterprisedb?topic=databases-for-enterprisedb-autoscaling&interface=ui) 
 Databases for etcd [Scaling Disk, RAM, and CPU](https://cloud.ibm.com/docs/databases-for-etcd?topic=databases-for-etcd-resources-scaling&interface=ui) [Autoscaling](https://cloud.ibm.com/docs/databases-for-etcd?topic=databases-for-etcd-autoscaling&interface=ui) 
 Databases for MongoDB [Scaling Disk, RAM, and CPU](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-resources-scaling&interface=ui) [Autoscaling](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-autoscaling&interface=ui) 
 Databases for PostgreSQL [Scaling Disk, RAM, and CPU](https://cloud.ibm.com/docs/databases-for-postgresql?topic=databases-for-postgresql-resources-scaling&interface=ui) [Autoscaling](https://cloud.ibm.com/docs/databases-for-postgresql?topic=databases-for-postgresql-autoscaling&interface=ui) 
 Databases for Redis [Scaling Disk, RAM, and CPU](https://cloud.ibm.com/docs/databases-for-redis?topic=databases-for-redis-resources-scaling&interface=ui) [Autoscaling](https://cloud.ibm.com/docs/databases-for-redis?topic=databases-for-redis-autoscaling&interface=ui)
Searching archive data by using the Data Engine service 

You can use the Data Engine service to query IBM Cloud Activity Tracker (AT) archive files that are stored in an Object Storage (COS) bucket in your account. You can run queries from the IBM Cloud UI, or programmatically.

Zoom

![IBM Cloud services integration scenario](https://cloud.ibm.com/docs-content/v1/content/6f3c07c34058f637b1c86044ca4f9b24214330a7/activity-tracker/images/Activity-Tracker-Search-03-Search-Archive.svg)

Figure 1. IBM Cloud services integration scenario

Each IBM Cloud Activity Tracker instance has a service plan associated that indicates the number of days that you can query data through the web UI. To have access to the events after this period, you must enable archiving in the IBM Cloud Activity Tracker instance. When you enable archiving, you get an archive file hourly. This file contains information for that hour. Notice that UTC timestamps are used to determine which records are included in each file. The file name includes the ID of your IBM Cloud Activity Tracker instance and the date of the records that are included in the file.

To query the archive data in a file, you can use the Data Engine service. The service offers an SQL editor through the UI, and also programmatic options such as a REST API.

Use the Data Engine user interface (UI) to develop and test your queries, and the [SQL Query REST API](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-sqlqueryrestapi) to automate them.

The Data Engine service provides a serverless, no-ETL solution to easily query data stored in Object Storage. Underneath, SQL Query uses Apache Spark SQL as its underlying query engine.

You can use the Data Engine to run SQL queries (that is, SELECT statements) to analyze, transform structured and semi-structured data, or clean up rectangular data. You cannot run actions such as CREATE, DELETE, INSERT, and UPDATE.
[IBM Developer<br><br>Visit IBM Developer for technical articles, code patterns, tutorials, and more.<br><br>![112-Developer-tools.svg](https://cloud.ibm.com/media/docs/images/homepage/112-Developer-tools.svg)](https://developer.ibm.com/depmodels/cloud/)[Architecture Center<br><br>Discover the architecture references available for this product.<br><br>![190-Application-Platform.svg](https://cloud.ibm.com/media/docs/images/homepage/190-Application-Platform.svg)](https://www.ibm.com/cloud/architecture)[IBM Training<br><br>Build your skills with IBM Cloud training.<br><br>![83-Elements-of-the-cloud.svg](https://cloud.ibm.com/media/docs/images/homepage/83-Elements-of-the-cloud.svg)](https://www.ibm.com/training/cloud)[Community<br><br>Join the IBM Cloud community and collaborate with others.<br><br>![35-Connectivity.svg](https://cloud.ibm.com/media/docs/images/homepage/35-Connectivity.svg)](https://community.ibm.com/community/user/publiccloud/home)

 Latest updates 

[View more](https://cloud.ibm.com/docs/messages-for-rabbitmq?topic=messages-for-rabbitmq-rabbitmq-relnotes)09 March 2023 | RabbitMQ Version 3.11 Release

IBM Cloud® Messages for RabbitMQ version 3.11 is now available.

19 October 2022 | Deploying and Connecting an IBM Cloud® Databases Instance Tutorial

This tutorial guides you through the process of deploying a Cloud Databases instance and connecting it to a web front end by creating a webpage that allows visitors to input a word and its definition. These values are then stored in a database running on Cloud Databases.
For more information, see [Data Engine overview](https://cloud.ibm.com/docs/sql-query?topic=sql-query-overview).



 Prerequisites 

To be able to use the Data Engine service to query archived event files, check the following prerequisites:



* You must have access to the COS instance where the logging data is available.

You must have access to a bucket that contains the Log Analysis archive files and a bucket to use to store results from your queries.
* You must have an Log Analysis instance provisioned in your account that has [archiving configured to a COS bucket in IBM Cloud](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-archiving-ov).
* You must have 1 or more archive files uploaded in the bucket.
* You must have a policy on the COS service with at least platform role reader so that you can view data in the COS bucket where the archive files are uploaded.
* You must have a policy on the COS instance with at least Writer access to one COS bucket so that result files (files containing output data) can be written there.
* You must have permissions to query the data by using the Data Engine service. For more information about roles, see [Required user roles](https://cloud.ibm.com/docs/sql-query?topic=sql-query-overviewuser-roles).







 Step 1. Provisioning an Data Engine instance 

To provision an instance, see [Create your Data Engine service instance](https://cloud.ibm.com/docs/sql-query?topic=sql-query-getting-started).

To see the list of supported regions, see [Availability zones](https://cloud.ibm.com/docs/sql-query?topic=sql-query-overviewavailability).





 Step 2. Granting user permissions to run a query 

To run queries with the SQL query service, a user needs a platform role and a service role.