vania-janet/multiturn-rag-retrieval-data huggingface dataset:
For more information, see the [authentication documentation](https://cloud.ibm.com/docs/sql-query?topic=sql-query-authentication).





 November 2019 

MULTILINE option
: You can specify a [MULTILINE](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceexternalTableSpec) option for JSON input data if individual JSON records are stored across multiple lines.





 October 2019 

New open source script for uploading large volumes
: The new open source [cos-upload](https://github.com/IBM-Cloud/data-lake/tree/master/upload/cos-upload) script can be used to efficiently upload large volumes of data to IBM Cloud Object Storage buckets with Aspera by merely providing an IAM API Key.

JSON parser
: Data Engine JSON parser is processing and extracting all JSON keys in lowercase, so it can work correctly on LogDNA data.

Run automatic SQL-based post processing
: Storing new objects in IBM Cloud Object Storage can trigger Data Engine executions. It is enabled by the IBM Cloud Object Storage [event provider](https://www.ibm.com/cloud/blog/announcements/cloud-object-storage-event-provider-enables-hello-serverless-and-more) for IBM Cloud® Functions. By combining it with the SQL [Cloud Functions](https://hub.docker.com/r/ibmfunctions/sqlquery/), you can automatically run SQL-based post processing for new objects.

Query hints
: Data Engine has query hints for SQL queries that have potential for faster execution by using certain features of Data Engine. These hints are flagged with a light bulb icon in the job list and the specific hint is available inside the Details pane.





 September 2019 

Support for ETL to IBM® Db2® on Cloud
Using IBM Cloudant Query FAQ 

[IBM® Cloudant® for IBM Cloud® Query](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-query) is an API for querying slices of data based on the values of a database's document attributes. It is a flexible API that must be used carefully to ensure that database performance can be maintained as the data size grows over time.



 How do I use IBM Cloudant Query? 

IBM Cloudant Query is accessed through the [POST /{db}/_find](https://cloud.ibm.com/apidocs/cloudantpostfind) API endpoint where the JSON specification of the query is passed in the HTTP POST body. For example, this query finds up to 10 documents where the firstname is "Charles" and the surname is "Dickens":

{
"selector": {
"firstname": "Charles",
"surname": "Dickens"
},
"limit": 10
}

For more information, see [Selector Syntax](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-queryselector-syntax).





 How do I create an index to support an IBM Cloudant Query? 

Without a suitable secondary index, IBM Cloudant Query scans each document in the database in turn until it has enough matches to satisfy the query. The larger the data set and the more documents it has to scan to find matching documents, the slower the response time. For faster performance, an IBM Cloudant Query _find must be backed by a suitable secondary index. A secondary index is a pre-calculated data structure that allows IBM Cloudant to quickly jump to the slice of data it needs without scanning irrelevant documents. For the surname fields, we call the [POST /{db}/_index](https://cloud.ibm.com/apidocs/cloudantpostindex) endpoint to pass the JSON index definition as the HTTP POST body:

{
"index": {
"fields": ["firstname", "surname"]
},
"ddoc": "jsonindexes",
[checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/icons/checkmark-icon.svg) 





* PDF files that are secured with a password or certificate are not supported. Vector objects, including SVG images and vectorized text, are not supported. Only images of the supported image file types that occur in the PDF are rendered.
* Only single-page image files are supported.
* Files within compressed archive files (ZIP, GZIP, TAR) are extracted. Discovery ingests the supported file types within the archive; it ignores all other file types. The file names must be encoded in UTF-8. Files with names that include Japanese characters, for example, must be renamed before they are added to the ZIP file.
* Discovery supports MacOS ZIP files only if they are generated by using a command such as: zip -r my-folder.zip my-folder -x ".DS_Store". ZIP files that are created by right-clicking a folder and clicking Compress are not supported.
* PDF files that you upload as part of an archive file are not displayed in the advanced view for a query result that you open from the Improve and customize page. If you want the file to be viewable from the advanced view, reimport the PDF file separately from the archive file.



When you add files to a Document Retrieval for Contracts project type, any file types that support SDU and OCR are processed with a pretrained Smart Document Understanding model and Optical Character Recognition automatically.





 Document limits 

The number of documents that are allowed per service instance depends on your Discovery plan type.

The document limit applies to the number of documents in the index. Upload fewer documents at the start if the enrichments that you plan to apply might increase the number of documents later. For example, the following configurations generate more documents:



* Splitting documents segments one document into multiple documents
* CSV files that you upload generate one document per line
* Database data sources that you crawl produce one document per database row
General Availability of IBM Cloud® Databases for Redis support for IBM Cloud Databases enabled by IBM Cloud Satellite.
: A distributed cloud provides consistent security and services across environments, centralized workload visibility, reduced latency, easier compliance and higher application development velocity. See blog post announcement [here](https://www.ibm.com/cloud/blog/announcements/deploy-managed-cloud-native-databases-anywhere-with-ibm-cloud-satellite).





 13 April 2020 

IBM Cloud® Databases for Redis autoscaling
: We are excited to announce that autoscaling of your deployments based on disk capacity and disk I/O utilization is now available for IBM Cloud® Databases for Redis via the UI, API, and CLI. See blog post announcement [here](https://www.ibm.com/cloud/blog/announcements/ibm-cloud-databases-portfolio-introduces-autoscaling).





 17 October 2019 

IBM Cloud® Databases for Redis 5 is Available in Preview
: Redis 5 is now available in Preview on IBM Cloud® Databases for Redis. This release has lots of interesting new features, but the most important is the introduction of Redis Streams. See blog post announcement [here](https://www.ibm.com/cloud/blog/announcements/databases-for-redis-5-is-available-in-preview).





 6 August 2019 

New Regions Available for IBM Cloud Database Services
: IBM Cloud® Databases for Redis is now available to be deployed in Seoul; South Korea; and Chennai, India. See blog post announcement [here](https://www.ibm.com/cloud/blog/announcements/new-regions-available-for-ibm-cloud-database-services).





 2 October 2018 

General Availability of IBM Cloud® Databases for Redis
: IBM Cloud® Databases for Redis added to the [IBM Cloud Databases](https://www.ibm.com/cloud/databases) family. See blog post announcement [here](https://www.ibm.com/cloud/blog/ibm-cloud-databases-for-postgresql-and-databases-for-redis-are-now-generally-available).
New enhancements to query API : Enhancements are now available for the query API (GET /v1/environments/{environment_id}/collections/{collection_id}/query). See the [Query your collection](https://cloud.ibm.com/apidocs/discoveryquery-your-collection) method in the API reference for information.

New support for 'passages' parameter in the query API : The query API now supports the passages parameter. If the parameter is set to true, the query returns a set of the most relevant passages from the documents in your collection. The passages are generated by sophisticated Watson algorithms to determine the best passages of text from all of the documents returned by the query. This enables you to find information and context more precisely. See the [Query your collection](https://cloud.ibm.com/apidocs/discoveryquery-your-collection) method in the API reference for information. : Specifying passages=true in your query can reduce performance as a result of increased processing to extract passages. With larger environments, the performance impact can be lessened. : The passages parameter is supported only on private collections. It is not supported in the Watson Discovery News collection. : The passages parameter currently returns a maximum of 10 results. The number of returned results cannot be changed. [Update](https://cloud.ibm.com/docs/discovery?topic=discovery-query-parameterspassages_count) : The passages parameter returns a maximum of three (3) passages from any given document in the collection. If a document contains more than three additional relevant passages, the parameter does not return them.





 7 April 2017 

New support for 'sort' parameter in the query API : The query API (GET /v1/environments/{environment_id}/collections/{collection_id}/query) now supports the sort parameter, which enables you to specify a comma-separated list of fields in the document to sort on. See the [Query your collection](https://cloud.ibm.com/apidocs/discoveryquery-your-collection) method in the API reference for information.
[Infrastructure availability ](https://cloud.ibm.com/docs/overview?topic=overview-services_regioniaas-service-infra)

We make it easy for you to implement, host, and scale services, infrastructure, and apps but not all services and infrastructure are available for purchase in every IBM Cloud location.

[Locations for resource deployment ](https://cloud.ibm.com/docs/overview?topic=overview-locations)

We have a resilient global network of locations consisting of regions, zones, and data centers to host your highly available cloud workloads.

[Choose your compute ](https://cloud.ibm.com/docs/cloud-infrastructure?topic=cloud-infrastructure-compute)

Depending on your needs, you have various options, or you can mix it up if that's what your environment requires.

[Set up your networking ](https://cloud.ibm.com/docs/cloud-infrastructure?topic=cloud-infrastructure-network)

You automatically get connected to the Virtual Private Network (VPN) for VPC when your IBM Cloud account is set up.

[Choose your storage ](https://cloud.ibm.com/docs/cloud-infrastructure?topic=cloud-infrastructure-storage)

Bare Metal Servers and Virtual Servers are provisioned with default storage.

 Get support 

If you experience problems, we're here to help.

[FAQ library ](https://cloud.ibm.com/docs/faqs)

Review the FAQs for the IBM Cloud platform and services.

[Support plans ](https://cloud.ibm.com/docs/get-support?topic=get-support-support-plans)

You can choose a Basic, Advanced, or Premium support plan to customize your IBM Cloud support experience for your business needs.

[Viewing IBM Cloud status ](https://cloud.ibm.com/docs/get-support?topic=get-support-viewing-cloud-status)

The Status page is the central place to find details about major incidents that affect the IBM Cloud platform and services.
Pricing 

An IBM Cloud® Databases for MongoDB Standard plan deploys as one highly available MongoDB cluster with three data members. Your data is replicated on all members. The Databases for MongoDB Enterprise plan is also deployed as one highly available MongoDB cluster but with three data members.

The Standard plan is priced based on the total amount of disk storage, RAM, and backup storage that is allocated to deployments, prorated hourly. Databases for MongoDB deployments have a minimum of 10 GB of disk and 1 GB of RAM per data member.

The Enterprise plan pricing is based on the total amount of disk storage, RAM, and backup storage that is allocated to deployments, prorated hourly. Databases for MongoDB deployments have a minimum of 20 GB of disk, 6 cores, and 14 GB of RAM per data member.

A Databases for MongoDB EE Analytics node is priced as a data member.



 Using the Pricing Calculator 

Templates are provided for ease of use and provide balanced resource allocations appropriate for general-purpose workloads. The Custom tab can be used to configure Disk, RAM, and vCPU, as desired.

For pricing estimation, use the Add to Estimate button at the bottom of the [Databases for MongoDB catalog page](https://cloud.ibm.com/catalog/databases-for-mongodb). Input your total consumption across two data members into the calculator. This is roughly double the size of your data because your data is replicated to both members. For example, 10 GB of disk and 1 GB of RAM across two data members would be priced at 20 GB of disk and 2 GB of RAM respectively.





 Backups Pricing 

You receive your total disk space purchased, per database, in free backup storage. For example, in a given month, if you have a Databases for MongoDB deployment that has 20 GB of disk per member, and has three data members, you receive 60 GB of backup storage free for that month. If your backup storage utilization is greater than 60 GB for the month (in this scenario), you are charged an overage of $0.03/ month per gigabyte.

By default, Cloud Databases provides a daily backup that is stored for 30 days.
Why don't I get results in the FAQ or tutorial library? 

If you filter for content in IBM Cloud® docs FAQs or tutorials libraries and don't get results, try searching the IBM Cloud docs for related content.

  What’s happening 

You filter for content in the IBM Cloud docs [FAQ library](https://cloud.ibm.com/docs?tab=faqs) or [tutorials library](https://cloud.ibm.com/docs?tab=tutorials) and see the following related message.

In the FAQ library:

> No FAQs found
>
> It seems we can't find what you're looking for. Try updating your filter selections.

In the tutorials library:

> No tutorials found
>
> Sorry, there are no tutorials that match your criteria. Try updating your filter selections.

  Why it’s happening 

There might not be any content with your selected filters, or a topic might not be set with the correct filter value.

  How to fix it 

Clear the selected filters, and try filtering the library again. If you still don't see any content, try searching all of the IBM Cloud docs for related content.



1.  Go to the [IBM Cloud docs search](https://cloud.ibm.com/docs/search).
2.  Enter search terms for the topic that you're trying to find, and press Enter. For example, if the page was about configuring Kubernetes clusters, you might enter config Kubernetes cluster.
3.  To search in the docs for a product or category of products, refine your search results by clicking the Filter icon ![Filter icon "Filter"](https://cloud.ibm.com/docs-content/v1/content/adcecdbe4e86955f88aac7f3c1e7ec3ff44992ef/icons/filter.svg) , selecting the category or product, and clicking Filter.



If you still can't find a tutorial or FAQ with information that you were looking for, let the IBM Cloud docs team know about the missing page by [submitting feedback](https://cloud.ibm.com/docs/overview?topic=overview-feedback).
Databases <br><br> * [Hyper Protect DBaaS for MongoDB](https://cloud.ibm.com/docs/hyper-protect-dbaas-for-mongodb?topic=hyper-protect-dbaas-for-mongodb-high-availability-disaster-recovery)<br> * [Hyper Protect DBaaS for PostgreSQL](https://cloud.ibm.com/docs/hyper-protect-dbaas-for-postgresql?topic=hyper-protect-dbaas-for-postgresql-high-availability-disaster-recovery)<br><br><br> 



In addition to the Financial Services Validated services in the reference architecture, see the following references for other important information:



* [How IBM Cloud ensures high availability and disaster recovery](https://cloud.ibm.com/docs/overview?topic=overview-zero-downtime)
* [High availability](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-shared-high-availability) in the VPC reference architecture
* [Responsibilities for operating services in your deployment](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-shared-responsibilities) of the VPC reference architecture







 Related controls in IBM Cloud Framework for Financial Services 

The following [IBM Cloud Framework for Financial Services controls](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-aboutframework-control-requirements) are most related to this guidance. However, in addition to following the guidance here, do your own due diligence to ensure you meet the requirements.



Table 2. Related controls in IBM Cloud Framework for Financial Services

 Family Control
For help using the ground truth editor, see [Annotating documents](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-user-guide).







 Experimental services and features: What does experimental mean? 

IBM releases experimental services and features for you to try out. These services might be unstable, change frequently in ways that are not compatible with earlier versions, and might be discontinued with short notice. These services and features are not recommended for use in production environments.

For more information about experimental services, see the [IBM Cloud documentation ![External link icon](https://cloud.ibm.com/docs-content/v1/content/icons/launch-glyph.svg)](https://cloud.ibm.com/docs/get-support/servicessupport.htmls-services-exporcont). For the full details of experimental services, see the latest version of the [IBM Cloud Service Description ![External link icon](https://cloud.ibm.com/docs-content/v1/content/icons/launch-glyph.svg)](https://www.ibm.com/software/sla/sladb.nsf/sla/bm?OpenDocument).





 Storage space issues 



 Symptoms 

You might see a message about having exceeded the allowed storage space when you attempt to perform one of these tasks:



* Upload documents or dictionaries
* Deploy a model or version a model
* Run a pre-annotator on documents







 Causes 

The storage limit has been met or would be exceeded if the action were to proceed.





 Resolving the problem 

The largest consumers of storage space are machine learning and rule-based models. To free up space, you can take the following actions:



* Delete snapshot versions of any models that you do not expect to need to revert to.
* Delete any models that you do not need.
* If your models are too important to delete, consider increasing the amount of storage in your deployment.









 Contacting IBM Support
That's the end of this part. The next part is called The Document ID.







 The _id video 

Learn how _ids work in IBM Cloudant, how they are different from relational databases, and how you can define your own _id.



* The _id video script

Welcome to the Introduction to IBM Cloudant course, an 17-part video series that gives you an overview of the IBM Cloudant database-as-a-service.

This video is part 3 - The Document _id.

In the previous section, we saw how data is stored in IBM Cloudant documents with flexibility on how your application stores JSON objects in IBM Cloudant databases. However, a few hard and fast rules exist.

One rule is that every document must contain a unique identifier that is called _id, which is a string. Two documents in the same database can have the same _id field. In other databases, you specify which column is the unique identifier, but in IBM Cloudant, it's always _id and can't be changed.

Also, unlike relational databases, IBM Cloudant does not have "auto-incrementing IDs" that is, an ID field that starts at 1 and increments for each document added.

IBM Cloudant's _id field is one of the following strings:



* A 32-character string generated by IBM Cloudant. The ID is a meaningless sequence of numbers and letters that are guaranteed to be unique.
* A string that is defined by you (if you know something unique about your data).



The following examples show how to supply your own document _id:

Using it to store something that you know is unique that is, the email address of a user. Your registration mechanism can enforce a one-user-per-email address policy. Some users choose to encode the document type in the _id, for example, user:56, book:55. The last example shows with a 32-digit string (generated in your app) that is designed to sort in approximate date and time order. This method makes it easy to retrieve the latest documents from the database, without a secondary index.

IBM Cloudant takes your document _ids and stores them in an index (like the contents page of book).
Logging for Hyper Protect DBaaS for MongoDB 

Hyper Protect DBaaS for MongoDB is deprecated. As of 15 April 2023, you can't create new instances, and access to free instances will be removed. Existing paid instances are supported until 30 November 2023. Existing instances will be deleted on 1 March 2024.

Migration options: (a) [Hyper Protect Virtual Servers](https://www.ibm.com/cloud/hyper-protect-virtual-servers) with client managed databases and (b) [IBM Cloud Databases](https://www.ibm.com/cloud/free/databases) with optional additional services for security, such as [Hyper Protect Crypto Services](https://www.ibm.com/cloud/hyper-protect-crypto) and Data Security Broker ([Baffle](https://baffle.io/)). To migrate your data, you can use common dump and restore utilities (follow similar steps in the [migration documentation](https://cloud.ibm.com/docs/hyper-protect-dbaas-for-mongodb?topic=hyper-protect-dbaas-for-mongodb-migration_mongodb)).

When you use IBM Cloud® Hyper Protect DBaaS for MongoDB, you can forward your database logs and database audit logs to IBM Log Analysis.

To receive logs of your Hyper Protect DBaaS for MongoDB instance, you need to have a Log Analysis instance in the same region, as the following table shows.



Table 1. Log Analysis regions

 Hyper Protect DBaaS for MongoDB region Log Analysis region 

 Dallas (us-south) Dallas (us-south) 
 Washington DC (us-east) Washington DC (us-east) 
 Frankfurt (eu-de) Frankfurt (eu-de) 
 Sydney (au-syd) Sydney (au-syd) 



You can only specify one Log Analysis instance to receive platform logs from platform services in one region. This instance will be used to receive logs from all Hyper Protect DBaaS for MongoDB service instances provisioned in this region.



 Prerequisite 

To enable logging, all three nodes of the database cluster need to be running.
Supported SQL statements 

Following SQL statements are supported in IBM® watsonx.data through different connectors.



Table 1. SQL statements and supporting connectors

 Connector CREATE SCHEMA CREATE TABLE INSERT INTO SELECT SELECT (Complex) ALTER TABLE ALTER SCHEMA DELETE GRANT REVOKE SHOW GRANTS SHOW ROLES SHOW ROLE GRANTS UPDATE DROP TABLE CREATE ROLE CREATE VIEW DROP SCHEMA DROP VIEW 

 Iceberg ✓ ✓ ✓ ✓ ✓ ✓ X ✓ X X ✓ X X X ✓ X X ✓ X 
 MySQL X ✓ ✓ ✓ ✓ X X X X X X X X X ✓ X X X X 
 PostgreSQL X ✓ ✓ ✓ ✓ ✓ X X X X X X X X ✓ X X X X 
 MongoDB X ✓ ✓ ✓ ✓ ✓ X X X X ✓ X X X ✓ X X X X 
 Hive ✓ ✓ ✓ ✓ ✓ X X ✓ ✓ ✓ X ✓ ✓ X ✓ ✓ ✓ ✓ ✓ 
 Kafka X X X ✓ ✓ X X X X X ✓ X X X X X X X X 
 Memory ✓ ✓ ✓ ✓ ✓ X X X X X ✓ X X X ✓ X ✓ ✓ ✓ 
 TPCH X X X ✓ ✓ X X X X X ✓ X X X X X X X X 
 TPCDS X X X ✓ ✓ X X X X X ✓ X X X X X X X X 
 System X X X ✓ ✓ X X X X X ✓ X X X X X X X X 
 JMX X X X ✓ ✓ X X X X X ✓ X X X X X X X X 
 Db2 ✓ ✓ ✓ ✓ -- ✓ X X X X ✓ X X X ✓ X ✓ ✓ X 
 Netezza ✓ ✓ ✓ ✓ -- ✓ X X X X ✓ X X X ✓ X ✓ ✓ X 
 Hudi X X X ✓ -- X X X X X ✓ X X X X X X X X 





 Limitations 



1. For CREATE TABLE, MySQL connector supports only CREATE TABLE AS.
2. For ALTER TABLE, MongoDB connector supports only table rename.
3. Db2 connector partially supports ALTER TABLE, CREATE VIEW, and DROP SCHEMA.
4. Netezza connector partially supports ALTER TABLE and CREATE VIEW.
5.