vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Getting started with IBM Cloud Data Engine 

In this IBM Cloud® Data Engine tutorial, we set you up to begin querying rectangular data in IBM Cloud® Object Storage and storing the results in Object Storage.



 Before you begin 

Before you can run SQL queries, you need to have one or more Cloud Object Storage buckets to hold the data to be analyzed and to hold the query results. Cloud Object Storage offers several plans, including a Lite plan at no cost. To create a Cloud Object Storage instance, do the following steps.



1. Go to the [IBM Cloud catalog > Storage > Object Storage](https://cloud.ibm.com/objectstorage/create) page.
2. Select one of the plans and create the service. If you want to find your Cloud Object Storage instance later, go to your [IBM Cloud resource list > Storage](https://cloud.ibm.com/dashboard/apps).



You can now manage and browse the buckets and data the instance contains. Click [here](https://cloud.ibm.com/docs/services/cloud-object-storage/getting-started.htmlgetting-started-console) for more information about how to use Cloud Object Storage.





 Create your Data Engine service instance 



1. Go to the [IBM Cloud catalog](https://cloud.ibm.com/catalog) and search for Data Engine.
2. Click Data Engine to open the catalog details page.
3. Select the Lite plan and click Create to create an instance of the service.
4. Click Launch Data Engine UI on the Dashboard page to open the Data Engine Console.

When you open the Console for the first time, the Data Engine service automatically creates a bucket for you in your Cloud Object Storage instance. It uses this bucket as the default target for your query results.







 Run one of the samples to see how to use the service 



1. Select a sample query. The query automatically gets loaded into the editor. The input data that is used by the sample query is available in a publicly accessible bucket.
2. The Target field is automatically completed with the unique resource identifier (URI) of your default bucket.
To summarize, IBM Cloudant Search indexes are defined with a supplied JavaScript function. They are built on Apache Lucene and are primarily used for free-text search matching, but the query language is useful for building flexible queries on a fixed set of indexed fields. It also has some powerful counting aggregations suitable for drill-down user interfaces.

IBM Cloudant Search also powers type=text IBM Cloudant Query indexes, so a subset of its capabilities is surfaced by using the _find API.

That's the end of this part. The next part is called Under the hood.







 Under the hood video 

Learn how the IBM Cloudant service is organized.



* Under the hood video script

Welcome to the Introduction to IBM Cloudant course, an 17-part video series that gives you an overview of the IBM Cloudant database-as-a-service.

This video is part 17 - Under the hood.

Let's look at how an IBM Cloudant service is organized: This overview applies to the IBM Cloudant services that map to CouchDB 2 and 3. CouchDB 4 is built on different technology.

IBM Cloudant is a distributed database with data that is stored around a cluster of storage nodes. Picture the IBM Cloudant service as ring of nodes, in this case twelve. Every node can deal with incoming API calls and every node has responsibility for storing some of the data: shards and associated secondary indexes of databases that exist in the cluster.

When data is written to IBM Cloudant, one of the nodes in the ring handles the request: Its job is to instruct three copies of the data to be stored in three storage nodes. Data is stored in triplicate in IBM Cloudant, so each shard of a database is stored multiple times, often across a region's availability zones.

When you make an API call to write data and get a response back, we write the data to at least two of the three storage nodes. Data is flushed to disk - it isn't cached in memory to be flushed data. We consider that technique too risky and prone to data loss.

When you create a database, a number of database shards are created (16 by default) which are spread around the cluster.
IBM Cloud overview docs 

The platform is built to support your needs whether it's working only in the public cloud or taking advantage of a multicloud deployment model. Leverage the docs, API & SDK reference, architecture references, and more to support your app development needs.

 Developer tools 

[API & SDK reference](https://cloud.ibm.com/docs?tab=api-docs&category=platform_services)[CLI reference](https://cloud.ibm.com/docs?tab=develop)[Terraform reference](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs)

 Recommended content 

[What's included in the platform? The platform consists of a network of interrelated components that work together to provide a consistent and dependable cloud experience to support your app development needs.](https://cloud.ibm.com/docs/overview?topic=overview-whatis-platformbuilt-into-platform)[Try out IBM Cloud, for free Get up and running quickly by setting up an account and exploring free products with this short tutorial.](https://cloud.ibm.com/docs/overview?topic=overview-tutorial-try-for-free)[Service and infrastructure by location Check out the regions and data centers that IBM Cloud products are available.](https://cloud.ibm.com/docs/overview?topic=overview-services_region)

 Video library 

[![carousel thumbnail](https://cloud.ibm.com/docs-content/v1/content/overview//images/welcome-thumbnail.png)<br><br>Learn how to navigate through the IBM Cloud console](https://cloud.ibm.com/docs/overview?topic=overview-uivideo-ui)

 Learn more 

[IBM Developer<br><br>Visit IBM Developer for technical articles, code patterns, tutorials, and more.<br><br>!
Reading a document 

The steps shown here demonstrate how to read a document:



1. Send a GET request to retrieve a document.
2. Run the following command: https://$ACCOUNT.cloudant.com/$DATABASE/$DOCUMENT_ID.



Recall that for a partitioned database the $DOCUMENT_ID is formed of a partition key part and a document key part.

If you don't know the _id for a particular document, you can [query the database](https://cloud.ibm.com/apidocs/cloudantpostalldocsqueries) for all documents.

Due to the distributed, eventually consistent nature of IBM Cloudant, reads might return stale data. In particular, data written recently, even by the same client, might not be returned from a read request immediately following the write request. To work around this behavior, a client can cache the state of data locally. Caching also helps to keep request counts down, increase application performance, and decrease load on the database cluster. This behavior also applies to other read requests such as to MapReduce and search indexes.

See an example of retrieving a document by using HTTP:

GET /$DATABASE/$DOCUMENT_ID HTTP/1.1

You can customize this section for the programming language that you want to use by selecting the language in the code examples.

See an example of retrieving a document:



* Curl
* Java
* Python
* Go
* Node



curl -H "Authorization: Bearer $API_BEARER_TOKEN" -X GET "$SERVICE_URL/products/small-appliances:1000042"

import com.ibm.cloud.cloudant.v1.Cloudant;
import com.ibm.cloud.cloudant.v1.model.Document;
import com.ibm.cloud.cloudant.v1.model.GetDocumentOptions;

Cloudant service = Cloudant.newInstance();

GetDocumentOptions documentOptions =
new GetDocumentOptions.Builder()
.db("products")
.docId("small-appliances:1000042")
HEAD /{db}/{docid} <br>[Cloudant](https://cloud.ibm.com/apidocs/cloudantheaddocument), [CouchDB](https://docs.couchdb.org/en/latest/api/document/common.htmldb-doc) Retrieve the HTTP headers that contain a minimal amount of information about the specified document. 
 GET /{db}/{docid} <br>[Cloudant](https://cloud.ibm.com/apidocs/cloudantgetdocument), [CouchDB](https://docs.couchdb.org/en/latest/api/document/common.htmlget--db-docid) Retrieve the document specified by the docid from the specified database. 
 PUT /{db}/{docid} <br>[Cloudant](https://cloud.ibm.com/apidocs/cloudantputdocument), [CouchDB](https://docs.couchdb.org/en/latest/api/document/common.htmlput--db-docid) Create a new named document or create a new revision of the existing document. 
 DELETE /{db}/{docid} <br>[Cloudant](https://cloud.ibm.com/apidocs/cloudantdeletedocument), [CouchDB](https://docs.couchdb.org/en/latest/api/document/common.htmldelete--db-docid) Delete the specified document. 
 COPY /{db}/{docid} <br>Cloudant, [CouchDB](https://docs.couchdb.org/en/latest/api/document/common.htmlcopy--db-docid) Copy an existing document to a new or existing document.
"id": "4eee973603bf77f30b1f880ed83df76a",
"key": "4eee973603bf77f30b1f880ed83df76a",
"value": {
"rev": "1-3b5e6b73e57745787ad5627fe8f268c1"
}
},
{
"id": "4eee973603bf77f30b1f880ed83f469a",
"key": "4eee973603bf77f30b1f880ed83f469a",
"value": {
"rev": "1-967a00dff5e02add41819138abb3284d"
}
}
...
Show more
If you supply include_docs=true, then another doc attribute is added to each "row" in the result set that includes the document body.<-- </section "id="section-what-is-the_all_docs_endpoint" "> --><-- <section "id="section-the-limit-startkey-endkey-parameters" "> --> The limit, startkey, and endkey parameters To access data from _all_docs in reasonably sized pages, you must supply the limit parameter to tell IBM Cloudant how many documents to return: get me 5 documents
GET $SERVICE_URL/$DATABASE/_all_docs?limit=5 HTTP/1.1
You can also limit the range of document _ids that you want by supplying one or more values to startkey or endkey.<-- <ul> --> * Curl * Java * Node * Python * Go<-- </ul> --> get me 5 documents from _id order00057 onwards
<br>IBM Cloud<br><br> Recognize terms that are mentioned in sentences that match a syntactic pattern that you teach Discovery to recognize. [Patterns (beta)](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain-pattern) Available as a beta feature for English-language collections in managed deployments only. The enrichment that is derived by defining patterns cannot be applied to Content Mining projects. You can export the resource and use it as an advanced rules model. 
 Recognizes entities that you identify as being significant by training an entity extractor machine learning model. [Entity extractor](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-entity-extractor) Supports starting from an imported Knowledge Studio corpus. 



Alternatively, you can apply built-in Watson NLP enrichments that find the following information in your collection:



* [Entities and keywords](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-nlu)
* [Sentiment](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-nlu-sentiment)



You can extract meaning from documents based on the document structure by defining a Smart Document Understanding (SDU) model. Use the Smart Document Understanding tool to identify new fields by which to target enrichments or to split large documents into more manageable chunks. For more information, see [Structural meaning with SDU](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-configuring-fields).

Dictionaries and classifiers that you add to one project can be used by other projects.

For more information about how to get the most from enrichments, read the [Enriching your documents can make search more effective](https://community.ibm.com/community/user/watsonai/blogs/bill-murdock1/2022/01/14/enriching-your-documents-can-make-search-more-effe) blog post.



 Choosing the right enrichment type 

The following diagram helps you to choose the right enrichment for your use case.
IBM Cloud® Databases for PostgreSQL docs 

PostgreSQL is a powerful, open source object-relational database that is highly customizable.

 Developer tools 

[API & SDK reference](https://cloud.ibm.com/apidocs/cloud-databases-api/cloud-databases-api-v5introduction)[CLI reference](https://cloud.ibm.com/docs/databases-cli-plugin)[Terraform reference](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/resources/database)

 Recommended content 

[Getting Started A brief tutorial to introduce using an IBM Cloud® Databases for PostgreSQL deployment.](https://cloud.ibm.com/docs/databases-for-postgresql?topic=databases-for-postgresql-getting-started)[Setting the Admin Password The IBM Cloud® Databases for PostgreSQL service is provisioned with a admin user. You must set the admin password before you use it to connect.](https://cloud.ibm.com/docs/databases-for-postgresql?topic=databases-for-postgresql-admin-password&interface=ui)[Getting Connection Strings To connect to IBM Cloud® Databases for PostgreSQL, you need some users and connection strings.](https://cloud.ibm.com/docs/databases-for-postgresql?topic=databases-for-postgresql-connection-strings&interface=ui)

 Learn more 

[IBM Developer<br><br>Visit IBM Developer for technical articles, code patterns, tutorials, and more.<br><br>![112-Developer-tools.svg](https://cloud.ibm.com/media/docs/images/homepage/112-Developer-tools.svg)](https://developer.ibm.com/depmodels/cloud/)[Architecture Center<br><br>Discover the architecture references available for this product.<br><br>!
(https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collections)[Adding custom fields with Smart Document Understanding Identify meaningful information based on the structure of your documents.](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-configuring-fields)[Adding domain-specific resources Teach Watson about the information that matters to your use case.](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain)

 Video library 

[![carousel thumbnail](https://cloud.ibm.com/docs-content/v1/content/discovery-data//images/thumb-intro.png)<br><br>Get started with Watson Discovery](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-aboutabout-video)[![carousel thumbnail](https://cloud.ibm.com/docs-content/v1/content/discovery-data//images/thumb-content-mining.png)<br><br>IBM Watson Discovery: Content Mining](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-contentminerappcmvideo)[![carousel thumbnail](https://cloud.ibm.com/docs-content/v1/content/discovery-data//images/thumb-extractor.png)<br><br>Defining custom entity types with Watson Discovery](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-entity-extractorentity-extractor-video)[![carousel thumbnail](https://cloud.ibm.com/docs-content/v1/content/discovery-data//images/thumb-sdu.png)<br><br>Creating a user-defined Smart Document Understanding model](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-configuring-fieldssdu-label)

 Learn more
For more information, see [Optical character recognition](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionsocr).

The tool cannot read documents with the following characteristics; remove them from your collection before you begin:



* Documents that appear to have text that overlays other text are considered double overlaid and cannot be annotated.
* Documents that contain multiple columns of text on a single page cannot be annotated.



When you build a custom Smart Document Understanding model, the conversion time for your collection can increase due to the resources that are required to apply the AI model to your documents.



 Start with representative documents 

Documents come in all shapes and sizes. Your collection might have a mix of different document structures. Smart Document Understanding works best when the documents in a single collection have similar style characteristics. For example, the documents use consistent font sizes and colors for titles and headers, and tables in the document have similar layouts. To create the best model for your collection, take this prerequisite step:



1. Review your documents to look for style and layout patterns, and then separate the documents into groups based on their style.

For example, if your data contains documents that follow four different formatting styles, break the documents into four separate collections, one for each style. Add documents with a uniform layout and style to each collection. A good target size per collection is 40 documents.
2. Use the SDU tool to annotate this representative set of documents and train Watson to recognize custom content in your data.
3. Apply the custom SDU model to the full collection. For more information, see [Reusing SDU models](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-configuring-fieldsimport).









 Creating the model 

To apply a user-trained Smart Document Understanding model to your collection, complete the following steps:



1. Open the Manage collections page from the navigation panel.
2. If your project has more than one collection, select the collection with documents that you want to annotate.
3. Open the Identify fields page.
4.
(https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-web-cloudconnector-web-cloud-max-starting-urls).

The number of web pages that are crawled is limited to 250,000, so the web crawler might not crawl all the specified websites.

The number of child URLs per URL that are crawled is limited to 10,000. If the number of child URLs within any crawled URL exceeds 10,000, the crawler cannot process any of the content in the child URLs.
9. If you want to limit the types of files to add to the collection, you can list the file extensions for file types to either include or exclude.

If the URLs for your website pages do not end in .html, use the exclude filter instead of the include filter. You must add at least one file extension to exclude.

For a list of supported file types, see [Supported file types](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionssupportedfiletypes).
10. If you want the web crawl to extract text from images on the site, expand More processing settings, and set Apply optical character recognition (OCR) to On.

When OCR is enabled and your documents contain images, processing takes longer. For more information, see [Optical character recognition](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionsocr).
11. Click Finish.



The collection is created quickly. It takes more time for the data to be processed as it is added to the collection.

If you want to check the progress, go to the Activity page. From the navigation pane, click Manage collections, and then click to open the collection.



 I need to crawl lots of sites. What's my limit? 

The service can support a total of 500 crawler connections per Discovery service instance. All of the data sources except Web crawl use one crawler connection each. For Web crawl, one connection is required for every 5 starting URLs.
If you have an account with IBM Cloud, your personal data is held by IBM Cloud. The IBM Data Processing Addendum (Addendum) applies to the processing of client's personal data by IBM on behalf of client in order to provide IBM standard services.
[IBM DPA](https://www.ibm.com/support/customer/zz/en/dpa.html)

Databases for EnterpriseDB processes limited client Personal Information (PI) in the course of running the service and optimizing the user experience.

Databases for EnterpriseDB provides a [Data Sheet Addendum (DSA)](https://www.ibm.com/software/reports/compatibility/clarity-reports/report/html/softwareReqsForProduct?deliverableId=CD09D2E06DC811E8A0B560E89C071ECC) with its policies as a Data Processor regarding content and data protection.





 HIPAA 

Databases for EnterpriseDB meets the required IBM controls that are commensurate with the Health Insurance Portability and Accountability Act of 1996 (HIPAA) Security and Privacy Rule requirements. These requirements include the appropriate administrative, physical, and technical safeguards required of Business Associates in 45 CFR Part 160 and Subparts A and C of Part 164. HIPAA must be requested at the time of provisioning and requires a representative to sign a Business Associate Addendum (BAA) agreement with IBM.





 PCI DSS 

Databases for EnterpriseDB are compliant with the Payment Card Industry Data Security Standard (PCI DSS). IBM Cloud completes annual PCI DSS assessments by using an approved Qualified Security Assessor (QSA), and the resulting Attestations of Compliance (AOCs) and Service Responsibility Matrix (SRM) guides are available upon customer request. Auditors reviewed Databases for EnterpriseDB for compliance under PCI DSS version 3.2.1 at Service Provider Level 1.

Customers are responsible for the storing, processing, and transmission of their cardholder data, and can create cardholder data environments (CDEs) that can store, transmit, or process cardholder data by using Databases for EnterpriseDB. Customers can request and use the IBM Cloud AOCs and SRM guides when they seek their own PCI DSS certifications.
[Checkmark](https://cloud.ibm.com/docs-content/v1/content/6f3c07c34058f637b1c86044ca4f9b24214330a7/activity-tracker//images/checkmark-icon.svg) [Location-based events](https://cloud.ibm.com/docs/databases-for-etcd?topic=cloud-databases-activity-tracker-integration) 
 [IBM Cloud Databases for Elasticsearch](https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=databases-for-elasticsearch-getting-started) databases-for-elasticsearch-group ![Checkmark](https://cloud.ibm.com/docs-content/v1/content/6f3c07c34058f637b1c86044ca4f9b24214330a7/activity-tracker//images/checkmark-icon.svg) [Location-based events](https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=cloud-databases-activity-tracker-integration) 
 [IBM Cloud® Messages for RabbitMQ](https://cloud.ibm.com/docs/messages-for-rabbitmq?topic=messages-for-rabbitmq-getting-started) messages-for-rabbitmq-group ![Checkmark](https://cloud.ibm.com/docs-content/v1/content/6f3c07c34058f637b1c86044ca4f9b24214330a7/activity-tracker//images/checkmark-icon.svg) [Location-based events](https://cloud.ibm.com/docs/messages-for-rabbitmq?topic=cloud-databases-activity-tracker) 
 [IBM Cloud Databases for MongoDB](https://cloud.ibm.com/docs/databases-for-mongodb) databases-for-mongodb-group !
Example document in IBM Cloudant dashboard

{
"_id": "color",
"_rev": "1-f413f4b74a724e391fa5dd2e9c8e9d3f",
"name": "blue"
}







 Updating a document 

You can use the /update-document action to update a document in your IBM Cloudant database.

Before you begin

Create a /whisk.system/cloudant[package binding](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-pkg_cloudantcloudant_db) that is configured for your IBM Cloudant account.

The following example updates the document that was created in the [Writing a document to an IBM Cloudant database](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-pkg_cloudantcloudant_write) section.

You can update a document in your database by replacing <test> with your database name and replacing the --param doc flag with the id and contents of the document in your database that you want to update.



1. You can update a document in the test database by running the following command. This example adds the shade value to the color document.

ibmcloud fn action invoke /_/myCloudant/update-document --blocking --result --param dbname test --param doc "{"_id":"color","name":"blue","shade":"indigo"}"

Example output

{
"id": "color",
"ok": true,
"rev": "2-8b904347bfe52e0f388ef6f39d6ba84f"
}
2. To see the update, fetch the document again.

ibmcloud fn action invoke /_/myCloudant/read --blocking --result --param dbname test --param id color

Example document

{