vania-janet/multiturn-rag-retrieval-data huggingface dataset:
[Yes](https://cloud.ibm.com/docs-content/v1/content/ca9232b3702768f24bb897f48f10778fcda28718/watson-assistant/images/checkmark-icon.svg) 
 video ![Yes](https://cloud.ibm.com/docs-content/v1/content/ca9232b3702768f24bb897f48f10778fcda28718/watson-assistant/images/checkmark-icon.svg) ![Yes](https://cloud.ibm.com/docs-content/v1/content/ca9232b3702768f24bb897f48f10778fcda28718/watson-assistant/images/checkmark-icon.svg) ![Yes](https://cloud.ibm.com/docs-content/v1/content/ca9232b3702768f24bb897f48f10778fcda28718/watson-assistant/images/checkmark-icon.svg) ![Yes](https://cloud.ibm.com/docs-content/v1/content/ca9232b3702768f24bb897f48f10778fcda28718/watson-assistant/images/checkmark-icon.svg) ![Yes](https://cloud.ibm.com/docs-content/v1/content/ca9232b3702768f24bb897f48f10778fcda28718/watson-assistant/images/checkmark-icon.svg)
productsDoc.SetProperty("brand", "Salter")
productsDoc.SetProperty("name", "Digital Kitchen Scales")
productsDoc.SetProperty("description", "Slim Colourful Design Electronic Cooking Appliance for Home / Kitchen, Weigh up to 5kg + Aquatronic for Liquids ml + fl. oz. 15Yr Guarantee - Green")
productsDoc.SetProperty("price", 14.99)
productsDoc.SetProperty("image", "assets/img/0gmsnghhew.jpg")

postDocumentOptions := service.NewPostDocumentOptions(
"products",
)
postDocumentOptions.SetDocument(&productsDoc)

documentResult, response, err := service.PostDocument(postDocumentOptions)
if err != nil {
panic(err)
}

b, _ := json.MarshalIndent(documentResult, "", " ")
fmt.Println(string(b))

const { CloudantV1 } = require('@ibm-cloud/cloudant');

const service = CloudantV1.newInstance({});

const productsDoc = {
_id: 'small-appliances:1000042',
_rev: '1-967a00dff5e02add41819138abb3284d'
type: 'product',
productid: '1000042',
brand: 'Salter',
name: 'Digital Kitchen Scales',
description: 'Slim Colourful Design Electronic Cooking Appliance for Home / Kitchen, Weigh up to 5kg + Aquatronic for Liquids ml + fl. oz. 15Yr Guarantee - Green',
price: 14.99,
image: 'assets/img/0gmsnghhew.jpg'
};

service.postDocument({
The specified image must be in .jpg, .gif, or .png format. Y 
 title string The title to show before the image. N 
 description string The text of the description that accompanies the image. N 
 alt_text string Descriptive text that can be used for screen readers or other situations where the image cannot be seen. N 







 Example 

This example displays an image with a title and descriptive text.

{
"generic":[
{
"response_type": "image",
"source": "https://example.com/image.jpg",
"title": "Example image",
"description": "An example image returned as part of a multimedia response."
}
]
}







 option 

Presents a set of options (such as buttons or a drop-down list) that users can choose from. The selected value is then sent to the assistant as user input. An options response is automatically defined when you choose the Options customer response type for a step (for more information, see [Collecting information from your customers](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-collect-info)).



 Integration channel support 



 Web chat Phone SMS Slack Facebook WhatsApp 

 ![Yes](https://cloud.ibm.com/docs-content/v1/content/ca9232b3702768f24bb897f48f10778fcda28718/watson-assistant/images/checkmark-icon.svg) ![Yes](https://cloud.ibm.com/docs-content/v1/content/ca9232b3702768f24bb897f48f10778fcda28718/watson-assistant/images/checkmark-icon.svg) !
"name" : "README1.md",
"blob" : "IyBF ... KWBgCg==",
"md5" : "270f60e62d3d37add3702ced7f6969a1 README.md"
}
}





 Syncing files to disk 

After the files are uploaded to the index, they can be synced to the disk. Call the /elasticsearch/file_syncs endpoint from the Cloud Databases API.

curl -X POST https://api.{region}.databases.cloud.ibm.com/v4/ibm/deployments/{id}/elasticsearch/file_syncs -H 'authorization: Bearer <token>'

The region is the region that your deployment is in, and the id (CRN) part of the URL needs to be url-encoded. More information is in the [API Reference](https://cloud.ibm.com/apidocs/cloud-databases-api).

The call starts and returns a [task](https://cloud.ibm.com/apidocs/cloud-databases-apiget-currently-running-tasks-on-a-deployment) so you can monitor its progress. After the returned task finishes, the contents in the index are present on all the nodes in your cluster.

Any number of files can be uploaded and synced. The contents of the files are not validated. Ensure that they can be processed by Elasticsearch.





 Using the files 

Elasticsearch features that use files on the file system do so by accepting the path to the file when defining the index. An uploaded file example.txt is at /data/ibm_file_sync/current/example.txt. This list contains examples and is not exhaustive.



* [Keep Words Token Filter](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-keep-words-tokenfilter.html)
[External link icon](https://cloud.ibm.com/docs-content/v1/content/ded4adc3ea0bd2a81b113c579f2b1183926da211/icons/launch-glyph.svg)](https://watson-developer-cloud.github.io/doc-tutorial-downloads/discovery/test-doc1.html), [test-doc2.html ![External link icon](https://cloud.ibm.com/docs-content/v1/content/ded4adc3ea0bd2a81b113c579f2b1183926da211/icons/launch-glyph.svg)](https://watson-developer-cloud.github.io/doc-tutorial-downloads/discovery/test-doc2.html), [test-doc3.html ![External link icon](https://cloud.ibm.com/docs-content/v1/content/ded4adc3ea0bd2a81b113c579f2b1183926da211/icons/launch-glyph.svg)](https://watson-developer-cloud.github.io/doc-tutorial-downloads/discovery/test-doc3.html), [test-doc4.html ![External link icon](https://cloud.ibm.com/docs-content/v1/content/ded4adc3ea0bd2a81b113c579f2b1183926da211/icons/launch-glyph.svg)](https://watson-developer-cloud.github.io/doc-tutorial-downloads/discovery/test-doc4.html)

In some browsers, the link open in a new window instead of saving locally. If this occurs, select Save As in your browser's File menu to save a copy of the file.





 Step 1: Quick tour of the Discovery data schema
[External link icon](https://cloud.ibm.com/docs-content/v1/content/ded4adc3ea0bd2a81b113c579f2b1183926da211/icons/launch-glyph.svg)](https://watson-developer-cloud.github.io/doc-tutorial-downloads/discovery/test-doc1.html), [test-doc2.html ![External link icon](https://cloud.ibm.com/docs-content/v1/content/ded4adc3ea0bd2a81b113c579f2b1183926da211/icons/launch-glyph.svg)](https://watson-developer-cloud.github.io/doc-tutorial-downloads/discovery/test-doc2.html), [test-doc3.html ![External link icon](https://cloud.ibm.com/docs-content/v1/content/ded4adc3ea0bd2a81b113c579f2b1183926da211/icons/launch-glyph.svg)](https://watson-developer-cloud.github.io/doc-tutorial-downloads/discovery/test-doc3.html), and [test-doc4.html ![External link icon](https://cloud.ibm.com/docs-content/v1/content/ded4adc3ea0bd2a81b113c579f2b1183926da211/icons/launch-glyph.svg)](https://watson-developer-cloud.github.io/doc-tutorial-downloads/discovery/test-doc4.html).

In some browsers, the preceding links open in a new window. To save locally, select Save As in your browser's File menu to save a copy of the file.





 Step 4: Upload the documents 



1. Now, add the example documents to your collection.
Some connectors support document-level security from a IBM Cloud Pak for Data deployment ![checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/icons/checkmark-icon.svg) 
 Programmatically configure external data source crawls ![checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/icons/checkmark-icon.svg) 
 Configure the normalization processes of document segmentation and HTML file inclusion or exclusion rules during ingestion ![checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/icons/checkmark-icon.svg) 
 Configure the JSON normalization process during ingestion and after enrichment ![checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/icons/checkmark-icon.svg) ![checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/icons/checkmark-icon.svg) 
 Configure dictionary tokenization ![checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/icons/checkmark-icon.svg)
POST /_users/_bulk_docs cloudantnosqldb.users.write 
 POST /_users/ cloudantnosqldb.users.write 
 GET/HEAD /_uuids cloudantnosqldb.cluster-uuids.execute 
 POST /$DATABASE/ cloudantnosqldb.data-document.write or cloudantnosqldb.design-document.write or cloudantnosqldb.local-document.write 
 POST /$DATABASE/_bulk_docs cloudantnosqldb.data-document.write either or both cloudantnosqldb.design-document.write either or both cloudantnosqldb.local-document.write 
 PUT /$DATABASE/$DOCUMENT_ID cloudantnosqldb.data-document.write 
 DELETE /$DATABASE/$DOCUMENT_ID cloudantnosqldb.data-document.write 
 PUT /$DATABASE/$DOCUMENT_ID/$ATTACHMENT cloudantnosqldb.data-document.write 
 DELETE /$DATABASE/$DOCUMENT_ID/$ATTACHMENT cloudantnosqldb.data-document.write 
 PUT/DELETE /$DATABASE/_local/$DOCUMENT_ID cloudantnosqldb.local-document.write 
 COPY (Depends on write document type.) /$DATABASE/_local/$DOCUMENT_ID cloudantnosqldb.any-document.read + cloudantnosqldb.design-document.write either or both cloudantnosqldb.local-document.write either or both cloudantnosqldb.data-document.write 
 GET/HEAD /_iam_session cloudantnosqldb.iam-session.read 
 POST /_iam_session cloudantnosqldb.iam-session.write 
 DELETE /_iam_session cloudantnosqldb.iam-session.delete 
 GET/HEAD /_session cloudantnosqldb.session.read 
 POST /_session cloudantnosqldb.session.write 
 DELETE /_session cloudantnosqldb.session.delete
b, _ := json.MarshalIndent(resource, "", " ")
fmt.Println(string(b))

const params = {
caseNumber: caseNumber,
crn: resourceCrn,
note: 'This resource is the service that is having the problem.',
};

caseManagementService.addResource(params)
.then(res => {
console.log(JSON.stringify(res.result, null, 2));
})
.catch(err => {
console.warn(err)
});







 Supported file types for cases 

When you create a case, you can attach a file. The following file types are supported:

7z, ace, ams, arm, asp, bash, history, bkp, big, bmp, bz2, ca, ca-bundle, ca-crt, cabundle, cap, cer, cert, cfg, cnf, crt, csr, csv, dat, dbs, debug, dib, dmesg, dmp, doc, docx, dotx, dump, email, eml, emz, env, eps, error, evt, evtx, fragment, gif, gz, gz_aa, gz_ab, gz_ac, har, hosts, htaccess, html, iaf, ics, id, img, info, jpb, jpe, jpeg, jpg, key, lic, log, logsm lon02, lst, lzh, mai, md5, mib, mjpg, msg, mso, odp, ods, odt, oft, openssh, out, ovf, ovpn, p7b, p7s, pages, pcap, pcf, pcx, pdb, pem, pfx, pic, pix,
(https://cloud.ibm.com/docs-content/v1/content/icons/checkmark-icon.svg) ![Checkmark icon.](https://cloud.ibm.com/docs-content/v1/content/icons/checkmark-icon.svg) ![Checkmark icon.](https://cloud.ibm.com/docs-content/v1/content/icons/checkmark-icon.svg) ![Checkmark icon.](https://cloud.ibm.com/docs-content/v1/content/icons/checkmark-icon.svg) ![Checkmark icon.](https://cloud.ibm.com/docs-content/v1/content/icons/checkmark-icon.svg) 
 [Data-At-Rest Encryption](https://cloud.ibm.com/docs/FileStorage?topic=FileStorage-mng-data) ![Checkmark icon.](https://cloud.ibm.com/docs-content/v1/content/icons/checkmark-icon.svg) ![Checkmark icon.](https://cloud.ibm.com/docs-content/v1/content/icons/checkmark-icon.svg) ![Checkmark icon.](https://cloud.ibm.com/docs-content/v1/content/icons/checkmark-icon.svg) ![Checkmark icon.](https://cloud.ibm.com/docs-content/v1/content/icons/checkmark-icon.svg) ![Checkmark icon.](https://cloud.ibm.com/docs-content/v1/content/icons/checkmark-icon.svg) ![Checkmark icon.](https://cloud.ibm.com/docs-content/v1/content/icons/checkmark-icon.svg) ![Checkmark icon.]
[checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/icons/checkmark-icon.svg) 
 Process text from scanned documents or other images ![checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/icons/checkmark-icon.svg) ![checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/icons/checkmark-icon.svg) 
 Extract meaning from tables ![checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/icons/checkmark-icon.svg) 
 Get insights from contracts (IBM Cloud Pak for Data, Enterprise, and Premium plans only) ![checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/icons/checkmark-icon.svg) 
 Apply the Part of Speech enrichment to your data ![checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/icons/checkmark-icon.svg) 
 Use the Entity Extraction, Document and Phrase Sentiment Analysis, and Keyword Extraction enrichments !
[Available](https://cloud.ibm.com/docs-content/v1/content/icons/checkmark-icon.svg) ![Available](https://cloud.ibm.com/docs-content/v1/content/icons/checkmark-icon.svg) 
 Metadata File Entry: Create/Modify ![Available](https://cloud.ibm.com/docs-content/v1/content/icons/checkmark-icon.svg) ![Available](https://cloud.ibm.com/docs-content/v1/content/icons/checkmark-icon.svg) 
 Network Pool: View ![Available](https://cloud.ibm.com/docs-content/v1/content/icons/checkmark-icon.svg) ![Available](https://cloud.ibm.com/docs-content/v1/content/icons/checkmark-icon.svg) ![Available](https://cloud.ibm.com/docs-content/v1/content/icons/checkmark-icon.svg) ![Available](https://cloud.ibm.com/docs-content/v1/content/icons/checkmark-icon.svg) ![Available](https://cloud.ibm.com/docs-content/v1/content/icons/checkmark-icon.svg) 
 Object Extensions: Manage ![Available](https://cloud.ibm.com/docs-content/v1/content/icons/checkmark-icon.svg) ![Available](https://cloud.ibm.com/docs-content/v1/content/icons/checkmark-icon.svg) 
 Object Extensions: View ![Available](https://cloud.ibm.com/docs-content/v1/content/icons/checkmark-icon.svg) ![Available](https://cloud.ibm.com/docs-content/v1/content/icons/checkmark-icon.svg) !
[Available](https://cloud.ibm.com/docs-content/v1/content/11f96f96f8f4b60cf6f3fa4d904856a2c4528bc2/icons/checkmark-icon.svg) ![Available](https://cloud.ibm.com/docs-content/v1/content/11f96f96f8f4b60cf6f3fa4d904856a2c4528bc2/icons/checkmark-icon.svg) ![Available](https://cloud.ibm.com/docs-content/v1/content/11f96f96f8f4b60cf6f3fa4d904856a2c4528bc2/icons/checkmark-icon.svg) 
 Object Extensions: Manage ![Available](https://cloud.ibm.com/docs-content/v1/content/11f96f96f8f4b60cf6f3fa4d904856a2c4528bc2/icons/checkmark-icon.svg) ![Available](https://cloud.ibm.com/docs-content/v1/content/11f96f96f8f4b60cf6f3fa4d904856a2c4528bc2/icons/checkmark-icon.svg) 
 Object Extensions: View ![Available](https://cloud.ibm.com/docs-content/v1/content/11f96f96f8f4b60cf6f3fa4d904856a2c4528bc2/icons/checkmark-icon.svg) ![Available](https://cloud.ibm.com/docs-content/v1/content/11f96f96f8f4b60cf6f3fa4d904856a2c4528bc2/icons/checkmark-icon.svg) !
If you store dates in other formats, you can add the format to the list of supported formats.

To add more date formats, complete the following steps:



1. From the Manage fields page for the collection, add a format as a new line in the Date formats field.

Specify a date format that is supported by the Java [SimpleDateFormat](https://docs.oracle.com/javase/8/docs/api/java/text/SimpleDateFormat.html) class.

For example, if your records store only year values for dates, add yyyy to the supported date formats list. You can then set the data type for the field that contains a year value to Date, and reprocess your collection. As a result, an occurrence of 2019 in the date field is stored as 2019-01-01T05:00:00Z in the index.

When you add a date format, you must specify an associated time zone for the date.
2. Specify a time zone.
3. Optionally, select a date locale.

The locale that you choose is used to parse a string value that represents the date for the date-type data set fields. For example, by using the EEE, MM dd, yyyy format, the English (United States) locale can parse the string value of "Wednesday, 07 01, 2020", and the Japanese (Japan) locale can parse the same string value of "水曜日, 07 01, 2020".
4. If you already imported documents with dates in formats that were not recognized, reprocess the documents.



Discovery cannot store a date that is mentioned within a text field as a Date field in the index. However, you can use an enrichment such as the Entities enrichment to identify dates that are mentioned in text.







 How file types are handled 

When you upload a document, data in the file is indexed. Different files types are handled differently by Discovery.



* [CSV files](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-index-overviewfile-type-csv)
[list-expansions](https://cloud.ibm.com/apidocs/discoverylist-expansions) Action username, password, iam_access_token, iam_apikey, iam_url, headers, headers[X-Watson-Learning-Opt-Out], url, environment_id, collection_id Get the expansion list. 
 [add-document](https://cloud.ibm.com/apidocs/discoveryadd-document) Action username, password, iam_access_token, iam_apikey, iam_url, headers, headers[X-Watson-Learning-Opt-Out], url, environment_id, collection_id, file, metadata, file_content_type Add a document. 
 [delete-document](https://cloud.ibm.com/apidocs/discoverydelete-document) Action username, password, iam_access_token, iam_apikey, iam_url, headers, headers[X-Watson-Learning-Opt-Out], url, environment_id, collection_id, document_id Delete a document. 
 [get-document-status](https://cloud.ibm.com/apidocs/discoveryget-document-status) Action username, password, iam_access_token, iam_apikey, iam_url, headers, headers[X-Watson-Learning-Opt-Out], url, environment_id, collection_id, document_id Get document details. 
 [update-document](https://cloud.ibm.com/apidocs/discoveryupdate-document) Action username, password, iam_access_token, iam_apikey, iam_url, headers, headers[X-Watson-Learning-Opt-Out], url, environment_id, collection_id, document_id, file, metadata, file_content_type Update a document.
When OCR is enabled and your documents contain images, processing takes longer. For more information, see [Optical character recognition](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionsocr).
11. Click Finish.



The collection is created quickly. It takes more time for the data to be processed as it is added to the collection.

If you want to check the progress, go to the Activity page. From the navigation pane, click Manage collections, and then click to open the collection.
When OCR is enabled and your documents contain images, processing takes longer. For more information, see [Optical character recognition](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionsocr).
12. Click Finish.



The collection is created quickly. It takes more time for the data to be processed as it is added to the collection.

If you want to check the progress, go to the Activity page. From the navigation pane, click Manage collections, and then click to open the collection.

Currently, not all documents are refreshed during scheduled recrawls. For more information, see the [release note](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-release-notesdiscovery-6december2022).
Now go ahead and go to one of your buckets by selecting it from the list. Click Add Objects. New objects overwrite existing objects with the same names within the same bucket. When you use the console to upload objects the object name always matches the file name. There doesn't need to be any relationship between the file name and the object key if you're using the API to write data. Go ahead and add a handful of files to this bucket.

Objects are limited to 200 MB when uploaded through the console unless you use the [Aspera high-speed transfer](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-upload) plug-in. Larger objects (up to 10 TB) can also be [split into parts and uploaded in parallel using the API](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-large-objects). Object keys can be up to 1024 characters in length, and it's best to avoid any characters that might be problematic in a web address. For example, ?, =, <, and other special characters might cause unwanted behavior if not URL-encoded.





 How do I invite a user to administer buckets and data? 

Bringing in another user and allow them to act as an administrator for the instance and any data stored in it is an important way to distribute responsibility for administering your IBM Cloud Object Storage instance.



1. To add the new user you first need to leave the current Object Storage interface and head for the IAM console. Go to the Manage menu and follow the link at Access (IAM) > Users. Click Invite users.

Zoom

![IAM invite users](https://cloud.ibm.com/docs-content/v1/content/63322f2f3a72050206eb05ba7f590fdcb741105c/cloud-object-storage/images/console_iam_invitebtn.png)

Figure 2: IAM invite users
2.
* For more information about size limits, which can differ per plan, see the following topics:



* [Collection limits](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionscollections-limits)
* [Document limits](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionscollections-doc-limits)









 Installing IBM Secure Gateway for on-premises data 

To connect to an on-premises data source, you first need to download, install, and configure IBM® Secure Gateway for IBM Cloud®.

After you install the client for one on-premises data source, you can reuse it for other data sources in the project.

The number of gateways that you can create is limited to 50.

For more information, see [About Secure Gateway](https://cloud.ibm.com/docs/SecureGateway?topic=SecureGateway-about-sg).

You can use the IBM Secure Gateway with the following connectors only:



* [Web crawl](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-web-cloudconnector-web-cloud-prereq-task)
* [Microsoft SharePoint On Prem](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-sharepoint-onprem-cloudconnector-sharepoint-onprem-cloud-prereq-task)



To install IBM® Secure Gateway for IBM Cloud®, complete the following steps:



1. From the data source configuration page, click Manage connection.
2. On the Download and install Secure Gateway client page, download the appropriate version of IBM® Secure Gateway for IBM Cloud®.
3. After you complete the download, click Download Secure Gateway and Continue.
4. When prompted, enter the Gateway ID and Token that are displayed.

For more information, see [Installing the client](https://cloud.ibm.com/docs/SecureGateway?topic=SecureGateway-client-install).
5.