vania-janet/multiturn-rag-retrieval-data huggingface dataset:
See an example of indexing a field:

function(doc) {
if (doc.name) {
emit("name", doc.name);
}
}





 An index for a one-to-many relationship 

If the object passed to emit has an _id field, a view query with include_docs set to true contains the document with the specific ID.

See an example of indexing a one-to-many relationship:

function(doc) {
if (doc.friends) {
for (friend in doc.friends) {
emit(doc._id, { "_id": friend });
}
}
}





 Complex keys 

Keys aren't limited to simple values. You can use arbitrary JSON values to influence sorting.

When the key is an array, view results can be grouped by a subsection of the key. For example, if keys have the form [year, month, day], then results can be reduced to a single value or by year, month, or day.

For more information, see [Using views](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-using-viewsusing-views).







 Reduce functions 

Design documents with options.partitioned set to true can't contain custom JavaScript reduce functions. Only built-in reduces are allowed.



 No reducer 

A view definition inside a design document is permitted to have no reduce attribute, indicating that no query-time aggregation is performed.

{
"views": {
"getVerifiedEmails": {
"map": "function(user) { if(user.email_verified === true) { emit(user.email); } }"
}
}
}

The previous map function generates a secondary index suitable for selection only. The index is always ordered by the key (the emit function's first parameter) - in this case user.email. This view is ideal for fetching documents by a known user email or ranges of users email addresses.





 Built-in reduce functions
The following JSON sample defines values for the $dessert string, $toppings_array array, $age number, and $full_name object context variables:

{
"context": {
"dessert": "cake",
"toppings_array": [
"onions",
"olives"
],
"age": 18,
"full_name": {
"first": "Jane",
"last": "Doe"
}
},
"output":{}
}

To define a context variable in JSON format, complete the following steps:



1. Click to open the dialog node to which you want to add the context variable.

Any existing context variable values that are defined for this node are displayed in a set of corresponding Variable and Value fields. If you do not want them to be displayed in the edit view of the node, you must close the context editor. You can close the editor from the same menu that is used to open the JSON editor; the following steps describe how to access the menu.
2. Click the Options icon ![Advanced response](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/kabob.png) that is associated with the response, and then click Open JSON editor.

![Shows how to access the JSON editor associated with a standard node response.](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/contextvar-json-response.png)

If the Multiple responses setting is On for the node, then you must first click the Edit response!
Get a list of the service accounts that are created by the ibmcloud-object-storage-plugin.

oc get ServiceAccount --all-namespaces -l app=ibmcloud-object-storage-plugin -o jsonpath='{range .items[]}{.metadata.namespace}{"t"}{.metadata.name}{"n"}{end}'
12. Delete the conflicting resources.
13. After you delete the conflicting resources, [retry the installation](https://cloud.ibm.com/docs/openshift?topic=openshift-storage_cos_install).
List fields across collections [GET /v1/environments/{environment_id}/fields](https://cloud.ibm.com/apidocs/discoverylistfields) [GET /v2/projects/{project_id}/fields](https://cloud.ibm.com/apidocs/discovery-datalistfields) 







 Configurations 

The v2 API does not have an endpoint that is dedicated to configurations. Instead, configuration settings for projects, collections, and queries are specified directly in the API for those objects. Not all of the configuration parameters that are available in v1 are available or applicable in v2.

In the [v1 configuration API](https://cloud.ibm.com/apidocs/discoverycreateconfiguration), the JSON object that is used to specify a configuration object contains several parameters that are either available in different formats from other v2 endpoints or are not available in v2. The following table describes how to find related parameters in v2.

You cannot customize the conversion of documents during the ingestion process in v2 as you can in v1.



Configuration setting details

 v1 configuration parameter v2 API 

 "conversions.html": { ... } Not available 
 "conversions.image_text_recognition": { ... } Not available from the API. However, you can enable optical character recognition (OCR) for a collection from the product user interface to extract text from images. OCR has other benefits, too. For example, if a page in a document can't be processed, OCR converts the page into an image and scans it to ensure that the document is uploaded successfully. 
 "conversions.json_normalizations": { ... } Moved to the [Collections API](https://cloud.ibm.com/apidocs/discovery-datalistcollections). 
 "conversions.pdf": { ... } Not available. If you used special parameters to extract text from images in PDFs, enable optical character recognition (OCR) from the product user interface for the collection that contains the PDFs instead. 
 "conversions.segment": { ...
On the Configure data screen, there are three tabs: Identify fields, Manage fields, and Enrich fields.



* Identify fields contains the SDU editor.
* Manage fields lists all indexed fields (all fields are indexed by default). Switch off any fields you do not want to index. For example, your PDFs might contain a running header or footer that does not contain useful information, so you can exclude those fields from the index. You can also split documents here, based on fields, see [Splitting documents](https://cloud.ibm.com/docs/discovery?topic=discovery-sdusplitting).
* Enrich fields is identical to the Enrich tab on the original screen. For more information about enrichments, see [Adding enrichments](https://cloud.ibm.com/docs/discovery?topic=discovery-configserviceadding-enrichments). The Upload sample documents option is not available with SDU collections.



If you did not upload any documents earlier, return to the Overview screen by clicking the name of your collection in the upper left, or click the ![Manage Data](https://cloud.ibm.com/docs-content/v1/content/ded4adc3ea0bd2a81b113c579f2b1183926da211/discovery//images/icon_yourData.png) icon and choose your collection. Drag and drop documents into your collection, or click browse from computer. After you initially upload your documents, the Upload documents button displays on the upper right.
4. Open the Identify fields tab. A subset of documents will be available for annotation purposes, between 20 - 50 documents will appear in the dropdown list. The number will depend on several factors, including the number of documents in your collection in the supported file formats.



You can use the toolbar at the top to complete the following actions:



* Choose a document to annotate
* Navigate the document displayed
* Adjust the page view (single page view, zoom in, zoom out), clear changes, and export/import models.
PNG, TIFF, and JPEG images embedded in PDF, Word, PowerPoint, and Excel files are also scanned and the text, if any, is extracted.





 Using the Smart Document Understanding editor 

The SDU editor is only available for collections that contain supported document types and do not have the Element Classification enrichment applied. If you do not want to use the SDU editor, you can set up your configuration using the API, see the [API reference](https://cloud.ibm.com/apidocs/discovery/).

The SDU editor functions are only available in the Discovery tooling, they are not available in the API.

Navigating the Smart Document Understanding editor:

If you did not yet create a Discovery instance and environment, see [Getting started](https://cloud.ibm.com/docs/discovery?topic=discovery-getting-started) for instructions.



1. On the Manage Data screen, click Upload your own data, and create a new private collection in Discovery.
2. Drag and drop documents into your collection, or click select documents to upload documents. After the upload is complete, this information displays:



* The fields identified from your documents.
* Enrichments applied to your documents. The Entity Extraction, Sentiment Analysis, Category Classification, and Concept Tagging enrichments are automatically applied to the text field by Discovery (unless you are importing documents using a connector). You can add (or remove) additional enrichments to the text (and other) fields.
* Pre-built queries you can run immediately.



3. Click Configure data on the upper right. On the Configure data screen, there are three tabs: Identify fields, Manage fields, and Enrich fields.



* Identify fields contains the SDU editor.
* Manage fields lists all indexed fields (all fields are indexed by default). Switch off any fields you do not want to index. For example, your PDFs might contain a running header or footer that does not contain useful information, so you can exclude those fields from the index.
Now that subtitles are indexed properly in Discovery, use them as the basis for splitting the PDF files into many smaller documents.



1. Return to the web browser tab where your Discovery project is displayed.
2. Open the Manage fields tab for the current collection.
3. In the Split document on each occurrence of field, choose subtitle, and then click Apply changes and reprocess.

Zoom

![Shows the split by field option in Discovery](https://cloud.ibm.com/docs-content/v1/content/ca9232b3702768f24bb897f48f10778fcda28718/watson-assistant/images/tut-neural-split-doc.png)

Figure 10. Split a document
4. From the navigation panel, click Manage collections, and then open the other collection.
5. Go to the Manage fields page, and then choose subtitle in the Split document on each occurrence of field.
6. Click Apply changes and reprocess.



The collections start to be reprocessed. After reindexing is finished, instead of containing one document each, the collections will contain several hundred documents each.

Zoom

![Shows the collections with many documents](https://cloud.ibm.com/docs-content/v1/content/ca9232b3702768f24bb897f48f10778fcda28718/watson-assistant/images/tut-neural-both-collections.png)

Figure 11. The collections with more documents

While the index is being rebuilt, let's get our assistant ready.





 Step 8: Add an extension to your assistant 

Connect your assistant to your NeuralSeek service instance.



1. Reopen the NeuralSeek service from IBM Cloud. You can find the instance in the AI and Machine Learning section of your [resource list](https://cloud.ibm.com/resources).
2. Click the Integrate tab and follow the instructions to set up the NeuralSeek custom extension for your assistant.
Review the number of denied requests from a given second that are shown by the number of 429: too many requests responses. Requests are denied when they exceed the provisioned throughput capacity set for the instance. The graph shows the denied requests that are broken down by reads, writes, and global queries.

Zoom

![The Denied Requests tab shows the denied requests that are broken down by read, writes, and global queries.](https://cloud.ibm.com/docs-content/v1/content/522c6f62358b063f921283400a601c3f9bc66f08/Cloudant/tutorials/images/denied-requests-tab.png)

Figure 8. Denied Requests
3. Click Storage.

Periodically review your storage, so you are prepared if your plan's provisioning needs to be changed.

Zoom

![The Storage tab includes your JSON documents, indexes, and attachements.](https://cloud.ibm.com/docs-content/v1/content/522c6f62358b063f921283400a601c3f9bc66f08/Cloudant/tutorials/images/storage-tab.png)

Figure 9. Storage



For more information, see [Plans and provisioning](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-ibm-cloud-public).
Only the account owner, or a user with the Manage Users classic infrastructure permission, can adjust the permissions.



For more information about permissions, see [Classic infrastructure permissions](https://cloud.ibm.com/docs/account?topic=account-infrapermissioninfrapermission) and [Managing device access](https://cloud.ibm.com/docs/virtual-servers?topic=virtual-servers-managing-device-access).

You might need to check your Windows firewall settings to ensure that you can import an image from IBM Cloud Object Storage.
2. Access the Image Templates page by selecting Devices > Manage > Images.
3. Click the Import Custom Image link to open the Import tool.
4. Complete the required fields (see Table 1).
5. When the import is complete from IBM Cloud Object Storage, the image appears on the Image Templates page.





Table 1. Values for importing a custom image from IBM Cloud Object Storage

 Field Value 

 Image Name Specify a descriptive name for your new image. This image is the image that you use to provision virtual server instances. 
 IBM Cloud API Key Specify the API key that gives access to your IBM Cloud Object Storage service instance. This key is the object storage service Service Credentials API key. When you import an encrypted image, the API Key must also have access to your key management service instance. The API key is only available to be copied or downloaded at the time of creation. If the API key is lost, you must create a new API key. For more information, see [Working with API keys](https://cloud.ibm.com/docs/account?topic=account-manapikeymanapikey). 
 Notes Add any notes that are related to the image that might be helpful to users. 
 COS service instance Select the IBM Cloud Object Storage service instance where the image that you want to import is stored. 
 Location Select the specific geographic region where your image is stored.
![Shows labeled subtitles in the SDU tool](https://cloud.ibm.com/docs-content/v1/content/ca9232b3702768f24bb897f48f10778fcda28718/watson-assistant/images/tut-neural-sdu-subtitle.png)

Figure 8. Applying the subtitle label
8. After every subtitle on the current page is labeled, click Submit page.

The next page of the PDF file is displayed.

Zoom

![Shows labeled subtitles in the SDU tool](https://cloud.ibm.com/docs-content/v1/content/ca9232b3702768f24bb897f48f10778fcda28718/watson-assistant/images/tut-neural-sdu-page2.png)

Figure 9. Next page is displayed for labeling
9. Repeat this process until the tool is able to label the headings correctly for you in a consistent way when new pages are loaded into the tool. At that point, click Apply changes and reprocess.

Congratulations! You have successfully trained a Smart Document Understanding (SDU) model that can recognize subtitles in your documents. Let's apply the same model to the other PDF file that you added to the project.
10. From the SDU editor toolbar, click the overflow menu icon ![Vertical overflow menu icon](https://cloud.ibm.com/docs-content/v1/content/ca9232b3702768f24bb897f48f10778fcda28718/watson-assistant/images/overflow-menu--vertical.svg) from the page header, and choose Export model.
11. Save the .sdumodel file to your system in a location where you can access it again shortly.
12. From the navigation panel, click Manage collections, and then open the Discovery docs part 2 collection.
13. Open the Identify fields tab.
14.
For more information, see [Storage tiers](https://cloud.ibm.com/docs/power-iaas?topic=power-iaas-about-virtual-serverstorage-tiers). 
 Region Select either us-east, us-south, br-sao, ca-tor, ca-mon, eu-de, or eu-gb, au-syd, jp-tok, jp-osa for the region. 
 Image file name Enter the file name of the image. The image file name must not contain spaces. Supported file formats are tar and ova. You can compress image files by using gzip. The supported file name extensions are .ova, .ova.gz, .tar, .tar.gz and .tgz. You must use the private endpoint domain. For example, Aix_7200-03-02-1846_cldrdy_112018.gz. 
 Bucket name Sub folders can be used and specified as bucketName/optional/folders. Optional folders are created automatically if they don’t exist. Optional folders can be added during an [export image](https://cloud.ibm.com/docs/power-iaas?topic=power-iaas-capturing-exporting-vmconsole-capture-export) operation to Cloud Object Storage. To identity your bucket name, select Menu icon ![Menu icon](https://cloud.ibm.com/docs-content/v1/content/1d36273917990fbf48078805e201d2e85e2fcb27/icons/icon_hamburger.svg) > Resource list > Storage > Cloud Object Storage name > Buckets. 
 Cloud Object Storage access key To identify your access key, select Menu icon ![Menu icon](https://cloud.ibm.com/docs-content/v1/content/1d36273917990fbf48078805e201d2e85e2fcb27/icons/icon_hamburger.svg) > Resource list > Storage > Cloud Object Storage name > Service credentials > View credentials.
For more information, see [Managing service ID API keys](https://cloud.ibm.com/docs/account?topic=account-serviceidapikeysserviceidapikeys). 
 Target file type Select the file type that you want to export. If you imported a VMDK image, you can export that image in VHD or VMDK format. If the file was originally in VHD format, you can export only as VHD. 







 Next Steps 

Because each image is a different size and has different characteristics, the export process might take several minutes before it completes.

After you export an image, the image remains in the list of image templates, but it is also available as a file in the IBM Cloud Object Storage location that is specified during the export process. You can download the image file from IBM Cloud Object Storage. From the Resource List in IBM Cloud console, access your Cloud Object Storage service instance. In the bucket where your image is stored, select the image files that you want to download and select Download objects.

When your image template is exported to IBM Cloud Object Storage, each block device (or disk) has its own associated file. For example, if your image file is named image.vhd, the first block device is exported as image-0.vhd. The second block device is exported as image-1.vhd, and so on.
Content that is indexed with a boost value greater than 1 is more relevant than content that is indexed without a boost value. Content with a boost value less than one isn't so relevant. A positive floating point number 1 (No boosting) 
 facet Creates a faceted index. For more information, see [Faceting](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-cloudant-searchfaceting). true false 
 index Whether the data is indexed, and if so, how. If set to false, the data can't be used for searches, but can still be retrieved from the index if store is set to true. For more information, see [Analyzers](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-search-analyzers). true, false true 
 store If true, the value is returned in the search result; otherwise, the value isn't returned. true, false false 



If you don't set the store parameter, the index data results for the document aren't returned in response to a query.

See the following example search index function:

function(doc) {
index("default", doc._id);
if (doc.min_length) {
index("min_length", doc.min_length, {"store": true});
}
if (doc.diet) {
index("diet", doc.diet, {"store": true});
}
if (doc.latin_name) {
index("latin_name", doc.latin_name, {"store": true});
}
if (doc.class) {
index("class", doc.class, {"store": true});
}
}



 Index guard clauses 

The index function requires the name of the data field to index as the second parameter. However, if that data field doesn't exist for the document, an error occurs. The solution is to use an appropriate "guard clause" that checks whether the field exists.
You can adjust the zone, the machine profile and image type, etc.

resource "ibm_is_instance" "vsi1" {
name = "${var.environment_name}-vsi1"
vpc = ibm_is_vpc.vpc1.id
zone = "${var.region}-1"
profile = "cx2-2x4"
image = data.ibm_is_image.vsi_image.id
keys = [ibm_is_ssh_key.generated_key.id]
resource_group = ibm_resource_group.group.id

primary_network_interface {
subnet = ibm_is_subnet.subnet11.id
}
}

The SSH key to access the VSI is generated, too. The private key part of this SSH key is written to the output directory and can be used later on to connect to the VSI.

IAM-enabled services like IBM Cloud Object Storage and IBM Cloudant are also created as resources within the group:

 a database
resource "ibm_resource_instance" "database" {
name = "database"
service = "cloudantnosqldb"
plan = "${var.cloudantnosqldb_plan}"
location = "${var.cloudantnosqldb_location}"
resource_group_id = "${ibm_resource_group.group.id}"
}

 a cloud object storage
resource "ibm_resource_instance" "objectstorage" {
name = "objectstorage"
service = "cloud-object-storage"
plan = "${var.cloudobjectstorage_plan}"
location = "${var.cloudobjectstorage_location}"
resource_group_id = "${ibm_resource_group.group.id}"
}
Show more







 Step 3: Deploy this environment in your account 



 Create a local working environment 

All of the operations will be done in a bash shell and making use of terraform and ibmcloud command.
To export a custom image by using the API, use [Create an image export job](https://cloud.ibm.com/apidocs/vpc/latestcreate_image_export_job).

For the$image_id variable, specify the ID of the custom image that you want to export. You can optionally specify a meaningful name for the image export job. In this example, the export job name is my-image-export. For the storage_bucketname subproperty, specify the name of the IBM Cloud Object Storage bucket where you want to export the custom image. In this example, the bucket name is bucket-27200-lwx4cfvcue. Alternatively, you can use the crn subproperty to specify the CRN of the bucket.

curl -X POST "$vpc_api_endpoint/v1/images/$image_id/export_jobs?version=2022-11-21&generation=2" -H "Authorization: Bearer $iam_token" -d '{
"name": "my-image-export",
"storage_bucket": {
"name": "bucket-27200-lwx4cfvcue"
}
}'







 Viewing and managing custom image export history 

You can view the custom image export history to see details such as the export job name, status, target bucket in IBM Cloud Object Storage, the IBM Cloud Object Storage object name that is assigned, the date that the export job started, and if the export job is complete, the completion date.

You can also delete a completed image export job. Or you can delete (and cancel) an in-progress image export job if you have an IAM role that includes the is.image.image.export action.

When you delete an image export job, it is removed only from the export history. The exported image file is not removed from the IBM Cloud Object Storage bucket. You must manage files that are stored in IBM Cloud Object Storage by using the IBM Cloud Object Storage interfaces.

Finally, you can rename an image export job. Renaming the image export job does not change the name of the object that is created in IBM Cloud Object Storage.
Importing your custom image into IBM Cloud VPC 

After your custom image is created and stored on IBM Cloud Object Storage, you must import that custom image directly to IBM Cloud VPC. This requirement includes any custom images that you plan to import into a private catalog. The custom images must be imported into IBM Cloud VPC first and then imported into the private catalog.



*  Upload your custom image to IBM Cloud Object Storage.
*  On the Objects page of your IBM Cloud Object Storage bucket, click Upload. You can use the Aspera high-speed transfer plug-in to upload images that are larger than 200 MB.
*  Import your custom image to IBM Cloud VPC. When a supported custom image is available in IBM Cloud Object Storage, the image is ready to import. For more information, see [Importing and validating custom images into VPC](https://cloud.ibm.com/docs/vpc?topic=vpc-importing-custom-images-vpc).