vania-janet/multiturn-rag-retrieval-data huggingface dataset:
If you prefer to use the Discovery API to train Discovery, see the [API reference](https://cloud.ibm.com/apidocs/discovery-datalisttrainingqueries).

You also can use the API to add curations. Curations is a beta feature that you can use to teach Discovery to return a specific document every time a certain query is submitted. For more information, see [Curations](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-curations).

Adding a custom stopwords list can also improve the relevance of results for natural language queries. For more information, see [Identifying words to ignore](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-stopwords).





 Understanding relevancy training 

Answers to common questions about training a project.



 How do I know whether my system is trained? 

Run a natural language query and check the document_retrieval_strategy. See [confidence scores](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-trainconfidence).

If you are using the API, see [List training queries](https://cloud.ibm.com/apidocs/discovery-datalisttrainingqueries).





 How long does it take to train a model? 

It can take between 45 minutes to an hour for the training to finish. The duration of the training differs depending on the amount and variety of the data that is used to train the relevancy model. Also, the training occurs asynchronously. It can be delayed if other data that it needs is unavailable because it is being searched or processed in some other way.





 How do I stop relevancy training from being applied to my project? 

Use the API to delete the relevancy model that is associated with your project. To delete the model, you delete that training data that is associated with the ranker model. For more information, see [Deleting training queries](https://cloud.ibm.com/apidocs/discovery-datadeletetrainingqueries).
* [Do you support using certificates from non-IBM Certificate Authorities (CAs)?](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-faq-v2-external-certs)
* [What is the recommended way to manage private keys?](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-faq-hsm)
* [Can I integrate my corporate LDAP server with the Certificate Authority (CA) in the IBM Blockchain Platform?](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-faq-ldap)
* [What is the process for rotating certificates on a periodic basis?](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-faq-cert-mgmt)



Developing applications and smart contracts



* [What languages are supported for smart contracts?](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-faq-v2-IBP-Overview-1-4)
* [What version of the IBM Blockchain Platform works with the Ansible collection?](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-faq-ansible-version)
* [How do I get support for running the IBM Blockchain Platform Ansible playbook?](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-faq-ansible-support)
* [Can the IBM Blockchain Platform monitor the health of a client application?]
* [Table understanding](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-understanding_tables)





 Watson NLP enrichments 

For example, the following screen capture shows a transcript of the US Declaration of Independence that was added to a Discovery collection where the Entities and Keywords enrichments are enabled. The mentions that are recognized by the enrichments are highlighted in the document text.

Zoom

![Shows an excerpt of the US Declaration of Independence with several terms highlighted.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/nlu-rich-both.png)

Figure 1. Excerpt of the US Declaration of Independence with highlighted terms

Some of the NLP enrichments are applied to projects automatically. You don't need to apply them yourself if you are using one of these project types.





 Default enrichments per project type 

Some prebuilt enrichments are applied automatically to collections in a project based on the project type. The following table shows the default enrichments that are applied to each project type.



Default enrichments per project type
This table has row and column headers. The row headers identify project types. The column headers identify different enrichments. To understand which enrichments are applied to a project type by default, go to the row that describes the enrichments, and find the columns for the project type that you are interested in.

 Enrichment Document Retrieval Document Retrieval for Contracts Conversational Search Content Mining 

 Contracts ![checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/icons/checkmark-icon.svg) 
 Entities !
The ICIAE can be requested by choosing a plan with the Contact IBM Sales option. For more information about ICIAE, see: [IBM Cloud Integrated Analytics Environment](http://www-01.ibm.com/common/ssi/ShowDoc.wss?docURL=/common/ssi/rep_ca/1/897/ENUS217-361/index.html&lang=en&request_locale=en).





 Getting help and support 

If you have any questions or issues while using IBM Informix on Cloud for IBM Cloud, you can get help by doing one or more of the following:



* Watch our [videos](https://developer.ibm.com/clouddataservices/docs/informix/).
* Navigate through or post questions on one of the forums.



* When using forums to ask a question, make sure to tag your question so that it is seen by the IBM Cloud development teams.
* If you have technical questions about deploying an app with Informix on Cloud, post your question on [Stack Overflow](http://stackoverflow.com/search?q=informixoncloud+bluemix) and tag your question with "ibm-cloud" and "informix".
* For questions about the service and getting started instructions, use the [IBM developerWorks® dW Answers](https://developer.ibm.com/answers/smartspace/bluemix/index.html) forum. Include the "ibm-cloud" and "informix" tags.
* See [Getting help](https://cloud.ibm.com/docs/get-support?topic=get-support-getting-customer-supportusing-avatar) for more details about using the forums.



* Open a support ticket. For information about opening an IBM support ticket, or about support levels and ticket severity, see [Contacting support](https://cloud.ibm.com/docs/get-support?topic=get-support-getting-customer-supporttypesofsupport).
html is produced for documents processed with Smart Document Understanding or Content Intelligence. The html field must be removed before relevancy training can complete successfully. * If the Part of Speech enrichment is not turned on: Dynamic facets will not be created, Dictionary suggestions cannot be used, Content Miner "extracted facets" will not generate. * [Update: fixed in version 2.1.1 Discovery for Content Intelligence and Table Understanding enrichments are configured out of the box to be applied on a field named html. When a user uploads a JSON document without a root-level field named html, these enrichments will not yield results in the index. To run the enrichments on this kind of JSON documents, users must re-configure the enrichments to run on an existing field (or fields) in the JSON document. * When viewing the Content Miner deploy page, sometimes the full application URL is not displayed for copying. To fix, refresh the page. * [Update: fixed in version 2.1.2 Deprovisioning a IBM Watson® Discovery for IBM Cloud Pak® for Data Instance will not delete the underlying data. Delete the collections and documents manually. * [Update: fixed in version 2.1.3 On the Improvement tools panel, the enrichment Sentiment of phrases is listed, but is not currently available. * In Content Mining projects, the dates fields may not be parsed properly for display in facets. * The Dynamic facets toggle should not appear in Content Mining projects. * A minimum of 50-100 documents should be ingested to see valid dynamic facets generated. * If you click Stop to stop a crawler and the converter processes slowly or has errors, you might see a status of the crawler running. * The total size limit of all non-HTML fields in uploaded and crawled documents is 1MB, which is equivalent to 1,048,576 bytes, and the total size limit of all HTML fields in these documents is 5MB. If you exceed either limit, you receive an error message stating The document has fields/HTML fields that exceed the 1 MB/5 MB limit., and the document is not ingested. For assistance on increasing either size limit, contact the IBM Support Center]] ! ! !
'Status': 'Enabled'|'Disabled'
}
},
]
}
)
Show more

Listing the versions of an object using the same client:

resp = cosClient.list_object_versions(Prefix='some-prefix', Bucket=BUCKET)

Note that the Python APIs are very flexible, and there are many different ways to accomplish the same task.





 Node.js 

Enabling versioning using the [IBM COS SDK for Node.js](https://ibm.github.io/ibm-cos-sdk-js/AWS/S3.htmlputBucketVersioning-property):

const IBM = require('ibm-cos-sdk');

var config = {
endpoint: '<endpoint>',
apiKeyId: '<api-key>',
serviceInstanceId: '<resource-instance-id>',
};

var cos = new IBM.S3(config);

var params = {
Bucket: 'STRING_VALUE', /* required /
ReplicationConfiguration: { /* required /
Role: 'STRING_VALUE', /* required /
Rules: [ /* required /
{
Destination: { /* required /
Bucket: 'STRING_VALUE', /* required /
},
Status: Enabled | Disabled, /* required /
Filter: {
And: {
Prefix: 'STRING_VALUE',
Tags:
{
Key: 'STRING_VALUE', /* required /
Value: 'STRING_VALUE' /* required /
},
/* more items /
]
},
Prefix: 'STRING_VALUE',
Tag: {
Key: 'STRING_VALUE', /* required /
Value: 'STRING_VALUE' /* required /
}
},
ID: 'STRING_VALUE',
Prefix: 'STRING_VALUE',
Priority: 'NUMBER_VALUE',
}
}
* [Using toolchains](https://cloud.ibm.com/docs/ContinuousDelivery?topic=ContinuousDelivery-toolchains-using)
* [Configuring tool integrations](https://cloud.ibm.com/docs/ContinuousDelivery?topic=ContinuousDelivery-integrations)
* [Working with Tekton pipelines](https://cloud.ibm.com/docs/ContinuousDelivery?topic=ContinuousDelivery-tekton-pipelines)
* [Working with Delivery Pipeline Private Workers](https://cloud.ibm.com/docs/ContinuousDelivery?topic=ContinuousDelivery-private-workers)
* [Creating toolchains with Git](https://cloud.ibm.com/docs/ContinuousDelivery?topic=ContinuousDelivery-toolchains_git)
* [Adding DevOps Insights to your toolchain](https://cloud.ibm.com/docs/ContinuousDelivery?topic=ContinuousDelivery-add-devops-insights)
* [Managing personal data for Continuous Delivery](https://cloud.ibm.com/docs/ContinuousDelivery?topic=ContinuousDelivery-cd_personal_data)



For more information about IBM Cloud Provider resources and data sources, see the [Terraform Registry documentation](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs).
Show more





 Node.js 

Enabling versioning using the [IBM COS SDK for Node.js](https://ibm.github.io/ibm-cos-sdk-js/AWS/S3.htmlputBucketVersioning-property):

'use strict';

// Required libraries
const ibm = require('ibm-cos-sdk');
const fs = require('fs');
const crypto = require('crypto');

function logError(e) {
console.log(ERROR: ${e.code} - ${e.message}n);
}

function logDone() {
console.log('DONE!n');
}

const COS_ENDPOINT = ""; //Choose endpoint from https://control.cloud-object-storage.cloud.ibm.com/v2/endpoints. Ex: https://s3.us-south.cloud-object-storage.appdomain.cloud
const COS_API_KEY_ID = ""; // API key of cos instance example: xxxd12V2QHXbjaM99G9tWyYDgF_0gYdlQ8aWALIQxXx4
const COS_AUTH_ENDPOINT = "";
const COS_RESOURCE_INSTANCE_CRN = ""; // example: crn:v1:bluemix:public:cloud-object-storage:global:a <CREDENTIAL_ID_AS_GENERATED>:<SERVICE_ID_AS_GENERATED>::

// Client Creation.
var config = {
endpoint: COS_ENDPOINT,
apiKeyId: COS_API_KEY_ID,
ibmAuthEndpoint: COS_AUTH_ENDPOINT,
serviceInstanceId: COS_RESOURCE_INSTANCE_CRN,
signatureVersion: 'iam'
};

var cos = new ibm.S3(config);
(https://search.maven.org/remotecontent?filepath=com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2-sources.jar),
ANNOTATIONS [MongoDB-Driver]
(https://search.maven.org/remotecontent?filepath=org/mongodb/mongodb-driver-core/4.0.5/mongodb-driver-core-4.0.5-sources.jar)

END OF CREATIVE COMMONS ATTRIBUTION 2.5 GENERIC NOTICES AND INFORMATION
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

CREATIVE COMMONS ATTRIBUTION 3.0 GENERIC

The Program includes some or all of the following works licensed under
the Creative Commons.
The url to the license is https://creativecommons.org/licenses/by/3.0/legalcode

GO GOPHER [Go] (https://golang.org/dl/go1.16.src.tar.gz),
TEMPLATES - PAGE CONTENT
[tools] (https://github.com/golang/tools/tree/2d16b83),
DOCUMENTATION [Atob] (http://registry.npmjs.org/atob/-/atob-2.1.2.tgz),
HTTP2/H2DEMO/TMPL.GO [net]
(https://github.com/golang/net/archive/eb5bcb51f2a31c7d5141d810b70815c05d9c9146.zip),
SPDX-EXCEPTIONS
(http://registry.npmjs.org/spdx-exceptions/-/spdx-exceptions-2.2.0.tgz),
for param in params:
print(param)
endpoint = params[0]
bucket_name = params[1]
log_config_file = params[2]
access_key = params[3]
secret_key = params[4]
cos = cos_utils.get_cos_object(access_key, secret_key, endpoint)

retCode = cos_utils.download_file(log_config_file, cos, bucket_name, "{}/{}".format(install_path, log_config_file))
if (retCode != 0):
print("non-zero return code while downloading file {}".format(str(retCode)))
sys.exit(retCode)
else:
print("Successfully downloaded file...")
Show more







 Using the library set created using script based customization 

When you use a library set that you created using a script, you need to include the path of the library in certain environment variables so that the library set can be accessed by your Spark application.

For example, if your custom library is a .so file, you need to add "EXTRA_LD_LIBRARY_PATH" to the "env" section of the Spark application submit call, with the value /home/spark/shared/user-libs/<libraryset_name>/custom/<subdir_if_applicable>. This prepends "EXTRA_LD_LIBRARY_PATH" to "LD_LIBRARY_PATH", ensuring that this is the first path to the .so file that is searched.

If your custom library is a JAR file and you need it to be accessible on the Spark classpath, you must specify the JAR file in the extra classpath for driver/executor depending on where you require the JAR file. For example, to add it in front of the driver classpath, add the following property, where the library set name is java_library:
* CC-BY-SA-3.0
* CC-BY-SA-4.0





 Creative Commons Attribution 2.5 Generic 

The Program includes some or all of the following works licensed under the Creative Commons. The URL to the license is [https://creativecommons.org/licenses/by/2.5/legalcode](https://creativecommons.org/licenses/by/2.5/legalcode).

JCIP-ANNOTATIONS [HttpCore] (Copyright not found) ([https://downloads.apache.org/httpcomponents/httpcore/source/](https://downloads.apache.org/httpcomponents/httpcore/source/))





 Creative Commons Attribution 3.0 Generic 

The Program includes some or all of the following works licensed under the Creative Commons. The URL to the license is [https://creativecommons.org/licenses/by/3.0/legalcode](https://creativecommons.org/licenses/by/3.0/legalcode).

(FONTAWESOME - DOCS) MONGO-R3.4.1/SRC/THIRD_PARTY/WIREDTIGER/TOOLS/WTSTATS/TEMPLATE/APP/LESS/FONTAWESOME.LESS [MongoDB] (Copyright not found)([http://registry.npmjs.org/mongodb/-/mongodb-3.4.1.tgz](http://registry.npmjs.org/mongodb/-/mongodb-3.4.1.tgz))

SPDX [npm] (Copyright not found)([https://github.com/nodejs/node/tree/v12.13.0/deps/npm](https://github.com/nodejs/node/tree/v12.13.0/deps/npm))





 Creative Commons Attribution 4.0 Generic 

The Program includes some or all of the following works licensed under the Creative Commons.
Or, if you prefer, there is a [downloadable spreadsheet](https://cloud.ibm.com/media/docs/downloads/framework-financial-services/IBM_Cloud_Framework_for_Financial_Services_-_Control_Requirements_v1.1.0.xlsx) available as well.





 Framework guidance and reference architectures 



 Interpreting the framework guidance 

We refer to the entity that uses (consumes) an application as the consumer and the entity that develops or deploys (provides) an application as the application provider. The consumer and application provider might be the same when the organization providing the application also consumes it (for example, a financial institution deploys an application for internal use by others in their company). Or, they might be different if a third-party ecosystem partner is providing an application for a financial institution to use.

You should use the information in the framework if you are an application provider (whether from a financial institution or ecosystem partner) who wishes to meet the requirements or leverage the best practices of IBM Cloud for Financial Services. However, the content has a different purpose depending on whether you represent an application provider from an ecosystem partner or an application provider within a financial institution.

Ecosystem partner
: The reference architectures and guidance form a set of requirements if you want to pursue a designation of [IBM Cloud for Financial Services Validated](https://cloud.ibm.com/docs/framework-financial-servicesbecoming-fs-validated) from IBM. The Financial Services Validated designation signifies that you have successfully evidenced compliance to the control requirements of the IBM Cloud Framework for Financial Services and may improve your ability to market to financial institutions. Even though these should be considered requirements, there is a process for requesting the approval of deviations when certain aspects of the guidance may not be feasible or make sense for a given application.

Financial institution
: The reference architectures and guidance form a set of recommendations. As a financial institution, you know your controls better than IBM does. So, you have latitude in terms of following the guidance depending on what your internal controls dictate and the level of risk you are willing to take. IBM does not formally validate the applications of financial institutions.
* Download the [scripts](https://github.com/watson-developer-cloud/doc-tutorial-downloads/tree/master/knowledge-studio).
* Review information about the script's use of the MinIO client. The client is required for the MinIO commands.



* The scripts download the client from the [MinIO website](https://dl.min.io/) if the client isn't installed.
* If you want the script to use your installed version, verify that you can run the client by issuing the command mc on the command line.









 all-backup-restore script 

The all-backup-restore.sh script backs up and restores the MongoDB, PostgreSQL, and Minio databases, and PVC.

Unless you only need to back up a single database, it is recommended that you use the all-backup-restore script, which deactivates and reactivates Knowledge Studio.

The script backs up or restores the data in the following order:



1. MongoDB
2. PostgreSQL
3. MinIO
4. PVC



all-backup-restore.sh [backup|restore] [RELEASE_NAME] [BACKUP_DIR] [-n NAMESPACE]

Use either the backup or restore command.

backup
: Backs up each database to a subdirectory of the `BACKUP_DIR` directory.

restore
: Restores the data from each database in the `BACKUP_DIR` directory.



 Arguments and options 

RELEASE_NAME
: Required. The release name that was specified when the Knowledge Studio Helm chart was installed in your cluster.
: You can find the release name as the prefix of the pod name, for example, {release_name}-ibm-watson-ks-yyy-xxx. For version 1.2.0, the value is always `wks`.

BACKUP_DIR
: Required. The base directory of each database where backups are stored to or restored from. Each database is stored in a subdirectory of the backup directory.
* UI
* CLI
* API
* Terraform







 Provisioning 

Provision a IBM Cloud® Databases for MongoDB deployment through the [catalog](https://cloud.ibm.com/catalog/services/databases-for-mongodb), the [Cloud Databases CLI](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-cdb-reference), the [Cloud Databases API](https://cloud.ibm.com/apidocs/cloud-databases-api/cloud-databases-api-v5), or through [Terraform](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/resources/database).



 Provisioning through the catalog 

Deploy from the catalog by specifying the following parameters:



* Service name - The name can be any string and is the name that is used on the web and in the CLI to identify the new deployment.
* Location - The deployment's public cloud region or Satellite location.
* Database Version

Set only at deployment

- The deployment version of your database. To ensure optimal performance, run the preferred version. The latest minor version is used automatically. For more information, see [Database Versioning Policy](https://cloud.ibm.com/docs/databases-for-mongodb?topic=cloud-databases-versioning-policy).
* Resource allocation - Specify initial RAM, disk, and cores for your databases. The minimum sizes of memory and disk are selected by default. With dedicated cores, your resource group is given a single-tenant host with a minimum reserve of CPU shares. Your deployments are then allocated the number of cores you specify. Once provisioned, disk cannot be scaled down.
* Endpoints

Set only at deployment

- Configure the [Service Endpoints](https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-service-endpoints) on your deployment. A MongoDB deployment cannot have both public and private endpoints simultaneously.
Display your tables and run an SQL statement:

spark.sql('show tables').show(truncate=False)

 replace yourTable with a valid table. Do not use the sample tables as you do not have access to the data!
spark.sql('select * from yourTable').show()





 Usage with IBM Analytics Engine 

The Hive metastore client is also already included in IBM® Analytics Engine. It also includes the convenience library to configure the connection to the Hive metastore. The following example shows a Spark batch job for a show tables example in Python:

import sys
from dataengine import SparkSessionWithDataengine

if __name__ == '__main__':
crn = sys.argv[1]
apikey = sys.argv[2]

print(" Start SparkSessionWithDataengine example")
session_builder = SparkSessionWithDataengine.enableDataengine(crn, apikey, "public")

print(" Setup IBM Cloud Object Storage access")
spark = session_builder.appName("AnalyticEngine DataEngine integration") 
.config("fs.cos.impl", "com.ibm.stocator.fs.ObjectStoreFileSystem") 
.config("fs.stocator.scheme.list", "cos") 
.config("fs.stocator.cos.impl", "com.ibm.stocator.fs.cos.COSAPIClient") 
.getOrCreate()

print(" Got a spark session, listing all tables")
spark.sql('show tables').show()

spark.stop()
Show more

Prepare a JSON file to start that program, as in the following example (listTablesExample.json):

{
"application_details": {
output "cd_toolchain_attributes" {
value = ibm_cd_toolchain.cd_toolchain
}

For more information about ibm_cd_toolchain, see the argument reference details in the [Terraform Registry Documentation](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/resources/cd_toolchain).
3. Initialize the Terraform CLI.

terraform init
4. Apply the Terraform execution plan with the refresh-only option. Terraform refreshes its state and displays the attributes of the toolchain resource.

terraform apply -refresh-only -auto-approve









 Take a tutorial: Using toolchains 

Check out the [Add a toolchain to an app](https://www.ibm.com/cloud/architecture/tutorials/add-a-toolchain-to-an-app?task=2) tutorial on the [IBM Cloud Garage Method](https://www.ibm.com/cloud/architecture/toolchains).