vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Watson services integration 

Share domain artifacts and models between Knowledge Studio and other Watson services available on IBM Cloud Pak for Data.

Use Knowledge Studio to perform the following tasks:



* Upload analyzed documents that are in [UIMA CAS XMI format](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-preannotationwks_uimaweximport). For example, you can upload UIMA CAS XMI files that were exported from IBM Watson Explorer content analytics collections or [IBM Watson Explorer Content Analytics Studio](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-preannotationwks_uimawexstudio).
* Export a [machine learning](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-publish-mlwks_madiscovery) or [rule-based](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-wks_rule_publishwks_rule_discovery) model to use with Watson Discovery for IBM Cloud Pak for Data.
* Export a [machine learning](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-publish-mlwks_manlu) to use with Natural Language Understanding for IBM Cloud Pak for Data.
* [Export a machine learning model](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-publish-mlwks_maexport) to use in IBM Watson Explorer.
* [Export a rule-based model PEAR file](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-wks_rule_publishwks_rule_export) to use in IBM Watson Explorer.
Cloud Pak for Data Unlimited 
 Premium 10 
 Enterprise 10 
 Plus (includes Trial) 3 



For each Knowledge Studio machine learning model, the maximum number of entities that can be detected is 50.





 Advanced rules models 

Add an advanced rules model to apply a text extraction model that was created and exported from the Advanced Rule editor of IBM Watson® Knowledge Studio to your collection.

Your model must be created with the appropriate Knowledge Studio deployment:



IBM Cloud Pak for Data

You can add models that were created and exported from the following places:



* IBM Watson® Knowledge Studio that was built with a IBM Cloud Pak® for Data deployment earlier than the 4.5 release.
* IBM Watson® Knowledge Studio that is hosted on IBM Cloud
* NLP Editor that is built by contributors to the Center for Open-source Data & AI Technologies



IBM Cloud

You can add models that were created with a IBM Watson® Knowledge Studio instance that is hosted on IBM Cloud only.





 Removal from Knowledge Studio 

Support for building models with the beta Advanced Rules Editor in Knowledge Studio ended. Any rules models that were exported from Knowledge Studio prior to the end of support date can continue to be used in Discovery.

End of support dates differ based on the deployment type:



IBM Cloud

30 June 2022
IBM Cloud Pak for Data

IBM Cloud Pak for Data release 4.5.1 on 3 August 2022.



IBM Cloud

As an alternative to using a model that is generated by the Knowledge Studio Advanced Rules Editor, you can define a rule by [adding a Patterns enrichment](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain-pattern).





 Adding an existing model 

To add an advanced rule model, complete the following steps:



1. Create the model and export the ZIP file that contains the model resources.

For more information about how to export the model, see the instructions for your model source:



* [Knowledge Studio for IBM Cloud Pak® for Data](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-create-advanced-rules-model)
* [Watson OpenScale](https://www.ibm.com/docs/SSQNUZ_4.6.x/svc-welcome/aiopenscale.html): Infuse your AI with trust and transparency. Understand how your AI models make decisions to detect and mitigate bias.
* [Watson Query](https://www.ibm.com/docs/SSQNUZ_4.6.x/svc-welcome/dv.html): Create data sets from disparate data sources so that you can query and use the data as if it came from a single source. When you provision this service, check the You must check this box if you updated the kernel semaphore parameter check box and use the recommended storage class. See [Storage](https://cloud.ibm.com/docs/cloud-pak-data?topic=cloud-pak-data-getting-startedstorage).
* [Watson Studio](https://www.ibm.com/docs/SSQNUZ_4.6.x/svc-welcome/wsl.html): Build custom models and infuse your business with AI and machine learning.







 Supported versions 

The current release of IBM Cloud Pak for Data on IBM Cloud is IBM Cloud Pak for Data Version 4.6.x.





 Before you begin 

Before you can install IBM Cloud Pak for Data, you must purchase a license through [IBM Passport Advantage](https://www.ibm.com/software/passportadvantage/index.html) or register for a 60-day trial license. See [Step 1. Assign the license](https://cloud.ibm.com/docs/cloud-pak-data?topic=cloud-pak-data-getting-startedassign_license).

You can deploy Cloud Pak for Data on Virtual Private Cloud (VPC) Gen2 infrastructure. You can use either a single or multi zone deployment. For more information, see [Creating a VPC cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-getting-started&interface=uivpc-gen2-gs).
* [Watson OpenScale](https://www.ibm.com/docs/SSQNUZ_4.6.x/svc-welcome/aiopenscale.html): Infuse your AI with trust and transparency. Understand how your AI models make decisions to detect and mitigate bias.
* [Watson Query](https://www.ibm.com/docs/SSQNUZ_4.6.x/svc-welcome/dv.html): Create data sets from disparate data sources so that you can query and use the data as if it came from a single source. When you provision this service, check the You must check this box if you updated the kernel semaphore parameter check box and use the recommended storage class. See [Storage](https://cloud.ibm.com/docs/cloud-pak-datastorage).
* [Watson Studio](https://www.ibm.com/docs/SSQNUZ_4.6.x/svc-welcome/wsl.html): Build custom models and infuse your business with AI and machine learning.







 Supported versions 

The current release of IBM Cloud Pak for Data on IBM Cloud is IBM Cloud Pak for Data Version 4.6.x.





 Before you begin 

Before you can install IBM Cloud Pak for Data, you must purchase a license through [IBM Passport Advantage](https://www.ibm.com/software/passportadvantage/index.html) or register for a 60-day trial license. See [Step 1. Assign the license](https://cloud.ibm.com/docs/cloud-pak-dataassign_license).

You can deploy Cloud Pak for Data on Virtual Private Cloud (VPC) Gen2 infrastructure. You can use either a single or multi zone deployment. For more information, see [Creating a VPC cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-getting-started&interface=uivpc-gen2-gs).

You must ensure that you have sufficient resources for the services that you plan to install.
Therefore, after the migration, you can recreate the model. The model that will be produced will perform the same as the models you generated prior to the migration because the resources that are used for training will be the same.

ATTENTION: If you have a model that is already deployed and you plan to delete the workspace after you back it up, withdraw the model from deployment. You can rebuild and redeploy the model after you restore the workspace from the backup. For information about undeploying models, see [Undeploying machine learning models](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-publish-ml) and [Undeploying rule-based models](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-wks_rule_publish).

Be aware that if you fail to withdraw the model from deployment, the result is an orphaned deployed model. Orphaned deployed models will continue to generate charges on your monthly bills.
4. Manage read-only dictionary information.

Read-only dictionaries cannot be migrated. Find out where the read-only dictionary was imported from, so you can reupload it to your workspace after the migration.
5. Make a list of current user roles.

NOTE: This step is optional. It is typically performed when a workspace is migrated across instances and the new workspace must be identical to the original workspace. If you want to migrate only the workspace data into a new workspace, you can skip this step.

If you are migrating workspaces across different instances, consider making a list of users and their roles for the instance that you are backing up. Someone with the Admin role can print the list from the User Account Management page. After the workspaces are recreated on the new instance, someone with the Admin role must add the users and assign their roles.

For more information about roles, see [User roles in Knowledge Studio](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-roles).
RAM-intensive bare metal, mb4c.20x192: Maximize the RAM available to your worker nodes. This bare metal includes 2nd Generation Intel® Xeon® Scalable Processors with Intel® C620 Series chip sets for better performance for workloads such as machine learning, AI, and IoT. 20 / 192 GB 2 TB HDD / 960 GB SSD 10000 Mbps 
 RAM-intensive bare metal, mb4c.20x384: Maximize the RAM available to your worker nodes. This bare metal includes 2nd Generation Intel® Xeon® Scalable Processors with Intel® C620 Series chip sets for better performance for workloads such as machine learning, AI, and IoT. 20 / 384 GB 2 TB HDD / 960 GB SSD 10000 Mbps 
 GPU bare metal, mg4c.32x384.2xp100: Choose this type for mathematically intensive workloads such as high-performance computing, machine learning, deep learning, or 3D applications. This flavor has two Tesla P100 physical cards that have two GPUs per card for a total of four GPUs. Note that this Pascal GPU flavor does not support the Data Center GPU Manager because of a known issue from NVIDIA. 32 / 384 GB 2 TB HDD / 960 GB SSD 10000 Mbps 
 Balanced bare metal, me4c.4x32: Use for balanced workloads that require more compute resources than virtual machines offer. This bare metal includes 2nd Generation Intel® Xeon® Scalable Processors with Intel® C620 Series chip sets for better performance for workloads such as machine learning, AI, and IoT. 4 / 32 GB 2 TB HDD / 2 TB HDD 10000 Mbps 









 Software-defined storage (SDS) machines 

Software-defined storage (SDS) flavors have additional raw disks for physical local storage. Unlike the primary and auxiliary local disk, these raw disks are not wiped during a worker node update or reload. Because data is co-located with the compute node, SDS machines are suited for high-performance workloads.

Software-defined storage flavor are available for classic clusters only and are not supported in VPC clusters.



 Planning considerations for SDS
This bare metal includes 2nd Generation Intel® Xeon® Scalable Processors with Intel® C620 Series chip sets for better performance for workloads such as machine learning, AI, and IoT. 20 / 192 GB 2 TB HDD / 960 GB SSD 10000 Mbps 
 RAM-intensive bare metal, mb4c.20x384: Maximize the RAM available to your worker nodes. This bare metal includes 2nd Generation Intel® Xeon® Scalable Processors with Intel® C620 Series chip sets for better performance for workloads such as machine learning, AI, and IoT. 20 / 384 GB 2 TB HDD / 960 GB SSD 10000 Mbps 
 GPU bare metal, mg4c.32x384.2xp100: Choose this type for mathematically intensive workloads such as high-performance computing, machine learning, deep learning, or 3D applications. This flavor has two Tesla P100 physical cards that have two GPUs per card for a total of four GPUs. Note that this Pascal GPU flavor does not support the Data Center GPU Manager because of a known issue from NVIDIA. 32 / 384 GB 2 TB HDD / 960 GB SSD 10000 Mbps 
 Balanced bare metal, me4c.4x32: Use for balanced workloads that require more compute resources than virtual machines offer. This bare metal includes 2nd Generation Intel® Xeon® Scalable Processors with Intel® C620 Series chip sets for better performance for workloads such as machine learning, AI, and IoT. 4 / 32 GB 2 TB HDD / 2 TB HDD 10000 Mbps 









 Software-defined storage (SDS) machines 

Software-defined storage (SDS) flavors have additional raw disks for physical local storage. Unlike the primary and auxiliary local disk, these raw disks are not wiped during a worker node update or reload. Because data is co-located with the compute node, SDS machines are suited for high-performance workloads.

Software-defined storage flavor are available for classic clusters only and are not supported in VPC clusters.



 Planning considerations for SDS
You can also deploy apps that don't need to process their workloads across both the GPU and CPU. After, you might find it useful to play around with mathematically intensive workloads such as the [TensorFlow](https://www.tensorflow.org/) machine learning framework with [this Kubernetes demo](https://github.com/pachyderm/pachyderm/tree/master/examples/ml/tensorflow).

Before you begin



* Create a [cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-clusters) or [worker pool](https://cloud.ibm.com/docs/openshift?topic=openshift-add_workersadd_pool) that uses a GPU bare metal flavor. Keep in mind that setting up a bare metal machine can take more than one business day to complete.
* Make sure that you are assigned a [service access role](https://cloud.ibm.com/docs/openshift?topic=openshift-userschecking-perms) that grants the appropriate Kubernetes RBAC role so that you can work with Kubernetes resources in the cluster.
* [Install the NVIDIA GPU operator for your cluster version](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/getting-started.htmloperator-install-guide).
* [Install the Node Feature Discovery and NVIDIA GPU operators for your cluster version](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/getting-started.html).

You must use NVIDIA GPU operator version 1.3.1 or later. When you install the Node Feature Discovery operator, select the update channel that matches your Red Hat OpenShift cluster version. Do not install the operators through another method, such as a Helm chart.



To run a workload on a GPU machine,



1. Create a YAML file.
In the Analyzed document export options area, select Export documents as XML files, select the Enable CAS as XMI format export check box, specify the output path for where the exported data is to be written, and click OK.
4. Stop and restart the parse and index services for the collection, and then do one of the following steps:



* If the collection already contains indexed documents that you want to use for training the machine learning model in the document cache, restart a full index build.
* If the collection does not contain indexed documents that you want to use for training the machine learning model, upload documents, configure at least one crawler to crawl the documents, and start the crawler.



5. In the Export area, check the status of the export request. The progress indicates how many documents are exported.
6. Go to the output folder that you specified when you configured export options. When documents are exported as XML files, the output folder name is based on the time stamp when the export occurs. The output folder contains XMI files (.xmi) and the UIMA TypeSystem descriptor file (exported_typesystem.xml).







 What to do next 

You must define a mapping between the UIMA types and Knowledge Studio entity types. You must also create a .zip file that contains all the files that are required to upload the analyzed data into a Knowledge Studio workspace.

Related information:

[Exporting crawled or analyzed documents](http://www.ibm.com/support/knowledgecenter/SS8NLW_11.0.0/com.ibm.discovery.es.ad.doc/iiysatexportca.htm)

[Output paths for exported documents](http://www.ibm.com/support/knowledgecenter/SS8NLW_11.0.0/com.ibm.discovery.es.ad.doc/iiysaexportoutput.htm)







 Exporting an analyzed collection from Content Analytics Studio 

You can export a collection of analyzed documents from Watson Explorer Content Analytics Studio, and upload the analyzed documents as XMI files into a Knowledge Studio project.



 Procedure 

To get analyzed documents from a Content Analytics Studio collection, follow these steps:



1.
Once the service is provisioned, Click Manage on the left pane and click Launch Application.
3. Click on Manual setup to manually setup the monitors.







 Selecting a deployment 

In this section, as part of preparing your model for monitoring you will set up and enable monitors for each deployment that you are tracking with IBM Watson OpenScale.



1. By clicking on the Edit icon on the Database tile, choose Free lite plan database as your Database type and click Save. This is to store your model transactions and model evaluation results.
2. Click on Machine learning providers



1. Click on Add machine learning provider and click the edit icon on the Connection tile.
2. Select Watson Machine Learning(V2) as your service provider type.
3. In the Deployment space dropdown, select the deployment space iris_deployment_space you created above.
4. Leave the Environment type to Pre-production.
5. Click Save.



3. On the left pane:



1. Click on Insights dashboard(first icon) to add a deployment
2. Click on Add to dashboard and select iris_deployment
3. Click on Configure.



4. Click on Configure monitors to setup your monitors.







 Provide model details 

Provide information about your model so that IBM Watson OpenScale can access the database and understand how the model is set up.



1. Provide the Model details by clicking the edit icon on the Model input tile and select



1. Data type: Numerical/categorical
2. Algorithm type: Multi-class classification
3. Click Save and continue



2. Click the edit icon on the Training data tile and select



1. Storage type: Database or cloud storage
2. Location: Cloud Object Storage
3. For Resource instance ID and API key, run the below command in the Cloud Shell. Make sure to change the value after --instance-name to match the name of the Object Storage instance you have been using for this tutorial.
Release notes for IBM Watson® Discovery Cartridge for IBM Cloud Pak® for Data 

Learn about features and changes that were included for each release and update of the product software.

IBM Cloud Pak for Data

This information applies only to instances of IBM Watson® Discovery that are installed on IBM Cloud Pak® for Data. For information about releases and updates for managed deployments, see [Release notes for Watson Discovery for IBM Cloud](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-release-notes).

For the list of Discovery known issues, see [Limitations and known issues in Watson Discovery](https://www.ibm.com/docs/SSQNUZ_4.6.x?topic=issues-watson-discovery).



 Knowledge Studio for IBM Cloud Pak for Data deprecation announcement 

After version 4.7, the operator for IBM Knowledge Studio will no longer be supported and will be removed from the IBM Watson Discovery Cartridge for IBM Cloud Pak for Data and from github.com. The service will not be displayed in the Cloud Pak for Data catalog. This change will not impact existing deployments of the operator.

Migrate your solutions to Watson Discovery, which has powerful custom natural language processing capabilities. Any existing Watson Knowledge Studio for Cloud Pak for Data rules-based or machine learning models can be imported to Watson Discovery and applied to your data as custom enrichments. And the recent release of the custom entities extraction feature brings equivalent function to label and train custom entity models into Watson Discovery. For more information about these features, see [Choose enrichments](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain).

For more information about migrating your solutions, see [Migrating Knowledge Studio solutions](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-migrate-wks).





 4.7.0 release, 28 June 2023 

IBM Watson® Discovery Cartridge for IBM Cloud Pak® for Data 4.7.0 is available.
CSV file that is used to train the document classifier machine learning model. The file can contain key data points per column. The data points can vary, but the file must include the following columns:



* Natural language text that you want to classify or label.
* Label or class name that categorizes the idea that is expressed in the document text. You can apply more than one label to a text sample. Separate multiple label values with a semicolon.



Test data
: Optional. CSV file that is used to test the document classifier machine learning model after it is trained. If you don't specify a separate file for testing, a subset of the training data content is used for testing purposes.

Target data
: Required. CSV file with the data that you want to classify.

All of the CSV files (training, test, and target) must have the same column names. The data in the columns must have the same data types, such as string, number, and so on.

You can use a CSV file that you uploaded at the time that you created the Content Mining project or you can create a new collection.

For more information, see the following topics:



* [Adding collections](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-add-collection)
* [Analyzing CSV files](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-csv-file)





 Document classifier training data sample 

The following table shows an example of the type of content that might be stored in CSV files that are used to train a document classifier.



Table 1. Sample data for CSV files

 Claim_id Date Product_line Product Client_segments Client_location Client age Feedback Label 

 0 2016/1/1 tea lemon tea Not Member Manhattan 20 The straw was peeled off from the juice pack. package_container 
 1 2016/1/2 ice cream vanilla ice cream Silver Member Queens 20 I got some ice cream for my children, but there was something like a piece of thread inside the cup. contamination_tampering
* Deploy a [machine learning](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-publish-mlwks_madiscovery) or [rule-based](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks_rule_publishwks_rule_discovery) model to use with the Watson Discovery service.
* Deploy a [machine learning](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-publish-mlwks_manlu) or [rule-based](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks_rule_publishwks_rule_nlu) model to use with the Natural Language Understanding service.
* [Export a machine learning model](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-publish-mlwks_maexport) to use in IBM Watson Explorer.
* [Export a rule-based model PEAR file](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks_rule_publishwks_rule_export) to use in IBM Watson Explorer.







 HIPAA support 

US Health Insurance Portability and Accountability Act (HIPAA) support is available for Premium plans in the Washington, DC location created on or after 1 April 2019. See [Enabling EU and HIPAA supported settings](https://cloud.ibm.com/docs/account?topic=account-eu-hipaa-supportedeu-hipaa-supported) for more information.