vania-janet/multiturn-rag-retrieval-data huggingface dataset:
If you want the crawler to extract text from images in documents, expand More processing settings, and set Apply optical character recognition (OCR) to On.

When OCR is enabled and your documents contain images, processing takes longer. For more information, see [Optical character recognition](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionsocr).
11. Click Finish.



The collection is created quickly. It takes more time for the data to be processed as it is added to the collection.

If you want to check the progress, go to the Activity page. From the navigation pane, click Manage collections, and then click to open the collection.





 Prerequisite steps for NFS storage 

Choose one of the following methods to enable the crawler pod to access the file system:



* [Configure an external NFS server](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-lfs-cp4duse-external-nfs)
* [Configure dynamic provisioning with an NFS storage class](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-lfs-cp4ddyn-prov-nfs)





 Configuring an external NFS server 

If the local file system files or folders that you want to crawl are stored in an external Network File System (NFS), you can use the external NFS server to create the persistent volume claim.



1. Create a file named crawler-pv-nfs.yaml with the following content:

apiVersion: v1
kind: PersistentVolume
metadata:
name: <persistent-volume-name>
labels:
pv-name: <persistent-volume-name>
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
nfs:
server: <NFS server hostname or IP address>
path: <Path of NFS exported folder>
<br>Some enrichments are applied to the service by default based on the type of project that you create. For more details, see [Default project settings](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-project-defaults). <br>The version of the Entities enrichment that is available in v2 doesn't include the disambiguation field, which in v1 contains the disambiguation information for the entity and includes the entity subtype information. <br>The following enrichments are not available in v2:<br><br><br><br> * Categories<br> * Concepts<br> * Emotion<br> * Relations<br> * Semantic roles<br> * Sentiment of Entities<br> * Sentiment of Keywords<br><br><br> 
 "normalizations": [ ... ] Moved to the [Collections API](https://cloud.ibm.com/apidocs/discovery-datalistcollections). 
 "source": { ... } Not available. Configure connections to external data sources through the user interface. For more information, see [Creating collections](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collections). 







 Collections 



Collections API support details

 Action v1 API v2 API 

 Create a collection [POST /v1/environments/{environment_id}/collections](https://cloud.ibm.com/apidocs/discoverycreatecollection) [POST /v2/projects/{project_id}/collections](https://cloud.ibm.com/apidocs/discovery-datacreatecollection) <br>The supported parameters and responses differ between the two versions. See the [collection notes](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-migrate-to-v2-apimigrate-to-v2-api-collection-notes).
Contextual entities understand values that you have not explicitly defined. The system makes predictions about additional entity values based on how your user examples are annotated, and uses those values to train other entities. Any similar user examples are added to the Annotation view, so you can see how this option impacts training.

If you do not want your contextual entities to use this expanded understanding of entity values, select all the user examples in the Annotation view for that entity, and then click Delete.



Tutorial: To walk through a tutorial that shows you how to define contextual entities before you add your own, go to [Tutorial: Defining contextual entities](https://www.ibm.com/cloud/architecture/demo/try-watson-assistant-contextual-entities).

The contextual entities tutorial shows a slightly older version of the user interface. But the concepts it covers are the same, so it is still a worthwhile exercise.



 What you don't annotate matters 

If you have an intent example with an annotation, and another word in that example matches the value or a synonym of the same entity, but the value is not annotated, that omission has impact. The model also learns from the context of the term you did not annotate. Therefore, if you label one term as a mention of an entity in a user example, be sure to label any other applicable mentions also.



1. The Customer_Care_Appointments intent includes two intent examples with the word visit.

![Visit examples intent](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/oe-counter1.png)
2. In the second occurrence of the word, you want to annotate the word visit as an entity value of the @meeting entity. This makes visit equivalent to other @meeting entity values such as appointment, as in I'd like to make an appointment or I'd like to schedule a visit.

!
Add entries for the dictionary or upload a file that contains dictionary terms.
6. Go to the Machine Learning Model > Pre-annotation page.
7. Click Run Pre-annotators.
8. Select Dictionaries, and then click Next.
9. If you want to erase existing annotations made by pre-annotators before running the pre-annotator, select Wipe previous pre-annotation results. Human annotations are preserved even if this is selected.
10. Select the check box for each document set that you want to pre-annotate and click Run.

Pre-annotation is applied to individual documents without regard for the various document sets or annotation sets that a document might belong to. A document that overlaps between a selected document set and an unselected document set will be pre-annotated in both document sets.



Related information:



* [Creating dictionaries](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-dictionaries)
* [Getting Started > Adding a dictionary](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks_tutintrowks_tutless4)









 Pre-annotating documents with the machine learning model 

You can use an existing machine learning model to pre-annotate documents that you add to your corpus.



 About this task 

After 10 to 30 documents are annotated, a machine learning model can be trained on the data. Don't use such a minimally trained model in production. However, you can use the model to pre-annotate documents to help speed up the human annotation of subsequent documents. For example, if you add documents to the corpus after you train a machine learning model, you can use the model to pre-annotate the new document sets. Never run a pre-annotator on the same documents that have been annotated by a person. Pre-annotators remove human annotation.





 Procedure
[Application setup](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-caveonix-step4) – Setting up the Service Provider and a Tenant or Organization so that the application becomes accessible for the users.



The automated installation provisions one VM and configures all the application components on that VM.



 Deployment sizing 

The sizing of the deployment is calculated by using the following volumes.



Table 1. Volumes

 Data type Volume 

 Scans per day 1 
 Scan data (MB) 20 
 Log data (MB) 500 
 Flow data (MB) 200 
 Asset data (MB) 46 
 Total storage per asset per day (MB) 766 
 Data replication multiplier 2 
 Total index storage per asset per day (MB) 1,532 



The Data Replication Multiplier is set to 2 as the Index store (Elastic Search) uses by default n+1 replication of the indexes.

The number of scaling VMs is calculated from the number of assets and the number of days of data to index.



Table 2. Scaling VM parameters

 Number of assets 100 500 5000 

 Days of data 30 30 30 
 Total index storage per asset per day (MB) 1532 1532 1532 
 Total index storage per asset per 30 days (TB) 4 22 219 
 Data supported per scaling node (TB) 0 8 8 
 Scaling VMs required 0 3 27 



The following table shows how the amount of storage that is required is calculated.



Table 3. Storage parameters

 Number of assets 100 500 5,000 

 Long-term data retention (months) 8 8 8 
 Total storage per asset per day (MB) 766 766 766 
 Days of data 30 30 30 
 Near-term data retention (TB) 7 33 329 
 Long-term data retention (TB) 18 88 877 



From a data perspective, data is used as follows:



* Scan data is used in compliance management.
* Log data is used in forensic management.
* Policy and flow data is used in risk management. Flow data is available from NSX Manager only.



Data storage has three tiers:



* Replicated
For data volumes attached to a virtual server instance, you can [increase volume capacity](https://cloud.ibm.com/docs/vpc?topic=vpc-expanding-block-storage-volumes) in GB increments up to 16,000 GB capacity, depending on your volume profile. You can also [increase or decrease IOPS](https://cloud.ibm.com/docs/vpc?topic=vpc-adjusting-volume-iops) for a volume that is attached to an instance.

You can also apply user tags and access management tags to your boot and data volumes anytime. Add tags when you create a volume or update an existing volume with the UI, CLI, API and Terraform. For more information, see [Tags for Block Storage for VPC volumes](https://cloud.ibm.com/docs/vpc?topic=vpc-block-storage-about&interface=uivpc-storage-about-tags).

When you create, view, or update a Block Storage for VPC volume, or [restore a volume from a snapshot](https://cloud.ibm.com/docs/vpc?topic=vpc-snapshots-vpc-restore), the volume health state is reported in the UI, CLI, or API. For more information, see [Block Storage for VPC volume health states](https://cloud.ibm.com/docs/vpc?topic=vpc-block-storage-vpc-monitoringblock-storage-vpc-health-states).





 Block Storage for VPC volume types 

Block Storage for VPC offers block-level volumes that are attached to an instance as a boot volume when the instance is created or attached as secondary data volumes. You can configure up to 300 Block Storage for VPC volumes per account in a region. You can request to increase this quota by [opening support case](https://cloud.ibm.com/docs/vpc?topic=vpc-manage-storage-limit) and specifying in which zone you need more volumes.
If you can, limit the number of databases per IBM Cloudant account to 500 or fewer. While this particular number is not magic (IBM Cloudant can safely handle more), several use cases exist that are adversely affected by large numbers of databases in an account.

The replicator scheduler has a limited number of simultaneous replication jobs that it is prepared to run. As the number of databases grows, the replication latency is likely to increase if you try to replicate everything contained in an account.

The flip side of the same coin is the operational aspect: IBM Cloudant’s operations team relies on replication, too, to move around accounts. By keeping down the number of databases, you help us help you if you need to shift your account from one location to another.

So when must you use a single database and distinguish between different document types by using views, and when must you use multiple databases to model your data? IBM Cloudant can’t federate views across multiple databases. If you have unrelated data that can never be “joined” or queried together, then that data can be a candidate for splitting across multiple databases.

If you have an ever-growing data set (like a log, sensor readings, or other types of time-series), it’s also not a good idea to create a single, ever-growing, massive database. This kind of use case requires time-boxing, which we cover in more detail later.





 Avoid the database per user anti-pattern like the plague 

If you’re building a multi-user service atop IBM Cloudant, it is tempting to allow each user store their data in a separate database under the application account. That works well, mostly, if the number of users is small.

Now add the need to derive cross-user analytics. The way that you do that is to replicate all the user databases into a single analytics database. All good. This app has suddenly become successful, and the number of users grew in the range of 150 - 20,000. You have 20,000 replications just to keep the analytics database current. If you also want to run in an active-active disaster recovery setup, add another 20,000 replications, and the system stops functioning.
Number of vSAN capacity disks 

Specify the number of capacity disks that you want to add.

For VMware vSphere 7, order up to 10 disks for Dual CPU models and order up to eight disks for Quad CPU models.





 Size for vSAN cache disks 

Review the Size for vSAN cache disks value.





 Number of vSAN cache disks 

Review the Number of vSAN cache disks value.





 Enable vSAN deduplication and compression 

vSAN storage depends on the number of servers and your total disk capacity, and the use of deduplication and compression.

The amount of storage reduction from deduplication and compression depends on many factors, including the type of data stored and the number of duplicate blocks. Larger disk groups tend to provide a higher deduplication ratio. For more information, see [Using deduplication and compression](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vsan.doc/GUID-3D2D80CC-444E-454E-9B8B-25C3F620EFED.html).





 vSAN license 

Use the IBM-provided VMware license for the vSAN component, which is included with purchase.

Bring Your Own License (BYOL) is no longer supported except for migrations or upgrades of existing BYOL clusters. Select I will provide and enter your own license key only if you are performing an upgrade or migration of an existing BYOL cluster.

If your initial cluster is a vSAN cluster, any additional vSAN clusters use the same vSAN license and have the same configuration as the initial one. This statement is also true for any initial or additional clusters in the instance for which you select vSAN. The first time, you're prompted for the vSAN license and the edition. The next time that you select vSAN for a new cluster, the license that is chosen initially is reused.









 Networking type 

Select Public and private network or Private network only.





 Uplink speed 

The uplink speed provides two options:



* 10 Gb, which is selected by default.
* 25 Gb, which is available only for specific pods and data center locations.
* UI
* CLI
* API







 Scaling Disk, Memory, and CPU 

You can manually adjust the amount of resources available to your IBM Cloud® Messages for RabbitMQ deployment to suit your workload and the size of your data.



 Resource Breakdown 

Messages for RabbitMQ deployments have three data members in a cluster, and resources are allocated to all three members equally. For example, the minimum storage of a RabbitMQ deployment is 3072 MB, which equates to an initial size of 1024 MB per member. The minimum RAM for a RabbitMQ deployment is 3072 MB, which equates to an initial allocation of 1024 MB per member.

Billing is based on the total amount of resources that are allocated to the service.



 Disk Usage 

Storage shows the amount of disk space that is allocated to your service. Each member gets an equal share of the allocated space. Your data is replicated across three data members in the RabbitMQ cluster, so the total amount of storage you use is approximately three times the size of your data set.

Disk allocation affects the performance of the disk, with larger disks having higher performance. Baseline input/output operations per second (IOPS) performance for disk is 10 IOPS for each GB. Reaching IOPS limits consistently causes throughput and message processing delays, which can be alleviated by scaling up disk space.

You cannot scale down storage. If your data set size has decreased, you can recover space by backing up and restoring to a new deployment.





 RAM 

RabbitMQ throttles publishing when it detects it is using 40% of available memory to keep memory usage from growing uncontrollably during spike of activity. If you find that you regularly reach the limit, you can allocate more memory to your deployment. Adding memory to the total allocation adds memory to the members equally.





 Queue Rebalancing 

If you notice that one RabbitMQ node is occupying significantly more resources than another, it is likely that the queues are not evenly distributed between the nodes. This can happen for the following possible reasons:



* You are connected to only one of the VIPs and all the queues are created on a single node.
"field": [ 1, 2, 3 ] "field": [ 1, 2, 3 ] "field": [ 1, 2, 3 ] No difference. 
 "field": [ "v6", "v7", "v8" ] "field": [ "v6", "v7", "v8"] "field": [ "v6", "v7", "v8"] No difference. 











 Verifying that your data was migrated successfully 

To verify that the migration was successful, compare the following metrics to the [metrics that you noted before the migration](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-migrate-to-v2migrate-to-v2-metrics).



* Number of collections

Be sure to re-create all of the collections that you used in v1 and want to keep. With the v2 [List collections](https://cloud.ibm.com/apidocs/discovery-datalistcollections) API method, you can get a list of collections, but you must submit a request per project. You cannot use one call to get the total number of collections per service instance.
* Number of documents per collection

For collections with uploaded data, check the number of documents in the collection by sending an empty query with the [Query a project](https://cloud.ibm.com/apidocs/discovery-dataquery) API method. Specify the collection ID parameter to limit the results to only documents in one collection. An empty query returns all documents. Therefore, you can get the total number of documents from the matching_results value in the response.

The number of documents per collection should be close to the number of documents that were stored in the same collection in v1. The numbers might not be the same.

For crawled data, do not be surprised if the v2 collection has fewer documents. The v1 connectors do not delete documents from a Discovery collection that are deleted from the external data source. Your v2 version of the collection has a fresher crawl of the data as it exists in the external data source today.
For example, "75% of the secondary volumes are under 250 GB and 25% are up to 16,000 GB.
* How many total volumes use customer-managed encryption? For example, "100 volumes (or 50%) are to use customer-managed encryption".
* Provide an estimate of when you expect or plan to provision all of the requested volumes. For example, "I expect to create these volumes within 90 days".
* Provide a 90-day forecast of expected average capacity usage of these volumes. For example, I expect 25% of the volumes to be used in 30 days, 50 percent to be used in 60 days and 75% to be used in 90 days. You might also want to break it down further, by primary and secondary volumes. For example, for secondary volumes, I expect to create 50% of the secondary volumes at less than 250 GB within 30 days and secondary volumes greater than 250 (if possible, estimate volume size) within 60 days.



Respond promptly to all questions and statements in your request. They're necessary for processing and approval. If some things are unclear, there's opportunity to clarify your answers.

You're notified about the update to your limits throughout the case process.
For example, if you have a worker pool with one worker node per three zones (three total worker nodes) and set the minSize to 4 per zone, the cluster autoscaler does not immediately provision an additional three worker nodes per zone (12 worker nodes total). Instead, the scale-up is triggered by resource requests. If you create a workload that requests the resources of 15 worker nodes, the cluster autoscaler scales up the worker pool to meet this request. Now, the minSize means that the cluster autoscaler does not scale down to fewer than four worker nodes per zone even if you remove the workload that requests the amount. For more information, see the [Kubernetes docs](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.mdwhen-does-cluster-autoscaler-change-the-size-of-a-cluster).
* "maxSize": 2: Specify the maximum number of worker nodes per zone that the cluster autoscaler can scale up the worker pool to. The value must be equal to or greater than the value that you set for the minSize.
* "enabled": false: Set the value to true for the cluster autoscaler to manage scaling for the worker pool. Set the value to false to stop the cluster autoscaler from scaling the worker pool. Later, if you want to [remove the cluster autoscaler](https://cloud.ibm.com/docs/containers?topic=containers-cluster-scaling-install-addonautoscaler-remove-console), you must first disable each worker pool in the ConfigMap.



3. Save the configuration file.
4. Get your cluster autoscaler pod.

kubectl get pods -n kube-system
5. Review the Events section of the cluster autoscaler pod for a ConfigUpdated event to verify that the ConfigMap is successfully updated. The event message for your ConfigMap is in the following format: minSize:maxSize:PoolName:<SUCCESS|FAILED>:error message.
Pre-annotating documents 

This tutorial helps you understand how to pre-annotate documents, which bootstraps the annotation process of human annotation.



 Learning objectives 

After you complete this tutorial, you will know how to pre-annotate documents with a machine learning model.

This tutorial should take approximately 5 minutes to finish. If you explore other concepts related to this tutorial, it could take longer to complete.





 Before you begin 



* You're using a supported browser. For information, see [Browser requirements](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-system-requirements).
* You successfully completed [Getting started with Knowledge Studio](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-wks_tutintro), which covers creating a workspace, creating a type system, and adding a dictionary.
* You successfully completed [Creating a machine learning model](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-wks_tutml_intro).
* You must have at least one user ID in either the Admin or Project Manager role. For information about user roles, see [User roles in Knowledge Studio](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-roles).







 Results 

After completing this tutorial, you will have a set of partially annotated documents. Then, you can assign the documents to human annotators to finish the annotation work.





 Lesson 1: Pre-annotating new documents with a machine learning model 

In this lesson, you will learn how to use a machine learning model to pre-annotate documents in Knowledge Studio.



 About this task 

After you train a machine learning model, you can use it to pre-annotate new documents that you add to the corpus.