vania-janet/multiturn-rag-retrieval-data huggingface dataset:
If a dictionary that is first in the order labels IBM as an Organization entity type, a machine learning model that is second in the order can't annotate IBM Watson as a Software Brand entity type because that would override the earlier annotation made to IBM.

You can view the current order of pre-annotators in the Order column on the Machine Learning Model > Pre-annotation page. To change the order, complete the following steps.



1. Click Order Settings.
2. Click the Move up and Move down arrow** buttons to move pre-annotation methods earlier or later in the order.
3. Click Save.
4. Double check the Order column on the Pre-annotation page to make sure that it matches the order that you want.







 Run pre-annotators 



1. After your pre-annotation methods are prepared and you have configured the order of your pre-annotators, click Run Pre-annotators.
2. Select the pre-annotators that you want to use, and then click Next.
3. If you want to erase existing annotations made by pre-annotators before running the pre-annotator, select Wipe previous pre-annotation results. Human annotations are preserved even if this is selected.
4. Select the document sets that you want to pre-annotate.
5. Click Run.









 Pre-annotating documents with a dictionary 

To help human annotators get started with their annotation tasks, you can create a dictionary and use it to pre-annotate documents that you add to the corpus.



 About this task 

When a human annotator begins work on documents that were pre-annotated, it is likely that a number of mentions will already be marked by entity types based on the dictionary entries. The human annotator can change or remove the pre-annotated entity types and assign entity types to unannotated mentions. Pre-annotation by a dictionary does not annotate relations, coreferences. Relations and coreferences must be annotated by human annotators.
[Data collection](https://cloud.ibm.com/docs-content/v1/content/5eb88a2575e30e316dbb83829c7f4c8b1ddbc67c/devsecops/images/data-collection.svg)

Figure 1. Data collection

By default, the sample application in the CI toolchain attempts to set the minimum number of reviewers to 1. If you want to change the number of reviewers, set the peer_review_approvers envronment property as required. For more information on setting the minimum number of reviewers required for a pull/merge request, see the following GitHub and GitLab resources:



* [GitHub: Require multiple reviewers for pull requests](https://github.blog/2018-03-23-require-multiple-reviewers/)
* [GitLab: Merge request approval rules](https://docs.gitlab.com/ee/user/project/merge_requests/approvals/rules.html)





 Data collected in continuous integration build runs 

This data collection contains a list of all of the commits for the pull/merge requests that were merged in app repos since the last build.

Pull request data is collected directly from the app repos. Data for each pull/merge request that is related to commits between the repo commit that triggered the previous build and the currently available commit is collected.

Commits that do not contain a pull/merge request create a compliance incident issue in the following releases of the pipeline. You cannot commit directly to the master branch.

A compliance incident typically holds the following information:



* List of pull/merge request URLs for the associated commit ID.
* Application repository.
* Commit ID.
* Required number of approvals.



Zoom

![Pull Request incident content](https://cloud.ibm.com/docs-content/v1/content/5eb88a2575e30e316dbb83829c7f4c8b1ddbc67c/devsecops/images/devsecops-pr-incident-issue.png)

Figure 2.
If you have a large number of entities, you might find it easier to import them from a comma-separated value (CSV) file than to define them one by one.

Entity annotations are not included in the import of an entity CSV file. You must import the entire dialog skill to retain the associated annotations for a contextual entity in that skill. If you export and import entities only, then any contextual entities that you exported are treated as dictionary-based entities after you import them.



1. Collect the entities into a CSV file, or export them from a spreadsheet to a CSV file. The required format for each line in the file is as follows:

<entity>,<value>,<synonyms>

where <entity> is the name of an entity, <value> is a value for the entity, and <synonyms> is a comma-separated list of synonyms for that value.

weekday,Monday,Mon
weekday,Tuesday,Tue,Tues
weekday,Wednesday,Wed
weekday,Thursday,Thur,Thu,Thurs
weekday,Friday,Fri
weekday,Saturday,Sat
weekday,Sunday,Sun
month,January,Jan
month,February,Feb
month,March,Mar
month,April,Apr
month,May

Importing a CSV file also supports patterns. Any string wrapped with / will be considered a pattern (as opposed to a synonym).

ContactInfo,localPhone,/(d{3})-(d{4})/
ContactInfo,fullUSphone,/(d{3})-(d{3})-(d{4})/
ContactInfo,internationalPhone,/^((?+?[0-9])?)?[0-9_- ()]$/
ContactInfo,email,/b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+.[A-Za-z]{2,}b/
The number of documents per collection should be close to the number of documents that were stored in the same collection in v1. The numbers might not be the same.

For crawled data, do not be surprised if the v2 collection has fewer documents. The v1 connectors do not delete documents from a Discovery collection that are deleted from the external data source. Your v2 version of the collection has a fresher crawl of the data as it exists in the external data source today.



Do not expect the search results to be the same for queries that you submit in the v1 and v2 instances.





 Using a news service with v2 

If you used the Watson Discovery News data source in v1 and want to create a data source with equivalent function in v2, find a news and events data provider service. Look for a service that offers a News API that extracts news articles in JSON format. You can then upload the JSON files to create a News collection in your v2 project.





 Delete your v1 service instance 

After your data is migrated and your applications are updated to use the new v2 service instance, be sure to delete your v1 service instance. You are charged for the v1 service instance until you delete it. For more information, see [Deleting a managed service instance](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-delete).
After you create a dictionary, you can use it to speed up human annotation tasks by pre-annotating the documents.







 Next steps 

Learn how to [create a machine learning model](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks_tutml_intro).





 Tutorial summary 

While learning about Knowledge Studio, you created a workspace and added artifacts to it.



 Lessons learned 

By completing this tutorial, you learned about the following concepts:



* Workspaces
* Type systems
* Dictionaries
Human annotations are preserved even if this is selected.
4. Select the document sets that you want to pre-annotate.
5. Click Run.









 Pre-annotating documents with Natural Language Understanding 

You can use the Natural Language Understanding service to pre-annotate documents that you add to your corpus.



 Before you begin 

Determine whether the Natural Language Understanding pre-annotator is likely to add value for your use case. Review the list of supported [Natural Language Understanding service entity types and subtypes](https://cloud.ibm.com/docs/natural-language-understanding?topic=natural-language-understanding-entity-types) to determine if there is a natural overlap between them and the types in your type system. If so, continue with this procedure. If not, choose a different pre-annotator to use.





 About this task 

Natural Language Understanding is a service that offers text analysis through natural language processing. When you use the Natural Language Understanding pre-annotator, it calls the Natural Language Understanding service to find and annotate entities in your documents.

You must specify the entity types that you want the service to look for by mapping the Natural Language Understanding entity types to corresponding Knowledge Studio entity types that you have added to the Knowledge Studio type system. Only mentions of entity types that you map will be found and annotated.





 Procedure 

To use the Natural Language Understanding service to pre-annotate documents, complete the following steps:



1. Log in as a Knowledge Studio administrator and select your workspace.
2. Go to the Machine Learning Model > Pre-annotation page.
3. Click the overflow menu button in the Natural Language Understanding row, and then click Map entity types.



* The drop-down list of the Natural Language Understanding entity types is pre-populated with entity types that are recognized by the Natural Language Understanding service.
* You must map at least one entity type.
* You cannot map an Natural Language Understanding entity type to a Knowledge Studio entity role, only Knowledge Studio entity types.
* You can map more than one Natural Language Understanding entity type to a single Knowledge Studio entity type, or the other way around.
* UI
* CLI
* API







 Scaling Disk, Memory, and CPU 

You can manually adjust the amount of resources available to your Databases for MongoDB deployment to suit your workload and the size of your data.



 Resource Breakdown 

Databases for MongoDB runs with three data members in a cluster, and resources are allocated to all members equally. For example, the minimum disk size of a MongoDB deployment is 30720 MB, which equates to an initial size of 10240 MB per member. Minimum RAM for a MongoDB deployment is 3072 MB, which equates to an initial allocation of 1024 MB per member.

Billing is based on the total amount of resources that are allocated to the service.

When you provision a deployment, you can select the initial resource allocation of disk and memory. After provision, you can scale your deployment as it needs more resources.



 Disk Usage 

Your disk allocation must be enough to store all of your data. Your data is replicated to both data members so the total amount of disk that you use is at least twice the size of your data set.

Disk allocation also affects the performance of the disk, with larger disks having higher performance. Baseline Input-Output Operations per second (IOPS) performance for disk is 10 IOPS for each GB. Scale disk to increase the IOPS that your deployment can handle.

You cannot scale down storage. If your data set size has decreased, you can recover space by backing up and restoring to a new deployment.





 RAM 

Memory resources are used for database operations and also control the amount of memory that is allocated to the [internal and file system cache](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-high-availability). If your database can serve most of the requests from the cache, then it doesn't have to read from disk and performs better.

The amount of memory you allocate to your deployment is split between all members. Adding memory to the total allocation adds memory to all members equally.





 Dedicated Cores 

You can enable or increase the CPU allocation to the deployment.
In v2, enrichments are associated with the collection configuration. Some enrichments are applied to your collection by default, depending on the type of project used. For more information, see [Default project settings](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-project-defaults). In v2, you can configure the collection to use any subset of available enrichments on the fields of your document.





 Retaining document IDs 

Document IDs are assigned to the documents that you add to a v2 collection when you upload them from the product user interface or add them by using the [Add a document](https://cloud.ibm.com/apidocs/discovery-dataadddocument) API method.

You might want to retain the IDs of your v1 documents in v2 if you are using processes that depend on these unique identifiers. For example, regression testing for the application might verify that specific documents are returned by checking the document IDs. Relevancy training uses the document IDs to track documents between training runs. These processes are easier to adapt if the document IDs are the same between your v1 and v2 instances. Otherwise, the processes that are used with the Discovery v1 instance must be remapped to the IDs that are assigned to the documents after they are added to the Discovery v2 instance.

If you specified your own documents IDs when you added documents to the v1 service instance, you can retain the IDs by using the Update a document method instead of the Add a document method. With the update method, you can assign a document ID to the document as you add it to the v2 collection. For more information, see [Update a document](https://cloud.ibm.com/apidocs/discovery-dataupdatedocument).

If your data is stored in a JSON file, an array in the original document generates a document ID with a number appended to it. For example, original_id_n. To retain the original document ID without the number suffix, remove the array in the JSON file.
* The patterns must be defined in an array, even if you plan to define only one pattern.
* Escape any backslash () characters with a backslash.
* For more information about the facet value options, see the Regular expression facet value options table.



2. Click Import, and then choose the JSON file where the patterns are defined.
3. Click Save.





 Regular expression limits 



Regular expression plan limits

 Plan Regex enrichments per service instance Regex patterns per service instance 

 Cloud Pak for Data Unlimited Unlimited 
 Premium 100 50 
 Enterprise 100 50 



Totals include enrichments that you create in this Content Mining project and in other projects in the same service instance.







 Applying the annotator 

After the annotator is created, you must apply it to your collection.



1. From the Create a custom annotator for your analytics solutions page of the Content Mining application, click custom annotator, and then select collection from the list.
2. In the tile for your collection, click the options icon, and choose Edit collection.

Zoom

![Collection tile overflow menu](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/cm-edit-colxn-icon.png)

Figure 1. Collection menu
3. Click the Enrichment tab, and then select the annotator that you created.

You might need to scroll to find it.
4. Click Save, and then confirm the action.



Give the index time to rebuild.





 Filtering documents with your facet 



1. Click the collection tile to open your collection in the data analysis page.
2. Do one of the following things:



* Your custom facets are listed in the Facets view. Scroll and click Load more repeatedly until your facets are displayed.
* Submit an empty search to return all documents. In the Facet analysis pane, select the facet that you created.
This practice is somewhat like normalizing your data. There's a balance to strike between reducing the number of HTTP requests and avoiding conflicts.

For example, see the medical record that includes a list of operations:

{
"_id": "Joe McIllness",
"operations": [
{ "surgery": "heart bypass" },
{ "surgery": "lumbar puncture" }
]
}

If Joe is unfortunate enough to have lots of operations at the same time, the concurrent updates to a document are likely to create conflicted documents. Better to break out the operations into separate documents, which refer to Joe's person document, and use a view to connect things together. To represent each operation, you’d upload documents like the following two examples:

{
"type": "operation",
"patient": "Joe McIllness",
"surgery": "heart bypass"
}

{
"type": "operation",
"patient": "Joe McIllness",
"surgery": "lumbar puncture"
}

Emitting the "patient" field as the key in your view would then allow querying for all operations for a specific patient. Again, views are used to help knit together a full picture of a specific entity from separate documents. Views help keep the number of HTTP requests low, even though IBM Cloudant splits up the data for a single-modeled entity.
* How do I avoid conflicts?

Avoiding conflicted documents helps speed up many operations on your IBM Cloudant databases. There’s a process that works out the current winning revision used each time that the document is read, for example, single document retrievals, calls with include_docs=true, view building, and so on.

The winning revision is a particular revision from the document’s overall tree. Recall that documents on IBM Cloudant are in fact trees of revisions. An arbitrary but deterministic algorithm selects one of the non-deleted leaves of this tree to return when a request is made for the document.
Scaling per Member 

Databases for Elasticsearch deployments have minimum and maximum allocation for disk and RAM as shown. Scaling deployments through the API/CLI provides more granularity and also allows a user to scale a database instance up to 4 TB of disk per member.



Table 1. Per Member Scaling Limits

 Resource Minimum Maximum Scaling Granularity (API/CLI) 

 Disk 5 GB per member 4 TB per member 1024 MB per member 
 RAM 1 GB per member 112 GB per member 128 MB per member 
 CPU (if enabled) 3 CPUs per member 28 CPUs per member 1 CPU per member
If bulk labeling is enabled, a notification is displayed to show the number of occurrences of the term that were found and labeled in the current document.
5. If you want to label occurrences of the term in all of the documents in the collection, click Apply to all documents.

When you enable this option, occurrences of the term are labeled in all of the documents in the collection, including documents that you already reviewed and marked complete.

Zoom

![Shows the notification that asks whether to apply bulk labeling to all documents.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/ee-bulk-label-all-docs.png)

Figure 5. Bulk labeling configuration

You are asked to confirm the action because it cannot be undone. If you don't want to have to confirm the action every time you choose to apply bulk labeling to all documents, select Do not ask for confirmation again. Click Run.

Zoom

![Shows the bulk labeling confirmation dialog box.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/ee-bulk-label-confirmation.png)

Figure 6. Bulk labeling configuration confirmation

For more information, see [Labeling examples in bulk](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-entity-extractorentity-extractor-bulk-label).
6. Scroll through the document to label every valid example of every entity type that you want your extractor to recognize.

The machine learning model learns as much from the terms that you don't label as the terms that you do.

If you miss labeling a valid example, the model learns that when the term is used in that context, it is not a valid mention of the entity type. In some cases, an omission is appropriate.
The volume exists on the account until you delete the volume or you reach the end of a billing cycle, whichever comes first.

Pricing is also affected when you [expand volume capacity](https://cloud.ibm.com/docs/vpc?topic=vpc-expanding-block-storage-volumes) or [adjust IOPS](https://cloud.ibm.com/docs/vpc?topic=vpc-adjusting-volume-iops) by specifying a different IOPS profile. For example, expanding volume capacity increases costs, and changing an IOPS profile from a 5-IOPS/GB tier to a 3-IOPS/GB tier decreases the monthly and hourly rate. Billing for an updated volume is automatically updated to add the prorated difference of the new price to the current billing cycle. The new full amount is then billed in the next billing cycle.

Pricing for Block Storage for VPC volumes is also set by region. For more information, see [Pricing](https://www.ibm.com/cloud/vpc/pricing).





 Are there limits on the number of volumes I can create? 

You can create up to 300 total Block Storage for VPC volumes (data and boot) per account in a region. To increase this [quota](https://cloud.ibm.com/docs/vpc?topic=vpc-quotasblock-storage-quotas), open a [support case](https://cloud.ibm.com/docs/vpc?topic=vpc-getting-help) and specify the zone where you need more volumes.





 After a data volume is created with a specific capacity, can the capacity later be increased? 

You can increase the capacity of data volumes that are attached to a virtual server instance. You can indicate capacity in GB increments up to 16,000 GB capacity, depending on your volume profile. For more information, see [Increasing Block Storage for VPC volume capacity](https://cloud.ibm.com/docs/vpc?topic=vpc-expanding-block-storage-volumes).





 Can I increase capacity of a boot volume? 

Boot volume capacity can be increased during instance provisioning or later, by directly modifying the boot volume.
This practice is somewhat like normalizing your data. There's a balance to strike between reducing the number of HTTP requests and avoiding conflicts.

For example, see the medical record that includes a list of operations:

{
"_id": "Joe McIllness",
"operations": [
{ "surgery": "heart bypass" },
{ "surgery": "lumbar puncture" }
]
}

If Joe is unfortunate enough to have lots of operations at the same time, the concurrent updates to a document are likely to create conflicted documents. Better to break out the operations into separate documents, which refer to Joe's person document, and use a view to connect things together. To represent each operation, you’d upload documents like the following two examples:

{
"type": "operation",
"patient": "Joe McIllness",
"surgery": "heart bypass"
}

{
"type": "operation",
"patient": "Joe McIllness",
"surgery": "lumbar puncture"
}

Emitting the "patient" field as the key in your view would then allow querying for all operations for a specific patient. Again, views are used to help knit together a full picture of a specific entity from separate documents. Views help keep the number of HTTP requests low, even though IBM Cloudant splits up the data for a single-modeled entity.





 How do I avoid conflicts? 

Avoiding conflicted documents helps speed up many operations on your IBM Cloudant databases. There’s a process that works out the current winning revision used each time that the document is read, for example, single document retrievals, calls with include_docs=true, view building, and so on.

The winning revision is a particular revision from the document’s overall tree. Recall that documents on IBM Cloudant are in fact trees of revisions. An arbitrary but deterministic algorithm selects one of the non-deleted leaves of this tree to return when a request is made for the document.