vania-janet/multiturn-rag-retrieval-data huggingface dataset:
: In addition, the following functionality was reorganized:



* In the previous version, management of dictionaries was included as part of the Pre-annotators page. Now, management of dictionaries is located on the Dictionaries page in the Assets section of the navigation.
* In the previous version, human annotation functionality was distributed across the Mentions, Relations, and Coreferences tabs in the Document Annotation section of the navigation. Now, the functionality is merged under the Annotation Tasks page in the Machine Learning Model section of the navigation.
* To manage human annotation tasks, in the previous version, you found the Tasks tab under the Assets & Tools > Documents page. Now, you add tasks and manage existing tasks on the Annotation Tasks page in the Machine Learning Model section of the navigation.
* In the previous version, the Rules, Regex and Dictionaries pages were separate pages in the Document Annotation section of the navigation. Now, the functionality is merged under the Rules page in the Rule-based Model section of the navigation.



For more details about the navigation changes, see Figure 1 and Table 3.

![Screen captures of the previous navigation (left side) and new navigation (right side).](https://cloud.ibm.com/docs-content/v1/content/50eddb8fd81f33092880335ff107a78ff5cd0f65/watson-knowledge-studio/images/nav3-0718.svg) Figure 1. Screen captures of the previous navigation (left side) and new navigation (right side).



Table 3. Navigation changes (July 2018)

 Feature Previous location Current location 

 Annotation tasks Assets & Tools > Documents > Tasks Machine Learning Model > Annotation Tasks 
 Coreferences tab Document Annotation Machine Learning Model > Annotation Tasks > task > annotation set > document 
 Dictionaries page (management) Assets & Tools > Pre-annotators > Manage Dictionaries Assets 
 Dictionaries tab (mapping to classes for rule-based model) Document Annotation Rule-based Model > Rules 
 Documents page Assets & Tools Assets 
 Entity Types page Assets & Tools Assets
You must create or upload a type system before you begin any annotation tasks.



 About this task 

For more information about type systems, see [Type systems](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-typesystemwks_typesystem).





 Procedure 



1. Download the [en-klue2-types.json ![External link icon](https://cloud.ibm.com/docs-content/v1/content/50eddb8fd81f33092880335ff107a78ff5cd0f65/icons/launch-glyph.svg)](https://watson-developer-cloud.github.io/doc-tutorial-downloads/knowledge-studio/en-klue2-types.json) file to your computer. This file contains an example KLUE type system.
2. Click Assets> Entity Types.
3. On the Entity Types page, click Upload.
4. Upload the en-klue2-types.json file from your computer. The uploaded type system is displayed in the table.
5. Browse the type system so you can see the data that was uploaded.
6. Edit an entity type:



1. Locate the MONEY entity type.
2. Double-click anywhere in the table row to edit the entity type.
3. In the Roles column, click the Delete a role icon ![The "Delete a role" icon.](images/wks_delete.jpg "The "Delete a role" icon.") next to the AWARD role.

![A screen capture of deleting a role from an entity type.](images/wks_tut_delete_role.png "A screen capture that shows the cursor hovering over the "Delete a role" icon.")
4. Click Save.









 What to do next
Delete the document as described in [Deleting a document that has not been associated with an annotation task](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-documents-for-annotationdeletenotask).

You can confirm that the annotations on the remaining documents are not lost by downloading the document sets and reviewing the documents in the gt folder.











 Data model 

The diagrams in this topic summarize the flow of documents in a Knowledge Studio system and the differences between documents in the corpus, an annotation task, and ground truth.

The corpus contains documents, which are partitioned into document sets:



* A document is nothing more than strings of text.
* A document set is a pointer to a group of documents. The document set does not contain copies of the documents themselves.
* Some document sets can point to a single document, a setup that you can control through the overlap parameter that you specify when you create annotation sets.



![This figure illustrates two document sets that point to three documents. The documents are divided between the sets.](https://cloud.ibm.com/docs-content/v1/content/148d3bd95f946aa1bb53ea1540475f522e6b61c9/watson-knowledge-studio-data/images/wks_datacorpus.svg) Figure 1. This figure illustrates two document sets that point to three documents. The documents are divided between the sets.

Ground truth comprises the annotations (mentions, relations, and coreferenced mentions) that are added to documents. Ground truth is singular for each document.

![This figures illustrates that ground truth consists of the annotations that are added to document 1, document 2, document 3, and so on.]
: Required: Annotation to specify a service that accepts private requests.

service.kubernetes.io/ibm-load-balancer-cloud-provider-vpc-subnets
: Required: Annotation to specify the dedicated subnet that the VPC NLB deploys to. The value can be specified as a VPC subnet ID, VPC subnet name, or VPC subnet CIDR. You must specify only one subnet. The subnet must exist in the same VPC as your cluster and in a zone where your cluster has worker nodes, but no worker nodes can be attached to this subnet. The worker nodes that exist in the same zone as this subnet are configured to receive traffic from the VPC NLB. To see subnets in all resource groups, run ibmcloud oc subnets --provider vpc-gen2 --vpc-id <vpc> --zone <zone>.

service.kubernetes.io/ibm-load-balancer-cloud-provider-vpc-node-selector
: Optional: Annotation to specify a worker node label selector. Within the same zone as the dedicated subnet for the VPC NLB, you can configure specific worker nodes to receive traffic by selecting one of the supported label selector keys. Note that you can include only one label selector in the annotation, and that the selector must be specified in the "key=value" format. If this annotation is not specified, all worker nodes in the same zone as the VPC subnet that you specified in the service.kubernetes.io/ibm-load-balancer-cloud-provider-vpc-subnets annotation are configured to receive traffic from the VPC NLB. If specified, any dedicated: edge labels on worker nodes are ignored.
: The following keys are permitted.



* ibm-cloud.kubernetes.io/internal-ip
* ibm-cloud.kubernetes.io/machine-type
* ibm-cloud.kubernetes.io/os
* ibm-cloud.kubernetes.io/region
* ibm-cloud.kubernetes.io/subnet-id
1.2.8 Ensure that the --authorization-mode argument includes Node Scored 1 Pass IBM 
 1.2.9 Ensure that the --authorization-mode argument includes RBAC Scored 1 Pass IBM 
 1.2.10 Ensure that the admission control plug-in EventRateLimit is set Not Scored 1 [Fail](https://cloud.ibm.com/docs/openshift?topic=openshift-cis-benchmark-411ibm-remediations-and-explanations-411) IBM 
 1.2.11 Ensure that the admission control plug-in AlwaysAdmit is not set Scored 1 Pass IBM 
 1.2.12 Ensure that the admission control plug-in AlwaysPullImages is set Not Scored 1 [Fail](https://cloud.ibm.com/docs/openshift?topic=openshift-cis-benchmark-411ibm-remediations-and-explanations-411) IBM 
 1.2.13 Ensure that the admission control plug-in SecurityContextDeny is set if PodSecurityPolicy is not used Not Scored 1 [Pass](https://cloud.ibm.com/docs/openshift?topic=openshift-cis-benchmark-411ibm-remediations-and-explanations-411) IBM 
 1.2.14 Ensure that the admission control plug-in ServiceAccount is set Scored 1 Pass IBM 
 1.2.15 Ensure that the admission control plug-in NamespaceLifecycle is set Scored 1 Pass IBM 
 1.2.16 Ensure that the admission control plug-in PodSecurityPolicy is set Scored 1 [Pass](https://cloud.ibm.com/docs/openshift?topic=openshift-cis-benchmark-411ibm-remediations-and-explanations-411) IBM 
 1.2.17 Ensure that the admission control plug-in NodeRestriction is set Scored 1 Pass IBM 
 1.2.18 Ensure that the --insecure-bind-address argument is not set. Scored 1 Pass IBM 
 1.2.19 Ensure that the --insecure-port argument is set to 0. Scored 1 Pass IBM
Human annotations are preserved even if this is selected.
10. Select the check box for each document set that you want to pre-annotate and click Run.

Pre-annotation is applied to individual documents without regard for the various document sets or annotation sets that a document might belong to. A document that overlaps between a selected document set and an unselected document set will be pre-annotated in both document sets.



Related information:

[Creating dictionaries](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-dictionaries)

[Getting Started > Adding a dictionary](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-wks_tutintrowks_tutless4)







 Pre-annotating documents with the machine learning model 

You can use an existing machine learning model to pre-annotate documents that you add to your corpus.



 About this task 

After 10 to 30 documents are annotated, a machine learning model can be trained on the data. Don't use such a minimally trained model in a production. However, you can use the model to pre-annotate documents to help speed up the human annotation of subsequent documents. For example, if you add documents to the corpus after you train a machine learning model, you can use the model to pre-annotate the new document sets. Never run a pre-annotator on the same documents that have been annotated by a person. Pre-annotators remove human annotation.





 Procedure 

To use an existing machine learning model to pre-annotate documents:



1. Log in as a Knowledge Studio administrator and select your workspace.
2. Go to the Machine Learning Model > Pre-annotation page.
3. Click Run Pre-annotators.
4. Select Machine Learning Model, and then click Next.
5.
CA-7 (1) - Independent Assessment 



  Control requirements 

CA-7 (1) - 0
:   The organization employs assessors or assessment teams with [Assignment: organization-defined level of independence] to monitor the security controls in the information system on an ongoing basis.





  NIST supplemental guidance 

Organizations can maximize the value of assessments of security controls during the continuous monitoring process by requiring that such assessments be conducted by assessors or assessment teams with appropriate levels of independence based on continuous monitoring strategies. Assessor independence provides a degree of impartiality to the monitoring process. To achieve such impartiality, assessors should not: (i) create a mutual or conflicting interest with the organizations where the assessments are being conducted; (ii) assess their own work; (iii) act as management or employees of the organizations they are serving; or (iv) place themselves in advocacy positions for the organizations acquiring their services.
If there are any other fields that you want to enrich that are not yet listed in Fields to be enriched, enter them in Add a field to enrich, and add the enrichments.
3. Click Apply changes to collection.







 How to annotate a document 

See [Best practices for annotating documents](https://cloud.ibm.com/docs/discovery?topic=discovery-sdubestpractices) before you begin annotating.



1. Review the field labels that you can use to annotate the document. They are displayed in the Field labels panel.

See the Default field labels table for a list of the fields and their descriptions.
2. To create a custom field label, click Create new.

The field label must consist of all lowercase letters with no spaces. For example, my_field is a valid field label.

The number of custom fields you can add varies by plan type:



* Lite: 0
* Advanced: 10
* Premium: 100



3. Click a field label from the list to activate it.
4. Click the content representing that field in the SDU editor. The content highlights with the corresponding field label color.



* Alternately, you can select a field label, and then drag it to the content in the SDU editor.
* To clear a change, click the Clear change button on the toolbar.



5. Click Submit page.

As you annotate, Watson is learning and starts to predict annotations. Continue annotating, until Watson correctly and consistently identifies fields.
6. After you finish annotating, click Apply changes to collection. The Upload your documents screen opens. Re-upload the documents in your collection. After uploading is complete, you are redirected to the Overview screen.





Default field labels

 Field Definition 

 answer In a Q/A pair (often in an FAQ), the answer to the question. 
 author Name of author (or authors). 
 footer Use this tag to denote meta-information about the document (such as the page number or references), that appear at the bottom of the
The crawler plug-in can apply more nuanced rules to what documents and what fields in the documents get added. For more information, see [Building a Cloud Pak for Data custom crawler plug-in](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-crawler-plugin-build).



 Data source requirements 

The following requirements and limitations are specific to IBM Watson® Discovery:



* The individual file size limit is 32 MB per file, which includes compressed archive files (ZIP, CZIP, TAR). When decompressed, the individual files within compressed files cannot exceed 32 MB per file. This limit is the same for collections in which you upload your own data.
* Depending on the type of installation (starter or production mode), the number of collections you can ingest simultaneously varies. A starter installation includes one crawler pod, which allows three collections to be processed simultaneously. A production installation includes two crawler pods, which can process six collections simultaneously.

If you are running a starter installation and you want to process more than three collections simultaneously, you must increase the number of crawler pods by running the following commands:

oc patch wd wd --type=merge --patch='{"spec": {"ingestion": {"crawler": {"replicas": <number-of-replicas> } } } }'

In a starter installation, the maximum number of simultaneous collections that can crawl an external data source is 3. If you start a fourth, that collection does not start to process until the prior three crawls finish.

Each number-of-replicas allows 3 simultaneous crawls, so number-of-replicas=2 increases the replicas to 6, and number-of -replicas=3 increases them to 9.







 Crawler plug-in settings 

When you deploy one or more crawler plug-ins, you can configure your collection to use one of the plug-ins.

These settings are only available when crawler plug-ins are deployed.
If your application relies on the previous behavior, specify a version number earlier than 2023-03-31, such as 2020-08-30, in your API calls.





 2 March 2023 

Now you can specify the types of files to add to a collection
: When you connect to an external data source, you can limit the types of files to add to the collection from the external data source. For example, you can choose to add only PDF files from a Box data source.





 21 Febuary 2023 

Optical character recognition v2 technology is used
: The latest version (OCR v2) is used automatically when you enable OCR for English, German, French, Spanish, Dutch, Brazilian Portuguese, and Hebrew collections in all IBM Cloud service plans.

The new optical character recognition model was developed by IBM Research to be better at extracting text from scanned documents and other images that have the following limitations:



* Low quality images due to incorrect scanner settings, insufficient resolution, bad lighting (such as with mobile capture), loss of focus, unaligned pages, and badly printed documents
* Documents with irregular fonts or a variety of colors, font sizes, and backgrounds



The entity extractor limits changed
: The number of documents that are allowed in the training data for the Plus plan increased from 100 to 200.

The number of entity types that you can create per plan decreased.



* For Premium plans, the limit changed from 75 to 18.
* For Enterprise plans, the limit changed from 50 to 18.
* For Plus plans, the limit changed from 20 to 12.



The string variation operator now works with phrases
: When you include the string variation operator with query input that contains a phrase, the variation is applied to each word in the phrase. For example, "tom cat"1 matches top hat in addition to tom cat. For more information about Discovery Query Language operators, see [Query operators](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-operators).





 10 Febuary 2023 

Entity extractor is generally available
: The Extract entities enrichment brings the powerful ability to build a custom type system into Discovery.
However, you can use the model to pre-annotate documents to help speed up the human annotation of subsequent documents. For example, if you add documents to the corpus after you train a machine learning model, you can use the model to pre-annotate the new document sets. Never run a pre-annotator on the same documents that have been annotated by a person. Pre-annotators remove human annotation.





 Procedure 

To use an existing machine learning model to pre-annotate documents:



1. Log in as a Knowledge Studio administrator and select your workspace.
2. Go to the Machine Learning Model > Pre-annotation page.
3. Click Run Pre-annotators.
4. Select Machine Learning Model, and then click Next.
5. If you want to erase existing annotations made by pre-annotators before running the pre-annotator, select Wipe previous pre-annotation results. Human annotations are preserved even if this is selected.
6. Select the check box for each document set that you want to pre-annotate and click Run.

Pre-annotation is applied to individual documents without regard for the various document sets or annotation sets that a document might belong to. A document that overlaps between a selected document set and an unselected document set will be pre-annotated in both document sets.









 Pre-annotating documents with the rule-based model 

You can use an existing rule-based model to pre-annotate documents that you add to your corpus.



 Procedure 

To use the rule-based model to pre-annotate documents, complete the following steps:



1. Log in as a Knowledge Studio administrator and select your workspace.
2. Go to the Machine Learning Model > Pre-annotation page.
3. Click the overflow menu button in the Rule-based Model row in the page, then click Map entity types and classes to map entity types that you defined in the Knowledge Studio type system to one or more rule-based model classes.
Because the test documents are used iteratively to improve the model, they can start to influence the model training indirectly. That's why the blind set of documents is so important. The blind set gives you a way to generate an unbiased evaluation of the model periodically.



The default split applies a ratio (70%-23%-7%) that is commonly used for machine learning training.



2. Click Train extractor.



When you train the extractor, Discovery uses documents from the training set to build a machine learning model. After the model is generated, it runs a test against the documents from the test set automatically. The results of the test are displayed for you to review.



 Troubleshoot training issues 

Learn about possible error messages and how to address them.

The training data is too large
: Your training data contains large text document or many entity types and resources that are needed to process the data is greater than the resources that are available to your service instance. This error can occur even when your workspace doesn't exceed the documented entity extractor limits. To resolve the issue, you can try one of the following approaches:



* Remove one or more entity types to decrease the size of your training data.
* Remove extra large documents from the training data. For example, if one of the labeled documents is extra large, change its status from Completed to In progress to omit it from the training data.
* Reduce the number of documents that are included in the training set. The default split ratio (70%-23%-7%) for the training data uses 70% of the documents in the training set. You can change the percentage of documents that are used in the training set to a smaller number. For example, you might change the split ratio to 60%-33%-7%.
IBM Cloud Pak for Data

Increase the capacity of your deployed service instance by scaling up service pods.









 Evaluating the extractor 

To review metrics from the test run of the entity extractor model that you created, click the Evalute extractor tab.

The following table describes the available evaluation metrics.



Metrics details

 Metric Description 

 Confusion matrix A table that provides a detailed numeric breakdown of annotated document sets.
Cloud Pak for Data Unlimited 
 Premium 100 
 Enterprise 100 
 Plus (includes Trial) 40 



The maximum number of documents that you can annotate to train an SDU model per collection depends on your Discovery plan type.



Training set limits

 Plan Documents per collection 

 Cloud Pak for Data 40 
 Premium 40 
 Enterprise 40 
 Plus (includes Trial) 40 







 Managing fields 

The Manage fields tab contains several options:

Identify fields to index
: For more information, see [Excluding content from query results](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-hide-data).

Improve query results by splitting your documents
: For more information, see [Split documents to make query results more succinct](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-split-documents).

Date format settings
: For more information, see [Date format settings](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-index-overviewfield-date-settings).

To access the Manage fields page, click the Manage collections icon on the navigation panel and open a collection. Click the Manage fields tab. For more information about collections, see [Creating collections](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collections).
Enrichments that you enable are applied to the documents in random order. For more information, see [Managing enrichments](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-manage-enrichments).

From the deployed Content Mining application, you can create a classifier or a custom annotator from a dictionary, regular expression, machine learning, or PEAR file and use it as an enrichment in collections that are stored in other project types. For more information, see [Adding facets](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-add-facets).