vania-janet/multiturn-rag-retrieval-data huggingface dataset:
If annotation guidelines were previously connected to the project, and you want help with choosing the correct annotation to apply, click View Guidelines. Depending on the access permissions set up on the site where the guidelines are hosted, you might be able to update the guidelines after you open them, for example, to add clarifications and examples.
10. To apply a different entity type to a mention, reject the current annotation, select the mention, such as Barack Obama, and then select the correct entity type, such as Person.
11. Continue accepting, rejecting, and revising other conflicts in the sentence. After you resolve all conflicts in a sentence, click the Select all annotations link and then click Accept.
12. Click the arrow icon to go to the next sentence. Continue until all mention conflicts in the document are resolved.

To save your work in progress, or to temporarily suspend the current adjudication session, click Save at any time. If you want to discard all changes that you made, click Discard. If you saved the previous adjudication session, and you are ready to resume adjudication, click Resume to start adjudicating conflicts where you last stopped. If you discarded the previous adjudication session, you must start a new adjudication session.
13. After resolving mention conflicts, switch the adjudication mode to resolve any conflicts that occur between relation annotations and coreference chain annotations.



* Resolving conflicts in relation mode is similar to how you resolve conflicts for mentions. You resolve conflicts on a sentence-by-sentence basis.
* Coreference chains can exist across multiple sentences. When you move to coreference mode, the system lists the coreference chains that were created by each human annotator in a pane on the right. Select the chains that you want to adjudicate and then click Reject Chain or Accept Chain to reject or accept annotations in a chain, or click Merge Chains to merge the two chains. If you want to remove a mention from a coreference chain, click the delete icon (-) in the label above the mention in the document area.



14. After all conflicts in the document are resolved, click Promote to Ground Truth.
* Involving domain subject matter experts to create the following resources, or to identify existing resources that can be re-used or modified for your domain:



* Annotation guidelines and examples to help human annotators learn how words and passages in your domain content are to be annotated.
* Type systems that define the domain-specific types (objects) and features (data classifications) that can be discovered in your domain content through text analysis. The type system controls the types of annotations that a human annotator can add to documents.
* Dictionaries of terms that are to be treated as equivalent terms in your domain content.



* Creating a corpus of documents that are representative of your domain content.
* Pre-annotating documents based on the dictionaries that you add to a Knowledge Studio workspace. After you create a machine learning model, you can use the model to pre-annotate new documents that you add to the corpus. Pre-annotation is a process of machine-annotating a document to the extent possible before a machine learning model is available to do so. Pre-annotation can reduce human-annotation labor by replacing some human annotation creation with mere verification of the correctness of machine annotation.
* Dividing documents among human annotators, who then use the IBM Watson® Knowledge Studio ground truth editor tool to manually add annotations to small sets of documents.
* Comparing the human annotation results and resolving conflicts. Adjudication in this phase is needed to ensure accurate and consistently annotated documents are promoted to ground truth, where they can be used to train and test a machine learning model.







 Model development 

This stage refers to the use of Knowledge Studio tools to create a model. After establishing ground truth, the human annotation results can be used to train an algorithm for automatically adding annotations to large collections of documents, such as collections that include millions of documents.





 Model evaluation 

This stage refers to the use of Knowledge Studio tools to refine the model and improve performance. The results generated by the model are evaluated against a test set of ground truth documents. Accuracy analysis identifies the causes of annotation errors. Headroom analysis helps you assess which errors require focus and where model refinements can yield the greatest impact.
Annotation tasks Assets & Tools > Documents > Tasks Machine Learning Model > Annotation Tasks 
 Coreferences tab Document Annotation Machine Learning Model > Annotation Tasks > task > annotation set > document 
 Dictionaries page (management) Assets & Tools > Pre-annotators > Manage Dictionaries Assets 
 Dictionaries tab (mapping to classes for rule-based model) Document Annotation Rule-based Model > Rules 
 Documents page Assets & Tools Assets 
 Entity Types page Assets & Tools Assets 
 Mentions tab Document Annotation Machine Learning Model > Annotation Tasks > task > annotation set > document 
 Performance page Model Management Machine Learning Model 
 Pre-annotators page Assets & Tools Machine Learning Model > Pre-annotation 
 Regex tab Document Annotation Rule-based Model > Rules 
 Relation Types page Assets & Tools Assets 
 Relations tab Document Annotation Machine Learning Model > Annotation Tasks > task > annotation set > document 
 Rules tab Document Annotation Rule-based Model 
 Tasks tab Assets & Tools > Documents Machine Learning Model > Annotation Tasks 
 Versions page (machine learning model) Model Management Machine Learning Model 
 Versions page (rule-based model) Model Management Rule-based Model 









 May 2018 



 New features and changes 

Configuration issue fixed
: A configuration issue was fixed that caused service instances in Sydney region to not appear in US South region.

Deploy Model window support changes
: In the Deploy Model window, if the region you're deploying to supports both IBM Cloud® Identity and Access Management resource groups and Cloud Foundry spaces, to see the list, you will need to choose the method of access management that your service instance uses.

Data collection setting added
: Added the data collection setting on the Service Details page. For more information about data collection, see [Troubleshooting, support, and FAQs](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-troubleshootingcontent)

Support for Chinese (Traditional)
: Added Chinese (traditional) language support.

Administrators can see number of workspaces
: Users who have the Admin role can now see the number of workspaces that are used.
See also [lemma](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-glossarygloss_L) and [surface form](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-glossarygloss_S).
* dictionary pre-annotator

A component that identifies mentions in text that match a specific set of words. By using domain-specific terminology to pre-annotate text, dictionary pre-annotators can accelerate a human annotator's ability to prepare a set of ground truth documents.
* document set

A collection of documents. Documents that are imported together become a document set. Annotated documents that are grouped together for training purposes (Test, Train, Blind) are generated as document sets.







 E 



* entity



1. A mention that is annotated by an entity type.
2. A person, object, or concept about which information is stored.
3. A set of details that are held about a real-world object such as a person, location, or bank account. An entity is a kind of item.



* entity type

The type of entity that a mention represents without consideration for context. For example, the mention IBM might be annotated by the entity type ORGANIZATION.

In an entity-relationship model, an entity type is the thing that is being modeled or the thing that a mention refers to, such as the name of a person or place. Different entity types have different sets of attributes such as "surname" or "home town", and are connected through relationships like "lives in". An entity type exists independently and can be uniquely identified.







 F 



* F1 score

A measure of a test's accuracy that considers both precision and recall to compute the score. The F1 score can be interpreted as a weighted average of the precision and recall values. An F1 score reaches its best value at 1 and worst value at 0.
* false negative

An answer or annotation that is correct, but was predicted to be incorrect.
* false positive
You can run a pre-annotator on documents that were added to the ground truth as part of the current workspace. Annotations that were added to the documents, reviewed, accepted, and promoted to ground truth within the current workspace are not stripped out.





 Running multiple pre-annotators 

Knowledge Studio allows you to run multiple pre-annotators at once. First, you need to prepare the pre-annotation methods that you want to use. For more information, see the following sections.



* [Dictionaries](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-preannotationwks_preannot)
* [Machine learning model](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-preannotationwks_preannotsire)
* [Rules-based model](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-preannotationwks_preannotrule)





 Configuring the order of pre-annotators 

When multiple pre-annotators are used, the first annotation made to a span of text is saved for the results, even if other pre-annotators attempt to annotate the same span of text later in the order. This doesn't apply to human annotations, which are preserved regardless of pre-annotation order.

For example, consider the example text IBM Watson. If a dictionary that is first in the order labels IBM as an Organization entity type, a machine learning model that is second in the order can't annotate IBM Watson as a Software Brand entity type because that would override the earlier annotation made to IBM.

You can view the current order of pre-annotators in the Order column on the Machine Learning Model > Pre-annotation page. To change the order, complete the following steps.



1. Click Order Settings.
2.
Click View Ground Truth for the training set or test set to see the annotations that were added through pre-annotation and by human annotators. The ground truth editor opens. Click to open individual documents and see how the mentions, relations, and coreferenced mentions were annotated.
3. On the Performance page, click View Decoding Results to see the annotations that the machine learning model added to documents in the test set. This button is available only after you evaluate the model. By viewing results, you can see how well the machine learning model labeled mentions, relations, and coreferenced mentions in the test data.
4. If you want to change how the documents are divided between training, test, and blind data sets, click Performance > Train and evaluate > Edit Settings. For example, if initial results seem acceptable, you might want to increase the number of documents in the test set to further test the machine learning model's results. You can change the ratio for how documents are automatically divided for different purposes, or you can select specific document sets to use as training data, test data, and blind data.
5. If you made any changes, click Train & Evaluate to retrain the model and re-evaluate the annotations.









 Deleting a machine learning model 

You cannot delete a machine learning model.

You can delete the workspace that was used to develop the model, but you cannot delete the model itself. Deleting a model is not the best approach. Instead, update or replace the artifacts that are used to train the model. Even if the model is not producing the results you expect, you can continue to refine it. Each time you create a new version, the model is built anew. You can edit artifacts like dictionaries and the type system, and choose to use different annotation sets when you train the next version.
For tokens that have annotations that were added indirectly from a dictionary annotation or regular expression match, you can choose whether the pattern should require any word with the same annotation type or the actual underlying words that were annotated instead.

In the lower layer of cells, you can see which cells are included in the pattern because a horizontal line connects them to one another. Where an annotation has been applied, there is a split. Cells with the original words are displayed below a cell with the annotation label. You can click one set of cells or the other to change the path of the line, and thus change the cells that are included in the pattern.

For example, you could choose to have the pattern use 50 instead of having it match the age regular expression.

![The "Create a rule" panel that shows an edit of the token, "50", that uses the annotation, "AGE_REGEX", in the rule. By default, the "AGE_REGEX" annotation is used, but you can edit the pattern so that the underlying word, "50", is used instead.](https://cloud.ibm.com/docs-content/v1/content/148d3bd95f946aa1bb53ea1540475f522e6b61c9/watson-knowledge-studio-data/images/rule_step5.png)
12. After you set the pattern order, you can annotate tokens in the text.

From the top layer of cells, click the cells that represents the tokens that you want to annotate, and then apply a class label to them. To select multiple cells, click one, press the Shift key, and click additional cells.

Assign a class to the selected cell or cells. If the class you want to assign does not exist, you can add it. Type the class name into the Assign class field and press Enter.

You cannot add more than 10 classes for the rule.

![The "Create a rule" section showing the "Assign class" window that opens when you click a token.
If so, select a mention and click Attribute View.
8. Click Save at any time to save your work.







 What to do next 

After you finish annotating all entity mentions, relation mentions, and coreferences in the document, as applicable, change the document status from In Progress to Completed, click Save, and then close the document.

After you finish annotating all documents and mark them Completed, the status of the annotation set changes to Submitted. That is how project managers know that they can start to evaluate the documents for inter-annotator agreement, reject or accept documents, and promote them to ground truth.







 Annotating repeating mentions 

You can optionally use the concordance tool to label multiple occurrences of a mention at once. The tool enables you to annotate the same text with the same entity type throughout a document and across annotation sets. Using the tool helps to ensure consistency in annotation across multiple documents. For example, you can label each occurrence of the mention encryption individually in mention mode, or you can label all occurrences of the mention encryption by using the concordance tool. Either way, the model learns from the entity type that you apply to the mention.



 About this task 

Although the concordance tool is optional, a good practice is to use the concordance tool to annotate mentions within a document or across documents before you start annotating mentions in individual documents. When you apply an entity type to a mention with the concordance tool, the system applies the entity type to all matching mentions, overriding any existing entity types that are assigned to a matching mention. To avoid conflicts, attributes (such as roles or subtypes) are removed from existing entity types when a new entity type is applied by the concordance tool.





 Procedure 

To annotate repeating mentions:



1. Log in as a human annotator (or as an administrator or project manager who was assigned documents to annotate). Workspaces that contain tasks that are assigned to you are displayed.
2. Open a workspace, and then click Machine Learning Model > Annotations. Click the Annotation Tasks tab. The annotation tasks that are assigned to you are displayed.
3.
For tokens that have annotations that were added indirectly from a dictionary annotation or regular expression match, you can choose whether the pattern should require any word with the same annotation type or the actual underlying words that were annotated instead.

In the lower layer of cells, you can see which cells are included in the pattern because a horizontal line connects them to one another. Where an annotation has been applied, there is a split. Cells with the original words are displayed below a cell with the annotation label. You can click one set of cells or the other to change the path of the line, and thus change the cells that are included in the pattern.

For example, you could choose to have the pattern use 50 instead of having it match the age regular expression.

![The "Create a rule" panel that shows an edit of the token, "50", that uses the annotation, "AGE_REGEX", in the rule. By default, the "AGE_REGEX" annotation is used, but you can edit the pattern so that the underlying word, "50", is used instead.](https://cloud.ibm.com/docs-content/v1/content/50eddb8fd81f33092880335ff107a78ff5cd0f65/watson-knowledge-studio/images/rule_step5.png)
12. After you set the pattern order, you can annotate tokens in the text.

From the top layer of cells, click the cells that represents the tokens that you want to annotate, and then apply a class label to them. To select multiple cells, click one, press the Shift key, and click additional cells.

Assign a class to the selected cell or cells. If the class you want to assign does not exist, you can add it. Type the class name into the Assign class field and press Enter.

You cannot add more than 10 classes for the rule.

![The "Create a rule" section showing the "Assign class" window that opens when you click a token.
To annotate occurrences regardless of case, add the terms in lowercase (such as hi-c). To annotate variations, add the variations as surface forms (MS-60-70 and MS 60 70).

Important: Do not use this approach if you are using the default tokenizer.









 Procedure 

To annotate entity mentions in a document:



1. Log in as a human annotator (or as an administrator who was assigned documents to annotate). Workspaces that contain tasks that are assigned to you are displayed.
2. Open a workspace, and then click Machine Learning Model > Annotations, then click the Annotation Tasks tab. The annotation tasks that are assigned to you are displayed.
3. Open the annotation task that you want to work on. The annotation sets that are assigned to you are displayed.
4. Click Annotate to open the annotation set you want to work on. The documents in the annotation set are displayed.
5. Open the document that you want to annotate. By default, the document opens in Mention mode, which is the mode you use to annotate entity mentions.
6. Begin to annotate entity mentions.



1. Click a word in the text that you recognize as a mention of a particular entity type from the type system. For entity mentions that consists of more than one word, click another word or drag the selection box edges to select multiple words or compound words.
2. Either select the entity type that you want to apply from the pane on the right, or enter the keyboard shortcut for the entity type.

If annotation guidelines were previously connected to the workspace, and you want help with choosing the correct annotation to apply, click View Guidelines. Depending on the access permissions set up on the site where the guidelines are hosted, you might be able to update the guidelines after you open them, for example, to add clarifications and examples.
3. Avoid creating overlapping mentions. But, if a valid overlapping mention is required, click Replace to more easily add it.
When a human annotator begins work on documents that were pre-annotated, it is likely that a number of mentions will already be marked by entity types based on the dictionary entries. The human annotator can change or remove the pre-annotated entity types and assign entity types to unannotated mentions. Pre-annotation by a dictionary does not annotate relations, coreferences. Relations and coreferences must be annotated by human annotators.

This task shows how to create a dictionary that is editable. If you want to upload and pre-annotate your documents with a read-only dictionary, click the Menu icon next to the Create Dictionary button. Select Upload Dictionary.





 Procedure 

To create an editable dictionary and pre-annotate documents, follow these steps:



1. Log in as a Knowledge Studio administrator and select your workspace.
2. Select the Assets > Dictionaries page.
3. Click Create Dictionary, enter a name, and then click Save.
4. From the Entity type list, select an entity type to associate with the dictionary.
5. Add entries for the dictionary or upload a file that contains dictionary terms.
6. Go to the Machine Learning Model > Pre-annotation page.
7. Click Run Pre-annotators.
8. Select Dictionaries, and then click Next.
9. If you want to erase existing annotations made by pre-annotators before running the pre-annotator, select Wipe previous pre-annotation results. Human annotations are preserved even if this is selected.
10. Select the check box for each document set that you want to pre-annotate and click Run.

Pre-annotation is applied to individual documents without regard for the various document sets or annotation sets that a document might belong to. A document that overlaps between a selected document set and an unselected document set will be pre-annotated in both document sets.



Related information:
Lesson 6: Adjudicating conflicts in annotated documents 

In this lesson, you will learn how to adjudicate conflicts in documents that overlap between document sets in Knowledge Studio.



 About this task 

When you approve a document set, only the documents that do not overlap with other document sets are promoted to ground truth. If a document is part of the overlap between multiple document sets, you must adjudicate any annotation conflicts before the document can be promoted to ground truth.

For more information about adjudication, see [Building the ground truth](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-build-groundtruth).





 Procedure 



1. Log in to Knowledge Studio as the administrator and select Machine Learning Model > Annotations. Click the Annotation Tasks tab, then click the Test task.
2. Verify that the two document sets are in an approved state.
3. Click Check Overlapping Documents for Conflicts.

You can see the overlapping documents that were annotated by more than one human annotator.
4. Because the tutorial instructed you to create a conflicting relation for the Technology - gmanews.tv document, find that document in the list and click Check for Conflicts.
5. Select the two conflicting annotation sets and click Check for Conflicts.

Adjudication mode opens. In adjudication mode, you can view overlapping documents, check for conflicts, and remove or replace annotations before you promote the documents to ground truth.
6. Select Relation conflicts, accept the founderOf relation, and reject the employedBy relation.
7. Click Promote to Ground Truth.

Alternatively, you can promote a document to ground truth by clicking Accept on the Documents page.







 Results 

After you resolve the annotation conflicts and promote the documents to ground truth, you can use them to train the machine learning model.







 Lesson 7: Creating a machine learning model 

In this lesson, you will learn how to create a machine learning model in Knowledge Studio.



 About this task 

When you create a machine learning model, you select the document sets that you want to use to train it.