vania-janet/multiturn-rag-retrieval-data huggingface dataset:
This value specifies that you want 100 percent of the documents in the base set to be included in all the new annotation sets so they can be annotated by all human annotators.
7. For each new annotation set, specify the required information.



* In the Annotator field, select a human annotator user ID to assign to the new annotation set. In a realistic scenario, each annotation set is assigned to a different human annotator.

If you have only a single administrator ID to use for the tutorial, assign that user to all annotation sets. In a realistic scenario, you would have multiple human annotators, but for the tutorial, the administrator can act as human annotator.
* In the Set name field, specify a descriptive name for the annotation set. For this tutorial, you can use the names, Set 1 and Set 2.



8. Click Generate.
9. Click Save.
10. As human annotators begin annotating documents, you can open tasks to see their progress.









 Lesson 4: Annotating documents 

In this lesson, you will learn how to use the ground truth editor to annotate documents in Knowledge Studio.



 About this task 

For more information about human annotation, see [Annotation with the ground truth editor](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-user-guidewks_hagte).





 Procedure 



1. Log in to Knowledge Studio as a user who is assigned to the annotation task that you created in [Lesson 3](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks_tutml_introwks_tutless_ml2).

If you have access only to a single administrator ID for this tutorial, you can use that ID to perform human annotation. However, remember that in a realistic scenario, human annotation is performed by different users with the Human Annotator role.
2.
This value specifies that you want 100 percent of the documents in the base set to be included in all the new annotation sets so they can be annotated by all human annotators.
7. For each new annotation set, specify the required information.



* In the Annotator field, select a human annotator user ID to assign to the new annotation set. In a realistic scenario, each annotation set is assigned to a different human annotator.

> Note: If you have only a single administrator ID to use for the tutorial, assign that user to all annotation sets. In a realistic scenario, you would have multiple human annotators, but for the tutorial, the administrator can act as human annotator.
* In the Set name field, specify a descriptive name for the annotation set. For this tutorial, you can use the names, Set 1 and Set 2.



8. Click Generate.
9. Click Save.
10. As human annotators begin annotating documents, you can open tasks to see their progress.









 Lesson 4: Annotating documents 

In this lesson, you will learn how to use the ground truth editor to annotate documents in Knowledge Studio.



 About this task 

For more information about human annotation, see [Annotation with the ground truth editor](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-user-guidewks_hagte).





 Procedure 



1. Log in to Knowledge Studio as a user who is assigned to the annotation task that you created in [Lesson 3](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-wks_tutml_introwks_tutless_ml2).

> Note: If you have access only to a single administrator ID for this tutorial, you can use that ID to perform human annotation. However, remember that in a realistic scenario, human annotation is performed by different users with the Human Annotator role.
2.
Make sure that all of the annotation sets that you want to include in the task appear under Selected Sets, then click Save to create the task.







 What to do next 

After the task is created, you can return to the Annotation Tasks tab on the Machine Learning Model > Annotations page to view the progress of each human annotator. Also, you can complete the following tasks:



* Check approved documents that overlap between annotation sets to resolve annotation conflicts.
* Open a task to add annotation sets to it. Ensure that the annotation sets that you add include documents that overlap with documents in the original annotation sets.



From the Settings tab of the main navigation, you can specify the following information:



* Specify preferences for using colors and keyboard shortcuts in the ground truth editor.
* Specify an inter-annotator agreement threshold, and then open a task to see how consistently multiple human annotators annotated the same documents.
* Specify a URL to connect your annotation guidelines to the ground truth editor.









 Configuring ground truth editor preferences 

An project manager can specify preferences for using colors and keyboard shortcuts in the ground truth editor.



 Procedure 

To specify visual preferences for working with the ground truth editor :



1. Log in as a Knowledge Studio administrator, and select your workspace.
2. From the navigation on the left, select the Settings > Document Annotation Settings.
3. Select the Entity Types or Relation Types tab.
4. Select the entity type or relation type that you want to change and then click Edit keyboard shortcuts and colors. For each type, you can define a:



* Keyboard shortcut, which means a user can enter <key> to apply the type label to highlighted text. For example, if you define o as the keyboard shortcut for ORGANIZATION, then a user can select text, and then press the o key to apply the ORGANIZATION entity type to the highlighted text. If you assign an upper case letter, then the user must press Shift+<key>.
* Text color. Ensure that the text color contrasts with the background color so the text will be visible after it is labeled.
* Background color.
To connect the ground truth editor and adjudication tool to your annotation guidelines:



1. Log in as a Knowledge Studio administrator, and select your workspace.
2. select the Settings > Annotation Guidelines tab.
3. Specify the URL to where your guidelines are hosted.
4. Click Save. The system connects the ground truth editor and adjudication tool to your annotation guidelines. Depending on the access permissions granted to users when you created the guidelines, human annotators and workspace administrators might be able to update the guidelines after opening them, for example, to add clarifications and examples.







 Annotation guidelines 

There is no prescribed format for how to document the guidelines, but it is important that the guidelines include detailed examples. Human annotators need to understand which entity type to apply to a mention given the context, and know which relation types are valid for a given pair of mentions. Examples drawn from your domain content are often the best way to convey the correct annotation choices to make.

Annotation guidelines are not static. As your project evolves, you'll likely discover instances of mentions and relationships that are not accurately captured in the guidelines. And you'll likely discover inconsistencies between multiple human annotators who interpret the guidelines in different ways. By updating the guidelines as situations arise, you can help improve the accuracy and consistency of annotations over time.

Before documents can be considered ground truth, any conflicts between how different human annotators annotated the same documents must be resolved. A key way to resolve the conflicts is by discussing what caused the confusion, thus helping human annotators learn from their mistakes. Improving and clarifying the guidelines can help reduce the number of conflicts and help ensure that accurately and consistently annotated documents are promoted to ground truth.

To help you manage the guidelines, you might want to divide what could become a long document into multiple parts, such as guidelines for annotating entities, guidelines for annotating relations, and guidelines for annotating the ways that mentions can be coreferenced. Changes that you make in one area must be evaluated and coordinated with changes that you make in another area.
See [Annotating document sets directly](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-annotating-document-sets-directly).<br>2. The project manager generally assigns annotation tasks to human annotators, configures the inter-annotator agreement threshold, and provides annotation guidelines for the human annotators to follow. See [Creating an annotation task](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-annotate-documentswks_hatask).<br>3. Human annotators use the ground truth editor to manually annotate documents. A human annotator identifies mentions of interest in your domain content and labels them with entity types. The human annotator also identifies relationships between mentions (for example, Mary is an employee of IBM) and how the mentions co-reference the same entity (such as an occurrence of "she" that refers to Mary). Refer to [Annotating documents](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-user-guide).<br><br><br> 
 Adjudicate and promote documents Accept or reject the ground truth that was generated by human annotators, and adjudicate any annotation differences to resolve conflicts. Evaluating the accuracy and consistency of the human annotation effort might be the responsibility of a senior human annotator or a user with stronger subject matter experience than the project manager. See [Adjudication](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-build-groundtruthwks_haperform). 
 Train the model Create the machine learning model. See [Creating a machine learning model](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-train-mlwks_madocsets). 
 Evaluate the model Evaluate the accuracy of the model.
See [Annotating document sets directly](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-annotating-document-sets-directly).<br>2. The project manager generally assigns annotation tasks to human annotators, configures the inter-annotator agreement threshold, and provides annotation guidelines for the human annotators to follow. See [Creating an annotation task](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-annotate-documentswks_hatask).<br>3. Human annotators use the ground truth editor to manually annotate documents. A human annotator identifies mentions of interest in your domain content and labels them with entity types. The human annotator also identifies relationships between mentions (for example, Mary is an employee of IBM) and how the mentions co-reference the same entity (such as an occurrence of "she" that refers to Mary). Refer to [Annotating documents](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-user-guide).<br><br><br> 
 Adjudicate and promote documents Accept or reject the ground truth that was generated by human annotators, and adjudicate any annotation differences to resolve conflicts. Evaluating the accuracy and consistency of the human annotation effort might be the responsibility of a senior human annotator or a user with stronger subject matter experience than the project manager. See [Adjudication](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-build-groundtruthwks_haperform). 
 Train the model Create the machine learning model. See [Creating a machine learning model](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-train-mlwks_madocsets). 
 Evaluate the model Evaluate the accuracy of the model.
Also, you can complete the following tasks:



* Check approved documents that overlap between annotation sets to resolve annotation conflicts.
* Open a task to add annotation sets to it. Ensure that the annotation sets that you add include documents that overlap with documents in the original annotation sets.



From the Settings tab of the main navigation, you can specify the following information:



* Specify preferences for using colors and keyboard shortcuts in the ground truth editor.
* Specify an inter-annotator agreement threshold, and then open a task to see how consistently multiple human annotators annotated the same documents.
* Specify a URL to connect your annotation guidelines to the ground truth editor.









 Configuring ground truth editor preferences 

An project manager can specify preferences for using colors and keyboard shortcuts in the ground truth editor.



 Procedure 

To specify visual preferences for working with the ground truth editor :



1. Log in as a Knowledge Studio administrator, and select your workspace.
2. From the navigation on the left, select the Settings > Document Annotation Settings.
3. Select the Entity Types or Relation Types tab.
4. Select the entity type or relation type that you want to change and then click Edit keyboard shortcuts and colors. For each type, you can define a:



* Keyboard shortcut, which means a user can enter <key> to apply the type label to highlighted text. For example, if you define o as the keyboard shortcut for ORGANIZATION, then a user can select text, and then press the o key to apply the ORGANIZATION entity type to the highlighted text. If you assign an upper case letter, then the user must press Shift+<key>.
* Text color. Ensure that the text color contrasts with the background color so the text will be visible after it is labeled.
* Background color. This is the color of the label that is applied to the entity after you annotate it.



When annotating documents, human annotators can use the keyboard shortcuts to quickly add annotations. And the annotation label and text colors help human annotators to instantly recognize types after they add annotations to a document.
Adjustments can be made repeatedly to improve performance until a satisfactory level of accuracy is achieved.





 Model deployment 

This stage refers to exporting components that enable the model to run in machine learning runtime environments and making the model accessible to other Watson cognitive applications. For example, you can export the machine learning model for use by IBM Watson™ Natural Language Understanding for IBM Cloud Pak® for Data or IBM Watson™ Discovery for IBM Cloud Pak® for Data







 Creating an annotation task 

Before human annotators begin adding annotations to documents, the annotation process manager can optionally create an annotation task.

Admins and project managers can annotate ground truth document sets directly. See [Annotating document sets directly](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-annotating-document-sets-directly).



 About this task 

The annotation task specifies which documents are to be annotated. To compare how well the human annotators perform, and to see how consistently they apply the annotation guidelines, you must include at least two human annotators in the task. In addition, some percentage of documents must occur in all of the annotation sets that you add to the task (you specify the overlap percentage when you create the annotation sets).



 Important 



* An annotation task is a temporal concept that exists to allow human annotators to annotate text in isolated spaces. It also ensures that annotations must be approved before they are promoted to ground truth.
* An annotation set can be included in one active task at a time. To add an annotation set in one task to a different task, you must first delete the task where the annotation set is active.
* If you delete a human annotator's user account, it affects their annotations too. Any annotations in documents that were assigned to that user but were not promoted to ground truth are deleted.
* If the type system or ground truth editor settings change after you create a human annotation task, you must decide whether to propagate the changes to the task.
Dave and Phil, the human annotators, annotate documents in the sets that are assigned to them.](https://cloud.ibm.com/docs-content/v1/content/148d3bd95f946aa1bb53ea1540475f522e6b61c9/watson-knowledge-studio-data/images/wks_datatask.svg) Figure 2. This figures illustrates that the project manager creates annotation sets and assigns them to an annotation task. Dave and Phil, the human annotators, annotate documents in the sets that are assigned to them.

After the project manager approves annotation sets in an annotation task, annotations in documents that do not overlap with other annotation sets become ground truth. For documents that overlap between annotation sets (represented by document 2 in this example), the project manager must adjudicate and resolve conflicts. The annotations in overlapping documents do not become ground truth until they are approved through adjudication.

Ground truth is then used for training and testing a machine learning model, or it can be used as the basis for the next iteration of model development. To use ground truth in a new iteration, you must create a new annotation task.

![This figure illustrates how annotations added by two human annotators become ground truth. One document, labeled document 2, is annotated by both human annotators. The annotations in this overlapping document must be adjudicated before they become ground truth.](https://cloud.ibm.com/docs-content/v1/content/148d3bd95f946aa1bb53ea1540475f522e6b61c9/watson-knowledge-studio-data/images/wks_datatruth.svg) Figure 3. This figure illustrates how annotations added by two human annotators become ground truth. One document, labeled document 2, is annotated by both human annotators. The annotations in this overlapping document must be adjudicated before they become ground truth.
The guidelines can provide a sanity check to human annotators as they evaluate documents. The guidelines should be a living and changing document, especially at the beginning of the annotation process. They provide a key feedback loop because a human annotator can capture things she learned while annotating a few documents, then as she or someone else annotates a few more documents, new tips and gotchas can be added to the guideline, and so on. Be sure to include examples of difficult decisions and their preferred resolutions. The best way to determine what you need to add to the annotation guidelines is to carefully review document conflicts. Real examples of annotations that real people disagreed upon and how they were resolved can be a great help to human annotators as they tackle the annotation of new documents.
* Update type system

You might need to update the type system for these reasons:



* The documents that comprise the training data have references to concepts that are important types in the domain but are not represented anywhere in the type system. This suggests that you might need to add types that capture the missing concepts or relationships. Be careful not to try to define a type for every concept in a field, or every entity that occurs in domain documents; the type system should be limited to only the most fundamental types.
* An existing type is being consistently misused by human annotators. If a type consistently causes confusion, then you might need to rename it or eliminate it if it is redundant.
* An existing type is never used by human annotators because it is never referenced in the documents. If the type is unlikely to ever be used in literature from this domain, then remove it from the type system.
* Two types are often interchanged when human annotators annotate documents. Consider whether the two types could be consolidated into one type that accurately represents the concept or relationship. For example, if the type system contains both PERSON and PEOPLE, which are often used interchangeably, it might be best to use one type named PERSONPEOPLE that covers both cases instead of two separate types.



Use caution when you update the type system.
After establishing ground truth, the human annotation results can be used to train an algorithm for automatically adding annotations to large collections of documents, such as collections that include millions of documents.





 Model evaluation 

This stage refers to the use of Knowledge Studio tools to refine the model and improve performance. The results generated by the model are evaluated against a test set of ground truth documents. Accuracy analysis identifies the causes of annotation errors. Headroom analysis helps you assess which errors require focus and where model refinements can yield the greatest impact. Adjustments can be made repeatedly to improve performance until a satisfactory level of accuracy is achieved.





 Model deployment 

This stage refers to exporting components that enable the model to run in machine learning runtime environments and makl ing the model accessible to other Watson cognitive applications. For example, you can deploy the machine learning model for use by IBM Watson® Natural Language Understanding service or IBM Watson® Discovery service, or export the model for use in IBM Watson Explorer.







 Creating an annotation task 

Before human annotators can begin adding annotations to documents, the annotation process manager must create an annotation task.

Admins and project managers can annotate ground truth document sets directly. See [Annotating document sets directly](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-annotating-document-sets-directly).



 About this task 

The annotation task specifies which documents are to be annotated. To compare how well the human annotators perform, and to see how consistently they apply the annotation guidelines, you must include at least two human annotators in the task. In addition, some percentage of documents must occur in all of the annotation sets that you add to the task (you specify the overlap percentage when you create the annotation sets).



 Important 



* An annotation task is a temporal concept that exists to allow human annotators to annotate text in isolated spaces. It also ensures that only approved annotations are promoted to ground truth.
* An annotation set can be included in one active task at a time.