vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Ensure that the text color contrasts with the background color so the text will be visible after it is labeled.
* Background color. This is the color of the label that is applied to the entity after you annotate it.



When annotating documents, human annotators can use the keyboard shortcuts to quickly add annotations. And the annotation label and text colors help human annotators to instantly recognize types after they add annotations to a document.



* If there are entity or relation types that you do not want human annotators to assign to mentions, you can hide them from the ground truth editor, which shortens and simplifies the list of type options that users see. To do so, deselect the Active check box for the type.



As you assign new shortcuts and colors, you can preview the changes.
5. You can also change the default selection highlight color. The highlight color is the color of the border that is displayed around text after human annotators select it. The default color is a light blue, but you can change the color on the Selection Highlight tab to make it easier to identify the boundaries of the text that is selected.





 Related tasks 

[Modifying a type system without losing human annotations](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-improve-mlwks_projtypesysmod)









 Setting the IAA threshold 

To help you decide whether to accept or reject an annotated document set, you can specify an inter-annotator agreement threshold. The threshold helps you compare how well or poorly inter-annotator agreement compares to the IAA score calculated by the system.



 About this task 

To compare how different human annotators annotated the same documents, specify an evaluation threshold. If the annotations made by one human annotator differ from the annotations made by another human annotator to the point where the difference results in a low score, it means that the annotators do not agree. The disagreement needs to be investigated and resolved.





 Procedure 

To set the inter-annotator agreement threshold:



1.
Human annotators can sometimes be thorough about labeling entity type mentions, but be less diligent about annotating relation types and coreferences. For any key types that have a low percentage of corpus density, you might want to focus on finding more annotations of those type in the source documents. But, don't worry too much about coreference and relation accuracy if mention accuracy is deficient. Relation mentions between entities and coreferences of entities cannot be accurate unless entity mentions are accurate to begin with.
* Fix human annotations

Check to see whether your training data is consistently and fully annotated. A machine learning model learns from your ground truth annotations. For example, if a sentence contains the phrase Obama family, and you label "Obama" as PERSON in one sentence and "Obama family" as PEOPLE in another sentence, the inconsistency means that the machine learning model cannot learn the correct annotation. Likewise, if you label "Obama" as PERSON in one sentence but do not label his name at all in another sentence, the annotation effort is incomplete and the machine learning model will be improperly trained. This type of inconsistent and partial labeling is sometimes referred to as type confusion. In most cases, just the act of having multiple human annotators review an overlapping set of documents will surface occurrences of type confusion-related mistakes. Pay attention to the issues that surface during document conflict resolution because they can provide insight into deeper issues with the type system itself. If there is no room to further improve or refine the type system, then it might be necessary to update the annotation guidelines and include examples. You can provide illustrations of common mistakes and how to annotate mentions properly under a given set of circumstances.

Another indicator of annotation inconsistency is if you have enough annotations, but the corpus density is low. Density can be impacted when a mention that is significant in the domain literature occurs often, but is annotated as different types across the document set.

Low precision is often an indication that you need to improve annotation consistency. To do so, review the annotation guidelines, better train the human annotators, and ensure that human annotators are working in concert and not in isolation from one another.
> Note: If the task contains completed documents, human annotators cannot alter those documents to assess type system changes until they are back in an editable state. To become editable, ask human annotators to submit the document sets so that you can reject them.



Related concepts:

[Type systems](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-typesystemwks_typesystem)







 Document set management 

Use the right sets of data to test and train the model at the right time.

The documents that you add to the system must be allocated to the following system-level data sets when you create a machine learning model:



* Training set

A set of documents that have been annotated through pre-annotation or by human annotators that is used to train the model. The goal of the training set is to teach the machine learning model about correct annotations, which includes teaching the model through text that was not annotated.
* Test set

A set of annotated documents that is used to test the trained model. After you run a test on the test set, perform a detailed diagnostic-purposed error analysis of the results. Close analysis helps you find weaknesses in the current model that can be addressed.
* Blind set

A set of annotated documents that is set aside and used to test the system periodically after several iterations of testing and improvement have occurred. To prevent accuracy from being tainted (for example, by making changes based only on annotations in known documents), blind data should be data that has not previously been viewed by users involved with creating the model. Reported results should come only from tests that are run on blind data. After you run a test on the blind set, look at only the most high-level scores, such as the overall mention and relation F1 scores. You don't want to learn too many details about the performance or it might influence the improvements that you choose to make to the model.



The goal of Knowledge Studio is to enable large teams to work together to build models.
On the Machine Learning Model > Pre-annotation > Dictionaries tab, click Apply This Pre-annotator.
4. Select the document set that you created in [Lesson 1](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-wks_tutml_introtut_lessml1).
5. Click Run.







 Results 

The documents in the selected sets are pre-annotated by using the dictionary that you created. If you like, you can use the dictionary to pre-annotate document sets or annotation sets that you add later.







 Lesson 3: Creating an annotation task 

In this lesson, you will learn how to create annotation sets and use annotation tasks to track the work of human annotators in Knowledge Studio.



 About this task 

An annotation set is a subset of documents from an uploaded document set that you assign to a human annotator. The human annotator annotates the documents in the annotation set. To later use inter-annotator scores to compare the annotations that are added by each human annotator, you must assign at least two human annotators to different annotation sets. You must also specify that some percentage of documents overlap between the sets.

> Note: In a realistic scenario, you would create as many annotation sets as needed, based on the number of human annotators who are working in the workspace. In this tutorial, you will create two annotation sets. If you do not have access to multiple user IDs, you can assign both annotation sets to the same user.

For more information about annotation sets and annotation tasks, see [Creating an annotation task](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-annotate-documentswks_hatask).





 Procedure 



1. Within your workspace, click Machine Learning Model > Annotations.
2. Click the Annotation Tasks tab, then click Add Task.
3.
Creating custom annotators 

You can create a dictionary, regular expression, or machine learning annotator to generate new facets that can help you to analyze your data.

Before you begin, have the following data ready.



Custom annotator prerequisite data

 Annotator type Description Data 

 Dictionary Assigns facets to terms that match dictionary entries that you define or upload. You can optionally upload a file of dictionary terms. 
 Machine learning Assigns facets to mentions that are recognized by a machine learning model that you upload. A compressed file of a machine learning model is required. 
 Regular expression Assigns facets to text that matches Java regular expression patterns that you define or upload. You can optionally upload a JSON file that contains regular expression patterns. 



To create a custom annotator, complete the following steps:



1. From the analysis view of your collection, click the Collections link in the breadcrumb to open the Create a collection for your analytics solutions page of the Content Mining application.
2. To create an annotator, click collection, and then select custom annotator from the list.

Zoom

![Shows the collection menu](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/cm-collection-menu.png)

Figure 1. Collection menu
3. Click Create custom annotator.
4. Name your annotator, and then optionally add a description.
5. Choose the annotator type, and then click Next.
6. Follow the on-screen instructions.

For more information about how to configure each annotator type, see one of the following sections:



* [Dictionary](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-custom-annotatorcm-custom-annotator-dictionary)
Low precision X X X X X 
 Low recall X X X X 
 Low annotation % X X X 
 Low density X X X 
 Few documents of type X X X 
 Incorrect categorization X X 
 Missed annotations X X 
 Gap between test and train results X 
 Low F1 for testing training data X X X 





 Descriptions of fixes 



* Add dictionaries

A dictionary contains example surface forms of a given entity type. You might need to add a new dictionary or add more entries to an existing dictionary if training statistics show that annotations of the type occur infrequently in the training data. If you know that the entity type is key to the domain and does occur often, then the omission might indicate that the surface forms associated with the type which are present in the training data are not being recognized by the machine learning model. Providing more surface form examples can help to solve this problem.
* Edit document sets

Ensure that you have enough overall training data. To learn, a machine learning model requires sufficient content. Aim to provide around 300,000 words. (You can run a word count tool on your source documents to check the amount.) If your training set is too small, add documents to the corpus. It is not enough to have a handful of documents that exercise each important entity type or relationship; you want many documents that illustrate how those types are used in typical domain literature.



* Ensure that the assortment of documents that is used for testing and training is similar. For example, do not use documents from one data source as the test data set and documents from another data source as the training data set. Different data sources might represent entity and relation type information in completely different ways. It is best to use a combination of documents from all of the data sources in both data sets to get the most comprehensive set of usage examples available from the industry literature. If the overall performances scores for a test run differ greatly from the training run, it might be due to inconsistencies in the data sets used for each.
* Don't waste time with incorrectly formatted documents. Text documents that you add to the corpus must be in UTF-8 format.
To ensure that your application receives enough context to accurately annotate large collections of documents, your test data must include a large sampling of the most important entity types and relation types.

After you train your first model, you will likely need to make modifications based on the performance statistics. However, to create a reliable statistical model for machine annotation, you want the type system to be as close to final as possible before you begin large-scale annotation tasks.





 When to define roles 

Using roles enables you to define more precise entity types.

Every entity that you add has a role. Unless you change it, the role name is the same as the entity name. You can choose to define a different role for an entity to capture a more nuanced usage of the mention. The word or phrase to which the role applies is literally an example of an entity type, but in the sentence it plays the role of another entity type. For example, in the phrase, the White House vetoed the bill, we can capture more precisely the meaning of the White House if we annotate both the entity type of FACILITY, and a role of ORGANIZATION or PERSON. White House is a building (FACILITY), but represents the government (ORGANIZATION) or the President of the United States (PERSON) in this construction. The entity type plus the role label creates a more precise classification of the mention in the text.

Roles can also provide you with a way to capture relation information without relying on the use of overlapping mentions. For example, you might want to annotate the text 31-year-old male such that it captures the idea that this is a reference to a person with the attribute age of 31 years and attribute gender of male. By examining the text, we determine that 31-year-old is an age, male is a gender, and together they signify a person. You are dealing with 3 entity types, basically: AGE, GENDER, and PERSON. One approach to representing this would be to annotate 31-year-old as AGE, male as GENDER, and because the distinct mention of a PERSON is missing, but male has come to represent person, the word male can have a role of PERSON. You can then capture relations between the PERSON role and all the attributes of a person that you want to capture.
To see the annotations that the trained machine learning model created on that same set of documents, click View Decoding Results.
8. To view details about the precision, recall, and F1 scores for the machine learning model, click the Performance page.
9. Click the Detailed Statistics links above each of the graphs. On these Statistics pages, you can view the scores for mentions, relations, and coreference chains by using the radio buttons.

You can analyze performance by viewing a summary of statistics for entity types, relation types, and coreference chains. You can also analyze statistics that are presented in a confusion matrix. To see the matrix, change Summary to Confusion Matrix. The confusion matrix helps you compare the annotations that were added by the machine learning model to the annotations in the ground truth.

In this tutorial, you annotated documents with only a single dictionary for organizations. Therefore, the scores you see are 0 or N/A for most entity types except ORGANIZATION. The numbers are low, but that is expected because you did not do any human annotation or correction.

Figure 2. Options on the Statistics page for a machine learning model

![This screen capture shows the Statistics page.](https://cloud.ibm.com/docs-content/v1/content/50eddb8fd81f33092880335ff107a78ff5cd0f65/watson-knowledge-studio/images/wks_tutanno9.png)
10. Click Versions. On the Versions page, you can take a snapshot of the model and the resources that were used to create it (except for dictionaries and annotation tasks). For example, you might want to take a snapshot before you retrain the model. If the statistics are poorer the next time you train it, you can promote the older version and delete the version that returned poorer results.







 Results 

You created a machine learning model, trained it, and evaluated how well it performed when annotating test data and blind data. By exploring the performance metrics, you can identify ways to improve the accuracy of the machine learning model.







 Tutorial summary
A subject matter expert who reviews, modifies, and augments the results of pre-annotation by identifying mentions, entity type relationships, and mention coreferences. By examining text in context, a human annotator helps determine ground truth and improve the accuracy of the machine learning model.







 I 



* inter-annotator agreement

A measure of how similarly a document in two or more document sets is annotated.







 K 



* knowledge graph

A model that consolidates typed entities, their relationships, their properties, and hierarchical taxonomies to represent an organization of concepts for a given domain. After the knowledge graph store is loaded with inputs from structured and unstructured data sources, users and applications can access the knowledge graph to explore key elements of knowledge for a specific domain, explore interactions, and discover additional relationships.







 L 



* lemma

The normalized or canonical form of a word. Typically, the lemma is the underived and uninflected form of a noun or a verb. For example, the lemma of the terms 'organizing' and 'organized' is 'organize'. See also [dictionary](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-glossarygloss_D) and [surface form](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-glossarygloss_S).







 M 



* machine learning

A method of data analysis that iteratively learns from past data and independently adapts when exposed to new data. The mathematical model at the core of machine learning is built from ground truth inputs. Through training and refinement of example input data, the model can deliver accurate, repeatable results when it analyzes new data.
* machine learning annotator

See [machine learning model](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-glossarygloss_M).
* machine learning model

A component that identifies entities and entity relationships according to a statistical model that is based on ground truth.
To prevent accuracy from being tainted (for example, by making changes based only on annotations in known documents), blind data should be data that has not previously been viewed by users involved with creating the model. Reported results should come only from tests that are run on blind data. After you run a test on the blind set, look at only the most high-level scores, such as the overall mention and relation F1 scores. You don't want to learn too many details about the performance or it might influence the improvements that you choose to make to the model.



The goal of Knowledge Studio is to enable large teams to work together to build models. As such, it assumes that models are being produced by a team that includes a group of human annotators and a separate person or group of people that builds and tests the model, and makes improvements to it. Due to this assumption, the application is configured to push an equally proportioned grouping of documents from a single document set into the test, train, and blind sets. However, if your team is not segregated - if the people doing human annotation are also reviewing model test results in detail, for example - then you might need to change the allocation of documents into these sets to more explicitly separate the documents that are being used in each one.



 Why do I need a blind set? 

Because you use test data to assess accuracy in detail, you get to know the documents and their features after a while. For example, you start to know which entity types, relation types, and text types in the documents are best understood by the machine learning model, and which are not. This information is important because it helps you focus on making the right improvements - refining the type system, supplementing the training data to fill gaps, or adding dictionaries, for example. As the test documents get used iteratively to improve the model, they can start to influence the model training indirectly. That's why the "blind" set of documents is so important.





 How do I control which documents are allocated to a set? 

When you create a machine learning model, you must specify the ratio of documents from the set to allocate to the train, test, or blind sets. Knowledge Studio automatically applies a ratio of 70/23/7 to the document sets that you use to build a machine learning model.
The document text will continue to contain the specific medicine name. It will just be enhanced with an annotation that identifies its type, which makes the information easier to find and extract programmatically.

As you get started, define 10 to 40 entity types and relation types. Stay to the lower end of the range if the human annotators who will be working on the workspace are not highly specialized in the field. Do not define more than 50.

Plan to spend a good amount of time defining the type system before your team begins any human annotation tasks. When the team does start to annotate documents, begin with a small set, perhaps no more than 50. Annotate only mentions initially, examine the results, and refine your annotation guidelines and the type system as needed. When you're satisfied with the results of mention annotation, move on to annotate relations and coreferences.

While fundamental work is underway to define a set of entity types and mention-annotation guidelines, avoid putting a lot of effort into annotating relation mentions. Mention changes will undo relation annotation work. Do take some time to define a set of relation types and their allowable entity type pairs so that relation type needs are taken into consideration before the entity type inventory is finalized.

Expect the type system to evolve with the experience of people trying to annotate to it. If you revise the type system after you create human annotation tasks, you must decide whether to apply the changes to each task. If you apply the changes, human annotators will have to revisit the documents that they annotated previously.

When you test the model, you can review statistics that show how frequently the entity types and relation types occur in your sample documents. Be sure to review these statistics. To ensure that your application receives enough context to accurately annotate large collections of documents, your test data must include a large sampling of the most important entity types and relation types.

After you train your first model, you will likely need to make modifications based on the performance statistics. However, to create a reliable statistical model for machine annotation, you want the type system to be as close to final as possible before you begin large-scale annotation tasks.