vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Annotation guidelines 

There is no prescribed format for how to document the guidelines, but it is important that the guidelines include detailed examples. Human annotators need to understand which entity type to apply to a mention given the context, and know which relation types are valid for a given pair of mentions. Examples drawn from your domain content are often the best way to convey the correct annotation choices to make.

Annotation guidelines are not static. As your project evolves, you'll likely discover instances of mentions and relationships that are not accurately captured in the guidelines. And you'll likely discover inconsistencies between multiple human annotators who interpret the guidelines in different ways. By updating the guidelines as situations arise, you can help improve the accuracy and consistency of annotations over time.

Before documents can be considered ground truth, any conflicts between how different human annotators annotated the same documents must be resolved. A key way to resolve the conflicts is by discussing what caused the confusion, thus helping human annotators learn from their mistakes. Improving and clarifying the guidelines can help reduce the number of conflicts and help ensure that accurately and consistently annotated documents are promoted to ground truth.

To help you manage the guidelines, you might want to divide what could become a long document into multiple parts, such as guidelines for annotating entities, guidelines for annotating relations, and guidelines for annotating the ways that mentions can be coreferenced. Changes that you make in one area must be evaluated and coordinated with changes that you make in another area. For example, if you add an entity type, review the guidelines for annotating relation types and specify how the new entity type can relate to other entity types.





 Annotation guidelines example 

Most annotation guidelines will need a lot of detail and examples to ensure that human annotators consistently annotate text.

The example presented here is a simple guideline that was created for a small domain that contains traffic incident reports.



 Task Goals 



* As project members, become familiar with the iterative process of manual annotation and machine learning model refinement.
* Annotate documents in the automotive domain with the ground truth editor and use the annotations to train a machine learning model.
Improving and clarifying the guidelines can help reduce the number of conflicts and help ensure that accurately and consistently annotated documents are promoted to ground truth.

To help you manage the guidelines, you might want to divide what could become a long document into multiple parts, such as guidelines for annotating entities, guidelines for annotating relations, and guidelines for annotating the ways that mentions can be coreferenced. Changes that you make in one area must be evaluated and coordinated with changes that you make in another area. For example, if you add an entity type, review the guidelines for annotating relation types and specify how the new entity type can relate to other entity types.





 Annotation guidelines example 

Most annotation guidelines will need a lot of detail and examples to ensure that human annotators consistently annotate text.

The example presented here is a simple guideline that was created for a small domain that contains traffic incident reports.



 Task Goals 



* As project members, become familiar with the iterative process of manual annotation and machine learning model refinement.
* Annotate documents in the automotive domain with the ground truth editor and use the annotations to train a machine learning model. Annotate the entity types, relation types, and coreference the entities as needed.







 Guideline Notations 



* Square brackets [ ] indicate the extent to be annotated when less than the entire quoted text is annotated.

Include negations as appropriate, for example [no injuries]ACCIDENT_OUTCOME. The type system is not using entity class to represent negation.







 Entity Types 

The type system does not use entity sub-types or roles, nor mention types or classes.



 Entity types Guidelines Examples 

 ACCIDENT_OUTCOME A consequence of an accident. Applies to both humans (e.g., death) and cars (e.g., dented). Can include "towed" and "air bag deployment" as indicators of severity of damage, and "transported to hospital" (but not funeral home) as indicators of severity of injury. Can include negation.
As you get started, define 10 to 40 entity types and relation types. Stay to the lower end of the range if the human annotators who will be working on the workspace are not highly specialized in the field. Do not define more than 50.

Plan to spend a good amount of time defining the type system before your team begins any human annotation tasks. When the team does start to annotate documents, begin with a small set, perhaps no more than 50. Annotate only mentions initially, examine the results, and refine your annotation guidelines and the type system as needed. When you're satisfied with the results of mention annotation, move on to annotate relations and coreferences.

While fundamental work is underway to define a set of entity types and mention-annotation guidelines, avoid putting a lot of effort into annotating relation mentions. Mention changes will undo relation annotation work. Do take some time to define a set of relation types and their allowable entity type pairs so that relation type needs are taken into consideration before the entity type inventory is finalized.

Expect the type system to evolve with the experience of people trying to annotate to it. If you revise the type system after you create human annotation tasks, you must decide whether to apply the changes to each task. If you apply the changes, human annotators will have to revisit the documents that they annotated previously.

When you test the model, you can review statistics that show how frequently the entity types and relation types occur in your sample documents. Be sure to review these statistics. To ensure that your application receives enough context to accurately annotate large collections of documents, your test data must include a large sampling of the most important entity types and relation types.

> Important: After you train your first model, you will likely need to make modifications based on the performance statistics. However, to create a reliable statistical model for machine annotation, you want the type system to be as close to final as possible before you begin large-scale annotation tasks.





 When to define roles 

Using roles enables you to define more precise entity types.

Every entity that you add has a role.
In large team settings, admins rarely participate in the model development process. 
 Project manager Responsible for the overall organization of the workspace that she or he is assigned to. Tasks include creating the type system, managing assets, managing annotation work, evaluating the machine learning model, and deploying models. Users in this role need industry subject-matter expertise because they create the type system, teach the human annotators how to correctly apply the type system, and evaluate the model quality. 
 Human annotator Performs the labeling of the entity mentions and relationship mentions in the training documents that he or she is assigned to. The work is assigned to human annotators and monitored by the project manager. Human annotators may not have industry subject-matter expertise, as long as they are taught by the project manager how to correctly apply the type system. 







 Knowledge Studio role permissions 

To compare the permissions of each role, see the following table. One permission is listed on each row. If a role has that permission, the role column is marked with a check mark (✓).



Table 2. Role permissions

 Permission Admin Project manager Human annotator 

 Manage user access and roles ✓ 
 Manage workspaces ✓ 
 Create the type system and annotation guidelines ✓ ✓ 
 Upload training documents ✓ ✓ 
 Manage rules ✓ ✓ 
 Pre-annotate documents ✓ ✓ 
 Manage annotation tasks ✓ ✓ 
 Resolve annotation conflicts with human annotation (adjudication) ✓ ✓ 
 Train and evaluate machine learning models ✓ ✓ 
 Export models ✓ ✓ 
 Annotate document sets directly ✓ ✓ 
 Perform document annotation in annotation tasks ✓ ✓ ✓
For example, if there is no PERSON mention in the document sets that were annotated for ground truth or as test data, the scores for the PERSON entity type is N/A.



For example, the following confusion matrix example shows results of a machine learning model run on documents that deal with traffic incidents.

Table 1. Confusion matrix example



 Entity types MANUFACTURER MODEL O 

 MANUFACTURER 515 5 44 



You can learn the following things from this matrix:



* The model correctly recognized 515 tokens as mentions of the MANUFACTURER entity type.
* The model incorrectly labeled 5 tokens as mentions of the MODEL type that should have been labeled as MANUFACTURER.
* There were 44 tokens that comprise mentions of the MANUFACTURER type that the model failed to annotate as any entity type.



When you evaluate model performance and view the decoding results, you can start by investigating the tokens that were mislabeled or missed completely.







 Performance improvement cheat sheet 

Use this cheat sheet to help you determine the steps you can take to improve the machine learning model performance.

It's difficult to dictate rules for improving performance scores that would be applicable across domains, where type system complexity, appropriateness of training documents, human annotator skills, and other factors influence the outcome. However, the following table can help you make an initial assessment and take steps to improve performance, especially during the early stages of model development and testing.

The following table suggests fixes for the most common machine learning model performance problems.



 Problem Add dictionaries Edit document sets Add type-specific documents Annotate corpus more Fix human annotations Enhance human annotator guidelines Update type system Investigate further 

 Low F1 X X X X X X X X 
 Low precision X X X X X 
 Low recall X X X X 
 Low annotation % X X X 
 Low density X X X 
 Few documents of type X X X 
 Incorrect categorization X X 
 Missed annotations X X 
 Gap between test and train results X 
 Low F1 for testing training data X X X 





 Descriptions of fixes 



* Add dictionaries

A dictionary contains example surface forms of a given entity type.
Most annotation guidelines will need a lot of detail and examples to ensure that human annotators consistently annotate text.

The example presented here is a simple guideline that was created for a small domain that contains traffic incident reports.



 Task Goals 



* As project members, become familiar with the iterative process of manual annotation and machine learning model refinement.
* Annotate documents in the automotive domain with the ground truth editor and use the annotations to train a machine learning model. Annotate the entity types, relation types, and coreference the entities as needed.







 Guideline Notations 



* Square brackets [ ] indicate the extent to be annotated when less than the entire quoted text is annotated.

Include negations as appropriate, for example [no injuries]ACCIDENT_OUTCOME. The type system is not using entity class to represent negation.







 Entity Types 

The type system does not use entity sub-types or roles, nor mention types or classes.



 Entity type Guidelines Examples 

 ACCIDENT_OUTCOME A consequence of an accident. Applies to both humans (e.g., death) and cars (e.g., dented). Can include "towed" and "air bag deployment" as indicators of severity of damage, and "transported to hospital" (but not funeral home) as indicators of severity of injury. Can include negation. "[casualty]", "[injury]", "sustained [total loss]", "[no injuries]", "[towed] due to [disabling damage]", [not towed], "air bag did [not deploy]" (air bag itself has to be PART_OF_CAR, related by sufferedFrom to this ACCIDENT_OUTCOME), and indications of severity 
 CONDITION Weather or road conditions; an aspect of the scene that could affect likelihood of accident and could change from day to day but is not about the car or driver. Can be driver error or mechanical failure, and must appear to be problematic. Should exclude STRUCTURE. "dry", "rainy", "construction", "heavy traffic", "daylight", but not "grassy" or "intoxicated".
* [Dictionary Lookup annotator](https://www.ibm.com/support/knowledgecenter/SS8NLW_11.0.0/com.ibm.discovery.es.ta.doc/iiystadictlookup.htm)
* [Named Entity Recognition annotator](https://www.ibm.com/support/knowledgecenter/SS8NLW_11.0.0/com.ibm.discovery.es.ta.doc/iiystasystemt.htm)









 Uploading UIMA CAS XMI files into a workspace 

To use the pre-annotated documents that you downloaded to train a model, you must create a .zip file that contains all the files required to upload the XMI files, and then upload the .zip file into a Knowledge Studio workspace.



 Before you begin 

Before you upload the .zip file, ensure that the type system in your Knowledge Studio workspace includes the entity types that you mapped the UIMA types to.

UIMA analysis engines allow annotations to span sentences. In Knowledge Studio, annotations must exist within the boundaries of a single sentence. If the XMI files that you upload include annotations that span sentences, those annotations do not appear in the ground truth editor.





 Procedure 

To upload pre-annotated documents into a Knowledge Studio workspace, follow these steps:



1. Create a .zip file that contains all of the files that are required by Knowledge Studio.



1. Select the folder that contains the XMI files, UIMA type system descriptor file, and cas2di.tsv file, or select all of the files in the folder.
2. Create a .zip file that includes all files. Make sure the cas2di.tsv and UIMA type system descriptor files are stored in the root directory of the .zip file. These files cannot be stored in a subfolder within the .zip file or Knowledge Studio will not be able to read them, and nothing will be imported.

In Windows, you can right-click and select Send to > Compressed (zipped) folder.



2.
Adding documents for annotation 

To train a machine learning model, you must add documents that contain subject matter knowledge, such as journal articles or other industry-specific texts, to your workspace.



 About this task 

This section describes how to add documents for annotation only. To define rules for the rule-based model, you add or upload documents from which you can draw patterns to define as rules. See [Adding documents for defining rules](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks_rule_anno_add) for more information.





 Documents 

To train a machine learning model, you need to collect documents that are representative of your domain content and of high value to your application.

Try to ensure that your training documents are truly representative of content that is of interest to your domain; that is, they contain many relevant mentions that can be annotated. To choose the best documents, follow these guidelines:



* Strive to provide a set of documents that have a total size of about 300,000 words. Provide more words for a complex type system, and fewer for a simpler one.
* Limit each document to a page or two of content (fewer than 2,000 words, and closer to 1,000 words per document is best). In the early stages of model development, keeping each document down to a few paragraphs is also a good practice. A human annotator can mark mentions and relations in a long document, but attempts to mark coreferences across multiple pages might prove unwieldy.
* Ensure that the data in the documents is distributed across all possible entity types, subtypes, and roles, and the relationships between them. A goal to aim for is to eventually have at least 50 annotations for each entity type and 50 for each relation type in the document collection.
* Again, documents should represent the breadth of the subject matter that the application will cover, but in the case of skewed frequency-of-occurrence of entity types and relation types, try to get at least 50 exemplars of each type, more for entity types that have mentions which tend to be phrases.
* The set that you create for training must contain at least 10 annotated documents.
In human annotation, a collection of documents that are extracted from the corpus that allow the workload to be shared by multiple human annotators. In machine-based annotation, a collection of documents that can be used as blind data, training data, or test data.
* annotation process manager

A role that is responsible for managing the full annotation lifecycle activities within a workspace. The project manager that is added to a workspace typically performs the activities of an annotation process manager.
* annotator

See [human annotator](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-glossarygloss_H) and [machine learning annotator](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-glossarygloss_M).
* attribute

A characteristic or trait of an entity that describes the entity; for example, the telephone number of an employee is one of the employee attributes.







 B 



* blind data

A set of documents annotated with the ground truth, such as question and answer pairs, semantic annotation, and passage judgment. Blind data is never released or seen by developers and is used to test the system periodically to evaluate performance on unseen data. Testing on blind data prevents accuracy from being tainted by over-fitting to known question sets or annotations. Reported results should only come from tests that are run on blind data. See also [testing data](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-glossarygloss_T) and [training data](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-glossarygloss_T).







 C 



* concordance

Provides a way to ensure that the same mention is annotated with the same entity type throughout a document and across annotation sets. Concordance helps ensure consistency across multiple occurrences of a mention without requiring the human annotator to manually label each occurrence.
* confusion matrix
Adding documents for annotation 

To train a machine learning model, you must add documents that contain subject matter knowledge, such as journal articles or other industry-specific texts, to your workspace.



 About this task 

This section describes how to add documents for annotation only. To define rules for the rule-based model, you add or upload documents from which you can draw patterns to define as rules. See [Adding documents for defining rules](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-wks_rule_anno_add) for more information.





 Documents 

To train a machine learning model, you need to collect documents that are representative of your domain content and of high value to your application.

Try to ensure that your training documents are truly representative of content that is of interest to your domain; that is, they contain many relevant mentions that can be annotated. To choose the best documents, follow these guidelines:



* Strive to provide a set of documents that have a total size of about 300,000 words. Provide more words for a complex type system, and fewer for a simpler one.
* Limit each document to a page or two of content (fewer than 2,000 words, and closer to 1,000 words per document is best). In the early stages of model development, keeping each document down to a few paragraphs is also a good practice. A human annotator can mark mentions and relations in a long document, but attempts to mark coreferences across multiple pages might prove unwieldy.
* Ensure that the data in the documents is distributed across all possible entity types, subtypes, and roles, and the relationships between them. A goal to aim for is to eventually have at least 50 annotations for each entity type and 50 for each relation type in the document collection.
* Again, documents should represent the breadth of the subject matter that the application will cover, but in the case of skewed frequency-of-occurrence of entity types and relation types, try to get at least 50 exemplars of each type, more for entity types that have mentions which tend to be phrases.
* The set that you create for training must contain at least 10 annotated documents.
Annotation setup 

Users who have knowledge of the industry and its language must annotate the documents.

Perform the following tasks to enable human annotators to access the workspace:



* Invite subject matter experts to the Knowledge Studio instance that you are using.
* Associate human annotators with the annotation set or sets that you want them to annotate.
* Create a task that assigns the human annotator to annotate the documents in the set.

It is not until you explicitly assign tasks to human annotators that they can see your workspace when they log in to Knowledge Studio.



Your users (the human annotators) will need to become familiar with [Annotating documents](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-user-guide), which contains detailed information about how to annotate documents.



 Model life cycle 

The model that you create with Knowledge Studio is a software component that can be plugged into a natural language processing (NLP) pipeline.

With Knowledge Studio, you can create, evaluate, and improve models for new domains. A model adds annotations (metadata) to text that appears in natural language content. The annotations, which identify mentions of entities of interest in your domain content, the relationships between them, and how the mentions co-reference the same entity, can be used by applications to automatically analyze and process text. Application users benefit from this level of analysis by being able to extract meaning, discover insights, and obtain answers in a natural language context.

The creation of a model is an iterative multiple-step process that involves several stages: knowledge curation, ground truth generation, model development, model evaluation, and runtime deployment.



 End-to-end domain adaptation 

The following diagram summarizes the interactions between these five stages of model development and the typical activities that occur at each stage.

![A summary of the five stages of model development and the activities that occur at each stage.]