vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Labeling tips 

Review these tips before you begin:



* The document collection that you label must contain a representative set of documents. The documents must have many and varied examples of the entity types that you want the entity extractor to recognize. If the collection you selected when you started to create the entity extractor does not meet the requirement, stop now and start over with a different document collection.
* Define entity types that are clearly distinct from one another.
* Aim to label at least 40 examples of each entity type.
* Label every valid example of an entity type. Do not skip any occurrences. To speed up the process, use the bulk label feature.







 Labeling entity examples 

Label terms in the document that represent examples of the entity types you defined. When you are done with one document, switch the document status from In progress to Complete, and then move on to the next document.

Only the first 40,000 characters from each document are available for labeling. 40,000 characters is approximately 20 pages.

To label entity examples, complete the following steps:



1. Review the text of the document. Look for entity examples to label.

The following table shows some examples.



Entity types and examples

 Entity type Examples to label in the document 

 color white, green, purple 
 car convertible, SUV, sedan 
 auto_model Explorer, Civic, Sorrento 
 auto_manufacturer Ford, Honda, Kia 
 clothing shirt, blouse, skort 
 instruments bonds, stocks, ETFs, munis 



If an entity type that you want to identify is not created yet, add the entity type. From the Entity types panel, click Create new. For more information about adding entity types, see [Defining entity types](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-entity-extractorentity-extractor-add-entities).
2. First, click the entity type from the Entity types panel.
3. In the document body, select the word or phrase that represents the entity example.

The term is selected and a color label is applied to the term.
Entity Types tab Type System Assets & Tools > Entity Types 
 Ground Truth Editor Human Annotation Document Annotation tab 
 Ground Truth Editor Settings tab Human Annotation Settings > Document Annotation Settings 
 Models, running and exporting Annotator Component Model Management > Versions 
 Rules tab Main navigation Document Annotation > Rules 
 Statistics tab Annotator Component > Details Model Management > Performance 
 Summary table (from the Statistics tab) Annotator Component > Details Model Management > Performance > Detailed Statistics link 
 Type Mapping tab (for the rule-based model) Annotator Component > Details Model Management > Versions > Rule-based model type mapping 









 July 2017 

Header bug fix
: Fixed a defect that, in some cases, caused the header to disappear

Security updates
: Applied security updates for infrastructure components





 June 2017 

Improved stability
: Improved stability for handling large data under certain conditions

Adjudication bug fix
: Fixed a defect for an adjudication error





 May 2017 

Rename ability added
: Added ability to rename various objects, such as projects, document sets, and so on

Ability to deploy NLU models
: Added ability to deploy NLU models to specific IBM Cloud regions

Improved performance
: Improved performance for displaying large table for IAA

Bug fixes
: Fixed minor defects
: Applied security fixes





 April 2017 

Improved stability
: Improved stability for Knowledge Studio Premium

Usage analysis added
: Added usage analysis for business metrics





 Releases prior to April 2017 

See [IBM release notes, document 283189](https://www.ibm.com/support/pages/node/283189).
To do so, you must create the classifier in the deployed Content Mining application and export it. You can then import the classifier and apply it to your collection as an enrichment. For more information, see [Creating and applying a document classifier](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-doc-classifier).

The text classifier uses Part of Speech information regardless of whether the Part of Speech enrichment is applied to the project.

Text classifiers that you add to one project can be used by other projects, including Content Mining projects.

A text classifier does not classify the target text field with confidence scores that are lower than 0.5. You cannot change the confidence threshold that is used by the text classifier. If you expected certain types of passages to be classified that weren't, you can add passages with similar characteristics to your training data and train another classifier.





 Text classifier limits 

The number of text classifiers and labels that you can create per service instance depends on your Discovery plan type.



Text classifier plan limits

 Limit Plus Enterprise Premium Cloud Pak for Data 

 Number of text classifiers per service instance 5 20 20 Unlimited 
 Number of labeled data rows 2,000 20,000 20,000 20,000 
 Maximum size in MB of training data after enrichment 16 1,024 1,024 1,024 
 Number of labels 100 1,000 1,000 1,000
See also [lemma](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-glossarygloss_L) and [surface form](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-glossarygloss_S).
* dictionary pre-annotator

A component that identifies mentions in text that match a specific set of words. By using domain-specific terminology to pre-annotate text, dictionary pre-annotators can accelerate a human annotator's ability to prepare a set of ground truth documents.
* document set

A collection of documents. Documents that are imported together become a document set. Annotated documents that are grouped together for training purposes (Test, Train, Blind) are generated as document sets.







 E 



* entity



1. A mention that is annotated by an entity type.
2. A person, object, or concept about which information is stored.
3. A set of details that are held about a real-world object such as a person, location, or bank account. An entity is a kind of item.



* entity type

The type of entity that a mention represents without consideration for context. For example, the mention IBM might be annotated by the entity type ORGANIZATION.

In an entity-relationship model, an entity type is the thing that is being modeled or the thing that a mention refers to, such as the name of a person or place. Different entity types have different sets of attributes such as "surname" or "home town", and are connected through relationships like "lives in". An entity type exists independently and can be uniquely identified.







 F 



* F1 score

A measure of a test's accuracy that considers both precision and recall to compute the score. The F1 score can be interpreted as a weighted average of the precision and recall values. An F1 score reaches its best value at 1 and worst value at 0.
* false negative

An answer or annotation that is correct, but was predicted to be incorrect.
* false positive
Troubleshooting the ground truth editor 

Find out what to do if you have questions or need help. Where to go for help depends on the type of issue you are having.



 Access issues 



* If you cannot access the Knowledge Studio instance, then work with the workspace owner to ensure that they have invited you to join the workspace.
* If you can get to the Knowledge Studio instance, but do not see any workspaces listed, then work with the administrator or manager of the workspace to resolve the issue. The administrator, which is a person with the Admin role, or manager, which is a person with the Project Manager role in Knowledge Studio, must 1.) associate you with an annotation set and 2.) create a task that assigns you to annotate the documents in the set. You will not see the workspace listed on the Knowledge Studio home page until someone has performed these actions.







 Using the ground truth editor 

First, review the information in [Annotating documents](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-user-guide). If you still have questions about how to use the ground truth editor, check one of the developer communities, which are linked to from the bottom of the table of contents. If you can't find the answer to your question, post the question yourself.





 Annotation decisions 

Can't decide which entity type to assign to an ambiguous term?



* If the workspace has annotation guidelines, refer to them. The purpose of the guideline is to help you answer just such types of questions. If you know there are guidelines but cannot find them, then ask the workspace administrator to add a link to the annotation guidelines from the ground truth editor. If there is no guideline, consider starting one. The first entry can address the question you are debating now, and can describe the resolution that you come to. This entry will save time for the next human annotator who stumbles upon the same issue.
* Discuss the question with other human annotators involved in the workspace, and try to decide together on the optimal solution. This type of discussion is expected and beneficial to the workspace overall.
Troubleshooting the ground truth editor 

Find out what to do if you have questions or need help. Where to go for help depends on the type of issue you are having.



 Access issues 



* If you cannot access the Knowledge Studio instance, then work with the workspace owner to ensure that he has invited you to join the workspace.
* If you can get to the Knowledge Studio instance, but do not see any workspaces listed, then work with the administrator or manager of the workspace to resolve the issue. The administrator, which is a person with the Admin role, or manager, which is a person with the Project Manager role in Knowledge Studio, must 1.) associate you with an annotation set and 2.) create a task that assigns you to annotate the documents in the set. You will not see the workspace listed on the Knowledge Studio home page until someone has performed these actions.







 Using the ground truth editor 

First, review the information in [Annotating documents](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-user-guide). If you still have questions about how to use the ground truth editor, check one of the developer communities, which are linked to from the bottom of the table of contents. If you can't find the answer to your question, post the question yourself.





 Annotation decisions 

Can't decide which entity type to assign to an ambiguous term?



* If the workspace has annotation guidelines, refer to them. The purpose of the guideline is to help you answer just such types of questions. If you know there are guidelines but cannot find them, then ask the workspace administrator to add a link to the annotation guidelines from the ground truth editor. If there is no guideline, consider starting one. The first entry can address the question you are debating now, and can describe the resolution that you come to. This entry will save time for the next human annotator who stumbles upon the same issue.
* Discuss the question with other human annotators involved in the workspace, and try to decide together on the optimal solution. This type of discussion is expected and beneficial to the workspace overall.
The following types are supported:<br><br><br><br> * Arabic: Contains a sequence of Arabic characters<br> * ChineseNumeral: Contains only Chinese numerals<br> * ClauseEndingPunctuation: Punctuation characters that separate one clause or sentence from the next<br> * Han: Contains Han characters<br> * Hangul: Contains Korean Hangul syllabic characters<br> * Hebrew: Contains a sequence of Hebrew characters<br> * Hiragana: Contains Japanese Hiragana syllabic characters<br> * Ideographic: Contains an ideogram, or symbol representing an idea or thing<br> * Katakana: Contains Japanese Katakana syllabic characters<br> * Lowercase: Contains only lower case alphabetic characters<br> * Numeric: Contains only numeric characters<br> * Punctuation: One or more characters that provide punctuation in the text<br> * Syllabic: Contains syllabic characters<br> * Thai: Contains Thai characters<br> * Titlecase: Starts with a single upper case alphabetic character, followed by one or more lower case alphabetic characters<br> * Uppercase: A token containing only upper case alphabetic characters<br><br><br> 



* Rule Match:

Table 4. Rule matching



 Setting option Description 

 Rule Match Must match the named class. Remember, a class can be derived from a regex, a dictionary, or a rule. If the class specified here was derived from a regular expression, for example, then this token must match the search pattern of the expression. 





11. For tokens that have annotations that were added indirectly from a dictionary annotation or regular expression match, you can choose whether the pattern should require any word with the same annotation type or the actual underlying words that were annotated instead.

In the lower layer of cells, you can see which cells are included in the pattern because a horizontal line connects them to one another. Where an annotation has been applied, there is a split. Cells with the original words are displayed below a cell with the annotation label. You can click one set of cells or the other to change the path of the line, and thus change the cells that are included in the pattern.
JAVAX.ANNOTATION.CONCURRENT [JSR305] ((c) 2016 SPDX Workgroup a Linux Foundation Collaborative Project)([https://repo1.maven.org/maven2/com/google/code/findbugs/jsr305/1.3.9/](https://repo1.maven.org/maven2/com/google/code/findbugs/jsr305/1.3.9/))





 Creative Commons Attribution 3.0 

The Program includes some or all of the following works licensed under the Creative Commons. The URL to the license is [https://creativecommons.org/licenses/by/3.0/legalcode](https://creativecommons.org/licenses/by/3.0/legalcode).



* PHP_MAN.HTML [elazarl/goproxy (go)] (Copyright (c) 2012 Elazar Leibovich)([https://github.com/elazarl/goproxy/tree/49ad98f6dac1e021cff612afa5a9d95843efde52](https://github.com/elazarl/goproxy/tree/49ad98f6dac1e021cff612afa5a9d95843efde52))
* SPDX [npm] (Copyright (c) npm, Inc. and Contributors)([http://registry.npmjs.org/npm/-/npm-6.14.6.tgz](http://registry.npmjs.org/npm/-/npm-6.14.6.tgz))
pre-annotation 

The process of annotating a set of documents prior to human annotation. Documents can be pre-annotated by using a rule-based model, a machine-learning model, IBM Watson Natural Language Understanding, or a dictionary. Pre-annotation can help human annotators more quickly prepare a set of ground truth documents.





 precision 

A measurement that specifies the proportion of results that are relevant. Precision, which is a positive predictive value, is determined by the number of correct positive results divided by the number of all positive results. Accuracy is best measured by using both precision and recall. See also [recall](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx2154357), [accuracy](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx3125742).





 private cloud 

A cloud computing environment in which access is limited to members of an enterprise and partner networks. See also [public cloud](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx4585370).





 private image repository 

The combination of an organization's IBM Cloud registry and its namespace. The private image repository is used when referencing an image in a command. See also [image](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx2024928), [namespace](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx2031005).





 private key 

An algorithmic pattern used to encrypt messages that only the corresponding public key can decrypt. The private key is also used to decrypt messages that were encrypted by the corresponding public key. The private key is kept on the user system and is protected by a password.





 private resource 

An entry that is visible only to account owners and their included accounts. When resources are created, they are private by default.
However, they do not always match one-to-one with words; other textual elements are considered tokens in some situations. For example, punctuation at the end of a sentence is considered a token, and contractions are often expanded into two tokens. In languages that do not use white spaces, such as Chinese, more complicated statistical algorithms are used to identify the tokens.

The tokenization process is important because it determines the groups of characters that users can highlight for annotation in the ground truth editor. Annotations of entity and relation mentions are generally aligned with token boundaries, and must be labeled within a sentence; they cannot span sentence boundaries.



 Supported types 

Knowledge Studio supports the following tokenizers:



* Machine learning-based tokenizer (default)

This is a more advanced tokenizer that identifies the tokens in the source documents based on the statistical learning it has done in the language of the source documents. This tokenizer finds tokens that capture the more natural and nuanced patterns of language. You cannot customize this tokenizer.
* Dictionary-based tokenizer

This tokenizer is based on linguistic dictionaries. It finds tokens that follow the rules of the source document language. Only advanced users can customize this tokenizer.



You must choose the tokenizer that you want to use when you create the workspace. You cannot switch to a different tokenizer later. For best results, use the default tokenizer. Only advanced users who want to modify the tokenizer behavior through a deterministic dictionary mechanism can choose the dictionary-based tokenizer. They can then customize it by adding new entries to the dictionary. However, customization must be done carefully because when you add new words to the dictionary, the changes can impact the machine learning model in unintended ways.







 Summary of inputs, outputs, and limitations 

Different stages of model development require different inputs and produce different outputs.

For each stage of the model development process, this table summarizes the typical activities that you perform, the supported input file formats, the outputs that can be produced, and any sizing limits or other requirements.



 All model types 

Table 1: All model types



 Task Typical usage Supported input formats Supported output formats Limits and requirements
Log in to Knowledge Studio as a user who is assigned to the annotation task that you created in [Lesson 3](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks_tutml_introwks_tutless_ml2).

If you have access only to a single administrator ID for this tutorial, you can use that ID to perform human annotation. However, remember that in a realistic scenario, human annotation is performed by different users with the Human Annotator role.
2. Open the My workspace workspace and click Machine Learning Model > Annotations.
3. Click the Annotation Tasks tab, then open the Test annotation task you created in [Lesson 3](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks_tutml_introwks_tutless_ml2).
4. Click Annotate for one of the assigned annotation sets.

Depending on how you set up the annotation tasks, you could have one or more annotation tasks assigned to the user ID you logged in with.
5. From the list of documents, find the Technology - gmanews.tv document and open it.

Notice that the term IBM was already annotated with the ORGANIZATION entity type. This annotation was added by the dictionary pre-annotator that was applied in [Lesson 2](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks_tutml_introwks_tutless_ml3). This pre-annotation is correct, so it does not need to be modified.

![This screen capture shows an open document with an existing pre-annotation for "IBM".](images/wks_tut_preannotation.png "This screen capture shows an open document with an existing pre-annotation for "IBM".")
6. Annotate a mention:



1. Click the Entity tab.
* [Regular expressions](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-managing-projects-and-extractorscreating-a-regular-expression): syntax-based pattern matching





 Creating extractors with linguistic patterns 

Defining a new extractor using natural language processing (NLP) concepts such as tokenization and part-of-speech requires developing an extractor using the Annotation Query Language (AQL).

For more information about how to write AQL and available functions, see [Annotation Query Language reference](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-annotation-query-language-reference)





 Creating a dictionary 

A dictionary is a list of terms that relate to a concept. Use a dictionary as a convenient substitute for individual literals when analyzing text for multiple terms that fit into the same category.

You can define a dictionary as a simple list of terms or as a mapping table that maps a value or term to a preferred term. Mapping tables are typically used to resolve synonyms, abbreviations, and encoded values.

Use of a dictionary can sometimes result in overlapping matches. For example, a dictionary used to extract information about people with a military rank such as Officer, Warrant Officer, or Chief Warrant Officer might result in three overlapping matches for every occurrence of Chief Warrant Officer. See [Eliminate duplicate and overlapping results](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-managing-projects-and-extractorseliminate-duplicate-and-overlapping-results) for information needed to address this issue.



 Defining a list 

When using standardized terms such as names of departments in your organization, sales priorities, congressional committees, or military ranks to extract values, you can identify the needed terms in a list.



1. Click !