vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Pre-annotating documents 

This tutorial helps you understand how to pre-annotate documents, which bootstraps the annotation process of human annotation.



 Learning objectives 

After you complete this tutorial, you will know how to pre-annotate documents with a machine learning model.

This tutorial should take approximately 5 minutes to finish. If you explore other concepts related to this tutorial, it could take longer to complete.





 Before you begin 



* You're using a supported browser. For more information, see [Browser requirements](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-system-requirements).
* You successfully completed [Getting started with Knowledge Studio](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks_tutintro), which covers creating a workspace, creating a type system, and adding a dictionary.
* You successfully completed [Creating a machine learning model](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks_tutml_intro).
* You must have at least one user ID in either the Admin or Project Manager role. For more information about user roles, see [User roles in Knowledge Studio](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-roles).







 Results 

After completing this tutorial, you will have a set of partially annotated documents. Then, you can assign the documents to human annotators to finish the annotation work.





 Lesson 1: Pre-annotating new documents with a machine learning model 

In this lesson, you will learn how to use a machine learning model to pre-annotate documents in Knowledge Studio.



 About this task 

After you train a machine learning model, you can use it to pre-annotate new documents that you add to the corpus.

Do not run a pre-annotator on documents that have been annotated by humans, but not been added to the ground truth yet.
For more information about creating an annotation task, see [Annotation setup](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-annotate-documents).
6. To view the annotations that were applied by the machine learning model to the new documents, open the annotation task.

Because the new documents were pre-annotated with the machine learning model, human annotation requires less time. For more information about adding annotations by human annotators, see [Annotating documents](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-user-guide).







 Results 

By using your machine learning model to pre-annotate new document sets, you can expedite human annotation tasks for those documents.
For example, in a domain about auto racing, it makes sense to include the term bank, which refers to a road feature, only if financial institutions are not frequently discussed in the text also. If both senses of the word occur often in the source documents, then it is best to leave it out of both types of dictionaries: the dictionary that is associated with road features, and the dictionary that is associated with financial institutions.

You can create dictionaries in Knowledge Studio by manually adding individual entries. Knowledge Studio also supports the ability to upload several types of dictionary files.



 How are dictionaries used? 

Dictionaries are used in a couple ways, all optional. They are used by the machine learning model to provide words or phrases that are equivalent for information-extraction purposes and during pre-annotation to bootstrap the annotation effort.



* Machine learning usage

The entity type that you associate with a dictionary is not used to define rules for the machine learning model. Machine learning evaluates mentions in the documents independently. It does not assume that a mention has a specific entity type just because the mention matches an entry in a dictionary that is associated with that entity type. It does take the information into account, but treats it as one piece of information among other pieces of information that it gathers through linguistic analysis. In fact, if none of the terms in a dictionary occur in the ground truth documents, then the dictionary is not used at all by the machine learning model.
* Pre-annotation usage

Dictionaries are important to the following pre-annotatation processes.



* Dictionary pre-annotator: You associate a dictionary with an entity type from the type system when you run the dictionary pre-annotator.
* Rule-based model: You can optionally associate a dictionary with a rule class. Classes are then mapped to entity types from the type system when you run the rule-based model to pre-annotate documents. As a result, dictionary terms are, although circuitously, mapped to entity types for the rule-based model also.



In both cases, the dictionaries provide terms that the system can find and annotate as mentions. It assigns to each mention the entity type that is associated with the dictionary that contains the term.
As another example, if a PDF document contains a very long word that is hyphenated at the end of a line, that word might be converted as two words. In cases like these, the documents might not be suitable for machine learning unless you pre-process the files to fix formatting limitations.





 Documents from another Watson Knowledge Studio workspace 

If you previously downloaded documents from a Knowledge Studio workspace, you can upload the .zip file that you downloaded. An option lets you specify whether you want the ground truth annotations to be included in the imported files.

After documents are annotated, the annotated documents are stored in JSON format. The markup language in these files, which shows how the original document text was parsed and tokenized, includes elements for all of the annotations that a human annotator added. To improve model accuracy over time, you can upload these files into another workspace, thus preserving all of the existing annotations. A human annotator can revise, delete, and add annotations to these documents, or you can bypass human annotation and use these files to create training, test, and blind document sets for evaluating and improving the model performance.





 UIMA CAS XMI files 

To help train a model, you can upload documents that were pre-annotated by a UIMA analysis engine. The pre-annotated files must be in XMI serialization of UIMA Common Analysis Structure (UIMA CAS XMI) format and combined into a .zip file. For example, you can upload documents that were annotated in an IBM Watson Explorer collection.

A human annotator can revise, delete, and add annotations to these documents, or you can bypass human annotation and use these files to create training, test, and blind document sets for evaluating and improving the model performance. For more information about how to create these files and requirements for uploading them, see [Uploading pre-annotated documents](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-preannotationwks_uima).





 Anonymizing data
(images/rule-anno2.png "Shows the user clicking Edit for the "DATE" entity type in the "Rule-based model type mapping" tab.")
2. Choose the RuleDate class from the list and click Save.

![Shows the user choosing the "RuleDate" class from the list.](images/rule-anno3.png "Shows the user choosing the "RuleDate" class from the list.")



3. To pre-annotate document sets or annotation sets with the rule-based model, select the Rule-based Model tab and click Run this model.

Attention: Run the rule-based model as a pre-annotator only on documents that were not already annotated by humans.









 Tutorial summary 

While learning about Knowledge Studio, you created a rule-based model.



 Lessons learned 

By completing this tutorial, you learned about the following concepts:



* Classes
* Regular expressions
* Rules
Changed the button label from Run to Pre-annotate, changed the title of the window from Run Annotator to Run Pre-annotation, and changed the error message to clarify that you can't add automated annotations after humans annotated the documents.

Dictionary-based tokenizer fix
: For projects or workspaces that use dictionary-based tokenizers, fixed an issue that showed empty sentences if you imported documents without ground truth.







 September 2017 



 New features and changes 

New front-end experience released
: Released a new front-end user experience for Knowledge Studio on IBM Cloud® as an [experimental](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks-faqsexperimental) service. Changes include a reorganized navigation and a new model performance analysis page. For a summary of the navigation changes, see the table in the known issues.

Support for Chinese (Simplified) and Dutch
: Added Chinese (simplified) and Dutch language support.





 Known issues 

Export button not enabled
: For a rule-based model, after you map classes and entity types, the Export button is not enabled until you refresh the browser window.

Experimental Cloud release
: For the IBM Cloud [experimental](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks-faqsexperimental) release, some of the documentation terminology does not match the new interface. The documentation matches the interface in IBM Marketplace. The following terms have changed in the [experimental](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks-faqsexperimental) release:



Table 1. Terminology changes for IBM Cloud version

 Term in IBM Marketplace Term in IBM Cloud Notes 

 project workspace This term was changed because IBM Cloud also uses the term project 
 import and export upload and download The terms import and export are now referred to as upload and download when used in terms of documents and entity types. The term export is still used when referring to exporting a model to applications such as Watson Explorer.
For example, you might want to take a snapshot before you retrain the model. If the statistics are poorer the next time you train it, you can promote the older version and delete the version that returned poorer results.







 Results 

You created a machine learning model, trained it, and evaluated how well it performed when annotating test data and blind data. By exploring the performance metrics, you can identify ways to improve the accuracy of the machine learning model.







 Tutorial summary 

You created a machine learning model.



 Lessons learned 

By completing this tutorial, you learned about the following concepts:



* Document sets
* Machine learning models
* Human annotation tasks
* Inter-annotator agreement and adjudication
A document that overlaps between a selected document set and an unselected document will appear pre-annotated in both document sets.









 Uploading pre-annotated documents 

You can jump-start the training of your model by uploading documents that were pre-annotated through an Unstructured Information Management Architecture (UIMA) analysis engine.

The pre-annotated documents must be in the XMI serialization form of UIMA Common Analysis Structure (UIMA CAS XMI). The .zip file that you upload must include the UIMA TypeSystem descriptor file and a file that maps the UIMA types to entity types in your Knowledge Studio type system.

UIMA CAS XMI is a standard format of Apache UIMA. Guidelines are provided for how to create files in the correct format from analyzed collections in IBM Watson Explorer. If you use another Apache UIMA implementation, adapt these guidelines for your purposes. Regardless of how you create the XMI files, the requirements for creating the type system mapping file and .zip file are the same for everyone.

If you assign the imported documents to human annotators, the documents appear pre-annotated in the ground truth editor and a number of mentions might already be annotated. The human annotator thus has more time to focus on applying the annotation guidelines to unmarked mentions. Alternatively, you can bypass the human annotation step and use the pre-annotated documents to immediately start training and evaluating a machine learning model.



 Exporting analyzed documents from Watson Explorer Content Analytics 

You can export documents that were crawled and analyzed in IBM Watson Explorer Content Analytics, and upload the analyzed documents as XMI files into a Knowledge Studio workspace.



 Procedure 

To get analyzed documents from a Watson Explorer Content Analytics collection, follow these steps:



1. Open the Content Analytics administration console in a web browser.
2. On the Collections view, expand the collection that you want to export documents from. In the Parse and Index pane, ensure that the parse and index process is running, and then click the arrow icon for Export analyzed document content and metadata.
3.
Rules 

Create a rule-based model that can recognize patterns in your documents. Use rules to capture patterns that occur in documents and convey information about underlying entity types.



 Class overview 

When you construct a rule, you use classes to represent types of information. These classes are similar to entity types. So, why don't we just use entity types when defining rules? Because as you build rules, you can define intermediate classes that are used only to build other more complex classes. These intermediate classes are solely utilitarian. They are not useful on their own. Intermediate classes work with other intermediate classes to define a more useful and complete class. An intermediate class is necessary, but not something you expose as part of a type system. To enable the rule-based model to do useful things like pre-annotate documents with entity mentions, you must map the complex classes that you use during rule creation to their equivalent entity types from the type system.

For example, you want a model that can recognize people's names. To train a machine learning model to recognize people's names, you would annotate many different names that are written in a variety of formats in documents in an annotation set with the PERSON entity type, and train a model to recognize people's names. To create a rule-based model to recognize people's names, you define a rule that describes the text patterns used to write people's names. So, you might create a FirstName class and a LastName class and use these intermediate classes to define a FullName class. You might define conditions that determine the placement of the FullName class in relation to common prefixes, such as Dr. and common suffixes, such as Jr.. When the rule-based model is used, the FullName class is mapped to the PERSON entity type.

Another reason to avoid mapping intermediate classes to entities in your type system is that if you pre-annotate documents with the rule-based model, and then add them to your ground truth for training a machine learning model, you do not want to define rules in such a way that they will produce overlapping entity mentions.
In English, tokens are often equated to words as delimited by white spaces in a sentence. However, they do not always match one-to-one with words; other textual elements are considered tokens in some situations. For example, punctuation at the end of a sentence is considered a token, and contractions are often expanded into two tokens. In languages that do not use white spaces, such as Chinese, more complicated statistical algorithms are used to identify the tokens.

The tokenization process is important because it determines the groups of characters that users can highlight for annotation in the ground truth editor. Annotations of entity and relation mentions are generally aligned with token boundaries, and must be labeled within a sentence; they cannot span sentence boundaries.



 Supported types 

Knowledge Studio supports the following tokenizers:



* Machine learning-based tokenizer (default)

This is a more advanced tokenizer that identifies the tokens in the source documents based on the statistical learning it has done in the language of the source documents. This tokenizer finds tokens that capture the more natural and nuanced patterns of language. You cannot customize this tokenizer.
* Dictionary-based tokenizer

This tokenizer is based on linguistic dictionaries. It finds tokens that follow the rules of the source document language. Only advanced users can customize this tokenizer.



You must choose the tokenizer that you want to use when you create the workspace. You cannot switch to a different tokenizer later. For best results, use the default tokenizer. Only advanced users who want to modify the tokenizer behavior through a deterministic dictionary mechanism can choose the dictionary-based tokenizer. They can then customize it by adding new entries to the dictionary. However, customization must be done carefully because when you add new words to the dictionary, the changes can impact the machine learning model in unintended ways.







 Summary of inputs, outputs, and limitations 

Different stages of model development require different inputs and produce different outputs.

For each stage of the model development process, this table summarizes the typical activities that you perform, the supported input file formats, the outputs that can be produced, and any sizing limits or other requirements.



 All model types
From the newly imported and unannotated documents, [create annotation sets](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-documents-for-annotation).
3. [Recreate the annotation tasks](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-annotate-documentswks_hatask). Give the task the same name, an appropriate due date, and assign annotation sets to the appropriate human annotators.
If not already completed, open the overflow menu in the Rule-based Model row and click Map entity types and classes to map entity types that you defined in the Knowledge Studio type system to one or more rule-based model classes.
5. Click Edit for each entity type you want to map.



* The drop-down list of the Class Name column is pre-populated with classes that are associated with the rule-based model.
* You must map at least one entity type to a class.



6. On the Machine Learning Model > Pre-annotation page, click Run Pre-annotators.

The Rule-based Model option is not available until you map at least one entity type to a class.
7. If you want to erase existing annotations made by pre-annotators before running the pre-annotator, select Wipe previous pre-annotation results. Human annotations are preserved even if this is selected.
8. Select the document sets or annotation sets that you want to pre-annotate.
9. Click Run.

Pre-annotation is applied to individual documents without regard for the various document sets that a document might belong to. A document that overlaps between a selected document set and an unselected document will appear pre-annotated in both document sets.









 Uploading pre-annotated documents 

You can jump-start the training of your model by uploading documents that were pre-annotated through an Unstructured Information Management Architecture (UIMA) analysis engine.

The pre-annotated documents must be in the XMI serialization form of UIMA Common Analysis Structure (UIMA CAS XMI). The .zip file that you upload must include the UIMA TypeSystem descriptor file and a file that maps the UIMA types to entity types in your Knowledge Studio type system.

UIMA CAS XMI is a standard format of Apache UIMA. Guidelines are provided for how to create files in the correct format from analyzed collections in IBM Watson Explorer. If you use another Apache UIMA implementation, adapt these guidelines for your purposes.
Click the overflow menu button in the Natural Language Understanding row, and then click Map entity types.



* The drop-down list of the Natural Language Understanding entity types is pre-populated with entity types that are recognized by the Natural Language Understanding service.
* You must map at least one entity type.
* You cannot map an Natural Language Understanding entity type to a Knowledge Studio entity role, only Knowledge Studio entity types.
* You can map more than one Natural Language Understanding entity type to a single Knowledge Studio entity type, or the other way around. For example, the following mappings are permitted:



Table 1. Sample mapping of entity types



 Watson Knowledge Studio Entity Type Natural Language Understanding Entity Type 

 ENGINEER <br> <br>SCIENTIST Person 
 LOCATION CityTown <br> <br>Country 



4. After mapping all the entity types that you want to apply, go the Machine Learning Model > Pre-annotation page. Click Run Pre-annotators.
5. Select Natural Language Understanding, and then click Next.

The Natural Language Understanding annotator is not available until you map at least one entity type.
6. ore running the pre-annotator, select Wipe previous pre-annotation results. Human annotations are preserved even if this is selected.
7. Select the check box for each document set that you want to pre-annotate.

If you are running this pre-annotator for the first time, first validate that the pre-annotator can find mentions of the mapped entities as expected. Create one document set that contains a representative document or documents from each distinct data source.
8. Click Run.

If you are doing a validation check of the pre-annotator, then open the annotated documents and review the annotations that were added. Make sure a sufficient number of accurate annotations were created. If the annotations are accurate, then you can run the annotator again on more and larger document sets. If the annotations are not accurate, then consider mapping different Natural Language Understanding entity types to your types.
)







 What to do next 

After you create a dictionary, you can use it to speed up human annotation tasks by pre-annotating the documents.







 Next steps 

Learn how to [create a machine learning model](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-wks_tutml_intro).





 Tutorial summary 

While learning about Knowledge Studio, you created a workspace and added artifacts to it.



 Lessons learned 

By completing this tutorial, you learned about the following concepts:



* Workspaces
* Type systems
* Dictionaries