vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Open a workspace, and then click Machine Learning Model > Annotations. Click the Annotation Tasks tab. The annotation tasks that are assigned to you are displayed.
3. Open the annotation task that you want to work on. The annotation sets that are assigned to you are displayed.
4. Click Annotate to open the annotation set you want to work on. The documents in the annotation set are displayed.
5. Open the document that you want to annotate. By default, the document opens in Mention mode, which is the mode you use to annotate entity mentions.
6. Click Relations.
7. To annotate a relation:



1. Click an entity mention in the text, and then click a second entity mention in the same sentence that you want to connect to the first mention.
2. Either select the relation type that you want to apply from the pane on the right, or enter the keyboard shortcut for the relation type. The list of available relation types is constrained by the first entity mention that you select, and further constrained by the second entity mention. In some cases, only one relation type remains; you still must explicitly select the relation type to apply it.

If annotation guidelines were previously connected to the workspace, and you want help with choosing the correct annotation to apply, click View Guidelines. Depending on the access permissions set up on the site where the guidelines are hosted, you might be able to update the guidelines after you open them, for example, to add clarifications and examples.



8. To remove a relation mention that you just added, press Ctrl+Z to undo the action. To remove a relation mention later, you can left-click the relation type and then press the Delete key or click X next to the relation type.
9. Click Save at any time to save your work.







 What to do next 

After you finish annotating all entity mentions, relation mentions, and coreferences in the document, as applicable, change the document status from In Progress to Completed, click Save, and then close the document.
To use the rule-based model to pre-annotate documents, complete the following steps:



1. Log in as a Knowledge Studio administrator and select your workspace.
2. Go to the Machine Learning Model > Pre-annotation page.
3. Click the overflow menu button in the Rule-based Model row in the page, then click Map entity types and classes to map entity types that you defined in the Knowledge Studio type system to one or more rule-based model classes.

You can also open the mapping page by selecting the Rule-based Model > Versions > Rule-based Model tab.
4. Click Edit for each entity type you want to map.



* The drop-down list of the Class Name column is pre-populated with classes that are associated with the rule-based model.
* You must map at least one entity type to a class.



5. On the Machine Learning Model > Pre-annotation page, click Run Pre-annotators.

The Rule-based Model option is not available until you map at least one entity type to a class.
6. If you want to erase existing annotations made by pre-annotators before running the pre-annotator, select Wipe previous pre-annotation results. Human annotations are preserved even if this is selected.
7. Select the document sets or annotation sets that you want to pre-annotate.
8. Click Run.

Pre-annotation is applied to individual documents without regard for the various document sets that a document might belong to. A document that overlaps between a selected document set and an unselected document will appear pre-annotated in both document sets.









 Uploading pre-annotated documents 

You can jump-start the training of your model by uploading documents that were pre-annotated through an Unstructured Information Management Architecture (UIMA) analysis engine.

The pre-annotated documents must be in the XMI serialization form of UIMA Common Analysis Structure (UIMA CAS XMI).
The pre-annotated documents must be in the XMI serialization form of UIMA Common Analysis Structure (UIMA CAS XMI). The .zip file that you upload must include the UIMA TypeSystem descriptor file and a file that maps the UIMA types to entity types in your Knowledge Studio type system.

UIMA CAS XMI is a standard format of Apache UIMA. Guidelines are provided for how to create files in the correct format from analyzed collections in IBM Watson Explorer. If you use another Apache UIMA implementation, adapt these guidelines for your purposes. Regardless of how you create the XMI files, the requirements for creating the type system mapping file and .zip file are the same for everyone.

If you assign the imported documents to human annotators, the documents appear pre-annotated in the ground truth editor and a number of mentions might already be annotated. The human annotator thus has more time to focus on applying the annotation guidelines to unmarked mentions. Alternatively, you can bypass the human annotation step and use the pre-annotated documents to immediately start training and evaluating a machine learning model.



 Exporting analyzed documents from Watson Explorer Content Analytics 

You can export documents that were crawled and analyzed in IBM Watson Explorer Content Analytics, and upload the analyzed documents as XMI files into a Knowledge Studio workspace.



 Procedure 

To get analyzed documents from a Watson Explorer Content Analytics collection, follow these steps:



1. Open the Content Analytics administration console in a web browser.
2. On the Collections view, expand the collection that you want to export documents from. In the Parse and Index pane, ensure that the parse and index process is running, and then click the arrow icon for Export analyzed document content and metadata.
3. In the Analyzed document export options area, select Export documents as XML files, select the Enable CAS as XMI format export check box, specify the output path for where the exported data is to be written, and click OK.
4. Stop and restart the parse and index services for the collection, and then do one of the following steps:



* If the collection already contains indexed documents that you want to use for training the machine learning model in the document cache, restart a full index build.
Defining a rule 

Use the rule editor to define rules.



 About this task 

Avoid simultaneous editing of rules, classes, and regular expressions by multiple users because it might result in unexpected overwrites or duplication.





 Procedure 

To define a rule, complete the following steps:



1. Log in as a Knowledge Studio administrator or project manager and click Rule-based Model > Rules.
2. Click the plus sign (+) next to the Documents heading to add a document.

For more information, see [Adding documents for defining rules](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks_rule_anno_add).

For example, you might add a document named My Document that contains this single line of text:

A 50-year-old driver was driving the 2017 Example Horizon.
3. If you plan to define a regular expression or add a dictionary, create a class to associate with it.



1. From the Class panel, click the plus sign (+) next to Class.
2. Add a class name.

If you will associate the class with a regular expression or dictionary, consider naming the class such that you can identify its origin. For example, if you plan to use a regular expression to define a pattern for the age in the example sentence, you can create a class named AGE_REGEX. If you plan to use a dictionary to annotate the car manufacturer in the sentence, you can add a class named MANUFACTURER_DICT.

Keep these naming rules in mind:



* The first character in a class name must be alphabetical.
* Use only the following alphanumeric ASCII characters and the underscore character in values that you add to the classes: A through Z, a through z, 0 through 9.
* Names cannot contain spaces.
* Names cannot be longer than 64 characters.





4. Optional: To quickly annotate classes in a document, you can associate a dictionary with the rule editor. Terms in a document that match entries in the dictionary are automatically annotated with the class that you select for the dictionary.



1. Click the Dictionaries tab.
The maximum number of group cells that can participate in a pattern is 15 when each cell has a repeat setting of 1 or less. Group cells include single tokens, annotations, or tokens where any token is allowed. The maximum number of total tokens allowed in a pattern is 20. Take the repeat setting of each cell into account as you define the pattern. For example, you can define a pattern that includes 15 tokens when each cell has a repeat setting of 1 or less. But, you can define no more than 4 tokens in the pattern if each one has a repeat setting of 1 or more because for each one, the token can repeat up to 5 times. Four tokens that are repeated 5 times brings you to the allowed maximum of 20.

To indicate that a particular type of token is required, you can define the following types of condition settings:



* Repeat Setting: Specifies how many times the current token must be included in the pattern. You can change the repeat setting, but only one repeat setting can be specified per token. The options are described in the following table.

Table 1. Repeating settings



 Setting option Description 

 Required (Exactly 1) This token must exist in the pattern one time. This option is applied by default, but can be changed. 
 Repeating 1 or more times This token must exist in the pattern at least one time, and can repeat more times. 
 Repeating 0 or more times This token can optionally repeat in the pattern many times, but does not have to repeat. 
 Occurring 0 or 1 time This token is optional. 
 Advanced: custom This token must repeat in the pattern the number of times that is specified here. To define a custom repeat setting, click Open Properties, select Advanced, and then select the exact number of repeats or range of repeats that you want to define. The maximum number of repeats that are allowed for a token is 5. 



* Feature Setting: At least one of the feature settings must be defined. You can add more features to add to the number of conditions that must be met for text to match this pattern. The options are described in the following table.

Table 2. Feature settings



 Setting option Condition it adds 

 Text Must match the exact text in this token.
Classes are then mapped to entity types from the type system when you run the rule-based model to pre-annotate documents. As a result, dictionary terms are, although circuitously, mapped to entity types for the rule-based model also.



In both cases, the dictionaries provide terms that the system can find and annotate as mentions. It assigns to each mention the entity type that is associated with the dictionary that contains the term. When a human annotator begins work on new documents that were pre-annotated, many mentions are already annotated based on the dictionary entries. The human annotator thus has more time to focus on assigning entity types to mentions that require deeper analysis.







 Language considerations 



* For Brazilian Portuguese, English, French, German, Italian, and Spanish, Knowledge Studio does not currently provide an option to specify case-insensitive dictionary-matching, but dictionary entries match text that has a higher case. For example, vehicle in the dictionary matches vehicle, Vehicle or VEHICLE in text, while Sat in the dictionary matches Sat or SAT in text, but not sat.
* For Japanese and Korean, dictionary matching during pre-annotation is case-sensitive.
* For Arabic, Knowledge Studio assumes that Arabic text is stored unshaped and treats numeric shaping as a storage-level property. For details about how Knowledge Studio handles Arabic character shaping and numeric shaping, see [Configuring support for Arabic](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-wks_langsupp_ar).







 CSV file dictionary 

Also referred to as the standard dictionary format, a dictionary in comma-separated value (CSV) format is a file that you can edit after you upload it. The maximum size of a CSV file that you can upload is 1 MB. If you have a larger dictionary file, then break the large file into multiple files and upload them one at a time into a single dictionary in your Knowledge Studio workspace.
The annotation is added to the entity you associated it with, and the system begins to train itself on the new data.

The term you annotated is added to the entity as a new dictionary value. If you associated the annotated term with an existing entity value, then the term is added as a synonym of that entity value instead of as an independent entity value.
9. To see all of the mentions you annotated for a particular entity, from the entity's configuration page, click the Annotation tab.

![Annotation view selector highlighted](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/oe-annotate2.png)

Contextual entities understand values that you have not explicitly defined. The system makes predictions about additional entity values based on how your user examples are annotated, and uses those values to train other entities. Any similar user examples are added to the Annotation view, so you can see how this option impacts training.

If you do not want your contextual entities to use this expanded understanding of entity values, select all the user examples in the Annotation view for that entity, and then click Delete.



To walk through a tutorial that shows you how to define contextual entities before you add your own, go to [Tutorial: Defining contextual entities](https://www.ibm.com/cloud/garage/demo/try-watson-assistant-contextual-entities).





 Enabling system entities 

Watson Assistant provides a number of system entities, which are common entities that you can use for any application. Enabling a system entity makes it possible to quickly populate your skill with training data that is common to many use cases.

System entities can be used to recognize a broad range of values for the object types they represent. For example, the @sys-number system entity matches any numerical value, including whole numbers, decimal fractions, or even numbers written out as words.

System entities are centrally maintained, so any updates are available automatically. You cannot modify system entities.



1.
The following example illustrates how aggregate functions can count the number of person name annotations, or compute sets of given names that are associated with each distinct surname that is identified in a document:

-- identify occurrences of given names in the document
create view FirstName as
extract dictionary 'firstNames.dict' on D.text as name
from Document D;

-- identify occurrences of surnames in the document
create view LastName as
extract dictionary 'lastNames.dict' on D.text as name
from Document D;

-- identify complete person names in the document
create view Person as
select F.name as firstname, L.name as lastname
from FirstName F, LastName L
where FollowsTok(F.name, L.name, 0, 0);

-- count the number of person annotations in the document
create view CountPerson as
select Count()
from Person;

-- for each distinct surname, output a list of given names associated with it in the document
create view FamilyMembers as
select GetText(P.lastname) as lastname, List(GetText(P.firstname)) as firstnames
from Person P
group by GetText(P.lastname);
Show more

The following example illustrates the use of the Min and Max functions:

-- Extract stop words from input text
create view StopWords as
extract
regex /s(the|in|a|an|as|to|from)s/ on D.text as match
from Document D;

-- Count the number of times each stop word matched above, was used in the text
create view StopWordsCount as
select
GetText(S.match) as stopword,
Count(S.match) as stopwordcount
from StopWords S
group by GetText(S.match);

-- Retrieve the most used and least used stop word count
create view StopWordUsageCount as
select Min(S.stopwordcount) as least, Max(S.stopwordcount) as most
from StopWordsCount S;
</xsd:enumeration>
<xsd:enumeration value="median">
<xsd:annotation>
<xsd:documentation>Compute the median value of the values of the group.</xsd:documentation>
</xsd:annotation>
</xsd:enumeration>
<xsd:enumeration value="minimum">
<xsd:annotation>
<xsd:documentation>Compute the minimum value of the values of the group.</xsd:documentation>
</xsd:annotation>
</xsd:enumeration>
<xsd:enumeration value="standardDeviation">
<xsd:annotation>
<xsd:documentation>Compute the standard deviation value of the values of the group.</xsd:documentation>
</xsd:annotation>
</xsd:enumeration>
<xsd:enumeration value="total">
<xsd:annotation>
<xsd:documentation>Compute the sum of the values of the group.</xsd:documentation>
</xsd:annotation>
</xsd:enumeration>
<xsd:enumeration value="variance">
<xsd:annotation>
<xsd:documentation>Compute the variance value of the values of the group.</xsd:documentation>
</xsd:annotation>
</xsd:enumeration>
<xsd:enumeration value="doNotUse">
<xsd:annotation>
<xsd:documentation>The related item must not be used as a grouping item and must not be aggregated. This value is used for the item that represents the property of a level of a dimension.</xsd:documentation>
</xsd:annotation>
</xsd:enumeration>
</xsd:restriction>
</xsd:simpleType>
Classes are then mapped to entity types from the type system when you run the rule-based model to pre-annotate documents. As a result, dictionary terms are, although circuitously, mapped to entity types for the rule-based model also.



In both cases, the dictionaries provide terms that the system can find and annotate as mentions. It assigns to each mention the entity type that is associated with the dictionary that contains the term. When a human annotator begins work on new documents that were pre-annotated, many mentions are already annotated based on the dictionary entries. The human annotator thus has more time to focus on assigning entity types to mentions that require deeper analysis.







 Language considerations 



* For Brazilian Portuguese, English, French, German, Italian, and Spanish, Knowledge Studio does not currently provide an option to specify case-insensitive dictionary-matching, but dictionary entries match text that has a higher case. For example, vehicle in the dictionary matches vehicle, Vehicle or VEHICLE in text, while Sat in the dictionary matches Sat or SAT in text, but not sat.
* For Japanese and Korean, dictionary matching during pre-annotation is case-sensitive.
* For Arabic, Knowledge Studio assumes that Arabic text is stored unshaped and treats numeric shaping as a storage-level property. For details about how Knowledge Studio handles Arabic character shaping and numeric shaping, see [Configuring support for Arabic](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks_langsupp_ar).







 CSV file dictionary 

Also referred to as the standard dictionary format, a dictionary in comma-separated value (CSV) format is a file that you can edit after you upload it. The maximum size of a CSV file that you can upload is 1 MB. If you have a larger dictionary file, then break the large file into multiple files and upload them one at a time into a single dictionary in your Knowledge Studio workspace.
10. Click Versions. On the Versions page, you can take a snapshot of the model and the resources that were used to create it (except for dictionaries and annotation tasks). For example, you might want to take a snapshot before you retrain the model. If the statistics are poorer the next time you train it, you can promote the older version and delete the version that returned poorer results.







 Results 

You created a machine learning model, trained it, and evaluated how well it performed when annotating test data and blind data. By exploring the performance metrics, you can identify ways to improve the accuracy of the machine learning model.







 Tutorial summary 

You created a machine learning model.



 Lessons learned 

By completing this tutorial, you learned about the following concepts:



* Document sets
* Machine learning models
* Human annotation tasks
* Inter-annotator agreement and adjudication
You might need to make modifications while you train a model, based on the performance statistics. But, generally, you want the type system to be as close to final as possible before you begin large-scale annotation tasks. If you change the type system after human annotators began their work, they must revisit the documents that they annotated. They must assess the applicability of the type system changes.



 About this task 

This process propagates the current type system, ground truth editor keyboard shortcuts, and color settings to all document sets in a task.





 Procedure 

To modify the type system without losing the work that was done by human annotators:



1. Change the type system. For example, you can add or remove entity types or relation types.
2. Decide whether you want to propagate the changes to existing human annotation tasks.
3. Open the Machine Learning Model > Annotations page and click the Annotation Tasks tab. Open each task that you want to update and click Apply Type System Updates.

If you removed entity types or relation types from the type system, all occurrences of those types are highlighted in gray in the documents. These invalid types are ignored by the machine learning model. They do not prevent you from submitting and approving document sets.
4. Provide details to the human annotators about what changed in the type system.
5. Ask human annotators to update their documents to reflect the changes in the type system. For example, if you added new entity types or relation types, they must review their documents and annotate them appropriately.

> Note: If the task contains completed documents, human annotators cannot alter those documents to assess type system changes until they are back in an editable state. To become editable, ask human annotators to submit the document sets so that you can reject them.



Related concepts:

[Type systems](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-typesystemwks_typesystem)







 Document set management 

Use the right sets of data to test and train the model at the right time.