vania-janet/multiturn-rag-retrieval-data huggingface dataset:
* Involving domain subject matter experts to create the following resources, or to identify existing resources that can be re-used or modified for your domain:



* Annotation guidelines and examples to help human annotators learn how words and passages in your domain content are to be annotated.
* Type systems that define the domain-specific types (objects) and features (data classifications) that can be discovered in your domain content through text analysis. The type system controls the types of annotations that a human annotator can add to documents.
* Dictionaries of terms that are to be treated as equivalent terms in your domain content.



* Creating a corpus of documents that are representative of your domain content.
* Dividing documents among human annotators, who then use the IBM Watson™ Knowledge Studio ground truth editor tool to manually add annotations to small sets of documents.
* Comparing the human annotation results and resolving conflicts. Adjudication in this phase is needed to ensure accurate and consistently annotated documents are promoted to ground truth, where they can be used to train and test a machine learning model.







 Model development 

This stage refers to the use of Knowledge Studio tools to create a model. After establishing ground truth, the human annotation results can be used to train an algorithm for automatically adding annotations to large collections of documents, such as collections that include millions of documents.





 Model evaluation 

This stage refers to the use of Knowledge Studio tools to refine the model and improve performance. The results generated by the model are evaluated against a test set of ground truth documents. Accuracy analysis identifies the causes of annotation errors. Headroom analysis helps you assess which errors require focus and where model refinements can yield the greatest impact. Adjustments can be made repeatedly to improve performance until a satisfactory level of accuracy is achieved.





 Model deployment 

This stage refers to exporting components that enable the model to run in machine learning runtime environments and making the model accessible to other Watson cognitive applications. For example, you can export the machine learning model for use by IBM Watson™ Natural Language Understanding for IBM Cloud Pak® for Data or IBM Watson™ Discovery for IBM Cloud Pak® for Data







 Creating an annotation task
Open a workspace, and then click Machine Learning Model > Annotations. Click the Annotation Tasks tab. The annotation tasks that are assigned to you are displayed.
3. Open the annotation task that you want to work on. The annotation sets that are assigned to you are displayed.
4. Click Annotate to open the annotation set you want to work on. The documents in the annotation set are displayed.
5. Open the document that you want to annotate. By default, the document opens in Mention mode, which is the mode you use to annotate entity mentions.
6. Click Relations.
7. To annotate a relation:



1. Click an entity mention in the text, and then click a second entity mention.
2. Either select the relation type that you want to apply from the pane on the right, or enter the keyboard shortcut for the relation type. The list of available relation types is constrained by the first entity mention that you select, and further constrained by the second entity mention. In some cases, only one relation type remains; you still must explicitly select the relation type to apply it.

If annotation guidelines were previously connected to the workspace, and you want help with choosing the correct annotation to apply, click View Guidelines. Depending on the access permissions set up on the site where the guidelines are hosted, you might be able to update the guidelines after you open them, for example, to add clarifications and examples.



8. To remove a relation mention that you just added, press Ctrl+Z to undo the action. To remove a relation mention later, you can left-click the relation type and then press the Delete key or click X next to the relation type.
9. Click Save at any time to save your work.







 What to do next 

After you finish annotating all entity mentions, relation mentions, and coreferences in the document, as applicable, change the document status from In Progress to Completed, click Save, and then close the document.
Defining a rule 

Use the rule editor to define rules.



 About this task 

Avoid simultaneous editing of rules, classes, and regular expressions by multiple users because it might result in unexpected overwrites or duplication.





 Procedure 

To define a rule, complete the following steps:



1. Log in as a Knowledge Studio administrator or project manager and click Rule-based Model > Rules.
2. Click the plus sign (+) next to the Documents heading to add a document.

For more information, see [Adding documents for defining rules](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-wks_rule_anno_add).

For example, you might add a document named My Document that contains this single line of text:

A 50-year-old driver was driving the 2017 Example Horizon.
3. If you plan to define a regular expression or add a dictionary, create a class to associate with it.



1. From the Class panel, click the plus sign (+) next to Class.
2. Add a class name.

If you will associate the class with a regular expression or dictionary, consider naming the class such that you can identify its origin. For example, if you plan to use a regular expression to define a pattern for the age in the example sentence, you can create a class named AGE_REGEX. If you plan to use a dictionary to annotate the car manufacturer in the sentence, you can add a class named MANUFACTURER_DICT.

Keep these naming rules in mind:



* The first character in a class name must be alphabetical.
* Use only the following alphanumeric ASCII characters and the underscore character in values that you add to the classes: A through Z, a through z, 0 through 9.
* Names cannot contain spaces.
* Names cannot be longer than 64 characters.





4. Optional: To quickly annotate classes in a document, you can associate a dictionary with the rule editor. Terms in a document that match entries in the dictionary are automatically annotated with the class that you select for the dictionary.



1. Click the Dictionaries tab.
Table 1. Role descriptions

 Role Description 

 Admin Responsible for administrative tasks, which include managing users, resource consumption, and monthly charges. In large team settings, admins rarely participate in the model development process. 
 Project manager Responsible for the overall organization of the workspace that she or he is assigned to. Tasks include creating the type system, managing assets, managing annotation work, evaluating the machine learning model, and deploying models. Users in this role need industry subject-matter expertise because they create the type system, teach the human annotators how to correctly apply the type system, and evaluate the model quality. 
 Human annotator Performs the labeling of the entity mentions and relationship mentions in the training documents that he or she is assigned to. The work is assigned to human annotators and monitored by the project manager. Human annotators may not have industry subject-matter expertise, as long as they are taught by the project manager how to correctly apply the type system. 







 Knowledge Studio role permissions 

To compare the permissions of each role, see the following table. One permission is listed on each row. If a role has that permission, the role column is marked with a check mark (✓).



Table 2. Role permissions

 Permission Admin Project manager Human annotator 

 Manage user access and roles ✓ 
 Manage workspaces ✓ 
 Create the type system and annotation guidelines ✓ ✓ 
 Upload training documents ✓ ✓ 
 Manage rules ✓ ✓ 
 Pre-annotate documents ✓ ✓ 
 Manage annotation tasks ✓ ✓ 
 Resolve annotation conflicts with human annotation (adjudication) ✓ ✓ 
 Train and evaluate machine learning models ✓ ✓ 
 Deploy and undeploy models to runtime services ✓ ✓ 
 Annotate document sets directly ✓ ✓ 
 Perform document annotation in annotation tasks ✓ ✓ ✓
Enabling cross-sentence relations (Experimental) 

English entities and relations workspaces that you create can support annotating relations between pairs of entity mentions that exist within spans of six sentences. In workspaces where the cross-sentence relations feature is not enabled, the ground truth editor and runtime models do not support annotating relations that cross sentence boundaries.

Cross-sentence relations is an experimental feature that might be discontinued without notice. It is not recommended to use this feature in a production environment.

Enabling cross-sentence relations might decrease the performance of the ground truth editor and the performance of any models that you deploy from the workspace at runtime. The cross-sentence relations setting cannot be changed for a workspace after it is created.

For example, if you plan to annotate a relation for an entity mention in the seventh sentence below, you are limited to selecting other entity mentions only in the seventh sentence unless you enable cross-sentence relations. With the feature enabled, you can annotate a related entity mention within a span of six sentences (across up to five sentence boundaries), so the second entity mention in the relation could be annotated anywhere from the second sentence to the twelfth sentence.

Sentence 1.
 Sentence 2.
Sentence 3.
Sentence 4.
Sentence 5.
Sentence 6.
-----> Sentence 7.
Sentence 8.
Sentence 9.
Sentence 10.
Sentence 11.
 Sentence 12.
Sentence 13.



  Procedure 



1.  Launch the Knowledge Studio application.
2.  If you have existing workspaces, click Create Workspace.
3.  Click Create entities and relations workspace.
4.  Click Advanced options.
5.  Click the Enable cross sentence relations box.
6.  Click Create.



In the new workspace, you will be able to annotate relations between pairs of entity mentions that exist within spans of six sentences.
* Documents that contain multiple columns of text on a single page cannot be annotated. Report these documents to your administrator.
* Consider only labeling footnotes when they appear at the bottom of the page and are referenced in the main body of text in the document.
* Consider labeling notes that appear within sections or lists, for example notes that are explicitly called out as “Notes,” as text.
* When you annotate a table, make sure to select the entire table before you apply the table label.
* If the visual structure of one set of documents is drastically different from another, group the documents that are structured similarly in separate collections and then retrain each collection.
</xsd:annotation>
</xsd:enumeration>
</xsd:restriction>
</xsd:simpleType>
<xsd:simpleType name="regularAggregateType">
<xsd:annotation>
<xsd:documentation>The aggregation rule. same as V5 aggregate name. </xsd:documentation>
</xsd:annotation>
<xsd:restriction base="xsd:NMTOKEN">
<xsd:enumeration value="none">
<xsd:annotation>
<xsd:documentation>This item should be used as a grouping item.</xsd:documentation>
</xsd:annotation>
</xsd:enumeration>
<xsd:enumeration value="average">
<xsd:annotation>
<xsd:documentation>Compute the average value for the values of the group.</xsd:documentation>
</xsd:annotation>
</xsd:enumeration>
<xsd:enumeration value="automatic">
<xsd:annotation>
<xsd:documentation>The aggregation rule is determined automatically at run time.</xsd:documentation>
</xsd:annotation>
</xsd:enumeration>
<xsd:enumeration value="calculated">
<xsd:annotation>
<xsd:documentation>Aggregation is performed on the elements before the rest of the expression is executed. For example, if the aggregate rule for Q1_a is set to total, and the rule for q1_b is set to average, then if this query item is based on the expression (QI_a / QI_b)* 100, the aggregate is computed as: ( total(QI_a) / average( QI_b ) ) 100.
</xsd:documentation>
</xsd:annotation>
</xsd:enumeration>
The repeat setting specifies how many times the token can be repeated in the text for it to match the pattern. The current repeat setting of Required (Exactly 1) is appropriate.

![Shows the repeat setting for the comma token which is set to "Exactly 1".](images/rule-add7.png "Shows the repeat setting for the comma token which is set to "Exactly 1".")
7. Assign a class to represent the pattern DictMonth + numeric token + comma + RegExpYear.

Notice the four empty cells that represent the four tokens that you selected from the document. To select all the cells, select the first cell, and then press Shift + click each additional cell. Enter RuleDate as the class name, and then click it to create the new class.

![Shows that all four cells in the top row have been selected and the span is being defined as the "RuleDate" class.](images/rule-add8.png "Shows that all four cells in the top row have been selected and the span is being defined as the "RuleDate" class.")
8. In the Rule name field, enter MyDateRule and click Save.

After you save the rule, it is automatically applied to the sample documents. If the Technology - computerworld.com document is still open in the rule editor, you will see that the February 3, 2009 text in the document is now annotated as a mention of the RuleDate class.

![Shows text from the "Technology - computerworld.com" document with only the text "February 3, 2009" annotated as a mention of the "RuleDate" class.](images/rule-add10.png "Shows text from the "Technology - computerworld.com" document with only the text "February 3, 2009" annotated as a mention of the "RuleDate" class.")

You can search for all occurrences of RuleDate class mentions in the sample documents by clicking the Search annotation in documents icon !
If annotation guidelines were previously connected to the workspace, and you want help with choosing the correct annotation to apply, click View Guidelines. Depending on the access permissions set up on the site where the guidelines are hosted, you might be able to update the guidelines after you open them, for example, to add clarifications and examples.
3. Avoid creating overlapping mentions. But, if a valid overlapping mention is required, click Replace to more easily add it. An overlap occurs when you apply more than one label to an entity mention. Review these suggestions:



* Annotate Sub-Saharan as a single mention, not just Saharan or just Sub.
* Do not create an overlapping PERSON annotation for the JFK reference in JFK International Airport. The entire JFK International Airport mention should be labeled as a FACILITY only.
* For the text CEOs, do not create a PERSON annotation for CEO and a PEOPLE annotation for CEOs. Annotate CEOs as a PEOPLE entity type only.



Typically, the existence of too many overlapping mentions means that the annotation guidelines are ambiguous and need to be improved to provide better examples of how to handle compound words in your source data.
4. To remove an annotation that you just added, press Ctrl+Z to undo the action. To remove an entity mention later, you can left-click a mention and press the Delete key, or click View Details, and then click X next to the entity type that is assigned to the mention.



7. Depending on the type system, you might be able to configure attributes for an entity mention, such as assigning an entity role or subtype or a mention class or type. If so, select a mention and click Attribute View.
8. Click Save at any time to save your work.







 What to do next 

After you finish annotating all entity mentions, relation mentions, and coreferences in the document, as applicable, change the document status from In Progress to Completed, click Save, and then close the document.

After you finish annotating all documents and mark them Completed, the status of the annotation set changes to Submitted.
You might need to make modifications while you train a model, based on the performance statistics. But, generally, you want the type system to be as close to final as possible before you begin large-scale annotation tasks. If you change the type system after human annotators began their work, they must revisit the documents that they annotated. They must assess the applicability of the type system changes.



 About this task 

This process propagates the current type system, ground truth editor keyboard shortcuts, and color settings to all document sets in a task.





 Procedure 

To modify the type system without losing the work that was done by human annotators:



1. Change the type system. For example, you can add or remove entity types or relation types.
2. Decide whether you want to propagate the changes to existing human annotation tasks.
3. Open the Machine Learning Model > Annotations page and click the Annotation Tasks tab. Open each task that you want to update and click Apply Type System Updates.

If you removed entity types or relation types from the type system, all occurrences of those types are highlighted in gray in the documents. These invalid types are ignored by the machine learning model. They do not prevent you from submitting and approving document sets.
4. Provide details to the human annotators about what changed in the type system.
5. Ask human annotators to update their documents to reflect the changes in the type system. For example, if you added new entity types or relation types, they must review their documents and annotate them appropriately.

> Note: If the task contains completed documents, human annotators cannot alter those documents to assess type system changes until they are back in an editable state. To become editable, ask human annotators to submit the document sets so that you can reject them.



Related concepts:

[Type systems](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-typesystemwks_typesystem)







 Document set management
The dark blue line after the Military Ranks extractor indicates placement of the Person extractor in the text pattern.

![Person extractor](https://cloud.ibm.com/docs-content/v1/content/148d3bd95f946aa1bb53ea1540475f522e6b61c9/watson-knowledge-studio-data/images/military_ranks.png)

When you release the mouse button, a sequence object is displayed on the canvas.
3. Click the sequence object on the canvas and, under Extractor Properties, open the General tab. Rename the sequence object as Military Personnel.
4. The final extractor rule is displayed on the canvas.

![Military Personnel](https://cloud.ibm.com/docs-content/v1/content/148d3bd95f946aa1bb53ea1540475f522e6b61c9/watson-knowledge-studio-data/images/military_personnel.jpg)







 Adding proximity rules 

Proximity rules to specify the maximum number of tokens that might occur between the desired terms. In text, each word or character is generally referred to as a token. The sentence "The CEO announced the earnings last week, missing analyst estimates by 5%." contains 15 tokens: 11 words separated by spaces, 3 special characters, and 1 number.

You might want to locate references to red brick houses, knowing that phrases such as house made of red brick, red house of brick and the brick on the house reflected red might appear in the text you are analyzing. Proximity rules can ensure that these phrases are extracted while excluding sentences or phrases that coincidentally include the target terms, for example, "The red car passed a house near the brick yard".



1. Right-click an extractor on the canvas and under one of the Add submenus, click Proximity Rule.
2. In the format minimum-maximum, specify the minimum and maximum number of words, special characters, or values to allow between terms. By default, the proximity pattern is set to match the number of tokens between the two numbers you specify in your range.
[Pre-annotating documents with a dictionary](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-preannotationwks_preannot)







 Adding dictionaries to a workspace 

Adding dictionaries is an optional step in creating a model. Dictionaries are helpful because they enable you to jump start the annotation process.



 About this task 

If you provide a dictionary, you can run the dictionary pre-annotator on the documents. The pre-annotator finds terms that are represented in your dictionary and automatically annotates them. This initial pass on the documents simplifies the human annotator's job because she can review the annotations that were added by the pre-annotator and correct them or add to them. She does not have to start entirely from scratch.

The following restriction apply to dictionaries:



* Maximum 15,000 entries per dictionary

> Note: This limit does not apply to dictionaries that you upload as a dictionary CSV file. Read-only dictionaries can contain more entries.
* Maximum 64 dictionaries per workspace







 Procedure 

To add a dictionary to your workspace:



1. Log in as a Knowledge Studio administrator or project manager and open the Assets > Dictionaries page.
2. Perform one of the following tasks:



* Next to the Create Dictionary button, click the Menu icon, and then select Upload Dictionary. Select a dictionary, and then click Upload. After you upload a dictionary, select it to view the dictionary and associate it with an entity type.



You can upload a ZIP file that contains a dictionary that you downloaded from another Knowledge Studio workspace. You must upload the type system that was downloaded from the other workspace in JSON format before you can upload the corresponding dictionary file. You can edit and add entries to a dictionary that you reuse from another Knowledge Studio workspace. See [Uploading resources from another workspace](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-exportimport) for more details.
Log in to Knowledge Studio as a user who is assigned to the annotation task that you created in [Lesson 3](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-wks_tutml_introwks_tutless_ml2).

> Note: If you have access only to a single administrator ID for this tutorial, you can use that ID to perform human annotation. However, remember that in a realistic scenario, human annotation is performed by different users with the Human Annotator role.
2. Open the My workspace workspace and click Machine Learning Model > Annotations.
3. Click the Annotation Tasks tab, then open the Test annotation task you created in [Lesson 3](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-wks_tutml_introwks_tutless_ml2).
4. Click Annotate for one of the assigned annotation sets.

Depending on how you set up the annotation tasks, you could have one or more annotation tasks assigned to the user ID you logged in with.
5. From the list of documents, find the Technology - gmanews.tv document and open it.

Notice that the term IBM was already annotated with the ORGANIZATION entity type. This annotation was added by the dictionary pre-annotator that was applied in [Lesson 2](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-wks_tutml_introwks_tutless_ml3). This pre-annotation is correct, so it does not need to be modified.

![This screen capture shows an open document with an existing pre-annotation for "IBM".](images/wks_tut_preannotation.png "This screen capture shows an open document with an existing pre-annotation for "IBM".")
6.