vania-janet/multiturn-rag-retrieval-data huggingface dataset:
A .pear archive file that includes an Unstructured Information Management Architecture (UIMA) analysis engine and all of the resources that are required to use it for custom analysis.







 R 



* recall

A measurement that specifies the percentage of relevant results returned, out of all available relevant results. Recall, which is a measure of sensitivity, is determined by the number of correct positive results divided by the number of positive results that should have been returned. Accuracy is best measured by using both precision and recall. See also [accuracy](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-glossarygloss_A) and [precision](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-glossarygloss_P).
* relation

Typically a verb that reflects how entities are related to one another. For example, "lives in" is a relation between a person and a town. A relation links two different entities in the same sentence.
* relation type

A binary, uni-directional relationship between two entities. For example, MaryemployedBy IBM is a valid relationship; IBM employedByMary is not.
* role

An attribute that provides a context-sensitive meaning of a mention. For example, in the phrase "I went to IBM today", IBM is the mention, Organization is the entity type, and Facility is the role of the entity type.
* rule set

A set of rules that define patterns for annotating text. If a pattern applies, then the actions of the rule are performed on the matched annotations. A rule typically specifies the condition that must match, an optional quantifier, a list of additional constraints that the matched text must fulfill, and the actions to be taken when a match occurs, such as creating a new annotation or modifying an existing annotation.







 S 



* subtype

A type that extends or implements another type; the supertype.
* surface form

The form of a word or multiword unit as it is found in the corpus.
To annotate repeating mentions:



1. Log in as a human annotator (or as an administrator or project manager who was assigned documents to annotate). Workspaces that contain tasks that are assigned to you are displayed.
2. Open a workspace, and then click Machine Learning Model > Annotations. Click the Annotation Tasks tab. The annotation tasks that are assigned to you are displayed.
3. Open the annotation task that you want to work on. The annotation sets that are assigned to you are displayed.
4. Click Annotate to open the annotation set you want to work on. The documents in the annotation set are displayed.
5. Open the document that you want to annotate. By default, the document opens in Mention mode, which is the mode you use to annotate entity mentions.
6. If you have not added any annotations yet, add at least one annotation. Select a word or word phrase that represents a mention of an entity type from your type system, and assign the appropriate type to it. Click Save to save your annotation.
7. Select a single occurrence of repeating text that you want to annotate, and then click Concordance.
8. Select the documents that you want to apply the selected entity type to. You can create the annotations in all documents that you have been assigned to annotate, all documents that you have begun annotating, or all documents that you have not yet started to annotate.
9. Click Preview to see the annotations that will be added.

If you want to view the annotations in greater context, click the icons to preview the document content or open the document in a new window.
10. Click Apply & Review to apply the selected entity types to mentions in the selected documents. You still have a chance to review the annotations that will be added. If an annotation is inaccurate in a particular context, you can remove that occurrence by clicking the Edit icon, and then removing the entity type assignment for the mention.
11.
Just be sure that your team chooses one of them and applies it consistently to any lists that occur in the documents.
* A coreference is used when mentions refer to the same real-world entity. Relations are used between distinct entities. So, no two mentions should be connected by both coreference and a relation.







 Annotation with the ground truth editor 

When a human annotator annotates a document, the document is opened in the ground truth editor. The ground truth editor is a visual tool that human annotators use to apply labels to text.

The goal of human annotation is to label mentions, relations, and coreferenced mentions so that the machine learning model can be trained to detect these patterns in unseen text. At a minimum, use the tool to annotate entity mentions. If the application that will use the resulting model does not need to find and extract coreferences and relation mentions, then you do not need to annotate coreferences and relation mentions.

Concordance is an optional tool that can be used by human annotators to expedite the annotation of repetitive mentions.

Choose a mode to use when manually annotating documents:



* Mention mode

In this mode, the human annotator associates entity types, as defined in the type system, with meaningful words or phrases in the text. For example, all mentions of person names might be associated with an entity type named PERSON. The annotation of mentions is required and must occur before you annotate relation types and mentions as coreferences.

The human annotator can optionally use the concordance tool to ensure that the same text is annotated with the same entity type throughout a document and across annotation sets.
* Relation mode

In this mode, the human annotator connects mentions by associating a relation type, as defined in the type system. For example, the mention John Smith might be connected to the mention IBM by the relation type employedBy. The annotation of relation types is optional and can occur before or after you annotate mentions as coreferences.
* Coreference mode

In this mode, the human annotator identifies mentions that mean the same thing, thus helping to ensure consistency in the annotations when words are not identical.
Small documents can help human annotators identify coreference chains throughout the document. As annotation accuracy improves, you can add more documents to the corpus to provide greater depth to the training effort.





 Procedure 

To add documents to a workspace:



1. Log in as a Knowledge Studio administrator or project manager, and select your workspace.
2. Select the Assets> Documents > Documentation sets tab.
3. Click Upload Document Sets to add documents to the corpus.
4. Upload documents in one of the supported formats. For information about the supported document types, size limits, and other information, see [Creating a workspace > Summary of inputs, outputs, and limitations](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-create-projectwks_formats).

Notes about ZIP files of documents downloaded from another workspace



* If you previously downloaded documents from a Knowledge Studio workspace, drag the ZIP file that contains the downloaded documents or click to locate and select the file. If you want to include annotations that were added to the documents before they were downloaded, ensure that the option to include ground truth is selected before you click Upload. Only annotations that were promoted to ground truth before the documents were downloaded will be imported.
* Restriction: When annotated documents are imported, they are re-tokenized. This process can change what Knowledge Studio considers to be the sentence boundaries. Because annotations are defined by sentence, some annotations might be invalidated during this process. After uploading documents from another workspace, do a quick review of the annotations to address any discrepancies.
* You must upload the type system from the original workspace into the current workspace before you upload ground truth annotations. For details, see [Uploading resources from another workspace](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-exportimport).



Notes about ZIP files of documents in UIMA CAS XMI format
Then, you can assign the documents to human annotators to finish the annotation work.





 Lesson 1: Pre-annotating new documents with a machine learning model 

In this lesson, you will learn how to use a machine learning model to pre-annotate documents in Knowledge Studio.



 About this task 

After you train a machine learning model, you can use it to pre-annotate new documents that you add to the corpus.

> Attention: Do not run a pre-annotator on documents that have been annotated by humans, but not been added to the ground truth yet. If you do, all current annotations will be stripped from the documents.

In this tutorial, you can add a second set of documents by using the documents-ml.csv file. Do not re-add the documents-new.csv file, since this addition would result in duplicate documents in the ground truth. Duplication causes the following problems:



* If annotations on each document do not match, they lower the quality of the machine learning model.
* If annotations on each document match, they over-train the machine learning model on the duplicated files.



For more information about pre-annotating documents, see [Bootstrapping annotation](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-preannotation), which covers additional methods of pre-annotation.





 Procedure 



1. Log in to Knowledge Studio as the administrator.
2. Upload more documents to the workspace. You can use the [documents-ml.csv![External link icon](https://cloud.ibm.com/docs-content/v1/content/icons/launch-glyph.svg)](https://watson-developer-cloud.github.io/doc-tutorial-downloads/knowledge-studio/documents-ml.csv) file.
* Use only the following alphanumeric ASCII characters and the underscore character in values that you add to the classes: A through Z, a through z, 0 through 9.
* Names cannot contain spaces.
* Names cannot be longer than 64 characters.





4. Optional: To quickly annotate classes in a document, you can associate a dictionary with the rule editor. Terms in a document that match entries in the dictionary are automatically annotated with the class that you select for the dictionary.



1. Click the Dictionaries tab.

Any dictionaries that you created are displayed.

If you have not added a dictionary, open the Assets > Dictionaries page from the main navigation bar to add one. See [Creating dictionaries](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-dictionaries) for more information.
2. Click a dictionary, associate a class with the dictionary, and then click Save.

For example, if you have a dictionary that contains organization names, you can create a rule class called ORGANIZATION, and associate the class with the dictionary. Any organization names that occur in your sample documents will be annotated as instances of the ORGANIZATION class.



If you want to remove the dictionary association from the rule editor later, then you can remove the class mapping. To do so, choose the empty option from the top of the drop-down list.
5. Optional: To define a regular expression to help you construct a rule, click the Regex tab.



1. Click the plus sign (+) next to the Regular Expressions heading.
2. Name the regular expression. For example, MyAgeRegex.

The name cannot be longer than 64 characters.
3. Associate the expression with a class. For example, AGE_REGEX.
4. Click Add Entry.
5. Add the expression.

For example, to capture a number that represents an age (up to age 99), you can specify [0-9]{1,2}.
Any string wrapped with / will be considered a pattern (as opposed to a synonym).
* Annotations associated with contexual entities are not downloaded. You must download the entire dialog skill to capture both the entity value and any associated annotations.





1. Go to the Entities page



* To download all entities, meaning the entities that are listed on this and any additional pages, do not select any individual entities. Instead, click the Download icon ![Download option](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/download.png).
* To download the entities that are listed on the current page only, select the checkbox in the header. This action selects all of the entities on the current page. Click the Download icon ![Download option](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/download.png).
* To download one or more specific entities, select the entities that you want to download, and then click the Download icon ![Download option](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/download.png).



2. Specify the name and location in which to store the CSV file that is generated, and then click Save.







 Uploading entities 

If you have a large number of entities, you might find it easier to upload them from a comma-separated value (CSV) file than to define them one by one.

Entity annotations are not included in the upload of an entity CSV file. You must upload the entire dialog skill to retain the associated annotations for a contextual entity in that skill.
This pre-annotation is correct, so it does not need to be modified.

![This screen capture shows an open document with an existing pre-annotation for "IBM".](images/wks_tut_preannotation.png "This screen capture shows an open document with an existing pre-annotation for "IBM".")
6. Annotate a mention:



1. Click the Entity tab.
2. In the document body, select the text Thomas Watson.
3. In the list of entity types, click PERSON. The entity type PERSON is applied to the selected mention.

![This screen capture shows the "PERSON" entity type applied to the mention, "Thomas Watson".](images/wks_tut_annotatemention3.png "This screen capture shows the "PERSON" entity type applied to the mention, "Thomas Watson".")



7. Annotate a relation:



1. Click the Relation tab.
2. Select the Thomas Watson and IBM mentions (in that order). To select a mention, click the entity type label above the text.
3. In the list of relation types, click founderOf. The two mentions are connected with a founderOf relationship.

![This screen capture shows two mentions connected by the relation type, "founderOf".](images/wks_tut_annotaterelation.png "This screen capture shows two mentions connected by the relation type, "founderOf".")



8. From the status menu, select Completed, and then click the Save button.
9. Click Open document list to return to the list of documents for this task and click Submit All Documents to submit the documents for approval.

In a realistic situation, you would create many more annotations and complete all the documents in the set before submitting.
10. Close this annotation set, and then open the other annotation set in the Test task.
: Admins and Project Managers can annotate document sets directly from the Ground Truth tab on the Annotations page.
: Annotation tasks are still available. You can manage them from the Annotation Tasks tab on the Annotations page.
: Annotations applied directly to ground truth are not applied to related active annotation tasks. It is recommended that you annotate document sets directly only when they are not associated with any active annotation tasks.
: The Annotation Sets tab has been removed from the Documents page. You can manage annotation sets from the Annotation Tasks tab of the Annotations page.

- To create new annotation sets, click Add task. Then, click Create Annotation Sets.
- To manage existing annotation sets, click an existing annotation task, then click Edit.







 March 2019 



 New features and changes 

Custom categories workspace (Experimental)
: Introduced an experimental custom categories workspace for Knowledge Studio service instances on Lite and Standard plans that are hosted in the Dallas location. With the new workspace, you can deploy your own custom text categorization model to Natural Language Understanding or Discovery.







 January 2019 

Migrating Cloud Foundry service instances
: You can now migrate Knowledge Studio Cloud Foundry service instances to a resource group.





 December 2018 



 New features and changes 

Support to deploy same model to multiple instances
: Introduced support to deploy the same machine learning model version to multiple service instances and general improvements to the Version History and Deployment page. For more information about deploying multiple instances of the same model version see [Deploying the same model version to multiple services](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-publish-mlwks_secdep)

Added models method
: Added the models method to the Natural Language Understanding service allowing users to list deployed Knowledge Studio models.







 September 2018 



 New features and changes 

Support for additional document types
: Introduced support for HTML, DOC, DOCX, and PDF files.
* Keyboard shortcut, which means a user can enter <key> to apply the type label to highlighted text. For example, if you define o as the keyboard shortcut for ORGANIZATION, then a user can select text, and then press the o key to apply the ORGANIZATION entity type to the highlighted text. If you assign an upper case letter, then the user must press Shift+<key>.
* Text color. Ensure that the text color contrasts with the background color so the text will be visible after it is labeled.
* Background color. This is the color of the label that is applied to the entity after you annotate it.



When annotating documents, human annotators can use the keyboard shortcuts to quickly add annotations. And the annotation label and text colors help human annotators to instantly recognize types after they add annotations to a document.



* If there are entity or relation types that you do not want human annotators to assign to mentions, you can hide them from the ground truth editor, which shortens and simplifies the list of type options that users see. To do so, deselect the Active check box for the type.



As you assign new shortcuts and colors, you can preview the changes.
5. You can also change the default selection highlight color. The highlight color is the color of the border that is displayed around text after human annotators select it. The default color is a light blue, but you can change the color on the Selection Highlight tab to make it easier to identify the boundaries of the text that is selected.





 Related tasks 

[Modifying a type system without losing human annotations](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-improve-mlwks_projtypesysmod)









 Setting the IAA threshold 

To help you decide whether to accept or reject an annotated document set, you can specify an inter-annotator agreement threshold. The threshold helps you compare how well or poorly inter-annotator agreement compares to the IAA score calculated by the system.



 About this task 

To compare how different human annotators annotated the same documents, specify an evaluation threshold.
There is no substitute for these extractors that is provided by the new tokenizer.
* Generic Splitters:



* Date Time Splitter
* Day First Splitter
* Month First Splitter
* Year First Splitter



* Syslog Splitter:



* Date Time Splitter





* Cross-sentence relations (Experimental): English entities and relations workspaces can now support annotating relations between entities within spans of six sentences. To get started, see [Enabling cross-sentence relations](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-enabling-cross-sentence-relations).







 Version 1.1.0 (27 November 2019) 



* Knowledge Studio for IBM Cloud Pak for Data 1.1.0 can be installed on IBM Cloud Pak for Data 2.5
* Dictionary suggestions: Create dictionaries faster. With only a few terms specified, Knowledge Studio will analyze your uploaded documents and suggest other terms possibly in the same domain. These dictionaries can be used as machine learning preannotators or within rules to create models.
* Advanced rules workspace (Beta): Create rules-based text extractors with an advanced rules workspace. Advanced rules expand well beyond the standard rules engine with a graphical interface that creates Annotation Query Language (AQL) rules, allowing users to build and manage complex rule sets.







 Version 1.0.1 (30 August 2019) 



* Added the ability to create multiple instances per installation.
* Added the ability to install on Red Hat OpenShift on IBM Cloud.
* Federal Information Security Management Act (FISMA) support is available for IBM Watson™ Knowledge Studio for IBM Cloud Pak for Data offerings purchased on or after August 30, 2019. FISMA support is also available to those who purchased the June 28, 2019 version and upgrade to the August 30, 2019 version. IBM Watson™ Knowledge Studio for IBM Cloud Pak for Data is FISMA High Ready.



Annotation tasks are no longer required to create ground truth.



* Admins and Project Managers can annotate document sets directly from the Ground Truth tab on the Annotations page.
* Annotation tasks are still available.
Pre-annotators remove human annotation.





 Procedure 

To use an existing machine learning model to pre-annotate documents:



1. Log in as a Knowledge Studio administrator and select your workspace.
2. Go to the Machine Learning Model > Pre-annotation page.
3. Click Run Pre-annotators.
4. Select Machine Learning Model, and then click Next.
5. If you want to erase existing annotations made by pre-annotators before running the pre-annotator, select Wipe previous pre-annotation results. Human annotations are preserved even if this is selected.
6. Select the check box for each document set that you want to pre-annotate and click Run.

Pre-annotation is applied to individual documents without regard for the various document sets or annotation sets that a document might belong to. A document that overlaps between a selected document set and an unselected document set will be pre-annotated in both document sets.









 Pre-annotating documents with the rule-based model 

You can use an existing rule-based model to pre-annotate documents that you add to your corpus.



 Procedure 

To use the rule-based model to pre-annotate documents, complete the following steps:



1. Log in as a Knowledge Studio administrator and select your workspace.
2. Go to the Machine Learning Model > Pre-annotation page.
3. Select the Rule-based Model > Versions > Rule-based Model tab.
4. If not already completed, open the overflow menu in the Rule-based Model row and click Map entity types and classes to map entity types that you defined in the Knowledge Studio type system to one or more rule-based model classes.
5. Click Edit for each entity type you want to map.



* The drop-down list of the Class Name column is pre-populated with classes that are associated with the rule-based model.
* You must map at least one entity type to a class.



6. On the Machine Learning Model > Pre-annotation page, click Run Pre-annotators.