vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Annotation setup 

Users who have knowledge of the industry and its language must annotate the documents.

Perform the following tasks to enable human annotators to access the workspace:



* Invite subject matter experts to the Knowledge Studio instance that you are using.
* Associate human annotators with the annotation set or sets that you want them to annotate.
* Create a task that assigns the human annotator to annotate the documents in the set.

It is not until you explicitly assign tasks to human annotators that they can see your workspace when they log in to Knowledge Studio.



Your users (the human annotators) will need to become familiar with [Annotating documents](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-user-guide), which contains detailed information about how to annotate documents.



 Model life cycle 

The model that you create with Knowledge Studio is a software component that can be plugged into a natural language processing (NLP) pipeline.

With Knowledge Studio, you can create, evaluate, and improve models for new domains. A model adds annotations (metadata) to text that appears in natural language content. The annotations, which identify mentions of entities of interest in your domain content, the relationships between them, and how the mentions co-reference the same entity, can be used by applications to automatically analyze and process text. Application users benefit from this level of analysis by being able to extract meaning, discover insights, and obtain answers in a natural language context.

The creation of a model is an iterative multiple-step process that involves several stages: knowledge curation, ground truth generation, model development, model evaluation, and runtime deployment.



 End-to-end domain adaptation 

The following diagram summarizes the interactions between these five stages of model development and the typical activities that occur at each stage.

![A summary of the five stages of model development and the activities that occur at each stage.]
Click Run.

If you are doing a validation check of the pre-annotator, then open the annotated documents and review the annotations that were added. Make sure a sufficient number of accurate annotations were created. If the annotations are accurate, then you can run the annotator again on more and larger document sets. If the annotations are not accurate, then consider mapping different Natural Language Understanding entity types to your types. If the types do not naturally overlap, then the Natural Language Understanding pre-annotator is not the best pre-annotator for you to use.

Pre-annotation is applied to individual documents without regard for the various document sets that a document might belong to. A document that overlaps between a selected document set and an unselected document set will be pre-annotated in both document sets.







 Results 

Ground truth that is produced by documents that were pre-annotated by the Natural Language Understanding service cannot be used directly outside of Knowledge Studio. You can download the ground truth (in non-readable form) to move it from one Knowledge Studio workspace to another. And you can continue to develop the ground truth and use it to build a machine learning model or rule-based model that can be deployed for use in services outside of Knowledge Studio.

Documents that were pre-annotated with Natural Language Understanding are obscured into a non-readable format when they are downloaded. But, all annotations in those documents are obscured, including annotations that were added to the documents by human annotators.

Related information:

[Natural Language Understanding](https://www.ibm.com/watson/services/natural-language-understanding/)







 Pre-annotating documents with a dictionary 

To help human annotators get started with their annotation tasks, you can create a dictionary and use it to pre-annotate documents that you add to the corpus.



 About this task 

When a human annotator begins work on documents that were pre-annotated, it is likely that a number of mentions will already be marked by entity types based on the dictionary entries.
Annotating documents 

The information in this section helps subject matter experts who have been asked to annotate industry documents use the ground truth editor to complete the task.



 Workspace access 

You cannot see any workspaces until someone else creates a workspace and gives you access to it.

When an administrator adds you to an instance of Knowledge Studio, you are added in the human annotator role. With that role, you cannot create a workspace. To get access to a workspace, the administrator must create a workspace. Then the administrator or a project manager that the administrator associates with the workspace must perform the following steps:



1. Create an annotation set and associate you with it.
2. Create a task that assigns you to annotate the documents in the set.



It is not until an annotation task is assigned to you that you can see the workspace.

If you were invited to participate in a Knowledge Studio workspace, but do not see any workspaces from the Workspaces page, contact the person who invited you, and ask her to perform the required steps.





 Annotation best practices 

These annotation best practices provide some guidance and examples as you start to annotate documents.



* Annotate all documents completely.

Machine learning learns from negative examples -- what is not annotated -- not just from what is annotated. So, be judicious in what you annotate, but do a complete job. If you carefully annotate only the first 5 of 10 documents in a set, the annotations that you do not capture in the last 5 documents will teach the model to ignore any entity or relation mentions that were missed in those documents. You could end up reversing any gains you made by doing a thorough job with the first 5 documents.
* Consistent annotation is at least as important as correct annotation.

Some decisions regarding annotation guidelines are arbitrary, like whether car trim lines should be considered part of model names, such as Camry or Camry LX. Which of the policies is chosen is far less important than that the project team agrees on one or the other and annotates in accordance with that policy consistently.
Analyzing machine learning model performance 

Review the annotations that were added by the trained model to determine whether any adjustments must be made to the model to improve its ability to find valid entity mentions, relation mentions, and coreferences in the documents.



 About this task 

You can analyze performance by viewing a summary of statistics for entity types, relation types, and coreferenced mentions. You can also analyze statistics that are presented in a confusion matrix. The confusion matrix helps you compare the annotations added by the machine learning model to the annotations in ground truth.

The model statistics provide the following metrics:



* F1 score

A measurement that considers both precision and recall to compute the score. The F1 score can be interpreted as a weighted average of the precision and recall values, where an F1 score reaches its best value at 1 and worst value at 0. See [Analyzing low F1 scores](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-evaluate-mlevaluate-mllowf1).
* Precision

A measurement that specifies what fraction of the machine learning model's output was accurate when compared to the human annotator output. Precision is determined by the number of correctly labeled annotations divided by the total number of annotations added by the machine learning model. A precision score of 1.0 for entity type A means that every mention that was labeled as entity type A does indeed belong to that classification. A low precision score helps you identify places where the machine learning model created incorrect annotations. The score says nothing about how many other mentions that were labeled as entity type A by the human annotator were missed by the machine learning model; the recall score reflects that information. See [Analyzing low precision scores](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-evaluate-mlevaluate-mllowp).
* Recall

A measurement that specifies how many mentions that should have been annotated by a given label were actually annotated with that label - the right mentions being those that human annotators identified in the same documents.
This process can change what Knowledge Studio considers to be the sentence boundaries. Because annotations are defined by sentence, some annotations might be invalidated during this process. After uploading documents from another workspace, do a quick review of the annotations to address any discrepancies.



* If you previously downloaded documents from a Knowledge Studio workspace, drag the .zip file that contains the downloaded documents or click to locate and select the file. If you want to include annotations that were added to the documents before they were downloaded, ensure that the option to include ground truth is selected before you click Upload. Only annotations that were promoted to ground truth before the documents were downloaded will be imported.
* You must upload the type system from the original workspace into the current workspace before you upload ground truth annotations. For more information, see [Uploading resources from another workspace](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-exportimport).



Notes about .zip files of documents in UIMA CAS XMI format



* If you previously downloaded annotated documents that are in UIMA CAS XMI format, you can upload the .zip file that contains the analyzed content. Specify that this is the type of content you want to upload before you click Upload. For more information about how to create these files and requirements for uploading them, see [Uploading pre-annotated documents](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-preannotationwks_uima).



5. After the documents have been added, click the document names to preview them and verify that the content looks OK. For example, verify that text files are in UTF-8 format and that no diacritical marks or character normalization issues are visible in the documents, and check for poor sentence breaks. If problems exist, you might need to pre-process the files before you add them to the corpus. You want the documents to be as clean and well-formatted as possible before dictionary or human annotation begins.







 What to do next
Select the document set that you created in [Lesson 1](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks_tutml_introtut_lessml1).
6. Click Run.







 Results 

The documents in the selected sets are pre-annotated by using the dictionary that you created. If you like, you can use the dictionary to pre-annotate document sets or annotation sets that you add later.







 Lesson 3: Creating an annotation task 

In this lesson, you will learn how to create annotation sets and use annotation tasks to track the work of human annotators in Knowledge Studio.



 About this task 

An annotation set is a subset of documents from an uploaded document set that you assign to a human annotator. The human annotator annotates the documents in the annotation set. To later use inter-annotator scores to compare the annotations that are added by each human annotator, you must assign at least two human annotators to different annotation sets. You must also specify that some percentage of documents overlap between the sets.

In a realistic scenario, you create as many annotation sets as needed, based on the number of human annotators who are working in the workspace. In this tutorial, you will create two annotation sets. If you do not have access to multiple user IDs, you can assign both annotation sets to the same user.

For more information about annotation sets and annotation tasks, see [Creating an annotation task](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-annotate-documentswks_hatask).





 Procedure 



1. Within your workspace, click Machine Learning Model > Annotations.
2. Click the Annotation Tasks tab, then click Add Task.
3. Specify the details for the task:



* In the Task name field, enter Test.
* In the Deadline field, select a date in the future.



4.
In human annotation, a collection of documents that are extracted from the corpus that allow the workload to be shared by multiple human annotators. In machine-based annotation, a collection of documents that can be used as blind data, training data, or test data.
* annotation process manager

A role that is responsible for managing the full annotation lifecycle activities within a workspace. The project manager that is added to a workspace typically performs the activities of an annotation process manager.
* annotator

See [human annotator](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-glossarygloss_H) and [machine learning annotator](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-glossarygloss_M).
* attribute

A characteristic or trait of an entity that describes the entity; for example, the telephone number of an employee is one of the employee attributes.







 B 



* blind data

A set of documents annotated with the ground truth, such as question and answer pairs, semantic annotation, and passage judgment. Blind data is never released or seen by developers and is used to test the system periodically to evaluate performance on unseen data. Testing on blind data prevents accuracy from being tainted by over-fitting to known question sets or annotations. Reported results should only come from tests that are run on blind data. See also [testing data](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-glossarygloss_T) and [training data](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-glossarygloss_T).







 C 



* concordance

Provides a way to ensure that the same mention is annotated with the same entity type throughout a document and across annotation sets. Concordance helps ensure consistency across multiple occurrences of a mention without requiring the human annotator to manually label each occurrence.
A value of true authenticates the credentials, and a value of false invokes the action without any authentication. When set to an integer or a string, this value must match the request X-Require-Whisk-Auth header value. 







 Package annotations 



Annotations specific to packages.

 Annotation Description 

 description A description of the package. 
 parameters An array that describes parameters that are scoped to the package. 







 Parameter annotations 



Parameter annotations.

 Annotation Description 

 name The name of the parameter. 
 description A pithy description of the parameter. 
 doclink A link to further documentation for the parameter (useful for OAuth tokens). 
 required True for required parameters and false for optional ones. 
 bindTime True if the parameter is specified when a package is bound. 
 type The type of the parameter, one of password, array (but can be used more broadly). 







 Activation annotations 

You can document activation records with the following annotations:



Activation annotations.

 Annotation Description 

 path The fully qualified path name of the action that generated the activation. If this activation was the result of an action in a package binding, the path refers to the parent package. 
 kind The kind of action executed, and one of the supported Cloud Functions runtime kinds. 
 limits The time, memory, and log limits that this activation were subject to. 



For sequence-related activations, the system generates the following annotations:



Sequence-related activation annotations.

 Annotation Description 

 topmost This annotation is only present for an outermost sequence action. 
 causedBy This annotation is only present for actions that are contained in a sequence. 
 waitTime The time that is spent waiting in the internal Cloud Functions system. This time is roughly the time that is spent between the receipt of the activation request and when the invoker provisioned a container for the action. This value is only present for non-sequence related activations. For sequences, this information can be derived from the topmost sequence activation record. 
 initTime The time that is spent initializing the function.
The documents are divided between the sets.](https://cloud.ibm.com/docs-content/v1/content/50eddb8fd81f33092880335ff107a78ff5cd0f65/watson-knowledge-studio/images/wks_datacorpus.svg) Figure 1. This figure illustrates two document sets that point to three documents. The documents are divided between the sets.

Ground truth comprises the annotations (mentions, relations, and coreferenced mentions) that are added to documents. Ground truth is singular for each document.

![This figures illustrates that ground truth consists of the annotations that are added to document 1, document 2, document 3, and so on.](https://cloud.ibm.com/docs-content/v1/content/50eddb8fd81f33092880335ff107a78ff5cd0f65/watson-knowledge-studio/images/wks_datagt.svg) Figure 2. This figures illustrates that ground truth consists of the annotations that are added to document 1, document 2, document 3, and so on.

When you create an annotation task, copies of the annotations are created for each document in the annotation set that you add to the task. Human annotators annotate the documents. The annotations are isolated from each other and from ground truth. An annotation task is a temporal concept that exists to allow human annotators to annotate text in isolated spaces. In contrast, ground truth is permanent and singular.

![This figures illustrates that the project manager creates annotation sets and assigns them to an annotation task. Dave and Phil, the human annotators, annotate documents in the sets that are assigned to them.]
> Note: If the task contains completed documents, human annotators cannot alter those documents to assess type system changes until they are back in an editable state. To become editable, ask human annotators to submit the document sets so that you can reject them.



Related concepts:

[Type systems](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-typesystemwks_typesystem)







 Document set management 

Use the right sets of data to test and train the model at the right time.

The documents that you add to the system must be allocated to the following system-level data sets when you create a machine learning model:



* Training set

A set of documents that have been annotated through pre-annotation or by human annotators that is used to train the model. The goal of the training set is to teach the machine learning model about correct annotations, which includes teaching the model through text that was not annotated.
* Test set

A set of annotated documents that is used to test the trained model. After you run a test on the test set, perform a detailed diagnostic-purposed error analysis of the results. Close analysis helps you find weaknesses in the current model that can be addressed.
* Blind set

A set of annotated documents that is set aside and used to test the system periodically after several iterations of testing and improvement have occurred. To prevent accuracy from being tainted (for example, by making changes based only on annotations in known documents), blind data should be data that has not previously been viewed by users involved with creating the model. Reported results should come only from tests that are run on blind data. After you run a test on the blind set, look at only the most high-level scores, such as the overall mention and relation F1 scores. You don't want to learn too many details about the performance or it might influence the improvements that you choose to make to the model.



The goal of Knowledge Studio is to enable large teams to work together to build models.
A subject matter expert who reviews, modifies, and augments the results of pre-annotation by identifying mentions, entity type relationships, and mention coreferences. By examining text in context, a human annotator helps determine ground truth and improve the accuracy of the machine learning model.







 I 



* inter-annotator agreement

A measure of how similarly a document in two or more document sets is annotated.







 K 



* knowledge graph

A model that consolidates typed entities, their relationships, their properties, and hierarchical taxonomies to represent an organization of concepts for a given domain. After the knowledge graph store is loaded with inputs from structured and unstructured data sources, users and applications can access the knowledge graph to explore key elements of knowledge for a specific domain, explore interactions, and discover additional relationships.







 L 



* lemma

The normalized or canonical form of a word. Typically, the lemma is the underived and uninflected form of a noun or a verb. For example, the lemma of the terms 'organizing' and 'organized' is 'organize'. See also [dictionary](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-glossarygloss_D) and [surface form](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-glossarygloss_S).







 M 



* machine learning

A method of data analysis that iteratively learns from past data and independently adapts when exposed to new data. The mathematical model at the core of machine learning is built from ground truth inputs. Through training and refinement of example input data, the model can deliver accurate, repeatable results when it analyzes new data.
* machine learning annotator

See [machine learning model](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-glossarygloss_M).
* machine learning model

A component that identifies entities and entity relationships according to a statistical model that is based on ground truth.