vania-janet/multiturn-rag-retrieval-data huggingface dataset:
You can continue to annotate the documents.



Give Discovery some time to import and process the machine learning model corpus. After the entity extractor is created, the extractor is opened to the Label documents page.





 Training the extractor 

After you label documents, review the training data that will be used to train the entity extractor model.

To train the extractor, complete the following step:



1. Decide whether you want to apply an advanced option. Most models do not require changes to these options.

The following customizations are available from the Review and finish page:



* Include documents that were not reviewed by a person in the training set.

Typically, only documents that a person labeled, reviewed, and explicitly marked complete can be candidates for inclusion in the training set. However, if you want to allow documents that were not marked complete to be included in the training set, you can do so.
* Change the ratio of documents that are included in the document sets that comprise your training data.

The documents from your collection are split at random into the following sets:



* Training set: The documents that you label and that are used to train the entity extractor machine learning model. The goal of the training set is to teach the machine learning model about correct labels.
* Test set: The documents that are used to test the trained model. After you run a test, you can review the results, closely analyze areas where the model got something wrong, and find ways to improve the model's performance.
* Blind set: Documents that are set aside and used to test the model periodically after several iterations of testing and improvement are completed. The documents in the blind set are intentionally roped off. As you test the model with documents from the test set and analyze the results, you become familiar with the underlying test documents. Because the test documents are used iteratively to improve the model, they can start to influence the model training indirectly. That's why the blind set of documents is so important. The blind set gives you a way to generate an unbiased evaluation of the model periodically.



The default split applies a ratio (70%-23%-7%) that is commonly used for machine learning training.



2. Click Train extractor.
[Named Entity Recognition annotator](http://www.ibm.com/support/knowledgecenter/SS8NLW_11.0.0/com.ibm.discovery.es.ta.doc/iiystasystemt.htm)







 Uploading UIMA CAS XMI files into a workspace 

To use the pre-annotated documents that you downloaded to train a model, you must create a .zip file that contains all the files required to upload the XMI files, and then upload the .zip file into a Knowledge Studio workspace.



 Before you begin 

Before you upload the .zip file, ensure that the type system in your Knowledge Studio workspace includes the entity types that you mapped the UIMA types to.

UIMA analysis engines allow annotations to span sentences. In Knowledge Studio, annotations must exist within the boundaries of a single sentence. If the XMI files that you upload include annotations that span sentences, those annotations do not appear in the ground truth editor.





 Procedure 

To upload pre-annotated documents into a Knowledge Studio workspace, follow these steps:



1. Create a .zip file that contains all of the files that are required by Knowledge Studio.



1. Select the folder that contains the XMI files, UIMA type system descriptor file, and cas2di.tsv file, or select all of the files in the folder.
2. Create a .zip file that includes all files. Make sure the cas2di.tsv and UIMA type system descriptor files are stored in the root directory of the .zip file. These files cannot be stored in a subfolder within the .zip file or Knowledge Studio will not be able to read them, and nothing will be imported.

In Windows, you can right-click and select Send to > Compressed (zipped) folder.



2. Upload the .zip file into a Knowledge Studio workspace.



1. Log in as a Knowledge Studio administrator or project manager, open the workspace that you want to add the documents to, and open the Assets> Documents page.
2. Click Upload Document Sets.
3.
[Pre-annotating documents with a dictionary](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-preannotationwks_preannot)







 Adding dictionaries to a workspace 

Adding dictionaries is an optional step in creating a model. Dictionaries are helpful because they enable you to jump start the annotation process.



 About this task 

If you provide a dictionary, you can run the dictionary pre-annotator on the documents. The pre-annotator finds terms that are represented in your dictionary and automatically annotates them. This initial pass on the documents simplifies the human annotator's job because she can review the annotations that were added by the pre-annotator and correct them or add to them. She does not have to start entirely from scratch.

The following restriction apply to dictionaries:



* Maximum 15,000 entries per dictionary

This limit does not apply to dictionaries that you upload as a dictionary CSV file. Read-only dictionaries can contain more entries.
* Maximum 64 dictionaries per workspace







 Procedure 

To add a dictionary to your workspace:



1. Log in as a Knowledge Studio administrator or project manager and open the Assets > Dictionaries page.
2. Perform one of the following tasks:



* Next to the Create Dictionary button, click the Menu icon, and then select Upload Dictionary. Select a dictionary, and then click Upload. After you upload a dictionary, select it to view the dictionary and associate it with an entity type.



You can upload a ZIP file that contains a dictionary that you downloaded from another Knowledge Studio workspace. You must upload the type system that was downloaded from the other workspace in JSON format before you can upload the corresponding dictionary file. You can edit and add entries to a dictionary that you reuse from another Knowledge Studio workspace.
(https://cloud.ibm.com/docs-content/v1/content/148d3bd95f946aa1bb53ea1540475f522e6b61c9/watson-knowledge-studio-data/images/wks_datacorpus.svg) Figure 1. This figure illustrates two document sets that point to three documents. The documents are divided between the sets.

Ground truth comprises the annotations (mentions, relations, and coreferenced mentions) that are added to documents. Ground truth is singular for each document.

![This figures illustrates that ground truth consists of the annotations that are added to document 1, document 2, document 3, and so on.](https://cloud.ibm.com/docs-content/v1/content/148d3bd95f946aa1bb53ea1540475f522e6b61c9/watson-knowledge-studio-data/images/wks_datagt.svg) Figure 2. This figures illustrates that ground truth consists of the annotations that are added to document 1, document 2, document 3, and so on.

When you create an annotation task, copies of the annotations are created for each document in the annotation set that you add to the task. Human annotators annotate the documents. The annotations are isolated from each other and from ground truth. An annotation task is a temporal concept that exists to allow human annotators to annotate text in isolated spaces. In contrast, ground truth is permanent and singular.

![This figures illustrates that the project manager creates annotation sets and assigns them to an annotation task. Dave and Phil, the human annotators, annotate documents in the sets that are assigned to them.](https://cloud.ibm.com/docs-content/v1/content/148d3bd95f946aa1bb53ea1540475f522e6b61c9/watson-knowledge-studio-data/images/wks_datatask.svg) Figure 2.
For more information about annotation sets and annotation tasks, see [Creating an annotation task](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-annotate-documentswks_hatask).





 Procedure 



1. Within your workspace, click Machine Learning Model > Annotations.
2. Click the Annotation Tasks tab, then click Add Task.
3. Specify the details for the task:



* In the Task name field, enter Test.
* In the Deadline field, select a date in the future.



4. Click Create Annotation Sets.

The Create Annotation Sets window opens. By default, this window shows the base set, which contains all documents, and fields where you can specify the information for a new annotation set.
5. Click Add another set and human annotator to add fields for an additional annotation set. You can click to add as many annotation sets as you want to create. For this tutorial, you need only two annotation sets.

![A screen capture of the Create Annotation Sets page.](https://cloud.ibm.com/docs-content/v1/content/50eddb8fd81f33092880335ff107a78ff5cd0f65/watson-knowledge-studio/images/wks_tutdocset2.png)
6. In the Overlap field, specify 100. This value specifies that you want 100 percent of the documents in the base set to be included in all the new annotation sets so they can be annotated by all human annotators.
7. For each new annotation set, specify the required information.



* In the Annotator field, select a human annotator user ID to assign to the new annotation set. In a realistic scenario, each annotation set is assigned to a different human annotator.

If you have only a single administrator ID to use for the tutorial, assign that user to all annotation sets.
For more information about adjudication, see [Building the ground truth](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-build-groundtruth).





 Procedure 



1. Log in to Knowledge Studio as the administrator and select Machine Learning Model > Annotations. Click the Annotation Tasks tab, then click the Test task.
2. Verify that the two document sets are in an approved state.
3. Click Check Overlapping Documents for Conflicts.

You can see the overlapping documents that were annotated by more than one human annotator.
4. Because the tutorial instructed you to create a conflicting relation for the Technology - gmanews.tv document, find that document in the list and click Check for Conflicts.
5. Select the two conflicting annotation sets and click Check for Conflicts.

Adjudication mode opens. In adjudication mode, you can view overlapping documents, check for conflicts, and remove or replace annotations before you promote the documents to ground truth.
6. Select Relation conflicts, accept the founderOf relation, and reject the employedBy relation.
7. Click Promote to Ground Truth.

Alternatively, you can promote a document to ground truth by clicking Accept on the Documents page.







 Results 

After you resolve the annotation conflicts and promote the documents to ground truth, you can use them to train the machine learning model.







 Lesson 7: Creating a machine learning model 

In this lesson, you will learn how to create a machine learning model in Knowledge Studio.



 About this task 

When you create a machine learning model, you select the document sets that you want to use to train it. You also specify the percentage of documents that are to be used as training data, test data, and blind data. Only documents that became ground truth through approval or adjudication can be used to train the machine learning model.
For more information about annotation sets and annotation tasks, see [Creating an annotation task](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-annotate-documentswks_hatask).





 Procedure 



1. Within your workspace, click Machine Learning Model > Annotations.
2. Click the Annotation Tasks tab, then click Add Task.
3. Specify the details for the task:



* In the Task name field, enter Test.
* In the Deadline field, select a date in the future.



4. Click Create Annotation Sets.

The Create Annotation Sets window opens. By default, this window shows the base set, which contains all documents, and fields where you can specify the information for a new annotation set.
5. Click Add another set and human annotator to add fields for an additional annotation set. You can click to add as many annotation sets as you want to create. For this tutorial, you need only two annotation sets.

![A screen capture of the Create Annotation Sets page.](https://cloud.ibm.com/docs-content/v1/content/148d3bd95f946aa1bb53ea1540475f522e6b61c9/watson-knowledge-studio-data/images/wks_tutdocset2.png)
6. In the Overlap field, specify 100. This value specifies that you want 100 percent of the documents in the base set to be included in all the new annotation sets so they can be annotated by all human annotators.
7. For each new annotation set, specify the required information.



* In the Annotator field, select a human annotator user ID to assign to the new annotation set. In a realistic scenario, each annotation set is assigned to a different human annotator.
* Use only the following alphanumeric ASCII characters and the underscore character in values that you add to the classes: A through Z, a through z, 0 through 9.
* Names cannot contain spaces.
* Names cannot be longer than 64 characters.





4. Optional: To quickly annotate classes in a document, you can associate a dictionary with the rule editor. Terms in a document that match entries in the dictionary are automatically annotated with the class that you select for the dictionary.



1. Click the Dictionaries tab.

Any dictionaries that you created are displayed.

If you have not added a dictionary, open the Assets > Dictionaries page from the main navigation bar to add one. See [Creating dictionaries](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-dictionaries) for more information.
2. Click a dictionary, associate a class with the dictionary, and then click Save.

For example, if you have a dictionary that contains organization names, you can create a rule class called ORGANIZATION, and associate the class with the dictionary. Any organization names that occur in your sample documents will be annotated as instances of the ORGANIZATION class.



If you want to remove the dictionary association from the rule editor later, then you can remove the class mapping. To do so, choose the empty option from the top of the drop-down list.
5. Optional: To define a regular expression to help you construct a rule, click the Regex tab.



1. Click the plus sign (+) next to the Regular Expressions heading.
2. Name the regular expression. For example, MyAgeRegex.

The name cannot be longer than 64 characters.
3. Associate the expression with a class. For example, AGE_REGEX.
4. Click Add Entry.
5. Add the expression.

For example, to capture a number that represents an age (up to age 99), you can specify [0-9]{1,2}. To capture expressions of time, like 12:30 AM, you might specify this regular expression:
[Resolve any annotation conflicts](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-build-groundtruthwks_haadjudicate).
4. When all the documents are part of the ground truth and the task is complete, delete the task as described in [Deleting a document that is associated with an annotation task and human annotation has not begun](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-documents-for-annotationdeletenoannop).
5. Delete the document as described in [Deleting a document that has not been associated with an annotation task](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-documents-for-annotationdeletenotask).

You can confirm that the annotations on the remaining documents are not lost by downloading the document sets and reviewing the documents in the gt folder.











 Data model 

The diagrams in this topic summarize the flow of documents in a Knowledge Studio system and the differences between documents in the corpus, an annotation task, and ground truth.

The corpus contains documents, which are partitioned into document sets:



* A document is nothing more than strings of text.
* A document set is a pointer to a group of documents. The document set does not contain copies of the documents themselves.
* Some document sets can point to a single document, a setup that you can control through the overlap parameter that you specify when you create annotation sets.



![This figure illustrates two document sets that point to three documents. The documents are divided between the sets.](https://cloud.ibm.com/docs-content/v1/content/50eddb8fd81f33092880335ff107a78ff5cd0f65/watson-knowledge-studio/images/wks_datacorpus.svg) Figure 1. This figure illustrates two document sets that point to three documents. The documents are divided between the sets.
Ground truth is then used for training and testing a machine learning model, or it can be used as the basis for the next iteration of model development. To use ground truth in a new iteration, you must create a new annotation task.

![This figure illustrates how annotations added by two human annotators become ground truth. One document, labeled document 2, is annotated by both human annotators. The annotations in this overlapping document must be adjudicated before they become ground truth.](https://cloud.ibm.com/docs-content/v1/content/50eddb8fd81f33092880335ff107a78ff5cd0f65/watson-knowledge-studio/images/wks_datatruth.svg) Figure 3. This figure illustrates how annotations added by two human annotators become ground truth. One document, labeled document 2, is annotated by both human annotators. The annotations in this overlapping document must be adjudicated before they become ground truth.
Don't worry. You don't need to label every page. As you apply labels and submit pages, Watson learns from what you annotate and starts to predict annotations.

Follow these guidelines:



* If there's nothing special about a section, leave it labeled as text, which is applied by default.
* A label cannot span multiple pages.
* Do not treat bold, italic, or underlined text differently. Label based on the context, not the style.
* Use consistent labeling on all documents.
* Work from the first page of a multipage document to the last.
* To remove a single annotation, choose another label (such as text) and apply it to the item to overwrite the previous annotation.
* To remove annotations that you added to an entire page, click the Clear changes icon in the toolbar.
* To annotate a table, click the text at the start of the table and then drag to select the text in the entire table.
* When you label one or more tables, the Table Understanding enrichment is enabled for the entire collection automatically. For more information, see [Understanding tables](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-understanding_tables).
* Images from the source documents are not rendered in the preview. If Optical Character Recognition (OCR) is enabled, any text from the image or diagram is extracted and rendered in the preview.
* Do not label white space.



7. When everything that you want to label is labeled, submit the page. Click Submit page.

Continue annotating documents until Watson can correctly and consistently map different types of content to the appropriate fields for you.
8. After you teach Watson to identify fields, click Apply changes and reprocess.



Custom fields that you define by using the SDU tool are indexed as root-level fields.



 What to do next 

When you build a user-trained model, you change where information is stored in your documents. Next, change how the search results are configured. By default, search results are retrieved from passages or the text field.
Each surface form must be 256 or fewer characters in length. You can change which of the surface forms is used as the lemma. For example, the lemma IBM might have surface forms like IBM Corp. and International Business Machines, Inc..
5. Select the appropriate part of speech for each lemma and surface form in the dictionary.

The part of speech information is used by the tokenizer, and during pre-annotation.
6. Click Save to store your changes.







 What to do next 

Run the pre-annotator, which uses the dictionaries that you created to do a preliminary pass of the source documents, and adds annotations to them.

Related tasks:

[Pre-annotating documents with a dictionary](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-preannotationwks_preannot)

Related reference:

[Language support](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-language-support)
Configuring the order of pre-annotators 

When multiple pre-annotators are used, the first annotation made to a span of text is saved for the results, even if other pre-annotators attempt to annotate the same span of text later in the order. This doesn't apply to human annotations, which are preserved regardless of pre-annotation order.

For example, consider the example text IBM Watson. If a dictionary that is first in the order labels IBM as an Organization entity type, a machine learning model that is second in the order can't annotate IBM Watson as a Software Brand entity type because that would override the earlier annotation made to IBM.

You can view the current order of pre-annotators in the Order column on the Machine Learning Model > Pre-annotation page. To change the order, complete the following steps.



1. Click Order Settings.
2. Click the Move up and Move down arrow** buttons to move pre-annotation methods earlier or later in the order.
3. Click Save.
4. Double check the Order column on the Pre-annotation page to make sure that it matches the order that you want.







 Run pre-annotators 



1. After your pre-annotation methods are prepared and you have configured the order of your pre-annotators, click Run Pre-annotators.
2. Select the pre-annotators that you want to use, and then click Next.
3. If you want to erase existing annotations made by pre-annotators before running the pre-annotator, select Wipe previous pre-annotation results. Human annotations are preserved even if this is selected.
4. Select the document sets that you want to pre-annotate.
5. Click Run.









 Pre-annotating documents with Natural Language Understanding 

You can use the Natural Language Understanding service to pre-annotate documents that you add to your corpus.



 Before you begin 

Determine whether the Natural Language Understanding pre-annotator is likely to add value for your use case.