vania-janet/multiturn-rag-retrieval-data huggingface dataset:
<br><br> * CSV file in UTF-8 format<br> * Text in UTF-8 format<br> * HTML<br> * PDF files (scanned and password-protected files are not supported)<br> * Microsoft Word DOC or DOCX files (password-protected files are not supported)<br> * ZIP file that contains documents downloaded from another workspace<br> * ZIP file that contains documents in UIMA CAS XMI format<br><br><br> ZIP archive file of documents <br><br> * 40,000 characters per document<br> * 10,000 documents per workspace<br> * 1,000 document sets (including annotation sets) per workspace<br> * 5 MB per file and 200 MB per upload (TXT, PDF, DOC, DOCX, and HTML files)<br><br><br> 
 Pre-annotation Use a dictionary or IBM Watson® Natural Language Understanding pre-annotator to provide a starting point for human annotation. <br> <br>You cannot re-annotate a corpus from IBM Watson Explorer. Raw documents. <br> <br>Note: Do not pre-annotate documents that a human annotator has already annotated, or you will lose the work done by the human annotator. Partly-annotated documents None 
 Document annotation Manage human annotation. Annotate entities, relations, and coreference chains to create ground truth Annotation task Ground truth <br><br> * 256 active anntation tasks per workspace<br><br><br> 
 Training and refinement Train a supervised machine learning model to extract domain-specific information from unstructured text. Evaluate and improve a supervised machine learning model. You cannot create a semi-supervised or unsupervised machine learning model. You cannot do extensive feature engineering. Not applicable Machine learning model <br><br> * 1 machine learning model per workspace<br> * 10 model versions per workspace<br> * Maximum number of workspaces is determined by your deployment.<br> * The maximum number of training actions you can perform per month is determined by your deployment.<br><br><br>
Human annotators can specify entity subtypes for each pre-annotated mention by working on an [annotation task](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-annotate-documentswks_hatask) with the pre-annotated document.
* Machine learning

Uses a machine learning model to automatically annotate documents. This option is only available to you if you have created a machine learning model with Knowledge Studio already. If you add a document set, you can run the machine learning annotator that you created previously to pre-annotate the new documents. If the new set of documents is similar to the documents that were used to train the machine learning annotator originally, then this is probably your best choice for pre-annotation.
* Rule

Uses a rule-based model to automatically annotate documents. This option is only available if you have created a rule-based model with Knowledge Studio already. If your documents contain common patterns of tokens from which you can derive meaning, this model might be a good choice. It can incorporate some of the function of the dictionary pre-annotator if you enable it, by identifying class types for dictionary terms that it finds in the documents also.



Alternatively, you can upload already-annotated documents, and use them to start training the machine learning model. You cannot run a pre-annotator on annotated documents that you upload or the existing annotations will be stripped from the documents and replaced with annotations produced by the pre-annotator only.

You can run a pre-annotator on documents that were added to the ground truth as part of the current workspace. Annotations that were added to the documents, reviewed, accepted, and promoted to ground truth within the current workspace are not stripped out.





 Running multiple pre-annotators 

Knowledge Studio allows you to run multiple pre-annotators at once. First, you need to prepare the pre-annotation methods that you want to use. For more information, see the following sections.
If the same text (USER_NAME) is used in the documents or text is redacted, you will basically be training the model to expect all names to have that same value or be redacted. When the model is used at runtime on new documents, and encounters never-seen-before names in all their variability, you want it to be able to recognize them as names.







 Adding documents to a workspace 

To train a model, you must add documents that are representative of your domain content to your workspace.



 About this task 

As a best practice, start with a relatively small collection of documents. Use these documents to train human annotators (if your workspace involves human annotation) and to refine the annotation guidelines. Small documents can help human annotators identify coreference chains throughout the document. As annotation accuracy improves, you can add more documents to the corpus to provide greater depth to the training effort.





 Procedure 

To add documents to a workspace:



1. Log in as a Knowledge Studio administrator or project manager, and select your workspace.
2. Select the Assets> Documents > Documentation sets tab.
3. Click Upload Document Sets to add documents to the corpus.
4. Upload documents in one of the supported formats. For more information about the supported document types, size limits, and other information, see [Creating a workspace > Summary of inputs, outputs, and limitations](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-create-projectwks_formats).

Notes about .zip files of documents downloaded from another workspace

When annotated documents are imported, they are re-tokenized. This process can change what Knowledge Studio considers to be the sentence boundaries. Because annotations are defined by sentence, some annotations might be invalidated during this process. After uploading documents from another workspace, do a quick review of the annotations to address any discrepancies.



* If you previously downloaded documents from a Knowledge Studio workspace, drag the .zip file that contains the downloaded documents or click to locate and select the file.
An option lets you specify whether you want the ground truth annotations to be included in the imported files.

After documents are annotated, the annotated documents are stored in JSON format. The markup language in these files, which shows how the original document text was parsed and tokenized, includes elements for all of the annotations that a human annotator added. To improve model accuracy over time, you can upload these files into another workspace, thus preserving all of the existing annotations. A human annotator can revise, delete, and add annotations to these documents, or you can bypass human annotation and use these files to create training, test, and blind document sets for evaluating and improving the model performance.





 UIMA CAS XMI files 

To help train a model, you can upload documents that were pre-annotated by a UIMA analysis engine. The pre-annotated files must be in XMI serialization of UIMA Common Analysis Structure (UIMA CAS XMI) format and combined into a ZIP file. For example, you can upload documents that were annotated in an IBM Watson Explorer collection.

A human annotator can revise, delete, and add annotations to these documents, or you can bypass human annotation and use these files to create training, test, and blind document sets for evaluating and improving the model performance. For details about how to create these files and requirements for uploading them, see [Uploading pre-annotated documents](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-preannotationwks_uima).





 Anonymizing data 

If you want to build a model that is optimized for your data, but do not want to upload the data as-is to Knowledge Studio for privacy reasons, you can strip the documents of any personally identifiable information (PII) first, and then use those anonymized documents to train the model. Do not redact the information or replace it wholesale with variables. For best results, replace the real information with fake information of the same type.
5. Choose the annotator type, and then click Next.
6. Follow the on-screen instructions.

For more information about how to configure each annotator type, see one of the following sections:



* [Dictionary](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-custom-annotatorcm-custom-annotator-dictionary)
* [Machine learning model](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-custom-annotatorcm-custom-annotator-ml)
* [Regular expressions](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-custom-annotatorcm-custom-annotator-regex)







 Dictionary configuration 

You can import an existing dictionary by uploading it or you can create a dictionary by adding terms one at a time.

If you plan to import a dictionary, the dictionary terms must be defined in a CSV file. Specify each term and its synonyms in a separate line. Use the following syntax to specify each term:

{term},{synonym},{synonym},...

To add a dictionary, complete the following steps:



1. Do one of the following things:



* To import the dictionary terms:



1. Click Import, and then browse for the file with your dictionary terms.
2. Click Import.



* To define the dictionary terms:



1. Click Add.
2. Click Word list to add the dictionary terms.
3. Click Add, and then add the term in the Base word field and any synonyms that you want to define for the term in the Other words field. Separate multiple synonyms with commas. Click OK.
4. Repeat the previous step to add more dictionary terms.
5. After you finish adding dictionary terms, click Basic settings.





2. Name the dictionary.
3.
* Predicted labels are the examples that the entity extractor identified and labeled as entity types.
* The ground truth has examples that a person labeled or that were bulk labeled and reviewed by a person. Labels in the ground truth are considered the correct labels.



The performance of the model is rated based on how closely the predicted labels match the ground truth.





 Improving the extractor 

The following table shows suggested fixes for common problems.



Improvement actions

 Problem Action to remedy the problem 

 Low overall scores You might not have enough documents with labeled examples in your training set. Label more examples in more of your documents. 
 Low recall Label more documents with new examples of the entity types that the extractor missed. 
 Low precision Look for entity types that are commonly confused. Find and label more examples of each entity type to help the entity extractor distinguish between the entity types. 







 Adding documents to the training data 

To add more documents, complete the following steps:



1. Open the Label documents tab.
2. From the Document list panel, choose Add documents.

This button is disabled if no other documents are available to add to the entity extractor from the current collection. To add more documents to the collection, go to the Activity page for the collection, and then click the Upload data tile to browse for and add more files.



You cannot choose the documents from the collection to show in the Document list for labeling purposes. If there are specific types of documents that you want to label, consider adding representative documents to a collection that you can use to create the entity extractor.

There are limits to the number of documents that can be included in the training data. If your training data includes documents with a combination of sections that are labeled and others that are not, the system might sample some examples from unlabeled sentences. Subsampling helps to balance the number of positive and negative examples that are used for training. Balancing the examples in the training set improves the training performance.







 Publishing the entity extractor as an enrichment 

When you think the entity extractor is ready, publish the entity extractor. How do you know when it's ready? If the score doesn't change after several test runs in which you make improvements, the model is ready.
[Natural Language Understanding](https://www.ibm.com/watson/services/natural-language-understanding/)







 Pre-annotating documents with a dictionary 

To help human annotators get started with their annotation tasks, you can create a dictionary and use it to pre-annotate documents that you add to the corpus.



 About this task 

When a human annotator begins work on documents that were pre-annotated, it is likely that a number of mentions will already be marked by entity types based on the dictionary entries. The human annotator can change or remove the pre-annotated entity types and assign entity types to unannotated mentions. Pre-annotation by a dictionary does not annotate relations, coreferences. Relations and coreferences must be annotated by human annotators.

This task shows you how to create a dictionary that is editable. If you want to upload and pre-annotate your documents with a read-only dictionary, click the Menu icon next to the Create Dictionary button, and then select Upload Dictionary.





 Procedure 

To create an editable dictionary and pre-annotate documents, follow these steps:



1. Log in as a Knowledge Studio administrator and select your workspace.
2. Select the Assets > Dictionaries page.
3. Click Create Dictionary, enter a name, and then click Save.
4. From the Entity type list, select an entity type to associate with the dictionary.

You can also associate an entity type with the dictionary from the Machine Learning Model > Pre-annotation page. Click the overflow menu button in the Dictionaries row in the page, then click Map entity types.
5. Add entries for the dictionary or upload a file that contains dictionary terms.
6. Go to the Machine Learning Model > Pre-annotation page.
7. Click Run Pre-annotators.
8. Select Dictionaries, and then click Next.
9. If you want to erase existing annotations made by pre-annotators before running the pre-annotator, select Wipe previous pre-annotation results.
It can incorporate some of the function of the dictionary pre-annotator if you enable it, by identifying class types for dictionary terms that it finds in the documents also.



Alternatively, you can upload already-annotated documents, and use them to start training the machine learning model. You cannot run a pre-annotator on annotated documents that you upload or the existing annotations will be stripped from the documents and replaced with annotations produced by the pre-annotator only.





 Running multiple pre-annotators 

Knowledge Studio allows you to run multiple pre-annotators at once. First, you need to prepare the pre-annotation methods that you want to use. For more information, see the following sections:



* [Natural Language Understanding](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-preannotationwks_preannotnlu)
* [Dictionaries](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-preannotationwks_preannot)
* [Machine learning model](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-preannotationwks_preannotsire)
* [Rules-based model](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-preannotationwks_preannotrule)





 Configuring the order of pre-annotators 

When multiple pre-annotators are used, the first annotation made to a span of text is saved for the results, even if other pre-annotators attempt to annotate the same span of text later in the order. This doesn't apply to human annotations, which are preserved regardless of pre-annotation order.

For example, consider the example text IBM Watson.
Recall, which is a measure of sensitivity, is determined by the number of correct positive results divided by the number of positive results that should have been returned. Accuracy is best measured by using both precision and recall. See also [accuracy](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-glossarygloss_A) and [precision](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-glossarygloss_P).
* relation

Typically a verb that reflects how entities are related to one another. For example, "lives in" is a relation between a person and a town. A relation links two different entities in the same sentence.
* relation type

A binary, uni-directional relationship between two entities. For example, MaryemployedBy IBM is a valid relationship; IBM employedByMary is not.
* role

An attribute that provides a context-sensitive meaning of a mention. For example, in the phrase "I went to IBM today", IBM is the mention, Organization is the entity type, and Facility is the role of the entity type.
* rule set

A set of rules that define patterns for annotating text. If a pattern applies, then the actions of the rule are performed on the matched annotations. A rule typically specifies the condition that must match, an optional quantifier, a list of additional constraints that the matched text must fulfill, and the actions to be taken when a match occurs, such as creating a new annotation or modifying an existing annotation.







 S 



* subtype

A type that extends or implements another type; the supertype.
* surface form

The form of a word or multiword unit as it is found in the corpus. For example, some surface forms of the lemma 'organize' are the terms 'organizing' and 'organized'.
Each time that you train the model, review the performance metric scores to determine whether your most recent changes increase or decrease the model's scores.



1. To compare one test run against another, click View score history.

The history view shows the last 5 training runs.

To retain the score information for more than the most recent 5 training runs, you can export the metrics in comma-separated value format, and track the scores in a separate application. Click the tabular representation icon ![Tabular representation icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/table-of-contents.svg), and then click Download as CSV.



If a subsequent training run results in lower scores, don't publish that version of the model.







 Deleting an entity extractor 

You can delete an entity extractor if it is not in use, meaning the enrichment that is published from the entity extractor is not applied to a collection.

You might want to delete an entity extractor if you hit the limit for the maximum number of extractors that are allowed for your plan, for example.

Remember, limits are defined per service instance, not per project. If you cannot create new entity extractors, but do not have the maximum number of extractors in the current project, check other projects in the same service instance. There might be entity extractors that aren't being used in other projects that can be deleted.



1. Remove the entity extractor enrichment that was published from the entity extractor that you want to delete from any collections where it is being used.

For more information, see [Deleting enrichments](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-manage-enrichmentsenrichments-delete).
2. From the Improvement tools panel of the Improve and customize page, expand Teach domain concepts, and then click Extract entities.
3. Find the entity extractor that you want to delete, click the Actions icon, and then select Delete.
Bootstrapping annotation 

Simplify the job of the human annotator by pre-annotating the documents in a workspace. A pre-annotator is a Knowledge Studio dictionary, rule-based model, or machine learning model that you can run to find and annotate mentions automatically.

Pre-annotation makes the job of human annotators easier because it covers the straightforward annotations, and gets the job of annotating the documents underway.

The method that you use to pre-annotate documents in no way restricts the ways that you can use the resulting model. For example, just because you use the Natural Language Understanding service to pre-annotate documents does not mean you must deploy the final machine learning model that you build to the Natural Language Understanding service.



 Pre-annotation methods 

The following pre-annotators are available:



* Natural Language Understanding

A pre-annotator that you can use to find mentions of entities in your documents automatically. If your source documents have general knowledge subject matter, then this pre-annotator is a good choice for you. If you are working with highly specialized documents that focus on a specific field, such as patent law research, for example, the dictionary pre-annotator or rule-based model might be a better choice.
* Dictionary

Uses a dictionary of terms that you provide and associate with an entity type to find mentions of that entity type in the documents. This choice is best for fields with unique or specialized terminology because this pre-annotator does not analyze the context in which the term is used in the way a machine learning pre-annotator does; it instead relies on the term being distinct enough to have a decipherable meaning regardless of the context in which it is used. For example, it is easier to recognize asbestos as a mineral entity type than to determine the entity type of squash, which can refer to a vegetable, a sport, or a verb meaning to crush something.

Dictionary pre-annotators do not recognize entity subtypes.
FISMA support is also available to those who purchased the June 28, 2019 version and upgrade to the August 30, 2019 version. IBM Watson™ Knowledge Studio for IBM Cloud Pak for Data is FISMA High Ready.



Annotation tasks are no longer required to create ground truth.



* Admins and Project Managers can annotate document sets directly from the Ground Truth tab on the Annotations page.
* Annotation tasks are still available. You can manage them from the Annotation Tasks tab on the Annotations page.
* Annotations applied directly to ground truth are not applied to related active annotation tasks. It is recommended that you annotate document sets directly only when they are not associated with any active annotation tasks.
* The Annotation Sets tab has been removed from the Documents page. You can manage annotation sets from the Annotation Tasks tab of the Annotations page.



* To create new annotation sets, click Add task. Then, click Create Annotation Sets.
* To manage existing annotation sets, click an existing annotation task, then click Edit.









 Version 1.0.0 (30 June 2019) 

Knowledge Studio for IBM Cloud Pak for Data is now available as an add-on to the IBM Cloud Private platform.