vania-janet/multiturn-rag-retrieval-data huggingface dataset:
If you want to view the annotations in greater context, click the icons to preview the document content or open the document in a new window.
10. Click Apply & Review to apply the selected entity types to mentions in the selected documents. You still have a chance to review the annotations that will be added. If an annotation is inaccurate in a particular context, you can remove that occurrence by clicking the Edit icon, and then removing the entity type assignment for the mention.
11. When you are happy with the list of annotations, click Go Back to Ground Truth Editor .







 Results 

The mentions are annotated in the document. There is no way to remove the set of mentions that you added through concordance at once. You must remove each mention, one at a time.







 Annotating mentions as coreferences 

To annotate mentions as coreferences to the same entity, a human annotator selects every occurrence of a mention that refers to the same thing. Coreference helps a model recognize that entities that are referred to in different ways are to be associated with the same entity, such as the name of a U.S. state and its abbreviation, the name of a company and its acronym, or a person's name and a pronoun that refers back to that person.



 Before you begin 

You must annotate mentions in the document before you can identify coreferences.





 About this task 

When you annotate mentions as coreferences, the system creates a coreference chain. The chain provides a way for you to view all of the mentions in context and verify that all of the occurrences belong together under the same entity. For example, "Barack", "Michelle", "he", and "she" are all of the same entity type, PERSON, but "Barack" and "he" are one entity, and "Michelle" and "she" are another entity. In this example, you create two coreference chains.

When you create a coreference chain, you must select mentions that have been marked by the same entity type. In some cases, however, you might want to include mentions of different types in the same coreference chain. To do this, you must create multiple chains and then merge them.
In this mode, the human annotator connects mentions by associating a relation type, as defined in the type system. For example, the mention John Smith might be connected to the mention IBM by the relation type employedBy. The annotation of relation types is optional and can occur before or after you annotate mentions as coreferences.
* Coreference mode

In this mode, the human annotator identifies mentions that mean the same thing, thus helping to ensure consistency in the annotations when words are not identical. For example, the mention of IBM in the first sentence, the mention of International Business Machines, and the mention of IBM in a later sentence refer to the same thing and would all be labeled by the same entity type, such as ORGANIZATION. The annotation of mentions as coreferences is optional and can occur before or after you annotate relation types.





 Tips for using the editor 



* Save your work as you go.
* If you make a mistake, you can press Ctrl+Z to undo the previous action. To redo the action after undoing it, press Ctrl+Y. You can undo the previous 10 actions that you performed while editing the current document. The actions are lost as soon as you close the document. The actions must be undone in reverse order, and you must switch to the mode that you were in when you performed the action to undo it. You cannot undo and redo concordance tool actions.









 Annotating entity mentions 

To annotate entity mentions, a human annotator selects a string of text in a document, and then applies a label that most appropriately describes what the string of text represents. The labels that can be applied are entity types defined in the workspace's type system.



 About this task 

Before starting to annotate entity mentions in a document, it's a good practice to read the entire document. Doing so can help keep the entire context in mind while annotating, and can help provide insights into how entity mentions might relate to each other and which mentions might need to be coreferenced in future passes through the document.
After the documents have been added, click the document names to preview them and verify that the content looks OK. For example, verify that text files are in UTF-8 format and that no diacritical marks or character normalization issues are visible in the documents, and check for poor sentence breaks. If problems exist, you might need to pre-process the files before you add them to the corpus. You want the documents to be as clean and well-formatted as possible before dictionary or human annotation begins.







 What to do next 

Before you start any human annotation tasks, divide the corpus into multiple document sets and assign the document sets to human annotators.

Admins and project managers can [annotate document sets directly](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-annotating-document-sets-directly) without creating annotation tasks.







 Deleting documents 

You can delete a document if you determine that it does not represent standard industry text that will benefit the model.

To delete a document, choose the option that applies to your situation:



* [Deleting a document that has not been associated with an annotation task](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-documents-for-annotationdeletenotask)
* [Deleting a document that is associated with an annotation task and human annotation has not begun](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-documents-for-annotationdeletenoanno)
* [Deleting a document that is associated with an annotation task and human annotation has begun](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-documents-for-annotationdeleteanno)





 Deleting a document that has not been associated with an annotation task 

If the document you want to delete is not associated with an annotation task, complete these steps to delete the document.



 Procedure
Annotating document sets directly 

Admins and project managers are able to annotate ground truth document sets directly without creating annotation tasks.



  Procedure 

Annotations that you apply directly to document sets are not applied to active annotation tasks. To avoid unintended overwrites, do not annotate document sets that are involved in active annotation tasks.



1.  [Launch the Knowledge Studio application](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks_tutintrolaunching-the-knowledge-studio-application).
2.  [Add documents to your workspace.](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-documents-for-annotationwks_projadd)
3.  From your workspace menu, click Machine Learning Model, then click Annotations.
4.  Click the Ground Truth tab.
5.  Click Annotate next to the document set that you want to use.
6.  Annotate your documents. See [Annotating documents](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-user-guide) for more details.
7.  Click Save to update the document set with your annotations.







  What to do next 



*  [Train a machine learning model](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-train-ml).
Before you start any human annotation tasks, divide the corpus into multiple document sets and assign the document sets to human annotators.

Admins and project managers can [annotate document sets directly](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-annotating-document-sets-directly) without creating annotation tasks.







 Deleting documents 

You can delete a document if you determine that it does not represent standard industry text that will benefit the model.

To delete a document, choose the option that applies to your situation:



* [Deleting a document that has not been associated with an annotation task](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-documents-for-annotationdeletenotask)
* [Deleting a document that is associated with an annotation task and human annotation has not begun](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-documents-for-annotationdeletenoanno)
* [Deleting a document that is associated with an annotation task and human annotation has begun](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-documents-for-annotationdeleteanno)





 Deleting a document that has not been associated with an annotation task 

If the document you want to delete is not associated with an annotation task, complete these steps to delete the document.



 Procedure to delete 



1. Log in as a Knowledge Studio administrator and select your workspace.
2. Select the Assets> Documents > Document sets tab.
3. Select the document set that contains the document you want to delete. The document set opens.
4. Find the document that you want to remove, and then click Delete.









 Deleting a document that is associated with an annotation task and human annotation has not begun
These changes are the result of migrating the Watson Assistant platform to Java 17, where locale values are updated by using specifications in [CLDR 39](https://cldr.unicode.org/index/downloads/cldr-39).

To avoid or minimize the impact of similar changes in the future, you can use [Actions display formats](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-actions-global-settingsactions-global-settings-display-formats).





 18 May 2023 

Differences in contextual entity detection for dialog skills with few annotations
: If you have 10 to 20 examples of contextual entities in your dialog skill, you might see differences in the entities detected due to updates made to address critical vulnerabilities. The impact of these differences is limited to only newly-trained models. Existing models are unaffected. You can mitigate these differences by annotating more examples. For more information, see [Annotation-based method](https://cloud.ibm.com/docs/assistant?topic=assistant-entitiesentities-annotations-overview).





 15 May 2023 

Change to dialog skill context variables named request
: If your dialog skill used a context variable that is named request, it was removed from the response payload of any /message calls in the V1 or V2 API, or through the Watson Assistant user interface. After 15 May 2023, this behavior changes. Watson Assistant doesn't remove context variables that are named request from the response payload anymore.





 3 May 2023 

Algorithm version Beta provides improved intent detection and action matching
: The algorithm version Beta now provides improved intent detection and action matching. It includes a new foundation model that is trained using a transformer architecture to improve intent detection and action matching for English.

Improvements include:



* Improved robustness to variations in user inputs such as typos and different inflection forms
* Less training data required to reach the same level of performance compared to previous algorithms



For more information, see [Algorithm version and training](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-algorithm-version).
: When you identify fields, instead of annotating documents with the SDU tool, you can choose to use a pretrained model. The pretrained model applies a non-customizable model that automatically extracts text and identifies tables, lists, and sections.

Improved contract analysis
: To enable the Contracts enrichment that recognizes and tags contract-related concepts in your data, you can choose to create a Document Retrieval project type, and then select Apply contracts enrichment. You no longer need to use an installation override YAML file to enable it. This change also means that you can choose which Document Retrieval projects use the Contracts enrichment; it is not applied to all Document Retrieval projects automatically.

New LDAP directory data source
: Connect to data that is stored in an external directory that supports the Lightweight Directory Access Protocol (LDAP), such as a corporate email directory. As the directory data is added to your collection, Discovery interprets and stores key attributes of each record, such as department and location information. Later, you can find relevant records by filtering on these attribute categories. For more information, see [LDAP directory](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-ldap-cp4d).

Improved SharePoint OnPrem connection process
: The steps you follow to connect to a SharePoint instance that is hosted on-premises were simplified. You no longer need to deploy a web services package on the SharePoint server before you can connect to the SharePoint OnPrem data source. For more information, see [SharePoint OnPrem](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-sharepoint-onprem-cp4d).

New Salesforce proxy support
: You can now connect to a Salesforce data source when using a proxy server. For more information, see [Salesforce](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-salesforce-cp4d).

Improved custom connector improvements
: Support was added for Optical character recognition (OCR)
: Support was added for Document-level security
Annotating document sets directly 

Admins and project managers are able to annotate ground truth document sets directly without creating annotation tasks.



  Procedure 

Annotations that you apply directly to document sets are not applied to active annotation tasks. To avoid unintended overwrites, do not annotate document sets that are involved in active annotation tasks.



1.  [Launch the Knowledge Studio application](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-wks_tutintrolaunching-the-knowledge-studio-application).
2.  [Add documents to your workspace.](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-documents-for-annotationwks_projadd)
3.  From your workspace menu, click Machine Learning Model, then click Annotations.
4.  Click the Ground Truth tab.
5.  Click Annotate next to the document set that you want to use.
6.  Annotate your documents. See [Annotating documents](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-user-guide) for more details.
7.  Click Save to update the document set with your annotations.







  What to do next 



*  [Train a machine learning model](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-train-ml).
For more information about how to use the entity extractor to add domain customization to your AI applications, see the [Entity Extractor Feature in Watson Discovery v2](https://community.ibm.com/community/user/watsonai/blogs/ananya-poddar/2022/06/09/entity-extraction-in-watson-discovery-v2) blog post.

Discovery also has a built-in Entities enrichment that can be applied directly to your collection. It doesn't require any training to recognize commonly-known proper nouns. For more information about the Watson NLP Entities enrichment, see [Entities](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-nlu).

You already built an entity type system in Knowledge Studio? You can use the corpus that is associated with your machine learning model as a starting point for your entity extractor training data. For more information, see [Importing a corpus](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-entity-extractorentity-extractor-import-wks).

For information about the languages with which the entity extractor can be used, see [Language support](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-language-support).



 Entity extractor overview video 

This video provides an overview of how to define custom entity types and then use them to extract terms of interest from your data.

To read a transcript of the video, [open the video on YouTube.com](https://www.youtube.com/watch?v=nqO_2mL0U7A0), click the More actions icon, and then choose Open transcript.





 Example 

If you are familiar with the built-in Entities enrichment, you know that the enrichment can recognize terms that match generalized categories, such as Person and Location. With the entity extractor, you control what constitutes terms or phrases that are meaningful.

The following image shows the terms that an enrichment that recognizes family members entity type mentions might extract from text. The example illustrates how family member mentions and other entity mentions (that are recognized by the built-in Entities enrichment) both might be predicted.
To take a snapshot of the current version, click Machine Learning Model > Versions, and then click Take Snapshot. The resources in version 1.0 are frozen, and a new version, labeled 1.1, becomes the current version. For each new version that you create, the minor version number is incremented, for example, 1.0 becomes 1.1 and then becomes 1.2.
4. Revise the workspace resources as needed, re-train, and re-evaluate the model.
5. If you are pleased with the performance results and want to store the new version before making future changes, create another version. Continue revising resources and re-training the model as needed, creating a new version for each iteration that you want to retain.
6. If performance results are worse, and you want to revert to a previous version before testing any further:



1. Open the Assets > Dictionaries page and download any dictionaries that you want to re-use in the restored model.
2. Click Machine Learning Model > Versions and click Promote for the version that you want to restore. The version that you promote becomes the current version, and the version number changes to 2.0. When you promote a version, the major version number is incremented and the minor version number becomes 0, for example, 1.1 becomes 2.0.
3. Open the Dictionaries page and upload the dictionaries that you downloaded.
4. If testing of the new version requires changes to ground truth, open the Machine Learning Model > Annotations page. Click the Annotation Tasks tab and create a new annotation task.











 Modifying a type system without losing human annotations 

You might need to make modifications while you train a model, based on the performance statistics. But, generally, you want the type system to be as close to final as possible before you begin large-scale annotation tasks. If you change the type system after human annotators began their work, they must revisit the documents that they annotated. They must assess the applicability of the type system changes.



 About this task
Apply a pretrained SDU model 

Apply a prebuilt Smart Document Understanding (SDU) model that can extract text and is trained to identify tables, lists, and sections in documents.

Use the pretrained model if your documents contain tables with valuable information that you want to capture. The model is also able to preserve the meaning inherent in the nesting structure of tables, lists, and sections. Using the pretrained model speeds up the process of capturing information from the structure of a document.

If you want to customize how the document structure is used to infer meaning from a document or you want to split documents with a field that is generated by an SDU model, create a user-trained model instead. For more information, see [Define a user-trained SDU model](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-configuring-fields).

A pretrained model is applied to Document Retrieval for Contracts projects automatically. Instead of you annotating contract-related content in your documents, the project applies a model that already knows how to recognize terms and concepts that are significant to contracts.



 Preparing documents 

You can apply a pretrained SDU model to the following file types only:



* Image files (PNG, TIFF, JPG)
* Microsoft PowerPoint
* Microsoft Word
* PDF



For a complete list of file types that Discovery supports, see [Supported file types](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionssupportedfiletypes).

The Smart Document Understanding tool uses optical character recognition (OCR) to extract text from images in the files that it analyzes. Images must meet the minimum quality requirements that are supported by OCR. For more information, see [Optical character recognition](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionsocr).

The tool cannot read documents with the following characteristics; remove them from your collection before you begin:



* Documents that appear to have text that overlays other text are considered double overlaid and cannot be annotated.
Using the machine learning model 

Leverage a machine learning model that you trained with Knowledge Studio by making it available to other Watson applications.

You can deploy or export a machine learning model. A dictionary or Natural Language Understanding pre-annotator can only be used to pre-annotate documents within Knowledge Studio.

Before you can deploy a model for use by a service, you must have a subscription to the service. IBM Watson services are hosted on IBM Cloud®, which is the cloud platform for IBM. For more information about the platform, see [What is IBM Cloud?](https://cloud.ibm.com/docs/overview). To subscribe to one of the IBM Watson services, create an account from the [IBM Cloud](https://cloud.ibm.com/) website.

For some of the services, you must know details about the service instance that you plan to deploy to, such as the IBM Cloud space name and service instance name. The space and instance name information is available from the IBM Cloud Services page.

You can also pre-annotate new documents with the machine learning model. See [Pre-annotating documents with the machine learning model](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-preannotationwks_preannotsire) for details.



 Deploying a machine learning model to IBM Watson Discovery 

When you are satisfied with the performance of the model, you can deploy a version of it to IBM Watson Discovery. This feature enables your applications to use the deployed machine learning model to enrich the insights that you get from your data to include the recognition of concepts and relations that are relevant to your domain.



 About this task 

When you deploy the machine learning model, you select the version of it that you want to deploy.





 Procedure 

To deploy a machine learning model, complete the following steps:



1. Log in as a Knowledge Studio administrator or project manager, and select your workspace.
2. Select Machine Learning Model > Versions.
3. Choose the version of the model that you want to deploy.

If there is only one working version of the model, create a snapshot of the current model.