vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Also, if target terms are based on a pattern, you can design a sequence pattern.

You can use the workspace to build extractors to be used with any language, such as Spanish and French, that is based on tokens defined by white space and punctuation. For example, the results of using the Person extractor with Spanish text might extract names as shown here.

(images/samplefilter.png "Sample filter")]



 Extending the provided extractors for use in a different language 

If your target language is based on tokens defined by white space and punctuation, you can extend a provided extractor that has customization points or dictionaries by adding terms to the appropriate dictionaries in the target language.



 Provided extractor Included in folder: How to extend 

 All generic extractors Generic extractors Define a regular expression for the target text pattern. If the provided extractor produces results, create a union with the provided extractor and the regular expression. 
 Email address, Phone number, Zip code Named Entity Recognition Define a regular expression for the target text pattern. If the provided extractor produces results, create a union with the provided extractor and the regular expression. 
 Currency Named Entity Recognition Combine a decimal number extractor with a literal to represent the local currency symbol in a sequence or define a regular expression. If more than one currency symbol is required, use a dictionary rather than a literal. 
 Date and time Named Entity Recognition For numeric formats, define a regular expression for the target text pattern.<br><br>For alphabetic formats, define a sequence using integers for the days and years and a mapping table to represent the months. Use the full names and abbreviations for each month in the mapping table. For example, map the French months juillet and juil to juillet or July, if translating to English. 
 City, County, Country, Person, Location, Organization, State or province, Town Named Entity Recognition European and similar languages<br><br>Add entries for the language to the appropriate dictionary.<br><br>Arabic, Japanese and similar languages 
 Continent Named Entity Recognition Define a dictionary and specify entries to contain the names of the continents in the target language. If English results are also to be included, form a union between the provided continent extractor with your custom one.
After uploading or adding entries, you can edit the entries.

Open an entry to specify equivalent terms, called surface forms. Each surface form must be 256 or fewer characters in length. You can change which of the surface forms is used as the lemma. For example, the lemma IBM might have surface forms like IBM Corp. and International Business Machines, Inc..
5. Select the appropriate part of speech for each lemma and surface form in the dictionary.

The part of speech information is used by the tokenizer, and during pre-annotation.
6. Click Save to store your changes.







 What to do next 

Run the pre-annotator, which uses the dictionaries that you created to do a preliminary pass of the source documents, and adds annotations to them.







 Adding dictionary suggestions 

As you add entries to your dictionary, Knowledge Studio searches the documents in your workspace for similar dictionary entries that might be useful to you.



* You need to upload documents in the Documents tab to get suggestions.
* It may take some time to prepare the suggestion engine after uploading documents.



The suggested entries are provided in the Entry Suggestions pane.

![Entry Suggestions pane](https://cloud.ibm.com/docs-content/v1/content/50eddb8fd81f33092880335ff107a78ff5cd0f65/watson-knowledge-studio/images/entry-suggestions-pane.png)

The following options available to you from the Entry Suggestions pane.



* Clicking the name of a suggested entry opens a window that displays the occurrences of the term in your documents. This can help you decide whether to add or exclude a suggested entry.
* Add this entry (+) adds the entry suggestion to your dictionary.
* Exclude this entry (-) removes the entry from the list of suggested entries and adds it to Excluded Terms.
* Add all adds the entire list of suggested entries to the dictionary (this does not add entries listed under Excluded Terms).
* Exclude all moves all current entry suggestions to Excluded Terms.



Related tasks:
* It impacts which nodes are included in the disambiguation options list

<-- </ul> -->

Sometimes a node is not listed as a disambiguation option as expected. This can happen if a condition value is also referenced by a node that is not eligible for inclusion in the disambiguation list for some reason. For example, an entity mention might trigger a node that is situated earlier in the dialog tree but is not enabled for disambiguation. If the same entity is the only condition for a node that is enabled for disambiguation, but is situated later in the tree, then it might not be added as a disambiguation option because your assistant never reaches it. If it matched against the earlier node and was omitted, your assistant might not process the later node.
<-- </ul> -->For each node that you opt in to disambiguation, test scenarios in which you expect the node to be included in the disambiguation options list. Testing gives you a chance to make adjustments to the node order or other factors that might impact how well disambiguation works at run time.When testing, the order in which the options are listed might change from test run to test run. In fact, the options themselves that are included in the disambiguation list might change from one test run to the next. Don't worry; this is the intended behavior. As part of development that is in progress to help the assistant learn automatically from user choices, the choices included and their order in the disambiguation list is being randomized on purpose. Changing the order helps to avoid bias that can be introduced by a percentage of people who always pick the first option without carefully reviewing all of their choices beforehand.<-- </section "id="section-dialog-runtime-choose-nodes" "> --><-- <section "id="section-dialog-runtime-handle-none" "> --> Handling none of the above When a user clicks the None of the above option, your assistant strips the intents that were recognized in the user input from the message and submits it again.
For example, if the source document contains the sentence Murder on the Orient Express made headlines, then how you define entity types to capture the key information in the sentence differs depending on the type of application that will use the model that you build with the type system.



* For a literary application, you might create types that capture this information:

[NOVEL] [CRITICAL_RECEPTION]

[Murder on the Orient Express] [made headlines]
* For a public safety application, you might create these types:

[EVENT_CRIME] [LOCATION]

[Murder] on the [Orient Express] made headlines



The structure is often driven by characteristics of the documents from which information will be extracted, but should always be tempered by what information the application will actually use from those documents. For example, you might be creating an application that gains insights from medical records. To build the type system, you start looking at patient records to see what kinds of information must be captured. The patient records might all contain a field that describes what the patient ate for lunch. However, if the application will not use that information, do not add it to the type system. And in making the decision to make this omission, recognize that this means that sections of your patient record documents will be left unannotated. That is alright and even expected.

Mentions annotate a span of text; they do not replace the text. A type system is not an ontology for a domain. Expect to have general entity types such as MEDICATION_NAME instead of having an entity type for each medicine type. The document text will continue to contain the specific medicine name. It will just be enhanced with an annotation that identifies its type, which makes the information easier to find and extract programmatically.

As you get started, define 10 to 40 entity types and relation types. Stay to the lower end of the range if the human annotators who will be working on the workspace are not highly specialized in the field. Do not define more than 50.

Plan to spend a good amount of time defining the type system before your team begins any human annotation tasks. When the team does start to annotate documents, begin with a small set, perhaps no more than 50.
[Edit icon](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/oe-intent-edit.png) is used to edit an intent user example; it is not related to adding annotations.
4. A Search box opens that you can use to search for the entity that the highlighted word or phrase is a mention of.

![Search box initial state](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/oe-intent-search1.png)

In this example, searching prod brings up matches for the @product entity.

![Search box with search parameter prod](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/oe-intent-search2.png)

If the entity has any existing entity values, they are displayed for informational purposes only. You are adding the annotation to the entity, not to any specific entity value.

If you want to teach the model that the mention is synonymous with an existing entity value, you can associate it with a specific entity value.

To associate the mention with a specific entity value, follow these steps:



1. Type the full entity name and value into the search field. For example, type @product:IT.
2. When the entity value is displayed in the drop-down menu, select it.



5. Select the entity to which you want to add the annotation.

In this example, computer is being added as an annotation for the @product entity.

Create at least 10 annotations for each contextual entity; more annotations are recommended for production use.
6.
Adding documents for defining rules 

Add documents with linguistic patterns that illustrate the types of rules you want to define.

Documents that you add for defining rules are kept separate from documents that you add for annotation. Their sole purpose is to be used for mining examples of patterns. They are not used for training nor are they ever downloaded.



  Ways you can add documents to the rule editor 



*  Author a text document in UTF-8 format

You can add text that illustrates a pattern directly into the rule editor. The name that you specify for the document cannot be longer than 256 characters.
*  Upload a CSV file

Upload a file in CSV format that contains the document text.
*  Copy a document that was imported for annotation

If a document that is part of a set that was imported for annotation purposes contains examples of patterns that you want to capture as rules, you can copy the document over to the rule editor. Any changes that you make to the document do not impact the original version of the document that is being used for annotation. See [Adding documents to a workspace](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-documents-for-annotationwks_projadd) for information about uploading documents.



For documents that you copy or upload, choose documents that contain distinct patterns that help to identify entity types in documents.
[Restore the models](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-backup-restorerestoremodels)
7. [Restore any incomplete annotation tasks](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-backup-restorerestoretasks)





 Data that can be backed up 

The following artifacts can be backed up and migrated manually:



* Editable dictionaries
* Type system
* Approved ground truth document sets



The following types of artifacts cannot be backed up and migrated manually:



* In-progress human annotation documents
* Annotation tasks
* Models and snapshots
* Read-only dictionaries







 Preparing for backup 

To prepare for backing up and restoring your data, complete the following steps:



1. Complete any work that is in progress in your workspace.



* Finish any in-progress annotation tasks. Only documents that have been annotated, adjudicated, and approved and promoted to ground truth can be backed up. If you do not finish the annotation work, you will lose any annotation effort that is in progress but not completely done.
* If you created annotation tasks to track work that you want to do, but none of the annotation work has begun and will not take place until after the workspace is restored, then make a list of the outstanding annotation tasks. Be sure to note the document sets that you imported but that have not been added to the ground truth yet. Also, make a note of whom you assigned to annotate each document set. Re-upload these document sets and re-create the annotation tasks after the workspace is restored.



2. Understand tokenizer use.

For machine learning models, workspaces use the machine learning-based tokenizer by default. If you are using a dictionary-based tokenizer and have a specific need to continue doing so, you can configure the workspace to use the dictionary-based tokenizer when you restore it.
You can deploy a custom categories model to use instead of the [standard categories](https://cloud.ibm.com/docs/natural-language-understanding?topic=natural-language-understanding-categories-hierarchy) offered by Natural Language Understanding and Discovery.





 Watson services integration 

Share domain artifacts and models between IBM Watson® Knowledge Studio and other Watson services.

Use Knowledge Studio to perform the following tasks:



* Bootstrap annotation by using the Natural Language Understanding service to automatically find and annotate entities in your documents. When human annotators begin to annotate the documents, they can see the annotations that were already made by the service and can review and add to them. See [Pre-annotating documents with Natural Language Understanding](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-preannotationwks_preannotnlu) for details.
* Upload analyzed documents that are in [UIMA CAS XMI format](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-preannotationwks_uimaweximport). For example, you can upload UIMA CAS XMI files that were exported from IBM Watson Explorer content analytics collections or [IBM Watson Explorer Content Analytics Studio](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-preannotationwks_uimawexstudio).
* Deploy a [machine learning](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-publish-mlwks_madiscovery) or [rule-based](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks_rule_publishwks_rule_discovery) model to use with the Watson Discovery service.
You can check which field the analyzable text is extracted from and change it or augment it by changing index type of another field. For more information, see [Identifying the text field](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-edit-collectiontext-field).

Click Next.
9. If you want to apply an enrichment to the text in your training data, select at least one field from the Target fields list where you want to apply enrichments.

Typically, you want to choose the field that contains the body of text that you want to classify. From the earlier example, the Feedback field is the best choice.

Next, select any annotators that you want to apply to enrich the text in the target field or fields, and then click Next.

The Part of speech annotator is selected by default.
10. On the Confirm page, review your classifier configuration settings. To make changes, use the Back button. Otherwise, click Save.

An Overview page is displayed.
11. Click New model to create and train your machine learning model.
12. You can optionally change the name of the model and add a description.

You can change the default ratio values that are specified for the following data sets:



* Training dataset: Updates the weights of the training model.
* Validation set: Monitors the accuracy of the training model during training. The accuracy result is used to draw a training loss graph.
* Test dataset: Calculates the score of the trained model.



13. Click Create.

It might take several minutes for model training to complete.







 Deploying the document classifier model 

After the model is trained, deploy the model as an enrichment.



1. Click the overflow menu icon in the Actions column, and then click Deploy model. Specify the name and other details, and then click Deploy.
2. Do one of the following things:
Adding documents for defining rules 

Add documents with linguistic patterns that illustrate the types of rules you want to define.

Documents that you add for defining rules are kept separate from documents that you add for annotation. Their sole purpose is to be used for mining examples of patterns. They are not used for training nor are they ever downloaded.



  Ways you can add documents to the rule editor 



*  Author a text document in UTF-8 format

You can add text that illustrates a pattern directly into the rule editor. The name that you specify for the document cannot be longer than 256 characters.
*  Upload a CSV file

Upload a file in CSV format that contains the document text.
*  Copy a document that was imported for annotation

If a document that is part of a set that was imported for annotation purposes contains examples of patterns that you want to capture as rules, you can copy the document over to the rule editor. Any changes that you make to the document do not impact the original version of the document that is being used for annotation. See [Adding documents to a workspace](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-documents-for-annotationwks_projadd) for information about uploading documents.



For documents that you copy or upload, choose documents that contain distinct patterns that help to identify entity types in documents.
Images must meet the minimum quality requirements that are supported by OCR. For more information, see [Optical character recognition](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionsocr).

The tool cannot read documents with the following characteristics; remove them from your collection before you begin:



* Documents that appear to have text that overlays other text are considered double overlaid and cannot be annotated.
* Documents that contain multiple columns of text on a single page cannot be annotated.



When you apply a Smart Document Understanding model, the conversion time for your collection can increase due to the resources that are required to apply the AI model to your documents.





 Apply a pretrained model 

To apply a pretrained Smart Document Understanding model to your collection, complete the following steps:



1. Open the Manage collections page from the navigation panel.
2. Select the collection to which you want to apply the model.
3. Open the Identify fields page.
4. Choose Pre-trained models

The Text extraction only option is used by default. With this model, any text that is recognized in the source documents is indexed in the text field.
5. Click Submit, and then click Apply changes and reprocess.







 Understanding the output 

If the SDU model finds and processes a structure, such as a table, in the document, it stores a representation of the structure in a field named enriched_{field}, where {field} is the field where the structure was stored.

The following excerpt shows the JSON representation of a table from the enriched_html field of a document that was processed by the pretrained SDU model.

Zoom

![Shows a JSON snippet that contains an enriched_html field with a table object that contains sections such as section_title, row_headers, table_headers, location, and so on.]
Added Performance page
: Added a Performance page for model quality evaluation and guidance about how to improve quality.







 November 2017 



 Changes 

Bug fixes
: Fixed an issue where some relation annotations were missing in the downloaded corpus.
: Fixed an issue where a model could not be withdrawn from deployment if its status was None.
: Fixed an issue where the model could not be evaluated for Korean.







 October 2017 



 Changes 

Export issue fixed
: Fixed the issue with the Export button not being enabled until you refreshed the browser window in the IBM Cloud [experimental](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks-faqsexperimental) release.

Button label issue fixed
: Fixed the button labels and tooltips to match the changes for the terms upload and download in the IBM Cloud [experimental](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks-faqsexperimental) release. These terms are used instead of import and export when referring to type systems, documents, and dictionaries.

Description delay issue fixed
: Fixed the delay in updating the descriptions on the Knowledge Studio User Account Management page in the IBM Cloud [experimental](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks-faqsexperimental) release.

UI changes to pre-annotation interface
: In the pre-annotation section of the interface, made a couple GUI changes to clarify the functionality of the machine learning model, the rule-based model, the dictionary, and IBM® AlchemyLanguage. Changed the button label from Run to Pre-annotate, changed the title of the window from Run Annotator to Run Pre-annotation, and changed the error message to clarify that you can't add automated annotations after humans annotated the documents.

Dictionary-based tokenizer fix
: For projects or workspaces that use dictionary-based tokenizers, fixed an issue that showed empty sentences if you imported documents without ground truth.







 September 2017