vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Define a user-trained SDU model 

Create a Smart Document Understanding (SDU) model that learns about the content of a document based on the document's structure.

Use the Smart Document Understanding tool to add custom fields to a collection so you can do the following things:



* Target prebuilt or custom enrichments at specific sections of a document.
* Break large documents into smaller documents.



For help with deciding whether SDU can help your use case, read [When to use Smart Document Understanding](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-configuring-fieldssdu-when).

If capturing information from tables is critical to your use case, consider using a pretrained model. For more information about creating a pretrained SDU model, see [Apply a pretrained SDU model](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-sdu-pretrained).



 When to use Smart Document Understanding 

The Smart Document Understanding (SDU) tool works better with some project types.



* The tool is most beneficial when used with Document Retrieval projects. Use the tool to break your documents into smaller, more consumable chunks of information. When you help Discovery index the correct set of information in your documents, you improve the answers that your application can find and return.

For example, your documents might contain tips that are shown in sections with an H4 heading. If you want to extract the information from these tips separately, you can add a field that is named tips, and teach the model to recognize it. After you apply the model to your collection, you can apply an enrichment to the tips field only. Later, you can limit the search to return content from only the tips field.

Or maybe you have extra large documents that contain subsections. You can teach the SDU model to recognize these subsections, and then split the large document into multiple, smaller, and easier-to-manage documents that begin with one of these subsections.
* The best way to prepare a collection for use in Conversational Search projects is to identify discrete question-and-answer pairs.
For more information about contextual entities, see [Annotation-based method](https://cloud.ibm.com/docs/assistant?topic=assistant-entitiesentities-annotations-overview).

For more information about how to use contextual entites to identify names of people, see the [Detecting Names And Locations With Watson Assistant](https://medium.com/ibm-watson/detecting-names-and-locations-with-watson-assistant-e3e1fa2a8427) blog post on Medium.

How legacy numeric system entities are processed has changed
: All new dialog skills use the new system entities automatically.

For existing skills that use legacy numeric system entities, how the entities are processed now differs based on the skill language.



* Arabic, Chinese, Korean, and Japanese dialog skills that use legacy numeric system entities function the same as before.
* If you choose to continue to use the legacy system entities in European-language dialog skills, a new legacy API format is used. The new legacy API format simulates the legacy system entities behavior. In particular, it returns a metadata object and does not stop the service from idenfifying multiple system entities for the same input string. In addition, it returns an interpretation object, which was introduced with the new version of system entities. Review the interpretation object to see the useful information that is returned by the new version.



Update your skills to use the new system entities from the Options>System Entities page.

Web chat security is generally available
: Enable the security feature of web chat so that you can verify that messages sent to your assistant come from only your customers and can pass sensitive information to your assistant.

When configuring the JWT, you no longer need to specify the Authentication Context Class Reference (acr) claim.





 1 July 2020 

Salesforce support is generally available
: Integrate your web chat with Salesforce so your assistant can transfer customers who asks to speak to a person to a Salesforce agent who can answer their questions. For more information, see [Integrating with Salesforce](https://cloud.ibm.com/docs/assistant?topic=assistant-deploy-salesforce).





 24 June 2020 

Get better answers from search skill
* POST /v1/customizations/{customization_id}/upgrade_model
* POST /v1/acoustic_customizations/{customization_id}/upgrade_model



For more information, see [Upgrading custom models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-custom-upgrade).

New support for concurrent requests to update different custom acoustic models
: The service now allows you to submit multiple simultaneous requests to add different audio resources to a custom acoustic model. Previously, the service allowed only one request at a time to add audio to a custom model.

New updated field for methods that list custom models
: The output of the HTTP GET methods that list information about custom language and custom acoustic models now includes an updated field. The field indicates the date and time in Coordinated Universal Time (UTC) at which the custom model was last modified.

Change to schema for warnings associated with custom model training
: The schema changed for a warning that is generated by a custom model training request when the strict parameter is set to false. The names of the fields changed from warning_id and description to code and message, respectively. For more information, see the [API & SDK reference](https://cloud.ibm.com/apidocs/speech-to-text).





 10 June 2019 

Processing metrics not available with synchronous HTTP interface
: Processing metrics are available only with the WebSocket and asynchronous HTTP interfaces. They are not supported with the synchronous HTTP interface. For more information, see [Processing metrics](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-metricsprocessing-metrics).





 17 May 2019 

New processing metrics and audio metrics features for speech recognition
: The service now offers two types of optional metrics with speech recognition requests:
For more information, see [audio/mulaw format](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-audio-formatsaudio-formats-mulaw).

New supported_features identified when listing models
: The GET /v1/models and GET /v1/models/{model_id} methods now return a supported_features field as part of their output for each language model. The additional information describes whether the model supports customization. For more information, see the [API & SDK reference](https://cloud.ibm.com/apidocs/speech-to-text).





 30 June 2016 

Beta asynchronous HTTP interface now supports all available languages
: The beta asynchronous HTTP interface now supports all languages that are supported by the service. The interface was previously available for US English only. For more information, see [The asynchronous HTTP interface](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-async) and the [API & SDK reference](https://cloud.ibm.com/apidocs/speech-to-text).





 23 June 2016 

New beta asynchronous HTTP interface now available
: A beta asynchronous HTTP interface is now available. The interface provides full recognition capabilities for US English transcription via non-blocking HTTP calls. You can register callback URLs and provide user-specified secret strings to achieve authentication and data integrity with digital signatures. For more information, see [The asynchronous HTTP interface](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-async) and the [API & SDK reference](https://cloud.ibm.com/apidocs/speech-to-text).

New beta smart_formatting parameter for speech recognition
: A beta smart formatting feature that converts dates, times, series of digits and numbers, phone numbers, currency values, and Internet addresses into more conventional representations in final transcripts.
A relationship between two words or phrases in which both refer to the same person or thing and one stands as a linguistic antecedent of the other. For example, there is a coreference between the two pronouns in the phrase "She taught herself" but not in the phrase "She taught her". A coreference links two equivalent entities in the same text.
* coreference chain

A list of entities that were annotated as coreferences. When you annotate mentions as coreferences, the system creates a coreference chain. The chain provides a way for you to view all of the mentions in context and verify that all of the occurrences belong together under the same entity type.
* corpus

A collection of source documents that were added to a workspace and used to train a machine learning model.
* curate

To select, collect, preserve, and maintain content relevant to a specific topic. Curation establishes, maintains, and adds value to data; it transforms data into trusted information and knowledge.







 D 



* dictionary

A collection of words that can be used to pre-annotate documents. A new annotation is created for each word in the document text that matches a term in the dictionary. A machine learning model can be configured with one or more independent dictionaries, which are typically domain-specific, such a dictionary for pharmaceuticals and a dictionary for wealth management. See also [lemma](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-glossarygloss_L) and [surface form](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-glossarygloss_S).
* dictionary pre-annotator

A component that identifies mentions in text that match a specific set of words. By using domain-specific terminology to pre-annotate text, dictionary pre-annotators can accelerate a human annotator's ability to prepare a set of ground truth documents.
* document set

A collection of documents. Documents that are imported together become a document set.
From the Skills menu of the dialog skill where you want to enable disabmiguation, click Options.2. On the Disambiguation page, switch the toggle to On.3. In the Disambiguation message field, add text to show before the list of dialog node options. For example, What do you want to do?4. In the Anything else field, add text to display as an additional option that users can pick if none of the other dialog node options reflect what the user wants to do. For example, None of the above.
Keep the message short, so it displays inline with the other options. The message must be less than 512 characters. For information about what your assistant does if a user chooses this option, see Handling none of the above]] !!.
5. If you want to limit the number of disambiguation options that can be displayed to a user, then in the Maximum number of suggestions field, specify a number between 2 and 5.
Your changes are automatically saved.
6. Now, click Dialog from the Skills menu. Review your dialog to decide which dialog nodes you want the assistant to ask for help with.
<-- <ul> -->

* You can pick nodes at any level of the tree hierarchy.
* You can pick nodes that condition on intents, entities, special conditions, context variables, or any combination of these values.

<-- </ul> -->

See Choosing nodes]]!Shows where to add the external node name information in the node edit view.Shows where to add the external node name information in the node edit view_.]!! ! ! for tips.

For each node that you want to make available from the disambiguation options list, complete the following steps:

<-- <ol> -->

1. Click to open the node in edit view.
2. In the external node name field, describe the user task that this dialog node is designed to handle. For example, Cancel an account.

!]
Move through the document and click each mention that means the same thing and is labeled by the same entity type. For example, click each occurrence of IBM, International Business Machines, and IBM Corp., assuming all these mentions have the entity type ORGANIZATION.
2. Double-click the last mention that you want to add to the chain. A coreference chain is created in the side panel. The name of the chain matches the first mention that you selected.
3. To highlight all mentions in a chain to review them in context, hover over the name of the chain in the side pane.



8. The Single Mention List displays terms in the document that have been annotated, but have not been added to a chain. If you notice a mention in the list that belongs in a chain, you can add it to the chain from here.



1. From the Single Mention List in the side panel, click the mention.
2. From the drop-down list below the mention description, choose the number that represents the chain that you want to add the mention to.
3. Click Merge to add the mention to the chain, and then click OK.



The mention is removed from the Single Mention List and the number of the chain that it now belongs to is displayed below the mention in the document.
9. You can undo your work by using the following methods:



* To remove a coreference chain that you just added, press Ctrl+Z to undo the action.
* To remove a coreference chain later, from the Coreference Chains side panel, click the X next to the chain that you want to remove.
* To remove a single mention from the chain, click the coreference ID to open a window that displays a list of the mentions in the chain, and then click the X next to the mention that you want to remove.



10. Click Save at any time to save your work.







 What to do next 

After you finish annotating all entity mentions, relation mentions, and coreferences in the document, as applicable, change the document status from In Progress to Completed, click Save, and then close the document.

After you finish annotating all documents and mark them Completed, the status of the annotation set changes to Submitted.
* To remove a single mention from the chain, click the coreference ID to open a window that displays a list of the mentions in the chain, and then click the X next to the mention that you want to remove.



10. Click Save at any time to save your work.







 What to do next 

After you finish annotating all entity mentions, relation mentions, and coreferences in the document, as applicable, change the document status from In Progress to Completed, click Save, and then close the document.

After you finish annotating all documents and mark them Completed, the status of the annotation set changes to Submitted. That status is how project managers know that they can start to evaluate the documents for inter-annotator agreement, reject them or accept them, and promote them to ground truth.







 Annotating relations 

To annotate relation mentions, a human annotator finds textual evidence of a relation between two entity mentions in a sentence, and then applies a label that most appropriately describes the relation type. The labels that can be applied are relation types defined in the workspace's type system.



 Before you begin 

You must annotate entity mentions in the document before you can define relation types between them.





 About this task 

The relation mention can only be defined if the text explicitly describes the relationship between the two entity mentions. Explicit textual evidence might include possessives, subject-verb-object structures, or appositives. For example, in the following sentence, it is not valid to add the ownedBy relation mention between dog and owner.

NOT VALID: The dog got a treat from its owner.

The valid relation mention is between its and owner, because it is this part of the sentence in which the text explicitly defines the relationship between the dog and its owner. Owner might be a home owner, or the owner of some other dog, but this text makes it clear that the same dog that is mentioned in the beginning of the sentence is owned by this person.

VALID: The dog got a treat from its owner.

|ownedBy^

The requirement that both entity mentions and the text that defines the relation type between them must exist within a single sentence might seem strict.
To deploy, attach each language skill to its own assistant that you can deploy in a way that optimizes its use by your target audience.





 Understanding the universal language model 

A skill that uses the universal language model applies a set of shared linguistic characteristics and rules from multiple languages as a starting point. It then learns from the training data that you add to it.

The universal language classifier can adapt to a single language per skill. It cannot be used to support multiple languages within a single skill. However, you can use the universal language model in one skill to support one language, such as Russian, and in another skill to support another language, such as Hindi. The key is to add enough training examples or intent user examples in your target language to teach the model about the unique syntactic and grammatical rules of the language.

Use the universal language model when you want to create a conversation in a language for which no dedicated language model is available, and which is unique enough that an existing model is insufficient.

For more information about feature support in the universal language model, see [Supported languages](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-language-support).





 Creating a skill that uses the universal language model 

To create a skill that uses the universal language model, complete the following steps:



1. From the Skills page, click Create skill.
2. Name your skill, and optionally add a description. From the Language field, choose Another language.

Remember, if the language you want to support is listed individually, choose it instead of using the universal model. The built-in language models provide optimal language support.
3. Create the skill.
4. If your skill will support a language with right-to-left directional text, such as Hebrew, configure the bidirectional capabilities.

Click the options icon from the skill tile ![Skill options icon](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/kebab.png), and then select Language preferences.
* [Adding variety](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-dialog-overviewdialog-overview-add-variety)







 Adding multiple lines 

If you want a single text response to include multiple lines separated by carriage returns, then follow these steps:



1. Add each line that you want to show to the user as a separate sentence into its own response variation field. For example:



Multiple line response

 Response variations 

 Hi. 
 How are you today? 



2. For the response variation setting, choose multiline.

If you are using a dialog skill that was created before support for rich response types was added to the product, then you might not see the multiline option. Add a second text response type to the current node response. This action changes how the response is represented in the underlying JSON. As a result, the multiline option becomes available. Choose the multiline variation type. Now, you can delete the second text response type that you added to the response.



When the response is shown to the user, both response variations are displayed, one on each line, like this:

Hi.
How are you today?





 Adding variety 

If your users return to your conversation service frequently, they might be bored to hear the same greetings and responses every time. You can add variations to your responses so that your conversation can respond to the same condition in different ways.

In this example, the answer that your assistant provides in response to questions about store locations differs from one interaction to the next:

![Shows a node that shows a user ask, Where are you located, and the dialog has three different responses defined.](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/variety.png)

You can choose to rotate through the response variations sequentially or in random order. By default, responses are rotated sequentially, as if they were chosen from an ordered list.
For example, type @product:IT.
2. When the entity value is displayed in the drop-down menu, select it.



5. Select the entity to which you want to add the annotation.

In this example, computer is being added as an annotation for the @product entity.

Create at least 10 annotations for each contextual entity; more annotations are recommended for production use.
6. If none of the entities are appropriate, you can create a new entity by choosing @(create new entity).
7. Repeat this process for each entity mention that you want to annotate.

Be sure to annotate every mention of an entity type that occurs in any user examples that you edit. See [What you don't annotate matters](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-entitiesentities-counter-examples) for more details.
8. Now, click the annotation you just created. A box opens that says, Go to: <entity-name>. Clicking that link takes you directly to the entity.

![Verify value computer for product entity](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/oe-verify-value.png)

The annotation is added to the entity you associated it with, and the system begins to train itself on the new data.

The term you annotated is added to the entity as a new dictionary value. If you associated the annotated term with an existing entity value, then the term is added as a synonym of that entity value instead of as an independent entity value.
9. To see all of the mentions you annotated for a particular entity, from the entity's configuration page, click the Annotation tab.

!
For both project types, the best way to test is to trigger search from the Watson Assistant preview. When you configure search support for an assistant, you can fine-tune the experience in ways that aren't available in Discovery.

And settings that are available from the Search results tool for a Document Retrieval project type are replaced by configuration settings that you specify in Watson Assistant. For example, the query response title and body are defined in Watson Assistant. And a passage length of 325 characters is applied to responses regardless of what you specify in the Max characters in a passage field.

The way that you deploy search support in your chatbot is the same regardless of the project type. You enable search support in your assistant and then publish your assistant.

If you decide that you want to use a Document Retrieval project type, you must create it before you add the search function to your virtual assistant. Otherwise, when you add search support to your assistant, a Conversational Search project is created for you automatically. When you have a service instance that contains a Document Retrieval project already, the Watson Assistant user interface shows the existing instance, and you can choose to use it.