vania-janet/multiturn-rag-retrieval-data huggingface dataset:
[Shows the Annotate entities toggle](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/oe-annotate.png)
4. Click any word, words, or punctuation that is part of a single entity mention from the intent examples.

In this example, mobile phones is the entity mention.

![Review intent examples](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/oe-click-tokens.png)

A Search box opens that you can use to search for the entity that the highlighted word or phrase is a mention of.

![Search box with search parameter prod](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/oe-tokens-clicked.png)
5. Enter the entity name to search for. You do not need to include the starting @ symbol.

Do one of the following things:



* If the entity has any existing entity values, they are displayed for informational purposes only. You are adding the annotation to the entity, not to any specific entity value.
* If you want to teach the model that the mention is synonymous with an existing entity value, add a colon (:) after the entity name to show a list of entity values. Choose an entity value from the list that is displayed. For example, @product:device.

!
Consider using contextual entities as a way to capture these types of proper nouns. Instead of trying to add a dictionary-based entity that covers every permutation of the names for people or cities, for example, you can teach your skill to recognize the context in which such names are used. For more information about contextual entities, see [Annotation-based method](https://cloud.ibm.com/docs/assistant?topic=assistant-entitiesentities-annotations-overview).

Action: Remove references to @sys-person and @sys-location from your dialogs. Turn off the @sys-person and @sys-location system entities to prevent yourself or others from adding them to a dialog inadvertently.
* Irrelevance detection

We revised the irrelevance detection classification algorithm to make it even smarter out of the box. Now, even before you begin to teach the system about irrelevant requests, it is able to recognize user input that your skill is not designed to address. For more information, see [Irrelevance detection](https://cloud.ibm.com/docs/assistant?topic=assistant-irrelevance-detection).

Action: In each dialog skill, go to the Options>Irrelevance detection page and turn on the new classification model. Make sure everything works as well, if not better, than it did before. Share your feedback.
* Old API version dates

v1 API versions that are dated on or before 2017-02-03 are being deprecated. When you send calls to the service with earlier API version dates, they will receive properly formatted and valid responses for a time, so you can gracefully transition to using the later API versions. However, the confidence scores and other results that are sent in the response will reflect those generated by a more recent version of the API.

Action: Do some testing of calls with the latest version to verify that things work as expected. Some functionality has changed over the last few years. After testing, change the version date on any API calls that you make from your applications.







 26 February 2020 

Slot Save it as field retains your edits
"location": {
"begin": 28,
"end": 36
}
}
],
"relevance": 0.596757
},
{
"text": "parents",
"mentions":
{
"text": "parents",
"location": {
"begin": 52,
"end": 59
}
}
],
"relevance": 0.568336
},
{
"text": "mother",
"mentions":
{
"text": "mother",
"location": {
"begin": 66,
"end": 72
}
}
],
"relevance": 0.755562
},
{
"text": "Mr. Collins",
"mentions":
{
"text": "Mr. Collins",
"location": {
"begin": 118,
"end": 129
}
}
],
"relevance": 0.945891
}
]
}
],
"url": "https://www.gutenberg.org/files/1342/1342-h/1342-h.htmlink2HCH0020",
"Subject": "Parental love",
"Year": "1813/01/01",
"Book": "Pride and Prejudice",
"Author": "Jane Austen",
"Quote": [
"From this day you must be a stranger to one of your parents. Your mother will never see you again if you do not marry Mr. Collins, and I will never see you again if you do."
],
"metadata": {
"name": "favorites2.json"
},
"Speaker": "Mr. Bennett"
},
"notices": []
}
Show more
* Both training and test data (for evaluation purposes) should reflect the distribution of intents in real usage. Generally, more frequent intents have relatively more examples, and better response coverage.
* You can include punctuation in the example text, as long as it appears naturally. If you believe that some users express their intents with examples that include punctuation, and some users will not, include both versions. Generally, the more coverage for various patterns, the better the response.







 How entity references are treated 

When you include an entity mention in a user example, the machine learning model uses the information in different ways in these scenarios:



* [Referencing entity values and synonyms in intent examples](https://cloud.ibm.com/docs/assistant?topic=assistant-intentsintents-related-entities)
* [Annotated mentions](https://cloud.ibm.com/docs/assistant?topic=assistant-intentsintents-annotated-mentions)
* [Directly referencing an entity name in an intent example](https://cloud.ibm.com/docs/assistant?topic=assistant-intentsintents-entity-as-example)





 Referencing entity values and synonyms in intent examples 

If you have defined, or plan to define, entities that are related to this intent, mention the entity values or synonyms in some of the examples. Doing so helps to establish a relationship between the intent and entities. It is a weak relationship, but it does inform the model.





 Annotated mentions 

As you define entities, you can annotate mentions of the entity directly from your existing intent user examples. A relationship that you identify in this way between the intent and the entity is not used by the intent classification model. However, when you add the mention to the entity, it is also added to that entity as new value. And when you add the mention to an existing entity value, it is also added to that entity value as new synonym. Intent classification does use these types of dictionary references in intent user examples to establish a weak reference between an intent and an entity.
* Relevance ranking is not supported.





 Dutch 



 Feature Standard Support [Custom model](https://cloud.ibm.com/docs/natural-language-understanding?topic=natural-language-understanding-customizing) support 

 Categories X 
 Classifications 
 Concepts 
 Emotion 
 Entities X* X 
 Keywords X 
 Metadata 
 Relations X 
 Semantic roles 
 Sentiment X 
 Syntax X 



* Relevance ranking is not supported.





 English 



 Feature Standard Support [Custom model](https://cloud.ibm.com/docs/natural-language-understanding?topic=natural-language-understanding-customizing) support 

 Categories X X (Beta) 
 Classifications X* X 
 Concepts X 
 Emotion X 
 Entities X X 
 Keywords X 
 Metadata X 
 Relations X X 
 Semantic roles X 
 Sentiment X 
 Syntax X 



* Indicates support for tone analysis; see [Tone analytics (Classifications)](https://cloud.ibm.com/docs/natural-language-understanding?topic=natural-language-understanding-tone_analytics) for more information.





 Finnish 



 Feature Standard Support [Custom model](https://cloud.ibm.com/docs/natural-language-understanding?topic=natural-language-understanding-customizing) support 

 Categories 
 Classifications 
 Concepts 
 Emotion 
 Entities X* 
 Keywords X 
 Metadata 
 Relations 
 Semantic roles 
 Sentiment X 
 Syntax X 



* Relevance ranking is not supported.





 French 



 Feature Standard Support [Custom model](https://cloud.ibm.com/docs/natural-language-understanding?topic=natural-language-understanding-customizing) support 

 Categories X 
 Classifications X* X 
 Concepts X 
 Emotion X 
 Entities X X 
 Keywords X 
 Metadata X 
 Relations X X 
 Semantic roles 
 Sentiment X 
 Syntax X 



* Indicates support for tone analysis; see [Tone analytics (Classifications)](https://cloud.ibm.com/docs/natural-language-understanding?topic=natural-language-understanding-tone_analytics) for more information.





 German
To return to the main Build queries screen at any time, click Build queries on the upper left. To return to the Manage data screen, click the name of the collection on the upper right.

If you would like to delete all of the training data in your collection at one time, you must do so via the API. See [Delete all training data for a collection](https://cloud.ibm.com/apidocs/discoverydelete-all-training-data) for more information. For more information about training via the API, see [Improving the relevance of your query results with the API](https://cloud.ibm.com/docs/discovery?topic=discovery-improving-result-relevance-with-the-api).





 Testing and iterating on the relevancy of results 

After you complete rating results and Watson applies the training, test to see if your query results improved. To do so, run test queries that are related, but not identical to, your training queries. Check to see if the results of your test queries improved.

If you would like to further improve results after testing, you could:



* Add more documents to your collection.
* Add more training queries.
* Rate more results, making sure to use both the Relevant and Not relevant ratings.



For additional training guidance, see [Relevancy training tips](https://cloud.ibm.com/docs/discovery?topic=discovery-relevancy-tips).





 Confidence scores 

Discovery returns a confidence score for both natural language queries and those written in the Discovery Query Language.

The confidence score is returned for both trained and untrained private collections (with the exception of filter-only queries of untrained collections). In addition, Discovery returns a document_retrieval_strategy field that indicates the source of the confidence score:



* untrained
* relevancy_training, or
* continuous_relevancy_training



The document_retrieval_strategy can be used along with the confidence score to determine how the results provided should be used. In cases where load is high, the document_retrieval_strategy returned might be untrained, even if the collection is trained.
" digital and <em>hybrid</em> <em>cloud</em> transformation.</p>n<p>URL: http://www.ibm.com/press/us/en/pressrelease/50837.wss</p>nnnn</body></html>",
" continuity software for nenterprise data centers and <em>cloud</em> infrastructure. Adding these ncapabilities"
] ! !
}
}
Show more
<-- </section "id="section-highlight" "> --><-- <section "id="section-passages" "> --> passages A Boolean that specifies whether the service returns a set of the most relevant passages from the documents that are returned by a query that uses the natural_language_query parameter. The passages are generated by sophisticated Watson algorithms that determine the best passages of text from all of the documents returned by the query. The default value for the parameter differs based on your project type. For more information about default values, see Default query settings]] .Discovery attempts to return passages that start at the beginning of a sentence and stop at the end by using sentence boundary detection. To do so, it first searches for passages approximately the length specified in the passages.characters parameter]passages.characters_ parameter] (for most project types, the default is 200). It then expands each passage to the limit of twice the specified length so as to return full sentences. If your passages.characters parameter is short or the sentences in your documents are long there might be no sentence boundaries close enough to return the full sentence without going over twice the requested length. In that case, Discovery stays within the limit of twice the passages.characters parameter, so the passages that are returned might not include the entire sentence and can omit the beginning, end, or both.Since sentence boundary adjustments expand passage size, the average passage length can increase. If your application has limited screen space, you might want to set a smaller value for passages.characters or truncate the passages that are returned by Discovery. Sentence boundary detection works for all supported languages and uses language-specific logic.Passages are grouped with each document result and are ordered by passage relevance.
[passages.characters](https://cloud.ibm.com/docs/discovery?topic=discovery-query-parameterspassages_characters) Length of passages passages=true&passages.characters=144 
 [highlight](https://cloud.ibm.com/docs/discovery?topic=discovery-query-parametershighlight) Highlight query matches highlight=true 
 [deduplicate](https://cloud.ibm.com/docs/discovery?topic=discovery-query-parametersdeduplicate) Deduplicate IBM Watson™ Discovery News returned results deduplicate=true 
 [deduplicate.field](https://cloud.ibm.com/docs/discovery?topic=discovery-query-parametersdeduplicate_field) Deduplicate returned results based on field deduplicate.field=title 
 [collection_ids](https://cloud.ibm.com/docs/discovery?topic=discovery-query-parameterscollection_ids) Query multiple collections collection_ids={1},{2},{3} 
 [similar](https://cloud.ibm.com/docs/discovery?topic=discovery-query-parameterssimilar) Enables the similar document finding similar=true 
 [similar.document_ids](https://cloud.ibm.com/docs/discovery?topic=discovery-query-parameterssimilar_document_ids) Which documents to find similar documents to similar.document_ids={id1},{id2} 
 [similar.fields](https://cloud.ibm.com/docs/discovery?topic=discovery-query-parameterssimilar_fields) Which fields to compare when finding similar documents similar.fields=text,title 





 Query limitations 

You cannot query on field names that contain the following:



* Numerical characters (0 - 9) in the suffix of the field name (for example extracted-content2).
* The characters _, +, and - in the prefix of the field_name (for example +extracted-content).
* The characters ., ,, and : in the field_name (for example new:extracted-content)









 Operators
When you specify both parameters, each with different values, they are treated separately.For example, the request might include the parameters "return": "docno"] and "passages":{"fields": "body"] ! ! ! ! ! ! ! ! ! ! ! . The body field is specified in passages.fields, but not in return. In the result, passages from the document body are returned, but the contents of the body field itself is not returned.<-- </section "id="section-passages_fields" "> --><-- <section "id="section-passages_count" "> --> passages.count The maximum number of passages to return. The search returns fewer passages if the specified count is the total number found. The default value is 10. The maximum value is 100.<-- </section "id="section-passages_count" "> --><-- <section "id="section-passages_characters" "> --> passages.characters The approximate number of characters that any one passage can have. The default value is 200. The minimum is 50. The maximum is 2,000. Passages that are returned can contain up to twice the requested length (if necessary) to get them to begin and end at sentence boundaries.<-- </section "id="section-passages_characters" "> --><-- <section "id="section-passages_max-per-doc" "> --> passages.max_per_document One passage is returned per document by default. You can increase the maximum number of passages to return per document by specifying a higher number in the passages.max_per_document parameter.<-- </section "id="section-passages_max-per-doc" "> --><-- </section "id="section-passages" "> --><-- <section "id="section-similar" "> --> similar Finds documents that are similar to documents that you identify as being of interest to you.
In the tool, the name of an entity is always prefixed with the @ character.

You can train the skill to recognize your entities by providing entity term values and synonyms, entity patterns, or by identifying the context in which an entity is typically used in a sentence. To fine tune your dialog, go back and add nodes that check for entity mentions in user input in addition to intents.



![Diagram of a more complex implementation that uses intent, entity, and dialog.](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/complex-impl.png)

As you add information, the skill uses this unique data to build a machine learning model that can recognize these and similar user inputs. Each time you add or change the training data, the training process is triggered to ensure that the underlying model stays up-to-date as your customer needs and the topics they want to discuss change.

For help creating a dialog skill, see [Creating a dialog skill](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-skill-dialog-add).





 Search skill 

When Watson Assistant doesn't have an explicit solution to a problem, it routes the user question to a search skill to find an answer from across your disparate sources of self-service content. The search skill interacts with the IBM Watson® Discovery service to extract this information from a configured data collection.

If you already have Discovery for IBM Cloud Pak for Data installed and an instance provisioned, you can mine your existing data collections for source material that you can share with customers to address their questions.

The following diagram illustrates how user input is processed when both a dialog skill and a search skill are added to an assistant. Any questions that the dialog is not designed to answer are sent to the search skill, which finds a relevant response from a Discovery data collection.

![Diagram of how a question is sent to the Discovery service from a search skill.]
1,000 100,000 100,000 150 contextual entities with 3000 annotations 



System entities that you enable for use count toward your totals.





 Editing entities 

You can click any entity in the list to open it for editing. You can rename or delete entities, and you can add, edit, or delete values, synonyms, or patterns.

If you change the entity type from synonym to pattern, or vice versa, the existing values are converted, but might not be useful as-is.





 Searching entities 

The search capability was introduced with the 1.5.0 release.

Use the Search feature to find entity names, values, and synonyms.



1. From the Entities page header, click the Search icon ![Search icon in the Intents page header](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/search_icon.png).

System entities are not searchable.
2. Enter a search term or phrase.

The first time you search for something, you might get a message that says the skill is being indexed. If so, wait a minute, and then resubmit the search term.

![Entity search term](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/entities-search-index.png)



Entities containing your search term, with corresponding examples, are shown.

![Entity search return](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/entities-search-results.png)





 Exporting entities 

You can export a number of entities to a CSV file, so you can then import and reuse them for another Watson Assistant application.
This documentation is for the classic Watson Assistant experience. To see the documentation for the new Watson Assistant, please go [here](https://cloud.ibm.com/docs/watson-assistant).



 Use a search skill to embed existing help content ![Plus or higher plan only](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/plus.png) 

Put your subject matter expertise to work by adding a search skill. The search skill gives your assistant access to corporate data collections that it can mine for answers.

When a search skill is added, your assistant can route complex customer inquiries to the IBM Watson® Discovery service. Discovery treats the user input as a search query. It finds information that is relevant to the query from an external data source and returns it to the assistant.

This feature is available only to paid plan users.

Add a search skill to your assistant to prevent the assistant from having to say things like, I'm sorry. I can't help you with that. Instead, the assistant can query existing company documents or data to see whether any useful information can be found and shared with the customer.

![Shows a search result in the Preview](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/search-preview-example.png)

To show the exact answer highlighted in bold font, enable the Emphasize the answer feature that is available with Discovery v2 instances.

Watch a 4-minute video that provides an overview of the search skill:

To learn more about how search skill can benefit your business, [read this blog post](https://medium.com/ibm-watson/adding-search-to-watson-assistant-99e4e81839e5).
The user examples should be tailored to your specific business. For example, if you are an insurance company, a user example might look more like this, I want to buy a new XYZ insurance plan.

The examples that you provide are used by your assistant to build a machine learning model that can recognize the same and similar types of utterances and map them to the appropriate intent.



Start with a few intents, and test them as you iteratively expand the scope of the application.

![Plus or higher plans only](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/plus.png) If you already have chat transcripts from a call center or customer inquiries that you collected from an online application, put that data to work for you. Share the real customer utterances with Watson and let Watson recommend the best intents and intent user examples for your needs. See [Get help defining intents](https://cloud.ibm.com/docs/assistant?topic=assistant-intent-recommendations) for more details.





 Creating intents 



1. Open your dialog skill. The skill opens to the Intents page.
2. Select Create intent.
3. In the Intent name field, type a name for the intent.



* The intent name can contain letters (in Unicode), numbers, underscores, hyphens, and periods.
* The name cannot consist of .. or any other string of only periods.
* Intent names cannot contain spaces and must not exceed 128 characters. The following are examples of intent names:



* weather_conditions
* pay_bill
* escalate_to_agent





A number sign is prepended to the intent name automatically to help identify the term as an intent. You do not need to add it.

Keep the name as short as possible. It's easier to read in the "Try it out" pane and conversation logs if you keep the intent name short and concise.
To improve response time, you can create an index that excludes documents with status: { $ne: archived } at index time by using the partial_filter_selector field that is shown in the following example:

POST /db/_index HTTP/1.1
Content-Type: application/json
Content-Length: 144
Host: localhost:5984

{
"index": {
"partial_filter_selector": {
"status": {
"$ne": "archived"
}
},
"fields": ["type"]
},
"ddoc" : "type-not-archived",
"type" : "json"
}
Show more

Partial indexes aren't currently used by the query planner unless specified by a use_index field, so you must modify the original query:

{
"selector": {
"status": {
"$ne": "archived"
},
"type": "user"
},
"use_index": "type-not-archived"
}

Technically, you don't need to include the filter on the status field in the query selector. The partial index ensures that this value is always true. However, if you include the filter, it makes the intent of the selector clearer. It also makes it easier to take advantage of future improvements to query planning (for example, automatic selection of partial indexes).