vania-janet/multiturn-rag-retrieval-data huggingface dataset:
You might want to index certain types of content in different fields or split up large documents so that the text field contains fewer passages per document. To do so, you can teach Discovery to recognize important fields in your documents by applying a Smart Document Understanding model to your collection.

Smart Document Understanding (SDU) is a technology that learns about the content of a document based on the document's structure. You can apply a prebuilt SDU model or create a custom SDU model.

Zoom

![Shows the SDU model options.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/gs-sample-id-fields.png)

Figure 8. Smart Document Understanding model options

To create a custom SDU model, you select the User-trained model option, and then annotate fields in your document. (You will not annotate documents as part of this tutorial.)

Zoom

![Shows the Smart Document Understanding tool.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/gs-sample-sdu.png)

Figure 9. Smart Document Understanding annotation tool

For more information about SDU, see [Using Smart Document Understanding](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-configuring-fields).
10. Click Manage fields.

The Manage fields page lists the indexed fields. From here, you can include or remove fields from the index. You can also split large documents into many smaller documents.

Zoom

![Shows the fields in the index.]
[Screen capture showing how to move or delete an example](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/move_example.png)







 Searching intents 

Use the Search feature to find user examples, intent names, and descriptions.



1. From the Intents page, click the Search icon.

![Search icon in the Intents page header](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/intent-search.png)
2. Submit a search term or phrase.

The first time you search for something, you might get a message that says the skill is being indexed. If so, wait a minute, and then resubmit the search term.

Intents that contain your search term are displayed.

![Shows the results from a search for intents](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/intent-search-results.png)







 Downloading intents 

You can download a number of intents to a CSV file, so you can then upload and reuse them in another Watson Assistant application.



1. Go to the Intents page.



* To download all intents, meaning the intents listed on this and any additional pages, do not select any individual intents. Instead, click the Download all intents icon. ![Download all intents icon](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/download-all.png)
{
"settings": {
"browserInsightsAdaptiveGroups": {
"enabled": true,
"maxDuration": 2592000,
"maxNumberOfFields": 30,
"maxPageSize": 10000,
"notOlderThan": 2595600
}
}
}
]
}
},
"errors": null
}





 Query limits 

The volume of data that a query can return is limited, and there are user limits on daily data volume. The following limits apply in addition to general rate limits enforced by the API:



* A zone-scoped query can include up to 10 zones.
* Queries can request up to 30 fields as indicated by maxNumberOfFields in settings.
* Responses can return up to 10,000 records. This limit is indicated by maxPageSize in settings.



Queries must explicitly specify the upper bounds of records to return using the limit argument.







 Sorting 

You can sort the order of query result elements using the orderBy argument. By default, results are sorted by the primary key of the dataset (table). If you specify another field to sort on, the primary key is included in the sorting key to keep consistent results for pagination.

Ordering within nested structures is not supported.



 Sorting examples 

Raw data sorting:

firewallEventsAdaptive (orderBy: [clientCountryName_ASC]) {
clientCountryName
}

Raw data sorting using multiple fields:

firewallEventsAdaptive (orderBy: [clientCountryName_ASC, datetime_DESC]) {
clientCountryName
datetime
}

Group sorting by aggregation function:

httpRequests1hGroups (orderBy: [sum_bytes_DESC]){
sum {
bytes
requests
}
dimensions {
datetime
}
}







 Pagination 

Pagination, breaking up query results into smaller parts, can be done using limit, orderBy, and filtering parameters.
"field": "STATE",
"count": 3,
"results":
{
"key": "CA",
"matching_results": 1210,
"aggregations":
{
"type": "term",
"field": "CITY",
"count": 2,
"results":
{
"key": "LOS ANGELES",
"matching_results": 77
},
{
"key": "SAN DIEGO",
"matching_results": 66
}
]
}
]
},
{
"key": "NY",
"matching_results": 693,
"aggregations":
{
"type": "term",
"field": "CITY",
"count": 2,
"results":
{
"key": "BROOKLYN",
"matching_results": 35
},
{
"key": "NEW YORK",
"matching_results": 21
}
]
}
]
},
{
"key": "FL",
"matching_results": 511,
"aggregations":
{
"type": "term",
"field": "CITY",
"count": 2,
"results":
{
"key": "JACKSONVILLE",
"matching_results": 33
},
{
"key": "TAMPA",
"matching_results": 29
}
]
}
]
}
]
}
],
"results": []
Show more

The order in which you specify the aggregations matters. For example, if you reverse the order of the term aggregations from the previous example, you get different results.

{
"query":"brake",
If you want to define a pattern for your assistant to look for in user input, such as a product order number or email address, define a pattern value instead. See [Adding entities that recognize patterns](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-entitiesentities-patterns) for more details.

Note: You can add either synonyms or patterns for a single entity value, not both.

Watson can also recommend synonyms for your entity values. The recommender finds related synonyms based on contextual similarity extracted from a vast body of existing information, including large sources of written text, and uses natural language processing techniques to identify words similar to the existing synonyms in your entity value.
3. To see synonym recommendations from Watson, click Recommend synonyms. Otherwise, skip this step.

Synonym recommendations are listed. The terms are displayed in lowercase, but your assistant recognizes mentions of the synonyms whether they are specified in lowercase or uppercase.

The more coherent your entity value synonyms are, the more relevant and better focused your recommendations will be. For example, if you have several words that are focused on a theme, you will get better suggestions than if you have one or two random words.

![Shows synonym recommendations for the word apple](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/entities-recommended-synonyms.png)



1. Select any synonyms you want to include, and then click Add selected.

You must click the Add selected button for any synonyms you selected to be added. If you move to the next set without clicking this button first, your selections will be lost.

The synonyms are added to your entity, and Watson suggests more synonyms.

If you receive no additional synonym recommendations, it could be because your entity is already well defined, or it contains content that the recommender is not currently able to expand upon.
Hitting IOPS limits can cause your databases to respond slowly or appear unresponsive.

Indexing uses disk, so if your use-case is write-heavy, your indexing speed can be limited by the IOPS available to your deployment. Some bottlenecks can be ameliorated by [tuning your indexes for disk usage](https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-disk-usage.html). In addition, searching can use disk if your working data set does not fit in the file system cache, increasing IOPS load. If your use-case involves searching a large data set, increasing the memory on your deployment can help Elasticsearch rely less on disk.

Another thing to note is the default Lucene file system management policy is niofs, which allows concurrent reads on a file, which also can be constrained by disk I/O limits. Information on file system storage types in the [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-store.html).

If you need more IOPS, you can increase the number IOPS available to your deployment by increasing disk space. If you are aware of trends in your indexing or usage that increases disk I/O, you can configure autoscaling to increase disk based on IOPS.
If you use the former collection_id, it continues to work; however, you might want to switch to the new collection_id for new projects.

Updated confidence score calculations : Query results return a score value, which indicates the relative relevancy between query results. Starting 30 November 2017, the way that score is calculated changed. It is recommended that you only use the score value to rank documents in a single search, not across searches or sessions. If you trained a collection, a score value is returned in the results of a natural language query. Because the score indicates the relative relevancy between query results, it is not recommended tht it be used as a threshold. Instead, use the confidence, which indicates the relevance of the result as compared to the trained model, to set thresholds. For more information about setting thresholds, see [Confidence scores](https://cloud.ibm.com/docs/discovery?topic=discovery-improving-result-relevance-with-the-toolingconfidence).

Improved sentence boundaries in Passages : Beginning with this release, Passage retrieval detects sentence boundaries - it attempts to return passages that start at the beginning of a sentence and stop at the end. Previously, many passages would begin or end somewhere mid-sentence. See [Passages](https://cloud.ibm.com/docs/discovery?topic=discovery-query-parameterspassages) for more information about Passage retrieval.





 15 November 2017 

New Relation Extraction enrichment : Added the [Relation Extraction](https://cloud.ibm.com/docs/discovery?topic=discovery-configservicerelation-extraction) enrichment, which includes the option to incorporate a custom relation model created with IBM Watson™ Knowledge Studio.

New custom entity model in Entity Extraction : The [Entity Extraction](https://cloud.ibm.com/docs/discovery?topic=discovery-configserviceentity-extraction) enrichment Discovery tooling now includes the option to incorporate a custom entity model created with IBM Watson™ Knowledge Studio.

Change to Japanese-language collections : The option to create a Japanese collection was removed from the Discovery tooling, however, the option to create a Japanese collection using the Discovery API remains.

New support for syndicated environments : Discovery Tooling now supports syndicated environments.
Instead of showing a standard message, such as I don't know how to help you with that. the assistant can say, Maybe this information can help: followed by the passage returned by the search. If a search skill is linked to your assistant, then whenever the anything_else node is triggered, rather than displaying the node response, you can set it up to trigger a search instead. The assistant passes the user input as the query to your search skill, and returns the search results as the response.

To see how this works at run time (where v2 API is used) from the Try it out pane (where v1 API is used), you must add a search skill response type to the Anything else node in your dialog. Doing so mimics the behavior that occurs at run time. Only keep this response type on the Anything else node for the duration of your dialog testing, and then remove it when you're done. The best way to see how the assistant will behave from end-to-end is to test by using the API. See [Test the search skill](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-skill-search-addsearch-skill-add-test-via-api).

You can prevent the search from being triggered from the anything_else node by following the steps in [Disabling search](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-skill-search-addsearch-skill-add-disable).
* Search response type: If you add a search skill response type to a dialog node, then your assistant retrieves a passage from an external data source and returns it as the response to a particular question. This type of search occurs only when the individual dialog node is processed.

This approach is useful if you want to narrow down a user query before you trigger a search. For example, the dialog branch might collect information about the type of device the customer wants to buy. When you know the make and model, you can then send a model keyword in the query that is submitted to the search skill, and get better results.
* To find and resubmit a test utterance, you can press the Up key to cycle through your recent inputs.
* To remove prior test utterances from the chat pane and start over, click the Clear link. Not only are the test utterances and responses removed, but this action also clears the values of any context variables that were set as a result of your interactions with the dialog. Context variable values that you explicitly set or change are not cleared.







 What to do next 

If you determine that the wrong intents or entities are being recognized, you might need to modify your intent or entity definitions.

If the correct intents and entities are being recognized, but the wrong nodes are being triggered in your dialog, make sure your conditions are written properly.

If you are ready to put the conversation to work helping your users, call the assistant from a client application. See [Building a client application](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-api-client).







 Searching your dialog 

The search capability was introduced with the 1.5.0 release.

You can search the dialog to find one or more dialog nodes that mention a given word or phrase.



1. From the Dialog page header, click the Search icon ![Search icon in the Intents page header](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/search_icon.png).
2. Enter a search term or phrase.

The first time you search, an index is created. You might be asked to wait for the text in your dialog nodes to be indexed and then resubmit your request.



Dialog nodes that contain your search term, with corresponding examples, are shown. Select a result to open it for editing.

!
2. List the contents and sort them by date.

ls -lrth
3. Find the name of the most recent BUAgent-x.XLOG and open it with /opt/BUAgent/xlogcat. It can't be opened with cat or vim.

./xlogcat BUAgent-1.XLOG
4. Review the log and determine the issue.



If your service plan covers it, you can get help by creating a [support case](https://cloud.ibm.com/unifiedsupport/supportcenter).
* Wait for user input (Default): Pauses the conversation and your assistant waits for the user to respond. In the simplest case, the text you specify here can more explicitly state the type of information you need the user to provide. If you use this action with a conditional response, be sure to word the conditional response such that you clearly state what was wrong with the user's answer and what you expect them to provide instead.
* Prompt again: After displaying the Not found response, your assistant repeats the slot prompt again and waits for the user to respond. If you use this action with a conditional response, the response can merely explain what was wrong about the answer the user provided. It does not need to reiterate the type of information you want the user to provide because the slot prompt typically explains that.

If you choose this option, consider adding at least one variation of the Not found response so that the user does not see the exact same text more than once. Take the opportunity to use different wording to explain to the user what information you need them to provide and in what format.
* Skip this slot: Instructs your assistant to stop trying to fill the current slot, and instead, move on to the prompt for the next empty slot. This option is useful in a slot where you want to both make the slot optional and to display a prompt that asks the user for information. For example, you might have a @seating entity that captures restaurant seating preferences, such as outside, near the fireplace, private, and so on. You can add a slot that prompts the user with, Do you have any seating preferences? and checks for @seating.values. If a valid response is provided, it saves the preference information to $seating_preferences. However, by choosing this action as the Not found response next step, you instruct your assistant to stop trying to fill this slot if the user does not provide a valid value for it.
* Skip to response: If, when the condition you define is met, you no longer need to fill any of the remaining slots in this node, choose this action to skip the remaining slots and go directly to the node-level response next. For example, if after capturing the one-way flight information, the slot prompt is, Are you buying round trip tickets?
"key": "TAMPA",
"matching_results": 29
}
]
}
]
}
]
}
],
"results": []
Show more

The order in which you specify the aggregations matters. For example, if you reverse the order of the term aggregations from the previous example, you get different results.

{
"query":"brake",
"aggregation": "term(field:CITY,count:3).term(field:STATE,count:1)"
}

The new order produces results that surface Chicago, a city that wasn't included in the previous set of results. When the request starts by grouping by state, Illinois, which has only one city with a high number of traffic incident reports, is not included in the results. New York and Florida, which both have more than one city with many incident reports, produce a higher number of statewide matches and therefore, were returned. When you group by city first, the results change.

{
"matching_results": 9064,
"retrieval_details": {
"document_retrieval_strategy": "untrained"
},
"aggregations": [
{
"type": "term",
"field": "CITY",
"count": 4,
"results":
{
"key": "LOS ANGELES",
"matching_results": 77,
"aggregations":
{
"type": "term",
"field": "STATE",
"count": 1,
"results":
{
"key": "CA",
"matching_results": 77
}
]
}
]
},
{
"key": "SAN DIEGO",
"matching_results": 66,
"aggregations":
{
"type": "term",
"field": "STATE",
"count": 1,
"results":
{
[Manage Data](https://cloud.ibm.com/docs-content/v1/content/ded4adc3ea0bd2a81b113c579f2b1183926da211/discovery//images/icon_yourData.png) icon, open your collection, and choose the Search settings tab.

See the [stopwords API reference](https://cloud.ibm.com/apidocs/discoverycreate-stopword-list) for the API commands to upload and delete custom stopword lists.

Notes about stopwords:



* Uploading a custom stopword list is only available for private collections on Advanced and Premium plans, with the exception of the COVID-19 collection available in Lite plans.
* The size limit for a custom stopword list txt file is 10,000 total characters or 1,000 words. If you upload a custom stopwords list with a large number of terms, you might negatively affect search accuracy. A good best practice would be to keep your list of stopwords under 200 total words.
* Only one custom stopword list can be uploaded per collection; if a second custom stopword list is uploaded, it replaces the first.
* All stopwords must be lowercase.
* To disable the custom stopword list, delete the custom stopword list.
* Do not upload or delete a custom stopword list at the same time documents are being ingested into your collection. This could cause the index to be unavailable for that brief period.
* Stopwords are removed at both index and query time. A good best practice is to upload your custom stopword list before uploading documents.



* If your documents are already indexed with the default stopwords and you then add a custom stopword list, the new stopwords are still included in the index. In that case, queries that contain these new stopwords filter them during querying.
* If a user searches for a word that was a stopword at one point in time but is later removed from the custom stopword list, they cannot find documents that match the original stopword because the term was removed during indexing.
For details about how to add a search skill response type, see [Adding a Search skill response type](https://cloud.ibm.com/docs/assistant?topic=assistant-dialog-overviewdialog-overview-add-search-skill).

For more tips about improving results, read the [Improve your natural language query results from Watson Discovery](https://developer.ibm.com/blogs/improving-your-natural-language-query-results-from-watson-discovery/) blog post.
* My response text is surrounded by brackets: If you notice that your response text is surrounded by brackets and quotation marks (["My response text"]) when you test it from the Preview, for example, you might need to change the source field that you're using in the configuration. The unexpected formatting indicates that the value is stored in the source document as an array. Any field that you extract text from must contain a value with a String data type, not an Array data type. When the chat integration shows a response that is extracted from a field that stores the data as an array, it does a straight conversion of the array value into a string, which produces a response that includes the array syntax.

For example, maybe the field in the source document contains an array with a single text value as its only array element:

"title": ["a single array element"]

The array value is converted by the Watson Assistant into this string value:

"title": "["a single array element"]"

As a result, the string is returned in this format in the chat; the surrounding brackets and quotation marks are displayed:

["a single array element"]

If you see this happening, consider choosing a different collection field from which to extract search results.

The Discovery document highlight field stores values in an array.







 Next steps 

After you create the skill, it appears as a tile on the Skills page.

The search skill cannot interact with customers until it is added to an assistant and the assistant is deployed. See [Creating assistants](https://cloud.ibm.com/docs/assistant?topic=assistant-assistant-add).



 Adding the skill to an assistant