vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Adding search 

Put your subject matter expertise to work by adding search. Search gives your assistant access to corporate data collections that it can mine for answers.

Your assistant can route complex customer inquiries as a search query. It finds information that is relevant to the query from an external data source and returns it to the assistant.

Add search to your assistant to prevent the assistant from having to say things like, I'm sorry. I can't help you with that. Instead, the assistant can query existing company documents or data to see whether any useful information can be found and shared with the customer.

You have two options to add search to your assistant:



* The search integration for IBM Watson® Discovery. For more information, see [Add the search integration for IBM Watson® Discovery](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-search-overviewsearch-overview-integration).
* A search extension for Coveo, Google, or NeuralSeek. For more information, see [Add a search extension](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-search-overviewsearch-overview-extension).





 Add the search integration for IBM Watson® Discovery 

Plus and Enterprise plans include a built-in search integration. You can embed your existing help content by integrating your assistant with search that is provided by IBM Watson® Discovery. This gives your assistant access to your organization's data collections that it can mine for answers. Customer questions are used as search queries to find relevant answers for your users.

For instructions on adding the built-in search integration, see [IBM Watson® Discovery search integration setup](https://cloud.ibm.com/docs/watson-assistant/?topic=watson-assistant-search-add).





 Add a search extension 

With a search extension, you can "bring your own search" and use content from your website, knowledge base, or content management system.

For instructions on adding a search extension, see:



* [Coveo search extension setup](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-search-extension-coveo)
Up to 60 passages are sent to the answer-finding service. How these 60 passages are chosen differs based on the passages.per_document parameter value.



* If passages.per_document is false, the top 60 passages from all of the documents that are returned by search are chosen based on their passage scores only.
* If passages.per_document is true, the returned documents are ranked first, and then the top 60 passages from these top documents are chosen.

For example, if you set the query to return 100 documents (count=100) and ask for 2 passages from each document (passages.max_per_document=2), then 2 passages are chosen from each of the 30 top-ranked documents (2 x 30 = 60 passages) only. No passages are chosen from the remaining 70 documents.



If your goal is to get the best 10 short answers, a good approach is to give the answer-finding feature various passages from more documents than just the top 10. To do so, set passages.per_document to true, and then request 20 documents and up to 3 passages from each document with the answer-finding feature enabled. The answer-finding feature searches for answers in up to 20*3 = 60 passages.

Answer finding does not use the transformed query string that is generated by query analysis. Instead, it uses a copy of the user's original input that is stored at query time to find the best short answer. If the answer-finding module is confident that it found an answer in one of the passages, the answer confidence score is combined with the document and passage scores to produce a final ranking, which can promote a document or passage that might otherwise be missed.





 Answer-finding API details 

The answer-finding API adds the following parameters to the passage section of the query API:



* find_answers is optional and defaults to false. If it is set to true (and the natural_language_query parameter is set to a query string), the answer-finding feature is enabled.
* max_answers_per_passage is optional and defaults to 1. In this case, the answer-finding feature finds the number of answers that are specified at most from any one passage.



A section is also added to the return value within each passage object.
sort number <br>subject <br>severity <br>updatedAt Only one of the sort options can be used at one time. You can use the prefix to reverse the sorting order. 
 status new <br>inProgress <br>waitingOnClient <br>resolutionProvided <br>resolved <br>closed Any number of options can be used. The available options can be entered as status:new,inProgress or as status:new status:inProgress. 
 page target page to view If you have several results from your search that spans multiple pages, you can view your results from any result page. For example, to view page 5 out of 10, use page:5. 
 pageSize 10 <br>25 <br>50 <br>100 The size of that page to be viewed. The page size refers to the number of results that you want to load. 



If you enter a term without a parameter, the search results are shown for the support case number and the case subject.



 Query and URL examples 

You can enter search queries in the search bar by stating a parameter and option separated by a colon (:) or you can use parameters in a URL to go directly to a specific case or group of cases on the Manage cases page.

To use parameters in the URL, add a question mark (?) to the end of the URL. Then, set your parameter equal to the option you select. You can also separate additional parameters with an ampersand (&).

To search for a case by keyword, you can enter the word in the search bar without a parameter. To search for a keyword with the URL, use search as a parameter and set it equal to your keyword, for example, search=server.



 Searching by case number 

Enter the following to search support cases by case number:

Query
: number:CS1234567

URL
: https://cloud.ibm.com/unifiedsupport/cases?number=CS1234567





 Searching with multiple parameters 

You might want to search for both new and in progress support cases, limit the results to 25 per page, and displayed by severity.
However, only 2 of the matching documents have a Company entity type.

By default, the top 10 matches are returned, sorted by relevance. You can change the number of results by adding the count parameter to the aggregation.

{
"filter": "enriched_text.entities:(text::Gilroy,type::Location)",
"aggregation": "term(enriched_text.entities.type,count:20)"
}



 Add a filter 

Use the filter() in the aggregation clause to filter results. For example, you can specify the same filter that was submitted separately in the previous example directly in the aggregation clause.

{
"aggregation": "filter(enriched_text.entities:(text::Gilroy,type::Location)).term(enriched_text.entities.type)"
}

In this case, the filter().term() aggregation finds the same result as the earlier example with the separate filter and aggregation clauses. However, results are ranked differently when the filter clause is used. You can leverage this difference by using the filter() clause within the aggregation clause to filter results from a sequence of expressions, as shown in the next example.





 Start with nested objects 

In the previous examples, the "matching_counts" value represents the number of documents that match the filter and aggregation. You might want to count how many nested objects are present in the query response. The nested() aggregation allows you to change the set of documents that is used as input to other aggregation terms.

For example, in the following query the nested() segment selects all enriched_text.entities nested objects as the input used by the filter() and term() segments.

{
"aggregation": "nested(enriched_text.entities).filter(enriched_text.entities.type::Organization).term(enriched_text.entities.text,count:3)"
}

The query results in an aggregations object that looks as follows:

{
"aggregations": [
{
"type": "nested",
"path": "enriched_text.entities",
"matching_results": 1993,
If false, ranks the passages from all of the documents by passage quality regardless of the document quality and returns them in a separate passages field in the response.
* When passages are returned for a query, you can also enable answer finding. When true, answer objects are returned as part of each passage in the query results. When find_answers and per_document are both set to true, the document search results and the passage search results within each document are reordered by using the answer confidences. The goal of this reordering is to place the best answer as the first answer of the first passage of the first document. Similarly, if the find_answers parameter is set to true and per_document parameter is set to false, then the passage search results are reordered in decreasing order of the highest confidence answer for each document and passage.
* Both v1 and v2 support custom stop words. However, there are a few differences in how custom stop words are used:



* There is no default custom stop words list for Japanese collections in v2.
* When you define custom stop words in v1, your stop words list replaces the existing stop words list. In v2, your list augments the default list. You cannot replace the list, which means you cannot remove stop words that are part of the default list in v2.







 Update how your application handles query results 

The way that your application shows query results might need to be updated due to the following differences between the query results document syntax between the v1 and v2 queries:



* At the entity enrichment level, the following information is not supported in v2:



* Disambiguation
* Emotion
* Sentiment



The Part of Speech enrichment is applied automatically to documents in most project types in v2, but the index fields that are generated by the enrichment are not displayed in the JSON representation of the document.

Zoom

![Difference in entities data structure](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/compare-result-syntax.png)

Figure 1.
You can then create a collection from a data source and configure your search skill to search this collection for answers to customer queries.

The following diagram illustrates how user input is processed when both a dialog skill and a search skill are added to an assistant. Any questions that the dialog is not designed to answer are sent to the search skill, which finds a relevant response from a Discovery data collection.

![Diagram of how a question gets routed to the search skill.](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/search-skill-diagram.png)





 Creating a skill 

You can add one conversation skill (actions or dialog) and one search skill to an assistant.



 Conversation 



* [Actions skill](https://cloud.ibm.com/docs/assistant?topic=assistant-skill-actions-add)
* [Dialog Skill](https://cloud.ibm.com/docs/assistant?topic=assistant-skill-dialog-add)







 Search 



* [Search Skill](https://cloud.ibm.com/docs/assistant?topic=assistant-skill-search-add)









 Skill limits 

The number of skills you can create depends on your Watson Assistant plan type. Any sample dialog skills that are available for you to use do not count toward your limit unless you use them. A skill version does not count as a skill.



Plan details

 Plan Maximum number of skills of each type per service instance 

 Enterprise 100 
 Premium (legacy) 100 
 Plus 50 
 Standard (legacy) 20 
 Lite, Trial 5 



* After 30 days of inactivity, an unused skill in a Lite plan service instance might be deleted to free up space.
Alternatively, you can add a generative language service named NeuralSeek between the Watson Discovery and Watson Assistant services. For more information, see [Use NeuralSeek to return polished answers from existing help content](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-tutorial-neuralseek).



 How the assistant calls Discovery 

When a user asks your assistant a question that triggers a search, the following API request is sent to Discovery if Emphasize the answer is enabled.

The Emphasize the answer feature is available from instances that are managed by IBM Cloud only.

{
"aggregation": "",
"sort": "",
"count": 10,
"return": [],
"filter": <custom_filter_specified_in_assistant>
"passages": {
"enabled": "true",
"fields": [
<search_config_body_field_specified_in_assistant>
],
"characters": 325,
"per_document": true,
"max_per_document": 3,
"find_answers": true,
"max_answers_per_passage": 1
},
"highlight": false,
"spelling_suggestions": false,
"table_results": {
"enabled": false
},
"suggested_refinements": {
"enabled": false
}
}
Show more

When Emphasize the answer is used ("find_answers": true), Discovery rescores and reorders the documents to ensure that documents with the highest-quality answers are returned first.





 Choosing a project type 

If the Conversational Search project type isn't providing the best answers and you want to understand why, switch to using a Document Retrieval project type.

Most often, the Conversational Search project type is the right choice. You get great results from the start, and when you enable extra features like Emphasize the answer, the answers are clear and concise.
If there are many top level fields, this can cause a performance impact for a natural_language_query when using training. Reducing the number of top level fields can improve performance. This can be done via normalization or by manually editing the JSON to put fields that are not helpful to finding relevant content in a nested structure. Changing the fields used for training also has an impact on the model so you’ll need to consider both the impact to performance and accuracy of results if making this change. For more information about relevancy training, see [Improving result relevance](https://cloud.ibm.com/docs/discovery?topic=discovery-improving-result-relevance-with-the-tooling).
3. Continuous Relevancy Training - Continuous relevancy training searches across all collections in an environment. The more collections in an environment, the larger the impact on performance. See [Continuous Relevancy Training](https://cloud.ibm.com/docs/discovery?topic=discovery-crt) for more information.







 Collection size and complexity 

The makeup of the documents in a private collection can also impact query performance:



1. Total number of documents - Performance might be affected when approaching the upper limits of Number of Documents for Advanced plan sizes.
2. Size of documents - Very large documents (those several MB in size) require movement of a large amount of data per request, which can negatively impact performance especially when using passage retrieval and relevancy training.
3. Number of enrichments - Enrichments add complexity to documents by adding a significant number of nested fields. If an enrichment is not necessary for your use case, consider disabling it at ingestion time. Enrichments are not directly used for natural_language_query search relevance. See [Adding enrichments](https://cloud.ibm.com/docs/discovery?topic=discovery-configserviceadding-enrichments) for more about information on enrichments.
Later, you can limit the search to return content from only the tips field.

Or maybe you have extra large documents that contain subsections. You can teach the SDU model to recognize these subsections, and then split the large document into multiple, smaller, and easier-to-manage documents that begin with one of these subsections.
* The best way to prepare a collection for use in Conversational Search projects is to identify discrete question-and-answer pairs. You can use the SDU tool to find and annotate them. If you configure the project to contain answers in an answer field, you must update the search configuration in Watson Assistant to get the body of the response from the custom answer field.
* A pretrained SDU model is applied to Document Retrieval for Contracts projects automatically. The pretrained SDU model knows how to recognize terms and concepts that are significant to contracts. As a result, you cannot apply a user-trained SDU model to this project type, but you also don't need to.
* The SDU tool is rarely used with Content Mining projects.



You can use the SDU tool to annotate the following file types only:



* Image files (PNG, TIFF, JPG)
* Microsoft PowerPoint
* Microsoft Word
* PDF



For a complete list of file types that Discovery supports, see [Supported file types](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionssupportedfiletypes).

The Smart Document Understanding tool uses optical character recognition (OCR) to extract text from images in the files that it analyzes. Images must meet the minimum quality requirements that are supported by OCR. For more information, see [Optical character recognition](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionsocr).

The tool cannot read documents with the following characteristics; remove them from your collection before you begin:



* Documents that appear to have text that overlays other text are considered double overlaid and cannot be annotated.
* Documents that contain multiple columns of text on a single page cannot be annotated.
Sometimes it is more efficient to store some items in the index rather than doing include_docs=true at query time.

Lucene has its own query language that creates queries that match combinations of clauses with logic, fuzzy matching, ranges, and term boosting.

See the following examples:



* Find documents whose title matches gastby and whose author starts with fitz. Notice the asterisk wildcard.
* Find documents whose author is in the range austen to dickens. This example shows a range query on a string field.
* Find documents whose price is 0 - 100AND whose year is in the 19th century or whose author matches charles dickens. This example shows how logic can be built into queries.



IBM Cloudant Search is useful not only for free-text search. It is also useful when you know which attributes you are going to search on, but the queries are varied, with different combinations of attributes each time. This flexibility is difficult to implement with fixed-order MapReduce indexes.

Faceting is a form of aggregation. You nominate individual indexed fields for faceting at index-time and activate the aggregation with parameters at query-time.

It has two uses:

Counting repeating values in the result set, such as the counts of products that belong to each category in a result set. Or, counting numbers of items in numeric ranges, such as the product counts in each of the price ranges. Both of these forms of counts can be presented to a front-end user as a means of further filtering an existing search to narrow the scope of the search. For example, a customer searches for Fender, then clicks Amps then on a price range under $500. All of this search and filtering can be powered by IBM Cloudant Search.

To summarize, IBM Cloudant Search indexes are defined with a supplied JavaScript function. They are built on Apache Lucene and are primarily used for free-text search matching, but the query language is useful for building flexible queries on a fixed set of indexed fields. It also has some powerful counting aggregations suitable for drill-down user interfaces.

IBM Cloudant Search also powers type=text IBM Cloudant Query indexes, so a subset of its capabilities is surfaced by using the _find API.
[Review intent examples](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/oe-click-tokens.png)

A Search box opens that you can use to search for the entity that the highlighted word or phrase is a mention of.

![Search box with search parameter prod](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/oe-tokens-clicked.png)
5. Enter the entity name to search for. You do not need to include the starting @ symbol.

Do one of the following things:



* If the entity has any existing entity values, they are displayed for informational purposes only. You are adding the annotation to the entity, not to any specific entity value.
* If you want to teach the model that the mention is synonymous with an existing entity value, add a colon (:) after the entity name to show a list of entity values. Choose an entity value from the list that is displayed. For example, @product:device.

![Search box with search parameter prod](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/oe-intent-review.png)



6. Select the entity or entity and value to which you want to add the annotation.

In this example, mobile phones is being added as an annotation for the @product entity value and as a synonym for the @product:device entity value.

Create at least 10 annotations for each contextual entity; more annotations are recommended for production use.
7. If none of the entities are appropriate, you can create a new entity by adding its name.
* Do not upload or delete a tokenization dictionary at the same time documents are being ingested into your collection.<-- </ul> --><-- </section "id="section-tokenization" "> --><-- <section "id="section-doc-similarity" "> --> Document similarity A document similarity query finds other documents similar to the currently viewed document, for example, a call center operator could be viewing the manuals for a product and use document similarity to find other documents with similar characteristics. You can query for similar documents by similar.document_ids, and can optionally refine the similarity by specifying additional similar.fields.Document similarity is determined by extracting the 25 most relevant terms from the original document and then searching for documents with similar relevant terms.Example query by similar.document_ids (do not insert a space after the comma if specifying multiple similar.document_ids):curl -u "apikey":"{apikey_value}" "{url}/v1/environments/{environment_id}/collections/{collection_id}/query?version=2019-04-30&similar.document_ids=4107b6f1-5d3f-4bea-bbcf-fb05bbf960b1,6057k6d1-7d7k-6aeh-cfbb-kj98ssf786c2"Example query with similar.fields added:curl -u "apikey":"{apikey_value}" "{url}/v1/environments/{environment_id}/collections/{collection_id}/query?version=2019-04-30&similar.document_ids=4107b6f1-5d3f-4bea-bbcf-fb05bbf960b1&similar.fields=title&return=title&count=100"Replace {apikey} and {url} with your API key and URL.See the document similarity API reference]API Reference] and query parameters]] ! ! ! !
"location": {
"begin": 52,
"end": 69
}
}
],
"relevance": 0.952591
},
{
"text": "Boston",
"mentions":
{
"text": "Boston",
"location": {
"begin": 74,
"end": 80
}
}
],
"relevance": 0.578079
},
{
"text": "MA",
"mentions":
{
"text": "MA",
"location": {
"begin": 82,
"end": 84
}
}
],
"relevance": 0.146905
}
],
"entities":
{
"text": "George Washington",
"type": "Location",
"mentions":
{
"text": "George Washington",
"confidence": 0.54922265,
"location": {
"begin": 52,
"end": 69
}
}
],
"model_name": "natural_language_understanding"
},
{
"text": "Boston, MA",
"type": "Location",
"mentions":
{
"text": "Boston, MA",
"confidence": 0.66049105,
"location": {
"begin": 74,
"end": 84
}
}
],
"model_name": "natural_language_understanding"
}
]
}
],
"metadata": {},
"enriched_html": [
{
"tables":
{
"body_cells":
{}
],
"location": {
"begin": 99,
"end": 183