vania-janet/multiturn-rag-retrieval-data huggingface dataset:
See [Confidence scores](https://cloud.ibm.com/docs/discovery?topic=discovery-improving-result-relevance-with-the-toolingconfidence) for details.

Adding a custom stopwords list can improve the relevance of results for natural language queries. See [Defining stopwords](https://cloud.ibm.com/docs/discovery?topic=discovery-query-conceptsstopwords) for more information.

See [Training data requirements](https://cloud.ibm.com/docs/discovery?topic=discovery-improving-result-relevance-with-the-apireqs) for the minimum requirements for training, as well as the training limits.

See [Usage monitoring](https://cloud.ibm.com/docs/discovery?topic=discovery-usage) for details on tracking usage and using this data to help you understand and improve your applications.

The components needed to train a Discovery instance include the following:



* Training data. This is the set of queries and examples the service uses to refine query results.
* Query. A natural-language query that applies to the training-data set. Each query has one or more associated examples, as described in the following bullet point. Each query must be unique within the training-data set.
* Example. This is a document indexed in a Discovery collection that acts as an exemplar, good or bad, for the associated query. When you add an example to a training-data query, you include a relevance label that indicates the relevance (or "goodness" versus "badness") of the document as it applies to the specified query.

Examples are identified by the indexed document ID. As noted, every example must include a label that indicates the "goodness" or "badness" of the document as it pertains to the query.

Examples can optionally specify a cross-reference query. The cross-reference query needs to return only the example document and must be independent of the unique Watson Discovery document ID. Cross-reference queries are not currently used automatically but can be used to repair training data in the event that new IDs are assigned to documents during an ingestion event.





 Training data requirements
Improved query builder navigation : Changes to the query builder to make it easier to toggle between the Discovery Query Language and Natural Language query options, as well as among query, filter, and aggregation.





 25 August 2017 

Improved 'passages' array : The passages array now includes field, start_offset, and end _offset. field is the name of the field the passage was extracted from. start_offset is the starting character of the passage text within the field. end_offset is the ending character of the passage text within the field.

New beta enhancement on the Build queries screen : Added the beta ability to write queries in the Discovery Query Language with a visual builder. Click Build in visual mode in the Search for documents and Limit which documents you query sections to try it out. As you build your query visually, it displays in the Discovery Query Language below it. : The visual query builder is currently supported only as a beta capability.





 18 August 2017 

New nested aggregations and conditions in beta visual aggregation builder : Added support for nested aggregations and conditions to the beta visual aggregation builder introduced [11 August 2017](https://cloud.ibm.com/docs/discovery?topic=discovery-release-notes11aug17). There is a limit of 3 conditions per aggregation row. : The visual aggregation builder is currently supported only as a beta capability.





 11 August 2017 

New pre-built sample queries and aggregations on the Build queries screen. : Added the option to select a query from a set of pre-built sample queries and aggregations. Click Use a sample query at the top right to access the list. If you are querying a private data collection, the samples use top entities, categories, etc. found in your collection. These queries can be used as a starting point for writing your own queries. Sample queries are available for both IBM Watson™ Discovery News and private collections.

New beta visual aggregation builder : Added the beta ability to write aggregations with a visual builder. Click Build in visual mode above the Write an aggregation query using the Discovery Query Language field to try it out. As you build your aggregation visually, the query displays in the Discovery Query Language below it.
See the following example response that shows which index was used to answer a query:

{
"dbname": "$ACCOUNT/movies",
"index": {
"ddoc": "_design/32372935e14bed00cc6db4fc9efca0f1537d34a8",
"name": "32372935e14bed00cc6db4fc9efca0f1537d34a8",
"type": "text",
"def": {
"default_analyzer": "keyword",
"default_field": {},
"selector": {},
"fields": []
}
},
"selector": {
"$and": [
{
"$default": {
"$text": "Pacino"
}
},
{
"year": {
"$eq": 2010
}
}
]
},
"opts": {
"use_index": [],
"bookmark": [],
"limit": 10000000000,
"skip": 0,
"sort": {},
"fields": "all_fields",
"r": [
49
],
"conflicts": false
},
"limit": 200,
"skip": 0,
"fields": "all_fields",
"query": "(($default:Pacino) AND (year_3anumber:2010))",
"sort": "relevance"
}
Show more

To instruct a query to use a specific index, add the use_index parameter to the query.

The value of the use_index parameter takes one of the following formats:



* "use_index": "$DDOC"
* "use_index": ["$DDOC","$INDEX_NAME"]
For example, a query for banana is converted to plantain OR fruit and does not contain the original term, banana. If you want an input term to be included in the query, then repeat the input term in the expanded terms list.

Unidirectional example:

{
"expansions": [
{
"input_terms":
"banana"
],
"expanded_terms":
"plantain",
"fruit"
]
},
{
"input_terms":
"car"
],
"expanded_terms":
"car",
"automobile",
"vehicle"
]
}
]
}
Show more

To enable query expansion, complete the following steps:



1. Create a synonyms list file. The file must be a JSON file with the json file extension.

Follow these guidelines:



* Specify the input_terms and expanded_terms in lowercase. Lowercase terms expand to uppercase.
* The synonyms files cannot contain terms that are specified as stop words. For example, if on is included in your stop words file, and you specify in your synonyms file that rotfl expands to rolling on the floor laughing, the expansion won't return the expected results. Check the words in the stop words file that is used by your collection by default to make sure that you don't use any of the same words. For more information, see [Identifying words to ignore](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-stopwords).



You can use the [expansions.json](https://watson-developer-cloud.github.io/doc-tutorial-downloads/discovery/expansions.json) file as a starting point when you build a query expansion list.
2. From the navigation pane, open the Improve and customize page.
3. Expand Improve relevance from the Improvement tools pane.
4. Click Synonyms, and then click Upload synonyms for the collection.
Get details about a query [GET /v1/environments/{environment_id}/collections/{collection_id}/training_data/{query_id}](https://cloud.ibm.com/apidocs/discoverygettrainingdata) [GET /v2/projects/{project_id}/training_data /queries/{query_id}](https://cloud.ibm.com/apidocs/discovery-datagettrainingquery) 
 Delete a training data query [DELETE /v1/environments/{environment_id}/collections/{collection_id}/training_data/{query_id}](https://cloud.ibm.com/apidocs/discoverydeletetrainingdata) [DELETE /v2/projects/{project_id}/training_data /queries/{query_id}](https://cloud.ibm.com/apidocs/discovery-datadeletetrainingquery) 
 List examples for a training data query [GET /v1/environments/{environment_id}/collections/{collection_id}/training_data/{query_id}/examples](https://cloud.ibm.com/apidocs/discoverylisttrainingexamples) [GET /v2/projects/{project_id}/training_data /queries/{query_id}](https://cloud.ibm.com/apidocs/discovery-datagettrainingquery) <br>The examples are in the list that is returned with the query. 
 Add example to training data query [POST /v1/environments/{environment_id}/collections/{collection_id}/training_data/{query_id}/examples](https://cloud.ibm.com/apidocs/discoverycreatetrainingexample) [POST /v2/projects/{project_id}/training_data /queries/{query_id}](https://cloud.ibm.com/apidocs/discovery-dataupdatetrainingquery) <br>Use the Create training query method in v2 and pass all examples when you create the query. Otherwise, use the update API.
Filter query reference 

The Watson Assistant service REST API offers powerful log search capabilities through filter queries. You can use the v2 /logs API filter parameter to search your skill log for events that match a specified query.

The filter parameter is a cacheable query that limits the results to those matching the specified filter. You can filter on various objects that are part of the JSON response model (for example, the user input text, the detected intents and entities, or the confidence score).

To see examples of filter queries, see [Examples](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-filter-referencefilter-reference-examples).

For more information about the /logs GET method and its response model, refer to the [API Reference](https://cloud.ibm.com/apidocs/assistant/assistant-v2listlogs).



 Filter query syntax 

The following example shows the general form of a filter query:



 Location Query operator Term 

 request.input.text :: "IBM Watson" 





* The location identifies the field that you want to filter on (in this example, request.input.text).
* The query operator, which specifies the type of matching you want to use (fuzzy matching or exact matching).
* The term specifies the expression or value you want to use to evaluate the field for matching. The term can contain literal text and operators, as described in the [next section](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-filter-referencefilter-reference-operators).



Filtering by intent or entity requires slightly different syntax from filtering on other fields. For more information, see [Filtering by intent or entity](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-filter-referencefilter-reference-intent-entity).

Note: The filter query syntax uses some characters that are not allowed in HTTP queries. Make sure that all special characters, including spaces and quotation marks, are URL encoded when sent as part of an HTTP query.
* v2 queries return results from all of the collections in the project. To restrict the query to use only certain collections within the project, use the collection_ids query parameter. You cannot query multiple collections that are added to different projects with one v2 query request.
* v2 results include a confidence field, but not a score field.

The confidence score replaced the score information in v1, but score was retained for backward compatibility. In v2, only the confidence field is returned.
* Use POST calls (instead of GET calls) to submit queries with v2.
* v1 queries accept many parameters. The Query parameters comparison table maps v1 parameters to v2 parameters.



Query parameters comparison

 v1 parameter v2 parameter Notes 

 N/A collection_ids Use this parameter in v2 to specify collection ids. 
 filter filter Same expression language. 
 query query Same expression language. 
 natural_language_query natural_language_query No notes. 
 passages passages The passage format changed and was enhanced in v2. The passages:true parameter changed to passages.enable:true. In addition to the count, characters, and fields options, you can specify per_document, which ranks the documents by document quality, and then returns the highest-ranked passages per document. You can also specify find_answers to return an answer object per passage, which contains a succinct answer to the query. 
 aggregation aggregation Same expression language. 
 count count No notes. 
 offset offset No notes. 
 return return No notes. 
 sort sort No notes. 
 highlight highlight If passages.enabled and passages.per_document are true, then passages are returned for each document instead of highlights. 
 spelling_suggestions spelling_suggestions No notes. 
 deduplicate N/A Not supported in v2. 
 similar similar The format changed in v2. The similar:true parameter changed to similar.enable:true. The document_ids and fields parameters changed from strings to string arrays. The document_ids parameter now is required if enabled is true. 
 bias N/A Not supported in v2. 











 Training data 

You can use the v1 training data API to work with two related objects:
* [columnReference](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencecolumnReference)
* [expression](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceexpression)
* [functionOrAggregate](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencefunctionOrAggregate)
* [identifier](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceidentifier)
* [number](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencenumber)
* [query](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencequery)
* [STRING](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceSTRING)
* [valueExpression](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencevalueExpression)
* [timeSeriesExpression](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencetimeSeriesExpression)









 Predicates 



 predicate 

The BETWEEN ... AND predicate compares a value with a range of values. If NOT is specified, the result is reversed.

The IN predicate compares a value or values with a collection of values. The range of values is either defined by a query or a list of expressions that are enclosed in parentheses. The query must identify a number of columns that are the same as the number of expressions that are specified to the left of the IN keyword. In addition, the number of elements in the list of expressions must be the same as the number of expressions that are specified to the left of the IN keyword.
<br>[GET or POST /v1/environments/{environment_id}/collections/{collection_id}/query](https://cloud.ibm.com/apidocs/discoveryqueryusingget) Queries a project. To specify a single collection, include the {collection_id} parameter. Supports a POST request only. <br>[POST /v2/projects/{project_id}/query](https://cloud.ibm.com/apidocs/discovery-dataquery) 
 Query multiple collections [GET or POST /v1/environments/{environment_id}/query](https://cloud.ibm.com/apidocs/discoveryfederatedquery) [POST /v2/projects/{project_id}/query](https://cloud.ibm.com/apidocs/discovery-dataquery) 
 Query system notices [GET /v1/environments/{environment_id}/collections/{collection_id}/notices](https://cloud.ibm.com/apidocs/discoveryquerynotices) [GET /v2/projects/{project_id}/collections/{collection_id}/notices](https://cloud.ibm.com/apidocs/discovery-dataquerycollectionnotices) 
 Query multiple collection system notices [GET /v1/environments/{environment_id}/notices](https://cloud.ibm.com/apidocs/discoveryfederatedquerynotices) [GET /v2/projects/{project_id}/notices](https://cloud.ibm.com/apidocs/discovery-dataquerynotices)
Query aggregations 

Use aggregations to group, analyze, or compare results that are returned by a query request.

An aggregation is defined by an aggregation parameter that you can specify in the Query API. The input to the aggregation parameter is the document set that is returned from the query, filter, or natural_language_query parameter that is specified as a separate parameter in the same query request. Otherwise, the aggregation is applied to all of the documents in the project.

You can use an aggregation to do calculatations from values in the result document set. For example, to get information about the highest dollar amount in the order.total field of the documents that are returned as query results, use max(order.total) as the value of the aggregation parameter.

Zoom

![Example aggregation query structure that shows max as the aggregation type and order.total as a grouping of field details](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/aggregation_structure.png)

Figure 1. Aggregation query structure example

The aggregation parameter returns data about the field with the highest value.

"aggregations": [
{
"type": "max",
"field": "order.total",
"value": 100668.00
}
]



 Grouping documents 

In addition to doing calculations, you can use an aggregation to group documents in the result set that match certain values, so you can count them or analyze them further. For example, you can use an aggregation to search a set of traffic incident reports for documents that mention the term brake. And from the returned documents, find reports from the US states with the most relevant mentions of the term.

In the following example request, the count parameter that returns only 3 aggregation results is included to make the example easier to follow.

{
"query":"brake",
"aggregation": "term(field:STATE,count:3,relevancy:true)"
}
If it is not, then you might want to check two things:



* If the document is not returned in the top 100, it might not be an example of a high-quality result. Reevaluate whether to use the document.
* If the document is not returned at all, then review why it is not returned and see whether any text in the document matches portions of the query.



This warning indicates that you might have one or more failed queries. It doesn't mean that the training cannot be completed.





 Error: Invalid training data found: Syntax error when parsing query 

A syntax error means that the query is invalid. Syntax errors can occur when you increase the complexity of the query by adding a filter to the natural language query, for example. Run the query against the collection outside of relevancy training by using the API. After you confirm that the query is valid and returns results, you can add it as a relevancy training query.





 Error: Training data quality standards not met: You will need additional training queries with labeled examples. (To be considered for training, each example must appear in the top 100 search results for its query.) 

You need to add more training data to train successfully. You need at least 49 unique training queries at a minimum, and each one needs at least one rated document. Minimum does not mean optimal; the size of the collection and other factors can increase the number of training examples that are needed to meet the minimum.





 Error: Training data quality standards not met: Insufficient number of unique training queries. Expected at least n, but found m. 

To meet the minimum training requirements, you need at least 50 unique training queries, and each query must have at least one rated document. If you have more queries than the minimum and are still receiving this error message, check your notices for other errors.





 Error: Training data quality standards not met: No documents found with non-zero relevance labels. 

Training data needs enough labeled data that specifies what documents are high value. Therefore, you need to rate some documents with nonzero values. You need to rate some documents as Relevant and some as Not relevant. At least one document must be rated Relevant.
[::](https://cloud.ibm.com/docs/discovery?topic=discovery-query-operatorsmatch) Exact match title::"Query building" 
 [!:](https://cloud.ibm.com/docs/discovery?topic=discovery-query-operatorsnotinclude) Does not include text:!computer 
 [!::](https://cloud.ibm.com/docs/discovery?topic=discovery-query-operatorsnotamatch) Not an exact match text::!winter 
| [\]Link 5] ] | Escape character | title::"Dorothy said: "There's no place like home"" || | Phrase query | enriched_text.concepts.text:"IBM Watson" || (), []Link 6] | Nested grouping | filter-entities:(text:Turkey,type:Location) || |]Link 7] | or | query-enriched.entities.text:Google|IBM || ,]Link 8] | and | query-enriched.entities.text:Google,IBM || <=, >=, >, <]lt_ _gt_ _gt_ _lt] ] | Numerical comparisons | enriched_text.sentiment.document.score>0.679 || [^x | Score multiplier | text:IBM^3 || *]Link 9] | Wildcard | query-enriched_text.concepts.text:pre* || n]n] | String variation | query-enriched_text.entities.text:cat1 || :*]Link 10] ! | Exists | title: || \]Link 11]! ! ! ! ! | Does not exist | title: |<-- </table ""> --><-- </section "id="section-operators" "> --><-- <section "id="section-aggregations" "> --> Aggregations Aggregations return a set of data values.
Query concepts 

IBM Watson™ Discovery offers powerful content search capabilities. Once your content is uploaded and enriched by Discovery, you can build queries, then integrate Discovery into your own projects, or create a custom application by using the Watson Explorer Application Builder.

The queries you write vary by collection because all collections contain unique content.

When you create a query or filter, Discovery looks at each result and tries to match the paths you define. When matches occur, they are added to the results set. When creating a query, you can be as vague or as specific as you want. The more specific the query, the more targeted the results.

You can write natural language queries (such as "IBM Watson partnerships") using the Discovery tooling or the API.



 Default query behavior 



* Stemming is applied to queries by default, which means that results match the root form of words. In contrast, stemming does not allow for matching on any arbitrary substring of a term. For example, words such as "connections," "connective," and "connected" are reduced to "connect." But words such as "boatbuilding," "boathouse," and "boatload," are not reduced to "boat."
* Tokenization occurs using a set of standard separators per language including whitespace, hyphen, underscore, and punctuation marks for English. For more information, see [Creating custom tokenization dictionaries](https://cloud.ibm.com/docs/discovery?topic=discovery-query-conceptstokenization).
* All private collections return a confidence score in the query results in most cases. For more information, see [Confidence scores](https://cloud.ibm.com/docs/discovery?topic=discovery-improving-result-relevance-with-the-toolingconfidence).
* Discovery returns query results that include special characters for the following languages: English, German, French, Dutch, Italian, and Portuguese. For example, if you query for aqui, you receive results for both aqui and aquí.







 Options for querying 



* You have the option to turn on passage retrieval. Passages are short, relevant excerpts extracted from the full documents returned by your query.
To find answers within a single known document (for example, a document review application with long, complex documents):



* Set passages.enabled to true
* Set passages.find_answers to true
* Set filter to select the document_id for the document



The following example shows a query that uses this API:

POST /v2/projects/{project_id}/query{
"natural_language_query": "Why did Nixon resign?",
"passages": {
"enabled": true, "find_answers":true
}
}

Example response:

{
"matching_results": 74, "retrieval_details": { "document_retrieval_strategy": "untrained"},
"results": [
{
"document_id": "63919442-7d5b-4cae-ab7e-56f58b1390fe",
"result_metadata":{"collection_id": "collection_id1234","document_retrieval_source":"search","confidence": 0.78214},
"metadata": {"parent_document_id": "63919442-7d5b-4cae-ab7e-56f58b1390fg"},
"title": "Watergate scandal",
"document_passages":
{
"passage_text": "With his complicity in the cover-up made public and his political support completely eroded, Nixon resigned from office on August 9, 1974. It is believed that, had he not done so, he would have been impeached by the House and removed from office by a trial in the Senate.",
"field": "text",
"start_offset": 281,
"end_offset": 553,
"answers":
{
"answer_text": "his complicity in the cover-up made public and his political support completely eroded",
"start_offset": 286, "end_offset": 373, "confidence": 0.78214
}