vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Query operators 

Operators are the separators between different parts of a query. For the complete list of available operators, see the [Query reference](https://cloud.ibm.com/docs/discovery?topic=discovery-query-referenceoperators).



 . [JSON delimiter] 

This delimiter separates the levels of hierarchy in the JSON schema

For example:

enriched_text.concepts.text





 : [Includes] 

This operator specifies a match for the query term.

For example:

enriched_text.concepts.text:"cloud computing"





 :: [Exact match] 

This operator specifies an exact match for the query term.

For example:

enriched_text.concepts.text::"Cloud computing"

Exact matches are case-sensitive.





 :! [Does not include] 

This operator specifies that the results do not contain a match for the query term

For example:

enriched_text.concepts.text:!"cloud computing"





 ::! [Not an exact match] 

This operator specifies that the results do not exactly match the query term

For example:

enriched_text.concepts.text::!"Cloud computing"

Exact matches are case-sensitive.





 \ [Escape character] 

Escape character for queries that require the ability to query terms by using string literals that contain control characters.

For example:

title::"Dorothy said: "There's no place like home""





 "" [Phrase query] 

All contents of a phrase query are processed as escaped. So no special characters within a phrase query are parsed, except for double quotes (") inside a phrase query, which must be escaped ("). Use phrase queries with full-text, rank-based queries, and not with boolean filter operations. Do not use wildcards () in phrase queries.

Single quotes (') are not supported.

For example:

enriched_text.entities.text:"IBM watson"





 (), [] [Nested grouping] 

Logical groupings can be formed to specify more specific information.

For example:
For example, !hello would match the text !hello. 
 "" Literal phrase. Use to encloses a query term that contains spaces or other special characters. No special characters within the quotation marks are parsed by the API. 
 Approximate match. Append this operator followed by a 1 or 2 to the end of the query term to specify the allowed number of single-character differences between the query string and a match in the response object. For example, car1 would match car, cat, or cars, but it would not match cats. This operator is not valid when filtering on log_id or any date or time field, or with fuzzy matching. 
 * Wildcard operator. Matches any sequence of zero or more characters. This operator is not valid when filtering on log_id, language, request.context.system.assistant_id, workspace_id, request.context.metadata.deployment, or any date or time field. 
 (), [] Grouping operators. Use to enclose a logical grouping of multiple expressions using Boolean operators. 
 Boolean or operator. 
 , Boolean and operator. 





 Filtering by intent or entity 

Because of differences in how intents and entities are stored internally, the syntax for filtering on a specific intent or entity is different from the syntax used for other fields in the returned JSON. To specify an intent or entity field within an intents or entities collection, you must use the : match operator instead of a dot.

For example, this query matches any logged event where the response includes a detected intent named hello:

response.output.intents:intent::hello

Note the : operator in place of a dot (intents:intent)

Use the same pattern to match on any field of a detected intent or entity in the response. For example, this query matches any logged event where the response includes a detected entity with the value soda:

response.output.entities:value::soda

Similarly, you can filter on intents or entities sent as part of the request, as in this example:

request.input.intents:intent::hello





 Filtering by other fields
* Group Specifies the group of users from which the query originates.
* Result rows Specifies the number of result rows that were returned by the query.
* Prep time Specifies the preparation time that was needed for the query.
* Status Specifies the completion status of the query.
* Plan ID Specifies the ID of the system-generated plan for the query.
* Client IP Specifies the IP of the client that ran the SQL query.
* GRA time Specifies the time that the query spent at the GRA.
* Checksum Specifies the checksum of the query.
* Client type Specifies the type of the client that ran the query.
* Client host Specifies the hostname of the client from which the query originates.
* Resource group name Specifies the resource group on which the query ran.
* Session ID Specifies the ID of the session in which the query ran.
* Client user ID Specifies the client user ID of the client that ran the query.
* Client application name Specifies the client application name from which the query originates.
* Client workstation name Specifies the client workstation name from which the query originates.
* Client accounting string Specifies the accounting string of the client that ran the query.
DQL overview 

The Discovery Query Language defines syntax you can use to filter, search, and analyze your data.



 How to write a Discovery Query Language query 

The Discovery Query Language leverages the structure of indexed documents. The following JSON snippet shows an indexed document from a collection where the Entities enrichment is applied. As a result of the enrichment, the JSON structure captures any mentions of known entities, such as city names, companies, or famous people.

In this example, the recognized entity is the company name IBM.

{
"document": {
"document_id": "f7f27ea30eb3e4c0ce21830618d9ee99",
"enriched_text": [
{
"entities":
{
"model_name": "natural_language_understanding",
"mentions": ],
"text":"IBM",
"type":"Organization"
}
]
}
]
}
}
Show more

To create a query that returns all of the documents in which the entity IBM is mentioned, use the following syntax:

Zoom

![The structure of the query enriched_text.entities.text:IBM, where text in enriched_text is the field where the enrichment is applied, and IBM is the term you are looking for in the enriched_text.entities.text subfield.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/query_structure.png)

Figure 1. Example query structure

This basic query contains a nested path expression before the : operator. Each path element is the name of a field in the document separated by a period (.). The : operator indicates that the text that follows the operator must be included in the result.

The :: operator indicates that the text must be matched exactly in the result.
Query expansion terms are usually synonyms, antonyms, or typical misspellings for common terms.

You can define two types of expansions:



* bidirectional - each expanded_term expands to include all expanded terms. For example, a query for ibm expands to ibm OR international business machines OR big blue).
* unidirectional - the input_terms in the query are replaced by the expanded_terms. For example, a query for banana could expand to plantain and fruit. input_terms are not used as part of the resulting query. In the previous banana example, the query banana would be converted to plantain OR fruit and not contain the original term.



This file can be used as a starting point when building a query expansion list: [expansions.json ![External link icon](https://cloud.ibm.com/docs-content/v1/content/ded4adc3ea0bd2a81b113c579f2b1183926da211/icons/launch-glyph.svg)](https://watson-developer-cloud.github.io/doc-tutorial-downloads/discovery/expansions.json). You can modify this file to create a custom query expansion list.

Bidirectional example:

{
"expansions": [
{
"expanded_terms":
"ibm",
"international business machines",
"big blue"
]
}
]
}

Unidirectional example:

{
"expansions": [
{
"input_terms":
"car"
],
"expanded_terms":
"car",
"automobile",
"vehicle"
]
},
{
"input_terms":
"banana"
],
"expanded_terms":
"banana",
"plantain",
"fruit"
]
}
]
}
Show more

Notes about query expansion:
Query parameters 

IBM Watson™ Discovery offers powerful content search capabilities through queries. After your content is uploaded and enriched by Discovery, you can build queries, integrate Discovery into your own projects, or create a custom application by using the Watson Explorer Application Builder. To get started with queries, see [Query concepts](https://cloud.ibm.com/docs/discovery?topic=discovery-query-concepts). For the complete list of parameters, see the [Query reference](https://cloud.ibm.com/docs/discovery?topic=discovery-query-referenceparameter-descriptions).

Search parameters

Search parameters enable you to search your collection, identify a result set, and perform analysis on the result set.

The results set is the group of documents identified by the combined searches of the search parameters. The results set might be significantly larger than the returned results. If an empty query is performed, the results set is equal to all the documents in the collection.



 query 

A query search returns all documents in your data set with full enrichments and full text in order of relevance. A query also excludes any documents that don't mention the query content. These queries are written using the [Discovery Query Language](https://cloud.ibm.com/docs/discovery?topic=discovery-query-operators).





 filter 

A cacheable query that excludes any documents that don't mention the query content. Filter search results are not returned in order of relevance. These queries are written using the [Discovery Query Language](https://cloud.ibm.com/docs/discovery?topic=discovery-query-operators).



 Differences between the filter and query parameters 

If you test the same search term on a small data set, you might find that the filter and query parameters return very similar (if not identical) results. However, there is a difference between the two parameters.



* Using a filter parameter alone returns search results in no specific order.
* Using a query parameter alone returns search results in order of relevance.



In large data sets, if you need results returned in order of relevance, it is advisable that you combine the filter and query parameters because using them together improves performance.
Use to enclose a query term that contains spaces or other special characters. No special characters within the quotation marks are parsed by the API. 
 Approximate match. Append this operator followed by a 1 or 2 to the end of the query term to specify the allowed number of single-character differences between the query string and a match in the response object. For example, car1 would match car, cat, or cars, but it would not match cats. This operator is not valid when filtering on log_id or any date or time field, or with fuzzy matching. 
 * Wildcard operator. Matches any sequence of zero or more characters. This operator is not valid when filtering on log_id, language, request.context.system.assistant_id, workspace_id, request.context.metadata.deployment, or any date or time field. 
 (), [] Grouping operators. Use to enclose a logical grouping of multiple expressions using Boolean operators. 
 Boolean or operator. 
 , Boolean and operator. 





 Filtering by intent or entity 

Because of differences in how intents and entities are stored internally, the syntax for filtering on a specific intent or entity is different from the syntax used for other fields in the returned JSON. To specify an intent or entity field within an intents or entities collection, you must use the : match operator instead of a dot.

For example, this query matches any logged event where the response includes a detected intent named hello:

response.output.intents:intent::hello

Note the : operator in place of a dot (intents:intent)

Use the same pattern to match on any field of a detected intent or entity in the response. For example, this query matches any logged event where the response includes a detected entity with the value soda:

response.output.entities:value::soda

Similarly, you can filter on intents or entities sent as part of the request, as in this example:

request.input.intents:intent::hello





 Filtering by other fields 

To filter on another field in the log data, specify the location as a path identifying the levels of nested objects in the JSON response from the /logs API.
Just be sure to design your dialog such that the search is not triggered unless any entities or context variables that you reference in the query have been set to valid values.

This field is equivalent to the Discovery natural_language_query parameter. For more information, see [Query parameters](https://cloud.ibm.com/docs/discovery?topic=discovery-query-parametersnlq).



* Filter: Optional. Specify a text string that defines information that must be present in any of the search results that are returned.



* To indicate that you want to return only documents with positive sentiment detected, for example, specify enriched_text.sentiment.document.label:positive.
* To filter results to includes only documents that the ingestion process identified as containing the entity Boston, MA, specify enriched_text.entities.text:"Boston, MA".
* To filter results to includes only documents that the ingestion process identified as containing a product name provided by the customer, you can specify enriched_text.entities.text:@product.
* To filter results to includes only documents that the ingestion process identified as containing a city name that you saved in a context variable named $destination, you can specify enriched_text.entities.text:$destination.



This field is equivalent to the Discovery filter parameter. For more information, see [Query parameters](https://cloud.ibm.com/docs/discovery?topic=discovery-query-parametersfilter).



If you add both a query and a filter value, the filter parameter is applied first to filter the data collection documents and cache the results. The query parameter then ranks the cached results.



3. Optional: Change the query type that is used for the search.

The search skill sends a natural language query to Discovery automatically. If you want to use the Discovery query language instead, you can specify it. To do so, open the JSON editor for the node response.

Edit the JSON code snippet to replace natural_language with discovery_query_language. For example:

{
"output": {
"generic": [
{
"query": "",
For relevancy training to be used, you must successfully train the project either programmatically ([Create training query method](https://cloud.ibm.com/apidocs/discovery-datacreatetrainingquery)) or by using the product user interface.

QPP
: A Query Performance Prediction algorithm that, given a query and a list of top results, produces a score that determines how relevant a document is. Used only if no Relevancy training ranker is available.

filter
: The filter parameter can be passed along with query and natural_language_query requests to remove documents that don't meet certain criteria from the result set. The filter is shown as the last step within the document retrieval phase. However, it is used at different times in the flow. Its placement in the diagram is chosen to emphasize the fact that any documents that don't match the filter definition are excluded from the result set. The exclusion applies even to documents that might be specified in a curation.

Passage retrieval
: Returns passages from documents when the passages.enabled=true parameter is included with a natural language query request.

Answer finding
: When the passages.find_answers=true parameter is included with a natural language query request, returns succinct answers from passages along with the passages that are extracted from documents. If answer finding is enabled, then the final confidence score for each search result is a combination of the confidence scores from answer finding, passage retrieval, and QPP or Reranked search, whichever method is used.

Table retrieval
: Returns information from tables in documents when the table_results.enabled=true parameter is included with a natural language query request.





 Query limits 

A query is any operation that submits a POST request to the /query endpoint of the API. Such operations include queries that are submitted by using the API. It does not include queries that are submitted from the search bar on the Improve and customize page of the product user interface.

A query is counted only if the request is successful, meaning it returns a response (with message code 200).

The number of search queries that you can submit per month per service instance depends on your Discovery plan type.



Number of queries per month
Example of searching logs for a natural language query that contains the term “train”:

{url}/v1/logs?version=2019-04-30&query=train

Replace {url} with your URL.

The response of a logs query includes results that appear similar to Discovery document results. Each result is either a query or event, specified in the document type field.

Example query log:

{
"customer_id": "",
"environment_id": "252e6e82-c2d2-450e-9670-0008b0a3a99c",
"natural_language_query": "how do i get from the airport to the city",
"query_id": "_Qs35yOoa7",
"document_results": {
"count": 2,
"results": [
{
"collection_id": "af9e1f41-6114-4d77-bca6-c9b99b2601b1",
"score": 6.724753491503856,
"position": 1,
"document_id": "4193eaa727d79b0f74597356dbcbb9a7"
},
{
"collection_id": "af9e1f41-6114-4d77-bca6-c9b99b2601b1",
"score": 5.791631414211986,
"position": 2,
"document_id": "bdcd6a9cc1438a3faa8c925f6a8d9429"
}
]
},
"event_type": "query",
"session_token": "1_LKczxWGEWx5TJAi1_Qs35yOoa7",
"created_timestamp": "2018-09-12T05:20:07.469Z"
}
Show more
"query":"enriched_text.entities.text:IBM^3"
}





 * (Wildcard) 

Matches unknown characters in a search expression. Do not use capital letters with wildcards.

For example:

{
"query":"enriched_text.entities.text:ib"
}





 n (String variation) 

The number of character differences that are allowed when matching a string. The maximum variation number that can be used is 2.

For example, the following query returns documents that contain car in the title field, as well as cap,cat,can, sat, and so on:

{
"query":"title:cat1"
}

The normalized version of the word is used for matching. Therefore, if the input contains "cats", the search looks for "cat", which is the normalized form of the plural cats.

When a phrase is submitted, each term in the phrase is allowed the specified number of variations. For example, the following input matches cat dog and far log in addition to car hog.

For example:

{
"query":"title:"car hog"1"
}





 :* (Exists) 

Used to return all results where the specified field exists.

For example:

{
"query":"title:"
}





 :!* (Does not exist) 

Used to return all results that do not include the specified field.

For example:

{
"query":"title:!"
}

For more information, see the Discovery [API reference](https://cloud.ibm.com/apidocs/discovery-dataquery).

For an overview of query concepts, see the [Query overview](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-concepts).
"expanded_terms":
"car",
"automobile",
"vehicle"
]
},
{
"input_terms":
"banana"
],
"expanded_terms":
"banana",
"plantain",
"fruit"
]
}
]
}
Show more

Notes about query expansion:



* Query expansion is only available for private collections. The number of available expansions arrays (total bidirectional and unidirectional arrays) and terms (the total input_terms plus expanded_terms) varies by plan. See [Discovery pricing plans](https://cloud.ibm.com/docs/discovery?topic=discovery-discovery-pricing-plans) for details.

All query terms, including both input_terms and expanded_terms, each count as one term. This example contains two objects in the expansions array and seven term strings.



{
"expansions": [
{
"input_terms":
"ibm"
],
"expanded_terms":
"ibm",
"watson"
]
},
{
"input_terms":
"banana"
],
"expanded_terms":
"banana",
"plantain",
"fruit"
]
}
]
}
Show more



* Only one query expansion list can be uploaded per collection; if a second expansion list is uploaded, it replaces the first.
* All input_terms and expanded_terms must be lowercase. Lowercase terms expand to uppercase.
* The query expansion list must be written in JSON.
* To disable query expansion, delete the query expansion list.
* Query expansion is performed on the query and multiple collection query methods.
* Each set of expansions is associated with a collection. When querying across [multiple collections](https://cloud.ibm.com/docs/discovery?topic=discovery-query-conceptsmultiple-collections), each collection is expanded individually.
Use phrase queries with full-text, rank-based queries, and not with boolean filter operations. Do not use wildcards () in phrase queries.

Single quotes (') are not supported.

For example:

enriched_text.entities.text:"IBM watson"





 (), [] [Nested grouping] 

Logical groupings can be formed to specify more specific information.

For example:

enriched_text.entities:(text:IBM,type:Company)





 | [or] 

Boolean operator for "or".

In the following example, documents in which Google or IBM are identified as entities are returned:

enriched_text.entities.text:Google|enriched_text.entities.text:IBM

The includes (:,:!) and match (::, ::!) operators have precedence over the OR operator.

For example, the following syntax searches for documents in which Google is identified as an entity or the string IBM is present:

enriched_text.entities.text:Google|IBM

The query is treated as follows:

(enriched_text.entities.text:Google) OR IBM





 , [and] 

Boolean operator for "and".

In the following example, documents in which Google and IBM both are identified as entities are returned:

enriched_text.entities.text:Google,enriched_text.entities.text:IBM

The includes (:,:!) and match (::, ::!) operators have precedence over the AND operator.

For example, the following syntax searches for documents in which Google is identified as an entity and the string IBM is present:

enriched_text.entities.text:Google,IBM

The query is treated as follows:

(enriched_text.entities.text:Google) AND IBM





 <=, >=, >, < [Numerical comparisons] 

Creates numerical comparisons of less than or equal to, greater than or equal to, greater than, and less than.

For example:

enriched_text.sentiment.document.score>0.679





 ^x [Score multiplier] 

Increases the score value of a search term.

For example:
* The query expansion list must be written in JSON.
* To disable query expansion, delete the query expansion list.
* Query expansion is performed on the query and multiple collection query methods.
* Each set of expansions is associated with a collection. When querying across [multiple collections](https://cloud.ibm.com/docs/discovery?topic=discovery-query-conceptsmultiple-collections), each collection is expanded individually.
* Query expansions are applied at query time, not during indexing, so the query expansion list can be updated without the need to re-ingest your documents.
* Do not upload or delete a query expansion list at the same time documents are being ingested into your collection. This could cause the index to be unavailable for that brief period.



To upload or delete query expansion files using the Discovery tooling, open the Search settings screen and choose Synonym. To open Search settings, click the ![Manage Data](https://cloud.ibm.com/docs-content/v1/content/ded4adc3ea0bd2a81b113c579f2b1183926da211/discovery//images/icon_yourData.png) icon, open your collection, and choose the Search settings tab.

See the [query expansion API reference](https://cloud.ibm.com/apidocs/discoveryget-the-expansion-list) for the API commands to upload and delete query expansion files.

Stopword and query expansion lists cannot contain overlapping terms. If a [stopword](https://cloud.ibm.com/docs/discovery?topic=discovery-query-conceptsstopwords) in your stopwords list is also included within your query expansion list, the query expansion does not work. For example, if on is included in your stopword list, and you specify in your query expansion list that rotfl must expand to rolling on the floor laughing, that expansion does not return the expected results.